Backup Compression

Introduction: Backup Requirements

Personal data includes family photos, documents, and work history. Business data includes financial records, customer information, intellectual property, and operational plans. Hardware failure, accidental deletion, software errors, or ransomware can make such data unavailable. Backups provide a separate recovery point and are a basic part of data protection. A backup policy can specify an RPO, the acceptable amount of data loss expressed as time, and an RTO, the required time to restore the service.

Backups have different requirements from day-to-day file storage. A backup system may cover entire disks or servers and keep multiple versions for months or years. An enterprise may copy terabytes of data every day. Without redundancy reduction, repository requirements grow with every retained version. The repository also has to keep metadata, chain dependencies, checksums, and information needed to verify a restore. Copies should also be separated from the source so that one hardware failure or attack does not remove both sets at once. Retention, isolation, and restore testing are part of the storage design.

Compression in a backup system reduces the size of data sent and stored. Deduplication removes repetitions across blocks or files. These mechanisms can be combined to limit new data in later recovery points and meet the required retention period. Data that is already compressed or encrypted usually offers less benefit from another compression pass, so a system should account for its format. The reduction ratio should be evaluated together with read cost, decompression, integrity checks, and the time needed to restore one recovery point. A repository can therefore choose different levels for active data, recent restore points, and cold archives rather than applying one setting to every workload. The choice should be validated with representative data, since text, virtual disks, databases, and encrypted media expose different redundancy patterns. Measurements should include restore throughput, repository I/O, and the final file size for each retention tier. The policy should also define copy count, retention, repository isolation, and restore-testing procedures. Frequently changing data emphasize write cost, while archives emphasize read cost and media durability. Tests should use representative data and check whether the declared RPO and RTO are met.

Compression and Deduplication in Backup Systems

Backup systems often combine two data-reduction strategies. They are distinct processes that remove redundancy at different levels.

Compression Algorithms in Backup Systems

Compression for backups has different constraints from compression for real-time network traffic. For long-retained data, the compression ratio matters, while administrators must also consider the backup window, CPU load, and restore time. Data may be compressed during a nightly window and stored for years, so the level is selected for the whole recovery lifecycle.

Backup software may offer several algorithms. The choice sets a trade-off between speed, CPU use, and repository size:

• LZ4: An algorithm with very short encoding and decoding times. It provides a moderate compression ratio with low CPU overhead, so it fits jobs where completion time is important. Its frame format can carry data size and a checksum, which helps with streaming block storage.
• Gzip/DEFLATE: DEFLATE combines LZ77-style repetition matching with Huffman coding. It provides lossless coding and a practical size-versus-time trade-off, so it is available in many tools. The search window and code tables affect the result, while the decoder reconstructs the original stream.
• Zstandard (zstd): A lossless algorithm with a broad range of levels. Lower levels target fast encoding, while higher levels increase compute cost in exchange for a smaller result. Comparisons with LZ4 or DEFLATE depend on the data and settings. Zstandard defines frames that can be processed as a stream and carry information required for decoding.
• LZMA/LZMA2: Algorithms used by tools such as 7-Zip. They can reach a high compression ratio, but encoding usually requires more time and memory. They suit cold-data archives where repository size matters more than backup creation time. Dictionary size and encoder parameters affect both the result and the memory requirement.

How Deduplication Works

Deduplication requires an index of fragment identifiers and a procedure for comparing new data with that index. The system stores unique fragments and maps that describe the fragment order needed to restore a file or recovery point. Implementations may use a fast fingerprint to find a candidate and a stronger hash to confirm it, while integrity policy can require block checks during reads.

File-Level and Block-Level Deduplication

A system may analyze an entire file or a part of a file. This leads to two main approaches:

• File-Level Deduplication (Single-Instance Storage): The system calculates a hash of the entire file and checks whether that hash is in its index. This recognizes identical files. Changing one byte changes the whole-file hash, so the modified version must be stored as a new object.
• Block-Level Deduplication: The file is split into smaller fragments, and each receives its own identifier. The system can recognize repetition within files and between similar but different files. Block size and boundary rules depend on the implementation.

Fixed-Size and Variable-Size Blocks

Block boundaries can be fixed in advance or derived from the content.

Fixed-Size Block Deduplication: The system divides a file into blocks of a constant size, for example 4 kilobytes ($4KB$). This is simple and fast. Inserting one byte at the beginning shifts later 4KB boundaries. Many blocks then receive different hashes even though much of their content is unchanged, reducing deduplication efficiency.

Variable-Size Block Deduplication (Content-Defined Chunking): The algorithm slides a window over the stream and chooses a boundary from the content. It may use a rolling fingerprint, such as a Rabin fingerprint, and a bit pattern in which the last 11 bits are zero. After a byte is inserted, earlier boundaries may shift, but the algorithm can resynchronize with the unchanged content after the modified region. Only some later chunks may then change. There is no guarantee that only the first or second block is affected, but unchanged chunks can often be reused. Smaller average chunks can increase matching opportunities while enlarging the index and reference set, so CDC parameters are selected for workload and restore time.

Order of Compression and Deduplication

The following model combines these pieces in an order used by systems that deduplicate raw blocks before compressing them. A client can send fingerprints in stages so the server does not receive blocks already present in the repository.

1. Data to be backed up is identified on the source machine.
2. The data is broken into variable-size blocks using a content-defined chunking algorithm.
3. A cryptographic hash, such as SHA-256, is calculated for each block.
4. These hashes are sent to the backup server, which checks its central index to see which blocks it has already stored. This is the deduplication step.
5. The server requests only the blocks that are new to it.
6. The source machine or server applies a lossless algorithm such as zstd or Gzip to new blocks. This is the compression step.
7. Only new blocks that are not already in the repository are sent over the network, then stored after compression.
8. The backup is then recorded as a metadata map of pointers referencing all the necessary blocks (both old and new) required to restore the data.

In this model, "deduplicate, then compress" limits compression work to new blocks. Other architectures may arrange the stages differently, but deduplicating compressed streams requires comparable representations and cannot retain every match available in raw blocks. During restore, the system reads the map, obtains the required blocks, and then decompresses them.

Relation to Backup Methods

Compression and deduplication affect how backup chains are built. They reduce transferred blocks while adding indexes, maps, and maintenance work. A chain must preserve point order and dependencies, since losing the base full backup or a required increment can prevent recovery of a selected state.

Full, Incremental, and Differential Backups

• Full Backups: A traditional full backup reads all selected data on each run. In a deduplicating system, the first full backup stores unique blocks, while later points can reuse unchanged blocks and store only new fragments. It remains a full recovery point, but its additional space depends on the changes. The word full describes the logical scope of the point, not the number of physical blocks written.
• Incremental and Differential Backups: An incremental backup stores changes since the previous point, while a differential backup stores changes since the base full backup. Block-level deduplication can also recognize unchanged fragments inside a modified file. Restoring an incremental point requires every chain element named by the metadata.

Synthetic Full Backup

A synthetic full backup is built from data already in the repository. An active full backup reads the complete scope from the source, while a synthetic full uses the previous full and the following increments.

The server merges the last full backup with blocks from later increments and writes a new full file or map, depending on the repository format. It does not need to reread the complete source, so a full transfer does not consume the production link. The operation still uses repository CPU and storage I/O, and its duration depends on the chain and consolidation method. After the synthetic point is created, later increments can use it as a new base, while earlier files remain until retention removes their dependencies. In practice, a tool can first create an increment containing the latest changes and then merge it with the full backup so that the new point represents the current source state.