File System Compression

File Systems: Organizing Data on Storage

Before discussing compression, we need to establish the role of a file system. It is the operating-system layer, provided by Windows, macOS, or Linux, that describes how data is stored and retrieved. A hard drive or Solid-State Drive (SSD) stores sequences of bits and does not interpret the concepts of a "file" or a "folder" on its own.

The file system maintains metadata and block mappings, including file names, locations, sizes, permissions, and other attributes. When a document is saved, it selects available space, writes the data, and updates those structures. When a file is opened, it reads the metadata, identifies the relevant blocks, and supplies the data to the application. Applications therefore use a logical file and directory structure independently of the physical layout.

As data grows from photos and videos to work documents and operating-system files, occupied capacity becomes more important. This motivates integrating compression into the file-system layer so fewer bytes are stored without changing the interface visible to applications. The operating system can still expose logical blocks and file size while physical block mapping remains an implementation detail. The file system must retain metadata that identifies compressed and uncompressed fragments and restores their order. Compression can also reduce the number of bytes transferred between memory and storage.

Manual and Transparent Compression

Manual compression happens outside the file system. The user selects files or folders, runs a program such as WinZip or 7-Zip, or uses the built-in "Compress to ZIP file" feature, and creates one archive, for example $archive.zip$. Accessing the contents requires opening or decompressing the archive. The operation is explicit and has its own output format.

File-system compression can instead use transparent compression. The file system compresses data during writes and decompresses it during reads. Applications receive the same logical contents and do not need to know the physical block format. Depending on the system, compression may be enabled for a file system, dataset, directory, file, or mount option. This configuration scope determines which newly written data receives the setting and how existing blocks are treated.

The operation can be described as follows:

1. A user saves a document in their word processor.
2. The file system receives the data before it is physically written to storage.
3. It runs the selected compression algorithm on the relevant data fragment.
4. It writes the compressed version when it saves space, or keeps the fragment uncompressed. The user and word processor see only a successfully saved file.
5. Later, when the user double-clicks the document to open it, the file system intercepts the read request.
6. It reads the compressed data from the disk.
7. It runs a decompression algorithm for compressed fragments in memory.
8. It passes the reconstructed data to the word processor, which opens the document normally.

From the perspective of applications, the file keeps its name, logical size, and opening behavior. The difference concerns physical space on the device. The system must track which fragments are compressed and restore their order, while the effect on execution time depends on compression ratio, storage device, and workload.

How Compression Affects Performance

Compression and decompression require extra CPU operations, but their effect on elapsed time depends on the device and workload. In many tasks the limiting factor is the I/O bottleneck, where storage latency or throughput dominates processing time.

A CPU can execute billions of operations per second, while storage latency and throughput are limited. A common simplification describes an SSD as thousands of times slower and a spinning hard drive as millions of times slower than processor work, but the values depend on the operation and hardware. When a large file is opened, part of the time may therefore be spent waiting for storage.

Transparent file-system compression can use this difference when computation costs less than the saved transfer. Read and write performance may differ for the same file because decompression is often simpler than compression and data varies in its compressibility.

• When reading a file, the system fetches less data from storage and then decompresses it in memory. If the saved transfer exceeds the computation cost, waiting time decreases. On a fast SSD or with poorly compressible data, the result can be neutral or slower, so an automatic speedup should not be assumed. I/O unit size and parallel access to several fragments also matter.
• When writing a file, the CPU compresses data before it is stored. Writing fewer bytes can reduce I/O, but compression adds CPU work and may lengthen the operation on fast storage. The capacity benefit is separate from this time trade-off. Frequent small updates can also require rewriting a compression unit.

Implementations: NTFS, ZFS, and Btrfs

Transparent compression is implemented by several file systems, but the details depend on the implementation. NTFS, OpenZFS, and Btrfs differ in algorithms, compression units, write behavior, and configuration.

NTFS Compression: A Windows Mechanism

NTFS (New Technology File System) has been the primary Microsoft Windows file system for decades. It exposes file and directory compression through the system interface and the compactcommand. In the graphical interface, the feature can be enabled by selecting "Compress contents to save disk space". Support depends on volume parameters, including cluster size, and differs from compression of executable files by Compact OS. NTFS compression also has constraints for particular file attributes and volume configurations.

How it Works: NTFS uses the LZNT1 format, which belongs to the LZ77 family also used by archive formats. File data is processed in chunks. The file is divided into compression units, or blocks of 16 clusters. A cluster is the smallest allocation unit, typically 4KB. NTFS attempts to compress each unit separately and stores the result when it saves at least one cluster. Otherwise, that unit remains uncompressed. Each decision therefore applies to a complete unit of 16 clusters.

Trade-offs:

• Advantages: The feature is available through Windows and can be enabled for a file or directory. Applications continue to use the ordinary file-system interface.
• Disadvantages: The compression ratio can be modest compared with newer algorithms. Updating a small fragment may require reading the entire unit, decompressing it, applying the change, recompressing it, and writing it again. This read-modify-write cycle increases write cost. If the result no longer fits its previous allocation, the allocation may change, contributing to file fragmentation.

ZFS and Btrfs: Inline Compression

ZFS, originally developed by Sun Microsystems, and Btrfs, the B-tree file system for Linux, support transparent inline compression. Their allocation and configuration mechanisms are not identical. OpenZFS compresses dataset records, while Btrfs stores compressed file extents and can select an algorithm for a file or mount point.

How it Works: Inline compression occurs during the write, before persistent blocks are allocated. It is integrated with their Copy-on-Write (CoW) architecture. When a file changes, new compressed extents can be written elsewhere and metadata can point to the new version. This reduces some read-modify-write work, but extent count and fragmentation still depend on the data and allocation policy.

Algorithm configuration is also important. ZFS normally selects a compression algorithm per dataset, while Btrfs can configure it through a mount option or file property:

• LZ4: OpenZFS offers this fast algorithm as a common choice for datasets. It provides a moderate ratio at low computational cost. Btrfs does not use LZ4 as one of its basic compression algorithms.
• Zstandard (zstd): This algorithm offers adjustable compression levels and is available in OpenZFS and Btrfs. Higher levels usually save more space at higher CPU cost. Results depend on the data, so no single level fits every workload.
• Gzip: OpenZFS offers variants from gzip-1 for faster compression to gzip-9 for greater space savings. Btrfs offers ZLIB levels from 1 to 9, related to deflate but not exposed as a Gzip interface.
• LZO: Btrfs provides this older algorithm, which favors speed over compression ratio. It is not a standard OpenZFS algorithm.

Both implementations use heuristics or storage thresholds that can leave a poorly compressible fragment uncompressed. JPEG data is one example. The exact threshold depends on the system, record or extent size, and selected algorithm. This limits computation for data where compression does not provide the required saving.

Considerations in Practice

Compressing Already-Compressed Data

File-system compression usually has little effect on data that is already compressed. This includes most multimedia files (JPEGs, MP3s, MP4 videos) and archives (ZIPs, RARs), which have already removed much of their predictability.

Specialized codecs and formats have already removed much of the redundancy, so their contents may look random to a general-purpose algorithm such as LZ4 or Gzip. Trying to compress them again can consume CPU time without saving space. In the worst case, the result becomes slightly larger because of format metadata. A file system may skip such a fragment, but the compression setting for a directory should be chosen after measuring the actual contents.

When Does File System Compression Make Sense?

File-system compression can help structured, repetitive data that has not already been compressed. Examples include:

• Text Documents and eBooks: Written language is full of repeating words and patterns.
• Program and OS Files: Executables and libraries can contain repeated fragments. Compression may reduce occupied space, but its effect on startup depends on the storage device and configuration.
• Database Files: Raw database files can be compressible. Less transfer may shorten I/O-bound queries, but CPU load and compression already performed by the database must be considered.
• Virtual Machine Disk Images: Virtual-machine disk images may contain empty or repeated blocks, so they can be suitable for compression when the image format has not already compressed them.

Transparent file-system compression primarily reduces occupied capacity and the number of transferred bytes. Enabling it for ZFS or Btrfs requires checking compression ratio, CPU cost, write latency, and the behavior of copies and snapshots. Measurements for the workload are a safer basis for configuration than a general assumption about speed. Measure reads, writes, occupied space, and behavior after changing the algorithm separately, including workloads with snapshots, backups, and frequent updates. The same checks help detect extra metadata, extent growth, or increased write amplification under sustained load and realistic concurrency on the target system. They also show whether the observed savings persist when the workload mixes reads, writes, snapshots and parallel operations.