Cloud Storage Compression

Cloud Storage

Photos, documents, and videos can be stored in a service described as the "cloud." Cloud storage means accessing resources in data centers over a network. These centers contain servers, hard drives, and solid-state drives connected by networking infrastructure. The logical object or file layer can be separate from the physical layer, where a provider uses replication, erasure coding, or media migration. A user sees an object name and metadata, while placement details remain part of the service implementation.

When you upload a file to Google Drive, Dropbox, or OneDrive, it travels over the internet to managed infrastructure. You can access the data from different devices without identifying a particular server or disk. The provider manages distribution, availability, replication, and media operations according to the service. In an object model, a key and metadata usually identify the object, while versioning can retain several states of the same object. Compression affects the data value, while indexes and metadata have their own overhead.

Distributed infrastructure has to operate at large scale. Global collections are sometimes described in zettabytes, or trillions of gigabytes. Costs include media, data centers, power, cooling, and networks. Providers therefore use size-reduction strategies such as compression and deduplication when the data type and service requirements allow them. Optimization must preserve availability and recoverability, so a reduction method is not suitable for every object. Choosing hot, cool, and archive tiers affects read cost and restore time.

Storage Space and Network Bandwidth

For a cloud provider, efficiency includes both storage space and data transfer. The first concerns media and maintenance, while the second concerns traffic between clients, services, and data centers. Operation costs, indexing, replication, and integrity checks also matter. Analysis should distinguish frequently read data from archives because latency and CPU constraints differ.

1. Compression in Cloud Systems

One way to reduce size is general-purpose lossless compression. Algorithms used in file systems and ZIP archives can also be used in cloud services when the object format and read requirements allow it. A text document or database file usually offers more redundancy than data that is already compressed. A service can compress during the write path, in an asynchronous job, or in an archive tier. Each option changes client, server, and read-path load.

Example algorithms include:

• Gzip (DEFLATE): The gzip format carries a lossless DEFLATE stream that combines repetition matching with Huffman coding. A gzip header can carry metadata, and its trailer contains a CRC32 checksum. Compression does not help every object type.
• LZ4: An algorithm designed for short encoding and decoding times. It generally gives less reduction than settings optimized for size, while keeping CPU overhead low. It fits hot data where latency matters.
• Zstandard (zstd): A lossless algorithm with levels that select a trade-off between speed and size. Comparisons with Gzip depend on the settings and data. Frame and dictionary choices can change the result for similar blocks, so parameters should be part of the storage format.

Compression analyzes the represented stream, usually within one object or block. It does not by itself recognize that many users uploaded identical data. Redundancy within a single file is handled by the compressor, while redundancy across objects can be handled by deduplication when a service provides it.

2. Deduplication in Cloud Storage

Deduplication compares fragments within a scope defined by the storage architecture. If 100 users upload the same photo of the Statue of Liberty, a system that deduplicates across accounts may store the photo data once and create references for the other 99 accounts. The actual benefit depends on sharing scope, privacy policy, and data type. A service may limit deduplication to one account, region, tenant, or repository so that object existence is not exposed to other users.

Hashing Data

For a quick preliminary comparison, the system computes a fixed-size "digital fingerprint" called a cryptographic hash. A hash reduces the number of byte-by-byte comparisons, but it does not by itself prove identity.

Algorithms such as $SHA-256$ (Secure Hash Algorithm, 256 bits) are used. For any input, from a small text file to a video, it returns a 256-bit value usually written as 64 hexadecimal characters. The same fragment supplied to $SHA-256$ produces the same result. This identifier has a very low practical collision probability, but it is not mathematically unique.

• Changing one bit usually changes many bits of the hash value, as described by the avalanche property.
• Collision resistance makes it difficult to find two different inputs with the same value, while a system can still verify a candidate match against the data.

Comparing short hash values quickly finds deduplication candidates. For data requiring integrity, the system can also verify fragment length and content. The index must also update references and delete a block only after checking that no retained object points to it.

File-Level and Block-Level Deduplication

Hashing can be applied to an entire file or to its blocks:

1. File-Level Deduplication (Single-Instance Storage): After upload, the system hashes the file and checks its index. A match can create a reference to existing data, while a new hash leads to a new stored file. This recognizes identical installers or photos. Changing one letter changes the whole-file hash, so a new version of a large document is treated as a new object.

2. Block-Level Deduplication: In this approach, a file is divided into smaller chunks and each chunk is hashed separately. A new file passes through the same partitioning, after which the system checks which blocks it already has. The scope may be one account or a larger collection, depending on the service. Fixed sizes simplify indexing, while content-defined boundaries better tolerate insertion in the middle of a file.

Consider a 500-page report ($report_v1.docx$) with one typo corrected on page 200 to create $report_v2.docx$. With a suitable partition, 99% of the blocks may remain identical. The repository can then store the new block and references to the rest. Dropbox has described a synchronization architecture that splits files into blocks and uses a hash list to request missing blocks, an example of a mechanism such as "Delta Sync." The 99% value is an assumption for the example: metadata changes or shifted boundaries can affect more blocks.

Order of Compression and Deduplication

One possible workflow combines the strategies in the following order. It applies to a system that identifies raw blocks before compressing them. The model stores hashes, an ordering map, and the algorithm used for each new block:

1. A file is uploaded to the system.
2. The file is broken into smaller blocks.
3. A hash (digital fingerprint) is calculated for each block.
4. The system checks its central hash store to see which of these hashes it has already seen. This is the deduplication step.
5. For any block that is new (its hash is not in the store), the system applies a fast lossless compression algorithm like LZ4 or zstd. This is the compression step.
6. The new, compressed block is then stored in the physical storage, and its hash is added to the central store.
7. Finally, the file in the user's account is represented by a metadata map that is simply a list of pointers to all the required blocks (both old and new).

In this model, hashing raw blocks before compression enables reuse of identical fragments. Compression changes the byte representation, so hashing only after compression requires the same format, level, and context for comparisons. Other architectures may arrange the stages differently. After storing a block, the system has to retain its hash, length, compression parameters, and relation to the file map so a read can rebuild the order.

Client-Side Security and Optimization

Client-Side and Server-Side Deduplication

Deduplication can happen in two places:

• Server-Side: The user uploads the complete file, and the server chunks, hashes, and compares it. This keeps the client simple, but data that will later be discarded has already crossed the network.
• Client-Side: The application chunks and hashes the file before uploading it. The client sends the list, and the server identifies missing blocks. Dropbox has described a model using 4 MB blocks and requesting blocks from a hash list. The benefit depends on computation cost and index scope. The client must protect the hash list and handle a file changing during the scan so the map matches the blocks actually sent.

Security and Privacy

Deduplication can reveal whether a fragment is already present. If an attacker can predict the content and observe a service response, they may try to confirm its presence from a hash or protocol behavior. Confirmation alone does not grant access to the data, but it is a side channel. The risk is higher for a small space of likely values, such as a known document or a predictable file format.

Protections depend on the trust model and may include:

• Deduplication per User: An index can be limited to one account or a trusted group. This limits cross-user information leakage at the cost of lower space savings.
• Encryption: Data is often encrypted in transit and at rest. Client-side encryption runs before upload, but two users with the same file will usually produce different ciphertexts, preventing cross-user deduplication. Convergent encryption retains deduplication, but it does not by itself remove confirmation-attack risk. Key-server schemes such as DupLESS address that risk with a different key model. The choice to share an index must therefore be tied to keys, read authorization, and deletion rules.

A service must therefore balance size, traffic, computation time, recoverability, and the information exposed by its protocol. Compression and deduplication are parts of that design choice, and their behavior depends on data format and security model. Configuration should be measured on representative files, with separate checks for upload, repeated upload, random access, deletion, and full restore. A favorable storage ratio is insufficient if the index becomes a bottleneck or if a recovery path cannot verify every referenced block. The same policy can also behave differently across tenants because their data have different repetition and encryption patterns. Operational limits, retention windows, and legal isolation can further restrict which objects are eligible for shared storage. These constraints belong in the design before a shared index is enabled. Recovery tests should confirm both the data path and the metadata path under the intended access controls, including deletion and recovery after a partial failure. Measurements should also cover access latency, index cost, and deletion propagation across replicas. An archive tier can lower media cost while increasing restore time. Retention and legal isolation may limit cross-tenant sharing.