Streaming Compression

Streaming and file downloading

When a file is downloaded, the receiver obtains data for storage and can start playback after it has accumulated the required portion. The application may save the complete file before use, and the player can seek freely through data that is already present. Streaming delivers data while it is being presented. A player downloads an initial buffer, starts playback, and continues receiving later segments in the background. Services such as YouTube and Netflix use this model.

This "play-as-you-go" model adds a time constraint. Over a longer interval, the data rate must match the playback rate, while the buffer can absorb short changes in throughput. A larger buffer lowers the risk of a stall but increases startup time and latency. When the buffer runs out, playback waits for more data. Compression therefore has to control both bitrate and delivery delay on networks with changing capacity. Buffer level does not replace a measurement of current throughput.

Streaming includes on-demand content and live transmissions. The first category can tolerate an initial buffer and prefetching, while the second requires a bounded delay between the source and the viewer. For live video, the delay includes capture, encoding, transport, receiver buffering, and display. The player must also handle new segments as they become available and possible gaps in their publication.

Compression efficiency, bandwidth, and latency

A streaming design combines a small bitrate with low . Stronger compression can reduce the required bitrate, but it often needs more buffering, more complex prediction, longer encoding, or frame reordering. The design must also account for device capacity, encoding cost, and short throughput losses. The balance depends on whether the content is on demand or interactive.

Inter-frame prediction and latency

High compression efficiency comes from inter-frame prediction. P-frames use reference pictures, while B-frames can use references from both earlier and later pictures. The encoder and decoder must account for coding, decoding, and display order.

P-frames (Predicted frames): These frames reference one or more earlier reference pictures and code a prediction with residual data. The decoder must have the required reference picture available to reconstruct the current picture.
B-frames (Bi-directionally predicted frames): These frames can reference pictures from the past and the future. Two references often improve the prediction and reduce residual data, but they do not guarantee the smallest size for every frame. A codec may use other reference structures as well.

A B-frame needs a reference picture from the future relative to its display time. The encoder buffers pictures and may send them in decoding order, while the decoder buffers them until display. Buffering and reordering add delay to the complete path, especially when the reference arrives after the dependent picture would otherwise be displayed.

For on-demand streaming such as Netflix, a few seconds of startup buffering can be acceptable. The encoder can use B-frames and a larger buffer to reduce bitrate. In live, real-time communication such as a Zoom call or a Twitch sports broadcast, a five-second delay makes reactions and conversation difficult. This profile limits buffering and reordering, accepting some loss of compression efficiency.

Strategy 1: configuring the codec for live streaming

Reducing latency starts with the video encoder configuration. The settings choose a balance between encoding speed, bitrate, the use of different frame types, and GOP length. Requirements differ between on-demand playback and interactive communication.

Group of Pictures (GOP)

A repeating sequence of frame types in a video stream is called a . Its structure and length affect bitrate, access points, fast seeking, and delay caused by references and buffering.

Long GOP (fewer access points, high compression): Suitable for on-demand content. An example GOP lasting 2 seconds contains one I-frame per 60 frames at 30 fps. The remaining frames can be predictive, reducing bitrate, while the player has fewer access points and may need a larger buffer. Decoding may start only after an available access point.
Short GOP (faster access, lower compression): Live broadcasts may use shorter GOPs when rapid recovery and switching matter. More frequent I-frames shorten the dependency chain and help decoding recover after an error, while increasing the average bitrate. They also simplify switching between representations.
Low-latency profiles (I-P-P-P...): Video conferencing profiles often disable B-frames. The GOP then contains I- and P-frames, and the absence of forward references limits buffering and reordering. Encoding and transport still contribute to the total delay.
I-frame only: Professional editing or selected surveillance systems may use a stream made entirely of I-frames. M-JPEG is an example of a sequence of independent JPEG images. This structure simplifies decoding and editing, but requires a high bitrate.

Strategy 2: Adaptive Bitrate Streaming (ABR)

A user's available throughput changes over time. Moving from strong Wi-Fi to a weak cellular connection or another download can reduce the available capacity. A single fixed-quality stream could then exhaust the buffer and freeze playback. The player observes download rate, segment time, and buffer level, while measurements still reflect earlier network conditions.

addresses this problem by providing multiple renditions of the same video. The renditions can differ in quality, resolution, frame rate, and bitrate, such as 480p, 720p, 1080p, and 4K. The server encodes them with compatible access points so switching can occur at segment boundaries. The player does not need to download the complete material at the highest quality.

The process includes these stages.

Segmentation: Each rendition is divided into media segments with a declared duration. HLS publishes the target duration with the #EXT-X-TARGETDURATION tag, and each playlist entry identifies a segment that the player can request independently. Their boundaries should align with points where the decoder can switch representations. Services choose segment duration according to latency, caching, and request overhead, so the value is a deployment parameter rather than a universal range.
Manifest file: The server provides a , such as an M3U8 playlist for HLS or an MPD document for DASH. The player reads renditions, segments, and their addresses from it. A manifest can also describe encryption, duration, and fragment order. A live playlist is updated as new segments become available.
Player: The player downloads the manifest, measures throughput, and observes the buffer. It also considers the device's capabilities, display resolution, and the predicted time needed for the next segment.
Segment requests: Using these measurements, the player requests the next segment from a rendition it can download without exhausting the buffer. With high throughput it might choose 1080p, and after a drop it might choose 720p. The switch normally occurs between segments to preserve the timeline.

The player changes renditions between segments. When continuous playback takes priority over quality, it selects a lower bitrate. With a stable buffer reserve it may return to a higher quality, while overly frequent changes can degrade the viewing experience. An algorithm can use a safety margin so it does not react to every short-lived drop. This reduces unnecessary quality oscillation. ABR is used for on-demand services and for selected live transmissions.

Strategy 3: low-latency protocols

TCP and head-of-line blocking

TCP provides a reliable, in-order byte stream. When a segment is lost, retransmission restores the stream, and the receiver does not deliver later bytes to the application before the missing data. This is head-of-line blocking. The mechanism suits file downloads and web pages, while a late audio retransmission in a live call may be less useful than moving to current audio. TCP protects data completeness, but loss recovery can increase waiting time and delay later data in the same stream.

UDP and real-time protocols

Real-time applications often use the with application-level mechanisms. UDP sends datagrams without delivery or ordering guarantees. Without transport-level retransmission, an application can decide whether to skip a late packet or recover it with another method. Packet loss can cause a brief artifact without stopping the entire presentation. A higher layer can add sequence numbers, a playout buffer, redundancy, or selected retransmissions.

RTP and RTCP provide media functions above UDP.

RTP (Real-time Transport Protocol): It carries audio and video data. Sequence numbers help detect loss and reorder packets, while timestamps support synchronization between streams. RTP does not guarantee complete delivery, so the application chooses how to handle loss.
RTCP (RTP Control Protocol): It operates alongside RTP and reports packet loss, the highest received sequence number, and interarrival jitter. An application can use this information to change its bitrate or buffer settings. These reports provide feedback but do not replace loss-tolerant coding. RTP and RTCP are separate message types.
SRT and WebRTC: Secure Reliable Transport uses UDP, a latency buffer, and ARQ retransmissions to recover selected packets. WebRTC uses RTP and SRTP and negotiates a path with ICE. Both approaches support low-latency configurations, but the result depends on the network, codec, buffer, and settings. Neither protocol guarantees one fixed delay for every deployment.