TCP Security & Hardening

TCP Security: Scope and Limitations

The Transmission Control Protocol (TCP) provides a reliable, ordered byte stream over an IP network. It assigns sequence numbers, uses acknowledgments, checksums, retransmission, and connection state to recover from ordinary loss and reordering. These mechanisms address transport reliability. They do not authenticate the endpoints, encrypt payloads, or establish that a segment came from the host named by its source address.

TCP was specified for an environment in which hosts and paths were expected to cooperate. Its designers concentrated on packet loss, ordering, and congestion. A malicious sender can therefore exploit state creation during the handshake, guess or observe sequence information, or inject control segments with forged addresses. The protocol's state machine is useful for reliability, but it is not an authentication protocol.

TCP security analysis starts with those mechanisms and identifies the assumptions they make. SYN flooding, sequence-number prediction, and RST injection expose different weaknesses, while current implementations use countermeasures for specific cases. Encryption and endpoint authentication require additional protocols and remain outside TCP itself.

SYN Flood: Exhausting Pending-Connection State

The SYN flood is a classic Denial-of-Service (DoS) attack that directly exploits the mechanics of the TCP three-way handshake.

Reviewing the Handshake's Vulnerability

To understand the attack, consider the server's behavior during an ordinary handshake.

1. A client sends a SYN packet to the server.
2. The server receives the SYN. A typical implementation allocates resources for pending state, including a Transmission Control Block (TCB) with the peer addresses, ports, sequence information, and timer data. It places the connection in the 'SYN_RCVD' state. The implementation also limits how many such entries may wait in its listen backlog.
3. The server sends a SYN-ACK packet back to the client.
4. The server waits for the final ACK from the client to complete the handshake and move the connection to the 'ESTABLISHED' state.

The exposure lies in Step 2. The server commits memory and state before it has evidence that the sender can receive the SYN-ACK and complete the handshake. The pending entry then remains until the final ACK arrives or a retransmission timer removes it.

Executing the SYN Flood Attack

An attacker can exploit this resource relationship with a high-rate stream of incomplete handshakes.

1. The attacker uses one or more machines to send a large number of SYN packets to the target server.
2. In a common variant, these SYN packets contain a spoofed (fake) source IP address. The attacker cannot receive replies sent to those addresses, so the final ACKs do not return. A SYN flood can also use reachable sources; source spoofing is not required for the resource-exhaustion mechanism.
3. The target server receives the SYN stream. For each accepted request, it records a half-open connection in the pending state and sends a SYN-ACK to the claimed source address.
4. For spoofed sources, these SYN-ACK packets travel across the internet to nonexistent or non-responsive machines, so the final ACK does not return to the server.

The result is resource pressure.The server may hold many half-open connections in 'SYN_RCVD'. Each entry consumes implementation resources and a position in a finite pending-connection queue. If the queue or related limits are reached, new SYNs can be delayed or dropped, including requests from legitimate clients. The service may then become unavailable or show a high connection-failure rate. The outcome depends on implementation limits and on the attack rate.

The Defense: SYN Cookies

A server can reduce this exposure with SYN cookies. The mechanism replies to an initial SYN without retaining the usual SYN-RECEIVED state. The final ACK carries information that lets the server reconstruct and validate the request before allocating the full connection state.

How SYN Cookies Work

1. Receiving the SYN: When a server uses SYN cookies and receives a SYN packet, it does not retain the ordinary pending TCB for that request. It can still use small, implementation-specific counters or policy state.
2. Crafting the Cookie:The server sends a calculated Initial Sequence Number (ISN) that encodes enough information to validate the request later. A common construction combines a keyed function of connection parameters with a time-dependent component; inputs can include the client's ISN and an encoded MSS:
   • The client's source IP address and port number.
   • The server's destination IP address and port number.
   • A time value and a secret known only to the server.
3. Sending the SYN-ACK: The server sends a standard SYN-ACK with the calculated ISN and does not retain the normal per-connection state. Exact cookie encoding is implementation-dependent, and some TCP options or data carried in the initial SYN may be unavailable or handled with reduced fidelity while cookies are active.

The Two Outcomes

• If the sender used a spoofed address: The SYN-ACK is sent to the claimed address, and the final ACK normally does not return. The server has not retained the usual pending TCB, so this request does not consume that per-connection state.
• If the sender can receive the reply:The client receives the SYN-ACK and sends the final ACK. Its acknowledgment number is the server's ISN, including the cookie value, plus one. That ACK supplies the value needed for validation.

Verification and Connection Establishment

When the server receives the final ACK, it subtracts one from the acknowledgment number to recover the cookie value and validates it against the peer addresses, ports, time value, secret, and any encoded option information. A valid value shows that the sender received the SYN-ACK or otherwise obtained its sequence number. Only after this check does the server reconstruct the pending state and establish the connection.

SYN cookies reduce exhaustion of pending per-connection state, but they do not remove CPU, bandwidth, backlog, or application-level limits. They may also limit TCP options, so implementations commonly enable them under configured pressure or attack conditions.

Sequence Number Prediction and Session Hijacking

Another major class of TCP attacks involves predicting the sequence numbers that TCP uses to order and acknowledge data. If an attacker can successfully guess the next sequence number in a connection, they can inject malicious data into the conversation. This is known as TCP Session Hijacking.

The Vulnerability: Predictable ISNs

TCP's segment acceptance rules assume that an off-path attacker does not know the current sequence numbers for an established connection. The connection itself is identified by the 4-tuple of addresses and ports. Sequence-number checks therefore provide limited origin validation for segments, but they do not authenticate an endpoint against an on-path observer.

In early TCP implementations, Initial Sequence Numbers (ISNs) could follow predictable patterns such as a global counter. Current guidance uses a clock-driven component combined with a keyed pseudorandom function of the connection tuple and a secret. This makes guessing difficult for an off-path attacker, while observation or traffic injection on the path remains a different threat.

The Hijacking Attack in Action

1. A legitimate client establishes an authenticated connection with a server (e.g., a Telnet session after logging in).
2. An attacker, who wants to take over this session, first observes the traffic to learn the IP addresses and port numbers.
3. The attacker then launches a brief DoS attack against the client to temporarily silence it. This is important to prevent the real client's machine from sending RST packets that would kill the hijacked session.
4. The attacker now needs to inject data. They predict the next sequence number the server is expecting from the client.
5. The attacker crafts a malicious packet. This packet has the client's spoofed source IP, the correct ports, and the predicted sequence number. The payload of this packet contains an unauthorized application command, for example a command that changes server data.
6. If the prediction is correct, the server receives the forged packet. If the 4-tuple, sequence number, and application protocol state all appear valid, the server may accept the payload. The session has then been influenced by the forged segment.

The Defense: Randomized ISNs

The primary defense against sequence-number prediction is an ISN algorithm that is difficult to predict. Current TCP guidance combines a clock-based value with a keyed pseudorandom function of the local and remote addresses, ports, and a secret. This substantially lowers the chance that an off-path attacker guesses the expected sequence number. It does not protect traffic from an on-path attacker, and it does not replace encryption or endpoint authentication.

RST Segment Injection

A forged segment with the RST flag set can terminate an existing TCP connection. The attacker must reproduce the IP addresses, ports, and a sequence number accepted by the endpoint.

Sequence-Number Validation

Legacy TCP behavior allowed an RST whose sequence number fell within the receive window to be accepted. RFC 5961 tightens this check. An RST is accepted immediately only when its sequence number equals RCV.NXT. An in-window segment that does not meet this condition causes a challenge ACK to be sent and is discarded.

Conditions for an RST Injection Attack

1. The attacker identifies the target connection by its source and destination IP addresses and ports.
2. The attacker prepares an RST segment whose source address matches the other endpoint.
3. An off-path attacker must guess the sequence number. RFC 5961 validation substantially reduces the probability of success for attempts that scan the receive window.
4. An on-path attacker can observe the connection parameters and select a valid sequence number, so challenge ACK does not remove this threat.
5. Once the RST is accepted, the endpoint terminates the connection.

This attack can be used to disrupt long-running connections like database transfers, large downloads, or persistent connections used by routing protocols like BGP.

Protection Against RST Injection

Exact sequence-number validation and the challenge ACK mechanism described in RFC 5961 provide the primary protection. Source-address validation and monitoring for unusual RST bursts can limit off-path attacks. TLS does not prevent RST injection because TCP processes RST below the TLS layer and an RST is not an encrypted TLS record. Applications that require TCP-segment authentication can consider TCP-AO, while IPsec can protect traffic at the network layer.

Conclusion: A Layered Approach to Security

TCP supplies transport reliability, not a complete security boundary. SYN floods, session hijacking, and RST injection show how state creation, sequence validation, and control segments can be abused when an attacker can send forged or observed traffic.

Countermeasures such as SYN cookies, less predictable ISNs, and stricter RST validation reduce specific risks. They do not cover every attack or every resource limit. Network security therefore uses several layers with separate responsibilities, including transport controls, cryptographic protocols, host configuration, and monitoring.

TLS can provide confidentiality and endpoint authentication above TCP, while firewalls and intrusion-detection systems can enforce policy and report traffic patterns. TCP remains the reliable transport layer; the required security properties must be supplied by the surrounding protocols, hosts, and network controls.