The Internet and IP Traffic Protection
The internet carries packets between networks through infrastructure that the communicating parties do not fully control. A party with access to part of the path may observe traffic metadata and, without additional protection, may try to read or modify the transmitted data.
Sensitive company documents, financial data, and medical information require confidentiality, integrity, and peer authentication. A VPN provides a logical protection channel over an existing public network. IPsec is one protocol suite used for this purpose. Depending on the deployment, a VPN connects a remote user to an organization or connects two local networks through gateways. The protection scope follows from tunnel endpoints, selected algorithms, and security policy rules. A VPN does not remove all traffic metadata, such as gateway addresses or communication times.
What is a Virtual Private Network (VPN)?
A creates a logical protection channel for data sent through a public network. The name points to two properties:
- Virtual:The network is "virtual" because it doesn't rely on its own dedicated physical wires. Instead, it uses the existing public internet infrastructure. It creates a logical, software-defined network on top of the physical one.
- Private: Protection comes from the selected security protocols. A configuration can encrypt data, verify its integrity, and authenticate the communicating parties. A VPN does not provide anonymity by itself.
Common uses for VPNs include an employee working from home securely accessing the company's internal network, or two company offices in different cities (e.g., Warsaw and Krakow) securely connecting their local networks together over the internet, making them behave as one single, private network. In both cases, the administrator defines which traffic may enter the tunnel, which addresses are protected, and how long keys remain valid. Protection applies to traffic selected by configuration, so correct firewall and routing rules remain necessary.
IPsec: A VPN Protocol Suite
IP traffic can be protected by a standardized set of rules and protocols. One such suite is , which stands for Internet Protocol Security.
IPsec is a suite of protocols that operates at the Network Layer, Layer 3 of the OSI model. It can protect traffic that uses the Internet Protocol, including HTTP over TCP, FTP over TCP, video transmission over UDP, and ICMP messages. Applications do not need to implement IPsec because the protection is applied below the application layer. An implementation uses a security policy to specify traffic that must be protected, rejected, or passed without IPsec. Active association parameters are kept in the Security Association Database (SAD), while traffic policy is kept in the Security Policy Database (SPD). These databases map a packet to the appropriate SA.
IPsec provides security through two primary protocols: the Authentication Header (AH) and the Encapsulating Security Payload (ESP).
IPsec Core Protocol 1: Authentication Header (AH)
The primary purpose of the protocol is to provide connectionless integrity and data origin authentication. It can also provide anti-replay protection.
- Data Integrity: AH calculates an integrity check value for the protected packet fields. The receiver verifies that value. A mismatch indicates that the data changed or that the packet failed authentication.
- Data Origin Authentication: AH lets the receiver verify that the packet came from a party holding the required authentication key.
AH does not provide confidentiality or encrypt data. Anyone who intercepts a packet protected only by AH can read its contents. The protocol provides integrity, data origin authentication, and optional anti-replay protection. Verification covers as much of the IP header as the protocol can protect, while fields changed during forwarding cannot be authenticated reliably. AH therefore accounts for mutable header fields when calculating the protected value.
IPsec Core Protocol 2: Encapsulating Security Payload (ESP)
The protocol provides confidentiality through encryption. It may also protect integrity and data origin.
ESP takes the data payload of a packet and encrypts it with an algorithm such as AES, producing ciphertext. Only the intended recipient, who has the correct key, can decrypt the data and read its contents.
In addition to encryption, ESP can provide connectionless integrity, data origin authentication, and anti-replay protection. The IPsec architecture requires implementations to support ESP, while AH support is optional. In practice, ESP often meets the protection requirements of a VPN on its own. Contemporary ESP use usually combines confidentiality with integrity and data origin authentication. An encryption-only configuration is not the preferred protection profile because the receiver would not detect a modified ciphertext.
IPsec Modes of Operation: Transport vs. Tunnel
IPsec can operate in one of two distinct modes, depending on what parts of the IP packet need to be protected. The choice of mode dictates the type of VPN being created.
Transport Mode
Transport mode is used to secure communications between two specific endpoints (hosts). In this mode, IPsec protects only the payload of the original IP packet. The original IP header remains intact and unencrypted.
With ESP, the data content is protected, while an eavesdropper can still see the original source and destination IP addresses, revealing who is talking to whom. This mode is typically used for host-to-host security within a trusted network. Transport mode keeps the original header for routing, so addresses and some metadata remain visible. It is primarily used when both hosts implement IPsec and act as the endpoints of the security associations.
Tunnel Mode
Tunnel mode protects communication between two gateways, such as routers or firewalls, or between a remote host and a gateway. IPsec takes the entire original IP packet, including its header and payload, and places it inside a new IP packet.
This new outer IP packet uses the IP addresses of the VPN gateways as its source and destination. With tunnel-mode ESP, the original packet, including its internal IP addresses, is encrypted. An eavesdropper on the internet can see that the Warsaw gateway is communicating with the Krakow gateway, but cannot read the internal addresses from the encrypted packet. This is why tunnel-mode ESP is typical for both site-to-site and remote-access VPNs. The outer header routes traffic between the tunnel endpoints, while the inner packet is processed according to the SA. Gateway addresses, packet sizes, and timing may still reveal limited information about the flow.
Protection Parameters: Security Associations (SA)
Before two devices protect traffic with AH or ESP, they must agree on the parameters. This agreement is called a . An SA specifies how traffic is processed in one direction.
The key parameters defined within an SA include:
- The security protocol to be used (AH or ESP).
- The specific encryption algorithm (e.g., AES-256) and integrity or authentication algorithm (e.g., HMAC-SHA-256).
- The secret keys used for encryption and authentication.
- The lifetime of the keys (how long they are valid before new keys must be generated).
- The operational mode (Transport or Tunnel).
An essential property of SAs is that they are unidirectional, meaning they only define the rules for traffic flowing in one direction. For a typical two-way conversation, two separate SAs are required: one for traffic from A to B, and another for traffic from B to A. The receiver selects an SA using its identifier and traffic selectors. The SA record specifies algorithms, keys, mode, and the counter used for anti-replay protection.
Parameter Negotiation: Internet Key Exchange (IKE)
Two devices must agree on SA parameters and generate shared keys across a network that an attacker may observe. IKE addresses this through algorithm negotiation, key exchange, and peer authentication.
The solution is the protocol. IKE automates SA negotiation and key management. In IKEv2, the main exchanges are IKE_SA_INIT, IKE_AUTH, and CREATE_CHILD_SA.
IKEv2: IKE_SA_INIT and IKE_AUTH
IKE_SA_INIT negotiates cryptographic algorithms, nonces, and Diffie-Hellman key-exchange material. IKE_AUTH authenticates the peers, for example with a pre-shared key or certificate, and establishes the IKE SA and the first Child SA. Further negotiations are protected by the established IKE SA. IKEv2 also supports reauthentication and rekeying. Each Child SA has its own traffic selectors and keys, while the IKE SA protects control exchanges between the peers.
IKEv2: CREATE_CHILD_SA
CREATE_CHILD_SA creates additional Child SAs or rekeys existing ones. The peers agree on the protocol, algorithms, and keys used to protect user data. In IKEv1, a similar operation was called Phase 2. In IKEv2, either peer may initiate the exchange, which runs under the protection of the IKE SA.