IPsec achieves confidentiality by encrypting IP packet data using a component called the Encapsulating Security Payload (ESP). ESP wraps the contents of each packet in strong symmetric encryption, making the data unreadable to anyone intercepting it in transit. But encryption alone isn’t the full picture. IPsec also includes a key exchange protocol that lets two devices agree on a shared secret without ever sending it over the network, plus a flexible framework for choosing which encryption algorithm to use.
ESP: The Core of IPsec Encryption
The Encapsulating Security Payload is the IPsec protocol responsible for confidentiality. When a packet is sent through an IPsec connection, ESP is inserted into the packet structure between the IP header and the actual data. It then encrypts the payload data and an ESP trailer using whatever encryption algorithm and key the two communicating devices have agreed upon.
Certain parts of the ESP packet are deliberately left unencrypted. The Security Parameter Index (SPI) and Sequence Number fields, which sit at the front of the ESP header, remain in plaintext so the receiving device can identify which security agreement the packet belongs to and check it against replay attacks. These fields are instead protected by integrity checks rather than encryption. The actual content you care about keeping private, the payload, is fully encrypted.
Tunnel Mode vs. Transport Mode
IPsec can encrypt packets in two different ways, and the choice affects how much of the original packet stays hidden.
In transport mode, ESP encrypts only the data portion of the packet. The original IP header, which contains the source and destination addresses, remains visible. This mode is typically used for direct communication between two devices on the same network, where hiding the addresses isn’t necessary.
In tunnel mode, the entire original IP packet, headers and all, gets encrypted and then wrapped inside a completely new IP packet with new headers. This means an eavesdropper can’t see the original source or destination addresses. Tunnel mode is the standard choice for site-to-site VPNs, where traffic passes between two gateways and the internal network addresses need to stay hidden. It also works through Network Address Translation (NAT), which transport mode can struggle with.
How Two Devices Agree on a Key
Encryption is only useful if both sides share a secret key, and sending that key across the internet in plaintext would defeat the purpose. IPsec solves this with the Internet Key Exchange protocol, version 2 (IKEv2), which establishes shared keys through a mathematical technique called a Diffie-Hellman exchange.
The process happens in two phases. In the first phase (called IKE_SA_INIT), the two devices negotiate which cryptographic algorithms they’ll use, exchange random values called nonces, and perform the Diffie-Hellman exchange. This exchange lets both sides independently calculate the same shared secret without ever transmitting it. From that shared secret, both devices derive a master value called SKEYSEED, which then generates all the individual keys needed: separate keys for encrypting data, verifying integrity, and authenticating identities.
In the second phase (IKE_AUTH), the devices authenticate each other using certificates or pre-shared credentials. This phase is already encrypted using the keys from the first phase, so the identities of both parties are hidden from eavesdroppers. By the end of these two exchanges, both devices have matching sets of encryption and integrity keys without those keys ever having crossed the network.
Encryption Algorithms Used by IPsec
IPsec doesn’t rely on a single encryption algorithm. Instead, it’s a framework that supports multiple algorithms, letting administrators choose based on security requirements and performance needs.
The current standard is AES (Advanced Encryption Standard), which NIST recommends in several modes: AES-GCM, AES-CTR, AES-CBC, and AES-CCM. Key sizes of 128, 192, or 256 bits are all approved. NIST’s guidance for IPsec VPNs specifically recommends AES-GCM with either 128-bit or 256-bit keys as the preferred option, since it handles both encryption and integrity verification in a single operation. Both 128-bit and 256-bit keys currently provide strong protection, though 256-bit is recommended when CPU load isn’t a concern. When performance matters, dropping to 128-bit is a reasonable tradeoff if deployment policy allows it.
A newer option is ChaCha20-Poly1305, a stream cipher paired with an authenticator that uses a 256-bit key. It was introduced as a standby cipher for IPsec, offering strong security with good performance, particularly on devices that lack dedicated AES hardware acceleration.
Older algorithms have been phased out. DES (56-bit keys) was abandoned long ago, and Triple DES (3DES) is also deprecated. 3DES uses a 64-bit block size, which creates a statistical vulnerability: ciphertext collisions become likely after roughly 4 billion blocks are encrypted with the same key. NIST has urged all users to migrate from 3DES to AES and plans to disallow 3DES in IPsec entirely.
Authenticated Encryption: Confidentiality Plus Integrity
Modern IPsec implementations don’t treat encryption and integrity as separate steps. Instead, they use Authenticated Encryption with Associated Data (AEAD), which combines both into a single cryptographic operation. AES-GCM and AES-CCM are the two primary AEAD algorithms used with IPsec.
In an AEAD operation, the algorithm takes a secret key, a nonce (a one-time-use value), the plaintext data, and associated data that needs integrity protection but not encryption (like the ESP header fields). It produces ciphertext that includes a built-in integrity check. When the receiving device decrypts the data, it automatically verifies that nothing was tampered with during transit. If an attacker modifies even a single bit of the encrypted packet, the integrity check fails and the packet is discarded. This matters for confidentiality because encryption without integrity can be vulnerable to subtle attacks where an adversary manipulates ciphertext to extract information about the plaintext.
Hardware Acceleration and Performance
Encrypting every packet adds computational overhead, which historically made IPsec slower than unencrypted connections. Modern processors largely eliminate this concern through dedicated hardware instructions for AES, commonly called AES-NI.
Intel introduced AES-NI in 2010, and the instructions are now standard across most server and desktop processors from both Intel and AMD. These hardware instructions accelerate AES operations by 3 to 10 times compared to purely software-based encryption. For parallelizable modes like CTR (used in AES-GCM), the improvement can reach 10x, processing data at roughly 1.3 CPU cycles per byte on optimized implementations. Even non-parallel modes like CBC encryption see a 2 to 3 times speedup.
This hardware support is one reason AES-GCM became the recommended default for IPsec. On any modern server or network appliance with AES-NI, the performance cost of full packet encryption is negligible for most workloads. ChaCha20-Poly1305, meanwhile, fills the gap on embedded or mobile devices that may lack AES-specific hardware, since it achieves strong performance through software alone.