What is Encapsulating Security Payload
One of the core protocols in the IP Security (IPsec) system is the IPsec Encapsulating Security Payload (ESP). ESP is one of the two protocols that the IETF has standardized for usage in IPsec, along with the Authentication Header (AH). The IP protocol number 50 designates ESP, which functions at the Network Layer (Layer 3).
As the foundation for the majority of Virtual Private Network (VPN) systems, ESP is frequently used to create secure communication channels across untrusted networks.
Core Security Functions of ESP
ESP is critical because it offers a comprehensive suite of security services:
- Confidentiality (Encryption): The main characteristic that sets ESP apart is its confidentiality (encryption). To stop unauthorized parties from reading private information, ESP encrypts the IP packet’s payload or user data. DES, 3DES, and Advanced Encryption Standard (AES) are the most often used encryption algorithms.
- Data Integrity: ESP makes sure that no changes were made to the received data while it was in transit.
- Authentication: Data origin authentication is provided by ESP. This confirms the sender’s identity and guarantees the packet originates from a legitimate source. ESP frequently makes use of hash functions like HMAC-SHA or HMAC-MD5 for providing authentication.
- Anti-Replay Protection: By employing a Sequence Number field, ESP helps stop attackers from intercepting and retransmitting legitimate packets at a later time.
- Traffic-Flow Confidentiality: The flow of traffic Limited traffic-flow confidentiality is provided by ESP, and it works best when tunnel mode is employed and set up at a security gateway.
At least one service (authentication or secrecy) needs to be chosen for ESP to work. Because encryption alone may expose z/OS IP security vulnerabilities, authentication is necessary.
You can also read Diffie Hellman Key Exchange Algorithm, Diffie Hellman Groups
ESP Structure and Components
ESP functions by adding a header and a trailer to the IP packet, creating a secure envelope around the original payload data.
The main components added by ESP are:
| Component | Location | Purpose |
|---|---|---|
| ESP Header | Before the encrypted payload | Contains the SPI and Sequence Number. |
| Encrypted Payload | The original data | The transport-layer segment or entire original IP packet that is protected by encryption. |
| ESP Trailer | After the encrypted payload | Contains Padding and the Next Header field. |
| Authentication Data (ICV) | At the very end | An Integrity Check Value (ICV) that provides data integrity and authentication. This field is variable in size and optional, used only if authentication is selected for the Security Association (SA). |
Key fields within the ESP format include:
- Security Parameters Index (SPI): A 32-bit mandatory field that uniquely identifies the specific security association (SA).
- Sequence Number: A 32-bit monotonically increasing counter value that provides the anti-replay function.
- Padding: Bytes (0–255 bytes) used to ensure the payload data fits the required cipher block size for encryption.
- Next Header (8 bits): Identifies the type of data contained in the payload data field.
How ESP Works
ESP operates directly on top of the Internet Protocol (IP), identified by IP Protocol Number 50. It functions by adding specific fields to an IP packet to create a secure envelope around the original data:
| Component | Location | Purpose |
| ESP Header | Before the encrypted payload | Contains the Security Parameter Index (SPI) to identify the security association (SA), and the Sequence Number for anti-replay. |
| Encrypted Payload | The original data | The encrypted transport-layer segment (TCP/UDP) or the entire original IP packet. |
| ESP Trailer | After the encrypted payload | Contains Padding (for block ciphers) and the Next Header field to identify the type of content in the payload. |
| Authentication Data (ICV) | At the very end | An Integrity Check Value (ICV) that provides data integrity and authentication. |
ESP Modes of Operation
The amount of the original IP packet that is protected by ESP is determined by its implementation in one of two primary modes:
- Transport Mode:
- Encrypts and authenticates only the payload data (e.g., the TCP/UDP segment).
- The original IP header remains intact and unencrypted, allowing network devices to read source and destination addresses.
- Typically used for end-to-end communication between two hosts running IPsec.
- Tunnel Mode (Most Common in VPNs):
- Encrypts and authenticates the entire original IP packet (header + payload).
- It then encapsulates this protected packet inside a new IP packet with a new, unencrypted outer IP header (often with the addresses of the VPN gateways).
- Used to create secure VPN tunnels between security gateways or between a host and a gateway (client-to-site VPNs).
Comparison and Combination with AH
Authentication Header (AH) is the other main IPsec protocol. One significant distinction is that ESP offers both encryption and authentication, whereas AH just offers authentication. ESP is the main technique utilized by IPsec since it can manage all IPsec capabilities, including confidentiality.
Although comparable techniques can be used by both protocols to achieve authentication:
- With the exception of variables like TTL that change while in transit, AH authenticates IP headers and their payloads.
- Only the IP datagram section of the IP packet is authenticated by the ESP authentication technique. ESP does not guarantee the integrity and authentication of the entire IP packet when it is in transport mode.
The same IP packet can be protected by combining ESP and AH. This combination guarantees that AH authenticates the entire IP packet and ESP encrypts the payload. IKEv2 does not, however, permit the use of both the ESP protocol for encryption and the AH protocol for authentication; if both are necessary, IKEv2 requires the use of ESP for both.
Additionally, ESP outperforms AH in terms of compatibility with Network Address Translation (NAT) devices. ESP can pass through NAT-enabled network infrastructure with ease by being contained in User Datagram Protocol (UDP) packets. A UDP header will cover the ESP header if Network Address Translation-Traversal (NAT-T) is enabled.
You can also read Cisco Dynamic Multipoint VPN DMVPN Phase 1 2 3 In Network