What is IPSec?

Laptop in darkness with half-closed lid
(Image credit: Unsplash / Christian Wiediger)

Internet Protocol Security (IPSec) is simply a group of protocols used to send and receive encrypted data between devices. This is very useful when sending information over the internet, so many of the best VPN providers use IPSec to protect your data by establishing an encrypted ‘tunnel’ on their virtual private network

Terrific tunneling 

Every VPN user connects to a VPN server via a special client. If your VPN provider uses IPSec, the secure connection is established in 6 stages. 

1. Key exchange

Before encryption can happen, the VPN client and server will use SA (Security Exchange) protocols to securely exchange a key that will be used to encrypt and decrypt data. Using public keys means it’s easy for the devices to verify that they’re genuinely communicating with each other. 

This is also where the devices will choose which encryption algorithm to use throughout the connection. IPSec supports AES, which is generally considered the most secure form of encryption available. 

2. Packet headers and trailers 

As we’ve already learned IPSec ensures the integrity of data when establishing a VPN tunnel to prevent others from manipulating it. IPSec also adds information to data packets it sends. 

In addition to the ‘payload’ in each packet (i.e. the data itself). It adds ‘headers’ and ‘trailers’ before and after each packet to ensure that the data has been correctly received. 

3. Authentication 

IP Sec also makes use of “Authentication Headers” in data packets. This simply ensures that any data sent across your VPN tunnel can be verified to have come from your authorized devices. This protects you from a “Man in the Middle” or “Replay Attack”, which would allow someone to intercept the connection between you and your VPN. 

4. Encryption 

If you’ve established an IPSec tunnel, at this stage IPSec will also encrypt data packets. It does this using an Encapsulating Security Protocol (ESP) that includes the entire ‘payload’ and IP headers.

This makes it almost impossible for hackers monitoring your connection to read personal data sent and received from your VPN provider.

5. Transmission 

Your VPN client can now communicate with the server using a transport protocol. Most commonly IPSec Tunnels use User Datagram Protocol (UDP) as it’s simple and fast to establish between two devices given that it doesn’t need to set up a dedicated connection.  

6. Decryption 

Whether it’s on your VPN client or server end, data exiting the tunnel is decrypted and your devices can use the data. Naturally, this happens transparently for you, so you simply see the data you requested, such as a website, appear on your device. 

Incredible IPSec 

As one of the oldest and most widely-used sets of protocols, IPSec is widely supported. It operates in the network layer, so it’s compatible with most applications. In plain English this means that you can use it transparently without having to configure each program. This gives it an edge over VPNs using SSL, for instance, as these need to be configured manually to work with individual applications. 

If your VPN provider connects in this way, your device will almost certainly support it. Due to its interoperability, IPSec tunneling is a popular choice when it comes to operating site-to-site VPNs. You can read more about these here.

Given that all data is encrypted, if your VPN client uses IPSec your ISP will be unable to monitor your personal data or throttle your connection depending on the type of content you’re using, such as streaming video.

Of course there are other VPN protocols that encrypt your data but IPSec also uses public/private key authentication to make sure that none of your data packets have been intercepted. This stops bad actors from impersonating your device or VPN provider and sending you false data.

IPSec inconveniences 

The process of encrypting each packet of data, as well as adding ‘headers’ and ‘tailers’ to the payload takes a toll on your machine’s CPU relative to other VPN protocols. If you’re using a mobile device or older machine, you may want to consider using a different protocol. You also may want to consider using a different protocol if you only plan to use a VPN for small amounts of data, as the toll it takes on your system isn’t worth it. 

As IPSec wasn’t originally designed for use specifically with a VPN, it doesn’t have some of the more advanced features of more modern protocols such as bypassing VPN blockers or firewalls. Using UDP means IPSec can usually connect through firewalls in any case. 

It can also be difficult to connect to other networks once you’re connected to an IPSec VPN, though this shouldn’t be an issue for most users. 

In 2013, whistleblower Edward Snowden alleged that his former employer, the National Security Agency had worked to deliberately weaken the security offered by IPSec. This supposedly was done by inserting deliberate vulnerabilities into the open-source code of the operating system OpenBSD and its cryptographic framework. Although the NSA certainly compromised certain Cisco firewalls that made use of IPSec, the way that they did this is still not clear. 

Bad actors are far more likely to use ‘zero day’ exploits on software and devices that haven’t been updated to compromise your data than theoretical vulnerabilities in IPSec’s source code. The best solution to this is to regularly update your software and hardware firmware.

IPSec in summary 

Now that you have a clear understanding of how IPSec tunnels secure your VPN data, you can decide if this is the right choice for you. 

IPSec certainly goes to some lengths to protect every bit of data communicating between your device and VPN server using powerful AES-256 bit encryption, even if it does so in quite a resource-intensive way. Take some time to decide if this is what you need. 

Whilst some VPN gateways using IPSec may be vulnerable to hackers and government agencies, any server can be targeted no matter how secure it is. Research both paid-for and free VPN providers to discover if they use IPSec and what other precautions they take to keep your data and theirs safe. 

We’ve also outlined other options in our guide What is a VPN protocol? - so you can see if one better matches your needs. 

Nate Drake is a tech journalist specializing in cybersecurity and retro tech. He broke out from his cubicle at Apple 6 years ago and now spends his days sipping Earl Grey tea & writing elegant copy.