What Is an IPsec VPN? How It Works and When to Use It

FundamentalsWhat IPsec actually protects

IPsec is not one protocol. It is a set of standards from the IETF that add three guarantees to IP traffic. Because it sits at the network layer, everything above it, from a database replication stream to a VoIP call, is covered without changing the application.

C
ConfidentialityNobody on the path can read the payload. ESP encrypts it with AES-GCM or AES-CBC.
I
IntegrityNobody can alter a packet in transit without the receiver noticing. Every packet carries an integrity check value.
A
AuthenticationEach end proves who it is before any data flows, with a pre-shared key or a certificate.

The word "VPN" in IPsec VPN just means the tunnel is used to join two private networks, or one remote host to a private network, across a public one. The public network is almost always the internet, but the same design runs over MPLS or a partner link.

FIG. 01 · SITE-TO-SITE TUNNEL

Figure 1. In tunnel mode the gateway hides the original packet, addresses included, behind a new IP header. The dots are encrypted packets in flight; the lower row shows the wrap order. Hover any node.

Packet anatomyAnatomy of an ESP packet

Every protected packet carries the same envelope. Two fields travel unencrypted on purpose: the SPI tells the receiver which key to use, and the sequence number lets it drop replayed packets without spending CPU on decryption.

081624

Security Parameter Index (SPI), 32 bits

Sequence number, 32 bits (anti-replay)

Payload data (variable): the original packet, encrypted

Padding (0 to 255 bytes)

Pad length

Next header

Integrity Check Value (ICV), variable: the authentication tag

Figure 2. The ESP packet from RFC 4303. Blue rows are encrypted, the white rows are sent in the clear so the receiver can find the right security association and reject replays before decrypting anything. Hover a field.

The handshakeHow an IPsec tunnel comes up

Before any user traffic flows, the two gateways negotiate. IKE (Internet Key Exchange) does that work in two phases, and almost every troubleshooting session starts by asking which phase failed.

IKE Phase 1: build the management channel
Both peers agree on encryption, hashing, and Diffie-Hellman group, then authenticate with a pre-shared key or certificates. Result: the IKE SA, a secure channel used only for negotiation.
GW A →IKE_SA_INIT: proposals, DH public value, nonce← GW BIKE_SA_INIT: chosen proposal, DH value, nonceGW A →IKE_AUTH: identity, certificate, AUTH payload← GW BIKE_AUTH: identity, AUTH, first child SA
IKE Phase 2: build the data channel
Inside the protected channel the peers negotiate the IPsec SA: which traffic to protect, which cipher, and how long keys live. Two one-way SAs are created, one per direction, each with its own SPI.
GW A →CREATE_CHILD_SA: TS 10.1.0.0/24 ↔ 10.2.0.0/24← GW BCREATE_CHILD_SA: accepted, SPI 0x4a91c2e0
ESP carries the traffic
Each packet that matches the selectors is encrypted and authenticated, wrapped in a new header, and sent. The receiving gateway looks up the SPI, checks the sequence number, verifies the integrity tag, decrypts, and forwards the inner packet into the LAN.
Rekey and dead peer detection
Keys rotate on a timer or a byte count so a captured key exposes only a slice of traffic. DPD probes tell a gateway when the other side has gone silent so it can tear down the stale SA and fail over.

The tunnel in numbers

What a healthy IKEv2 site-to-site tunnel looks like once it is up.

4messages to build the IKE SA

2one-way SAs per tunnel

1400bytes, recommended tunnel MTU

ConfigurationWhat the configuration looks like

The same tunnel, three vendors. Every version answers the same five questions: who is the peer, how do we authenticate, which subnets, which ciphers, which IKE version.

strongSwan (Linux)Cisco IOSpfSense

# /etc/swanctl/swanctl.conf, site-to-site, IKEv2, certificates
connections {
  hq-branch {
    local_addrs  = 203.0.113.1
    remote_addrs = 198.51.100.7
    local  { auth = pubkey  certs = hqCert.pem }
    remote { auth = pubkey  id = branch.example.com }
    children {
      lan {
        local_ts  = 10.1.0.0/24
        remote_ts = 10.2.0.0/24
        esp_proposals = aes256gcm16-x25519
        start_action = start
      }
    }
    version = 2
    proposals = aes256-sha256-x25519
  }
}

! IKEv2 site-to-site with a virtual tunnel interface
crypto ikev2 proposal PB-PROP
 encryption aes-gcm-256
 prf sha256
 group 19
crypto ikev2 profile PB-PROFILE
 match identity remote address 198.51.100.7 255.255.255.255
 authentication remote pre-share
 authentication local pre-share
 keyring local PB-KEYS
crypto ipsec transform-set PB-TS esp-gcm 256
 mode tunnel
interface Tunnel0
 ip address 172.16.0.1 255.255.255.252
 tunnel source 203.0.113.1
 tunnel destination 198.51.100.7
 tunnel mode ipsec ipv4
 tunnel protection ipsec profile PB-IPSEC

# VPN > IPsec > Tunnels (values as shown in the GUI)
Phase 1
  Key Exchange        IKEv2
  Remote Gateway      198.51.100.7
  Authentication      Mutual PSK
  Encryption          AES256-GCM, 128 bit
  Hash                SHA256
  DH Group            19 (nist ecp256)
Phase 2
  Mode                Tunnel IPv4
  Local Network       10.1.0.0/24
  Remote Network      10.2.0.0/24
  Protocol            ESP
  Encryption          AES256-GCM
  PFS                 19 (nist ecp256)

ComparisonIPsec VPN vs SSL VPN vs WireGuard

The three are not rivals for the same job. IPsec is the site-to-site standard that every firewall speaks. SSL VPN is the remote-access default because it passes through almost any network. WireGuard is the newest and simplest, but its key model fits fleets you control more than partners you do not.

Criterion	IPsec VPN	SSL VPN	WireGuard
OSI layer	3 (network)	4 to 7 (TLS session)	3 (network)
Ports	UDP 500, 4500; proto 50	TCP 443 (usually)	UDP 51820 (default)
Typical use	Site to site, cloud VPC links	Remote users, BYOD	Cloud, homelab, mobile
Key exchange	IKEv1 / IKEv2	TLS handshake	Noise protocol, static keys
Cipher agility	Negotiated	Negotiated	Fixed suite
Passes guest Wi-Fi	Often blocked	Almost always	Sometimes blocked
Vendor support	Every firewall and cloud	Vendor specific clients	Growing; some firewalls
Config size	Large	Medium	~10 lines

Which VPN do you actually need?

Pick the situation that matches yours.

Connect two offices Laptops on hotel Wi-Fi Servers in two clouds Partner needs one app

IPsec VPN, site to siteBoth firewalls already speak it, the tunnel stays up permanently, and nobody at either office has to run a client. Use IKEv2 with certificates if you have more than a handful of sites.

SSL VPN (or IKEv2 with a good client)Guest networks block UDP 500 and 4500 more often than you would think. TCP 443 gets through anywhere. If your firewall bundles an IKEv2 client with MOBIKE, that works too.

WireGuard, or the cloud provider’s IPsecFor a fleet you fully control, WireGuard is ten lines of config and very fast. If one side is a managed cloud gateway (AWS, Azure, GCP), use their IPsec endpoint; it is what they support.

Probably not a VPN at allZero trust network access (ZTNA) or a reverse proxy in front of that one application gives the partner exactly one door instead of a route into your network.

ModesTunnel mode vs transport mode

Tunnel mode encrypts the whole original packet and adds a new IP header, so the internal addresses never appear on the internet. It is the mode for gateways. Transport mode keeps the original header and encrypts only the payload; it is used host to host, or as the inner layer under L2TP.

FIG. 03 · TUNNEL VS TRANSPORT

Figure 3. Solid blocks are added by IPsec; dashed blocks are the original packet. Transport mode leaves the original IP header readable, which is why it cannot hide internal addresses.

TroubleshootingWhen it breaks: read the phase, not the vendor

Ninety percent of IPsec tickets fall into five buckets, and the log tells you which one before you open a single config file.

ike.log, what a failed tunnel looks like
IKE_SA_INIT✓ proposal accepted (AES-GCM-256 / SHA256 / DH19)Phase 1 crypto matches
IKE_AUTH✗ AUTHENTICATION_FAILEDFix: PSK or certificate identity mismatch. Check the remote ID string, not just the key.
CREATE_CHILD_SA● TS_UNACCEPTABLEFix: traffic selectors differ. Both sides must list the same local/remote subnets.
ESP● tunnel up, large transfers hangFix: MTU. Set tunnel MTU 1400 and clamp TCP MSS to 1360.
DPD✗ peer not responding, 3 retriesFix: UDP 4500 blocked mid-path, or the far side rebooted. Check NAT-T keepalives.

DecisionWhen to use an IPsec VPN, and when not to

Use it when you connect networks: a branch to headquarters, an on-premises firewall to an AWS or Azure VPC, or two data centers. Every mainstream firewall and every cloud provider exposes it as a first-class feature.

Skip it for individual remote users unless your firewall already bundles a good client. Hotel and airport networks routinely block UDP 500 and 4500, which SSL VPN on TCP 443 sails through. For a fleet of servers you control end to end, WireGuard needs a fraction of the configuration and CPU.

CiscoFortinetPalo AltoJuniperpfSenseMikroTikstrongSwanAWS VPNAzure VPN GatewayCloud VPN

TimelineThirty years in five stops

1995RFC 1825 defines the IP security architecture. AH and ESP are born.
1998IKEv1 (RFC 2409) standardizes key exchange. Main mode and aggressive mode appear.
2005IKEv2 (RFC 4306) fixes the mess: fewer messages, NAT-T built in, dead peer detection.
2014RFC 7296 consolidates IKEv2; MOBIKE makes roaming clients practical.
TodayAES-GCM and elliptic-curve groups are the default; every cloud sells a managed IPsec endpoint.

Tunnel mode with NAT traversal. The original packet becomes the payload of a new one, and only the outer header stays readable.

ComparisonIPsec, an SSL VPN and WireGuard on the criteria that decide it

Criterion	IPsec with IKEv2	SSL or TLS VPN	WireGuard
What it carries	Whole IP packets, and every protocol above them	Whatever the client proxies, in practice TCP applications or a browser session	Whole IP packets, and every protocol above them
Ports on the wire	UDP 500 for IKE, UDP 4500 once a NAT is in the path, otherwise ESP as IP protocol 50	TCP 443, the one port open on every network you will ever sit on	UDP on a single port you choose, 51820 by default
Getting out of a hotel network	Often blocked, because ESP is neither TCP nor UDP and the gear in front of you drops it	Almost always passes, because it looks exactly like HTTPS	Passes wherever outbound UDP is allowed, fails where only TCP is
Cipher negotiation	Negotiated per tunnel, which is why two vendors can disagree for an afternoon	Negotiated in the handshake, though TLS 1.3 cut the list down	Nothing to negotiate. One fixed suite: ChaCha20, Poly1305, Curve25519, BLAKE2
Two different vendors at each end	The only one where a firewall from one vendor reliably talks to a firewall from another	In practice the vendor client talks to the vendor gateway	The same implementation at both ends
Client moves to a new address	MOBIKE, RFC 4555, keeps the association alive across the change when both ends support it	A new connection opens and the session usually resumes	The peer updates its endpoint from the first authenticated packet that arrives from the new address
How narrow you can make access	Traffic selectors between networks, so everything inside the tunnel is reachable	Per user and per application, which is the answer an auditor is looking for	Allowed addresses per peer, explicit but coarse
What runs on the client	Already built into Windows, macOS, iOS and Android, so there is nothing to install	A vendor client, or nothing at all for browser based access	A small client everywhere, and in the Linux kernel since 5.6 in March 2020
Where it hurts	MTU and fragmentation, and phase 2 selectors that do not match	Throughput, because a TCP tunnel carrying TCP stalls twice under loss	No address assignment, user directory or group policy of its own
Reach for it when	Two offices, or an office and a cloud network, need to behave like one network	A handful of remote people need a handful of internal applications	You control both ends and want a link that comes back instantly

FAQFrequently asked questions

Is IPsec the same as a VPN?

No. VPN is the goal, a private path over a public network. IPsec is one way to build it. SSL/TLS, WireGuard, and older protocols like PPTP are other ways.

What ports does an IPsec VPN use?

IKE negotiates on UDP 500. When either side is behind NAT, both IKE and the encrypted data move to UDP 4500 (NAT-T). Without NAT, ESP travels as IP protocol 50, which has no port at all.

Which is more secure, IPsec or SSL VPN?

Configured well, both are secure. IPsec protects everything at the IP layer; SSL VPN is easier to expose accidentally because it lives on TCP 443 next to web traffic.

What is the difference between IKEv1 and IKEv2?

IKEv2 is faster to negotiate, has built-in NAT traversal and dead peer detection, supports MOBIKE for roaming clients, and is the version you should deploy today.

Does IPsec slow down the connection?

Encryption adds a few percent of overhead. On modern firewalls with AES-NI the tunnel usually saturates the WAN link before the CPU becomes a limit. MTU is the more common cause of slowness.

Can I run IPsec through a NAT router?

Yes, with NAT-T. The gateways detect the NAT during IKE and encapsulate ESP inside UDP 4500 so the router can track it like any other UDP flow.

Also worth reading

One packet a weekA short, illustrated explainer every Tuesday. No vendor pitches, unsubscribe in one click.

What Is an IPsec VPN? How It Works, IKE Phases, and When to Use It