MTU and Fragmentation, and the Numbers Worth Memorizing

The two numbersMTU and MSS, which are not the same number

Almost every question about MTU and fragmentation starts here, because two numbers describe the same limit from two layers and confusing them is the source of most bad advice on the subject.

MTU is a property of a link. It is the largest IP packet an interface will put on the wire, measured without the Ethernet header. On Ethernet it is 1500 bytes, and it has been 1500 bytes since the 1980s.

The MSS is a property of a TCP connection. The maximum segment size is the largest payload TCP will put in one segment, and each side announces its own value in the SYN packet options when the connection opens. It is derived from the MTU by subtracting the IP and TCP headers that will be added.

MTU 1500
  minus IPv4 header    20
  minus TCP header     20
  = MSS 1460

On IPv6 the IP header is 40 bytes rather than 20, so the same 1500 byte MTU yields an MSS of 1440.

The relationship matters because they solve the problem at different points. The MSS prevents oversized packets from being created. The MTU is where an oversized packet meets a limit and something has to give. A network with the TCP MSS set correctly rarely exercises the fragmentation rules at all, which is why MSS clamping is the fix that works.

Three outcomesWhat actually happens to an oversized packet

Three outcomes, and they are not equally likely or equally visible.

It gets fragmented. On IPv4, a router that receives a packet too large for the outgoing link may split it into fragments, each with its own IP header carrying part of the payload, and forward them on. Only the final destination reassembles them.

Fragmentation works and it is expensive: more packets, more headers, and if any one fragment is lost the entire original packet is lost with it.

It gets rejected with an error. If the sender set the do not fragment bit among the IP flags, the router may not split the packet. It discards it and returns an ICMP message, type 3 code 4, naming the MTU that would have fit. The sender lowers its packet size and continues. This is PMTUD working correctly.

It disappears. The router discards the packet and the ICMP message never reaches the sender, because a firewall somewhere blocked it. The sender learns nothing, retransmits the same oversized packet, and that is discarded too.

That third case is the one worth recognizing, because the symptom is specific and misleading. Small packets fit, so the connection opens, the handshake completes and a login succeeds. Anything carrying real data hangs. A website starts loading and stops. A file share connects and a copy freezes at zero bytes.

Inside a fragment, and why the size has to be a multiple of eight

Fragmentation is an Internet Protocol function, defined in RFC 791 and handled at layer 3, so it applies to whatever transport protocol sits above it. Three fields in the IPv4 header do the work.

Identification carries one value copied into every fragment cut from the same original packet, so the destination can tell which pieces belong together. Flags holds the do not fragment bit and the more fragments bit, which is set on every fragment except the last. Fragment offset says where this fragment's data sits inside the original payload.

The offset is counted in units of 8 bytes. Every fragment except the last must therefore carry a payload size that is a multiple of 8, so a router does not split a packet into even halves. It fills the first fragment to the largest multiple of 8 that fits the link, then puts the remainder in the next one.

Reassembly happens only at the destination host, never at a router in the middle. The destination holds the fragments until the set is complete. If one of them never arrives, it discards the rest and the whole original packet is lost.

That is the reason fragmentation is fragile on a lossy path. Cutting a packet into three fragments gives three chances to lose the payload instead of one, so the effective loss rate for the data is higher than the loss rate the network reports for packets.

Only the first fragment carries the TCP or UDP header, so only the first fragment has port numbers in it. A security device that filters on ports sees the rest of the set as payload with nothing to match, which is why many of them drop fragments rather than reason about them.

PMTUDPath MTU discovery, and why it is fragile

Path MTU discovery, PMTUD, is how a sender learns the smallest MTU anywhere along a route without being told in advance.

The sender marks its packets do not fragment and sends them at its local MTU. If a link along the way is smaller, the router there discards the packet and returns an ICMP fragmentation needed message naming the size that fits. The sender caches that number for the destination and resends smaller. Repeat until the packets arrive.

PMTUD is an elegant mechanism with one dependency that fails constantly: it needs an ICMP message to travel back to the sender through every firewall on the return path. Blocking all ICMP is still a common hardening instruction, and this is what it costs.

The failure is worth stating plainly because the diagnosis is unintuitive. Nothing reports an error. No log line says the packet was too big. The connection simply stops working once the data gets large, and it works perfectly for anything small enough. A tunnel that comes up and then passes nothing is the classic presentation.

IPv6 changed the fragmentation rules to make the problem more visible rather than less common. IPv6 routers never fragment at all: a packet too large is always discarded and a Packet Too Big message is returned to the sender.

That makes PMTUD mandatory rather than optional on IPv6, and it makes blocking ICMPv6 messages a way to break the network rather than a way to harden it.

Tunnel overheadThe numbers, and what every tunnel costs

MTU and fragmentation issues on a small business network almost always come down to encapsulation. Every tunnel wraps the original packet in new headers, and those bytes come out of the space available for data.

Each layer of encapsulation costs again, so a tunnel carried inside another tunnel subtracts twice and the packet size available for payload drops accordingly.

Two notes on reading that table.

The IPsec figure is a range because it depends on the cipher, the mode and whether NAT traversal is wrapping it in UDP. Rather than calculate precisely, most designs set the tunnel MTU to 1400 and stop thinking about it, which wastes a hundred bytes per packet and removes a whole category of problem.

Jumbo frames go the other way. An MTU of 9000 on a storage or data center network reduces per packet overhead considerably, and it only works if every device on the path agrees. One switch left at 1500 in a jumbo frame network produces exactly the failure described above.

MSS clampingMSS clamping, which is the fix

The correct response to a tunnel with reduced MTU is usually not to fix path MTU discovery, because you do not control the firewalls on the far side. It is to prevent oversized packets from being created at all.

MSS clamping is a router adjusting the maximum segment size value in the TCP SYN options as the packets pass through. Both endpoints think they negotiated a smaller segment size, so they never send anything that needs fragmenting, and PMTUD never has to run.

It is a small, sturdy, slightly inelegant fix with three properties worth knowing.

It only works for TCP. UDP carries no MSS option and has no negotiation to adjust. A UDP application sending large datagrams over a tunnel still needs the MTU set correctly or the application configured smaller.

It is applied on the tunnel interface, in both directions, on the device that owns the tunnel. Most firewall and router platforms have a single setting for it, sometimes called adjust MSS or clamp MSS to MTU. On Linux it is an iptables rule in the mangle table:

iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
         -j TCPMSS --clamp-mss-to-pmtu

The --clamp-mss-to-pmtu form derives the value from the outgoing interface MTU. --set-mss 1360 sets a fixed number instead, which is what you want when the path MTU is smaller than the local interface suggests.

It rewrites a field in transit, which is the sort of thing a purist objects to and everybody does anyway, because the alternative is depending on ICMP surviving a path you do not administer.

PitfallsWhere people go wrong

Blocking all ICMP and then debugging the tunnel. Type 3 code 4 is what path MTU discovery runs on. Blocking it converts a self correcting mechanism into a silent failure.

Setting the MTU on the wrong interface. The tunnel interface needs the reduced MTU, not the physical one underneath it. Lowering the physical MTU makes every other flow slower and does not fix the tunnel.

Testing with ping and concluding it is fine. A default ping is small enough to fit through anything, so it clears a path that real traffic cannot use. Test with a large packet and the do not fragment bit set, which is what ping -f -l on Windows and ping -M do -s on Linux are for.

Enabling jumbo frames on some devices. It has to be every device on the path, including the ones nobody remembers, or large packets vanish between the ones that agree and the one that does not.

Assuming fragmentation is harmless. It multiplies packet counts, it makes a single lost fragment cost the whole payload, and many security appliances cannot inspect a fragment properly so they drop it. For example, a firewall filtering on port numbers finds ports in the first fragment of the set and nowhere else.

Clamping the MSS and expecting UDP to improve. There is no MSS option in UDP to clamp. If the traffic that fails is UDP, the MTU itself has to be right.

Calculating IPsec overhead precisely. The number moves with the cipher and the encapsulation. Set 1400 and move on unless you are optimizing a link where the last hundred bytes matter.

ComparisonThree numbers, and the only one you should actually adjust

The last two rows together are the practical advice. Path MTU discovery is the mechanism that should handle this and cannot be relied on, so the number you actually adjust is the MSS.

FAQFrequently asked questions

Keep readingRelated concepts

Read next · Remote access What Is an IPsec VPN? A tunnel that comes up and then passes nothing is this, and IPsec is where it happens most. Open this next11 min

• Protocols · 11 min What Is ICMP? Type 3 code 4 is the message path MTU discovery runs on, and blocking it is what turns this into a silent failure.
• Switching · 12 min What Is VXLAN? Fifty bytes of encapsulation on a fabric nobody set to jumbo frames is the data center version of the same fault.
• Fundamentals · 10 min The IPv4 Header, Field by Field and in Order of Usefulness What the fragment fields in this header actually do.
• Remote access · 10 min GRE vs IPsec, and the Reason They Are Normally Used Together What the stacked overhead costs, and how to set it.
• Network fundamentals · 9 min Ethernet Frame Format, Field by Field The MTU that sets the frame payload ceiling explained in full.
• Network fundamentals · 9 min TCP Window Size, and Why It Caps Your Throughput Per-segment size versus total in-flight data.