Why architecture matters here
The architecture matters because the failure it prevents is uniquely deceptive. A PMTU problem does not look like a network problem — the connection establishes perfectly, because the handshake packets are small and cross any path. It only breaks when the application sends something large: a big HTTP response, a file upload, a database result set. So the symptom is 'small requests work, big ones hang,' which points suspicion at the application, the server, or the payload, when the real cause is a link three hops away silently dropping oversized packets. An engineer can burn hours in the wrong layer because the failure mode is invisible to every tool that only sends small probes.
It matters because it is fundamentally a per-path, changeable quantity, not a fixed setting. There is no single MTU you can configure once; the path MTU to one destination may be 1500 and to another 1400, and either can change when routing shifts a flow onto a different link with a different MTU. That is why the value must be discovered dynamically and cached per destination, and why the cache must age out and re-probe — a PMTU learned an hour ago may be stale after a reroute. An architecture that treats MTU as a static host setting will be right on the paths it was tuned for and silently wrong on the rest, so the discovery-and-cache design is what makes correct behavior portable across a heterogeneous network.
It matters because the trade-offs of the alternatives are all bad, which is why discovery is worth the trouble. You could fragment every oversized packet, but fragmentation is inefficient, hurts performance, is a reassembly and security headache, and is simply not available for IPv6 routers or DF-set IPv4. You could set every packet conservatively small (a low MSS everywhere), but then you pay per-packet overhead on every transfer across every path, wasting bandwidth on the many links that could have carried full-size frames. PMTUD exists precisely to get the best of both — full-size packets where the path allows, smaller only where it must — and the cost of that optimization is the discovery machinery and its dependence on ICMP, which is the fragility the rest of the architecture works around.
Finally, it matters because tunnels and overlays have made sub-1500 path MTUs the common case, not the exception. VPNs, GRE, IPsec, VXLAN, and cloud overlay networks all wrap packets in extra headers, shrinking the usable MTU below the 1500 that endpoints assume. A modern data-center or cloud path frequently has an effective MTU of 1400-ish somewhere in the middle, so PMTUD (or a clamp that stands in for it) is not an edge case but a routine requirement. This is why MSS clamping at tunnel boundaries — rewriting the TCP handshake's maximum segment size to fit the tunnel — has become standard practice: it makes the endpoints agree to a safe size up front so the black hole never forms.
The architecture: every piece explained
Top row: discovery working as designed. The sender starts with its local link MTU (1500 on Ethernet) as its guess and sends full-size packets with the don't-fragment bit set. Somewhere along the path is a link with a smaller MTU — say a 1400-byte tunnel. That link cannot forward the 1500-byte packet without fragmenting, and DF forbids fragmenting, so instead of dropping silently it returns an ICMP 'fragmentation needed' message (type 3, code 4 in IPv4; 'packet too big' in ICMPv6) that carries the next-hop MTU, 1400. Receiving it, the sender lowers its path MTU for that destination from 1500 to 1400 and resends the data in 1400-byte packets that now cross the whole path. Discovery has converged in one round.
Middle-left: the failure the architecture must survive. If the ICMP is filtered — a firewall blocking ICMP, a load balancer not relaying it, an asymmetric route — the 'fragmentation needed' message never reaches the sender. This is the blackhole: the router still drops the oversized packet (it must, with DF set), but the sender never learns why. So big packets vanish while small ones pass, and because the handshake used small packets, the connection came up fine and then hangs the instant it sends bulk data. The sender keeps retransmitting the same too-big packets, each silently dropped, until the connection times out.
Middle-right: the two robustness mechanisms that do not depend on ICMP. PLPMTUD (packetization-layer PMTUD) has the transport itself probe: it sends progressively larger packets and infers the path MTU from which sizes get acknowledged versus lost, so it discovers the right size without any ICMP message — immune to the blackhole. MSS clamping takes a different angle: a device at the tunnel edge rewrites the maximum-segment-size option in the TCP handshake down to what the tunnel can carry, so both endpoints agree from the start to send segments that fit and never emit an oversized packet at all. One recovers from the blackhole; the other prevents it.
Bottom row and ops: caching and the operational surface. The sender caches the discovered PMTU per destination, reuses it for subsequent packets, ages it out after a while, and periodically re-probes upward in case the path changed to allow larger packets again — so discovery is amortized, not repeated per packet, yet still adapts. When it all works the path is stable at the right size: no fragmentation, no blackhole, full-size packets wherever the path allows and smaller only where it must. The ops strip names the practical rules: allow ICMP type 3 code 4 (never blanket-block ICMP), clamp MSS on tunnels so sub-1500 overlay paths are safe by construction, keep a PLPMTUD fallback for paths where ICMP is unreliable, and watch for stalled large transfers as the fingerprint of a blackhole.
End-to-end flow
Trace a single HTTPS download crossing a VPN tunnel with a 1400-byte MTU, first in a healthy network and then through a firewall that blocks ICMP, to see discovery succeed and then collapse — and to see the clamp save it.
The handshake succeeds either way: the client opens the connection. The TCP SYN, the TLS ClientHello, and the early handshake packets are all small — well under 1400 bytes — so they cross the 1400-byte tunnel with room to spare regardless of MTU discovery. The connection establishes cleanly. This is the trap: everything looks healthy, because nothing large has been sent yet.
Healthy path — discovery converges: the server begins sending the response body in full 1500-byte segments with DF set. The first one hits the tunnel ingress, which cannot forward 1500 bytes and returns an ICMP 'fragmentation needed' carrying MTU 1400. The server receives it, lowers its cached path MTU for this client to 1400, and retransmits the data in 1400-byte segments. From here the download streams at full speed; the server reuses the cached 1400 for the rest of the transfer. One brief round of discovery, then steady state — exactly as designed.
Blackholed path — the hang: now put a firewall in front of the server that drops inbound ICMP. The server again sends 1500-byte DF segments; the tunnel again drops them and sends the ICMP 'fragmentation needed' — but the firewall discards it, so the server never hears back. The server's cached PMTU stays at 1500, so it retransmits the same oversized segments, which the tunnel drops again. The client, meanwhile, received the handshake but no body: the download stalls at zero or a few bytes and eventually times out. Every diagnostic that uses small packets (ping, a quick curl of a small page) works perfectly, which sends the operator hunting in the wrong place. This is the PMTU blackhole in its natural habitat.
The fixes bite: two changes rescue it. First, allow the ICMP 'fragmentation needed' message (type 3 code 4) through the firewall — it is not a threat, it is essential signaling — and discovery converges as in the healthy case. Second, and more robustly, clamp the MSS at the tunnel edge: a device on the tunnel rewrites the MSS option in the TCP handshake down to 1360 (1400 minus TCP/IP headers), so the server and client agree from the very first data segment to send pieces that fit the tunnel, and no oversized packet is ever emitted — the blackhole cannot form because there is nothing for the tunnel to reject. With the clamp in place the transfer runs full-speed and the operator never sees the hang at all — which is exactly why clamping sub-1500 tunnels is standard practice rather than relying on ICMP to always get through.
Path MTU discovery on Linux: find the path MTU by hand, read what the kernel cached, and the sysctls that control it
The architecture above is what the kernel does on your behalf. When a transfer hangs, you want to do it by hand. Four tools, in the order you would use them:
# 1. Probe a size with the don't-fragment bit set: 1472 bytes of ICMP payload + 28 bytes of headers = 1500
ping -c 3 -M do -s 1472 203.0.113.10 # replies: the path carries 1500-byte packets
ping -c 3 -M do -s 1372 203.0.113.10 # a 1400-byte packet
# "ping: local error: message too long, mtu=1400" -> the kernel already learned this path is 1400
# 100% loss for the large size, replies for small -> a blackhole: some device drops the ICMP
# 2. Let the kernel walk the path and report what it learns hop by hop
tracepath 203.0.113.10 # prints "pmtu 1400" when a hop reports a smaller MTU
tracepath -6 2001:db8::10 # IPv6: Packet Too Big is ICMPv6 type 2; the minimum MTU is 1280
# 3. What a live TCP connection is actually using
ss -tin dst 203.0.113.10 | grep -o 'pmtu:[0-9]*\|mss:[0-9]*\|advmss:[0-9]*'
# 4. The per-destination exception the kernel cached, and when it expires
ip route get 203.0.113.10 # shows mtu 1400 and an expires value while the exception lives| Sysctl | Default | What it does |
|---|---|---|
net.ipv4.ip_no_pmtu_disc | 0 | 0 performs PMTUD and sets DF; 1 turns it off and allows fragmentation; 2 sets DF but ignores ICMP; 3 is the hardened mode that relies on probing |
net.ipv4.tcp_mtu_probing | 0 | 0 off; 1 starts packetization-layer probing (RFC 4821) after a blackhole is detected; 2 probes always |
net.ipv4.tcp_base_mss | 1024 | The MSS probing starts from once a blackhole is detected |
net.ipv4.tcp_probe_interval | 600 | Seconds between upward probes once probing is on |
net.ipv4.route.min_pmtu | 552 | An ICMP message asking for a smaller path MTU than this is ignored |
net.ipv4.route.mtu_expires | 600 | Seconds a learned path MTU exception lives before the kernel tries full size again |
Setting tcp_mtu_probing=1 on servers that talk to the internet is the cheapest insurance against the blackhole: when retransmissions of a full-size segment keep failing, the stack falls back to tcp_base_mss and probes upward, so the connection recovers without any ICMP. The clamp at a tunnel edge is still the better fix because it prevents the first loss:
# MSS clamping on the forwarding device of a 1400-byte tunnel: 1400 - 40 bytes of IPv4 + TCP headers = 1360
iptables -t mangle -A FORWARD -o wg0 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1360
# or let netfilter derive it from the route MTU of each path
iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
nft add rule inet filter forward oifname "wg0" tcp flags syn tcp option maxseg size set rt mtu
# And never drop the signal: fragmentation-needed and packet-too-big are not attack traffic
iptables -A INPUT -p icmp --icmp-type fragmentation-needed -j ACCEPT
ip6tables -A INPUT -p icmpv6 --icmpv6-type packet-too-big -j ACCEPTCloud paths are where this bites most. Inside an AWS VPC the MTU is 9001 but traffic through an internet gateway or over peering to another Region is 1500 and VPN tunnels are smaller again; a Google Cloud VPC defaults to 1460. A single tracepath from the application host to the destination tells you which of those a hanging transfer is crossing.