The premiere source of truth powering network automation. Open and extensible, trusted by thousands.
NetBox is now available as a managed cloud solution! Stop worrying about your tooling and get back to building networks.
Disabling unreachables breaks PMTUD
By stretch | Thursday, October 9, 2008 at 12:03 a.m. UTC
After reading the reposting of my article on path MTU discovery in NIL's CT3 wiki, Steve Milsom wrote in to share an interesting observation. Disabling ICMP unreachable messages on a router with no ip unreachables (as is recommended by many BCP documents) has a side effect of breaking path MTU discovery. Let's take a look at what happens.
Recall that path MTU discovery is accomplished by setting the Don't Fragment (DF) bit in an outgoing IP packet, and receiving an ICMP unreachable message in the event that the packet sent exceeded the MTU at some point in the path. We can create an artificially small outgoing IP MTU on an interface with the ip mtu command in IOS to see path MTU discovery in action. We'll set the IP MTU of R2's F0/0 interface to 1200 bytes.
R2(config)# interface f0/0 R2(config-if)# ip mtu 1200
Now, by setting the DF bit in oversized ping from R1 to R3 we can see that we receive ICMP unreachable messages from R2, indicating that packet fragmentation is required to meet R2's outbound MTU.
R1# ping ip 10.0.0.6 df-bit size 1400 Type escape sequence to abort. Sending 5, 1400-byte ICMP Echos to 10.0.0.6, timeout is 2 seconds: Packet sent with the DF bit set M.M.M Success rate is 0 percent (0/5)
The letter M (for MTU, I suppose) in the output signifies the reception of an ICMP unreachable packet with code 4 (fragmentation needed). (The alternating dots indicating missed responses are due to IOS' default ICMP rate-limiting.)
Now we'll disable the transmission of ICMP unreachables out of R2's F0/1 interface with the no ip unreachables command:
R2(config)# interface f0/1 R2(config-if)# no ip unreachables
Observe what happens when we try the oversized ping from R1 again:
R1# ping ip 10.0.0.6 df-bit size 1400 Type escape sequence to abort. Sending 5, 1400-byte ICMP Echos to 10.0.0.6, timeout is 2 seconds: Packet sent with the DF bit set ..... Success rate is 0 percent (0/5)
Instead of responding with helpful ICMP messages, R2 simply drops our oversized packets. Think carefully about where you disable ICMP unreachables, as our experiment shows that path MTU discovery simply doesn't work without them.
Posted in Design
Comments
October 9, 2008 at 9:11 a.m. UTC
I also wonder how firewalls interfere with PMTUD?
We block all ICMP packets on our firewalls believing it to be security best practice. Is this incorrect?
October 9, 2008 at 2:41 p.m. UTC
Yep, firewalls can interfere with this. You need to make sure that the firewall permits ICMP unreachable (type 3, code 4).
October 9, 2008 at 5:47 p.m. UTC
And Cisco "autosecure" feature automatically disables "ICMP unreach" on all interfaces. ;-(
October 10, 2008 at 1:56 a.m. UTC
You need to be extra careful when disabling unreachables on VPN tunnel interfaces. tunnel pmtud is also broken when unreachables are disabled.
October 10, 2008 at 7:56 p.m. UTC
IPV6 will fix this issue....
October 11, 2008 at 1:33 a.m. UTC
To the commenter who blocks all ICMP traffic, I don't think this is a good practice for this very reason.
Regarding PMTUD and the no ip unreachables command, I learned this the hard way a few years ago. Cisco routers doing Lan to Lan VPN. Disabled the unreachables on all interfaces. Everything went totally to crap. Found that the inside interfaces should not have "no ip unreachables". Problem resolved.
October 13, 2008 at 2:52 p.m. UTC
Yeah, funny that so many people with more than a mont's experience have been bitten in the ass by this but it's still taught as default secure template stuff. The number of legacy apps no ip unreachables breaks is simply stunning, and the number of "I spent days and sleepless nights trying to figure out what had changed on our network..." stories I've read which eventually boiled down to "someone stuck this line in a config somewhere because . . . well, because." gives me heartburn - moreso because I've been there myself.
November 24, 2009 at 10:33 p.m. UTC
I don't see any security benefit when disabling ICMP unreachables. I am also against blindly implementing (or disabling) features because they appear in some security templates from the 90s.
Check my post about 'Why disabling ICMP unreachables is a bad thing'.
March 4, 2010 at 11:59 a.m. UTC
My understanding is this, and I'd welcome corrections or additions...
Disabling ICMP unreachables is useful because they are generated by the router CPU, and you are potentially exposed to a Denial of Service through CPU resource exhasustion by allowing ICMP to be generated.
In many Cisco routers the solution is simply not to punt packets which would generate ICMP messages to the CPU, and this is done using "no ip unreachables".
Unfortunately the side-effect is that Path MTU Discovery breaks, because that also uses one of the ICMP unreachable messages.
The solution is to selectively punt packets which you want ICMP messages for, but I think this feature is only available on certain 6500 models. To allow ICMP for everything but ACL-denied packets (a reasonable default) use the command:
mls rate-limit unicast ip icmp unreachable acl-drop 0
And if you search for that command on cisco.com up will come the command reference with more details.
HTH, oliver.
May 24, 2012 at 7:35 a.m. UTC
Agree with above. We had an issue with very high CPU utilization. Implementing "no ip unreachables" on specific interfaces resolved the problem. Take a look on this Cisco document:
http://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note09186a00804916e0.shtml#solution
November 16, 2012 at 1:08 a.m. UTC
In the context of pmtud, I suppose you only need to send unreachable if your interfaces have mismatched MTU and only from the interface with the larger MTU. To my knowledge, the no unreachables command only applies to locally generated unreachables and not those transiting the device.
November 26, 2012 at 2:43 p.m. UTC
I'm running a large DMVPN (200+ nodes) and I can assure you that
"no ip unreachables" affects all traffic, including those transiting the router. I've met dozen of routers breaking my nodes because of that.
March 23, 2013 at 6:21 p.m. UTC
breaking PMTUD is not as bad a situation as having sophisticated intruders who can exploit your perimeter using tricked up ICMP packets. If you don't believe me, go google one of Ofir Arkin's articles on the subject. I don't allow A N Y ICMP thru and that keeps those unwanted probes off my back. ICMP diagnostics were a good thing once upon a time when the Internet was primarily a well-behaved environment, used by developers, scientists and the like. Those times are long past now, and there are too many people intent on devious means to circumvent routers, ACLs, firewalls. Anyone requiring access to a site buried behind some archaic narrow link, can leave this site and go poke around using a laptop, presumably running a USB live distro to prevent any permanent infection..
April 11, 2013 at 3:06 p.m. UTC
Like others have mentioned, ICMP Type 3 Code 4 should always be allowed for PMTUD to work. Otherwise, You may need to choose to get your router to clear df bit, but that will of course cause fragmentation.
But if there is concern about security, we can do ICMP filtering with ACL or more efficiently, just do Control Plane Policing.
My 2cents - www.netxplane.com