Traceroute timeouts
If you spend a lot of time performing traceroutes to Cisco routers you've probably noticed that they usually end like this:
R1# traceroute 10.0.34.4 Type escape sequence to abort. Tracing the route to 10.0.34.4 1 10.0.12.2 16 msec 8 msec 12 msec 2 10.0.23.3 16 msec 16 msec 16 msec 3 10.0.34.4 16 msec * 20 msec
Notice that the second reply from the last hop has timed out. This is easily repeated with subsequent traceroutes, and it is always the second attempt which times out. Strange, eh?
The reason for this is IOS' default ICMP rate limiting. Back in May I wrote an article explaining the common "U.U.U" response that results from pinging an unreachable destination, and the same logic is at work here. Inspecting the default ICMP rate limits reveals that ICMP unreachable messages are only sent every 500ms:
R4# show ip icmp rate-limit DF bit unreachables All other unreachables Interval (millisecond) 500 500 Interface # DF bit unreachables # All other unreachables --------- --------------------- ------------------------ FastEthernet0/1 0 3 Greatest number of unreachables on FastEthernet0/1
This rate limiting configuration effectively ignores every other traceroute packet (see the timeline illustration in the previous article). Makes sense, but why do all the requests to routers prior to the last hop receive replies without this problem?
Those replies are of a different ICMP type, namely the "TTL exceeded" type. Remember that traceroute works by sequentially incrementing the TTL of UDP (or ICMP on Windows) packets destined for a host and recording the replies received from intermediate routers. All hops except the last one will (or should) return a "TTL exceeded in transit" message, whereas the last hop should return a "destination unreachable/port unreachable" message indicating that it cannot handle the received traffic (UDP traceroute packets are typically addressed to a pseudorandom high port on which the end host is not likely to be listening).
The packet capture attached at the end of this article includes the traffic from the traceroute demonstrated above.
Interestingly, we can remove the ICMP rate limit with no ip icmp rate-limit:
R4(config)# no ip icmp rate-limit unreachable R4(config)# ^Z R4# show ip icmp rate-limit
Now our traceroute from R1 completes fully:
R1# traceroute 10.0.34.4 Type escape sequence to abort. Tracing the route to 10.0.34.4 1 10.0.12.2 32 msec 12 msec 20 msec 2 10.0.23.3 60 msec 56 msec 24 msec 3 10.0.34.4 32 msec 44 msec 36 msec
However, removing the ICMP rate-limit may open an avenue for denial of service attacks on your routers, so in practice you probably want to leave it applied.
About the Author
Jeremy Stretch is a networking engineer and the maintainer of PacketLife.net. He currently lives in the Raleigh-Durham area of North Carolina. Although employed full-time out of necessity, his true passion lies in improving the field of network engineering around the world. You can contact him by email or follow him on Twitter.
Comments
I've seen that a million times but never realized what was going on. I feel much more enlightened now. Thanks, Stretch.
Good timing - I was doing some testing with this very config option last week, as I'm preparing to deploying a new router and also creating a new configuration template for our Cisco gear.
I've found that I still like to keep my unreach's rate-limited but the default setting of 500ms is a little too restrictive for me.
FYI, the 'show/clear ip icmp' commands are in the 12.4(2)T train, not the 12.4 mainline train. :)
Here, you can find the official Cisco documentation for ip icmp rate-limit command: http://www.cisco.com/en/US/docs/ios/12_4t/12_4t2/hticmpun.html
Another note on the firewall note, UDP Traceroute is considered a complex packet so it gets inspected by firewalls. For firewalls using Acceleration ie: Nokia Checkpoint ADP, this disables the templates, which causes everything else below the rule base to not be accelerated.
Might be not really related to this but interesting too but on Linux and Solaris boxes you can use the -l switch to force traceroute to use ICMP instead of the default behaviour which is to use UDP. On Windows this is not needed as ICMP is the default. It is not uncommon that one of the hops is either a firewall or is a router with an access list that blocks unknown UDP traffic which means that you will generally get better path information from using ICMP based traceroutes. This is also a problem imho with the Cisco implementation of the traceroute utility on IOS as it does not support sending ICMP packets instead of UDP packets.