Traceroute timeouts

If you spend a lot of time performing traceroutes to Cisco routers you've probably noticed that they usually end like this:

R1# traceroute 10.0.34.4

Type escape sequence to abort.
Tracing the route to 10.0.34.4

1 10.0.12.2 16 msec 8 msec 12 msec
  2 10.0.23.3 16 msec 16 msec 16 msec
  3 10.0.34.4 16 msec *  20 msec

Notice that the second reply from the last hop has timed out. This is easily repeated with subsequent traceroutes, and it is always the second attempt which times out. Strange, eh?

The reason for this is IOS' default ICMP rate limiting. Back in May I wrote an article explaining the common "U.U.U" response that results from pinging an unreachable destination, and the same logic is at work here. Inspecting the default ICMP rate limits reveals that ICMP unreachable messages are only sent every 500ms:

R4# show ip icmp rate-limit

DF bit unreachables       All other unreachables   
Interval (millisecond)     500                       500

Interface                  # DF bit unreachables     # All other unreachables 
---------                  ---------------------     ------------------------ 
FastEthernet0/1            0                         3

Greatest number of unreachables on FastEthernet0/1

This rate limiting configuration effectively ignores every other traceroute packet (see the timeline illustration in the previous article). Makes sense, but why do all the requests to routers prior to the last hop receive replies without this problem?

Those replies are of a different ICMP type, namely the "TTL exceeded" type. Remember that traceroute works by sequentially incrementing the TTL of UDP (or ICMP on Windows) packets destined for a host and recording the replies received from intermediate routers. All hops except the last one will (or should) return a "TTL exceeded in transit" message, whereas the last hop should return a "destination unreachable/port unreachable" message indicating that it cannot handle the received traffic (UDP traceroute packets are typically addressed to a pseudorandom high port on which the end host is not likely to be listening).

The packet capture attached at the end of this article includes the traffic from the traceroute demonstrated above.

Interestingly, we can remove the ICMP rate limit with no ip icmp rate-limit:

R4(config)# no ip icmp rate-limit unreachable
R4(config)# ^Z
R4# show ip icmp rate-limit

Now our traceroute from R1 completes fully:

R1# traceroute 10.0.34.4

Type escape sequence to abort.
Tracing the route to 10.0.34.4

1 10.0.12.2 32 msec 12 msec 20 msec
  2 10.0.23.3 60 msec 56 msec 24 msec
  3 10.0.34.4 32 msec 44 msec 36 msec

However, removing the ICMP rate-limit may open an avenue for denial of service attacks on your routers, so in practice you probably want to leave it applied.