TechWiseTV

Posted by stretch in Random on Wednesday, 20 Aug 2008 at 12:53 a.m. GMT

It turns out Cisco has been sponsoring an hour-long monthly webcast called TechWiseTV since late 2006, which I only recently found out about. Deviating from the normal sales-oriented, Powerpoint style of Cisco presentations, the show is decidedly "television-like" and mildly entertaining. Why they haven't done more to promote it is beyond me.

While one should expect to find some amount of product placement in any production bearing the Cisco logo, I was pleasantly surprised in the level of technical detail present in the TechWiseTV back episodes I perused (especially the botnet episode). Their next episode airs Thursday, August 21st, entitled, "Energy Efficiency in the Data Center." I suspect its emphasis will be more on APC and EMC/VMware than on Cisco, but it still looks worth a watch. Preview below.

Unfortunately, registration and/or a guest-level CCO account is required for access, and the video is only viewable in RealPlayer (which actually wasn't as bad as I thought it would be).

Path MTU Discovery

Posted by stretch in Networking on Monday, 18 Aug 2008 at 12:57 a.m. GMT

When a host needs to transmit data out an interface, it references the interface's Maximum Transmission Unit (MTU) to determine how much data it can put into each packet. Ethernet interfaces, for example, have a default MTU of 1500 bytes, not including the Ethernet header or trailer. This means a host needing to send a TCP data stream would typically use the first 20 of these 1500 bytes for the IP header, the next 20 for the TCP header, and as much of the remaining 1460 bytes as necessary for the data payload. Encapsulating data in maximum-size packets like this allows for the least possible consumption of bandwidth by protocol overhead.

Unfortunately, not all links which compose the Internet have the same MTU. The MTU offered by a link may vary depending on the physical media type or configured encapsultion (such as GRE tunneling or IPsec encryption). When a router decides to forward an IPv4 packet out an interface, but determines that the packet size exceeds the interface's MTU, the router must fragment the packet to transmit it as two (or more) invidiual pieces, each within the link MTU. Fragmentation is expensive both in router resources and in bandwidth utilization; new headers must be generated and attached to each fragment. (In fact, the IPv6 specification removes transit packet fragmentation from router operation entirely, but this discussion will be left for another time.)

Fragmentation performed by the router

To utilize a path in the most efficient manner possible, hosts must find the path MTU; this is the smallest MTU of any link in the path to the distant end. For example, for two hosts communicating across three routed links with independent MTUs of 1500, 800, and 1200 bytes, the smallest (800 bytes) must be assumed by each end host to avoid fragmentation.

Path MTU

Of course, it's impossible to know the MTU of each link through which a packet might travel. RFC 1191 defines path MTU discovery, a simple process through which a host can detect a path MTU smaller than its interface MTU. Two components are key to this process: the Don't Fragment (DF) bit of the IP header, and a subcode of the ICMP Destination Unreachable message, Fragmentation Needed.

IP DF flag

Setting the DF bit in an IP packet prevents a router from performing fragmentation when it encounters an MTU less than the packet size. Instead, the packet is discarded and an ICMP Fragmentation Needed message is sent to the originating host. Essentially, the router is indicating that it needs to fragment the packet but the DF flag won't allow for it. Conveniently, RFC 1191 expands the Fragmentation Needed message to include the MTU of the link necessitating fragmentation. A Fragmentation Needed message can be seen in packet #6 of this packet capture.

Path MTU discovery

Now that the actual path MTU has been learned, the host can cache this value and packetize future data for the destination to the appropriate size. Note that path MTU discovery is an ongoing process; the host continues to set the DF flag so that it can detect further decreases in MTU should dynamic routing influence a new path to the destination. RFC 1191 also allows for periodic testing for an increased path MTU, by occasionally attempting to pass a packet larger than the learned MTU. If the packet succeeds, the path MTU will be raised to this higher value.

You can test path MTU discovery across a live network with a tool like tracepath (part of the Linux IPutils package) or mturoute (Windows only). Here's a sample of tracepath output from the lab pictured above, with the MTU of F0/1 reduced to 1400 bytes using the ip mtu command:

Host$ tracepath -n 192.168.1.2
 1:  192.168.0.2       0.097ms pmtu 1500
 1:  192.168.0.1       0.535ms 
 1:  192.168.0.1       0.355ms 
 2:  192.168.0.1       0.430ms pmtu 1400
 2:  192.168.1.2       0.763ms reached
    Resume: pmtu 1400 hops 2 back 254

Cheat sheets update

Posted by stretch in Announcements on Saturday, 16 Aug 2008 at 1:29 p.m. GMT

Should the cheat sheets be called something else?

  • No, "cheat sheets" is fine (79%)
  • Yes, but I don't know what (11%)
  • Yes, and I have a suggestion I'll post in the comments (7%)
  • What are cheat sheets? (0%)

The votes are in! 164 people want to keep the term "cheat sheets," 40 people would like it changed, and two people apparently weren't sure what the poll was about. There were some good suggestions in the comments, but the majority of readers seem to think the current name is fine. Keeping the current name also requires the least amount of work on my part, so I suppose I'm content with that decision.

Several readers have written in asking for some sort of PIX/ASA cheat sheet; sadly I have very limited experience with these devices. However, I've started watching some of the CBT Nuggets for the CCSP track (by Jeremy Cioara), so I might be able to put together something useful soon. For now I have a number of cheat sheets already in the pipeline, and I've been considering a mailing list for people who want to help me review beta cheat sheets for errors before their public release. We'll see how that goes.

EIGRP goodbyes

Posted by stretch in Networking on Thursday, 14 Aug 2008 at 9:31 a.m. GMT

EIGRP includes a special type of hello packet, known as a "goodbye." Goodbyes are transmitted by an EIGRP router to its peer whenever a neighbor adjacency fails but the interface to reach that neighbor is still up. This could be the result of administrative reconfiguration, but sometimes the physical link between two EIGRP neighbors will fail without the routers realizing it (the interface status remains up/up).

In this case, the router will continue to transmit hellos (among other data) to the neighbor without receiving any hellos in return until the hold timer expires and the adjacency is torn down. When this happens, a router will transmit a goodbye message as an explicit notification that it is tearing down the adjacency.

%DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 10.0.0.1 (FastEthernet0/0) is down:
Interface Goodbye received

You're likely to wonder what the point of this is if the link to the neighbor has failed. However, goodbyes provide an excellent troubleshooting tool when dealing with links that have only partially failed and have become unidirectional. This is a common scenario with fiber connections, and as I have come to find, with satellite links. Often one satellite station will be receiving from another, but for some reason (such as poor weather) will not be transmitting at a high enough power for its signal to be received by the distant station. When EIGRP is run across such a link, the distant station will tear down the adjacency and transmit a goodbye packet, giving the operator a clear indication of the direction in which the link is still functional.

Unidirectional link between satellite stations

The goodbye packet itself is actually a hello packet with all its K values set to 255. An example can be seen in packet #9 of this EIGRP traffic capture.

802.1X guest VLANs

Posted by stretch in Networking on Tuesday, 12 Aug 2008 at 8:02 a.m. GMT

Reader Marcus suggested an extension of last week's post on configuring wired 802.1X discussing the use of a guest VLAN. 802.1X includes the capability of defining a guest VLAN in which unauthenticated clients are placed. This can be handy to provide a sort of user-facing demilitarized zone, where guests can connect to the network but only to receive limited services, or to provide temporary restricted access to download software updates.

Configuring a guest VLAN is simple, so long as you understand the concept. We'll continue from the previous example, adding a guest VLAN with access only to the Internet, as depicted here:

Lab with guest VLAN

This time, however, we'll assume our client is not 802.1X-capable (if continuing from the previous lab, be sure to disable 802.1X authentication under the network connection properties dialog). Configuring a guest VLAN is as simple as defining the VLAN number appropriately under the desired 802.1X interfaces:

Switch(config)# interface g0/12
Switch(config-if)# dot1x guest-vlan 99
Switch(config-if)# do show run int g0/12
Building configuration...

Current configuration : 159 bytes
!
interface GigabitEthernet0/12
 switchport access vlan 10
 switchport mode access
 dot1x port-control auto
 dot1x guest-vlan 99

We can see VLAN 99 specified as the guest VLAN under the interface's 802.1X properties.

Switch# show dot1x int g0/12
Supplicant MAC <Not Applicable>
   AuthSM State      = CONNECTING
   BendSM State      = IDLE
PortStatus        = UNAUTHORIZED
MaxReq            = 2
MaxAuthReq        = 2
HostMode          = Single
PortControl       = Auto
QuietPeriod       = 60 Seconds
Re-authentication = Disabled
ReAuthPeriod      = 3600 Seconds
ServerTimeout     = 30 Seconds
SuppTimeout       = 30 Seconds
TxPeriod          = 30 Seconds
Guest-Vlan        = 99

Now we can connect our client. The switch will prompt for authentication but, receiving no response, will eventually (after roughly 90 seconds) give up and place the client in the guest VLAN.

Switch# show dot1x int g0/12
Supplicant MAC <Not Applicable>
   AuthSM State      = AUTHENTICATED(GUEST_VLAN)
   BendSM State      = IDLE
PortStatus        = AUTHORIZED(GUEST-VLAN)
MaxReq            = 2
MaxAuthReq        = 2
HostMode          = Multi(GUEST VLAN)
PortControl       = Auto
QuietPeriod       = 60 Seconds
Re-authentication = Disabled
ReAuthPeriod      = 3600 Seconds
ServerTimeout     = 30 Seconds
SuppTimeout       = 30 Seconds
TxPeriod          = 30 Seconds
Guest-Vlan        = 99

Switch# show int g0/12 switchport | i Access
Access Mode VLAN: 99 (VLAN0099)

The unauthenticated client can now only send traffic to VLAN 99; it will have to authenticate successfully before it can be placed in its permanent VLAN, VLAN 10.

Note that the guest VLAN is intended to be used for clients which have no ability to authenticate; those clients which fail authentication deserve additional consideration. Prior to IOS 12.2(25)SE, clients which fail authentication are simply placed in the guest VLAN. With the introduction of 12.2(25)SE, clients which fail authentication remain in their assigned VLAN, unable to communicate. This behavior can be reverted to the older style with the command dot1x guest-vlan supplicant under global configuration.

IOS 12.2(25)SED introduced another option: restricted VLANs. A restricted VLAN can be defined in addition to a guest VLAN, to handle clients which attempt and fail authentication. Defining a restricted (or "authentication failure") VLAN is similar to defining a guest VLAN:

Switch(config-if)# dot1x auth-fail vlan <VLAN>

A reference guide by any other name

Posted by stretch in Announcements on Monday, 11 Aug 2008 at 7:27 a.m. GMT

Should the cheat sheets be called something else?

  • No, "cheat sheets" is fine (79%)
  • Yes, but I don't know what (11%)
  • Yes, and I have a suggestion I'll post in the comments (7%)
  • What are cheat sheets? (0%)

When I first set about producing the collection of cheat sheets available on this site, I wasn't sure what to call them. Never expecting my work to grow as popular as it has, I decided to borrow the term "cheat sheet" from common slang, referring to a short document compiled of various details key to a certain technology or other element.

However, since publishing the cheat sheets a number of people have commented that they found the term to be confusing, or at least odd. Often I've considered renaming the documents to avoid any accidental confusion with less than reputable study aides, especially with the ever growing drama concerning certification integrity.

I decided to open the issue to the people who use the cheat sheets: you. You can vote in the poll provided here (RSS readers see the link at bottom) to have your say. The poll will run through Friday, August 15, and if you would like to propose a replacement term please leave a comment. Keep suggestions sensible; "Handy 8.5x11 Inch Network Technology Reference Guides" isn't very catchy.

Cisco SRNDs

Posted by stretch in Resources on Friday, 8 Aug 2008 at 3:16 a.m. GMT

Wouldn't it be nice if there was an archive of detailed best practices guidelines for each aspect of building a network? And, dare we dream, even free? Fortunately, Cisco has such a hidden little stockpile of knowledge: documents detailing all the various aspects of network design. They call them Solution Reference Network Design guides, or SRNDs.

The acronym is a bit of a kludge, but the content is pretty impressive. Cisco publishes SRNDs on many diverse topics related to network planning and implementation. Subjects range from enterprise routing to wireless to quality of service to the data center. The documents, most of which are available in PDF format, vary in size from fifteen-minute reviews to short books. Each reads like a cross between a study guide and a whitepaper, maintaining an appropriate level of technical depth while avoiding too much dry wording.

I've found the SRNDs also make great supplementary study material, and provide very real examples of technologies being deployed to meet business needs. From what I've read, the authors also stay appreciably distant from market speak or sales pitching.

A simple wired 802.1X lab

Posted by stretch in Networking on Wednesday, 6 Aug 2008 at 2:18 a.m. GMT

IEEE 802.1X is a very cool security feature. It was developed to provide real security for wired and wireless networks at layer two. A client connected to an 802.1X-protected port can't send any traffic other than EAP to the switch until he successfully authenticates with the proper credentials or certificate. This article demonstrates how you can setup a simple 802.1X lab using a Windows XP-based client and RADIUS server.

802.1X Operation

A network switch acts as the middleman between an authenticating client and an authentication server. The switch implements two protocols: EAP is used to communicate with the client at the network perimeter, while RADIUS is used to relay authentication details to the server inside the network. EAP offers a number of authentication mechanisms, but our setup will use simple username/password authentication with an MD5 challenge. The flow of a successful authentication is illustrated here:

Authentication process

For a better idea of what this exchange looks like on the wire, check out these packet captures of 802.1X and RADIUS traffic.

Server Configuration

For my setup, I chose to install FreeRADIUS on my Gentoo Linux workstation, but any RADIUS service should work. The configurations in this section correspond to a bare FreeRADIUS deployment using cleartext credentials stored in a text file. Obviously, real-world deployments would dictate a much more robust and secure authentication method such as LDAP.

The base server configuration is located in radiusd.conf (on Linux, this file should reside in /etc/raddb/), but we shouldn't need to change any of the default values for this lab. However, we will need to add the subnet address from which we expect to receive authentication requests (10.0.0.0/24) in clients.conf. Remember that although the 802.1X client resides in VLAN 10, the RADIUS client (the switch) will be sending requests from its 10.0.0.1 interface. Copy and paste this block to enable the network with the shared secret of MyRadiusKey:

client 10.0.0.0/24 {
    secret      = MyRadiusKey
    shortname   = Lab
}

We'll also define a user/password combination for testing. I've created the user John.McGuirk with the password S0cc3r. Feel free to pick your own username and password, but make sure to maintain the spacing in the configuration file (the reply message is optional):

John.McGuirk    Cleartext-Password := "S0cc3r"
                Reply-Message = "Hello, %u"

After completing this configuration remember to (re)start the RADIUS service.

Switch Configuration

Port-based 802.1X authentication allows for some really cool security measures (like dynamic VLAN assignment and per-user ACLs), but for this lab we'll establish a base configuration just for demonstration's sake.

A preliminary step, if you haven't done so already, is to enable IP routing on the switch:

Switch(config)# ip routing

Before diving into the actual 802.1X configuration, we'll need to enable Authentication, Authorization, and Accounting (AAA) for the switch (this step can be skipped if AAA is already active). A word of caution: enabling AAA changes the authentication method used by the VTY (telnet) lines to fit the AAA model. It's a good idea to define a local username and password to authenticate to the switch if you haven't done so (this account is unrelated to our 802.1X configuration, just a way for us to log in again if we need to).

Switch(config)# aaa new-model
Switch(config)# username admin secret MyPassword

Next we'll configure the switch with the address and shared key of our RADIUS server. By default, Cisco switches will use UDP port 1645 for RADIUS authentication and port 1646 for accounting. Depending on the RADIUS daemon you chose to implement, you may need to modify these ports to match those used by your RADIUS daemon. FreeRADIUS, for example, uses the more recent port specification defined in RFC 2138, and requires additional configuration on the switch to reflect the port changes:

Switch(config)# radius-server host 10.0.0.100 auth-port 1812 acct-port 1813 key
 MyRadiusKey

Now we'll tie these two components together by configuring AAA to reference the RADIUS server for 802.1X authentication requests:

Switch(config)# aaa authentication dot1x default group radius

This takes care of the RADIUS portion of the configuration. Configuring 802.1X from this point is simple: enable it globally for the switch, and individually per interface:

Switch(config)# dot1x system-auth-control
Switch(config)# interface g0/12
Switch(config-if)# switchport mode access
Switch(config-if)# dot1x port-control auto

Note that the interface must be set to static access mode. If left in dynamic mode (where DTP is used to negotiate the port's function as either access or trunking), the switch will issue an error message stating that 802.1X cannot be configured on dynamic ports.

If you're inquisitive like me and issue a question mark to invoke the context-sensitive help in the midst of issuing a new command, you might have noticed that the dot1x port-control interface command has three options. These are:

  • auto - Normal 802.1X authentication
  • force-authorized - No 802.1X authentication is used (this is the default setting, to prevent service interruption while deploying 802.1X)
  • force-unauthorized - Ignores authentication attempts, port is always unauthorized

You can use the show dot1x command to verify the configuration of your client-facing interface:

Switch# show dot1x interface g0/12
Supplicant MAC <Not Applicable>
   AuthSM State      = N/A
   BendSM State      = N/A
PortStatus        = N/A
MaxReq            = 2
MaxAuthReq        = 2
HostMode          = Single
PortControl       = Auto
QuietPeriod       = 60 Seconds
Re-authentication = Disabled
ReAuthPeriod      = 3600 Seconds
ServerTimeout     = 30 Seconds
SuppTimeout       = 30 Seconds
TxPeriod          = 30 Seconds
Guest-Vlan        = 0

Client Configuration

The last element to configure is the supplicant software on the client. If your client is currently connected, unplug it temporarily before continuing (reconnecting after the configuration has been completed will make it easier to observe the 802.1X behavior). For my lab, I used a Windows XP box with SP2.

To enable the Windows 802.1X service, open Services from the control panel, and select and start the Wireless Zero Configuration service. ("But isn't this a wired connection?" I hear you ask. Thank you, Microsoft.) (Edit: Wired 802.1X is enabled by a separate service, Wired AutoConfig, in XP SP3. Thanks to Dude for pointing this out!) Next, open Network Connections from the control panel and open the Connection Properties dialog for the adapter you're using. You should have an Authentication tab within this window; if not, the 802.1X service isn't running and you'll need to do some troubleshooting.

Connection properties dialog

Enable 802.1X authentication and set the EAP type to MD5-Challenge. This will allow us to use basic username/password credentials instead of a more secure (and much more complex) PKI scheme. You can safely deselect the "authenticate as computer" and "authenticate as guest" options.

Authenticating

If everything is configured correctly, you should now be able to authenticate via 802.1X. Verify your IP addressing and connect your client to the switch. After roughly thirty seconds you should be prompted for authentication credentials by a little balloon. (In a more ideal setup, your Windows credentials and/or a client certificate would be sent without your interaction and 802.1X authentication would occur transparently.) Your prompt may differ from the example shown here.

Authentication prompt

Enter the username and password you configured on the RADIUS server in the authentication dialog.

Authentication dialog

Your client will notify you after a bit if the authentication fails. If you receive no notification, authentication has succeeded and you should be able to send traffic through the switch port (try pinging through to the RADIUS server to verify this). Issue the show dot1x command on the switch again to verify that the port is now in the "authorized" state.

Switch# show dot1x interface g0/12
Supplicant MAC 0014.22e9.545e
   AuthSM State      = AUTHENTICATED
   BendSM State      = IDLE
PortStatus        = AUTHORIZED
MaxReq            = 2
MaxAuthReq        = 2
HostMode          = Single
PortControl       = Auto
QuietPeriod       = 60 Seconds
Re-authentication = Disabled
ReAuthPeriod      = 3600 Seconds
ServerTimeout     = 30 Seconds
SuppTimeout       = 30 Seconds
TxPeriod          = 30 Seconds
Guest-Vlan        = 0

One other detail to note: Initially, the client's port on the switch will only transition to up/down (interface up, line protocol down) when you first connect. Only after successfully authenticating via 802.1X will it transition fully to up/up.

If authentication fails for some reason you'll have to do some sleuthing to determine the cause. Keep the following tips in mind:

  • Ensure that the switch is trying to authenticate to the correct RADIUS server on the correct UDP port
  • Ensure the RADIUS server is configured to accept authentication requests from the correct subnet
  • Review the RADIUS daemon logs for messages concerning failed authentication or misconfiguration
  • Use a variation of the debug dot1x command on the switch or a packet sniffer to verify EAP and RADIUS traffic
  • Try using the free NTRadPing RADIUS Test Utility to independently verify operation of the RADIUS server

EUI-64 in IPv6

Posted by stretch in Networking on Monday, 4 Aug 2008 at 2:13 a.m. GMT

One of IPv6's key benefits over IPv4 is its capability for automatic interface addressing. By implementing the IEEE's 64-bit Extended Unique Identifier (EUI-64) format, a host can automatically assign itself a unique 64-bit IPv6 interface identifier without the need for manual configuration or DHCP. This is accomplished on Ethernet interfaces by referencing the already unique 48-bit MAC address, and reformatting that value to match the EUI-64 specification.

RFC 2373 dictates the conversion process, which can be described as having two steps. The first step is to convert the 48-bit MAC address to a 64-bit value. To do this, we break the MAC address into its two 24-bit halves: the Organizationally Unique Identifier (OUI) and the NIC specific part. The 16-bit hex value 0xFFFE is then inserted between these two halves to form a 64-bit address.

Step 1

Why 0xFFFE? As explained in the IEEE's Guidelines for EUI-64 Registration Authority, this is a reserved value which equipment manufacturers cannot include in "real" EUI-64 address assignments. In other words, any EUI-64 address having 0xFFFE immediately following its OUI portion can be recognized as having been generated from an EUI-48 (or MAC) address.

The second step is to invert the universal/local (U/L) flag (bit 7) in the OUI portion of the address. Globally unique addresses assigned by the IEEE originally have this bit set to zero, indicating global uniqueness. Likewise, locally created addresses, such as those used for virtual interfaces or a MAC address manually configured by an administrator, will have this bit set to one. The U/L bit is inverted when using an EUI-64 address as an IPv6 interface ID.

Step 2

Again, you're probably wondering why this is done. The answer lies buried in section 2.5.1 of RFC 2373:

The motivation for inverting the "u" bit when forming the interface identifier is to make it easy for system administrators to hand configure local scope identifiers when hardware tokens are not available. This is expected to be case for serial links, tunnel end-points, etc. The alternative would have been for these to be of the form 0200:0:0:1, 0200:0:0:2, etc., instead of the much simpler ::1, ::2, etc.

The important part to remember here is that the scope of the address never changes: global addresses are still global and local addresses are still local. Rather, the meaning of the bit is inverted for convenience, so the value of the bit must be inverted as well.

We can see this conversion in action when we assign an IPv6 address to a router interface. First, take note of the interface's MAC address (this is typically the same as its burned-in address, or BIA).

Router# show interface f0/0
FastEthernet0/0 is up, line protocol is down
  Hardware is Gt96k FE, address is 0012.7feb.6b40 (bia 0012.7feb.6b40)
...

After assigning an EUI-64-designated IPv6 address to the interface, we can verify that the interface ID has been drawn from the MAC address in the process described for both the assigned and the link local address:

Router(config)# interface f0/0
Router(config-if)# ipv6 address 2001:db8::/64 eui-64
Router(config-if)# do show ipv6 interface f0/0
FastEthernet0/0 is up, line protocol is down
  IPv6 is enabled, link-local address is FE80::212:7FFF:FEEB:6B40 [TEN]
  No Virtual link-local address(es):
  Global unicast address(es):
    2001:DB8::212:7FFF:FEEB:6B40, subnet is 2001:DB8::/64 [EUI/TEN]
...

Removing line jumps

Posted by stretch in Network Diagrams on Saturday, 2 Aug 2008 at 12:36 a.m. GMT

A cardinal rule of network diagramming is to avoid intersecting lines wherever possible. Line intersections, particularly many in close proximity, make casual inspection difficult. Of course, sometimes such intersections cannot be avoided.

Line jumps

Microsoft Visio adds line "jumps" to more clearly indicate flow, but these aren't always desirable. To prevent a line from jumping at intersections, select it and navigate to Format -> Behavior and select the Connector tab. Here we have the option to modify the placement of line jumps. We can opt to always add jumps to the selected line, to always add jumps to the lines intersected, or to remove them completely.

Connector dialog

After setting line jump addition to "Never" our drawing looks a bit more tidy:

No line jumps

You might have noticed there are several styles of line jump available beside the default arc. Try out some of these and see if any better suite your style (I recommend "gap" for a minimalist feel).

Putting the twit in twitter

Posted by stretch in Announcements on Thursday, 31 Jul 2008 at 11:06 a.m. GMT

Not wanting to be left out of the loop, I decided to try out Twitter. You can find my ramblings at twitter.com/packetlife. Though to be honest, I'm not sure why you'd want to. I'm not a very interesting person.

Policing versus shaping

Posted by stretch in Networking on Wednesday, 30 Jul 2008 at 9:43 a.m. GMT

There are two methods to limit the amount of traffic originating from an interface: policing and shaping. When an interface is policed outbound, traffic exceeding the configured threshold is dropped (or remarked to a lower class of service). Shaping, on the other hand, buffers excess (burst) traffic to transmit during non-burst periods. Shaping has the potential to make more efficient use of bandwidth at the cost of additional overhead on the router.

All this is just dandy, but doesn't mean much until you see its effects on real traffic. Consider the following lab topology:

Client and server separated by R1

We'll be using Iperf on the client (192.168.10.2) to generate TCP traffic to the server (192.168.20.2). In the middle is R1, a Cisco 3725. Its F0/1 interface will be configured for policing or shaping outbound to the server.

Iperf

Iperf is able to test the bandwidth available across a link by generating TCP or UDP streams and benchmarking the throughput of each. To illustrate the effects of policing and shaping, we'll configure Iperf to generate four TCP streams, which we can monitor individually. To get a feel for how Iperf works, let's do a dry run before applying any QoS policies. Below is the output from Iperf on the client end after running unrestricted across a 100 Mbit link:

Client$ iperf -c 192.168.20.2 -t 30 -P 4
------------------------------------------------------------
Client connecting to 192.168.20.2, TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[1916] local 192.168.10.2 port 1908 connected with 192.168.20.2 port 5001
[1900] local 192.168.10.2 port 1909 connected with 192.168.20.2 port 5001
[1884] local 192.168.10.2 port 1910 connected with 192.168.20.2 port 5001
[1868] local 192.168.10.2 port 1911 connected with 192.168.20.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[1900]  0.0-30.0 sec  84.6 MBytes  23.6 Mbits/sec
[1884]  0.0-30.0 sec  84.6 MBytes  23.6 Mbits/sec
[1868]  0.0-30.0 sec  84.6 MBytes  23.6 Mbits/sec
[1916]  0.0-30.0 sec  84.6 MBytes  23.6 Mbits/sec
[SUM]  0.0-30.0 sec   338 MBytes  94.5 Mbits/sec

Iperf is run with several options:

  • -c - Toggles client mode, with the IP address of the server to contact
  • -t - The time to run, in seconds
  • -P - The number of parallel connections to establish

We can see that Iperf is able to effectively saturate the link at around 95 Mbps, with each stream consuming a roughly equal share of the available bandwidth.

Policing

Our first test will measure the throughput from the client to the server when R1 has been configured to police traffic to 1 Mbit. To do this we'll need to create the appropriate QoS policy and apply it outbound to F0/1:

policy-map Police
 class class-default
  police cir 1000000
!
interface FastEthernet0/1
 service-policy output Police

We can then inspect our applied policy with show policy-map interface. F0/1 is being policed to 1 Mbit with a 31250 bytes burst:

R1# show policy-map interface
 FastEthernet0/1

  Service-policy output: Police

    Class-map: class-default (match-any)
      2070 packets, 2998927 bytes
      5 minute offered rate 83000 bps, drop rate 0 bps
      Match: any
      police:
          cir 1000000 bps, bc 31250 bytes
        conformed 1394 packets, 1992832 bytes; actions:
          transmit
        exceeded 673 packets, 1005594 bytes; actions:
          drop
        conformed 57000 bps, exceed 30000 bps

Repeating the same Iperf test now yields very different results:

Client$ iperf -c 192.168.20.2 -t 30 -P 4
------------------------------------------------------------
Client connecting to 192.168.20.2, TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[1916] local 192.168.10.2 port 1922 connected with 192.168.20.2 port 5001
[1900] local 192.168.10.2 port 1923 connected with 192.168.20.2 port 5001
[1884] local 192.168.10.2 port 1924 connected with 192.168.20.2 port 5001
[1868] local 192.168.10.2 port 1925 connected with 192.168.20.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[1884]  0.0-30.2 sec   520 KBytes   141 Kbits/sec
[1916]  0.0-30.6 sec  1.13 MBytes   311 Kbits/sec
[1900]  0.0-30.5 sec   536 KBytes   144 Kbits/sec
[1868]  0.0-30.5 sec   920 KBytes   247 Kbits/sec
[SUM]  0.0-30.6 sec  3.06 MBytes   841 Kbits/sec

Notice that although we've allowed for up to 1 Mbps of traffic, Iperf has only achieved 841 Kbps. Also notice that, unlike our prior test, each flow does not receive an equal proportion of the available bandwidth. This is because policing (as configured) does not recognize individual flows; it merely drops packets whenever they threaten to exceed the configured threshold.

Using Wireshark's IO graphing feature on a capture obtained at the server, we can observe the apparently random nature of the flows. The black line measures the aggregate throughput, and the colored lines each represent an individual TCP flow.

Policed throughput

Shaping

In contrast to policing, we'll see that shaping handles traffic in a very organized, predictable manner. First we'll need to configure a QoS policy on R1 to shape traffic to 1 Mbit. When applying the Shape policy outbound on F0/1, be sure to remove the Police policy first with no service-policy output Police.

policy-map Shape
 class class-default
  shape average 1000000
!
interface FastEthernet0/1
 service-policy output Shape

Immediately after starting our Iperf test a third time we can see that shaping is taking place:

R1# show policy-map interface
 FastEthernet0/1

  Service-policy output: Shape

    Class-map: class-default (match-any)
      783 packets, 1050468 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: any
      Traffic Shaping
           Target/Average   Byte   Sustain   Excess    Interval  Increment
             Rate           Limit  bits/int  bits/int  (ms)      (bytes)
          1000000/1000000   6250   25000     25000     25        3125

        Adapt  Queue     Packets   Bytes     Packets   Bytes     Shaping
        Active Depth                         Delayed   Delayed   Active
        -      69        554       715690    491       708722    yes

This last test concludes with very consistent results:

Client$ iperf -c 192.168.20.2 -t 30 -P 4
------------------------------------------------------------
Client connecting to 192.168.20.2, TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[1916] local 192.168.10.2 port 1931 connected with 192.168.20.2 port 5001
[1900] local 192.168.10.2 port 1932 connected with 192.168.20.2 port 5001
[1884] local 192.168.10.2 port 1933 connected with 192.168.20.2 port 5001
[1868] local 192.168.10.2 port 1934 connected with 192.168.20.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[1916]  0.0-30.4 sec   896 KBytes   242 Kbits/sec
[1868]  0.0-30.5 sec   896 KBytes   241 Kbits/sec
[1884]  0.0-30.5 sec   896 KBytes   241 Kbits/sec
[1900]  0.0-30.5 sec   896 KBytes   241 Kbits/sec
[SUM]  0.0-30.5 sec  3.50 MBytes   962 Kbits/sec

With shaping applied, Iperf is able to squeeze 962 Kbps out of the 1 Mbps link, a 14% gain over policing. However, keep in mind that the gain measured here is incidental and very subject to change under more real-world conditions. Also notice that each stream receives a fair share of bandwidth. This even distribution is best illustrated graphically through an IO graph of a second capture:

Shaped throughput

MPLS cheat sheet

Posted by stretch in Announcements on Monday, 28 Jul 2008 at 9:07 a.m. GMT

MPLS cheat sheet

Added another cheat sheet to the pile, this one on Multiprotocol Label Switching, or MPLS. I have to admit that although I was a bit skeptical of its application when first reading about MPLS, I've grown to love the technology for its simplicity and for the very interesting approaches it offers to common problems. One such solution is described in an earlier post, Getting to know MPLS, which discusses how MPLS can be used to transport customer traffic across a provider's network without the P routers having to maintain customer routes (a very common scenario).

As always, write in with any errors you find and I'll correct them ASAP. Comments and constructive criticism are also welcome. Nonconstructive criticism should be sent in the form of cash.

Disabling password recovery

Posted by stretch in Networking on Saturday, 26 Jul 2008 at 7:11 a.m. GMT

The decision by a San Francisco network admin to hold the city's network hostage has received a lot of media attention recently, but not much has been said about this was technically accomplished. The article linked above explains that the admin had disabled password recovery on key Cisco IOS devices:

According to an affidavit from James Ramsey, an inspector with the San Francisco Police Department, he and other investigators discovered dial-up and DSL modems that would allow an unauthorized connection to the FiberWAN. He also found that Childs had configured several of the Cisco devices with a command that would erase critical configuration data in the event that anyone tried to restore administrative access to the devices, something Ramsey saw as dangerous because no backup configuration files could be found.

Disabling the password recovery service prevents anyone with physical access to a device to obtain its configuration via the normal password recovery procedure. Although devices can be power-cycled normally and remain accessible with the right credentials, anyone attempting to perform password recovery is forced to wipe all configuration from the device before it can be utilized.

Clearly a dangerous command, it is hidden from the CLI context-sensitive help (at least in the IOS tested):

R1(config)# no service pass?
password-encryption

Disabling password recovery:

R1(config)# no service password-recovery
WARNING:
Executing this command will disable password recovery me
chanism.
Do not execute this command without another plan for
password recovery.

Are you sure you want to continue? [yes/no]: y
R1(config)# ^Z
R1# reload
Proceed with reload? [confirm]

System Bootstrap, Version 12.2(8r)T2, RELEASE SOFTWARE (fc1)
TAC Support: http://www.cisco.com/tac
Copyright (c) 2002 by cisco Systems, Inc.
c3725 processor with 262144 Kbytes of main memory

Main memory is configured to 64 bit mode with parity disabled

Readonly ROMMON initialized

PASSWORD RECOVERY FUNCTIONALITY IS DISABLED
...

While disabling password recovery can add a valuable layer of security, always ensure you have secured backups stored elsewhere for any devices on which it is implemented.

Childs' play

Posted by stretch in Opinion on Friday, 25 Jul 2008 at 4:56 a.m. GMT

The recent fiasco involving a San Francisco network admin has been making headlines for over a week now. Public opinion can be found both in favor of and against Terry Childs, who has been jailed since July 13th for (until recently) holding hostage administrative access to the city's WAN infrastructure. Many sympathize with his situation in dealing with incompetent management, while others assert that he should be prosecuted for treating a metropolitan network as his own. Of course, anyone with experience in the field will immediately realize that both parties are at fault.

In one corner, you have a clearly negligent network admin. Even if his motivations for denying all others control of the network were benevolent -- and this is debatable -- one has to wonder if he considered himself invulnerable to harm. In what position would the city be left if he had an unfortunate encounter with a speeding bus? With no suitable personnel to replace him, and in the apparent absence of any documentation, how did he suppose the network would fare?

In the other corner, the ineptitude of the city's IT management is difficult to ignore. How do they excuse having relied exclusively on a single individual for the operation and maintenance of critical infrastructure? Why weren't these issues addressed before escalating to such a degree? A number of people higher on the chain than Childs should also be looking for a new job.

This conflict isn't so much about demarcation of responsibility as it is about ego, but it does serve as reminder that as networkers, we must mentally decouple our expertise from the networks we maintain. Territorial behavior not only damages the health of a network and the organization which owns it, but can hinder an engineer's ability to adapt to new settings when the time comes.