Evaluating Network Gear Performance
By stretch | Thursday, December 20, 2012 at 4:05 a.m. UTC
Choosing the right equipment for your network is hard. Even ignoring the ever-growing roster of features one must account for when evaluating candidate hardware, it's important not to overlook performance limitations. Below I describe some of the most crucial characteristics to consider when doing your research.
Throughput is the rate at which a device can convert input to output. This is different from bandwidth, which is the rate at which data travels across a medium. An Ethernet switch, for example, might have 48 ports running at an individual bandwidth of 1 Gbps each but be able to switch only a total of 12 Gbps among the ports at any given time. This is said to be the switch's maximum throughput.
Throughput is measured in two units: bits per second (bps) and packets per second (pps). Most people are most familiar with bits per second. This is the amount of data which flows through a particular point within a duration of one second, typically expressed as megabits (Mbps) or gigabits (Gbps) per second. Capitalization is important here. A lowercase 'b' indicates bits, whereas an uppercase 'B' indicates bytes. Speed is always measured in bits per second, with a lowercase 'b' (Kbps or Mbps).
Packets per second, similarly expressed most often as Kpps or Mpps, is another way of evaluating throughput. It conveys the number of packets or frames which can be processed in one second. This approach to measuring throughput is used to expose limitations of the processing power of devices, as shorter packets demand more frequent forwarding decisions. For example, a router might claim a throughput of 30 Mbps per second using full-size packets. However, it might also be limited to processing 40 Kpps. If each packet received was the minimum size of 64 bytes (512 bits), the router would be limited to just 20.48 Mbps (512 * 40,000) of throughput.
Cisco maintains often cited baseline performance measurements for its most popular routers and switches. If you work out the math, you can see that the Mbps numbers listed in the router performance document were derived using minimum-length (64 byte) packets. These numbers thus present a worst case scenario. Packets on a production network typically vary widely in size, and larger packets will yield higher bits-per-second rates.
Keep in mind that these benchmarks were taken with no features other than IP routing enabled. Adding additional features and services such as access control lists or network address translation may reduce throughput. Unfortunately, it's impractical for a vendor to list throughput rates with and without myriad features enabled, so you'll have to do some testing yourself.
Ethernet switches are often built with oversubscribed backplanes. Oversubscription refers to a point of congestion within a system where the potential rate of input is greater than the potential rate of output. For example, a switch with 48 1 Gbps ports might have a backplane throughput limitation of only 16 Gbps. This means that only 16 ports can transmit at wire rate (the physical maximum throughput) at any point in time. This isn't usually a problem at the network edge, where few users or servers ever need to transmit at these speeds for a prolonged time. However, oversubscription imposes much more critical considerations in the data center or network core.
As an example, let's look at the 16-port 10 Gbps Ethernet module WS-X6816-10G-2T for the Cisco Catalyst 6500 switch. Although the module provides an aggregate of 160 Gbps of potential throughput, its connection to the chassis backplane is only 40 Gbps. The module is oversubscribed at a ratio of 4:1. This module should only be used in situations where the aggregate throughput demand from all interfaces is not expected to exceed 40 Gbps.
IP Route Capacity
The maximum number of routes a router can hold in its routing table is limited by the amount of available content-addressable memory (CAM). Although a low-end router may be able to run BGP and exchange routes with BGP peers, it likely won't have sufficient memory to accept the full IPv4 Internet routing table, which comprises over 400 thousand routes. (Of course, low-end routers should never be implemented in a position where they would need to receive the full routing table.) Virtual routing contexts, in which a router stores multiple copies of a route in separate forwarding tables, increase routing table size exponentially, further elevating the importance of properly sizing routers for the role they play.
Maximum Concurrent Sessions
Firewalls and intrusion prevention systems perform stateful inspection of traffic transiting from one trust zone to another. These devices must be able to keep up with the demand for throughput not only in terms of bits per second and packets per second but also in the number of concurrent stateful sessions. A single web request might trigger the initiation of one or two dozen TCP connections to various content servers from an internal host. The firewall or IPS must be able to track the state of and inspect potentially thousands of sessions at any point in time. If the device's maximum capacity is reached, attempts to open new sessions may be rejected until a number of current sessions are closed or expire. Such devices are likewise limited in how fast they can create new sessions.
Posted in Design
December 20, 2012 at 4:34 a.m. UTC
thanks stretch, i just can fully agree with you !
December 20, 2012 at 4:36 a.m. UTC
December 20, 2012 at 1:45 p.m. UTC
Since you mentioned TCAM another note to be aware of is that the numbers listed for maximum number of routes is usually and OR not an AND. On the Sup720 you can have (up to) 1 million IPv4 routes and 0 IPv6 routes. Each IPv6 route costs you 2 IPv4 routes. Just something to keep in mind, it's not always mentioned on the datasheets, at least not that way.
December 20, 2012 at 3:31 p.m. UTC
December 20, 2012 at 3:50 p.m. UTC
Hey stretch,you do not post on your blog very often as compared to your past frequency of posting articles.
December 21, 2012 at 3:40 a.m. UTC
hi Stretch, I just want to catch your attention for a moment, I just finished my university materials, and won a Cisco contest, called "Cisco Netriders", I won over all my country and I'm going to Cisco campus on San Jose Califorina next january and febraury. And why I'm telling that, just to thank you, a lot of your posts helped me on my general study, so thanks, and great post.
P.D. The contest were for members of the cisco networking academy at the CCNA level
December 22, 2012 at 5:05 a.m. UTC
@chuco21 Congratulations! I'm happy the blog helped you!
December 22, 2012 at 3:39 p.m. UTC
Trust but verify. iperf is an easy to use client/server packet blaster with stats.
December 23, 2012 at 7:45 p.m. UTC
I don't how you find time to work, be productive, do personal study and then make time to share all that experience!!
Just wanted to take time to thank you for everything you do and wish you the best of holidays for you and your family!!
December 28, 2012 at 3:13 p.m. UTC
I think another thing that should be considered along with throughput is the overall forwarding architecture. Platforms like the Cisco 6500/7600 have a distributed forwarding architecture and in the case of many non-punted features can get up to the backplane connected bandwidth per card. While a platform like the ASR1000 with centralized forwarding usually is limited by the ESP. For the ASR, even for traffic forwarded between two interfaces on the same SPA, it will still traverse the ESP and consumes some of that resource. This requires some consideration for features like multicast replication, where say 1 to 8 in to out replication requires 8 times the ESP bandwidth as the input stream. Finally, in the data center there are also serious considerations on the end to end forwarding delay, where extremely low speed port to port delay is important for things like clustered computing and high-speed trading applications.
December 29, 2012 at 2:07 p.m. UTC
Thanks for the valuable article. I am trying to become stronger with network design/architecture and this was a helpful article.
January 4, 2013 at 5:33 p.m. UTC
Thanks for your article. This article is very helpful information for me.
January 9, 2013 at 6:53 p.m. UTC
You might also want to consider additional services such as a large QoS policy, or Netflow can greatly reduce the throughput of a router.
For instance a 2811 is very capable of 3M throughput (2xT1s) up to 100% saturation of the WAN link. However, were you to enable per se QoS, Netflow, and some other services I've seen the throughput drop to about 1.5-2Mb (assuming full sized packets). After said throughput over-subscription, the CPU usually spikes at 100, and hovers around 85% or higher.
Also the ISR G2's throughput aren't affected as adversely by the enabling of aforementioned services due to (based on my understanding) of these features being imbedded into the silicon as opposed to being software based (pre ISR G2 series).
January 10, 2013 at 9:44 a.m. UTC
good post . have you ever written or plan to write about how best to test our own hardware ? Just think it could be a very interesting topic.
January 27, 2013 at 7:38 a.m. UTC
Great tips , really great , if possible compare different vendor's
products for performance evaluation , and tell us little about hot to
generate test traffic on our network gear
March 15, 2013 at 7:33 a.m. UTC
Great article ..... Carry on the good job
October 23, 2013 at 8:03 p.m. UTC
Is there any way to calculate what the forwarding rate will be with 1500 bytes packets? Since the article stated that it is calculated for worst case scenario of 64 byte packets. It there a way to see how much the router/switch will be able to forward with 1500 bytes big packets? Will it be the same amount of pps and a bigger number of bps?
October 23, 2014 at 9:03 p.m. UTC
Great article - Never read it explained so clearly and easy. Takes Cisco at least 40 pages of kill yourself of boredom typeface to explain the same thing.
January 11, 2015 at 9:55 p.m. UTC
Just another note as well. A 1Gbps port is able to do 1Gbps in and out at the same time = 2Gbps. For a 48 port switch to not be oversubscribed, the backplane needs atleast 96Gbps (4948).
June 10, 2015 at 10:11 a.m. UTC
Very well explained!