Catalyst 2960S iSCSI Optimization

Thanks for the article Stretch.

I am interested in how you'd manage performance monitoring, to see if these config changes were improving things. Would iperf be the tool to reach for? I note netflow isn't supported on 2960, and perhaps it is the wrong tool anyway.

I would think that switching on jumbo frames would have make a very noticeable difference, but I'm not sure how to prove that.

I also wonder whether your disks will be able to even reach the maximum throughput on each server NIC and switch port.

thanks Stretch. Good article.

Keep up the good work

Greets
Belgium

Great article!

I had such a deployment with the exact same switches (except it was 2 48 ports).

On our side we used HP P2000 G3 SAN with 8 host ports.

We had a problem with host ports going down then up again during a few seconds. HP's solution was: disable auto negotiation on ports were san is plugged.

So far, so good (even if i keep thinking that disabling auto negotiation is not a best practice).

Anyway, great article!

Regards

Shure you want to use a low-end cisco switch for iSCSI traffic at all?

Quote from a recent cisco-nsp post:

On the 2960 the buffer depth is 77 or 78.
On the 2960S the buffer depth is closer to 40 (don't have an exact figure).
The buffers on these are extremely small for a gig/10GE switch (less than 1ms).

Outbound flow control is necessary with a blocking backplane only, when input buffers are congested. Most Cisco switches (like the 2960S) have a non-blocking backplane and no input buffers at all, thus outbound flow control doesn't make any sense.

Please also read Vendors on flow control.

Flow control is not intended to solve the problem of steady-state overloaded networks or links

Of course you can saturate the intra-stack links, but for storage with stacked switches I would stick to 3750X series anyway. When using 2960S in a stack, intra-stack bandwidth requirements needs to be considered. I wouldn't use stacked 2960S in environments with high-bandwidth requirements like storage anyway.

As for disabling auto negotiation is best pratice: this is a myth! In fact, flow control requires auto negotiation. Also read Blessay: Autonegotiation on Ethernet – It Works, It Should Be Mandatory!

The above quote is a great example of why you should never blindly follow networking advice from server admins

I can't agree more with you on this, but this is valid also for the networking 'myths' we still see floating around in the internet and real life, like auto-negotation.

Thanks for the post Stretch.

My company uses the 2960G at three sites for EqualLogic SANs and they have performed just fine though on a few support calls with EQL they have started down the road of pointing at the switch as "not on their supported hardware list" so that has been a concern in the back of my mind.

We've set up our switches pretty much the same way outlined here and I see you referenced Dell's guidelines so nothing much new but it is good to see that others are using these switches in similar scenarios.

Some other notes:

We have the switches set up in a redundant fashion, utilizing EtherChannel between switches and I'm considering using the stackable flavor in a new implementation but I'm concerned about the bandwidth and am currently investigating a single chassis (Force 10) in lieu of multi chassis stacked or EtherChannel... I'm wondering if anyone has any comments on the pro / con here, some people really seem to hate stackables but two large chassis for true redundancy just isn't in the budget - is a single large chassis with dual everything good enough??

Also, you mentioned QOS and that made me think of some problems we have that are worth mentioning. In order to prevent bandwidth overages I use ingress policing on the next hop switch in two sites to rate limit. This is maybe more of an EQL issue but it seems to wreck havoc with the communication between groups at different sites, they constantly balk about timeouts, etc. I wish EQL would build in a rate limit feature for replication then I wouldn't need to see a bunch of misleading errors about re-transmits.

D Stromblad said: "jumbo traffic on an Gig interface that is destined for a FE interface it will be dropped"

Citation please?

We always use HP Procurve 2910-AL switches for our SAN. But it's worth to take a look at the 2960S switches.

Great Article btw

Here's a great link from cisco http://www.cisco.com/en/US/products/hw/switches/ps700/products_configuration_example09186a008010edab.shtml#c4a

Chrismarget: this is were I found that info "If Gigabit Ethernet interfaces are configured to accept frames greater than the 10/100 interfaces, jumbo frames received on a Gigabit Ethernet interface and sent on a 10/100 interface are dropped."

I see you existed config mode to run a show command:
Switch(config)# spanning-tree mode rapid-pvst
Switch(config)# ^Z
Switch# show spanning-tree

You are aware you can do the following right?:
Switch(config)# spanning-tree mode rapid-pvst
Switch(config)# do show spanning-tree

Does anyone know of the routing performance in the 2960's? when you change the SDM mode to routing, does this happen in software or hardware?

@g1: Yes, I'm very familiar with the 'do' command. ;) But it can throw off people who aren't well-versed in IOS command syntax when used in examples.

http://packetlife.net/wiki/ios-tips/#do-the-do

"Does anyone know of the routing performance in the 2960's? when you change the SDM mode to routing, does this happen in software or hardware?"

I haven't heard of 2960S's before now, but I'm assuming they're still L2 only switches? As such, I'm also assuming there's no SDM and / or "routing" issues / concerns - right? I guess I can always visit Cisco's website...

At my site we are working with 3750's and EMC SAN. We had some issues, too, and it seems that the 9000 MTU size is standard.

Your recommendations are right on target. We are moving from 3750's to Nexus 5K soon to minimize latency on the iSCSI side. So, from where I stand, you got it!

Yes the 2960(S) are still considered L2 from Cisco, but they support static unicast routing on SVI: http://www.cisco.com/en/US/docs/switches/lan/catalyst2960/software/release/12.2_55_se/configuration/guide/swipstatrout.html

I was hoping someone could confirm if the switches could handle full throughput when being used as a router?

I would recommend ammending your QoS piece, and seeing about doing something that is a VLAN based QoS, so that your iSCSI VLAN would take priority when traversing trunks.

This might be good for a Layer 3 switch article...

I'm in the middle of looking at switches for a HP Lefthand iSCSI solution, and the bloated marketing numbers in spec sheets from the vendors have to be translated to reality.

I know at least the following items need to be considered: blocking, flow-control, queueing method and depth, latency, and backplane bandwidth.

Switches under consideration are HP 2910AL, Cisco 4948, and a Cisco WS-X6748-GE-TX 6500 blade.

Jumbo frames are also being considered, but I wasn't a fan of making a system mtu change that required a reload on my aggregation switches and apparently my Cisco 6500s already default jumbo frames to 9216.

s-oc4-n2-agg1 uptime is 5 years, 2 weeks, 3 hours, 41 minutes
...
s-oc4-n2-agg1(config)#system jumbomtu ?
    Jumbo mtu size in Bytes, default is 9216

This article is well written and exactly what one needs to do on 2960-S to prepare for iSCSI; particularly EQL iSCSI.

There is a big miss though.

The 2960-S is an access switch. Over-saturate it, encouter problems with it, and you'll end up with a TAC rep telling you that you're using an access switch in the distribution layer. They will be of little help.

Skip the 2960-S as an iSCSI switch. A low to medium storage load will bring it to its knees. The inability to xmit pause frames keeps it from tapping out when an array or initiator (or both) blast it with packets. It has an issue with buffer miss representation that makes it nearly impossible to troubleshoot.

Why pay $60k/array and hook them to a $3k switch? You're asking for heartbreak.

Breakfix is correct. The 2960s (G and S) do not have enough internal buffers to handle the overlapping 300ms bursts of 1Gbps traffic that occur on iSCSI Sans. It's fine as long as only one iSCSI stream is active on a given port, but (for example) if a host requests data from two or more iSCSI LUNs using the same NIC port on the host, the concurrent 1Gb streams from the LUNs fill the internal 2960 buffers causing drops.

If you are running iSCSI on an older 2960G and you are not seeing QDROPs on the interface displays, it's because your IOS is old and not counting platform drops. Use "sh platform port-asic stats drop" and you will see them.

Supposedly, the 3750/49xx etc have 32MB or more of internal buffering which is right around 300ms of 1Gbps traffic.

I use 2960's for iSCSI and they work flawless. To get more bang for the buck, don't forget to tune your queues so that ALL the buffers are lumped into one queue. That will not only consolidate your available buffers, but you can also set it so the reserved per-port buffers are minimal, allowing them to shift as needed between ports.

This can be done with the following commands:

mls qos mls qos queue-set output 1 threshold 1 100 100 1 400 mls qos queue-set output 1 threshold 2 3200 3200 10 3200 mls qos queue-set output 1 threshold 3 100 100 1 400 mls qos queue-set output 1 threshold 4 100 100 1 400

You'll notice I gave the second queue most of the buffers. That's because this is the default queue that outgoing traffic maps to that doesn't have any QoS markings.

You can verify that the traffic is in fact using the second queue via this command:

show mls qos int gi0/1 stat

at the bottom of the output, you will see the number of packets enqueued. Note that the output shows queues 0-3, however, when you configure it, you use queues 1-4. It's a little confusing at first.

Also, buffers are per ASIC and each switch ASIC is attached to several physical ports. So if you have several high traffic ports that get overrun often because they hook to multiple iSCSI targets, find out which ports group to each ASIC, and distribute the high traffic load between them. That will give you better use of the buffers.