I was recently tasked with configuring a number of 24-port Catalyst 2960S switches for deployment as standalone iSCSI switches for a storage area network (SAN). I haven't dealt much with SAN architecture so I wasn't sure what was needed. Obviously, just about any switch will support iSCSI right out of the box (it's just TCP/IP traffic, after all), but there are certain tweaks necessary to achieve the best possible performance.
Dell's PS Series Array Network Performance Guidelines outlines its recommendations for EqualLogic SAN arrays, including network configuration. This article parallels the Network Requirements and Recommendations section of that document.
Evaluating Switch Performance
According to the Catalyst 2960S data sheet, the 2960S-24xS-L series is capable of moving 41.7 Mpps worth of 64-byte packets, and its forwarding ceiling of 88 Gbps (which we could never reach with 24 GigE ports) leaves plenty of headroom. Since the traffic traversing this switch will be mostly iSCSI, which uses very long frames, the overall forwarding rate is much more important to us than the 64-byte packets-per-second limit (which is fairly high anyway).
Similar to their big brother, the Catalyst 3750, multiple 2960S switches can be combined into a single managed unit through the use of proprietary stacking cables. Although the requirement here is for only a single switch, it's worth keeping in mind that the stack backplane introduces a potential 20 Gbps choke point for traffic switched among stack members. Obviously, this is more of a concern regarding 48-port switches than it is for 24-port switches.
From the Dell tech report:
Because STP can increase the time it takes to recover from a PS Series array control module failover or a network switch failure, Dell recommends that you do not use STP on switch ports that connect end nodes (iSCSI initiators or storage array network interfaces).
The above quote is a great example of why you should never blindly follow networking advice from server admins. There is a far more elegant way to address this issue than simply turning off spanning tree (which the guide does reluctantly acknowledge). First, because the default spanning tree protocol on the 2960S is still legacy IEEE 802.1D-1998, we'll enable rapid spanning tree (IEEE-802.1D-2004, which includes 802.1w):
Switch(config)# spanning-tree mode rapid-pvst Switch(config)# ^Z Switch# show spanning-tree VLAN0001 Spanning tree enabled protocol rstp Root ID Priority 32768 Address 0026.622f.4788 Cost 38 Port 23 (GigabitEthernet1/0/23) Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec ...
Next, we'll issue the command
spanning-tree portfast on all interfaces which connect to endpoints; that is, anything other than another switch. This command designates an interface as an edge port and reduces the time it takes to transition to forwarding state to about a second.
Switch(config)# interface range g1/0/1 -22 Switch(config-if-range)# spanning-tree portfast %Warning: portfast should only be enabled on ports connected to a single host. Connecting hubs, concentrators, switches, bridges, etc... to this interface when portfast is enabled, can cause temporary bridging loops. Use with CAUTION %Portfast will be configured in 22 interfaces due to the range command but will only have effect when the interfaces are in a non-trunking mode.
Dell's recommendation is to enable bidirectional flow control on all interfaces. Unfortunately, the 2960S supports only the reception of flow control signals; it won't generate PAUSE frames itself. We'll enable negotiation of the "receive" direction of flow control on all server- and SAN-facing interfaces:
Switch(config-if-range)# flowcontrol receive desired
Unicast Storm Control
The recommendation here is to disable unicast storm control. This feature is actually disabled by default on our switch, so we can move right along.
You can verify that storm control has been disabled with the command
show storm-control; no interfaces should be listed.
Switch# show storm-control Interface Filter State Upper Lower Current --------- ------------- ----------- ----------- ----------
"Jumbo frame" refers to an Ethernet frame which has a maximum transmission unit (MTU) of greater than 1500 bytes, typically around 9000 bytes. It is recommended to enable jumbo frames on all hosts and switches comprising the iSCSI network because available throughput is consumed more efficiently using longer frames (a lower header-to-payload ratio).
When dealing with GigE Catalyst switches, there are two classifications of MTU with which we are concerned: the system MTU and the jumbo MTU. The system MTU applies to frames which are processed by the switch CPU (e.g. SSH, SNMP, etc.). The system MTU has a limitation of 1998 bytes and is set to 1500 bytes by default.
The jumbo MTU applies to transit traffic (traffic which is received on one interface and switch out via another) and is also set to 1500 bytes by default. However, we can raise the jumbo MTU to a maximum of 9198 bytes. We'll set it to 9000 bytes as that seems to be the standard.
Switch# show system mtu System MTU size is 1500 bytes System Jumbo MTU size is 1500 bytes System Alternate MTU size is 1500 bytes Routing MTU size is 1500 bytes Switch# configure terminal Switch(config)# system mtu ? MTU size in bytes jumbo Set Jumbo MTU value for GigabitEthernet or TenGigabitEthernet interfaces Switch(config)# system mtu jumbo ? Jumbo MTU size in bytes Switch(config)# system mtu jumbo 9000 Changes to the system jumbo MTU will not take effect until the next reload is done
Note that it is necessary to reload the switch in order for the new MTU to take effect. Note also that the MTU setting does not appear in the running or startup configuration; this is important to remember when replicating configurations among multiple switches.
Remember to verify that the new jumbo MTU has taken effect after a reload:
Switch# show system mtu System MTU size is 1500 bytes System Jumbo MTU size is 9000 bytes System Alternate MTU size is 1500 bytes Routing MTU size is 1500 bytes
VLANs and QoS
Use VLANs to segment iSCSI traffic into separate layer two domains as appropriate. Limiting each VLAN to a single IP network is a good rule of thumb but your SAN design may dictate otherwise.
Dell recommends not implementing QoS on iSCSI networks and I have to agree. Your storage network should only carry storage traffic, all of which you want to move from source to destination as fast as possible. Even management traffic to and from the switches themselves is best isolated to the out-of-band management interface.
That's what I've managed to come up with, anyway. What have I missed?