Some Cisco Nexus Design Considerations

I've been involved with a moderate datacenter deployment of Cisco Nexus switches over the past couple months, and I have learned a good deal about the architecture along the way. Here are some of the design considerations I've encountered, and my preferred solution to each.

Drawing the Layer Three Boundary

I work for a managed services provider which offers datacenter colocation, including private line and VPN connectivity to customer sites. One of the primary requirements for network design in this field is to maintain strict customer delineation across all segments of the network. The typical go-to tools for traffic isolation are IEEE 802.1Q VLANs at layer two and MPLS/VRFs at layer three. It is preferable to extend layer three (MPLS) forwarding as close to the datacenter edge as possible, to take advantage of the flexibility and inexpensive redundancy offered by equal-cost multipath routing.

Unfortunately, the only model in the Nexus switch series which supports MPLS is the 7000 (and that wasn't until NX-OS 5.2 was released just last year). (The Nexus 5500 series offers IP-only layer three functionality with an upgraded daughter card.(This is disappointing, because it necessitates deploying expensive Nexus 7000 chassis as access switches if you need to have MPLS at the edge. Otherwise, you're stuck with an extended layer two topology by way of Nexus 5000s.

Nexus 2000s Don't do Local Forwarding

A Nexus 2000 series switch, often referred to as a fabric extender or FEX, relies entirely upon its parent Nexus 5000 or 7000 switch for all forwarding. Using the topology below as a reference, for host A to send a packet to host B, that packet must be sent up to the parent Nexus 5000 for a forwarding decision to be made and then back down to the FEX.

For this reason, it is critical to provide plenty of throughput to downstream FEXes; 20 Gbps for 24 GE interfaces and 40 Gbps for 48 GE interfaces are the recommended budgets. This effects a slight oversubscription ratio of 6:5, or 1.2:1. Provided that each FEX is physically located within a few racks of its parent switch, 40 Gbps can be delivered relatively cheaply using Cisco-proprietary fabric extender transceivers.

Static FEX Interface Pinning

Speaking of FEX interface oversubscription, NX-OS provides a mechanism called static pinning to maintain deterministic oversubscription ratios in the event of a fabric uplink outage. Assume we connect a Nexus 2248 to its parent 5500 using four 10 Gbps Ethernet uplinks bonded into a single 40 Gbps port-channel. The default behavior of the FEX is to make available the entire 40 Gbps to all ports at any time, resulting in the aforementioned 1.2:1 oversubscription ratio on a 48-port FEX.

If one of the links in the port-channel fails, the aggregate uplink bandwidth is reduced to 30 Gbps (ignoring port-channel load-balancing concerns) and the oversubscription ratio increases to 1.6:1. Should two uplinks fail, the oversubscription ratio jumps to 2.4:1. This is normal behavior for a typical Ethernet switch.

The static pinning feature allows groups of access interfaces to be bound to particular fabric (uplink) interfaces with the command pinning max-links. If a fabric interface fails, its pinned access interfaces are automatically shut down. This of course disconnects whatever devices are attached to those interfaces, but it preserves the oversubscription ratio for the remaining interfaces.

The implication of shutting down a group of interfaces on the switch is that the connected devices are dual-homed to another switch as well, in the B fabric, and will begin using their secondary uplinks.

Fabric Extenders: Single- or Dual-Home?

A Nexus 2200 can have all its uplinks to a single Nexus 5500, or its uplinks can be split between two upstream 5500s. This second option might seem appealing on the surface, but it presents a more complex design and imposes several critical limitations. First, dual-homed FEXes do not support dual-homed hosts in active/active operation, e.g. a LACP port-channel (active/standby is fine, though). Second, both upstream 5500s must be configured identically to share the downstream dual-homed FEXes.

Personally, I don't see any practical advantages to dual-homing a FEX. Given that a FEX is, essentially, a remote line card of an upstream switch, and we (should) already have redundant upstream switches, there is no need for FEX uplink redundancy to multiple parent switches.

Building Virtual Port-Channels (VPCs)

Virtual port-channels, also refered to as multichassis Etherchannel (MEC), allow a single downstream switch to form a port-channel to two upstream switches. This is similar to a pair of Catalyst 6500s joined as a Virtual Switching System (VSS) pair, except that both upstream Nexuses function and are managed as independent devices. A VPC can be formed not only in the upstream direction but downstream as well, resulting in one massive port-channel among two aggregation switches and two access switches in a grotesque orgy of redundant layer two connectivity.

The verdict is still out on this one, as I haven't yet had any experience configuring a VPC in a production environment. While the aspect of eliminating spanning tree-blocked links in the datacenter in very appealing, much careful attention to the configuration of the upstream switches is necessary to keep everything running smoothly.

Allocating Virtual Device Contexts (VDCs)

Virtual device contexts are to the Nexus 7000 what virtual machines are to VMware ESXi: discrete autonomous systems which are independent from one another but which share common physical hardware. The Nexus 7000 supports up to four VDCs, one of which is the default VDC. Physical interfaces are assigned among the different VDCs, which are administered independently. One VDC can be rebooted without affecting the operation of the others.

VDCs are an intriguing feature, though with a limit of four per chassis, their potential is fairly limited. They are probably best suited to serve different operational domains; for example, one non-default VDC each for development, production, and operational networks.