First impressions of the Nexus 5000

About a year ago, Cisco announced the Nexus 5000, a combined 10 Gbps Ethernet and Fibre Channel aggregation switch targeted for datacenter deployment. The idea is that by carrying SAN Fibre Channel traffic across a high-speed Ethernet infrastructure (using Fibre Channel over Ethernet), operational and administrative costs can be reduced.

Recently I had the opportunity to experiment with a Nexus 5020 equipped with 40 onboard 10 Gbps Ethernet interface and an expansion module sporting an additional four 10 Gbps Ethernet and four native Fibre Channel interfaces. I wanted to share my experience for those who are planning on getting more familiar with the platform in the near future.

The Nexus lines runs an entirely new operating system dubbed NX-OS, which bears zero relation to IOS, and in fact runs on a Linux kernel. As such, one can catch a few hints of its open source heritage in the initial boot sequence:

Booting kickstart image: bootflash:/n5000-uk9-kickstart.4.0.1a.N1.1.bin....
......................................................Image verification OK

Starting kernel...
Usage: init 0123456SsQqAaBbCcUu
INIT: version 2.85 booting
Starting Nexus5020 POST...
  Executing Mod 1 1 SEEPROM Test......done
  Executing Mod 1 1 GigE Port Test.......done
  Executing Mod 1 1 Inband GigE Test.....done
  Executing Mod 1 1 NVRAM Test....done
  Executing Mod 1 1 PCIE Test..............................done
  Mod 1 1 Post Completed Successfully
  Executing Mod 1 2 SEEPROM Test....done
  Mod 1 2 Post Completed Successfully
  Mod 2 Post Completed Successfully
POST is completed
Checking all filesystems..... done.

[...]

Nexus 5000 Switch
switch login: admin
Password: 
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2008, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained herein are owned by
other third parties and are used and distributed under license.
Some parts of this software may be covered under the GNU Public
License or the GNU Lesser General Public License. A copy of 
each such license is available at
http://www.gnu.org/licenses/gpl.html and
http://www.gnu.org/licenses/lgpl.html
switch#

Although a completely different animal under the hood, NX-OS presents the user with a very IOS-like interface. If you're reasonably familiar with IOS, you're sure to find the NX-OS CLI very comfortable.

To begin with, you can view the running configuration of an NX-OS device just as with IOS. While the output has been rearranged slightly, you'll notice many lines have been ported verbatim from IOS:

switch# sh run
version 4.0(1a)N1(1)
role name access-admin
  rule 1 permit read-write  
username admin password 5 $1$nMsnY4cy$GOflFn9RQ2kZkPjTENO7T/  role network-admin
ssh key rsa 1024 force
ip host switch 10.3.1.15
snmp-server user admin network-admin auth md5 [removed] priv [removed]
 localizedkey
snmp-server host 10.3.1.2 version 2c public  udp-port 1163
vrf context management
  ip route 0.0.0.0/0 10.3.1.1
vlan 1
fcdomain fcid database
  vsan 1 wwn 10:00:00:00:c9:66:22:78 fcid 0x990002 dynamic

interface Ethernet1/1

interface Ethernet1/2

[...]

interface Ethernet1/39

interface Ethernet1/40

interface Ethernet2/1

interface Ethernet2/2

interface Ethernet2/3

interface Ethernet2/4

interface mgmt0
  ip address 10.3.1.15/24
snmp-server enable traps license
boot kickstart bootflash:/n5000-uk9-kickstart.4.0.1a.N1.1.bin
boot system bootflash:/n5000-uk9.4.0.1a.N1.1.bin 

show version provides all the information you'd expect, as well as an unnecessarily granular reset clock:

switch# sh version
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2008, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained herein are owned by
other third parties and are used and distributed under license.
Some parts of this software are covered under the GNU Public
License. A copy of the license is available at
http://www.gnu.org/licenses/gpl.html.

Software
  BIOS:      version 1.2.0
  loader:    version N/A
  kickstart: version 4.0(1a)N1(1)
  system:    version 4.0(1a)N1(1)
  BIOS compile time:       06/19/08
  kickstart image file is: bootflash:/n5000-uk9-kickstart.4.0.1a.N1.1.bin
  kickstart compile time:  12/3/2008 5:00:00 [12/03/2008 13:08:37]
  system image file is:    bootflash:/n5000-uk9.4.0.1a.N1.1.bin
  system compile time:     12/3/2008 5:00:00 [12/03/2008 13:44:18]

Hardware
  cisco Nexus5020 Chassis ("40x10GE/Supervisor")
  Intel(R) Celeron(R) M CPU    with 2074308 kB of memory.
  Processor Board ID [removed]

Device name: switch
  bootflash:    1003520 kB

Kernel uptime is 0 day(s), 0 hour(s), 7 minute(s), 40 second(s)

Last reset at 516981 usecs after  Tue Mar 31 21:06:52 2009

Reason: Reset Requested by CLI command reload
  System version: 4.0(1a)N1(1)
  Service:

plugin
  Core Plugin, Ethernet Plugin

Our old stomping grounds of global configuration mode are revisited as well:

switch# configure
switch(config)# hostname Nexus_5K
Nexus_5K(config)# 

Notice that specifying the terminal parameter after configure is unnecessary in NX-OS, as it's default (although, thankfully, typing config t will be silently accepted just as well). There are a few other nice (albeit long overdue) improvements to the CLI over IOS, such as the ability to finally use CIDR notation (e.g. 192.168.0.0/24) in IP address assignment and access lists:

Nexus_5K(config)# int mgmt0
Nexus_5K(config-if)# ip address 10.3.1.15/24

This alone seems like it could be a primary selling point for the Nexus line.

You may be wondering about the mgmt interface type. When I first saw this I thought back to CatOS' sc0 interface, a virtual layer three interface which exists solely for management purposes, and that's essentially the function of the mgmt0 interface on the Nexus 5000 (remember that the 5000 is a layer two-only device). Although none of the physical interfaces can be addressed at layer three, the 5000 does have the ability to perform per-interface and per-VLAN filtering (with access lists) up to layer four.

One of the 5000's primary features, beside raw speed, is the ability to consolidate Ethernet and Fibre Channel networks using FCoE. As mentioned above, the Nexus 5000 used in this lab has four native Fibre Channel interfaces, though an observant reader might have noted them absent from display of the running configuration.

This is because FCoE is a separately licensed, independent feature of NX-OS, and must be explicitly enabled:

Nexus_5K(config)# feature ?
  fcoe            Enable feature
  interface-vlan  Enable/Disable interface vlan
  lacp            Enable/Disable LACP
  private-vlan    Enable/Disable private-vlan
  tacacs+         Enable/Disable tacacs+
  udld            Enable/Disable UDLD

Nexus_5K(config)# feature fcoe
Nexus_5K(config)# 2009 Mar 31 21:46:26 Nexus_5K %$ VDC-1 %$ %PLATFORM-2-FC_LICENSE_DESIRED:
FCoE/FC feature will be enabled after the configuration is saved followed by a reboot

After rebooting the device (with reload), we can verify that our Fibre Channel interfaces now appear:

Nexus_5K# sh int brief

-------------------------------------------------------------------------------
Interface  Vsan   Admin  Admin   Status          SFP    Oper  Oper   Port
              Mode   Trunk                          Mode  Speed  Channel
                     Mode                                 (Gbps)
-------------------------------------------------------------------------------
fc2/1      1      auto   on      down             swl    --           --
fc2/2      1      auto   on      down             swl    --           --
fc2/3      1      auto   on      sfpAbsent        --     --           --
fc2/4      1      auto   on      sfpAbsent        --     --           --

--------------------------------------------------------------------------------
Ethernet      VLAN   Type Mode   Status  Reason                   Speed     Port
Interface                                                                   Ch #
--------------------------------------------------------------------------------
Eth1/1        1      eth  access up      none                        10G(D) --
Eth1/2        1      eth  access down    SFP not inserted            10G(D) --
Eth1/3        1      eth  access down    SFP not inserted            10G(D) --
Eth1/4        1      eth  access down    SFP not inserted            10G(D) --
[...]

Not all features are separately licensed. However, most will need to be explicitly enabled, as, unlike IOS, NX-OS is implemented in truly independent processes with their own memory space (again, it's powered by a Linux kernel). This provides for more efficient use of available memory, and provides exponentially greater protection against the emergence of rootkits targeting network infrastructure.

Obviously there's a lot more to the 5000 platform, but the goal here is to provide a measurable amount of exposure so that you have some idea what to expect if or when you encounter the platform for the first time. Check out the official Nexus 5000 configuration guide for a much more comprehensive review.

Also, if you didn't see it on Twitter, here's short video of some of the example CLI interactions above: