First impressions of the Nexus 5000
By stretch | Monday, April 6, 2009 at 3:13 a.m. UTC
About a year ago, Cisco announced the Nexus 5000, a combined 10 Gbps Ethernet and Fibre Channel aggregation switch targeted for datacenter deployment. The idea is that by carrying SAN Fibre Channel traffic across a high-speed Ethernet infrastructure (using Fibre Channel over Ethernet), operational and administrative costs can be reduced.
Recently I had the opportunity to experiment with a Nexus 5020 equipped with 40 onboard 10 Gbps Ethernet interface and an expansion module sporting an additional four 10 Gbps Ethernet and four native Fibre Channel interfaces. I wanted to share my experience for those who are planning on getting more familiar with the platform in the near future.
The Nexus lines runs an entirely new operating system dubbed NX-OS, which bears zero relation to IOS, and in fact runs on a Linux kernel. As such, one can catch a few hints of its open source heritage in the initial boot sequence:
Booting kickstart image: bootflash:/n5000-uk9-kickstart.4.0.1a.N1.1.bin.... ......................................................Image verification OK Starting kernel... Usage: init 0123456SsQqAaBbCcUu INIT: version 2.85 booting Starting Nexus5020 POST... Executing Mod 1 1 SEEPROM Test......done Executing Mod 1 1 GigE Port Test.......done Executing Mod 1 1 Inband GigE Test.....done Executing Mod 1 1 NVRAM Test....done Executing Mod 1 1 PCIE Test..............................done Mod 1 1 Post Completed Successfully Executing Mod 1 2 SEEPROM Test....done Mod 1 2 Post Completed Successfully Mod 2 Post Completed Successfully POST is completed Checking all filesystems..... done. [...] Nexus 5000 Switch switch login: admin Password: Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2008, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained herein are owned by other third parties and are used and distributed under license. Some parts of this software may be covered under the GNU Public License or the GNU Lesser General Public License. A copy of each such license is available at http://www.gnu.org/licenses/gpl.html and http://www.gnu.org/licenses/lgpl.html switch#
Although a completely different animal under the hood, NX-OS presents the user with a very IOS-like interface. If you're reasonably familiar with IOS, you're sure to find the NX-OS CLI very comfortable.
To begin with, you can view the running configuration of an NX-OS device just as with IOS. While the output has been rearranged slightly, you'll notice many lines have been ported verbatim from IOS:
switch# sh run version 4.0(1a)N1(1) role name access-admin rule 1 permit read-write username admin password 5 $1$nMsnY4cy$GOflFn9RQ2kZkPjTENO7T/ role network-admin ssh key rsa 1024 force ip host switch 10.3.1.15 snmp-server user admin network-admin auth md5 [removed] priv [removed] localizedkey snmp-server host 10.3.1.2 version 2c public udp-port 1163 vrf context management ip route 0.0.0.0/0 10.3.1.1 vlan 1 fcdomain fcid database vsan 1 wwn 10:00:00:00:c9:66:22:78 fcid 0x990002 dynamic interface Ethernet1/1 interface Ethernet1/2 [...] interface Ethernet1/39 interface Ethernet1/40 interface Ethernet2/1 interface Ethernet2/2 interface Ethernet2/3 interface Ethernet2/4 interface mgmt0 ip address 10.3.1.15/24 snmp-server enable traps license boot kickstart bootflash:/n5000-uk9-kickstart.4.0.1a.N1.1.bin boot system bootflash:/n5000-uk18.104.22.168a.N1.1.bin
show version provides all the information you'd expect, as well as an unnecessarily granular reset clock:
switch# sh version Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2008, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained herein are owned by other third parties and are used and distributed under license. Some parts of this software are covered under the GNU Public License. A copy of the license is available at http://www.gnu.org/licenses/gpl.html. Software BIOS: version 1.2.0 loader: version N/A kickstart: version 4.0(1a)N1(1) system: version 4.0(1a)N1(1) BIOS compile time: 06/19/08 kickstart image file is: bootflash:/n5000-uk9-kickstart.4.0.1a.N1.1.bin kickstart compile time: 12/3/2008 5:00:00 [12/03/2008 13:08:37] system image file is: bootflash:/n5000-uk22.214.171.124a.N1.1.bin system compile time: 12/3/2008 5:00:00 [12/03/2008 13:44:18] Hardware cisco Nexus5020 Chassis ("40x10GE/Supervisor") Intel(R) Celeron(R) M CPU with 2074308 kB of memory. Processor Board ID [removed] Device name: switch bootflash: 1003520 kB Kernel uptime is 0 day(s), 0 hour(s), 7 minute(s), 40 second(s) Last reset at 516981 usecs after Tue Mar 31 21:06:52 2009 Reason: Reset Requested by CLI command reload System version: 4.0(1a)N1(1) Service: plugin Core Plugin, Ethernet Plugin
Our old stomping grounds of global configuration mode are revisited as well:
switch# configure switch(config)# hostname Nexus_5K Nexus_5K(config)#
Notice that specifying the
terminal parameter after
configure is unnecessary in NX-OS, as it's default (although, thankfully, typing
config t will be silently accepted just as well). There are a few other nice (albeit long overdue) improvements to the CLI over IOS, such as the ability to finally use CIDR notation (e.g. 192.168.0.0/24) in IP address assignment and access lists:
Nexus_5K(config)# int mgmt0 Nexus_5K(config-if)# ip address 10.3.1.15/24
This alone seems like it could be a primary selling point for the Nexus line.
You may be wondering about the
mgmt interface type. When I first saw this I thought back to CatOS'
sc0 interface, a virtual layer three interface which exists solely for management purposes, and that's essentially the function of the
mgmt0 interface on the Nexus 5000 (remember that the 5000 is a layer two-only device). Although none of the physical interfaces can be addressed at layer three, the 5000 does have the ability to perform per-interface and per-VLAN filtering (with access lists) up to layer four.
One of the 5000's primary features, beside raw speed, is the ability to consolidate Ethernet and Fibre Channel networks using FCoE. As mentioned above, the Nexus 5000 used in this lab has four native Fibre Channel interfaces, though an observant reader might have noted them absent from display of the running configuration.
This is because FCoE is a separately licensed, independent feature of NX-OS, and must be explicitly enabled:
Nexus_5K(config)# feature ? fcoe Enable feature interface-vlan Enable/Disable interface vlan lacp Enable/Disable LACP private-vlan Enable/Disable private-vlan tacacs+ Enable/Disable tacacs+ udld Enable/Disable UDLD Nexus_5K(config)# feature fcoe Nexus_5K(config)# 2009 Mar 31 21:46:26 Nexus_5K %$ VDC-1 %$ %PLATFORM-2-FC_LICENSE_DESIRED: FCoE/FC feature will be enabled after the configuration is saved followed by a reboot
After rebooting the device (with
reload), we can verify that our Fibre Channel interfaces now appear:
Nexus_5K# sh int brief ------------------------------------------------------------------------------- Interface Vsan Admin Admin Status SFP Oper Oper Port Mode Trunk Mode Speed Channel Mode (Gbps) ------------------------------------------------------------------------------- fc2/1 1 auto on down swl -- -- fc2/2 1 auto on down swl -- -- fc2/3 1 auto on sfpAbsent -- -- -- fc2/4 1 auto on sfpAbsent -- -- -- -------------------------------------------------------------------------------- Ethernet VLAN Type Mode Status Reason Speed Port Interface Ch # -------------------------------------------------------------------------------- Eth1/1 1 eth access up none 10G(D) -- Eth1/2 1 eth access down SFP not inserted 10G(D) -- Eth1/3 1 eth access down SFP not inserted 10G(D) -- Eth1/4 1 eth access down SFP not inserted 10G(D) -- [...]
Not all features are separately licensed. However, most will need to be explicitly enabled, as, unlike IOS, NX-OS is implemented in truly independent processes with their own memory space (again, it's powered by a Linux kernel). This provides for more efficient use of available memory, and provides exponentially greater protection against the emergence of rootkits targeting network infrastructure.
Obviously there's a lot more to the 5000 platform, but the goal here is to provide a measurable amount of exposure so that you have some idea what to expect if or when you encounter the platform for the first time. Check out the official Nexus 5000 configuration guide for a much more comprehensive review.
Also, if you didn't see it on Twitter, here's short video of some of the example CLI interactions above:
Posted in Hardware
April 6, 2009 at 9:20 a.m. UTC
Interesting with a little introduction on NX-OS. The CIDR notation is very welcome.
April 6, 2009 at 1:09 p.m. UTC
Nice overview. From what I remember hearing a lot of the modular and linux like features were pulled in from the SAN-OS used on the fiber channel switches. And I have to agree, CIDR notation is an extremely welcome change (maybe we can look forward to that in 12.5? ;-).
April 6, 2009 at 4:05 p.m. UTC
Why not go with BSD? They could avoid all their legal troubles they've been having with their Linksys line against the Free Software Foundation.
April 6, 2009 at 4:47 p.m. UTC
For anyone interested in history, NX-OS is actually just SAN-OS renamed. SAN-OS, along with the Cisco MDS Fiberchannel switches, were developed by Nuova, one of Cisco's famous spin-off companies that it reacquired later.
April 6, 2009 at 11:17 p.m. UTC
Not that I condone this, but I wonder how long it will take someone to hack this up as its own distro. I know, it probably has some encrypted chip in it that it checks against, but it would still be interesting.
April 7, 2009 at 7:23 a.m. UTC
I noticed that IOS XR also support CIDR notation.
April 7, 2009 at 11:22 a.m. UTC
I love the look of it, also i agree it looks like it would not be hard to get used to.
love the site :)
April 8, 2009 at 1:20 p.m. UTC
Actually that's what FOSS is all about. The changes between the GPLv2 and GPLv3 (Linux still uses v2) explicitly adds that end-users must have the ability to not only change the code, but use that code to change how the device operates. I think they called it tivoization, where tivo released their code (as required by the GPL) but had hardware checks that would not allow any modified code to run. This leads back to why I'm confused why Cisco would go with Linux rather than BSD, which does not use the GPL license.
Either way nice write-up, stretch. Now that they've moved from a micro to a monolithic kernel, what kind of boot time are you seeing compared to traditional IOS-based switches?
April 8, 2009 at 2:26 p.m. UTC
you should have mentioned "show run all" which includes the defaults :)