ERSPAN from NX-OS to IOS
By stretch | Tuesday, May 14, 2013 at 6:27 p.m. UTC
Most readers are probably familiar with the switchport analysis (SPAN) feature on Cisco's Catalyst switches. SPAN replicates all ingress and/or egress traffic from one or several interfaces to another for the purposes of packet capture or traffic monitoring. This is especially helpful when deploying a network-based IDS. Unfortunately, it's often not possible to install the IDS on the same physical switch as the ports from which you want to capture.
Remote SPAN (RSPAN) can be employed to extend a SPAN session between source and destination points on disparate switches, however it requires a layer two path end-to-end. When we need to replicate layer two traffic across a layer three network, we turn to encapsulated remote SPAN (ERSPAN). ERSPAN transports traffic inside a point-to-point GRE tunnel between arbitrary IP endpoints.
For this lab, we'll configure an ERSPAN session from an NX-OS source (a Nexus 7K) to an IOS destination (a Cisco 7600) to provide an example configuration for both platforms. MPLS transport is used between the two switches and routing of the ERSPAN tunnel will take place inside a VRF named Capture.
First, we'll configure the Nexus switch as the ERSPAN source per the documentation. We'll use Lo1 in the Capture VRF as our ERSPAN tunnel source.
interface loopback1 vrf member Capture ip address 172.16.0.2/32
The monitor session must be created of the
erspan-source type. We'll assign a unique ERSPAN ID of 100 (valid values are 1 through 1023 inclusive) and assign the ERSPAN tunnel to the Capture VRF. The destination IP is set to the loopback interface of our destination switch (which must be routable within the Capture VRF) and we configure VLAN 142 as the SPAN source; this could alternatively be a physical interface or set of interfaces. Lastly,
no shut enables the ERSPAN session.
monitor session 1 type erspan-source description ERSPAN to 7600 erspan-id 100 vrf Capture destination ip 172.16.0.1 source vlan 142 both no shut
Our source configuration is almost complete, but an additional global command is necessary for ERSPAN to function. We need to designate Lo1 as the origin IP address for the GRE tunnel. The
global keyword here signifies that the command applies across all Nexus virtual device contexts (VDCs).
monitor erspan origin ip-address 172.16.0.2 global
We can verify the operation of ERSPAN with the command
show monitor session.
Nexus7K# show monitor session 1 session 1 --------------- description : ERSPAN to 7600 type : erspan-source state : up erspan-id : 100 vrf-name : Capture acl-name : acl-name not specified ip-ttl : 255 ip-dscp : 0 destination-ip : 172.16.0.1 origin-ip : 172.16.0.2 (global) source intf : rx : tx : both : source VLANs : rx : 142 tx : 142 both : 142 filter VLANs : filter not specified
For the destination configuration, we define the physical destination interface, ERSPAN ID, transport VRF, and local IP address. Note that we are specifying the IP address used as the destination, not the origin/source IP. This tells ERSPAN what interface to listen on for incoming GRE-encapsulated traffic.
interface Loopback1 ip vrf forwarding Capture ip address 172.16.0.1 255.255.255.255 ! monitor session 1 type erspan-destination description ERSPAN from Nexus 7K destination interface Gi1/48 source erspan-id 100 ip address 172.16.0.1 vrf Capture
7600# show monitor session 1 Session 1 --------- Type : ERSPAN Destination Session Status : Admin Enabled Description : ERSPAN from Nexus 7K Destination Ports : Gi1/48 Source IP Address : 172.16.0.1 Source IP VRF : Capture Source ERSPAN ID : 100
We can now connect our collection device to Gi1/48 and start sniffing. All traffic entering VLAN 142 on the source switch will be replicated out this interface.
About the Author
Jeremy Stretch is a network engineer living in the Raleigh-Durham, North Carolina area. He is known for his blog and cheat sheets here at Packet Life. You can reach him by email or follow him on Twitter.
Posted in Switching
May 15, 2013 at 11:20 a.m. UTC
May 15, 2013 at 9:16 p.m. UTC
Thanks a lot. I'll use this as a ref if I ever need it. Few random comments below:
My biggest gripe with ERSPAN is Change Control. Most change control I see allows for ad-hoc SPAN sessions (good). Most do not allow for ad-hoc tunnel standups. That makes ERSPAN basically useless as it's so much quicker to find other means to get a SPAN.
Also when considering NX-OS related devices the interfaces are typically 10Gb or bigger. So that hinders ERSPAN even more.
May 17, 2013 at 7:25 p.m. UTC
Thats a shitton cheaper than a bunch of APCON/GIGAMON patch chassis and a dedicated hardware sniffer.
May 18, 2013 at 7:32 p.m. UTC
Nice info very useful for isp environment capturing. Thanks Jeremy.
May 19, 2013 at 8:08 p.m. UTC
ERSPAN is awesome. The only problem as I see it is its limited deployment. Most access switches, such as the 3560, will only support Remote-SPAN. However, the Nexus does NOT support Remote-SPAN but supports ERSPAN. I can't think of a work around...
May 23, 2013 at 9:01 a.m. UTC
Great explanation... thanks
May 24, 2013 at 2:13 p.m. UTC
"Unfortunately, it's often not possible to install the IDS on the same physical switch as the ports from which you want to capture."
Can you tell me which switch model that have this problem? I thought most switches support local SPAN?
May 27, 2013 at 5:19 p.m. UTC
@Nizar, lack of available ports might be the problem rather than the platform.
September 12, 2013 at 1:43 p.m. UTC
Awesome work Jeremy,
July 3, 2014 at 7:21 p.m. UTC
can you send ERSPAN traffic to a non-cisco device like a laptop?
i know it can be done with VMWARE distributed switch.
if i do not set a cisco device as destination IP I get an error in "show monitor" that the session is down with reason "Destination IP cannot resolve over a FEX interface"
November 13, 2014 at 8:33 p.m. UTC
I want to send encapsulated erspan from two of the 5548 switches and 1 7010 switch, to the other 7010 switch , do i need separate L3 link between the switches to do this or the vpc link and the peer link is sufficient to handle traffic load?