Etherchannel costs and failover

By stretch | Thursday, December 10, 2009 at 5:12 a.m. UTC

IOS Etherchannel allows multiple physical links to be bonded via a single virtual interface so that their bandwidth is aggregated and each link bears a (roughly) equal share of the traffic load. However, extra consideration should be paid when designing Etherchannel links, as member links can fail, decreasing the aggregate link bandwidth without taking down the link.

Layer Two

layer2.png

In the above topology, three Etherchannels have been configured between the three switches, each composed of three 100 Mbps member links. S1 is the spanning tree root. The Etherchannels were deployed with two design goals in mind:

  • Support up to 200 Mbps of traffic between any two switches.
  • Provide n + 1 redundancy (the Etherchannel will remain up with a single failed link).

We can see that each Etherchannel, having an aggregate bandwidth of 300 Mbps, is assigned a spanning tree cost of 9:

S1# show spanning-tree vlan 1

VLAN0001
  Spanning tree enabled protocol rstp
  Root ID    Priority    1
             Address     0013.c412.0f00
             This bridge is the root
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

Bridge ID  Priority    1      (priority 0 sys-id-ext 1)
             Address     0013.c412.0f00
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time 300

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Fa0/1               Desg FWD 19        128.1    P2p 
Fa0/3               Desg FWD 19        128.3    P2p 
Fa0/5               Desg FWD 19        128.5    P2p 
Fa0/9               Desg FWD 19        128.9    P2p 
Fa0/19              Desg FWD 19        128.19   P2p Peer(STP) 
Fa0/20              Desg FWD 19        128.20   P2p Peer(STP) 
Fa0/21              Desg FWD 19        128.21   P2p Peer(STP) 
Po13                Desg FWD 9         128.65   P2p
Po12                Desg FWD 9         128.66   P2p

What happens if one of the member links between S1 and S2 fails? The aggregate bandwidth of the Etherchannel is recalculated as 200 Mbps, and the STP cost rises from 9 to 12:

S2# show spanning-tree vlan 1
...
Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
...
Po23                Altn BLK 9         128.65   P2p 
Po12                Root FWD 12        128.66   P2p

Our spanning topology remains unchanged: although the cost of S2's direct path to root has been raised from 9 to 12, 12 is still lower than the aggregate cost to root (via S3) of 18 (9 + 9).

However, if a second link in the Etherchannel fails, leaving only a single 100 Mbps member link, its bandwidth is further reduced to 100 Mbps and its cost raised to 19. At this point, the alternate path to root via S3 has a lower cost. The spanning tree topology reconverges to reflect this:

S2# show spanning-tree vlan 1
...
Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
...
Po23                Root FWD 9         128.65   P2p
Po12                Altn BLK 19        128.66   P2p 

Layer Three

layer3.png

Port-channel interfaces can operate as routed interfaces with IP addresses. The following snippet shows how a simple layer three Etherchannel is configured:

interface Port-channel12
 no switchport
 ip address 10.0.12.1 255.255.255.0
!
interface FastEthernet0/13
 no switchport
 no ip address
 channel-group 12 mode active
!
interface FastEthernet0/14
 no switchport
 no ip address
 channel-group 12 mode active
!
interface FastEthernet0/15
 no switchport
 no ip address
 channel-group 12 mode active

OSPF is a good choice as an IGP for this setup because it bases interface metrics on bandwidth. However, the default OSPF reference bandwidth is only 100 Mbps; any interface equal to or higher than 100 Mbps receives a cost of 1, which doesn't allow differentiation between healthy and partially-failed Etherchannels.

S1# show ip ospf interface brief
Interface    PID   Area            IP Address/Mask    Cost  State Nbrs F/C
Lo0          1     0               10.0.0.1/32        1     P2P   0/0
Po12         1     0               10.0.12.1/24       1     BDR   1/1
Po13         1     0               10.0.13.1/24       1     BDR   1/1

To resolve this, we raise the OSPF reference bandwidth to something much higher (say, 100 Gbps):

S1(config)# router ospf 1
S1(config-router)# auto-cost reference-bandwidth ?
    The reference bandwidth in terms of Mbits per second

S1(config-router)# auto-cost reference-bandwidth 100000
% OSPF: Reference bandwidth is changed. 
        Please ensure reference bandwidth is consistent across all routers.
S1(config-router)# ^Z
S1# show ip ospf interface brief
Interface    PID   Area            IP Address/Mask    Cost  State Nbrs F/C
Lo0          1     0               10.0.0.1/32        1     P2P   0/0
Po12         1     0               10.0.12.1/24       333   BDR   1/1
Po13         1     0               10.0.13.1/24       333   BDR   1/1

As you've probably predicted, the cost for S2 to reach the loopback interface of S1 (10.0.0.1/32) is 334 (333 for the Etherchannel plus a metric of 1 for the loopback interface):

S2# show ip route 10.0.0.1
Routing entry for 10.0.0.1/32
  Known via "ospf 1", distance 110, metric 334, type intra area
  Last update from 10.0.12.1 on Port-channel12, 00:00:16 ago
  Routing Descriptor Blocks:
  * 10.0.12.1, from 10.0.0.1, 00:00:16 ago, via Port-channel12
      Route metric is 334, traffic share count is 1

Revisiting our scenario with a failed member link between S1 and S2, we can observe very similar failover behavior (or rather, a lack thereof):

S2# show ip route 10.0.0.1
Routing entry for 10.0.0.1/32
  Known via "ospf 1", distance 110, metric 501, type intra area
  Last update from 10.0.12.1 on Port-channel12, 00:00:02 ago
  Routing Descriptor Blocks:
  * 10.0.12.1, from 10.0.0.1, 00:00:02 ago, via Port-channel12
      Route metric is 501, traffic share count is 1

The failed Etherchannel, now operating at only 200 Mbps, is assgined a higher OSPF cost of 500 (for a total metric of 501). However, 501 is still lower than the alternate route's aggregate cost of 667 (333 + 333 + 1), so our routing topology remains unchanged.

Removing a second link from the etherhchannel, leaving a lone member link operating at 100 Mbps, increases its OSPF cost to 1000 (for a total path cost of 1001). This cost is high enough to now favor the alternate route with a cost of 667:

S2# show ip route 10.0.0.1
Routing entry for 10.0.0.1/32
  Known via "ospf 1", distance 110, metric 667, type intra area
  Last update from 10.0.23.3 on Port-channel23, 00:00:49 ago
  Routing Descriptor Blocks:
  * 10.0.23.3, from 10.0.0.1, 00:00:49 ago, via Port-channel23
      Route metric is 667, traffic share count is 1

Finally, some higher-end platforms such as the Catalyst 6500 series support the port-channel min-link command, which forces an Etherhchannel to a down state if it has fewer than the specified number of member links.

About the Author

Jeremy Stretch is a network engineer living in the Raleigh-Durham, North Carolina area. He is known for his blog and cheat sheets here at Packet Life. You can reach him by email or follow him on Twitter.

Posted in Design

Comments


WaspRank
December 10, 2009 at 7:00 a.m. UTC

Cool! Great job


Chris (guest)
December 10, 2009 at 7:18 a.m. UTC

Awesome as usual, thanks Stretch! =]


nfangs
December 10, 2009 at 7:51 a.m. UTC

Interesting, to see etherchannel setup with odd no. of ports. I was taught to use contiguous even numbers of port for etherchannel.


stretch
December 10, 2009 at 8:51 a.m. UTC

@nfangs: Yeah, that's typically good practice to achieve even load-sharing. But this is what I had in the lab. =)


A guest
December 10, 2009 at 11:04 a.m. UTC

Thanks for the info. Is the layer 3 drawing correct? Shouldn't the link between S2-S3 be in a different subnet?

Greetings


Evan (guest)
December 10, 2009 at 3:26 p.m. UTC

Stretch, nicely put together! You do appear to have a typo in your layer 3 diagram where you have two instances of 10.0.12.0/24. I suspect it should be subnet 23 between S2 and S3.

Keep up the good work.


irom
December 10, 2009 at 3:27 p.m. UTC

In the second example, Layer Three, can we disable STP ?


smail (guest)
December 10, 2009 at 6:11 p.m. UTC

@irom

yes of course. OSPF or other dynamic routing protocol will take care for possible routing loops.


rene_booches
December 10, 2009 at 7:23 p.m. UTC

Great post, simple and to the point


lost-carrier (guest)
December 10, 2009 at 7:27 p.m. UTC

@irom: there is no need, because only switchports (or better VLANs) participate in STP. A routed Port (no switchport) will never do STP.


Skas (guest)
December 10, 2009 at 9:43 p.m. UTC

Nice writeup, thanks.


aconaway
December 11, 2009 at 2:12 p.m. UTC

Good one, Stretch. I always enjoy the posts.


William (guest)
December 14, 2009 at 5:01 a.m. UTC

Thanks for the post, I always love how relevant they are to the majority of work that I do.


Kamlesh Sharma (guest)
December 31, 2009 at 7:33 a.m. UTC

port-channel min-link feature is only available when you use LACP protocol only. thanks CiscoGeek


semak
June 14, 2012 at 8:22 p.m. UTC

Great Tutorial


Arif (guest)
November 7, 2012 at 8:20 a.m. UTC

Nice Article .....It was just so informative and nothing less than awesome buddy,,,,,thanks for such beautiful article ...keep rocking..


A guest
January 4, 2016 at 4:03 p.m. UTC

Great explanation :)

Comments have closed for this article due to its age.