Etherchannel costs and failover

IOS Etherchannel allows multiple physical links to be bonded via a single virtual interface so that their bandwidth is aggregated and each link bears a (roughly) equal share of the traffic load. However, extra consideration should be paid when designing Etherchannel links, as member links can fail, decreasing the aggregate link bandwidth without taking down the link.

Layer Two

In the above topology, three Etherchannels have been configured between the three switches, each composed of three 100 Mbps member links. S1 is the spanning tree root. The Etherchannels were deployed with two design goals in mind:

• Support up to 200 Mbps of traffic between any two switches.
• Provide n + 1 redundancy (the Etherchannel will remain up with a single failed link).

We can see that each Etherchannel, having an aggregate bandwidth of 300 Mbps, is assigned a spanning tree cost of 9:

S1# show spanning-tree vlan 1

VLAN0001
  Spanning tree enabled protocol rstp
  Root ID    Priority    1
             Address     0013.c412.0f00
             This bridge is the root
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

Bridge ID  Priority    1      (priority 0 sys-id-ext 1)
             Address     0013.c412.0f00
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time 300

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Fa0/1               Desg FWD 19        128.1    P2p 
Fa0/3               Desg FWD 19        128.3    P2p 
Fa0/5               Desg FWD 19        128.5    P2p 
Fa0/9               Desg FWD 19        128.9    P2p 
Fa0/19              Desg FWD 19        128.19   P2p Peer(STP) 
Fa0/20              Desg FWD 19        128.20   P2p Peer(STP) 
Fa0/21              Desg FWD 19        128.21   P2p Peer(STP) 
Po13                Desg FWD 9         128.65   P2p
Po12                Desg FWD 9         128.66   P2p

What happens if one of the member links between S1 and S2 fails? The aggregate bandwidth of the Etherchannel is recalculated as 200 Mbps, and the STP cost rises from 9 to 12:

S2# show spanning-tree vlan 1
...
Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
...
Po23                Altn BLK 9         128.65   P2p 
Po12                Root FWD 12        128.66   P2p

Our spanning topology remains unchanged: although the cost of S2's direct path to root has been raised from 9 to 12, 12 is still lower than the aggregate cost to root (via S3) of 18 (9 + 9).

However, if a second link in the Etherchannel fails, leaving only a single 100 Mbps member link, its bandwidth is further reduced to 100 Mbps and its cost raised to 19. At this point, the alternate path to root via S3 has a lower cost. The spanning tree topology reconverges to reflect this:

S2# show spanning-tree vlan 1
...
Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
...
Po23                Root FWD 9         128.65   P2p
Po12                Altn BLK 19        128.66   P2p 

Layer Three

Port-channel interfaces can operate as routed interfaces with IP addresses. The following snippet shows how a simple layer three Etherchannel is configured:

interface Port-channel12
 no switchport
 ip address 10.0.12.1 255.255.255.0
!
interface FastEthernet0/13
 no switchport
 no ip address
 channel-group 12 mode active
!
interface FastEthernet0/14
 no switchport
 no ip address
 channel-group 12 mode active
!
interface FastEthernet0/15
 no switchport
 no ip address
 channel-group 12 mode active

OSPF is a good choice as an IGP for this setup because it bases interface metrics on bandwidth. However, the default OSPF reference bandwidth is only 100 Mbps; any interface equal to or higher than 100 Mbps receives a cost of 1, which doesn't allow differentiation between healthy and partially-failed Etherchannels.

S1# show ip ospf interface brief
Interface    PID   Area            IP Address/Mask    Cost  State Nbrs F/C
Lo0          1     0               10.0.0.1/32        1     P2P   0/0
Po12         1     0               10.0.12.1/24       1     BDR   1/1
Po13         1     0               10.0.13.1/24       1     BDR   1/1

To resolve this, we raise the OSPF reference bandwidth to something much higher (say, 100 Gbps):

S1(config)# router ospf 1
S1(config-router)# auto-cost reference-bandwidth ?
    The reference bandwidth in terms of Mbits per second

S1(config-router)# auto-cost reference-bandwidth 100000
% OSPF: Reference bandwidth is changed. 
        Please ensure reference bandwidth is consistent across all routers.
S1(config-router)# ^Z
S1# show ip ospf interface brief
Interface    PID   Area            IP Address/Mask    Cost  State Nbrs F/C
Lo0          1     0               10.0.0.1/32        1     P2P   0/0
Po12         1     0               10.0.12.1/24       333   BDR   1/1
Po13         1     0               10.0.13.1/24       333   BDR   1/1

As you've probably predicted, the cost for S2 to reach the loopback interface of S1 (10.0.0.1/32) is 334 (333 for the Etherchannel plus a metric of 1 for the loopback interface):

S2# show ip route 10.0.0.1
Routing entry for 10.0.0.1/32
  Known via "ospf 1", distance 110, metric 334, type intra area
  Last update from 10.0.12.1 on Port-channel12, 00:00:16 ago
  Routing Descriptor Blocks:
  * 10.0.12.1, from 10.0.0.1, 00:00:16 ago, via Port-channel12
      Route metric is 334, traffic share count is 1

Revisiting our scenario with a failed member link between S1 and S2, we can observe very similar failover behavior (or rather, a lack thereof):

S2# show ip route 10.0.0.1
Routing entry for 10.0.0.1/32
  Known via "ospf 1", distance 110, metric 501, type intra area
  Last update from 10.0.12.1 on Port-channel12, 00:00:02 ago
  Routing Descriptor Blocks:
  * 10.0.12.1, from 10.0.0.1, 00:00:02 ago, via Port-channel12
      Route metric is 501, traffic share count is 1

The failed Etherchannel, now operating at only 200 Mbps, is assgined a higher OSPF cost of 500 (for a total metric of 501). However, 501 is still lower than the alternate route's aggregate cost of 667 (333 + 333 + 1), so our routing topology remains unchanged.

Removing a second link from the etherhchannel, leaving a lone member link operating at 100 Mbps, increases its OSPF cost to 1000 (for a total path cost of 1001). This cost is high enough to now favor the alternate route with a cost of 667:

S2# show ip route 10.0.0.1
Routing entry for 10.0.0.1/32
  Known via "ospf 1", distance 110, metric 667, type intra area
  Last update from 10.0.23.3 on Port-channel23, 00:00:49 ago
  Routing Descriptor Blocks:
  * 10.0.23.3, from 10.0.0.1, 00:00:49 ago, via Port-channel23
      Route metric is 667, traffic share count is 1

Finally, some higher-end platforms such as the Catalyst 6500 series support the port-channel min-link command, which forces an Etherhchannel to a down state if it has fewer than the specified number of member links.