Etherchannel costs and failover
By stretch | Thursday, December 10, 2009 at 5:12 a.m. UTC
IOS Etherchannel allows multiple physical links to be bonded via a single virtual interface so that their bandwidth is aggregated and each link bears a (roughly) equal share of the traffic load. However, extra consideration should be paid when designing Etherchannel links, as member links can fail, decreasing the aggregate link bandwidth without taking down the link.
Layer Two
In the above topology, three Etherchannels have been configured between the three switches, each composed of three 100 Mbps member links. S1 is the spanning tree root. The Etherchannels were deployed with two design goals in mind:
- Support up to 200 Mbps of traffic between any two switches.
- Provide n + 1 redundancy (the Etherchannel will remain up with a single failed link).
We can see that each Etherchannel, having an aggregate bandwidth of 300 Mbps, is assigned a spanning tree cost of 9:
S1# show spanning-tree vlan 1 VLAN0001 Spanning tree enabled protocol rstp Root ID Priority 1 Address 0013.c412.0f00 This bridge is the root Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Bridge ID Priority 1 (priority 0 sys-id-ext 1) Address 0013.c412.0f00 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Aging Time 300 Interface Role Sts Cost Prio.Nbr Type ------------------- ---- --- --------- -------- -------------------------------- Fa0/1 Desg FWD 19 128.1 P2p Fa0/3 Desg FWD 19 128.3 P2p Fa0/5 Desg FWD 19 128.5 P2p Fa0/9 Desg FWD 19 128.9 P2p Fa0/19 Desg FWD 19 128.19 P2p Peer(STP) Fa0/20 Desg FWD 19 128.20 P2p Peer(STP) Fa0/21 Desg FWD 19 128.21 P2p Peer(STP) Po13 Desg FWD 9 128.65 P2p Po12 Desg FWD 9 128.66 P2p
What happens if one of the member links between S1 and S2 fails? The aggregate bandwidth of the Etherchannel is recalculated as 200 Mbps, and the STP cost rises from 9 to 12:
S2# show spanning-tree vlan 1 ... Interface Role Sts Cost Prio.Nbr Type ------------------- ---- --- --------- -------- -------------------------------- ... Po23 Altn BLK 9 128.65 P2p Po12 Root FWD 12 128.66 P2p
Our spanning topology remains unchanged: although the cost of S2's direct path to root has been raised from 9 to 12, 12 is still lower than the aggregate cost to root (via S3) of 18 (9 + 9).
However, if a second link in the Etherchannel fails, leaving only a single 100 Mbps member link, its bandwidth is further reduced to 100 Mbps and its cost raised to 19. At this point, the alternate path to root via S3 has a lower cost. The spanning tree topology reconverges to reflect this:
S2# show spanning-tree vlan 1 ... Interface Role Sts Cost Prio.Nbr Type ------------------- ---- --- --------- -------- -------------------------------- ... Po23 Root FWD 9 128.65 P2p Po12 Altn BLK 19 128.66 P2p
Layer Three
Port-channel interfaces can operate as routed interfaces with IP addresses. The following snippet shows how a simple layer three Etherchannel is configured:
interface Port-channel12 no switchport ip address 10.0.12.1 255.255.255.0 ! interface FastEthernet0/13 no switchport no ip address channel-group 12 mode active ! interface FastEthernet0/14 no switchport no ip address channel-group 12 mode active ! interface FastEthernet0/15 no switchport no ip address channel-group 12 mode active
OSPF is a good choice as an IGP for this setup because it bases interface metrics on bandwidth. However, the default OSPF reference bandwidth is only 100 Mbps; any interface equal to or higher than 100 Mbps receives a cost of 1, which doesn't allow differentiation between healthy and partially-failed Etherchannels.
S1# show ip ospf interface brief Interface PID Area IP Address/Mask Cost State Nbrs F/C Lo0 1 0 10.0.0.1/32 1 P2P 0/0 Po12 1 0 10.0.12.1/24 1 BDR 1/1 Po13 1 0 10.0.13.1/24 1 BDR 1/1
To resolve this, we raise the OSPF reference bandwidth to something much higher (say, 100 Gbps):
S1(config)# router ospf 1 S1(config-router)# auto-cost reference-bandwidth ? The reference bandwidth in terms of Mbits per second S1(config-router)# auto-cost reference-bandwidth 100000 % OSPF: Reference bandwidth is changed. Please ensure reference bandwidth is consistent across all routers. S1(config-router)# ^Z S1# show ip ospf interface brief Interface PID Area IP Address/Mask Cost State Nbrs F/C Lo0 1 0 10.0.0.1/32 1 P2P 0/0 Po12 1 0 10.0.12.1/24 333 BDR 1/1 Po13 1 0 10.0.13.1/24 333 BDR 1/1
As you've probably predicted, the cost for S2 to reach the loopback interface of S1 (10.0.0.1/32) is 334 (333 for the Etherchannel plus a metric of 1 for the loopback interface):
S2# show ip route 10.0.0.1 Routing entry for 10.0.0.1/32 Known via "ospf 1", distance 110, metric 334, type intra area Last update from 10.0.12.1 on Port-channel12, 00:00:16 ago Routing Descriptor Blocks: * 10.0.12.1, from 10.0.0.1, 00:00:16 ago, via Port-channel12 Route metric is 334, traffic share count is 1
Revisiting our scenario with a failed member link between S1 and S2, we can observe very similar failover behavior (or rather, a lack thereof):
S2# show ip route 10.0.0.1 Routing entry for 10.0.0.1/32 Known via "ospf 1", distance 110, metric 501, type intra area Last update from 10.0.12.1 on Port-channel12, 00:00:02 ago Routing Descriptor Blocks: * 10.0.12.1, from 10.0.0.1, 00:00:02 ago, via Port-channel12 Route metric is 501, traffic share count is 1
The failed Etherchannel, now operating at only 200 Mbps, is assgined a higher OSPF cost of 500 (for a total metric of 501). However, 501 is still lower than the alternate route's aggregate cost of 667 (333 + 333 + 1), so our routing topology remains unchanged.
Removing a second link from the etherhchannel, leaving a lone member link operating at 100 Mbps, increases its OSPF cost to 1000 (for a total path cost of 1001). This cost is high enough to now favor the alternate route with a cost of 667:
S2# show ip route 10.0.0.1 Routing entry for 10.0.0.1/32 Known via "ospf 1", distance 110, metric 667, type intra area Last update from 10.0.23.3 on Port-channel23, 00:00:49 ago Routing Descriptor Blocks: * 10.0.23.3, from 10.0.0.1, 00:00:49 ago, via Port-channel23 Route metric is 667, traffic share count is 1
Finally, some higher-end platforms such as the Catalyst 6500 series support the port-channel min-link command, which forces an Etherhchannel to a down state if it has fewer than the specified number of member links.
Posted in Design
Comments
December 10, 2009 at 7:00 a.m. UTC
Cool! Great job
December 10, 2009 at 7:18 a.m. UTC
Awesome as usual, thanks Stretch! =]
December 10, 2009 at 7:51 a.m. UTC
Interesting, to see etherchannel setup with odd no. of ports. I was taught to use contiguous even numbers of port for etherchannel.
December 10, 2009 at 8:51 a.m. UTC
@nfangs: Yeah, that's typically good practice to achieve even load-sharing. But this is what I had in the lab. =)
December 10, 2009 at 11:04 a.m. UTC
Thanks for the info. Is the layer 3 drawing correct? Shouldn't the link between S2-S3 be in a different subnet?
Greetings
December 10, 2009 at 3:26 p.m. UTC
Stretch, nicely put together! You do appear to have a typo in your layer 3 diagram where you have two instances of 10.0.12.0/24. I suspect it should be subnet 23 between S2 and S3.
Keep up the good work.
December 10, 2009 at 3:27 p.m. UTC
In the second example, Layer Three, can we disable STP ?
December 10, 2009 at 6:11 p.m. UTC
@irom
yes of course. OSPF or other dynamic routing protocol will take care for possible routing loops.
December 10, 2009 at 7:23 p.m. UTC
Great post, simple and to the point
December 10, 2009 at 7:27 p.m. UTC
@irom: there is no need, because only switchports (or better VLANs) participate in STP. A routed Port (no switchport) will never do STP.
December 10, 2009 at 9:43 p.m. UTC
Nice writeup, thanks.
December 11, 2009 at 2:12 p.m. UTC
Good one, Stretch. I always enjoy the posts.
December 14, 2009 at 5:01 a.m. UTC
Thanks for the post, I always love how relevant they are to the majority of work that I do.
December 31, 2009 at 7:33 a.m. UTC
port-channel min-link feature is only available when you use LACP protocol only. thanks CiscoGeek
June 14, 2012 at 8:22 p.m. UTC
Great Tutorial
November 7, 2012 at 8:20 a.m. UTC
Nice Article .....It was just so informative and nothing less than awesome buddy,,,,,thanks for such beautiful article ...keep rocking..
January 4, 2016 at 4:03 p.m. UTC
Great explanation :)