EtherChannel considerations
By stretch | Monday, January 18, 2010 at 4:04 a.m. UTC
EtherChannel is Cisco's term for bundling two or more physical Ethernet links for the purposes of aggregating available bandwidth and, to a lesser extent, providing a measure of physical redundancy. Under normal conditions, all but one redundant physical link between two switches will be disabled by STP at one end.
With EtherChannel configured, multiple links are grouped into a port-channel, which is assigned its own configurable virtual interface. The bundle is treated as a single link.
EtherChannel Negotiation
An EtherChannel can be established using one of three mechanisms:
- PAgP - Cisco's proprietary negotiation protocol
- LACP (IEEE 802.3ad) - Standards-based negotiation protocol
- Static Persistence ("On") - No negotiation protocol is used
Any of these three mechanisms will suffice for most scenarios, however the choice does deserve some consideration. PAgP, while perfectly able, should probably be disqualified as a legacy proprietary protocol unless you have a specific need for it (such as ancient hardware). That leaves LACP and "on", both of which have a specific benefit.
LACP helps protect against switching loops caused by misconfiguration; when enabled, an EtherChannel will only be formed after successful negotiation between its two ends. However, this negotiation introduces an overhead and delay in initialization. Statically configuring an EtherChannel ("on") imposes no delay yet can cause serious problems if not properly configured at both ends.
To configure an EtherChannel using LACP negotiation, each side must be set to either active or passive; only interfaces configured in active mode will attempt to negotiate an EtherChannel. Passive interfaces merely respond to LACP requests. PAgP behaves the same, but its two modes are refered to as desirable and auto.
Only a single line is needed to configure a group of ports as an EtherChannel:
S1(config)# interface range f0/13 -15 S1(config-if-range)# channel-group 1 mode ? active Enable LACP unconditionally auto Enable PAgP only if a PAgP device is detected desirable Enable PAgP unconditionally on Enable Etherchannel only passive Enable LACP only if a LACP device is detected S1(config-if-range)# channel-group 1 mode active Creating a port-channel interface Port-channel 1
As noted, a virtual port-channel interface Port-channel1 has been created to represent the logical link. Switchport configurations applied to this interface are replicated to the physical member interfaces. We can inspect the health of the EtherChannel with the show etherchannel summary
command:
S1# show etherchannel summary Flags: D - down P - bundled in port-channel I - stand-alone s - suspended H - Hot-standby (LACP only) R - Layer3 S - Layer2 U - in use f - failed to allocate aggregator M - not in use, minimum links not met u - unsuitable for bundling w - waiting to be aggregated d - default port Number of channel-groups in use: 1 Number of aggregators: 1 Group Port-channel Protocol Ports ------+-------------+-----------+----------------------------------------------- 1 Po1(SD) LACP Fa0/13(D) Fa0/14(D) Fa0/15(D)
The opposite side of the LACP EtherChannel will typically be configured as passive, however it can be active as well.
S2(config-if-range)# channel-group 1 mode passive Creating a port-channel interface Port-channel 1
When the member ports on both sides of the EtherChannel are enabled, the port-channel interface also transitions to the up state. However, note the timing of the system messages:
*Mar 1 00:45:50.647: %LINK-3-UPDOWN: Interface FastEthernet0/14, changed state to up *Mar 1 00:45:50.683: %LINK-3-UPDOWN: Interface FastEthernet0/13, changed state to up *Mar 1 00:45:50.691: %LINK-3-UPDOWN: Interface FastEthernet0/15, changed state to up *Mar 1 00:45:53.487: %LINK-3-UPDOWN: Interface Port-channel1, changed state to up
Almost a full three seconds elapsed between the member ports transitioning to the up state and the port-channel interface coming up. Once it did, we can see the state of the EtherChannel has changed to "in use":
S1# show etherchannel summary Flags: D - down P - bundled in port-channel I - stand-alone s - suspended H - Hot-standby (LACP only) R - Layer3 S - Layer2 U - in use f - failed to allocate aggregator M - not in use, minimum links not met u - unsuitable for bundling w - waiting to be aggregated d - default port Number of channel-groups in use: 1 Number of aggregators: 1 Group Port-channel Protocol Ports ------+-------------+-----------+----------------------------------------------- 1 Po1(SU) LACP Fa0/13(P) Fa0/14(P) Fa0/15(P)
Note the S indicating layer two operation; on multilayer platforms, EtherChannel interfaces can be configured for routed operation as well.
For comparison, let's reconfigure the EtherChannel to function without a negtiation protocol ("on" mode):
S1(config)# no interface po1 S1(config)# interface range f0/13 -15 S1(config-if-range)# channel-group 1 mode on Creating a port-channel interface Port-channel 1 S1(config-if-range)# no shutdown
This time we observe that the port-channel interface is enabled as soon as its first member port comes up, as there is no delay imposed by negotiation:
*Mar 1 00:56:12.271: %LINK-3-UPDOWN: Interface FastEthernet0/13, changed state to up *Mar 1 00:56:12.287: %LINK-3-UPDOWN: Interface Port-channel1, changed state to up *Mar 1 00:56:12.291: %LINK-3-UPDOWN: Interface FastEthernet0/14, changed state to up *Mar 1 00:56:12.307: %LINK-3-UPDOWN: Interface FastEthernet0/15, changed state to up
In the Campus Network High Availability Design Guide, Cisco recommend forgoing the use of a negotiation protocol and configuring EtherChannels for static "on/on" operation; however they also caution that this approach offers no protection against the effect of misconfigurations.
EtherChannel Load-Balancing
Another consideration to make when implementing EtherChannels is the type of load-balancing in effect. EtherChannel provides load-balancing only per frame, not per bit. A switch decides which member link a frame will traverse by the outcome of a hash function performed against one or more fields of each frame. Which fields are considered is dependent on the switch platform and configuration. For example, a Catalyst 3550 can match only against a frame's destination or source MAC address:
S1(config)# port-channel load-balance ? dst-mac Dst Mac Addr src-mac Src Mac Addr
The show etherchannel load-balance
command reveals that source MAC address load-balancing is default on the Catalyst 3550:
S1# show etherchannel load-balance EtherChannel Load-Balancing Configuration: src-mac EtherChannel Load-Balancing Addresses Used Per-Protocol: Non-IP: Source MAC address IPv4: Source MAC address
More powerful platforms can match against IP address(es) or layer four port(s). Generally speaking, higher layer fields are more favorable as they tend to be more dynamic, resulting in a more granular distribution of traffic across member links.
Direction of flow is also an important detail. For example, consider the following topology:
Routed packets entering the subnet from S1 are always sourced from the MAC address of the VLAN interface. If source MAC load-balancing is in use, these frames will be forwarded down only one member link, because the outcome of the hash function will always be the same. Configuring destination MAC load-balancing on S1 is recommended to achieve a more varied distribution of frames and make better use of the available bandwidth.
The opposite is true on S2: Since all frames entering the EtherChannel from LAN hosts are destined for the MAC address of the gateway (VLAN interface), source MAC address load-balancing works better here.
EtherChannel Bandwidth and Costs
Finally, remember that the perceived bandwidth of a port-channel interface is equal to the sum of its active member links. For example, an EtherChannel with three active 100 Mbps members will show a bandwidth of 300 Mbps. Because members can still fail individually, the bandwidth of a port-channel interface can fluctuate without going down.
For more information on EtherChannel bandwidth and spanning tree considerations, see Etherchannel costs and failover.
Posted in Switching
Comments
January 18, 2010 at 8:09 a.m. UTC
Nice write out =).
January 18, 2010 at 10:57 a.m. UTC
newer models can also perform mixed type of load-balancing: src-dst-mac, src-dst-ip, src-dst-port (a simple source XOR destination). A portchannel may become more balanced with them.
January 18, 2010 at 2:15 p.m. UTC
Nice article! They'd been a little 'thin' lately...
January 18, 2010 at 2:25 p.m. UTC
I've often configured etherchannels this way with the one line configuration and it does work without any problems. I'm wondering why there is even an option to define which protocol to use with the "channel-protocol lacp" or "channel-protocol pagp" command since you define the protocol with which mode is used: ie active/passive for lacp and desirable/auto for pagp. If anyone knows the reason or if anyone has tried or misconfigured it so that you defined the channel-protocol as one thing and the mode as another I would like to know the results. My lab is offline right now so I can't test without taking users offline.
January 18, 2010 at 4:46 p.m. UTC
@Dinger: I can write often, or I can write thoroughly. But not both. =)
January 18, 2010 at 5:32 p.m. UTC
nice write up on how to look at load-balancing techniques....been a perspective I have been wanting to see for a while...bringing in a little "real-world". Thanx
January 18, 2010 at 5:52 p.m. UTC
One thing to add. The number of links is also something to consider when planning an ether-channel. I think it was in the BCMSN book that discussed it, but even though you can have from 2 - 8 links in your ether-channel there are distribution advantages to the number of links related to the algorithm that is ran. 2, 4, and 8 links are the most efficient.
Read more. REF : http://en.wikipedia.org/wiki/EtherChannel
Awesome articles Stretch! Keep em' coming!
January 19, 2010 at 3:32 a.m. UTC
@ ianmprice: Yeah 2, 4, and 8 are the most efficient ways to go.
On a 2 (0-1) member port-channel the load balancing uses the last binary bit.
0 or 1: so if the last bit is 0 it goes down member 0, and if the last bit is 1 it goes down member 1
On a 4 (0-3) member port-channel the load balancing uses the last 2 binary bits. 00, 01, 10, or 11: so if the last two bits are 01 it goes down member 1, or if they are 11 it goes down member 3
On a 8 (0-7) member port-channel the load balancing uses the last 3 binary bits. 000, 001, 010, 011, 100, 101, 110, 111: so if the last three bits are 110 it goes down member 6, or if they are 011 it goes down member 3
So as you can see PO’s with 3, 5,6,or 7 members could not evenly distribute.
On a 4 member PO using mac load-balancing the following is true:
If the last hex character is 1 the last two binary bits are 01 so it will use member number 1. If the last hex character is 8 the last two binary bits are 00 so it will use member number 0. If the last hex character is E the last two binary bits are 10 so it will use member number 2. If the last hex character is F the last two binary bits are 11 so it will use member number 3.
4 member IP load balancing same concept.
192.168.10.26 --> last two bits = 01 --> goes down member 1 of members 0-3
So if you were using src-mac load balancing on a layer-3 port-channel all the data would go down the same member, because the packet gets the interfaces source mac.
So the moral of the story is use 2,4,or 8 member Port-channels and your port-channels on distro switches need to use IP based load-balancing. While access can use MAC.
I think. Maybe.
Hopefully my hex to binary math was right. I just banged this out real quick.
January 25, 2010 at 11:12 a.m. UTC
This Doc on Cisco.com gives lots of detail on this type of info on various switch platforms ( one cool thing is in a 6500 to test where a packet will be sent in a etherchannel using the command "test etherchannel load-balance interface port-channel" )
January 26, 2010 at 12:41 p.m. UTC
any one have used etherchannel over wireless link ?
February 15, 2010 at 6:07 p.m. UTC
Pretty handy.. Thanks.
February 16, 2010 at 2:52 p.m. UTC
While I never deploy PAgP anymore I would not totally dismiss it just yet. Cisco is recommending it on VSS as ePAgP can be used for dual active detection.
February 18, 2010 at 3:16 a.m. UTC
Also something to note with regard to Switch stacks from Cisco's docs located here:
https://www.cisco.com/en/US/products/hw/switches/ps5023/products_configuration_example09186a00806cb982.shtml#stack
For cross-stack EtherChannel configurations, ensure that all ports targeted for the EtherChannel are either configured for LACP or are manually configured to be in the channel group. Use the channel-group channel-group-number mode on interface configuration command in order to manually configure ports to be in the channel group. The PAgP protocol is not supported on cross-stack EtherChannels.
April 2, 2010 at 1:37 a.m. UTC
I really like this article,it's simple but quite good for reference.thanks and hope that you'll post more.
May 13, 2010 at 10:01 a.m. UTC
Q: When configuring etherchannel between two switches, should they have the same number in both ends?
April 8, 2011 at 10:56 a.m. UTC
nice article, easy to understand. Thanks
April 18, 2011 at 5:59 a.m. UTC
It was usefull for me to understand the load balancing feature very well.
thanks
September 7, 2011 at 5:35 p.m. UTC
do you think i can achieve something with ethernet channel ( discard private ip addresses)
I run an IPTV company and I have high bandwidth requirements, I have 10Gbps incoming connection from my ISP.. so clients request my streams ( live channels streams) from my servers .. i'd like to build a server farm so they can all make requests to www.mychannels.com for example and then maybe ethernet channels will forward it equally to all my servers .. and if one of my servers goes down i don't want them to notice that, the traffic will again be forwarded to other servers, I get the server outta server farm fix it and get it back.
what do you think?
November 1, 2012 at 5:43 p.m. UTC
Jeremy, everyday when I have spare time at work I've been reading your blog, I have to say: it's been a great decision on my end. You have a skill to take the core of these concepts and express it in a brief, concise way. Man, that's some skill...
November 21, 2012 at 3:04 a.m. UTC
Good information!!!! Very useful!!!!!
Do you have any information how to configure Etherchannel using NAS box? or how it will communicate with switches using Etherchannel
January 19, 2013 at 9:42 p.m. UTC
Hello, Nice article! I'd like to mention an issue that I am having while adding links to an existing bundle. I have a 10 gig 12 links (12x10) bundle between cisco CRS1 and Cisco Nexus 7010. I want o add 4 more 10 gig links to the bundle to make it 16x10. On nexus I was able to create 16x10 bundle but on CRS1 I was only able to add one link but 3 other links getting error. The 3 links are up and running just fine but when I give this command "bundle id 50 mode active" and then do "commit" it does not commit and asked me to run "show configuration failed" command. The "show configuration failed" command doesn't really give any specific reason as to why the link can not be added. I am not sure what I am doing wrong!
January 23, 2013 at 6:27 p.m. UTC
Nervermind. I figured it out! I was configuring mtu on the interface which I shouldn't be doing. Interface takes the bubdle mtu that's why individual interface does not need mtu configured.
November 28, 2013 at 8:18 a.m. UTC
Hi Jeremy,
About the last part. I mean "Bandwidth and Cost"
I've heard that the number of links in Etherchanel must be 2 or 4 or 8 and so on.
For example having 3 links in a channel group doesn't give you more bandwidth than 2 links.
Is it right?
Thank you
March 10, 2015 at 5:02 p.m. UTC
I have configured the etherchannel in L2 switch for 2 links but when a single link gets down packet loss is not coming but when the same link comes up i am getting the packet loss can any one suggest how resolve the problem.
February 18, 2016 at 1:44 p.m. UTC
Finally, a comprehensive explanation of the difference between LACP and static LAG's. Most "explanations" assume the reader knows what static is, passing it over in their explanation.
The site looks excellent!