EtherChannel considerations

By stretch | Monday, January 18, 2010 at 4:04 a.m. UTC

EtherChannel is Cisco's term for bundling two or more physical Ethernet links for the purposes of aggregating available bandwidth and, to a lesser extent, providing a measure of physical redundancy. Under normal conditions, all but one redundant physical link between two switches will be disabled by STP at one end.

without_etherchannel.png

With EtherChannel configured, multiple links are grouped into a port-channel, which is assigned its own configurable virtual interface. The bundle is treated as a single link.

with_etherchannel.png

EtherChannel Negotiation

An EtherChannel can be established using one of three mechanisms:

  • PAgP - Cisco's proprietary negotiation protocol
  • LACP (IEEE 802.3ad) - Standards-based negotiation protocol
  • Static Persistence ("On") - No negotiation protocol is used

Any of these three mechanisms will suffice for most scenarios, however the choice does deserve some consideration. PAgP, while perfectly able, should probably be disqualified as a legacy proprietary protocol unless you have a specific need for it (such as ancient hardware). That leaves LACP and "on", both of which have a specific benefit.

LACP helps protect against switching loops caused by misconfiguration; when enabled, an EtherChannel will only be formed after successful negotiation between its two ends. However, this negotiation introduces an overhead and delay in initialization. Statically configuring an EtherChannel ("on") imposes no delay yet can cause serious problems if not properly configured at both ends.

To configure an EtherChannel using LACP negotiation, each side must be set to either active or passive; only interfaces configured in active mode will attempt to negotiate an EtherChannel. Passive interfaces merely respond to LACP requests. PAgP behaves the same, but its two modes are refered to as desirable and auto.

negotiation_modes.png

Only a single line is needed to configure a group of ports as an EtherChannel:

S1(config)# interface range f0/13 -15
S1(config-if-range)# channel-group 1 mode ?
  active     Enable LACP unconditionally
  auto       Enable PAgP only if a PAgP device is detected
  desirable  Enable PAgP unconditionally
  on         Enable Etherchannel only
  passive    Enable LACP only if a LACP device is detected

S1(config-if-range)# channel-group 1 mode active
Creating a port-channel interface Port-channel 1

As noted, a virtual port-channel interface Port-channel1 has been created to represent the logical link. Switchport configurations applied to this interface are replicated to the physical member interfaces. We can inspect the health of the EtherChannel with the show etherchannel summary command:

S1# show etherchannel summary
Flags:  D - down        P - bundled in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        U - in use      f - failed to allocate aggregator

M - not in use, minimum links not met
        u - unsuitable for bundling
        w - waiting to be aggregated
        d - default port

Number of channel-groups in use: 1
Number of aggregators:           1

Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
1      Po1(SD)         LACP      Fa0/13(D)   Fa0/14(D)   Fa0/15(D)   

The opposite side of the LACP EtherChannel will typically be configured as passive, however it can be active as well.

S2(config-if-range)# channel-group 1 mode passive
Creating a port-channel interface Port-channel 1

When the member ports on both sides of the EtherChannel are enabled, the port-channel interface also transitions to the up state. However, note the timing of the system messages:

*Mar  1 00:45:50.647: %LINK-3-UPDOWN: Interface FastEthernet0/14, changed state to up
*Mar  1 00:45:50.683: %LINK-3-UPDOWN: Interface FastEthernet0/13, changed state to up
*Mar  1 00:45:50.691: %LINK-3-UPDOWN: Interface FastEthernet0/15, changed state to up
*Mar  1 00:45:53.487: %LINK-3-UPDOWN: Interface Port-channel1, changed state to up

Almost a full three seconds elapsed between the member ports transitioning to the up state and the port-channel interface coming up. Once it did, we can see the state of the EtherChannel has changed to "in use":

S1# show etherchannel summary
Flags:  D - down        P - bundled in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        U - in use      f - failed to allocate aggregator

M - not in use, minimum links not met
        u - unsuitable for bundling
        w - waiting to be aggregated
        d - default port

Number of channel-groups in use: 1
Number of aggregators:           1

Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
1      Po1(SU)         LACP      Fa0/13(P)   Fa0/14(P)   Fa0/15(P)   

Note the S indicating layer two operation; on multilayer platforms, EtherChannel interfaces can be configured for routed operation as well.

For comparison, let's reconfigure the EtherChannel to function without a negtiation protocol ("on" mode):

S1(config)# no interface po1
S1(config)# interface range f0/13 -15
S1(config-if-range)# channel-group 1 mode on
Creating a port-channel interface Port-channel 1

S1(config-if-range)# no shutdown

This time we observe that the port-channel interface is enabled as soon as its first member port comes up, as there is no delay imposed by negotiation:

*Mar  1 00:56:12.271: %LINK-3-UPDOWN: Interface FastEthernet0/13, changed state to up
*Mar  1 00:56:12.287: %LINK-3-UPDOWN: Interface Port-channel1, changed state to up
*Mar  1 00:56:12.291: %LINK-3-UPDOWN: Interface FastEthernet0/14, changed state to up
*Mar  1 00:56:12.307: %LINK-3-UPDOWN: Interface FastEthernet0/15, changed state to up

In the Campus Network High Availability Design Guide, Cisco recommend forgoing the use of a negotiation protocol and configuring EtherChannels for static "on/on" operation; however they also caution that this approach offers no protection against the effect of misconfigurations.

EtherChannel Load-Balancing

Another consideration to make when implementing EtherChannels is the type of load-balancing in effect. EtherChannel provides load-balancing only per frame, not per bit. A switch decides which member link a frame will traverse by the outcome of a hash function performed against one or more fields of each frame. Which fields are considered is dependent on the switch platform and configuration. For example, a Catalyst 3550 can match only against a frame's destination or source MAC address:

S1(config)# port-channel load-balance ?
  dst-mac  Dst Mac Addr
  src-mac  Src Mac Addr

The show etherchannel load-balance command reveals that source MAC address load-balancing is default on the Catalyst 3550:

S1# show etherchannel load-balance
EtherChannel Load-Balancing Configuration:
        src-mac

EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Source MAC address
  IPv4: Source MAC address

More powerful platforms can match against IP address(es) or layer four port(s). Generally speaking, higher layer fields are more favorable as they tend to be more dynamic, resulting in a more granular distribution of traffic across member links.

Direction of flow is also an important detail. For example, consider the following topology:

topology.png

Routed packets entering the subnet from S1 are always sourced from the MAC address of the VLAN interface. If source MAC load-balancing is in use, these frames will be forwarded down only one member link, because the outcome of the hash function will always be the same. Configuring destination MAC load-balancing on S1 is recommended to achieve a more varied distribution of frames and make better use of the available bandwidth.

The opposite is true on S2: Since all frames entering the EtherChannel from LAN hosts are destined for the MAC address of the gateway (VLAN interface), source MAC address load-balancing works better here.

EtherChannel Bandwidth and Costs

Finally, remember that the perceived bandwidth of a port-channel interface is equal to the sum of its active member links. For example, an EtherChannel with three active 100 Mbps members will show a bandwidth of 300 Mbps. Because members can still fail individually, the bandwidth of a port-channel interface can fluctuate without going down.

For more information on EtherChannel bandwidth and spanning tree considerations, see Etherchannel costs and failover.

About the Author

Jeremy Stretch is a network engineer living in the Raleigh-Durham, North Carolina area. He is known for his blog and cheat sheets here at Packet Life. You can reach him by email or follow him on Twitter.

Posted in Switching

Comments


Juristium
January 18, 2010 at 8:09 a.m. UTC

Nice write out =).


dude
January 18, 2010 at 10:57 a.m. UTC

newer models can also perform mixed type of load-balancing: src-dst-mac, src-dst-ip, src-dst-port (a simple source XOR destination). A portchannel may become more balanced with them.


Dinger (guest)
January 18, 2010 at 2:15 p.m. UTC

Nice article! They'd been a little 'thin' lately...


tim_wier
January 18, 2010 at 2:25 p.m. UTC

I've often configured etherchannels this way with the one line configuration and it does work without any problems. I'm wondering why there is even an option to define which protocol to use with the "channel-protocol lacp" or "channel-protocol pagp" command since you define the protocol with which mode is used: ie active/passive for lacp and desirable/auto for pagp. If anyone knows the reason or if anyone has tried or misconfigured it so that you defined the channel-protocol as one thing and the mode as another I would like to know the results. My lab is offline right now so I can't test without taking users offline.


stretch
January 18, 2010 at 4:46 p.m. UTC

@Dinger: I can write often, or I can write thoroughly. But not both. =)


willroute4food (guest)
January 18, 2010 at 5:32 p.m. UTC

nice write up on how to look at load-balancing techniques....been a perspective I have been wanting to see for a while...bringing in a little "real-world". Thanx


ianmprice
January 18, 2010 at 5:52 p.m. UTC

One thing to add. The number of links is also something to consider when planning an ether-channel. I think it was in the BCMSN book that discussed it, but even though you can have from 2 - 8 links in your ether-channel there are distribution advantages to the number of links related to the algorithm that is ran. 2, 4, and 8 links are the most efficient.

Read more. REF : http://en.wikipedia.org/wiki/EtherChannel

Awesome articles Stretch! Keep em' coming!


sacox31s
January 19, 2010 at 3:32 a.m. UTC

@ ianmprice: Yeah 2, 4, and 8 are the most efficient ways to go.

On a 2 (0-1) member port-channel the load balancing uses the last binary bit.
0 or 1: so if the last bit is 0 it goes down member 0, and if the last bit is 1 it goes down member 1

On a 4 (0-3) member port-channel the load balancing uses the last 2 binary bits. 00, 01, 10, or 11: so if the last two bits are 01 it goes down member 1, or if they are 11 it goes down member 3

On a 8 (0-7) member port-channel the load balancing uses the last 3 binary bits. 000, 001, 010, 011, 100, 101, 110, 111: so if the last three bits are 110 it goes down member 6, or if they are 011 it goes down member 3

So as you can see PO’s with 3, 5,6,or 7 members could not evenly distribute.

On a 4 member PO using mac load-balancing the following is true:

If the last hex character is 1 the last two binary bits are 01 so it will use member number 1. If the last hex character is 8 the last two binary bits are 00 so it will use member number 0. If the last hex character is E the last two binary bits are 10 so it will use member number 2. If the last hex character is F the last two binary bits are 11 so it will use member number 3.

4 member IP load balancing same concept.

192.168.10.26 --> last two bits = 01 --> goes down member 1 of members 0-3

So if you were using src-mac load balancing on a layer-3 port-channel all the data would go down the same member, because the packet gets the interfaces source mac.

So the moral of the story is use 2,4,or 8 member Port-channels and your port-channels on distro switches need to use IP based load-balancing. While access can use MAC.

I think. Maybe.

Hopefully my hex to binary math was right. I just banged this out real quick.


vsaltao
January 25, 2010 at 11:12 a.m. UTC

This Doc on Cisco.com gives lots of detail on this type of info on various switch platforms ( one cool thing is in a 6500 to test where a packet will be sent in a etherchannel using the command "test etherchannel load-balance interface port-channel" )


ciscocrank
January 26, 2010 at 12:41 p.m. UTC

any one have used etherchannel over wireless link ?


Spikes
February 15, 2010 at 6:07 p.m. UTC

Pretty handy.. Thanks.


Steve (guest)
February 16, 2010 at 2:52 p.m. UTC

While I never deploy PAgP anymore I would not totally dismiss it just yet. Cisco is recommending it on VSS as ePAgP can be used for dual active detection.


Deep6
February 18, 2010 at 3:16 a.m. UTC

Also something to note with regard to Switch stacks from Cisco's docs located here:

https://www.cisco.com/en/US/products/hw/switches/ps5023/products_configuration_example09186a00806cb982.shtml#stack

For cross-stack EtherChannel configurations, ensure that all ports targeted for the EtherChannel are either configured for LACP or are manually configured to be in the channel group. Use the channel-group channel-group-number mode on interface configuration command in order to manually configure ports to be in the channel group. The PAgP protocol is not supported on cross-stack EtherChannels.


cisnux (guest)
April 2, 2010 at 1:37 a.m. UTC

I really like this article,it's simple but quite good for reference.thanks and hope that you'll post more.


Talal (guest)
May 13, 2010 at 10:01 a.m. UTC

Q: When configuring etherchannel between two switches, should they have the same number in both ends?


Vishnu (guest)
April 8, 2011 at 10:56 a.m. UTC

nice article, easy to understand. Thanks


koroush (guest)
April 18, 2011 at 5:59 a.m. UTC

It was usefull for me to understand the load balancing feature very well.
thanks


edon
September 7, 2011 at 5:35 p.m. UTC

do you think i can achieve something with ethernet channel ( discard private ip addresses)

I run an IPTV company and I have high bandwidth requirements, I have 10Gbps incoming connection from my ISP.. so clients request my streams ( live channels streams) from my servers .. i'd like to build a server farm so they can all make requests to www.mychannels.com for example and then maybe ethernet channels will forward it equally to all my servers .. and if one of my servers goes down i don't want them to notice that, the traffic will again be forwarded to other servers, I get the server outta server farm fix it and get it back.

what do you think?

Rejected cartoons


Josué (guest)
November 1, 2012 at 5:43 p.m. UTC

Jeremy, everyday when I have spare time at work I've been reading your blog, I have to say: it's been a great decision on my end. You have a skill to take the core of these concepts and express it in a brief, concise way. Man, that's some skill...


Rupesh (guest)
November 21, 2012 at 3:04 a.m. UTC

Good information!!!! Very useful!!!!!
Do you have any information how to configure Etherchannel using NAS box? or how it will communicate with switches using Etherchannel


Ciscotime (guest)
January 19, 2013 at 9:42 p.m. UTC

Hello, Nice article! I'd like to mention an issue that I am having while adding links to an existing bundle. I have a 10 gig 12 links (12x10) bundle between cisco CRS1 and Cisco Nexus 7010. I want o add 4 more 10 gig links to the bundle to make it 16x10. On nexus I was able to create 16x10 bundle but on CRS1 I was only able to add one link but 3 other links getting error. The 3 links are up and running just fine but when I give this command "bundle id 50 mode active" and then do "commit" it does not commit and asked me to run "show configuration failed" command. The "show configuration failed" command doesn't really give any specific reason as to why the link can not be added. I am not sure what I am doing wrong!


Ciscotime (guest)
January 23, 2013 at 6:27 p.m. UTC

Nervermind. I figured it out! I was configuring mtu on the interface which I shouldn't be doing. Interface takes the bubdle mtu that's why individual interface does not need mtu configured.


Alireza (guest)
November 28, 2013 at 8:18 a.m. UTC

Hi Jeremy,

About the last part. I mean "Bandwidth and Cost"
I've heard that the number of links in Etherchanel must be 2 or 4 or 8 and so on.
For example having 3 links in a channel group doesn't give you more bandwidth than 2 links.
Is it right?

Thank you

Leave a Comment


Optional; will not be displayed publicly or given out.
No commercial links. Only personal (e.g. blog, Twitter, or LinkedIn) and/or on-topic links, please.
_____ resolves an IPv4 address to a MAC address.