EtherChannel considerations

EtherChannel is Cisco's term for bundling two or more physical Ethernet links for the purposes of aggregating available bandwidth and, to a lesser extent, providing a measure of physical redundancy. Under normal conditions, all but one redundant physical link between two switches will be disabled by STP at one end.

With EtherChannel configured, multiple links are grouped into a port-channel, which is assigned its own configurable virtual interface. The bundle is treated as a single link.

EtherChannel Negotiation

An EtherChannel can be established using one of three mechanisms:

• PAgP - Cisco's proprietary negotiation protocol
• LACP (IEEE 802.3ad) - Standards-based negotiation protocol
• Static Persistence ("On") - No negotiation protocol is used

Any of these three mechanisms will suffice for most scenarios, however the choice does deserve some consideration. PAgP, while perfectly able, should probably be disqualified as a legacy proprietary protocol unless you have a specific need for it (such as ancient hardware). That leaves LACP and "on", both of which have a specific benefit.

LACP helps protect against switching loops caused by misconfiguration; when enabled, an EtherChannel will only be formed after successful negotiation between its two ends. However, this negotiation introduces an overhead and delay in initialization. Statically configuring an EtherChannel ("on") imposes no delay yet can cause serious problems if not properly configured at both ends.

To configure an EtherChannel using LACP negotiation, each side must be set to either active or passive; only interfaces configured in active mode will attempt to negotiate an EtherChannel. Passive interfaces merely respond to LACP requests. PAgP behaves the same, but its two modes are refered to as desirable and auto.

Only a single line is needed to configure a group of ports as an EtherChannel:

S1(config)# interface range f0/13 -15
S1(config-if-range)# channel-group 1 mode ?
  active     Enable LACP unconditionally
  auto       Enable PAgP only if a PAgP device is detected
  desirable  Enable PAgP unconditionally
  on         Enable Etherchannel only
  passive    Enable LACP only if a LACP device is detected

S1(config-if-range)# channel-group 1 mode active
Creating a port-channel interface Port-channel 1

As noted, a virtual port-channel interface Port-channel1 has been created to represent the logical link. Switchport configurations applied to this interface are replicated to the physical member interfaces. We can inspect the health of the EtherChannel with the show etherchannel summary command:

S1# show etherchannel summary
Flags:  D - down        P - bundled in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        U - in use      f - failed to allocate aggregator

M - not in use, minimum links not met
        u - unsuitable for bundling
        w - waiting to be aggregated
        d - default port

Number of channel-groups in use: 1
Number of aggregators:           1

Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
1      Po1(SD)         LACP      Fa0/13(D)   Fa0/14(D)   Fa0/15(D)   

The opposite side of the LACP EtherChannel will typically be configured as passive, however it can be active as well.

S2(config-if-range)# channel-group 1 mode passive
Creating a port-channel interface Port-channel 1

When the member ports on both sides of the EtherChannel are enabled, the port-channel interface also transitions to the up state. However, note the timing of the system messages:

*Mar  1 00:45:50.647: %LINK-3-UPDOWN: Interface FastEthernet0/14, changed state to up
*Mar  1 00:45:50.683: %LINK-3-UPDOWN: Interface FastEthernet0/13, changed state to up
*Mar  1 00:45:50.691: %LINK-3-UPDOWN: Interface FastEthernet0/15, changed state to up
*Mar  1 00:45:53.487: %LINK-3-UPDOWN: Interface Port-channel1, changed state to up

Almost a full three seconds elapsed between the member ports transitioning to the up state and the port-channel interface coming up. Once it did, we can see the state of the EtherChannel has changed to "in use":

S1# show etherchannel summary
Flags:  D - down        P - bundled in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        U - in use      f - failed to allocate aggregator

M - not in use, minimum links not met
        u - unsuitable for bundling
        w - waiting to be aggregated
        d - default port

Number of channel-groups in use: 1
Number of aggregators:           1

Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
1      Po1(SU)         LACP      Fa0/13(P)   Fa0/14(P)   Fa0/15(P)   

Note the S indicating layer two operation; on multilayer platforms, EtherChannel interfaces can be configured for routed operation as well.

For comparison, let's reconfigure the EtherChannel to function without a negtiation protocol ("on" mode):

S1(config)# no interface po1
S1(config)# interface range f0/13 -15
S1(config-if-range)# channel-group 1 mode on
Creating a port-channel interface Port-channel 1

S1(config-if-range)# no shutdown

This time we observe that the port-channel interface is enabled as soon as its first member port comes up, as there is no delay imposed by negotiation:

*Mar  1 00:56:12.271: %LINK-3-UPDOWN: Interface FastEthernet0/13, changed state to up
*Mar  1 00:56:12.287: %LINK-3-UPDOWN: Interface Port-channel1, changed state to up
*Mar  1 00:56:12.291: %LINK-3-UPDOWN: Interface FastEthernet0/14, changed state to up
*Mar  1 00:56:12.307: %LINK-3-UPDOWN: Interface FastEthernet0/15, changed state to up

In the Campus Network High Availability Design Guide, Cisco recommend forgoing the use of a negotiation protocol and configuring EtherChannels for static "on/on" operation; however they also caution that this approach offers no protection against the effect of misconfigurations.

EtherChannel Load-Balancing

Another consideration to make when implementing EtherChannels is the type of load-balancing in effect. EtherChannel provides load-balancing only per frame, not per bit. A switch decides which member link a frame will traverse by the outcome of a hash function performed against one or more fields of each frame. Which fields are considered is dependent on the switch platform and configuration. For example, a Catalyst 3550 can match only against a frame's destination or source MAC address:

S1(config)# port-channel load-balance ?
  dst-mac  Dst Mac Addr
  src-mac  Src Mac Addr

The show etherchannel load-balance command reveals that source MAC address load-balancing is default on the Catalyst 3550:

S1# show etherchannel load-balance
EtherChannel Load-Balancing Configuration:
        src-mac

EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Source MAC address
  IPv4: Source MAC address

More powerful platforms can match against IP address(es) or layer four port(s). Generally speaking, higher layer fields are more favorable as they tend to be more dynamic, resulting in a more granular distribution of traffic across member links.

Direction of flow is also an important detail. For example, consider the following topology:

Routed packets entering the subnet from S1 are always sourced from the MAC address of the VLAN interface. If source MAC load-balancing is in use, these frames will be forwarded down only one member link, because the outcome of the hash function will always be the same. Configuring destination MAC load-balancing on S1 is recommended to achieve a more varied distribution of frames and make better use of the available bandwidth.

The opposite is true on S2: Since all frames entering the EtherChannel from LAN hosts are destined for the MAC address of the gateway (VLAN interface), source MAC address load-balancing works better here.

EtherChannel Bandwidth and Costs

Finally, remember that the perceived bandwidth of a port-channel interface is equal to the sum of its active member links. For example, an EtherChannel with three active 100 Mbps members will show a bandwidth of 300 Mbps. Because members can still fail individually, the bandwidth of a port-channel interface can fluctuate without going down.

For more information on EtherChannel bandwidth and spanning tree considerations, see Etherchannel costs and failover.