Errdisable autorecovery
By stretch | Monday, September 14, 2009 at 7:10 a.m. UTC
There are a number of events which can disable a link on a Catalyst switch, such as the detection of a loopback, UDLD failure, or a broadcast storm. One of the most common causes of error disabling I've seen isn't technically an error, but a violation of a port security policy. Port security is a feature which allows for the restriction of incoming MAC addresses on a layer two switched interface. This is handy for mitigating the use of rogue devices customers purchase at Best Buy to help out with your network infrastructure design. In aggressive configurations, only a single MAC address (corresponding to the attached workstation) will be allowed inbound on a port; any other MAC address will trigger an error and the port will subsequently be disabled.
A default port security policy has been applied to FastEthernet0/1 in this example:
interface FastEthernet0/1 switchport access vlan 10 switchport mode access switchport port-security spanning-tree portfast
We can verify that the port is currently up and associated with a MAC address. Note that the violation mode is "shutdown."
Switch# show port-security interface f0/1 Port Security : Enabled Port Status : Secure-up Violation Mode : Shutdown Aging Time : 0 mins Aging Type : Absolute SecureStatic Address Aging : Disabled Maximum MAC Addresses : 1 Total MAC Addresses : 1 Configured MAC Addresses : 0 Sticky MAC Addresses : 0 Last Source Address:Vlan : 001d.60b3.0add:10 Security Violation Count : 0
When a violation is detected, the switch automatically places the port in the "err-disabled" shutdown state.
%PORT_SECURITY-2-PSECURE_VIOLATION: Security violation occurred, caused by MAC address 001d.60b3.0aff on port FastEthernet0/1. Switch# show interface f0/1 FastEthernet0/1 is down, line protocol is down (err-disabled) ...
By default, manual intervention by an administrator is necessary to restore the interface to working order; this can be done by issuing shutdown
followed by no shutdown
on the interface. The idea behind requiring administrative action is so that a human engineer can intercede, assess, and (ideally) correct the issue. However, some configurations may be prone to accidental violations, and a steady recurrence of these can amount to a huge time sink for the administrative staff.
This is where autorecovery can be of great assistance. We can configure the switch to automatically re-enable any error-disabled interfaces after a specified timeout period. This gives the offending issue a chance to be cleared by the user (for example, by removing an unapproved device) without the need for administrative intervention.
Switch(config)# errdisable recovery cause psecure-violation Switch(config)# errdisable recovery interval 300
The above configuration enables autorecovery for port security violations after five minutes. As evident in the list, autorecovery can apply to far more than just port security violations.
Switch# show errdisable recovery ErrDisable Reason Timer Status ----------------- -------------- arp-inspection Disabled bpduguard Disabled channel-misconfig Disabled dhcp-rate-limit Disabled dtp-flap Disabled gbic-invalid Disabled inline-power Disabled l2ptguard Disabled link-flap Disabled mac-limit Disabled link-monitor-failure Disabled loopback Disabled oam-remote-failure Disabled pagp-flap Disabled port-mode-failure Disabled psecure-violation Enabled security-violation Disabled sfp-config-mismatch Disabled storm-control Disabled udld Disabled unicast-flood Disabled vmps Disabled Timer interval: 300 seconds Interfaces that will be enabled at the next timeout:
Now when we force a violation on FastEthernet0/1, we see that it has been placed in the queue for automatic recovery:
Switch# show errdisable recovery ... Interfaces that will be enabled at the next timeout: Interface Errdisable reason Time left(sec) --------- ----------------- -------------- Fa0/1 psecure-violation 237
And two hundred and thirty-seven seconds later...
%PM-4-ERR_RECOVER: Attempting to recover from psecure-violation err-disable state on Fa0/1 Switch# show interface f0/1 FastEthernet0/1 is up, line protocol is up (connected) ...
Posted in Switching
Comments
September 14, 2009 at 8:28 a.m. UTC
Very useful feature, good coverage stretch! :)
September 14, 2009 at 10:40 a.m. UTC
Very helpful. Thank you!
September 21, 2009 at 3:35 a.m. UTC
First comment as a member! :)
September 23, 2009 at 6:48 p.m. UTC
Said it in twitter, but for the record - great post, and nx-os works about the same so double bonus!
September 24, 2009 at 2:39 a.m. UTC
I'm trying to figure out how this is a good thing. If port security is configured to shutdown the interface, why would you want it to automatically recover? Is it not effectively the same as just restricting the MAC without shutting down the interface?
September 26, 2009 at 5:19 p.m. UTC
Good question, The reason we decided to let it recover is that we do not have the staffing to log into the switch and babysit everyone that trips the port into error disable. So, what we did was have it reset itself after 10 minutes but if the person still has the violation it will go down again. So the bad device is never on the network. We will also get the alert and know that someone violated the policy and will be contacting their manager.
Self healing.
September 30, 2009 at 7:30 p.m. UTC
I have just enabled UDLD in a setup involving an etherchannel between two 3750s. The good thing is that the ports go into errdisable mode as soon as UDLD is detected but the bad thing is that I have to manually bring the port up. I was wondering if the ports can bring themselves up as soon as UDLD gets fixed. Any say?
October 1, 2009 at 1:44 p.m. UTC
Gr8 stuff...Very helpful! THNX
October 1, 2009 at 2:47 p.m. UTC
Wow, really nice post stretch. I'm just now learning about networking, on my 2nd year and just started working on switches about 2 months ago. this really helps with the learning process since the cisco site is really lacking. Taking my ccna's in may hopefully haha. Keep blogging i'm gonna need your help xD.
October 13, 2011 at 7:51 a.m. UTC
Helpful and well explained..
Thanks.
Tanveer.
January 28, 2012 at 8:49 a.m. UTC
packetlife is great source to solve many problems one of them is err-disable,
great source guys
Thanks
April 2, 2013 at 11:32 a.m. UTC
Great post.
Is there any way to set the "errdisable recovery" mode for bpduguard back to disabled?
I can't see a way from the configure>errdisable options.
Thanks
August 9, 2013 at 12:57 p.m. UTC
Hey so if a port has portfast enabled along with bpdugaurd . and it receives a bpdu . It moves to error disabled state . so when we do a shut no shut. does the portfast autorecover in 1) Normal STP 2) RSTP
February 22, 2015 at 9:27 p.m. UTC
So the question I'm trying to answer about err-disable in relation to psecure is this: when auto recovery is set, and port security is set, as in your example above, how does the switchport behave in the event of a persistent violation? Does the switch port simply keep bouncing up and down every however many seconds? Or does the errdisable recover/psecure somehow detect that the error condition is not resolved and wait to start the holddown timer until it is? If the port is shut down, I don't see how psecure could be tracking violations, unless errdisable is different that an ordinary shutdown?