RFC 5798 Brings IPv6 to VRRP
RFC 5798 was published this week, formalizing the latest incarnation of Virtual Router Redundancy Protocol (VRRP). VRRPv3 introduces support for IPv6 in addition to IPv4.
One might wonder why VRRP is necessary for IPv6 at all, given that IPv6 includes robust router discovery as part of its Neighbor Discovery (ND) protocol. Section 1.3 of the RFC explains it well (emphasis mine):
Neighbor Discovery (ND) includes a mechanism called Neighbor Unreachability Detection to detect the failure of a neighbor node (router or host) or the forwarding path to a neighbor. This is done by sending unicast ND Neighbor Solicitation messages to the neighbor node. To reduce the overhead of sending Neighbor Solicitations, they are only sent to neighbors to which the node is actively sending traffic and only after there has been no positive indication that the router is up for a period of time. Using the default parameters in ND, it will take a host about 38 seconds to learn that a router is unreachable before it will switch to another default router. This delay would be very noticeable to users and cause some transport protocol implementations to time out.
VRRP, with its sub-second timers, is able to fail over much more quickly than IPv6's built-in neighbor discovery. At the time of this writing, Cisco IOS does not yet support VRRPv3, though it appears likely that it will join Cisco's proprietary first hop redundancy protocols HSRP and GLBP in the near future.
As I've mentioned before, the IETF's New RFCs RSS feed is an excellent way to stay on top of new standards like this as they come out.
The Science of Network Troubleshooting
A number of people have written asking me what happened to a paper I wrote back in 2008 entitled "The Science of Network Troubleshooting." Unfortunately, I neglected to republish the paper after revamping packetlife.net in late 2009, so here it is again as a blog article.
Troubleshooting is not an art. Along with many other IT methodologies, it is often referred to as an art, but it's not. It's a science, if ever there were one. Granted, someone with great skill in troubleshooting can make it seem like a natural talent, the same way a professional ball player makes hitting a home run look easy, when in fact it is a learned skill. Another common misconception holds troubleshooting as a skill derived entirely from experience with the involved technologies. While experience is certainly beneficial, the ability to troubleshoot effectively arises primarily from the embrace of a systematic process, a science.
It's said that troubleshooting can't be taught, but I disagree. More accurately, I would argue that troubleshooting can't be taught easily, or to great detail. This is because traditional education encompasses how a technology functions; troubleshooting encompasses all the ways in which it can cease to function. Given that it's virtually impossible to identify and memorize all the potential points of failure a system or network might hold, engineers must instead learn a process for identifying and resolving malfunctions as they occur. To borrow a cliché analogy, teach a man to identify why a fish is broken, rather than expecting him to memorize all the ways a fish might break.
Troubleshooting as a Process
Essentially, troubleshooting is the correlation between cause and effect. Your proxy server experiences a hard disk failure, and you can no longer access web pages. A backhoe digs up a fiber, and you can't call a branch office. Cause, and effect. Moving forward, the correlation is obvious; the difficulty lies in transitioning from effect to cause, and this is troubleshooting at its core.
Consider walking into a dark room. The light is off, but you don't know why. This is the observed effect for which we need to identify a cause. Instinctively, you'll reach for the light switch. If the light switch is on, you'll search for another cause. Maybe the power's out. Maybe the breaker's been tripped. Maybe someone stole all the light bulbs (it happens). Without much thought, you investigate each of these possible causes in order of convenience or likelihood. Subconsciously, you're applying a process to resolve the problem.
Even though our light bulb analogy is admittedly simplistic, it serves to illustrate the fundamentals of troubleshooting. The same concepts are scalable to exponentially more complex scenarios. From a high-level view, the troubleshooting process can be reduced to a few core steps:
- Identify the effect(s)
- Eliminate suspect causes
- Devise a solution
- Test and repeat
- Mitigate
Step 1: Identify the Effect(s)
If you've been a network engineer for more than a few hours, you've been told at least once that the Internet is down. Yes, the global information infrastructure some forty years in the making has fallen to its knees and is in a state of complete chaos. All this is, of course, confirmed by Mary in accounting. Last time it was discovered her Ethernet cable had come unplugged, but this time she's certain it's a global catastrophe.
Correctly identifying a the effects of an outage or change is the most critical step in troubleshooting. A poor judgment at this first step will likely start you down the wrong path, wasting time and resources. Identifying an effect is not to be confused with deducing a probable cause; in this step we are focused solely on listing the ways in which network operation has deviated from the norm.
Identifying effects is best done without assumption or emotion. While your mind will naturally leap to possible causes at the first report of an outage, you must force yourself to adopt an objective stance and investigate the noted symptoms without bias. In the case of Mary's doomsday forecast, you would likely want to confirm the condition yourself before alerting the authorities.
Some key points to consider:
What was working and has stopped?
An important consideration is whether an absent service was ever present to begin with. A user may report an inability to reach FTP sites as an outage, not realizing FTP connections have always been blocked by the firewall as a matter of policy.
What wasn't working and has started?
This is can be a much less obvious change, but no less important. One example would be the easing of restrictions on traffic types or bandwidth, perhaps due to routing through an alternate path, or the deletion of an access control mechanism.
What has continued to work?
Has all network access been severed, or merely certain types of traffic? Or only certain destinations? Has a contingency system assumed control from a failed production system?
When was the change observed?
This critical point is very often neglected. Timing is imperative for correlation with other events, as we'll soon see. Also remember that we are often limited to noting the time a change was observed, rather than when it occurred. For example, an outage observed Monday morning which could have easily occurred at any time over the preceding weekend.
Who is affected? Who isn't?
Is the change limited to a certain group of users or devices? Is it constrained to a geographical or logical area? Is any person or service immune?
Is the condition intermittent?
Does the condition disappear and reappear? Does this happen at predictable intervals, or does it appear to be random?
Has this happened before?
Is this a recurring problem? How long ago did it happen last? What was the resolution? (You do keep logs of this sort of thing, right?)
Correlation with planned maintenance and configuration changes
Was something else being changed at this time? Was a device added, removed, or replaced? Did the outage occur during a scheduled maintenance window, either locally or at another site or provider?
Step 2: Eliminate Suspect Causes
Once we have a reliable account of the effect or effects, we can attempt to deduce probable causes. I say probable because deducing all possible causes is impractical, if not impossible. One possible cause is a power failure. Another possible cause is spontaneous combustion. Only one of these possible causes is probable.
There is a popular mantra of "always start with layer one," suggesting that the physical connectivity of a network should be verified before working on the higher layers. I disagree, as this is misleading and often impractical. You're not going to drive out to a remote site to verify everything is plugged in if a simple ping verifies end-to-end connectivity. Similarly, it's unlikely that any cables were disturbed if you can verify with relative certainty no one has gone near the suspect devices. Perhaps this is an oversimplified argument, but verifying physical connectivity is often needlessly time consuming and superseded by alternative methods.
Instead, I suggest narrowing causes in order of combined probability and convenience. For example, there might be nothing to indicate DNS is returning an invalid response, but performing a manual nslookup takes roughly two seconds, so this is easily justified. Conversely, comparing a current device configuration to its week-old backup and accounting for any differences may take a considerable amount of time, but this presents a high probability of exposing a cause, so it too is justified.
The order in which you decide to eliminate suspect causes is ultimately dependent on your experience, your familiarity with the infrastructure, and your allowance for time. Regardless of priority, each suspect cause should undergo the same process of elimination:
Define a working condition
You can't test for a condition unless you know what condition to expect. Before performing a test, you should have in mind what outcome should be produced in the absence of an outage. For example, performing a traceroute to a distant node is meaningless if you can't compare it against a traceroute to the same destination under normal conditions.
Define a test for that condition
Ensure that the test you perform is in fact evaluating the suspect cause. For instance, pinging an E-mail server doesn't explicitly guarantee that mail services are available, only the server itself (technically, only that server's address). To verify the presence of mail services, a connection to the relevant daemon(s) must be established.
Apply the test and record the result
Once you've applied the test, record its success or failure in your notes. Even if you've eliminated the cause under suspicion, you have a reference to remind you of this and avoid wasting time repeating the same test again unnecessarily.
It is common to uncover multiple failures in the course of troubleshooting. When this happens, it is important to recognize any dependencies. For example, if you discover that E-mail, web access, and a trunk link are all down, the E-mail and web failures can likely be ignored if they depend on the trunk link to function. However, always remember to verify these supposed secondary outages after the primary outage has been resolved.
Step 3: Devise a Solution
Once we have identified a point of failure, we want to continue our systematic approach. Just as with testing for failures, we can apply a simple methodology to testing for solutions. In fact, the process very closely mirrors the steps performed to eliminate suspect causes.
Define the failure
At this point you should have a comfortable idea of the failure. Form a detailed description so you have something to test against after applying a solution. For example, you would want to refine "The Internet is down" to "Users in building 10 cannot access the Internet because their subnet was removed from the outbound ACL on the firewall."
Define the proposed solution
Describe exactly what changes are to be made, and exactly what the expected outcome is. Blanket solutions such as arbitrarily rebooting a device or rebuilding a configuration from scratch might fix the problem, but they prevent any further diagnosis and consequently impede mitigation.
Apply the solution and record the result
Once we have a defined failure and a proposed solution, it's time to pit the two against each other. Be observant in applying the solution, and record its product. Does the outcome match what you expected? Has the failure been resolved?
In addition to our defined process, some guidelines are well worth mentioning.
Maintain focus
Far too often I encounter a technician who, upon becoming frustrated with a failure or failures, opts to recklessly reboot, reconfigure, or replace a device instead of troubleshooting systematically. This is the high-tech equivalent of pounding something with a hammer until it works. Focus on one failure at a time, and one solution at a time per failure.
Watch out for hazardous changes
When developing a solution, remember to evaluate what effect it might have on systems unrelated to those being troubleshot. It's a horrible feeling to realize you've fixed one problem at the expense of causing a much larger one. The best course of action when this happens is typically to immediately reverse the change which was made. Note that this is only possible with a systematic approach.
Step 4: Test and Repeat
Upon implementing a solution and observing a positive effect, we can begin to retrace our steps back toward the original symptoms. If any conditions were overlooked because they were decided to be dependent on the recently resolved failure, test for them again. Refer to your notes from the initial report and verify that each symptom has been resolved. Ensure that the same tests which were used to identify a failure are used to confirm functionality.
If you notice that a failure or failures remain, pick up where you left off in the testing cycle, annotate it and press forward.
Step 5: Mitigate
The troubleshooting process does not end when the problem has been resolved and everyone is happy. All of your hard work up until this point amounts to very little if the same problem occurs again tomorrow. In IT, problems are commonly fixed without ever being fixed. Band-aids and hasty installations are not acceptable substitutes for implementing a permanent and reliable solution. So to speak, many people will go on mopping the floor day after day without ever fixing the leak.
A permanent solution may be as complex as redesigning the routed backbone, or as simple as moving a power strip somewhere it won't be tripped on anymore. A permanent solution also doesn't have to be 100% effective, but it should be as effective as is practical. At the absolute minimum, ensure that you record the observed failure and the applied solution, so that if it the condition does reoccur you have an accurate and dated reference.
A Final Word
Everyone has his or her own preference in troubleshooting, and by no means do I consider this paper conclusive. However, if there's only one concept you take away, make it this: above all else, remain calm. You're no good to anyone in a panic. One's ability to manage stress and maintain a professional demeanor even in the face of utter chaos is what makes a good engineer great.
Most network outages, despite what we're led to believe by senior staff, are not the end of the world. There are instances where downtime can lead to loss of life; fortunately, this isn't the case with most networks. Money may be lost, time may be wasted, and feelings may be hurt, but when the problem is finally resolved, odds are you've learned something valuable.
Navigating Cisco.com Documentation
From what I've seen interacting with other engineers, it seems that most of us, when we need to research something relevant to Cisco networking, go straight to Google. This is typically the most expedient path toward an answer, but it isn't necessarily the most reliable or the most accurate. The problem is that Google or another search engine will direct you to whatever resource on cisco.com it considers most relevant; one must be careful to consider the hardware platform and/or IOS version specific to the situation at hand.
For example, an engineer who googles for documentation concerning the configuration of a particular IOS feature will often find what he needs, but may not be aware that the documentation he is looking at is for a newer or older version of software, or that it applies to a family of hardware different from the device he is configuring. This might not be a problem all the time, but it is a good way to get bitten by minor differences in supported features and configuration syntax.
A more reliable approach is to navigate Cisco's documentation manually to find the exact information you need. People often complain that Cisco's online documentation is too difficult to navigate, but after digging through it numerous times to find references for the many blog articles I've written, I think it merely takes a bit of strategy. That's why I decided to write this article.
Starting at http://cisco.com/, select Support from the navigation menu at the top of the page. This brings you to Cisco's main support page, which is essentially a collection of pointers to other places all around the site. The portion of the page we're most interested in is the part with two blue boxes of links:
Two navigation options are presented to the user here. The first option is to navigate by product or technology name (e.g. routers, security, wireless, etc.). This option also includes the ability to search. The second option is to navigate by task (e.g. troubleshoot, configure, design, etc.).
This is, in my opinion, bad design; if a user intends to configure a router, which link is he supposed to follow, "Routers" or "Configure"? That question is ultimately moot, because all of these links (with the exception of "Download Software") effectively all point to the same place anyway. Like I said, this page is just a collection of references to other resources on cisco.com; the master documentation index is actually at http://www.cisco.com/cisco/web/psa/default.html.
Depending on which link you chose on the prior page, you will be directed to a subsection of this master index. Personally, I prefer to always start from the root of the master index itself and dig into whatever I need from there. Note that many of the subnavigation links down the left side of the page are the same links shown on the main support page.
There are two basic paths down which you can begin navigating to the specific documentation you desire: "Products" and "Technology". Choosing "Products" allows one to navigate down a hierarchy of all of Cisco's hardware and software offerings and select the specific device or version of code (e.g. IOS 12.T) for which documentation is desired. The "Technology" path is more general in nature, and is better suited for theory and design discussion for technologies which span many products.
As it is obviously impractical to provide a complete list of what can be found where in the world of Cisco documentation, hopefully a few common examples will suffice.
Example 1: Configuring EIGRP on a Router
You might assume that to find the documentation for this, you should navigate to "Products" > "Routers" and select your specific platform (for example, the 2800 series). However, this is not the case. Most aspects of IOS configuration will be the same across all routers running the same version of IOS, regardless of the underlying hardware platform. For this reason, most IOS (and other operating system) configuration tasks are found under "Products" > "Cisco IOS and NX-OS Software".
Selecting the major version of software you'll be working with (as indicated by the output of show version on the device to be configured) takes you to the introduction page for that software. This example assumes IOS 12.4T is in use.
From here, select "Configuration Guides" to see the collection of configuration guides available for this version of IOS. Select the guide most relevant to the task to be performed. For the purpose of this example, we want "Cisco IOS IP Routing: EIGRP Configuration Guide, Release 12.4T".
This takes us to the manual specific to EIGRP configuration on IOS 12.4T.
The manual can be viewed online, navigated using the links at left, or optionally downloaded in part or entirely as a PDF file. A word of caution: while it may be tempting to download all available manuals in PDF for offline access, make sure that they are refreshed from time to time as the online documentation is updated and corrected.
To summarize, the path we followed to find the correct documentation for our example is:
- Products
- Cisco IOS and NX-OS Software
- Cisco IOS
- Cisco IOS Software Release 12.4 Family
- Cisco IOS
- Cisco IOS and NX-OS Software
Example 2: Configuring a Catalyst 3560
Catalyst switches are approached differently from routers, as much of their available features and configuration options may vary from platform to platform. If we look again under "Products" and navigate to the IOS version in use on the Catalyst switch, we see that the section is lacking any configuration guides.
To access the configuration guides for switches, we must first navigate to "Products" > "Switches", and select the model of switch we wish to configure. On the introduction page for that model, we then follow "Configuration Guides" and select the specific version of IOS running on the switch. This example assumes IOS version 12.2(52)SE.
This directs us to the manual specific to both our hardware platform and the version of IOS which it runs.
Example 3: Researching the Purpose of a Command
Assume we notice the command no ip unreachables present in the configuration of an interface on a router, and wish to learn more about its purpose. Starting once more at the root of the master documentation index, we navigate to "Products" > "Cisco IOS and NX-OS Software" and select the relevant major version of IOS. Whereas the first example had us looking for a configuration guide, this time we want a command reference.
There are two ways of locating a command. If you know the general category of the command, select "Command References" and find the relevant reference. If not, select "Master Index" and select "Cisco IOS Master Command List" for the appropriate software release. Commands are sorted alphabetically within the reference (omitting the leading negation keyword no).
- Products
- Cisco IOS and NX-OS Software
- Cisco IOS
- Cisco IOS Software Release 12.4 Family
- Cisco IOS
- Cisco IOS and NX-OS Software
Note that the master command reference is not a redundant resource; following it ultimately directs you to the correct categorized reference for a command.
Hopefully these examples have provided a better understanding of how Cisco's documentation is arranged, and how to navigate it more easily.
Cisco "Go" links reference in the wiki
One of my (very) slow-paced background projects has been building a list of all the handy Cisco.com "go" links I come across. "Go" links are easily remembered shortcut links to various content throughout Cisco.com. For example, http://cisco.com/go/fn redirects to Cisco's feature navigator tool at http://tools.cisco.com/ITDIT/CFN/jsp/index.jsp. Unfortunately, I've been unable to find a master index of all "go" links, so I started the Cisco.com "go" links wiki article to record the ones I've found useful.
At the time of this writing, the article contains just a few dozen links; I invite readers to add any others they've found handy. However, please keep in mind that this isn't intended to be a master index by any means. Google says there are thousands of these links; the list in the wiki only needs the links most useful to engineers (e.g., no marketing links please).
Teaching binary and other bases
I've seen a lot of people struggle when first learning decimal-to-binary and other conversions, mostly because they find themselves overwhelmed with conversion charts and don't quite grasp the concept of a numbering base. I decided to write this post when I realized the method I use to teach people binary and hexadecimal isn't used in any of the books I have (which isn't to say it's unique by any means, but perhaps not very widespread).
If you do use the methodology described here, I would appreciate feedback on how well it worked (or didn't) so that it can be improved.
Step 1: Relearn to Count in Decimal
Think back to kindergarten or whenever you first learned to count. As a typical example, let's say you have seventeen apples. To express this as a decimal value, start with zero and begin counting: 1, 2, 3... up to 9. When you reach the tenth apple, you've run out of digits. This is because we count in a base-10 numbering system, more commonly known as decimal, which employs the digits 0 through 9.
To express values greater than the highest available digit (e.g. values greater than 9), we must use multiple digits. We increase the column to the left (which is zero if not present) by one and reset the current column to zero; 09 becomes 10. This is a recursive process; it is repeated as necessary until we reach the leftmost column of the number (e.g. counting from 999 to 1000).
Here's the key concept to absorb: every numbering base works this way. The only difference is the amount of digits you can use.
Step 2: Counting with Fewer Digits
Now let's try counting to seventeen again, but this time you can't use the digits 8 or 9. We begin counting, 1, 2, 3... 7. Since we're out of available digits at this point, we increase the column to the left from 0 (or nothing) to 1, and reset the current column to zero. We continue increasing the value of this column again until we run out of digits a second time. The column to the left is again increased and the current column reset. We continue increasing the value of the rightmost column again until we run out of apples to count.
Congratulations, you now know how to count in octal (or base-8), which uses only the digits 0 through 7.
Step 3: Counting with Even Fewer Digits
Let's extend this concept to its extreme; for this exercise, you can only use the digits 0 and 1. Start counting: 0, 1 -- we're already out of digits, so add another column, resume counting, and repeat. Notice that we have to increase the column to the left and start over for every other apple we count.
And now you can count in binary.
One note here: binary numbers are typically written in groups of eight bits (_binary digits_), with leading zeros for padding. This is done out of convention simply because this is typically the smallest amount of bits a computer deals with at any one time.
Step 4: Counting with More Digits
"What do you mean, more digits? We only have ten!" True, we only have ten numeric digits; that's why we begin adding letters to the mix for bases like hexadecimal (base-16) or "hex" for short.
Note that hexadecimal numbers are normally denoted with a leading "0x"; the number in the above example would typically be written 0x11. Padded to fill a 32-bit variable, it would be written 0x00000011.
Step 5: Converting Back to Decimal
To convert a number from another base to a decimal number, each digit is multiplied by the value of its place. To illustrate this, first we example a the decimal number - 742, for instance - can be expressed as the sum of its place values:
The value of each place is derived as the base number (10 for decimal, 8 for octal, etc.) to the power of its column (starting with zero). In other words:
(A quick math tip: remember that any number to a power of zero equals one.)
Realizing this, we can easily extend this concept to converting other bases to decimal. Let's try our octal number 21:
Conversion from binary takes a bit longer but works exactly the same way:
It just takes a bit of practice to begin memorizing the first eight powers of two (1, 2, 4, 8, 16, 32, 64, and 128). Once you've got them locked in your head, converting between decimal and binary becomes second nature.
OSPFv2 versus OSPFv3
OSPFv3 is to IPv6 what OSPFv2 is to IPv4. The two versions of OSPF naturally have much in common, however there are several important differences in the way the two protocols operate. This article seeks to highlight some of the more noteworthy deviations. (For a more thorough discussion, see section 2 of RFC 5340.) For illustration, both OSPFv2 and OSPFv3 have been configured on the example topology below.
For some IOS OSPFv3 configuration fundamentals, see IPv6 and OSPFv3.
New LSA Types
OSPFv3 carries over the seven basic LSA types we're familiar with from OSPFv2. However, the type 1 and 2 LSAs have been re-purposed, as will be discussed in a bit. OSPFv3 also introduces two new LSA types: Link and Intra-area Prefix.
| OSPFv3 | OSPFv2 | ||
| 0x2001 | Router LSA | 1 | Router LSA |
| 0x2002 | Network LSA | 2 | Network LSA |
| 0x2003 | Inter-area Prefix LSA | 3 | Network Summary LSA |
| 0x2004 | Inter-area Router LSA | 4 | ASBR Summary LSA |
| 0x4005 | AS-External LSA | 5 | AS-External LSA |
| 0x2006 | Group Membership LSA | 6 | Group Membership LSA |
| 0x2007 | Type-7 LSA | 7 | NSSA External LSA |
| 0x0008 | Link LSA | ||
| 0x2009 | Intra-area Prefix LSA | ||
Separation of Addressing from the SPF Tree
One of the biggest advantages of OSPFv3 over its predecessor is the separation of IP addressing from the calculation of the SPF tree. One of OSPFv3's new LSAs, the Intra-area Prefix LSA (type 9), handles intra-area network information that was previously included in OSPFv2 type 2 LSAs. Because IP addressing is communicated independent of the LSAs used for SPF tree calculation, adding or modifying IP subnets within the OSPF domain will not affect the integrity of the SPF tree (which is concerned only with nodes and the links between them) and avoid forcing an SPF recalculation.
For illustration, following are the two OSPF databases from R2 in our example topology:
R2# show ip ospf database OSPF Router with ID (2.2.2.2) (Process ID 1) Router Link States (Area 0) Link ID ADV Router Age Seq# Checksum Link count 2.2.2.2 2.2.2.2 1697 0x80000001 0x00264B 2 4.4.4.4 4.4.4.4 1693 0x80000002 0x005216 2 Summary Net Link States (Area 0) Link ID ADV Router Age Seq# Checksum 10.0.123.0 2.2.2.2 1683 0x80000002 0x00DEC5 Router Link States (Area 1) Link ID ADV Router Age Seq# Checksum Link count 1.1.1.1 1.1.1.1 1656 0x80000003 0x00BB55 1 2.2.2.2 2.2.2.2 1655 0x80000003 0x008086 1 3.3.3.3 3.3.3.3 1656 0x80000003 0x003FBF 1 Net Link States (Area 1) Link ID ADV Router Age Seq# Checksum 10.0.123.3 3.3.3.3 1656 0x80000001 0x007B0E Summary Net Link States (Area 1) Link ID ADV Router Age Seq# Checksum 10.0.0.0 2.2.2.2 1694 0x80000001 0x003BB2
R2# show ipv6 ospf database OSPFv3 Router with ID (2.2.2.2) (Process ID 1) Router Link States (Area 0) ADV Router Age Seq# Fragment ID Link count Bits 2.2.2.2 1694 0x80000002 0 1 B 4.4.4.4 1695 0x80000002 0 1 None Inter Area Prefix Link States (Area 0) ADV Router Age Seq# Prefix 2.2.2.2 1692 0x80000001 2001:DB8:0:123::/64 Link (Type-8) Link States (Area 0) ADV Router Age Seq# Link ID Interface 2.2.2.2 1696 0x80000002 6 Se1/0 4.4.4.4 1699 0x80000002 6 Se1/0 Intra Area Prefix Link States (Area 0) ADV Router Age Seq# Link ID Ref-lstype Ref-LSID 2.2.2.2 1696 0x80000001 0 0x2001 0 4.4.4.4 1699 0x80000001 0 0x2001 0 Router Link States (Area 1) ADV Router Age Seq# Fragment ID Link count Bits 1.1.1.1 1652 0x80000005 0 1 None 2.2.2.2 1652 0x80000005 0 1 B 3.3.3.3 1649 0x80000005 0 1 None Net Link States (Area 1) ADV Router Age Seq# Link ID Rtr count 3.3.3.3 1661 0x80000001 4 3 Inter Area Prefix Link States (Area 1) ADV Router Age Seq# Prefix 2.2.2.2 1693 0x80000001 2001:DB8::/64 Link (Type-8) Link States (Area 1) ADV Router Age Seq# Link ID Interface 1.1.1.1 1697 0x80000002 4 Fa0/0 2.2.2.2 1698 0x80000002 4 Fa0/0 3.3.3.3 1696 0x80000002 4 Fa0/0 Intra Area Prefix Link States (Area 1) ADV Router Age Seq# Link ID Ref-lstype Ref-LSID 3.3.3.3 1662 0x80000001 4096 0x2002 4
Note that the OSPFv3 database appears larger, because network addressing is now stored separately in the new Link and Intra-area LSAs.
LSA Flooding Scope
Looking again at the above table of LSA types, one might note that OSPFv3 LSA numbers begin with differing numbers (for example, 0x2001 versus 0x4005). The second and third most-significant bits in an LSA number (referenced as bits S2 and S1, respectively) indicate its flooding scope:
| S2 | S1 | Flooding Scope |
| 0 | 0 | Link-local |
| 0 | 1 | Area |
| 1 | 0 | AS (OSPF domain) |
| 1 | 1 | Reserved |
Most LSAs begin with 0x2, indicating that they are flooded within an area. Two LSAs of note are the AS-External LSA (which advertises external routes) with a domain-wide scope, and the Link LSA with a scope appropriately restricted to the local link.
Link-local Addressing
An OSPFv2 router forms adjacencies using its configured IPv4 interface address:
R1# show ip ospf neighbor detail
Neighbor 2.2.2.2, interface address 10.0.123.2
In the area 1 via interface FastEthernet0/0
Neighbor priority is 1, State is FULL, 6 state changes
DR is 10.0.123.3 BDR is 10.0.123.2
Options is 0x52
LLS Options is 0x1 (LR)
Dead timer due in 00:00:37
Neighbor is up for 00:15:32
Index 2/2, retransmission queue length 0, number of retransmission 0
First 0x0(0)/0x0(0) Next 0x0(0)/0x0(0)
Last retransmission scan length is 0, maximum is 0
Last retransmission scan time is 0 msec, maximum is 0 msec
...
OSPFv3, however, makes use of IPv6's link-local address scope (FE80::/10). All OSPFv3 adjacencies are formed using link-local addresses:
R1# show ipv6 ospf neighbor detail
Neighbor 3.3.3.3
In the area 1 via interface FastEthernet0/0
Neighbor: interface-id 4, link-local address FE80::C003:7DFF:FE07:0
Neighbor priority is 1, State is FULL, 6 state changes
DR is 3.3.3.3 BDR is 2.2.2.2
Options is 0x6670B96D
Dead timer due in 00:00:32
Neighbor is up for 00:16:48
Index 1/1/1, retransmission queue length 0, number of retransmission 0
First 0x0(0)/0x0(0)/0x0(0) Next 0x0(0)/0x0(0)/0x0(0)
Last retransmission scan length is 0, maximum is 0
Last retransmission scan time is 0 msec, maximum is 0 msec
Also note in the second output that neighboring routers are referred to not by IP address, but by OSPF ID, demonstrating OSPFv3's fundamental separation of the SPF tree and IP addressing. It is worth pointing out that OSPFv3 router IDs are not IPv4 addresses; they are merely unique 32-bit identifiers expressed in the familiar dotted-decimal notation.
Support for Multiple Instances Per Link
OSPFv3 includes support for multiple instances of OSPF running in parallel across a common link. This is especially handy for shared network segments such as those found in Internet exchange points. On Cisco IOS, OSPFv3 instances are configured by appending the instance argument to the ipv6 ospf statement:
Router(config-if)# ipv6 ospf 100 area 0 instance 1
Removal of OSPF-specific Authentication
OSPFv2 authentication is achieved by implementing a shared secret and MD5 HMAC supported as part of the OSPFv2 protocol. OSPFv3 does away its own support for authentication entirely, instead relying on the more flexible IPsec framework offered by IPv6. For more detail on how OSPFv3 authentication is configured, see OSPFv3 authentication.
Improved Handling of Unknown LSA Types
OSPFv2 routers simple discard LSAs of an unknown type. OSPFv3 LSAs may be discarded, or optionally stored and flooded as though they were understood. Which action is taken is determined by the most-significant bit (the U bit) in the LSA type number. This option allows for the introduction of new LSA types across OSPF networks where only some routers support the new types.
Cisco IOS Configuration Differences
Cisco notes some differences in the way OSPFv2 and OSPFv3 are configured here. A brief summary:
- Enabling OSPF on an IPv6 interface automatically enables OSPFv3; an explicit OSPFv3 routing process does not need to be administratively created (though it likely still will be for other purposes).
- OSPFv3 interfaces must be designated under interface configuration; there is no option to designate interfaces using the
networkcommand under router configuration. - NBMA neighbors must be identified by router ID, not by IP address.
- Like OSPFv2, OSPFv3 will take its router ID from the highest-numbered IPv4 loopback interface; however, as this IPv4 address is likely irrelevant your IPv6 network, it is recommended to manually specify a router ID with the
router-idcommand under OSPFv3 router configuration.
IGP posters on sale for $9.99 this week
Quick announcement today: the 36x24" interior routing protocol posters are on sale this week for $9.99 plus shipping. They make great lab decor!
Experimenting with VLAN hopping
Most network engineers have been told at one point or another never to use VLAN 1 for user access. The motivation behind this warning is the result of fear concerning VLAN hopping attacks, wherein an attacker can send packets with a specially-crafted 802.1Q header(s) to "hop" from one VLAN to another. This theory relies on the assumption that a switch will happily forward 802.1Q-tagged frames ingressing an access port.
We can devise a simple scenario to test this by injecting crafted packets toward an access switch port on VLAN 1 and monitoring the traffic traversing the upstream trunk. If the attack succeeds, we will see our crafted packets appear on the trunk side of the switch with the VLAN ID of our choosing.
Both switches used for the lab are Catalyst 3550s running IOS 12.2(44)SE2. FastEthernet0/11 on S1 has been configured as a static access port in VLAN 1, and FastEthernet0/13 has been configured to trunk all VLANs, with VLAN 1 as the (default) native VLAN:
interface FastEthernet0/11 switchport mode access switchport nonegotiate ! interface FastEthernet0/13 switchport trunk encapsulation dot1q switchport mode trunk switchport nonegotiate
We can use Scapy to craft arbitrary frames with the necessary 802.1Q headers. The following command within Scapy generates an ICMP echo request (ping) with two 802.1Q headers (VLAN 1 and VLAN 10):
Welcome to Scapy (2.0.0.5 beta) >>> sendp(Ether(dst='ff:ff:ff:ff:ff:ff', src='00:01:02:03:04:05')/Dot1Q(vlan=1)/Dot1Q(vlan=10)/ IP(dst='255.255.255.255', src='192.168.0.1')/ICMP()) . Sent 1 packets.
Monitoring eth0 with Wireshark, we can verify that the frame was sent out with two 801.Q headers with the desired VLAN IDs.
If the VLAN hopping attack theory is valid, we should observe our frame exiting S1's FastEthernet0/13 onto the trunk with an 802.1Q tag specifying VLAN 10. However, by monitoring eth1, we can observe that this frame is not switched out onto the trunk. Rather, because S1 detected an 802.1Q-tagged frame ingress on an access port, the frame was discarded. (Interestingly, though, the receipt of such a frame does not increase any interface error counters.)
I've tried crafting the malicious frame in several ways, including swapping the order of the headers and sending only one header, but none of my attempts were successful in getting a tagged frame onto the trunk in any VLAN, even the native (untagged) VLAN. Only by transmitting the frame without any 802.1Q header was it successfully switched onto the trunk (untagged). These observations suggest that VLAN hopping attacks are not effective against modern switches (or at least the Catalyst 3550 running 12.2(44)SE2), in contrast to the findings of an @stake security assessment (PDF link) performed in 2002.
Still, an engineer might wish to enable the vlan dot1q tag native feature on platforms which support it. This forces the tagging of all traffic traversing 802.1Q trunk links, including that belonging to the native VLAN. Note that it is important that this feature be enabled on both ends of a trunk.
Finally, I feel inclined to include a footnote here regarding DTP, as it will no doubt come up in the comments if I do not. Switchports configured as switchport mode dynamic are left vulnerable to the unintended formation of trunks when an attacker spoofs DTP negotiation with the switch. This is not VLAN hopping; rather, it is a simple (yet very dangerous) misconfiguration issue. Note that DTP is enabled on Cisco switch ports by default.
SOHO configuration management
In large enterprise and service provider environments, it is considered best practice to routinely back-up and archive the configurations of all network devices. These backups can be used in an emergency to restore a device's configuration which was inadvertently altered or erased. Such a scheme is typically accomplished via automated FTP or SCP transfers to a centralized database, often integrated with a network management system like CiscoWorks.
However, this approach is often poorly suited for the small office/home office (SOHO) and similar small-scale environments. In such situations, where only one or few devices are present, it may not be justifiable or practical to run an on-site server dedicated for configuration backups.
Cisco IOS includes a configuration archive feature which allows for the creation of both manual and automated configuration snapshots, stored locally on the router's own filesystem.
To enable this feature, we enter archive configuration and specify a path where archived configurations are to be stored. In the following example, we've created a directory appropriately named archive on the Flash filesystem.
Demarc# mkdir archive
Create directory filename [archive]?
Created dir flash:archive
Demarc# dir
Directory of flash:/
1 -rw- 23587052 Jan 9 2010 17:16:58 -05:00 c181x-advipservicesk9-mz.124-24.T.bin
4 drw- 0 Feb 19 2010 22:47:02 -05:00 archive
7 -rw- 720 Jan 9 2010 03:23:34 -05:00 vlan.dat
128237568 bytes total (104640512 bytes free)
Demarc# conf t
Enter configuration commands, one per line. End with CNTL/Z.
Demarc(config)# archive
Demarc(config-archive)# path flash:/archive/
Here, we can also optionally set the maximum number of configurations to store at any time (the default is 10, maximum is 14), and/or enable automatic archiving with the time-period command. Here we'll increase the default number of configurations stored to 14, since we have plenty of space on the filesystem, and enable weekly automatic archivals (seven days are equal to 10080 minutes).
Demarc(config-archive)# maximum 14 Demarc(config-archive)# time-period 10080
We can view the current archive with the show archive command:
Demarc# show archive The maximum archive configurations allowed is 14. There are currently no configuration saved. The next archive file will be named flash:/archive/-0 Archive # Name 1 2 3 4 5 6 7 8 9 10 11 12 13 14
As you can see, the archive is currently empty. Let's begin by archiving the router's current configuration with the archive config command:
Demarc# archive config
Looking at the archive again, we see the first configuration slot has been filled:
Demarc# show archive The maximum archive configurations allowed is 14. There are currently 1 archive configurations saved. The next archive file will be named flash:/archive/-1 Archive # Name 1 flash:/archive/-0 <- Most Recent 2 3 4 5 6 7 8 9 10 11 12 13 14 Demarc# dir flash:/archive/ Directory of flash:/archive/ 8 -rw- 7758 Feb 19 2010 22:58:50 -05:00 -0 128237568 bytes total (104632320 bytes free)
Now, to illustrate the benefits of configuration archiving, we'll configure interface FastEthernet1, which currently has no configuration, to support a routed link:
Demarc# show run interface f1 Building configuration... Current configuration : 81 bytes ! interface FastEthernet1 no ip address duplex auto speed auto end Demarc# conf t Enter configuration commands, one per line. End with CNTL/Z. Demarc(config)# interface f1 Demarc(config-if)# ip address 192.0.2.1 255.255.255.0 Demarc(config-if)# ^Z Demarc# show run interface f1 Building configuration... Current configuration : 92 bytes ! interface FastEthernet1 ip address 192.0.2.1 255.255.255.0 duplex auto speed auto end Demarc# write Building configuration... [OK]
Note that configuration archival is performed separately from start-up configuration synchronization; write or copy run start is still needed to save the running configuration to the default start-up configuration.
After archiving the configuration again, we see that there are now two copies in the archive:
Demarc# archive config Demarc# show archive The maximum archive configurations allowed is 14. There are currently 2 archive configurations saved. The next archive file will be named flash:/archive/-2 Archive # Name 1 flash:/archive/-0 2 flash:/archive/-1 <- Most Recent 3 4 5 6 7 8 9 10 11 12 13 14
The show archive config differences command can be used to generate a Diff-style list of differences between any two files:
Demarc# show archive config differences flash:/archive/-0 flash:/archive/-1 Contextual Config Diffs: interface FastEthernet1 +ip address 192.0.2.1 255.255.255.0 interface FastEthernet1 -no ip address
(Note that the files are referenced here by their location in the flash: filesystem. Referencing them via the archive: filesystem doesn't seem to work; if anyone knows how to use it, please mention it in the comments.)
The more command can be used to view any individual file in its entirety:
Demarc# more flash:/archive/-0 ! version 12.4 service timestamps debug datetime msec service timestamps log datetime msec service password-encryption ! hostname Demarc ...
Now let's assume we want to revert to the prior configuration. All we need to do is issue the config replace command referencing the file we want to revert to:
Demarc# configure replace flash:/archive/-0 This will apply all necessary additions and deletions to replace the current running configuration with the contents of the specified configuration file, which is assumed to be a complete configuration, not a partial configuration. Enter Y if you are sure you want to proceed. ? [no]: y Total number of passes: 1 Rollback Done Demarc# show run interface f1 Building configuration... Current configuration : 71 bytes ! interface FastEthernet1 no ip address duplex auto speed auto end
Another handy feature of the configuration archive is its ability to log individual configuration commands entered by users. This feature is enabled with log config under archive configuration:
Demarc(config)# archive Demarc(config-archive)# log config Demarc(config-archive-log-cfg)# ? commands for controlling config logging: default Set a command to its defaults exit Exit from the log config submode hidekeys suppress output (e.g. passwords) when displaying logged commands logging Modify config logging parameters no Negate a command or set its defaults notify Send logged commands to target applications record What to record in the config logger Demarc(config-archive-log-cfg)# logging enable Demarc(config-archive-log-cfg)# logging size 500 Demarc(config-archive-log-cfg)# hidekeys
The log has been configured to record the last 500 configuration commands. The hidekeys command censors passwords and other sensitive information.
The log can be viewed with the show archive log config command. We can see that it has started recording commands as soon as logging was enabled (including that command itself):
Demarc# show archive log config all
idx sess user@line Logged command
1 1 stretch@vty0 | logging enable
2 1 stretch@vty0 | logging size 500
3 1 stretch@vty0 | hidekeys
Finally, we save our running configuration to start-up and commit it to the archive once more:
Demarc# wr Building configuration... [OK] Demarc# archive config Demarc# show archive The maximum archive configurations allowed is 14. There are currently 3 archive configurations saved. The next archive file will be named flash:/archive/-3 Archive # Name 1 flash:/archive/-0 2 flash:/archive/-1 3 flash:/archive/-2 <- Most Recent 4 5 6 7 8 9 10 11 12 13 14
Network Address Translation (NAT) cheat sheet
Announcing cheat sheet #25: Network Address Translation (NAT).
I realize it's been a while since I came out with the prior cheat sheet (though in fairness I did produce a pretty slick poster more recently). I promise I'll try not to wait so long before the next one.
