Monday, April 27, 2015 at 2:05 p.m. UTC by stretch
I came across an odd little issue recently involving equal-cost multipath (ECMP) routing and traceroute. Traceroutes from within our network to destinations out on the Internet were following two different paths, with one path being one hop longer than the other. This resulted in mangled traceroute output, impeding our ability to troubleshoot.
The relevant network topology comprises a mesh of two edge routers and two core switches. Each edge router has a number of transit circuits to different providers, and advertises a default route via OSPF to the two core switches below. The core switches each load-balance traffic across both default routes to either edge routers.
Because each edge router has different providers, some destinations are routed out via edge1 and others via edge2, which means sometimes a packet will be routed to edge2 via edge1, or vice versa.
Routers typically employ a hash function using layer three and four information from each packet to pseudo-randomly distribute traffic across equal links. Typically, all packets belonging to a flow (e.g. all packets with the same source and destination IP and port numbers) follow the same path.
However, in this case traceroute packets were being split across two path of unequal length, which made traceroute output pretty much unreadable. We noticed that only UDP traceroutes were affected; ICMP traceroutes reported one path as normal.
Monday, January 26, 2015 at 2:50 p.m. UTC by stretch
- At what OSI layer does protocol X operate?
- What's the difference between a router and a multilayer switch?
- What's the difference between forwarding and control planes?
- What's the difference between MTU and MSS?
- What's the difference between a VLAN interface and a BVI?
- How do tunnel interfaces work?
- What do NAT terms like "inside local" mean?
- Can I use the network and broadcast addresses in a NAT pool?
- Why do we need IP addresses? Can't we just use MAC addresses for everything?
- Does QoS provide more bandwidth?
Friday, January 16, 2015 at 1:12 a.m. UTC by stretch
- Where do IP addresses and domain names come from?
- Did we really run out of IPv4 addresses?
- Can I buy more IP addresses?
- Does IPv6 really provide a bazillion addresses?
- Why does IPv6 use hexadecimal addressing?
- What is IPAM?
- How do I create an IP addressing scheme?
- How does IPv6 subnetting work?
- What prefix length should I use on point-to-point links?
- How should I name devices on my network?
Friday, January 9, 2015 at 1:20 a.m. UTC by stretch
- What are the most popular certifications in networking?
- How much is certification X worth?
- How should I study for certification exams?
- What's a "brain dump?"
- What is the exam experience like?
- I only just barely passed! Does it still count?
- My employer will pay for me to get a certification. Should I do it?
Friday, January 2, 2015 at 1:29 p.m. UTC by stretch
This post is the first in a series I plan to publish over the next few months regarding frequently asked questions in networking. Each post will cover a different subject, roughly following the outline I shared last summer. I hope people find this useful!
- What kind of networking jobs are there?
- What are the different networking specialties?
- How do I get into networking?
- Do I need a college degree to be a networker?
- Do I need certifications?
- Do I need to know a programming language?
- What should I list on my resume?
- How much do networkers make?
- How do I find a job?
- Do you have any interview tips?
- What are the negative aspects of networking?
Tuesday, November 18, 2014 at 2:05 a.m. UTC by stretch
Not so long ago, if you wanted to build a data center network, it was perfectly feasible to place your layer three edge on the top-of-rack switches and address each rack as its own subnet. You could leverage ECMP for simple load-sharing across uplinks to the aggregation layer. This made for an extremely efficient, easily managed data center network.
Then, server virtualization took off. Which was great, except now we had this requirement that a virtual machine might need to move from one rack to another. With our L3 edge resting at the top of the rack, this meant we'd need to re-address each VM as it was moved (which is apparently a big problem on the application side). So, now we have two options: We can either retract the L3 edge up a layer and have a giant L2 network spanning dozens of racks, or we could build a layer two overlay on top of our existing layer three infrastructure.
Most people opt for some form of the L2 overlay approach, because no one wants to maintain a flat L2 network with dozens or hundreds of thousands of end hosts, right? But why is that?
Friday, October 31, 2014 at 2:20 a.m. UTC by stretch
"After all, what's the best part of Halloween?" Jimmy pleaded over the phone. He was trying yet again to convince Tom to skip work for the night and head over to the party he was throwing. Tom and Jimmy were good friends, but he already knew how the conversation was going to end.
"I dunno, the candy?" Tom played dumb.
"No, the eye candy! I'm telling you bro, you don't want to miss it. Rachel will be there." Jimmy sang the last bit tauntingly.
"I told you," Tom countered. "I've got work." It was around 6pm now, and he was just pulling into the parking lot outside the data center where he planned to spend the night recabling several racks of equipment. The scariest part of his Halloween would be picking through years' worth of undressed patch cabling.
"I don't get why you have to do that shit at night anyway. Why can't you do it during the day when you're stuck at work anyway?" Jimmy prodded.
Tom parked across from the building's entrance and turned off his car. Other than a couple vehicle belonging to the operations staff, the parking lot was deserted. He grabbed his tool bag from the passenger seat and headed toward the building's entrance.
Wednesday, October 1, 2014 at 2:05 a.m. UTC by stretch
Typically, when you buy a network router or switch, it comes bundled with some version of the manufacturer's operating system. Cisco routers come with IOS (or some derivative), Juniper routers come with Junos, and so on. But with the recent proliferation of merchant silicon, there seem to be fewer and fewer differences between competing devices under the hood. For instance, the Juniper QFX3500, the Cisco Nexus 3064, and the Arista 7050S are all powered by an off-the-shelf Broadcom chipset rather than custom ASICs developed in-house. Among such similar hardware platforms, the remaining differentiator is the software.
One company looking to benefit from this trend is Cumulus Networks. Cumulus does not produce or sell hardware, only a network operating system: Cumulus Linux. The Debian-based OS is built to run on whitebox hardware you can purchase from a number of partner Original Device Manufacturers (ODMs). (Their hardware compatability list includes a number of 10GE and 40GE switch models from different vendors.)
Cumulus Linux is, as the name implies, Linux. There is no "front end" CLI as on, for example, Arista platforms. Upon login you are presented with a Bash terminal and all the standard Linux utilities (plus a number of not-so-standard bits). Like any OS, Cumulus handles interactions with the underlying hardware and among processes.
Wednesday, August 13, 2014 at 11:46 p.m. UTC by stretch
As I announced earlier this summer, I'm working on writing a book targeted to people entering the field of computer networking. I've got a fair amount of content fleshed out already, but figured it might help to get some feedback on the tentative structure. The book is being written in a question-and-answer style, organized into chapters by subject.
Below is the preliminary table of contents. It's still very much a work in progress, but I'm curious what people think of this approach. Constructive criticism and suggestions for additional content are welcome!
Monday, July 14, 2014 at 1:03 p.m. UTC by stretch
I received an email last week from a reader seeking advice on a fairly common predicament:
Our CIO has recently told us that he wants to get rid of MPLS because it is too costly and is leaning towards big internet lines running IPSEC VPNs to connect the whole of Africa.
As you can imagine, this has caused a huge debate between the networks team and management, we run high priority services such as Lync enterprise, SAP, video conferencing etc. and networks feel we need MPLS for guaranteed quality for these services but management feels the Internet is today stable enough to run just as good as MPLS.
What is your take on the MPLS vs Internet debate from a network engineer's point of view? And more so, would running those services over Internet work?
This is something I struggled with pretty frequently in a prior job working for a managed services provider. MPLS WANs are great because they provide flexible, private connectivity with guaranteed throughput. Most MPLS providers also allow you to choose from a menu of QoS schemes and classify your traffic so that real-time voice and video services are treated higher preference during periods of congestion.
Unfortunately, MPLS WANs tend to be considerably more expensive than Internet circuits. A dedicated 3 Mbps MPLS circuit might cost three or four times as much as a 50 Mbps business class broadband Internet circuit: These numbers are hard to justify to management who may not appreciate the contexts of reliability and QoS controls. Since private connectivity can be achieved using a VPN overlay on top of plain Internet circuits, can we still justify the cost MPLS WANs? Should we?
My advice would be to stick with the MPLS WAN if you can afford it. A VPN overlaid on top of Internet circuits might work most of the time, but when it doesn't perform adequately, you'll have little immediate recourse. Should you decide on moving to a VPN overlay, do so in phases: Keep the MPLS WAN around for a few months in case the overlay strategy doesn't work out. But if you find that your Internet circuits provide sufficient throughput so that congestion of real-time services never becomes a problem, maybe that's an acceptable solution.
PacketLife.net is the work of a network engineer named Jeremy Stretch. It began as a repository for Cisco certification study notes in 2008, but quickly grew into a popular community web site.
The site's goal is to offer free, quality technical education to networkers all over the world, regardless of skill level or background.