Writing a Custom IPAM Application

Monday, August 10, 2015 at 1:14 a.m. UTC by stretch

Four years ago, I lamented the lackluster selection of IPAM applications available for service providers. Unfortunately, it seems not much has changed lately. I was back to exploring IPAM offerings again recently, this time with the needs of a cloud hosting provider in mind. I demoed a few tools, but none of them seemed to fit the bill (or they did, but were laughably overpriced).

So, I decided to write my own. In my rantings a few years back, I had considered this option:

Could I create a custom IPAM solution with everything we need? Sure! The problem is that I'm a network engineer, not a programmer (a natural division of labor which, it seems, is mostly to blame for the lack of robust IPAM solutions available). Even if I had the time to undertake such a project, I have little interest in providing long-term maintenance of it.

My opinion has not changed, but I've come to realize that if I want a tool that fits my requirements, I will need to build it. And after surprisingly little time, I'm happy to report that I have now have a kick-ass IPAM tool that does exactly what I want it to.

While I can't share the code, which is owned by and purpose-built for my employer, I would like to share some tips I picked up in the process with the hope of convincing others that rolling your own tools really isn't that hard. And in many cases, it may be well worth the effort.

Continue reading 12 comments

What the CCIE does not Prove

Friday, May 29, 2015 at 2:27 p.m. UTC by stretch

I came across an article today about a 19-year-old who earned his CCIE. It reminded me of a Reddit post from a few weeks ago. Someone asked why, when evaluating a CCIE, hiring managers still demand a number of years of practical experience in the field.

I'm in a situation where I'm a CCNP with 3 years of experience. I want to get my CCIE but I keep being told left and right I don't have enough experience and I'll never get a CCIE job without 7 years of experience. Am I supposed to just laze around and wait until I get more experience? It just doesn't make sense.

This is a fairly common misunderstanding among people new to our field, and is largely the result of vendor marketing. People want so badly to believe that a certification proves their worth as an individual, when in reality its value is much more narrowly defined.

Continue reading 39 comments

Traceroute and Not-so-Equal ECMP

Monday, April 27, 2015 at 2:05 p.m. UTC by stretch

I came across an odd little issue recently involving equal-cost multipath (ECMP) routing and traceroute. Traceroutes from within our network to destinations out on the Internet were following two different paths, with one path being one hop longer than the other. This resulted in mangled traceroute output, impeding our ability to troubleshoot.

The relevant network topology comprises a mesh of two edge routers and two core switches. Each edge router has a number of transit circuits to different providers, and advertises a default route via OSPF to the two core switches below. The core switches each load-balance traffic across both default routes to either edge routers.


Because each edge router has different providers, some destinations are routed out via edge1 and others via edge2, which means sometimes a packet will be routed to edge2 via edge1, or vice versa.


Routers typically employ a hash function using layer three and four information from each packet to pseudo-randomly distribute traffic across equal links. Typically, all packets belonging to a flow (e.g. all packets with the same source and destination IP and port numbers) follow the same path.

However, in this case traceroute packets were being split across two path of unequal length, which made traceroute output pretty much unreadable. We noticed that only UDP traceroutes were affected; ICMP traceroutes reported one path as normal.

Continue reading 10 comments

MAC Address Aggregation and Translation as an Alternative to L2 Overlays

Tuesday, November 18, 2014 at 2:05 a.m. UTC by stretch

Not so long ago, if you wanted to build a data center network, it was perfectly feasible to place your layer three edge on the top-of-rack switches and address each rack as its own subnet. You could leverage ECMP for simple load-sharing across uplinks to the aggregation layer. This made for an extremely efficient, easily managed data center network.

Then, server virtualization took off. Which was great, except now we had this requirement that a virtual machine might need to move from one rack to another. With our L3 edge resting at the top of the rack, this meant we'd need to re-address each VM as it was moved (which is apparently a big problem on the application side). So, now we have two options: We can either retract the L3 edge up a layer and have a giant L2 network spanning dozens of racks, or we could build a layer two overlay on top of our existing layer three infrastructure.

Most people opt for some form of the L2 overlay approach, because no one wants to maintain a flat L2 network with dozens or hundreds of thousands of end hosts, right? But why is that?

Continue reading 21 comments


Friday, October 31, 2014 at 2:20 a.m. UTC by stretch

"After all, what's the best part of Halloween?" Jimmy pleaded over the phone. He was trying yet again to convince Tom to skip work for the night and head over to the party he was throwing. Tom and Jimmy were good friends, but he already knew how the conversation was going to end.

"I dunno, the candy?" Tom played dumb.

"No, the eye candy! I'm telling you bro, you don't want to miss it. Rachel will be there." Jimmy sang the last bit tauntingly.

"I told you," Tom countered. "I've got work." It was around 6pm now, and he was just pulling into the parking lot outside the data center where he planned to spend the night recabling several racks of equipment. The scariest part of his Halloween would be picking through years' worth of undressed patch cabling.

"I don't get why you have to do that shit at night anyway. Why can't you do it during the day when you're stuck at work anyway?" Jimmy prodded.

Tom parked across from the building's entrance and turned off his car. Other than a couple vehicle belonging to the operations staff, the parking lot was deserted. He grabbed his tool bag from the passenger seat and headed toward the building's entrance.

Continue reading 12 comments

Cumulus Linux: First Impressions

Wednesday, October 1, 2014 at 2:05 a.m. UTC by stretch

Typically, when you buy a network router or switch, it comes bundled with some version of the manufacturer's operating system. Cisco routers come with IOS (or some derivative), Juniper routers come with Junos, and so on. But with the recent proliferation of merchant silicon, there seem to be fewer and fewer differences between competing devices under the hood. For instance, the Juniper QFX3500, the Cisco Nexus 3064, and the Arista 7050S are all powered by an off-the-shelf Broadcom chipset rather than custom ASICs developed in-house. Among such similar hardware platforms, the remaining differentiator is the software.

One company looking to benefit from this trend is Cumulus Networks. Cumulus does not produce or sell hardware, only a network operating system: Cumulus Linux. The Debian-based OS is built to run on whitebox hardware you can purchase from a number of partner Original Device Manufacturers (ODMs). (Their hardware compatability list includes a number of 10GE and 40GE switch models from different vendors.)

Cumulus Linux is, as the name implies, Linux. There is no "front end" CLI as on, for example, Arista platforms. Upon login you are presented with a Bash terminal and all the standard Linux utilities (plus a number of not-so-standard bits). Like any OS, Cumulus handles interactions with the underlying hardware and among processes.


Continue reading 10 comments

About Packet Life

PacketLife.net is the work of a network engineer named Jeremy Stretch. It began as a repository for Cisco certification study notes in 2008, but quickly grew into a popular community web site.

The site's goal is to offer free, quality technical education to networkers all over the world, regardless of skill level or background.

Blog Splotlight

Network Janitor

Kurt Bales