Lessons Learned Writing a Custom Config Builder

By stretch | Friday, July 26, 2013 at 2:34 p.m. UTC

A while back, I set about developing a modest configuration templating system for my employer. When I first joined the company, new network devices were being provisioned using configuration templates stored as Microsoft Word files, which, as you can imagine, was pretty painful. Each variable had to be identified and replaced by hand in a tedious and error-prone process. I wanted something better, but also cheap (or free) and simple. So I started building something.

To kick off my crazy project, I first decided to build a web application based on the Django Python framework (the same platform on which PacketLife.net runs). Django and similar frameworks handle most of the mundane tasks involved in writing a web application and allow for rapid prototyping. It also includes a built-in administration interface for creating and manipulating data independent of the front-end user interface. I spun up a modest internal VM running Ubuntu Server, installed Django, and initiated a working project in under an hour.

Once I had a development environment in place, I started to think really hard about exactly what I wanted to build. (Admittedly, this step probably should have come first.) I liked the idea of raw text-based config templates because they're nice and portable: You can copy a sample config out of a vendor's documentation or running device, tweak it, and extract the variables to produce a working template. This is a much more straightforward approach than, say, abstracting a command hierarchy into XML.

Here's a simple Cisco IOS BGP configuration snippet we'll use as an example:

router bgp {LOCAL-ASN}
 address-family ipv4
  no synchronization
  neighbor {PEER-IP} remote-as {PEER-ASN}
  neighbor {PEER-IP} activate
  neighbor {PEER-IP} soft-reconfiguration inbound
  network {LOCAL-NETWORK} mask {LOCAL-NETWORK-MASK}

The tedious part is having to replace these variables by hand when deploying a new configuration. Our primitive approach was to denote variables as capitalized text surrounded by brackets or braces, but this can be easy to miss. Even using bright yellow highlighting in a full-blown editor like Word doesn't guarantee that each variable will be caught and replaced.

The other challenge is that, even if you could ensure every variable was appropriately filled in every time, there's no contextual validation of the provided value. Someone might typo an IPv4 address as 192.0..1, and that's what gets dumped into the device when the configuration is applied. This obviously would trigger an error, which may or may not be noticed prior to deployment.

Contextual Variables

To address the first problem, detecting variables, we can use a simple regular expression. For example, the following regex could be used to match the variable naming scheme used in the sample config template above (all capital letters and hyphens surrounded by curly braces):

{([A-Z\-]+)}

We can easily write a few lines of code to parse a config template and pull out all the variables using this regex. However, it doesn't address our second problem: assigning context to each variable. For example, what if we have a variable named CUSTOMER? Does that mean we should specify a full customer name, a shorthand customer identifier, a human point of contact, or something else? This takes a bit more thought.

First we have to decide what types of contextual information we want to store about our variables. Most programming languages, for example, have different types of variables: integers, strings, lists, booleans, and so on. We need to classify variables in a similar manner: IPv4 addresses, IPv6 addresses, VLAN IDs, interface identifiers, etc. Each variable has a set of rules governing its form. For example, a VLAN ID needs to be a positive integer between 1 and 4094, whereas an IPv4 address must be expressed in dotted-decimal notation with a mask length.

I began listing the types of variables we might need, organized by the constraints needed to be placed on them:

Type Constraints Examples
Numeric Minimum and maximum values BGP ASN, VLAN ID, TCP/UDP port number
String Minimum and maximum lengths, regex-matching Interface name, city, password, SSID
IP address Version, network or host, minimum and maximum mask lengths Host address, network, interface address
Static choice One of a given set of values Direction (in or out), L4 protocol (TCP or UDP)

(The table above is clearly not inclusive, and is intended only to convey the concept of classifying variables by type.)

Once I was satisfied that my list was mostly complete, I set about creating a Django model for each type of variable: NumberVar, StringVar, etc. Through the app's administration interface, I could then create arbitrary instances of each. For example, I created a NumberVar named ASN with a minimum value of 1 and a maximum value of 65535. I created a second NumberVar named PRIVATE-ASN with a minimum value of 64512 and a maximum value of 65535. I would use one or the other to represent an ASN in my templates depending on which type I wanted, providing a modest degree of context.

NumberVars.png

I began creating more and more variable types with finer and finer tuning. I could create a specific IPv4 network type with a set mask length of /30 for use on point-to-point links. I created a specific string type with a minimum length of 12 characters to represent passwords. Now I was getting somewhere.

Next, I needed to adjust my regex to accommodate the inclusion of a variable type. I decided on the form {TYPE:NAME}. Further, since this was all going to be parsed by the application anyway, I decided to use a more human-friendly form of variable naming, with mixed cases and underscores representing spaces. This would allow me to dynamically generate clean field names within the user interface (more on this in a bit). The regex grew to:

{([A-Z0-9\-]+):([A-Za-z0-9_\-]+)}

With the new scheme in place, the template snippet earlier in this article would be rewritten as:

router bgp {ASN:Local_ASN}
 address-family ipv4
  no synchronization
  neighbor {IP4-HOST:Peer_IP} remote-as {ASN:Peer_ASN}
  neighbor {IP4-HOST:Peer_IP} activate
  neighbor {IP4-HOST:Peer_IP} soft-reconfiguration inbound
  network {IP4-NETWORK:Local_Network} mask {IP4-MASK:Local_Network_Mask}

Display Filters

There's one bit that's still missing, though. Why should we have to declare the local network and its mask as two separate variables? They are, after all, two expressions of a common value. For example, for a given network -- let's say 192.168.0.0/24 -- there are several pieces of information we might want to display:

  • The network ID (192.168.0.0)
  • The mask length in CIDR notation (24)
  • The mask length in dotted-decimal notation (255.255.255.0)
  • The mask length expressed as a wildcard mask (0.0.0.255)

All of this information is already contained in the variable; we just need a way to tweak how it gets displayed in the generated template. We can add an optional third component to the variable declaration scheme, a display filter to output differing formats of a variable. We'll use a pipe character appended to the variable name to add a display filter on the local network variable:

network {CUSTOMER-IP4-NETWORK:Local_Network|ip} mask {CUSTOMER-IP4-NETWORK:Local_Network|ddmask}

Given a value of 192.168.0.0/24, the line above would generate the following:

network 192.168.0.0 mask 255.255.255.0

Our variable-matching regex is finally extended to the following to accommodate optional display filters:

{([A-Z0-9\-]+):([A-Za-z0-9_\-]+)(?:\|([a-z0-9]+))?}

At this point, we have a reliable way to assign context to named variables and display their values in arbitrary formats. The next step is dynamically generating a form to be completed by the user based on those variables.

Dynamic Forms

Django provides a pretty robust tool set for generating and processing web forms. Armed with our variable declaration regex, we can parse a configuration template and create a new form field for each variable we find and present the resulting set of fields to the user for completion. Without getting too deep into the inner-workings of Django, as I want to keep this article platform-agnostic, it's possible to dynamically generate a web form with an individual field for each variable in the template.

This bit of template config:

router bgp {ASN:Local_ASN}
 address-family ipv4
  no synchronization
  neighbor {IP4-HOST:Peer_IP} remote-as {ASN:Peer_ASN}
  neighbor {IP4-HOST:Peer_IP} activate
  neighbor {IP4-HOST:Peer_IP} soft-reconfiguration inbound
  network {IP4-NETWORK:Local_Network} mask {IP4-MASK:Local_Network_Mask}

Becomes this form:

BGP_form.png

And when the form is correctly completed, will produce output like this:

router bgp 65000
 address-family ipv4
  no synchronization
  neighbor 192.0.2.1 remote-as 65100
  neighbor 192.0.2.1 activate
  neighbor 192.0.2.1 soft-reconfiguration inbound
  network 192.168.0.0 mask 255.255.255.0

I want to stress that the definition for this form has not been defined somewhere else in the application: It has been rendered solely from the configuration template. If we add or remove a variable in the template, that change is reflected when the form is regenerated. Also note that although the peer IP variable has been declared three times in the template, we only need to specify it once. Similarly, the local network variable is prompted for only once and displayed twice using two different filters.

Reusable Configlets

What if we need to add more than one network statement to our BGP configuration? We could generate one line and then replicate it by hand, but then we're right back to where we started. A more elegant solution is to break our template into two components: The base BGP protocol configuration and the network statement. Whereas the first component is needed only once, we can replicate the second part as many times as we need. In fact, that's exactly what the green "Add another" button does in the screenshot. We can also give the user the option to include additional configlets, such as one for a BGP aggregate address.

BGP_form2.png

The above form will produce the following output:

router bgp 65000
 address-family ipv4
  no synchronization
  neighbor 192.0.2.1 remote-as 65100
  neighbor 192.0.2.1 activate
  neighbor 192.0.2.1 soft-reconfiguration inbound
  network 192.168.0.0 mask 255.255.255.0
  network 192.168.1.0 mask 255.255.255.0
  network 192.168.2.0 mask 255.255.255.0
  network 192.168.3.0 mask 255.255.255.0
  aggregate-address 192.168.0.0 255.255.252.0 summary-only

As you can see, with a little thought and effort, it's possible to create a very robust and lightweight configuration templating system. Obviously, I've glossed over the mechanics behind the concepts and I am unfortunately unable to release the code as this was built on company time, but I want to encourage readers to give it a shot themselves and see what they come up with. I have no doubt that someone with stronger programming skills can come up with something much more impressive than what I did.

About the Author

Jeremy Stretch is a network engineer living in the Raleigh-Durham, North Carolina area. He is known for his blog and cheat sheets here at Packet Life. You can reach him by email or follow him on Twitter.

Comments


Reed (guest)
July 26, 2013 at 6:18 p.m. UTC

Brilliant! Oh if only you could post the code, then we'd have a template for a template.


tonhe
July 26, 2013 at 6:29 p.m. UTC

I've actually been recently working on something along the same lines. Mine is much more of a hack currently... all written in bash and expect. The configuration templates are not stored in separate files, and it is obviously all console based.

My script was built to aide in keeping SVI/VLAN deployments consistent between devices and across VLANs. I've had some issues here with jr engineers using copy/paste with bad results.

My script also gives the user the option for deploying the code to the remote device, and does some minor error checking before doing so.

It's currently up on github, but I'm not making a big announcement about it until I get a full rewrite in. I want to make it much more modular.


abester1
July 26, 2013 at 8:17 p.m. UTC

Great work... Having a tool to help build device templates and drive consistency for all your device deployment can be very handy especially as you scale your network size.

While I have a poor mans version of this idea available to me (which works more or less for what I need it for). I have always wondered why no one has built a specialized tool with database support to store all the variables for each device and be able to generate configuration on the fly.

Storing the variables in a db would make it a lot more sense for long term support especially when you're deploying 100's of the same model device. I think it would be great to be able to generate the configuration on the fly based whatever variables are already stored in the db. For example: multi-site deployment using standard kit/device specs, the different configuration parameters are generally 20% local customization, while 80% is generally the same for all devices (e.g. snmp community, ssh access, vty configuration, etc).

any thoughts ?


Ken (guest)
July 27, 2013 at 2:45 a.m. UTC

I have been doing this since my days in the Air Force back at ramstein. Started with html forms to php variables. Now I use perl with cpan module html::template, and I have gone through multiple input methods; html forms, excel reader, and MySQL Ajax editor. Now I also have an audit tool to verify data as well.

Office installation config building went from a full week to less than an hour. One side effect is the engineers rely on the tool completely and tend not to have as much knowledge of the configs they are putting in.


edwin (guest)
July 29, 2013 at 4:21 a.m. UTC

Cool stuff.
But why not use Chef from Opscode for the config deployement and automation?


David (guest)
July 29, 2013 at 9:41 a.m. UTC

Hello,
I have been doing something very similar with django using jinja2 as the template system. It is really powerful and allows you even to add loops and if statements inside the template.

Regards.


stretch
July 29, 2013 at 3:15 p.m. UTC

@edwin: Chef seems like extreme overkill for our scale, but may work well for others. I'd love to hear any experiences using Chef for network config management.


abester1
July 29, 2013 at 10:33 p.m. UTC

@Ken. Yea, I've noticed that as well the junior engineers are relying on the configuration files being generated and they don't take the time to ask "why is this configured this way?" Type of questions. But the upside working with a standardizes customer deployment in many sites to be effortless, this really makes the work almost idiot proof, almost :D


tom (guest)
July 31, 2013 at 3:59 a.m. UTC

I have been looking for a good way to dynamic web forms and it looks like django could be a good way... I would love to see any code you could share on the django side


dave (guest)
August 2, 2013 at 12:23 p.m. UTC

Very nice. I've been wanting to build a system similar to this for a number of years and for a number of reasons: Building out prototype labs, provisioning equipment, monitoring configs on devices. (kind of similar to what RANCID does, but if you define your network to have a specific configuration and it deviates from that overnight... red flag goes up.)


jimmyjmar (guest)
August 5, 2013 at 12:09 a.m. UTC

Very informative, I've been searching for a appropriate template solution. I second the motion of sharing your "template" code!


guest (guest)
August 6, 2013 at 1:43 p.m. UTC

If you had all the templates in Word-format, you could connect the word-document to an excel-file containing all info needed to configure the device, based on column, acting as a database, with the mail-merge fnkction in Word... Then all variables would change as you change the line word reads from the excel-file...


guest (guest)
August 7, 2013 at 7:48 a.m. UTC

Have you checked out Netomata Config Generator. It is open source and can view as a puppet for cisco. It is basically a template system and you choose how to deploy the configuration. Brent Chapman suggests using RANCID.


JSS (guest)
August 7, 2013 at 12:08 p.m. UTC

Great article and it gives me new ideas. I am doing a migration from a VPN3000 to ASA and built a config generator in Excel using VBA to create the ACL's, Group Policies, Tunnel Groups, etc. Like Stretch, the guy before me was doing a search/replace in Word. Now I open the template, enter 10 variables, and voila - we have a config.


Matt Gee (guest)
August 13, 2013 at 9:17 a.m. UTC

I created something similar recently, however our configs have just 10 variable that need replacing. I used the php str_replace function to replace variable in an existing template. Works perfectly: http://php.net/manual/en/function.str-replace.php


Binary (guest)
August 15, 2013 at 2:34 a.m. UTC

hmm.. I'm having difficulties on my templates, it still error every time I put it. I don't know why.. I hope this will help. Thanks much for sharing :)
Network support.


layer4down
August 20, 2013 at 5:10 p.m. UTC

Thumbs up bro!


fisher (guest)
August 22, 2013 at 7:52 p.m. UTC

Interesting. I might have to check out Django now. I've built this same templating tool the old-school way with Perl and CGI. Might be time to learn something new. Thanks!


regisu (guest)
August 25, 2013 at 6:10 p.m. UTC

I created something similar some time ago also in Django framework. I'm not a programmer so it's poorly writen app but it's working and it saving me alot of time. It's generating configuration and filters <prefix-lists> for BIRD route-servers in a small Polish Internet Exchange point that I manage. I've added few additional features and now I need only add new member in Django admin panel and then use few scripts to have everything configured for me ... like BIRD route-servers, revDNS and smokeping monitoring.
So I think that is good to take some time and learn some minimal programming skills.


jemery27
September 6, 2013 at 2:22 a.m. UTC

I have done a lot of dynamic config generation but almost always using spreadsheets (usually Excel and VBScript). The nice thing about a spreadsheet is it generally scales out - like if you need 10 configs, and it also allows you to save it in the spreadsheet for reference indefinitely. Also to make it much easier when using spreadsheet I have built a lot of user defined functions that allows ip address manipulation.

Lately I have been using google apps and google sheets so I converted my Excel VB to google scripts. Here is a google spreadsheet as an example that does something similar to this bgp config:

https://docs.google.com/spreadsheet/ccc?key=0AtvdVN89Xo5KdGs4OVRIeUZ6YmZnOU9RYlpxZWYtY2c&usp=sharing

It uses a couple IP functions in the formulas. For more of the functions and capabilities, here is a sample sheet:

https://docs.google.com/spreadsheet/ccc?key=0AtvdVN89Xo5KdEhSaWctWTV1d1RpQUt1TTF1blFtVnc&usp=sharing

And helpfile:
https://docs.google.com/document/d/18uB0Cbs37WOe1C-em5Rae6Hu8y5rUOvrMo2MoOyVlG4/edit?usp=sharing


Guest (guest)
September 7, 2013 at 3:16 p.m. UTC

Have you looked at Hatch?


network management (guest)
September 10, 2013 at 8:52 p.m. UTC

Great work! Will definitely give this a try and see where it gets me. Definitely a lot of people will benefit from your shared insight in writing custom configuration builder codes.


ChadH
December 4, 2013 at 9:16 a.m. UTC

Great post! This would be a great community project. Although, I prefer PHP web apps :)


vinman1000 (guest)
December 27, 2013 at 9:36 p.m. UTC

I did something similar using OLE (Object Linking and Embedding) with Word and Excel.

I was migrating 100 or so sites to frame relay and had to build configs for every one. I had all my hostnames, IP addresses, DLCIs, etc in a spreadsheet. When you wanted to build a config just take the row for the site and copy/paste it into the first row.

Then you would open a Word document that linked to that spreadsheet and got all the variables from row 1 and boom, you had your custom config.

It took a while to setup but saved time as I deployed the sites. It was easy to show my other engineers how to use it and they benefited too.

Thanks,
Vin


Nick (guest)
June 13, 2014 at 7:58 p.m. UTC

Very nice forms!

What are you using for frontend? Is it pure jQuery?

Leave a Comment


Optional; will not be displayed publicly or given out.
No commercial links. Only personal (e.g. blog, Twitter, or LinkedIn) and/or on-topic links, please.
What is the maximum decimal value that can be expressed by eight bits?