MAC Address Aggregation and Translation as an Alternative to L2 Overlays

Not so long ago, if you wanted to build a data center network, it was perfectly feasible to place your layer three edge on the top-of-rack switches and address each rack as its own subnet. You could leverage ECMP for simple load-sharing across uplinks to the aggregation layer. This made for an extremely efficient, easily managed data center network.

Then, server virtualization took off. Which was great, except now we had this requirement that a virtual machine might need to move from one rack to another. With our L3 edge resting at the top of the rack, this meant we'd need to re-address each VM as it was moved (which is apparently a big problem on the application side). So, now we have two options: We can either retract the L3 edge up a layer and have a giant L2 network spanning dozens of racks, or we could build a layer two overlay on top of our existing layer three infrastructure.

Most people opt for some form of the L2 overlay approach, because no one wants to maintain a flat L2 network with dozens or hundreds of thousands of end hosts, right? But why is that?

Challenges of a Layer Two Network

Layer two networks in the data center pose two major challenges. First, L2 networks must be free of forwarding loops. Traditionally this has been solved using some flavor of Spanning Tree to block all redundant paths, but this approach severely limits the amount of throughput available to each rack, as only one uplink (or LAG) can be active at a time. Other solutions to the loop problem include various multi-chassis LAG and virtual chassis implementations from different vendors. These each have their pros and cons and are better discussed elsewhere.

What I've been pondering lately is the second problem: MAC table size. Assuming you can solve the L2 loop problem reasonably well, you still have to worry about every switch learning every MAC address across the entire L2 domain. Most switches appropriate for a dense data center environment can handle 128 thousand or more MAC addresses, but a topology change (such as transitioning from an active uplink to a backup) might trigger a flush of nearly the entire table. The switch needs to re-learn thousands and thousands of MAC addresses as fast as it can, which might not be fast enough to avoid impacting network availability.

Why does this problem affect only L2 topologies, and not layer three designs? The key difference is that L3 endpoint addresses can be aggregated. A router doesn't need to know explicitly that the hosts at 192.0.2.36, 192.0.2.55, 192.0.2.187, and 192.0.2.208 are each reachable via a common interface. It only needs to know one route, for the 192.0.2.0/24 network. This function is inherent to IP addressing. MAC addresses, in contrast, can not be aggregated. Or can they?

MAC Aggregation

Historically, the problem with MAC addresses has been that they are both globally unique (mostly) and pseudo-random. It's extremely difficult, if not impossible, to predict the MAC address of each physical interface to be installed in each device in a data center, since each vendor has its own OUI(s) and the lower 24-bits of a MAC are effectively random.

It's very subtle, but the industry's shift to virtualization is actually solving this problem for us. When you create a virtual machine today, each of its network interfaces gets assigned one or more completely arbitrary MAC addresses by the hypervisor or by some central orchestration tool. For instance, VMware VMs typically use the OUI 00:50:56, with the lower 24-bits generated by some dynamic function. Unlike in the physical world, there's nothing preventing us from writing rules dictating what MAC addresses gets assigned to a VM.

Let's say we needed to deploy 100 racks with 10,000 VMs each in a single L2 domain. This would push us to the current limit of MAC table sizes on even the beefiest of switches (and we would need one or two of those beefy switches in every rack). But what if we decided to assign MAC addresses to each VM based on the physical location of its hypervisor? Assuming we use a privately-allocated 24-bit OUI (designated XX-XX-XX below), that leaves 24 bits of address space to carve up, which is analogous to a /8 prefix in IPv4. In our admittedly contrived scenario, we can simply designate the upper 8 bits as the rack identifier, and the lower 16 bits as the rack-specific VM identifier.

If our virtualization infrastructure is programmed to create VMs following this scheme, it allows us to aggregate at the L2 core. Instead of learning 1,000,000 individual MAC addresses, the core would only need to know 100 "MAC routes" (one /32 MAC route per rack). Each top-of-rack switch would still need to know the addresses of every VM interface in the rack, but at around 10 or 20 thousand MACs that's not problem. Each ToR switch would also need to maintain a "default" MAC route of 00-00-00-00-00-00/0 pointing northbound toward the core layer.

Of course, each MAC route would need to be installed within each VLAN in the domain, so the total number of aggregate routes would equate to the number of racks multiplied by the number of VLANs in use. That's still a tremendous savings over using individual host addresses.

If the necessary hardware support existed today, configuration syntax for a Juniper switch might look something like the snippet below. (These are not real commands.) Alternatively, it should be feasible to learn MAC aggregate routes dynamically via an extension to BGP or other routing protocol.

ethernet-switching-options {
    static {
        vlan 123 {
            route xx:xx:xx:01:00:00/32 next-hop xe-0/0/0.0;
            route xx:xx:xx:02:00:00/32 next-hop xe-0/0/1.0;
            route xx:xx:xx:03:00:00/32 next-hop xe-0/0/2.0;
            route xx:xx:xx:04:00:00/32 next-hop xe-0/0/3.0;
        }
    }
}

This concept scales well to the insertion of an aggregation layer between the ToR and core layers, too. Each aggregation switch would need to know only the summary MAC routes for the racks it feeds and a default MAC route toward the core.

MAC Address Translation

I know what you're thinking. "This is all just cupcakes and unicorns, but what happens when I need to move a VM to a different rack?" While perhaps less disruptive than changing the IP address of a VM, changing its MAC still requires some degree of modification to the VM's operating system (for example, Linux udev rules), which is best avoided. So instead of changing its MAC, we can simply translate it on the hypervisor.

Hang on, NAT is bad, right? Everyone knows that! The biggest sticking point with IP NAT is that it breaks end-to-end transparency: The IP address from which your traffic originates is not how the far end sees you (or vice versa). But at layer two, this isn't a concern. Do you know what the MAC address of the server at packetlife.net is? No, nor would you care. L2 addressing is significant only to the local link. Rewriting the source and/or destination MAC address on a frame as it passes between two L3 nodes has zero effect on its payload. (Remember, there's no encapsulation happening here.)

The ability to rewrite MAC addresses on the hypervisor simplifies our solution even further: VMs can be created with entirely random MAC addresses as they are today, and each hypervisor would simply MAT -- that's MAC Address Translation -- each "real" MAC to a MAC from its assigned pool (which is derived from its rack's aggregate).

Why Can't We do this Today?

Two features would be necessary to implement this crazy scheme today. First, we'd need the ability to perform one-to-one MAC address translation from an administratively defined pool on a hypervisor. This should be trivial to accomplish: We already have L3 NAPT in the hypervisor today. L2 translation is actually easier as it's completely stateless.

The second requirement is the primary barrier to adoption. As far as I'm aware, mainstream network hardware does not support longest prefix matching (LPM) for layer two addresses. LPM obviously has been around forever: It's how an IP router determines the best matching route for a destination address. But because MAC addresses have historically been considered unaggregatable, chipset manufacturers have understandably not built in this capability. That said, it should be perfectly feasible to include in future chipset iterations.

There are probably a number of caveats to this scheme that I haven't considered. I'm also not the first person to think of it, of course. But I do think the approach holds some merit, and it would at least be a pleasant departure from the industry's current approach of simply tunneling everything.

Please feel free to berate my theories and question my professional competence in the comments below as per established social convention.