WAN Circuit Topologies
By stretch | Wednesday, February 6, 2013 at 4:26 a.m. UTC
There are a variety of approaches to connecting multiple sites across a wide area network (WAN), ranging from leased lines to MPLS to IPsec VPN tunnels. Unfortunately, many of the options and potential WAN topologies are often misunderstood or confused with one another.
A point-to-point circuit, as its name implies, connects exactly two points. This is the simplest type of WAN circuit. Packets sent from one site are delivered to the other and vice versa. There is typically some amount of processing and encapsulation performed across the carrier's infrastructure, but it is all transparent to the customer.
A point-to-point circuit is typically delivered as a layer two transport service, which allows the customer to run whatever layer three protocols they want with an arbitrary addressing scheme. A customer can change or add an IP subnet in use on a layer two circuit without coordinating with their provider.
The primary detractor of point-to-point circuits is that they don't scale efficiently. Suppose you wanted to deploy a hub-and-spoke style WAN topology with twenty branch sites connecting to a single main office. You could deploy twenty point-to-point circuits, one to each spoke site from the hub, but that would result in a clutter of twenty separate physical connections at the hub site. Installing twenty circuits would be rather expensive, and you might not even be able to fit them all on the same device. This is where a point-to-multipoint circuit would be ideal.
With a point-to-multipoint circuit, we only need a single circuit to the hub site to be shared by all spoke circuits. Each spoke is assigned a unique tag which identifies its traffic across the hub circuit. The type of tagging is medium-dependent; we'll use an Ethernet circuit with IEEE 802.1Q VLAN trunking as an example. (The spoke circuits may have their traffic tagged or untagged, depending on the specific carrier's service parameters.) Each spoke gets a virtual circuit and routed subinterface on the hub circuit. The resulting layer three topology is the same as using several point-to-point circuits but is more practical in implementation.
interface GigabitEthernet0/0.102 description Site B encapsulation dot1q 102 ip addr 10.0.0.1 255.255.255.252 ! interface GigabitEthernet0/0.103 description Site C encapsulation dot1q 103 ip addr 10.0.0.5 255.255.255.252 ! interface GigabitEthernet0/0.104 description Site D encapsulation dot1q 104 ip addr 10.0.0.9 255.255.255.252
While more efficient than a tree of point-to-point circuits, the major drawback of a point-to-multipoint circuit is that spoke-to-spoke traffic must traverse the hub site. This isn't a problem if the spoke sites never need to talk to one another, but can quickly lead to problems if users at one spoke site need to access resources at another.
The logical evolution of the topology may be to expand to a multipoint-to-multipoint design, in which all sites are effectively hub sites. Each site is configured with a virtual circuit to each of the other sites, forming a full mesh of virtual point-to-point circuits across the WAN. But as you can see, multipoint-to-multipoint is hindered by the same scaling limitations as point-to-point: For n sites, n(n-1)/2 virtual circuits are necessary to form a full mesh. Although the number of physical circuits grows linearly as sites are added, the administrative effort to maintain so many virtual circuits can be a nightmare.
Metro Ethernet is a layer two metropolitan area network (MAN) service which simplifies the WAN topology greatly by effectively emulating one big LAN switch spanning an entire metro area. All sites connected into the metro Ethernet cloud are placed into a single multi-access segment. This allows each site to communicate directly across the carrier's infrastructure with any other site.
The catch here is that, as its name implies, metro Ethernet is typically limited to within a single geographic area. One could not, for example, order metro Ethernet connectivity among Los Angeles, Dallas, and New York. Its multi-access nature also introduces design considerations pertaining to the mapping of routing protocol adjacencies that should not be overlooked.
Unlike the previous classes, MPLS VPN is a layer three WAN technology. It can be delivered over any layer two transport which supports IP. A point-to-point circuit is installed between the customer site and the MPLS provider's nearest point of presence. A dynamic routing protocol such as OSPF or BGP is run between the customer edge (CE) and the provider edge (PE) routers to exchange routing information which then is propagated to the other sites within the MPLS VPN.
MPLS is a great choice for WAN connectivity because it offers national and international reach, high redundancy (through the carrier's multipath routed infrastructure), and linear scaling. Another important benefit is that MPLS removes the need to maintain a full mesh of routing adjacencies, which is of particular concern when using BGP. Rather than forming a mesh of adjacencies among all site routers, each site router needs only to peer with the provider and the provider handles all inter-site routing across its infrastructure.
The term MPLS can lead to confusion, because MPLS is only run across the carriers infrastructure (the P and PE routers), not on the customer's actual circuit. It is referred to as an MPLS VPN (as opposed to just MPLS) because the customer's network is implemented as a distinct VPN among many others belonging to the carrier's other customers.
Internet VPN Overlay
Finally, we have the do-it-yourself approach. If none of the aforementioned topologies are right for you, or if you find yourself severely limited by budget, you might opt for an Internet VPN overlay. With this design, each site has only an Internet circuit. Point-to-point or multipoint VPN tunnels (like DMVPN) are built among the sites to deliver secure, private connectivity. The biggest drawback to this design is that, underneath the overlay, you're still subject to the limitations of the public Internet, including non-guaranteed throughput rates and the absence of QoS controls. An Internet VPN overlay is however well-positioned as a backup to a more robust WAN solution.
About the Author
Jeremy Stretch is a network engineer living in the Raleigh-Durham, North Carolina area. He is known for his blog and cheat sheets here at Packet Life. You can reach him by email or follow him on Twitter.
Posted in WAN
February 6, 2013 at 5:34 a.m. UTC
Great overview! Thanks! Keep up the good work
February 6, 2013 at 5:39 a.m. UTC
February 6, 2013 at 9:23 p.m. UTC
Awesome write-up, man.
February 7, 2013 at 8:34 a.m. UTC
February 8, 2013 at 12:25 p.m. UTC
Thanks Jeremy. It's a great overview.
February 9, 2013 at 5:47 a.m. UTC
Good explanation mann...!!!
February 10, 2013 at 5:51 p.m. UTC
Great article. Thank you.
VPLS is another choice that can be delivered as layer 2 but you need to be careful about the number of MAC addresses a client is sending because that can fill the PE switch MAC table leading to outages.
February 25, 2013 at 5:15 p.m. UTC
Realy good explained, Jeremy. Thanks
February 26, 2013 at 2:59 p.m. UTC
This is a great overview. One thing I find fairly common is that engineers only look at options from the local telcos and don't consider private microwave when feasible between sites in a similar metropolitan area. I've consulted with a number of companies and found them higher bandwidth private circuits for less than typical leased lines.
March 1, 2013 at 4:17 a.m. UTC
Nice article. Short and sweet.
March 10, 2013 at 6:45 a.m. UTC
Very simple overview. Good read.
March 27, 2013 at 3:29 a.m. UTC