Routed or Switched Access Layer: Why not Both?

This is the same way I build my networks generally. With a mixture of MST or PVST+ I have found it to be quite stable.

Its little more painful in the hosting/SP side of things if you are adding vlans constantly, but it still works (assuming your MST doesnt have a hissy-fit).

Kurt
@networkjanitor

Mu opinion is that this should not even be thought of when designing a enterprise network. This is really a pain to keep track of if you have like a 1000+ access switches I think even 10 with be enough if you ask me. One of the big reasons you don't do this with Cisco devices is because the pricing of switches able to do dynamic routing which perhaps would be the solution. I strongly recommend people not to do this in a enterprise environment. Instead I would say that this is possible for what ever reason a administrator would need it.. A typical scenario where this is used is to tie up a OSPF square in a datacenter solution where you might have a L2 link connecting two DCs but also using L3 on the WAN side. It's kind of hard to explain if you don't draw that exampel but use your imagination.

So come on Jeremy you can do a lot better then this!

Love your blog though, keep it up!

I love this design, but for larger enterprises, I could see some concerns. Moving routing to the access layer requires switches robust enough to perform dynamic routing and maintain enterprise routing tables. Not a problem for newer equipment with the right feature set, but there's a cost and lifecycle factor to consider in larger networks.

Another concern would be complexity. In a large environment, where part your support staff may not have high level networking knowledge, the added complexity of overlapping L2/L3 topologies may become difficult for them to understand, manage, and troubleshoot effectively.

This is a great design for hierarchical campus environments though as it eliminates spanning-tree blocking from those VLANs that do not span multiple switches and potentially allows for L3 load-balancing to distribution pairs. It also allows you to move L3 security features to the access layer which can be advantageous for PCI compliance and HIPAA concerns.

How about access control? I believe that one could end up in a management nightmare with Access-lists or some sort of firewalling between segments. Those lists would have to end up in all sorts of places to provide necessary security :) (meaning; in the access layer, distribution layer and the core).

I think if your concern is to utilize more links one solution is to do that with multiple groups of HSRP each on different Vlan and distribute the Root Bridge priority for those Vlans across the distribution/core switches.

What do you think ?

Whether or not any of the VLANs you define on the trunk are native is irrelevant. For the purposes of this article, it doesn't matter whether a given VLAN is tagged or not.

@GT: Actually, routing to the access edge is the solution endorsed by Cisco, and is becoming quite common for a number of reasons. Also, I would argue that with proper documentation and change controls in place, this design actually scales very well, given that you don't extend the distribution-level L2 VLANs too far.

@Santino: One of the advantages of the hybrid L2/L3 design is that it's a cookie cutter. Admin staff only have to learn one design, which will be configured similarly across all access nodes. Further, this isn't "high-level" networking by any means; a CCNA should have no problems understanding the design.

@Jay: Access controls must exist on the distribution nodes for the switched VLANs, and can exist either on the distribution or access nodes for the VLANs carrying routed traffic. (Generally, you want to avoid performing policy enforcement on core devices.) You can still have all access lists on distribution nodes just as you would have for a L2-only access edge. Also, note that HSRP and root bridge prioritization does not achieve the real load balancing that equal cost path routing gives us.

Hi ,all

What is really the difference between two approaches ?

Is L3 faster than L2 or what ? I don't get it

In L3 we have those point to point SVI and CEF is directing the packets to distribution layer whereas in L2 we have the same but in L2 only can you explain it to me ?

Stretch,

Tried to catch you on irc but you weren't there. Shouldn't there be a Layer2 connection between the two access layer switches for vlan100 to exist on both switches? How is it possible to have 10.0.100.0/24 on each access layer switch if vlan100 isn't being trunked?

Cisco is moving towards layer 2 routing in their new Nexus series switches and NX-OS. Spanning tree is going away and being replaced with VPC (virtual port channels). This is really the future for enterprise wide 10G connectivity. There is a good white paper that discusses your topology here contrasted with the newer L2 Nexus model.

Link

Cheers.

I think this design is not a good one and should be avoided.

First, it requires careful planning in regards of STP, VTP and VLAN pruning, much more than in the "traditional well-known" routed / switched access layer designs.

Secondly, this design might sound feasible, it has the potential to grow to a nightmare when devices are moved to another access switch and they should keep their IP addresses "because its easier and so much better" (in regards of firewall rules, application policies, ... - take a reason of your choice...). If this will be done, you're going to have one of the worst case traffic flow you can imagine, crossing the distribution layer at least twice (once at L2 + once at L3). Additionaly, troubleshooting becomes more difficult, compared to the well-known designs....

Thirdly, L3 convergence depends on STP. If STP blocks the VLAN for the L3 tranfer link between access and distribution switch for some reason, it will take quite a long time until the routing protocol will notice it.

In my opinion, it's just another kludge and kludges are bad...

@Christoph: The hybrid design doesn't place any more emphasis on STP planning than does a plain L2-only topology. With regard to your second point, simply which VLAN you place a device into determines whether or not it will need to change its IP when being moved to a new distribution block. Finally, yes, there will be negligible overhead from running STP on point-to-point links, but RSTP does not pose the problem you describe.

Remember, this isn't a new design; it's simply running both an L2 and L3 topology side by side. I've seen it deployed in the real world and it works quite well so long as diligence is applied when spanning L2 VLANs beyond a distribution block. It's definitely not a kludge.

I would also avoid this type of mixed topology. Enterprises should be using cookie cutters as much as possible. The more similar your topologies, the less you need diagrams and documentation when troubleshooting, the faster you can identify and fix problems.

If you want to avoid having STP blocked ports look at a VSS, vPC, or a pure routed access model.

Great article. Very clever. This design also means that the access layer vlans don’t need to be on the distribution switches (or in our case the core). And anyway, if you’ve got layer 3 capable access switches – you paid for that capability, why not use it?

Hey Christoph, how does STP block a point-to-point vlan? Ou est le loop?

On the use of the pejorative term – kludge. None other than the great Radia Perlman described NAT as a kludge. Maybe it is from a Utopian technological point of view, but NAT is digging us out of a big hole at the moment. So not all kludges are bad. Just a compromise. Stretch’s design is pretty far from being a just compromise, in my opinion.

@ strech

With regard to your second point, simply which VLAN you place a device into determines whether or not it will need to change its IP when being moved to a new distribution block.

Sure, but the point is that every VLAN can be made available everywhere, therefore every IP subnet. And from my expirence some people tend to do almost everything that can be done for whatever reasons.

Remember, this isn't a new design; it's simply running both an L2 and L3 topology side by side. I've seen it deployed in the real world and it works quite well so long as diligence is applied when spanning L2 VLANs beyond a distribution block. It's definitely not a kludge.

I've also seen it deployed and I agree with you that it can work quite well if you take care at some aspects (keep VLAN spreading as small as possible, use RSTP, ...) But I've also seen it failing in deployments, where no one really took care about this resulting performance issues and other strange things... Maybe "kludge" is exaggerated, compromise would be a better term

@alan

Hey Christoph, how does STP block a point-to-point vlan? Ou est le loop?

It should never do, but with STP, there should never be a loop too although loops happen. But think about a real-life network where things like accidentally mis-configuration may happen from time to time...

Here's a real world example of why you might want to do this:

Our network has a wifi SSID that has to be available at every site. This wifi network we have is not a Cisco wifi network so it does not use LWAPP tunnels etc. Therefore it HAS TO BE layer 2 everywhere. We would normally have a problem with this because every link back to home would have to be layer 2 just to make wifi happy. We do that with the hybrid VLAN concept with 3560's, and for our desktop hosts, we provide them a local layer 3 VLAN.

The person who made the point about the complexity of this is correct. It is kind of hard to understand looking at show runs. The way I see it is the EIGRP routing process looks in all the VLANS for interesting layer 3 stuff like those hello's and acts accordingly. All you have to do is advertise your local VLAN(s). These local VLANS do not go over the trunk, but the layer 2 VLANS do.

Dano

Geert,

Come on man, we all know that Wifi is the future and is therefore more important than wired . We would sooner go back to using typewriters before re-engineering the ridiculous wifi.

I agree about the layer 2 loop possibility. Good thing we got iterations of STP protecting us. Only problem is most network people don't know how to troubleshoot STP all too much. In our network we turn off STP for some VLANS, but not on all switches where that VLAN exists. I'm not really sure how we have been able to avoid bridge loop disaster.

Dano

I guess you could add into this VTP domains.

At Access Layer, you could use L2 switches with trunks (uplinks) to Distribution Layer where SVI's perform inter-VLAN routing. If you use two L3 switches, you can have HSRP/VRRP/GLBP for L3 SVI resilience (and pseudo or real load balancing if you alter priorities for VLANs, alternating between the two L3 Distribution switches or use GLBP).

That would be one VTP Domain (say VTP1). CIDR block would be say, 10.0.0.0 /16 further subnetted for each VLAN

Now replicate this entire model. VTP2 would be 10.1.0.0 /16

You can use the same VLAN numbers on each VTP domain, for consistency. You can add a core layer and route between subnets to reach devices on other VTP domains / subnets / vlans.

This model would be very scalable. You can also change core layer from say Frame-Relay to MPLS - in other words the design is modular. QoS trust boundaries and where you mark is easy to determine and manage, it's all good.

Normally the Access Layer connects devices to the network. The Distribution Layer performs inter-VLAN routing / packet filtering / IGP Routing

The Core Layer's job is to shift data as quickly as possible. No ACL's or other stuff, just shift data.

This is only my opinion of course.

I used this hybrid method at our corporate offices, for about 20 IDF closets that were Cat 4507R's. The solution was effective and allowed us to go active/active on the IDF-to-Core uplinks. However, the solution is a pain to support and is a litle more complicated to troubleshoot.

We had this in place for years with no issues, but went back to L2 when we upgraded the core to Nexus 7k's. Now we run VPC's and are back to straight L2.

If I had to do it all over again in a non-Nexus world, I'd probably have passed on the hybrid and gone straight L3.

Juniper is always pushing this approach. I am not sure the slight performance boost as they put it, trumps the complexity of the design. Its not complex by itself, but when you factor in the security and business requirements, I can see this thing growing out of control if you work for a company as dynamic as mine.

If your goal is just eliminating spanning tree, and you are talking about a controlled data center environment ( not user switches) I would recommend disabling spanning tree and configuring flex links. Its a common practice in the ISP environment.

I will say that this scheme is slightly easier to manage with juniper switches, as you can manage all the switches in your company as one giant virtual switch.

Ok, can I throw another scenario in for your comments? What if the distribution layer consists of a single stack of 3750s and the Access layer consists of blocks of 3560s connected redundantly to one another using the SFPs. I guess you'd need to run Spanning-Tree within the 3560 Access blocks, but I'm not sure how I would go about configuring dual uplinks to distribution layer considering there will only be the one SVI per VLAN (because the distribution layer is a single stack).

Port-channelling isn't an option here because the 3560s are not stackable and I would want the uplinks to come from different switches in the block.

Great blog btw!

Nice Post. Add VRF into the mix and you can have path isolation to a central firewall across L3 boundaries.

To the naysayers...study up. You will see the benefit. Do not dismiss something simply because you do not understand.

Layer 2 adjacency requirements are becoming a thing of the past, and the need to span vlans across access layers switches is going away. Keep in mind that Cisco's tried and true L2 design doesn’t have vlan's spanning across access layer switches to avoid STP convergence, solely relying on HSRP failover. With layer 3 access layer links, both links are forwarding and load balancing.

This kind of design is very applicable for server farms where much of data are being process from time to time, campuses which taking online lessons (audio, video, and live conversation) but for some company, the old style of access-distribution-core layer design will be more consider for normal activities due to cost-budgeting.

But for companies who really rely on the internet for everything like what I've mention in the first place, it would be a good practice.

@stretch -> Thank you sir again for this additional knowledge. "Compilation of words and ideas makes a better world!"