I ran into a problem recently when rolling out EIGRP on top of an Internet VPN for backup connectivity to customer sites, and the problem isn't even with the more complex bits of such an implementation. Cisco, it seems, has taken an unfortunately forward approach in attempting to design my network for me, stemming from a relatively obscure MP-BGP "feature."
Our lab topology features a service provider network and a customer site with a single router. The customer site has two paths to the provider: a dedicated circuit serves as the primary path and a backup DMVPN tunnel over the Internet provides failover connectivity. BGP is run across the dedicated circuit and EIGRP is run across the DMVPN tunnel.
The customer network exists as a VRF named ABC within the service provider network. The customer router advertises a single network (10.0.123.0/24) to the provider.
The basic configuration to bring up the router adjacencies might seem a bit tedious for those not accustomed to such a setup, but it's really pretty simple. First, we'll configure MP-BGP for the ABC VRF on R1 so that it can learn the 10.0.123.0/24 network advertised via the dedicated link toward the service provider network core.
router bgp 100 no synchronization bgp log-neighbor-changes neighbor 172.16.0.5 remote-as 100 neighbor 172.16.0.5 soft-reconfiguration inbound no auto-summary ! address-family vpnv4 neighbor 172.16.0.5 activate neighbor 172.16.0.5 send-community extended exit-address-family ! address-family ipv4 vrf ABC no synchronization exit-address-family
We can see that BGP is up and the customer route has been learned via the core:
R1# show ip bgp vpnv4 vrf ABC 10.0.123.0 BGP routing table entry for 100:123:10.0.123.0/24, version 31 Paths: (1 available, best #1, table ABC) Not advertised to any peer 65000 172.16.0.5 from 172.16.0.5 (172.16.0.9) Origin IGP, metric 0, localpref 100, valid, internal, best Extended Community: RT:100:123 mpls labels in/out nolabel/16
So far, so good. Next we'll bring up EIGRP. (We can disregard the VPN configuration since we're focused on routing for the purposes of this discussion.) Note that EIGRP is VRF-aware on R1.
router eigrp 100 auto-summary ! address-family ipv4 vrf ABC network 172.16.0.0 0.0.0.3 auto-summary autonomous-system 65000 exit-address-family
router eigrp 65000 network 10.0.123.0 0.0.0.255 network 172.16.0.0 0.0.0.3 no auto-summary
Finally, we'll redistribute EIGRP into MP-BGP on R1. We'll use a simple route-map to set the route's local preference to 90 (which is less than the default of 100) so that the BGP-learned route is preferred over the EIGRP-learned route.
route-map EIGRP-In permit 10 set local-preference 90 ! router bgp 100 address-family ipv4 vrf ABC redistribute eigrp 65000 route-map EIGRP-In no synchronization exit-address-family
With our configuration work completed everything should function as desired. Unfortunately, this is where we run into a rather sticky problem. When we go to verify that the correct route is being taken, we see that it has actually disappeared: Only the EIGRP-learned route is present in the BGP table for VRF ABC.
R1# show ip bgp vpnv4 vrf ABC 10.0.123.0 BGP routing table entry for 100:123:10.0.123.0/24, version 36 Paths: (1 available, best #1, table ABC) Advertised to update-groups: 1 Local 172.16.0.2 from 0.0.0.0 (172.16.0.6) Origin incomplete, metric 409600, localpref 90, weight 32768, valid, sourced, best Extended Community: RT:100:123 Cost:pre-bestpath:128:409600 0x8800:32768:0 0x8801:65000:153600 0x8802:65281:256000 0x8803:65281:1500 mpls labels in/out 18/nolabel
What gives? Let's look on one of the core routers.
Core# show ip bgp vpnv4 vrf ABC 10.0.123.0 BGP routing table entry for 100:123:10.0.123.0/24, version 30 Paths: (3 available, best #1, table ABC) Flag: 0x820 Advertised to update-groups: 1 Local 172.16.0.6 from 172.16.0.6 (172.16.0.6) Origin incomplete, metric 409600, localpref 90, valid, internal, best Extended Community: RT:100:123 Cost:pre-bestpath:128:409600 0x8800:32768:0 0x8801:65000:153600 0x8802:65281:256000 0x8803:65281:1500 mpls labels in/out nolabel/18 65000 172.16.0.10 from 172.16.0.10 (10.0.123.1) Origin IGP, metric 0, localpref 100, valid, external Extended Community: RT:100:123
Okay, there are our two routes. The path through the dedicated circuit (toward 172.16.0.10) has a local preference of 100, and the path through the VPN tunnel (toward R1 at 172.16.0.6) has a local preference of only 90, so the former route should be the best route, right? But here we see that the VPN route is clearly preferred. What's going on?
The answer lies in this innocent-looking little extended BGP community:
Local 172.16.0.6 from 172.16.0.6 (172.16.0.6) Origin incomplete, metric 409600, localpref 90, valid, internal, best Extended Community: RT:100:123 Cost:pre-bestpath:128:409600 0x8800:32768:0 0x8801:65000:153600 0x8802:65281:256000 0x8803:65281:1500
This is a type of BGP cost community, which are used to administratively influence the BGP path selection process. And this particular community is of the especially volatile pre-bestpath variety, which essentially hijacks the entire BGP path selection process. The community itself holds the metric carried over from the original EIGRP route, and because it is a pre-bestpath community, this metric is evaluated before the traditional legacy BGP path selection algorithm starts.
Since the contending BGP route doesn't have a cost community assigned, it is considered to have a pre-bestpath metric of 2,147,483,647. This is the midpoint between 0 and 2^32, and obviously much, much higher than any sane EIGRP metric. So, the VPN route is declared the winner before BGP even gets a chance to weigh in (no pun intended).
But where did this cost community even come from? As it turns out, this behavior is actually baked into IOS to support a specific design scenario which Cisco calls EIGRP MPLS VPN PE-CE with Backdoor Links:
The "pre-bestpath" point of insertion (POI) was introduced in the BGP Cost Community feature to support mixed EIGRP VPN network topologies that contain VPN and back door links. This POI is applied automatically to EIGRP routes that are redistributed into BGP. The "pre-best path" POI carries the EIGRP route type and metric. This POI influences the best path calculation process by influencing BGP to consider this POI before any other comparison step. No configuration is required. This feature is enabled automatically for EIGRP VPN sites when Cisco IOS Release 12.0(27)S is installed to a PE, CE, or back door router.
"No configuration is required." Well thanks, Cisco. That sure is a nifty trick you've got there, but how about you let me design my network? I'll tell the router if I want it to do stuff like that.
At this point you might naively hope, as I did, for a simple off switch for this "feature" given that this hot-wiring destroys the BGP path selection process we all know and love (or at least know). But there isn't one. I played around with route-maps and community strings for a few hours before I opened a case with Cisco TAC, who confirmed that, no, there isn't a way to disable this behavior.
Defeated, I had a few options to resolve the issue:
- Enable the command
bgp bestpath cost-community ignoreunder the advisement of TAC on all core BGP peers. (Obviously this is not a trivial change.) This won't actually remove the community, but will neutralize its effect.
- Tweak the original EIGRP metric to something insanely high so that the default pre-bestpath cost is actually preferred.
- Drop EIGRP in favor of OSPF as my DMVPN routing protocol.
I opted for interim solution number two as it was admittedly dirty but by far the least disruptive, all things considered. Eventually, I'll implement
bgp bestpath cost-community ignore across the core routers, or possibly even switch everything to OSPF just to avoid this mess altogether, but why should I have to? Am I unreasonable in expecting this to not be a default or behavior, or even if it is, that the IOS developers would include a mechanism to disable it?
I've submitted a
bug report feature request via TAC to implement some control over this behavior in the future, but I'm not holding my breath.