The premiere source of truth powering network automation. Open and extensible, trusted by thousands.
NetBox is now available as a managed cloud solution! Stop worrying about your tooling and get back to building networks.
Comcast vs. Level 3
By stretch | Wednesday, December 1, 2010 at 8:26 p.m. UTC
There has been much in the news this week concerning the clash between Level 3 and Comcast regarding Level 3's new position as the content delivery network (CDN) for Netflix. While the disagreement itself isn't a terribly disruptive ordeal (to date, there has been no interruption of service) it marks an important point in the ongoing evolution of the Internet.
Last month, Level 3 became a primary CDN for Netflix. This is interesting in itself as Level 3 has traditionally stuck to tier one transit services, but has recently entered the CDN industry. To put into perspective the sheer volume of traffic generated by Netflix, Level 3 says it will add 2.9 Tbps of additional CDN capacity. Naturally, Level 3 set about increasing its peering capacities with other networks to handle this new load.
This is where the dispute with Comcast arose. A letter from Comcast to the FCC (see the PDF for the full version) says that as part of their upgrades, Level 3 "demanded 27 to 30 new interconnection ports" be added to the existing settlement-free peering between the two networks. Prior to the additional connections, the traffic exchange ratio was reportedly around 2:1, with Level 3 sending about twice as much traffic as it was receiving from Comcast, which is generally considered to be acceptable for free peering. Comcast claims that increasing the peering capacity as requested "would result in a traffic imbalance that could be in the range of about 5:1." Comcast responded by demanding that Level 3 begin paying an undisclosed fee for the increased traffic flow.
The Issue
If Comcast were charging Level 3 for transit service, this would be business as usual, not even worth commenting on. But this situation is markedly different as Comcast is demanding payment for traffic terminating on its own network. The traffic in question ultimately must traverse Comcast's infrastructure, regardless of who the immediate peer is. Comcast effectively is charging two parties for a single service: it wants Level 3 to pay for sending data, and its own subscribers to pay for receiving that same data.
In an ideal economy, Comcast should be able to charge what it likes. And Level 3 should be able to decline. And Comcast subscribers should be able to switch to another provider if they want to watch Netflix. Sadly, this is not the situation of Internet access in the US. For millions of residents, Comcast is their only choice for broadband Internet access. Terminating peering with Comcast would mean an inability to deliver Netflix to those millions of potential customers, and Comcast knows it.
For now, Level 3 has agreed "under protest" to pay the fees demanded by Comcast to avoid interrupting service to end users.
The Bigger Issue
Comcast is primarily a residential service provider. Generally, it will receive more traffic from peers than it sends, as is the nature of the end user as a content consumer rather than a content generator. Following the logic employed in its decision to charge Level 3 for asymmetric peering, what is to deter Comcast from charging other peering networks as well? The structure of the Internet provides no mechanism to prevent Comcast from holding hostage its millions of subscribers so long as it maintains a monopoly over their Internet service.
The bottom line is that Level 3 needs Comcast to reach the users who pay Comcast for Internet service. Likewise, Comcast relies on Level 3 to make its share of the Internet (e.g. Netflix) available to subscribers. When one side attempts to tip the scales, the stability of the public Internet is damaged.
Further Reading
- Comcast's letter to the FCC (PDF)
- Level 3 vs. Comcast: More Than A Peering Spat? - Data Center Knowledge
- How Comcast became a toll-collecting, nuke-wielding hydra - Ars Technica
- Internet peering disputes: follow the money - Ivan Pepelnjak
- UPDATED: Statement from Level 3
Comments
December 1, 2010 at 8:50 p.m. UTC
The biggest issue in my mind is the fact that Comcast provides Internet access for users. As applications on the Internet get more and more advanced, we're going to see more and more of this type of thing going on. Cisco Umi is a fair example of that. What if that takes off and everyone is now streaming 1080p (or whatever resolution it runs) across the Internet? Who pays to ship that content to and fro? What if Comcast doesn't peer directly with the CDN serving "the next big thing"? I don't have an answer, but none of this is going to end well.
Thanks, Comcast, for living up to your reputation of being the worst company in the country.
December 1, 2010 at 8:51 p.m. UTC
If Comcast has the audacity to attempt something like this and gets away with it, imagine what all other service providers will do. The cost of online services will increase, might even have to start paying Microsoft for Windows Updates...
December 1, 2010 at 9:21 p.m. UTC
See, the way the letter sounded, Level3 wants to ramp up the amount of traffic it sends into Comcast's network. And from what I understand by today's peering standards this would entail a typical move from a free agreement to a peering agreement in which Level 3 would compensate Comcast for the larger amount of traffic.
The problem here is that Level 3 doesn't want to pay for the increase in traffic. They just want to get it for free.
If you read the pdf you will find that Level 3 itself has been in this same boat before. Another network wanted to do to Level 3 what they are trying to do to Comcast and Level 3 basically said 'No, it would be unfair.'
I will admit I have not looked at Level3's side so I do not know what rationalization they are using for this move, but so far I agree with Comcast. It's a douche bag move to undercut other CDN's by forcing Comcast to take accept the unbalanced amount of traffic for free.
December 1, 2010 at 9:22 p.m. UTC
I think Comcast did have an agreement with the previous CDN that NetFlix used, Akamai. Akamai may have been charging NetFlix the fee and could be one reason why NetFlix looked elsewhere and found a cheaper CDN, Level 3 who might not have known about the Comcast/Akamai agreement.
I'm not a big fan of bandwidth providers provide anything other than bandwidth. In this case, I could see potential conflicts occurring between Level 3 customers being favored by Level 3 CDN versus Akamai CDN.
December 1, 2010 at 9:24 p.m. UTC
@Robert: The instance referenced in the letter was between Level 3 and Cogent, and affected transit traffic, which is an entirely different ballgame. The dispute between Level 3 and Comcast involves traffic destined for Comcast customers.
December 1, 2010 at 9:42 p.m. UTC
Regarding your statements that L3 is the new CDN for netflix...
-
netflix has a multi-vendor approach, also for parts of their CDN, nothing new here
-
24 milion in capex on CDN tech in SIX MONTHS doesnt buy you much if you talk the scale of netflix
"Akamai spokesman Jeffrey Young said in an interview with Tech Trader Daily that reports of poor performance by the Akamai network are "simply untrue." He said the company continues to work with Netflix closely, and that they will continue to be a customer into next year. Young notes that it has been widely reported that Netflix is taking a multi-vendor strategy for content delivery."
December 1, 2010 at 9:50 p.m. UTC
@Job: As the article states, Level 3 (not to be confused with L3 Communications) is a new CDN for Netflix; I'm sure they have others as well.
I'm not sure what the "24 million" you mention is in reference to.
December 1, 2010 at 10:08 p.m. UTC
December 1, 2010 at 10:29 p.m. UTC
@Robert
"It's a douche bag move to undercut other CDN's by forcing Comcast to take accept the unbalanced amount of traffic for free."
If I remember correctly this is the same Comcast that was looking at charging their customers for bandwidth usage, so if they were going to charge Level 3 for "Access" to their network and charge "customers" for having the audacity to use that bandwidth I would say they are getting their cake and eating it too...
So I would have to say that the FCC or someone is going to have start to do something here otherwise the only people that are going to lose is the customers.
December 2, 2010 at 2:55 a.m. UTC
What's amusing here, and I haven't seen much discussion of (here, or on NANOG or the like), is how this same scenario would look to a smaller ISP.
Eyeballs want content. ISP of eyeballs has to pay for transit. If 'small ISP' went to any CDN and said 'Pay me, so that you can send traffic to my users', they'd laugh and watch said 'small ISP' promptly fall off the face of the earth.
I fail to see how this is any different when 'large ISP' asks a CDN to pay to access their eyeballs. It's the responsibility of the ISP, regardless of how big/small to ensure their users have access to the content. If large, that might mean accommodating a large number of private peering scenarios, or if small, paying for more transit.
One other thing that bugs me here is the ratio argument. 90% of Comcast's services, by definition are an unbalanced ratio (10 down / 1 up). They can't realistically think they can maintain a somewhat equal ratio.
IMHO, yeah, Level3 is probably double dipping a bit, but they shouldn't have succumbed to Comcast's demands. I would just let them pay someone else to transit the traffic. Users suffer, complain, and someone with clue at Comcast might finally get the hint.
December 2, 2010 at 3:07 a.m. UTC
I can understand Comcast’s position regarding the leveraging of the peering agreement to cut Netflix a better deal, especially since they are a competitor in what I presume is Comcast’s most profitable space. It's an obvious profit center for Level 3 and an obvious cost center for Comcast. It is the non-obvious benefit to their customers that Comcast is overlooking. If they were just an ISP, they might even advertise that Netflix works great on their network because of the relationship.
Given that Comcast is primarily residential and small business customers I'm surprised the current ratio is only 2:1. Must be all the bittorrent traffic...
December 3, 2010 at 12:53 a.m. UTC
AT&T tried to pull this stunt on Google a couple years ago, and to my recollection, it didn't happen. Bottom line, Comcast is getting greedy (and they aren't alone). They are charging their customers for accessing the data and they are charging Level 3 for providing the data to their customers.
December 4, 2010 at 12:30 a.m. UTC
Welcome to the world of peering. Level3 is hardly the only company Comcast charges for traffic. My former employer had a peering arrangement with Comcast. We paid them money for it. Why? Because it was cheaper to pay Comcast per meg for the amount of traffic we had destined for their network than it was to pay our actual transit providers.
This should never have been news. As much as I hate to say it, Comcast is in the right here. This is a peering argument that should never have gone public.
December 6, 2010 at 5:26 p.m. UTC
@drakonblayde
see hoyosa’s argument three comments up. This isn’t “peering”. “peering” is what large scale networks do. Verizon, Qwest, Level 3, AT&T, NTT, THOSE are peering networks.
When Comcast can get from Chicago to New York, or New York to San Jose, without going through Level 3, or when Level 3 has to use Comcast’s network to get from Point A to Point B, then its PEERING.
Level 3 is a Tier-1 global carrier and Comcast is a Tier-2 network of consumer endpoints spread to hell and back across 1/3 of a continent. Comcast, like ANY other consumer/business ISP in the US, couldn’t exist without buying or agreeing on service from Level 3, and/or other similar Tier-1 providers. And I’m willing to bet my salary this month that your former employer’s agreement with Comcast was “peering” in name only. “peer” implies some form of reciprocal action, that both parties are playing on the same level. Your employer was most likely a “Customer” of Comcast, not a peer.
December 8, 2010 at 3:09 a.m. UTC
If comcast wants to prevail in this struggle should consider becoming a Tier 1. Doing this would be a company with a profitable future in both markets. The broadband market services is growing like wildfire. This isn't a easy decision to make ($$$), but if they do it, will have a good harvest in the near future.
March 4, 2011 at 9:06 p.m. UTC
Stretch, thanks for your cogent analysis of this issue. I was able to use this post to bolster my arguments in an ongoing dialogue with Alan Roth of USTelecom (a broadband ISP trade group) over at my blog Legally Sociable.