Steve Gibbard
Reprinted from the Any2 Exchange Quarterly Newsletter, November, 2007.
A nice bonus to consulting on strategy for the Any2 project is that after spending several years working on infrastructure elsewhere in the world, IÕve been getting to do some thinking about what the US peering infrastructure should look like in a couple of years. Most predictions of the future turn out in hindsight to have been laughably wrong, and this will probably be no exception. That said, here are some random thoughts.
As I talk to backbone engineers and attend operational conferences, I hear a sense of doom. For the last decade, weÕve kept up with amazing amounts of traffic growth on the Internet by indiscriminately adding capacity. If a connection filled, we just replaced it with a faster one. WeÕve gone from T1s to DS3s, OC3s, OC12s, OC48s, OC192s, and now OC768s. On the Ethernet side, weÕve gone from 10 megabits per second to 100 megabits per second, 1 gigabit per second, 10 gigabits per second, and now 40 gigabits per second. And now, from the complaints IÕm hearing, weÕre stuck. The optical interfaces needed to increase capacity are unavailable. It seems unlikely that weÕll be stuck at 40 gigabits per second for very long – after all, faster optics have always come along eventually and thereÕs no reason to think theyÕll stop being invented – but we may well start seeing a situation where supply outstrips demand.
The situation reminds me a lot of urban planning models of the 1950s. WeÕve probably all seen the displays in museums – the futuristic cities with elevated freeways whisking drivers quickly from the suburbs to their destinations. And weÕve all experienced the results, sitting on those completely jammed freeways, sometimes for hours, trying in vain to get somewhere. WeÕve seen the highway engineersÕ attempts to deal with the situation – building wider and wider roads only to see them get clogged again. Among urban planners, that vision has been discredited. ÒTraffic is a gas: it expands to fill the available space,Ó an urban planner told me recently. Instead, they talk about Òsmart growth.Ó Rather than trying to build enough road capacity to move everybody between arbitrary locations miles away, they try to put people close to their likely destinations – in effect moving back to the development pattern that was forced before World War II by the lack of cars.
On the Internet – particularly the American Internet – weÕve got a similar history and a similar problem. WeÕve gone from a telecom world where everything was local – where making a long distance phone call was a big deal – to one where weÕre frequently told, Òdistance is dead.Ó In many ways this is good. It gives us access to literally a world of information. It lets us easily stay in touch with far away friends. But when I ask backbone engineers what all this traffic thatÕs causing them such agony is, they tell me itÕs mostly American video being shipped from China to individual American viewers because the file sharing applications make no effort to distinguish local and distant sources. The backbone operators can keep adding capacity, but the videos are going to get bigger too, and the number of viewers seems likely to explode as people become accustomed to the idea of video on demand. It would make much more sense to apply some of the principles of Òsmart growthÓ to this.
That means putting the content close to the users. There will be some economies of scale involved in determining how close, but itÕs worth noting that just 20,000 people downloading iTunes video at its 2 Mb/s viewing rate would fill one of our currently state of the art 40 gigabit per second backbone links. How many simultaneous primetime TV viewers are there in a typical medium-sized city?
In much of the world, particularly in places where long distance capacity has long been in short supply, exchange points can be easily categorized as big international exchanges or local exchanges. The international exchanges are often only a few per continent – think LINX, AMS-IX, and DE-CIX in Europe, or HKIX and the various exchanges in Singapore and Tokyo in Asia. Keeping the number of global exchanges small allows international networks to build out to a limited number of locations, and be assured of getting fairly full connectivity. Local exchanges serve a very different purpose – allowing networks in a specific geographic area to hand off traffic to other networks in the same area, without the cost in money or performance of hauling the traffic to an international exchange. ItÕs the local exchanges where the most popular video content presumably belongs. In the US, weÕve either blurred that distinction or done away with the local exchanges altogether.
The US has some of the worldÕs most important exchanges for international traffic, mostly on the East and West Coasts where international exchanges would be expected to be. The few in the middle – mainly Chicago and Dallas – serve such wide areas that they can hardly be thought of as local exchanges either. While local networks do connect to exchanges in their areas and peer with each other, pricing designed to squeeze the maximum amount of money out of those with the resources to operate big global networks makes those exchanges increasingly hard for local networks to justify.
This is poised to be an exciting time for US exchange points. After several years of having only two major exchange operators, there are now several investment firms buying up telecom buildings and planning new exchange points. CRG West is one of these, and after some considerable success with the first Any2 exchange in Los Angeles theyÕre launching exchanges in several more locations. Boston is the one of those I find exciting because itÕs a previously unserved market. But the other locations will be interesting as well. Even in areas with well-established exchange points, CRG WestÕs pricing model should enable local peering for those who canÕt cost-justify it at the existing exchanges. And as other real estate companies see whatÕs possible and follow suit, IÕm hopeful that weÕll start seeing more exchanges in more places, bringing us closer to what should be a sustainable model for the American Internet as it continues to grow.