A big secret to Internet traffic routing

TCP/IP is the backbone of the internet and it controls how information gets from one place to another. The basic rule is that each router only knows about the ones next to it and should send traffic to the router it knows about that is closer to the destination that the current router. So each router should look at the routers around it and say "who's closest to where this traffic is going?" In turn the router that gets the traffic will do the same. Information flows from one router to the next in a zig zag line but it generally heads towards the destination.
 That's what the TCP/IP protocol says but in reality traffic can take very strange paths. Take a look at the diagram below:

In this example there are two major backbone providers. AT&T and Verizon. AT&T has connections from San Diego to San Francisco and from San Diego to Chicago. Verizon has connections from Richmond to Chicago and from Richmond to San Francisco. Each connection is assigned a "cost" of 1x or 2x. For this example let's assume that the cost for longer connections is higher because there is more fiber optic cable to maintain.

 

When a home user buys internet access from Verizon they are purchasing a connection to every other computer on the internet. A customer who buys internet access from AT&T is doing the same. The two providers have to connect their networks in order to provide Internet service to their customers. On one day a lot of traffic might go from AT&T to Verizon and the next day a lot of traffic might come from Verizon to AT&T. Figuring out who should pay who turns out to be almost impossible. What happens is that AT&T and Verizon sign a "Peering" contract which is an agreement for both to connect to the other's network without charging each other. They figure the traffic back and forth will probably even out in the long run.

 

Here are two "packets" of data to consider:

Packet 1: Starts in Richmond and needs to go to San Francisco

Packet 2: Starts in San Diego and needs to go to Chicago

 

The most obvious path for each packet is this:

Packet 1: Goes directly from Richmond to San Francisco on the Verizon 2x link

Packet 2: Goes directly from San Diego to Chicago on the AT&T 2x link 

 

Verizon would "pay" 2x to ship packet 1

AT&T would "pay" 2x to ship packet 2

Total cost to both providers: 4x

 

In reality this is how the packets travel:

Packet 1: From Richmond to Chicago on the Verizon 1x link, from Chicago to San Francisco in the AT&T network 

Packet 2: From San Diego to San Francisco on the AT&T 1x link, from San Francisco to Chicago on the Verizon network

 

Verizon would "pay" 1x for packet 1 but 3x for packet 2

AT&T would "pay" 1x for packet 2 but 3x for packet 1 

Total cost to both providers: 8x

 

Why? 

Because of "Peering" contracts each provider uses "hot potato routing." They try to get packets out of their network as quickly as possible and let someone else foot the bill of shipping it across country. Each provider sees a 1x link that is the shortest path to send the packet on.

 

What's really strange is that both providers end up paying more in total than if they had sent the packets directly to their destinations on their own network. The packets also ended up taking a much longer route than needed. But, there is no way for AT&T to know what Verizon is doing with packets and no way for Verizon to know what AT&T is doing with packets. If opponents can't tell what the other is doing they will naturally try to get the other to take more of the traffic load.

 

This is a classic example of the game theory prisoner's dilemma