Material from Computer Networking: A Top Down Approach, 6th edition. J.F. Kurose and K.W. Ross

ISPs, Backbones and Peering

14-740: Fundamentals of Computer NetworksBill Nace

Administrivia

• Norton2010 Paper Review for today

• Lab0 is posted

• I'm in Japan!

2

Last Lecture• Internet TCP/IP architecture• Layering not strictly enforced• “Hourglass” -- IP ueber alles• Allows rapid innovation at layers below• Flexible applications / services above• IETF process• “Rough consensus and running code”

• End-to-end argument• Does it still apply?

3

traceroute

• ISPs and Backbones• Peering and Settlements

• Peering Evolution

• Interconnections

4

A Packet’s Journey• Packets travel across many networks• Particular protocols will be studied later• This lecture motivates why routing

mechanics are necessaryCMU Pgh

PennRen

Some NW

Tier 1 ISP

Tier 1 ISP

Tier 1 ISPSome ISP

Little ISP

Cheap ISP

Huge ISP

CMU Kobe

AB

C

E

F

D

G

H

J

5

CMU’s Infrastructure• Two 10 Gigabit Ethernet links

(singlemode fiber) to Pittsburgh Supercomputer Center

• From Cisco 6500s located in Cyert and Stever House

• “Redundancy” links to PennRen / Level3 over 1Gbps rate-limited 10 Gigabit Ethernet

• Money

• $40K per year to PennRen

• A PA state education network

• $100K to PSC, $100K to Level3

• $300K per year on fiber leases

1Gbps/10Gbps10Gbps

10Gbps 1Gbps

Internal

External

CMUInternal

PennRen

PSCInternet2

Level3Comodity Internet

Backbones• In the beginning ... of Internet time• Single backbone: NSFNet• Everyone on the “Internet” was on NSFNet• A backbone network enables all connected

end-hosts (users and companies) to communicate with each other

• No interconnection problems

7

Backbones (2)• Commercial backbone providers emerged• Technology transfer!• A “Good Thing”• If US Government was managing the

Internet, might not be so successful• We all expect universal Internet

connectivity

8

Interfaces: Transit

• Transit / provider-customer

• ISP sells access to another ISP or company

• e.g. CMU buys transit (or access to Internet) from Level3

9

Interfaces: Peering• Reciprocal access to each other’s customers

• Usually free exchange of traffic

• DO NOT serve as transit for 3rd party data• E.g. Google and MSN peer with each other, so email

messages between Gmail and Hotmail are transferred directly, without going through their transit providers

• These relationships are confidential business secrets

• Roughly hierarchical, though the topology is flattening

10

Tier 1 ISP• Internet backbone providers• Peers with every other Tier-1 in “Internet Region”

• Who is a major ISP in your country?• US?

• Japan?

• India?

• China?

• Careful: “Tier-1” is an overloaded and misleading term

11

Images courtesy of Level-3

Images courtesy of Verizon>4200 networks in 142 nations, many at OC-768 speeds

Image courtesy of tatacommunications.com/network

Tata: "Discover the World's Largest Global Footprint"

“Typical” Infrastructure• High speed links• Level3 operates 40Gbps DWDM networks

• High performance routers

• Over-provisioned bandwidth• 40 ms delay within region• <1% packet loss

• Global presence (or at least multi-continent)

15

“Typical” Tier-1 Relationships

• Directly connected to other Tier-1 ISPs (i.e. peer with)

• Connected to a large number of Tier-2 ISPs

• Vertically integrated: sell services directly to customers

• International in coverage

• Ecosystem: do not buy transit from another provider in order to reach the whole Internet

16

Why do Tier-1 ISPs need to peer with each other?

• No single Tier-1 ISP can reach the whole Internet on its own• Internet is a network of networks

• But Tier-1 ISPs have a restrictive peering policy• Do not peer with other non-Tier-1 ISPs• They are potential revenue generating

customers• No incentive to accept additional peers

17

What is a Tier-2 ISP?• Network infrastructure is usually regional

• Customer of Tier-1 ISP(s) – needs to buy transit

• Provider of customers – also re-sells this transit

• Peers with other Tier-2 ISPs – settlement free

18

Tier 1 ISP

Tier 1 ISP

Tier 1 ISP

IXP

Tier-2 ISP

Tier-2 ISPTier-2 ISP

Tier-2 ISPTier-2 ISP pays Tier-1 for connectivity to rest

of internet

Multi-connection relationships possible

Tier-2 ISPs also peer privately. Interconnection can be direct or via IXP

Thinking time

19

• Why do Tier-2 ISPs need to buy transit from a Tier-1 provider?

Thinking time

20

• Why do Tier-2 ISPs peer with each other?

Open Peering Policy• Tier-2 ISPs have an open peering policy• Peer with anyone possible

• Costs of peering have to be balanced against gains for a Tier-2 ISP• Management cost: usually settlement-free

peering means both parties should send approx equal amounts of traffic to each other

• Maintenance cost: extra equipment, transmission capacity to meeting point, exchange and other fees

21

Content Providers• Do not sell transit

• Category A: Focus on content creation• Do not want to operate a network• Do not have expertise in maintaining peering

relationships — router, policy, negotiations, …• No Peering Policy

• Category B: Sophisticated, large-scale players• Use peering to improve user experience• Open Peering Policy

22

Peering Ecosystem

Tier 1 ISPs

Tier 2 ISPs

Content / Enterprise Companies

Full MeshPeering

Partial MeshPeering

GenerallyNo Peering

Must BuyTransit

Transit Free

Must BuyTransit

Internet Region

Things are never so clear-cut . . .

traceroute

• ISPs and Backbones• Peering and Settlements

• Peering Evolution

• Interconnections

24

US Evolution• 1999/2000 economic collapse • Telecom sector• Need to rethink their business model• Need to cut costs

• General dotcom bust• Lots of cheap equipment on eBay• Transit rates drop• Upstream provider @Home for cable companies went

bankrupt

• Peer-to-peer file sharing grows exponentially in popularity

25

Transit Prices are Falling• Price per Mbs ➙ 63% per year decline

• Luckily, volume is increasing even more

• Somewhat bad for Tier-1 ISPs

26

Content Companies are Peering

• Network savvy, large scale enterprises• MSN, Yahoo, Google, Ebay, Walmart, …• Yahoo has an open peering policy• Reduce transit costs• Improve end-user experience• Good for content companies• Bad for Tier-1 ISPs

27

traceroute

• ISPs and Backbones• Peering and Settlements

• Peering Evolution

• Interconnections

28

Interconnection

• How do two networks interconnect with each other?• We are talking about routers in two

different companies, under separate administrative control

29

Public Peering• U.S. government decided to let commercial companies

take over management of the backbone networks• 1991: Commercial Internet Exchange (CIX)• 1 router in Santa Clara, CA

• 1995: Network Access Points (NAPs) in SF, Chicago, NY, D.C.

• More recent: Internet Exchange Point (IXP)• 30 in US, 6 in Japan

• “More is good” because of congestion, at one point 20% of traffic going through the 1st NAP was dropped!

30

ISP Connections

Tier 1 ISP

Tier 1 ISP

Tier 1 ISP

IXP(Meet-me

room)

Tier 1 ISP

Tier-1 providers interconnect

privately (Peer)

Tier-1 providers also interconnect at Internet Exchange Points (IXP) (30 in US)

Private Peering• Two networks bypass IXPs and directly peer• In response to congestion at some IXPs• Business reasons

• Also more cost-effective• There are limited number of IXPs at major regions• Can privately peer in other locations

• Sometimes take a hybrid approach• Public exchange for some peers• Private meeting point for others

32

Provider-customer Interconnection

• Point-of-presence (PoP)

• Customer typically leases a high-speed link from a 3rd party telecomm provider

• Directly connects a router at a provider’s PoP

33

Various Methods

• How do two networks interconnect with each other?• Many methods: Public Peering, Private

Peering, Provider-customer PoP

• There are really no industry-specific regulations, mostly purely driven by commercial and economic forces

34

How does the world connect to Facebook?

Facebook’s network

IXP in Germany

IXP in Japan

IXP in Sweden

Level3

Amazon

Directly Connected Peer?

Image courtesy of Teun Vink (teun.tv)

Lesson Objectives• Now, you should be able to:• analyze business practices of various enterprises

using the multi-tier network model (Tier-1, 2, etc) and common peering practices

• describe the relationships and associated motivations for enterprises on the internet

• analyze the effect of recent trends in internet usage patterns on the various business enterprises on the internet

• describe interconnection methods between enterprise networks

36