Internet Video: The Next Wave of Massive Disruption to the U.S.

Peering EcosystemWilliam B. Norton

Co-Founder & Chief Technical LiaisonEquinix, Inc.

Clean Slate Networking Research WorkshopWednesday, March 21, 2007 13:30-14:00Frances C. Arrillaga Alumni Centerv0.3

Merit @ UMich – NSFNET Project

Logistics

NSF funded$

1987-1994

Regional Techs Meetings

NSFNETR

R R R R R R

R

RRRRR

Host

Agenda

Then port 80 grew, NSF stopped funding…Open sharing of Ops Data

Merit @ UMich – NSFNET Project

Logistics

1987-1994

Regional Techs Meetings

NSFNETR

R R R R R R

R

RRRRR

Host

Agenda

NSF funded$

NSFNETR

R R R R R R

R

RRRRR

NSP

NSP

NSP

NSP

NAP

Commercial Internet R

RR

RR

R

1994

North American Network Operators

Group (NANOG)

$ Self-Sustaining

Regional Techs Meeting

Slow transitionLittle sharingWeak SLAs&NDAs1998->EQIX

Equinix

• Carrier Neutral Internet Exchange• Bay Area, LA, CHI, DAL, DC, NYMetro• Singapore, Hong Kong, Tokyo, Sydney• IPO in 2001, Market Cap $2.5B today• 1200 Customers, 200 Networks• Tier 1 ISPs, Cable Cos, Content, etc.• Efficient interconnection of services

Catbird seat90% documenting Internet Ops Practices

Internet Operations White Papers1) “Interconnection Strategies for ISPs”2) “Internet Service Providers and Peering”3) “A Business Case for Peering”4) “The Art of Peering: The Peering Playbook”5) “The Peering Simulation Game”6) “Do ATM-based Internet Exchanges Make Sense

Anymore?”7) “Evolution of the U.S. Peering Ecosystem”8) “The Asia Pacific Internet Peering Guidebook”9) “The Great (Public vs. Private) Debate”10) “The Folly of Peering Traffic Ratios?”11) “Video Internet: The Next Wave….”

Internet makes anyone a publisher, similar effect now emerging for video

On the Internet,Everyone if a Publisher

Massive Disruption in U.S. Peering Ecosystem Short Videos

• YouTube – founded 2005– Short video clips – 50 million view per day!– 20Gbps of peering traffic Feb 2006– $1M/month in Sept 2006!– Entering Peering Ecosystem– 30 Other competitors 600Gbps peerable?

• DoveTail• Video may dwarf current peered traffic

– 2010 – 80-90% Internet is Video– Inculcate video guys into peering ecosystem

Now, On the Internet Everyone is a Broadcaster

Short video clips…Full TV shows…

Source: http://www.nanog.org/mtg-0606/norton.htmlSource: http://digg.com/tech_news/YouTube_Gets_Bandwidth_Boost_from_Level_3

Massive Disruption in U.S. Peering Ecosystem Full Episodes

• “Desperate Housewives” – 210MB/hour– For 320x240 H.264 Video iTunes image

• 10,000,000 households• 2,100,000,000 MB = 2.1 peta-Bytes• How long will that take to download?

Source: http://www.pbs.org/cringely/pulpit/pulpit20060302.html

3 days @ 64Gbps non-stop !Just one showTry 250M*180 Channels*HDTV

The Point: Massive Wave of Incremental Traffic to document…

The Research Questions

How to distribute video across the Internet ?How much does it cost per video?

Models A:10 videos B: 100 C: 1000

1: Transit Model 1A Model 1B Model 1C

2: CDN Model 2A Model 2B Model 2C

3: Hybrid Model 3A Model 3B Model 3C

4: P2P Model 4A Model 4B Model 4C

Modeling

$ per video?

Transit =Metered pipe to the Internet

CDN =Content Distribution Network

Peering =free & reciprocal access to each others

CustomersP2P =PeerToPeer

Shift from Avg to more typical demand curve...

Varying Sized Loads

Small =Distribute 10 videos every 5 minutes on avg.

Medium =Distribute 100 videos every 5 minutes on avg.

Large =Distribute 1000 videos every 5 minutes on avg.

Demand Curve Modeling

α = 10 videos/5 min=400Mbps ω =1.6Gbps ρ = 2.64Gbpsα = 100 videos/5 min=4,000Mbps ω =16Gbps ρ = 26.4Gbpsα = 1000 videos/5 min=40,000Mbps ω =160Gbps ρ = 264Gbps

ρ=6.6xα

95th % ω~ 4 x α

α = average loadρ = peak load (6.6*α)ω =95th percentile (4*α)

Model 1C – Large Load Commodity Transit

Server1 GigE Switch1

Upstream ISPs

:

Server24

8 * 10GE to upstreams each

::

Router1

10G

Server262

Server263

Server264

:

:

GigE Switch14

:

Router2Router2

Router4

10G

Average Load 40,000 Mbps95th Percentile Load 4 160,000 MbpsPeak Load 6.6 264,000 MbpsModel 1C - Simple Commodity Transit for Video Distribution

monthlyTransit Fee 160,000 mbps@ $10 perMbps $1,600,000Colo 14 rack@ $2,000 perRack $28,000Network Equip 4 6509 $150,000 3yrAmort $16,667AggregationSw 14 $10,000 3yrAmort $3,889Servers 264 $4,000 3yrAmort $29,333Maintenance 15% $7,483Staff 3 $180,000 $45,000Total $1,730,372

# videos downloaded per month 8,640,000Cost per video downloaded $0.20

Model 2C: CDN Large LoadAverage Load 40,000 Mbps95th Percentile Load 4 160,000 MbpsPeak Load 6.6 264,000 MbpsModel 2C - Content Delivery Network for Video Distribution

monthlyTransit Fee 160,000 mbps@ $13 perMbps $2,080,000Colo 1 rack@ $1,500 perRack $1,500Network Equip 1 6503 $30,000 3yrAmort $833Servers 1 $4,000 3yrAmort $111Maintenance 15% $367Staff 0.5 $180,000 $7,500Total $2,090,311

# videos downloaded per month 8,640,000Cost per video downloaded $0.24

Model 3C: Transit/Peering Large LoadAverage Load 40,000 Mbps95th Percentile Load 4 160,000 MbpsPeak Load 6.6 264,000 MbpsModel 3C - Blended Transit and Peering for Video distribution3 site 25% peering monthlyTransit Fee 120,000 mbps@ $10 perMbps $1,200,000Colo 42 rack@ $2,000 rack+port $84,000Network Equip 12 6509 $150,000 3yrAmort $50,000AggregationSw 42 $10,000 3yrAmort $3,889Servers 792 $4,000 3yrAmort $88,000Maintenance 15% $21,283Staff 3 $180,000 $45,000Total $1,492,172

# videos downloaded per month 8,640,000Cost per video downloaded $0.17

Model 4C: P2P Large LoadAverage Load Mbps Mbps95th Percentile Load 4 160,000 MbpsPeak Load 6.6 264,000 MbpsModel 4C - Peer-to-Peer Network for Video Distributionsingle-site stormcasting monthlyTransit Fee 100 mbps@ $50 perMbps $5,000Colo 1 rack@ $1,500 perRack $1,500Network Equip 1 6503 $30,000 3yrAmort $833Servers 1 $4,000 3yrAmort $111Maintenance 15% $367Staff 0.5 $180,000 $7,500Total $15,311

# videos downloaded per month 8,640,000Cost per video downloaded $0.0018

SummaryInternet Video Distribution Methods

$0.00$0.10$0.20$0.30$0.40$0.50$0.60$0.70$0.80$0.90

A B C

Load Models

$/Vi

deo

Dow

nloa

d

Model 1:Transit

Model2: CDN

Model3: Transit/Peering

Model 4: P2P

Models A:10 videos B: 100 C: 1000

1: Transit 1A: $0.60 1B: $0.36 1C: $0.20

2: CDN 2A: $0.77 2B: $0.44 2C: $0.24

3: Hybrid 3A: $0.69 3B: $0.31 3C: $0.17

4: P2P 4A:$0.18 4B: $0.0177

4C: $0.0018

Per Video Cost Of delivery

Observations• Internet Transit Supply ▼Price ▲• Internet Transit Model src/dst specific• Bottlenecks

– IX Power, Router Capacity, Peer’s Capacity,– Backbone Capacity, Last Mile bottleneck, 100G NIC?– Do I need to upgrade $$$$ gear to support my

competitor (peer)?• Identify Players, Positions, Motivations, Behavior• Geoff Huston: “P2P has won. Telco/Cable co

trying to keep its 1998 biz plan relevant.”

Predictions from the Field

Content Providers: "We are sitting around this table in 2010 and we are commenting how remarkable the last few years have been, specifically that:“

1. Video streaming volume has grown 100 fold2. Last mile wireless replaced local loop3. Botnets (DDOS attacks) are still an issue4. Non-mechanical (i.e. Flash) Drives replaced internal hard drives on laptops5. 10% of all cell phones are now video phones6. We have cell phones that we actually like7. The U.S. is insignificant traffic wise relative to the rest of the world8. Most popular question discussed around the table: 'How do we operate business

in China?‘9. No online privacy. And the gov't watches everything10.18-25 demographic is best reached w/ads on the Internet11.Next Gen 3D on-line Social Networks are so successful12.No physical network interfaces are needed13.We will big brother ourselves (video cams 'who scraped my car?')14.So many special purpose Internet apps - in car google maps, livetraffic updates,

etc.15.So much of our personal information is on the net16.Video IM emerged as a dominant app17.P2P will emerge for non-pirated videos - DRM in place and embraced18.Voice calls are free, bundled with other things

Source: May 2006 Web Content Private Forum

“ISPs: We are sitting around this table in 2010 at NANOG and we are commenting how remarkable the last few years have been, specifically that:“

1. We have 10G network interface(s) on laptops (I assumed wired, but someone else might have been thinking wireless)

2. $5/mbps is the common/standard price of transit (other prediction was $30/mbps)3. Internet traffic is now so heavily localized (as in 75% of telephone calls are across town type of thing but for

the Internet)4. Ad revenue will cover the cost/or subsidize significantly of DSL5. 90% of Internet bits will be video traffic6. VoIP traffic exceeds the PSTN traffic7. Private networks predominantly migrate to overlays over the Internet8. Wireless Internet Service Providers (WISPs) are serious competitive threat to DSL and Cable Internet9. Sprint is bought by Time Warner10. Cable companies form cabal & hookup with Sprint or Level 311. Government passes Net Neutrality Law of some flavor12. Earthlink successfully reinvents themselves as Wireless Metro player in Response to ATT and Verizon13. 40% paid or subscription as opposed to Content Click Ads. Like Cable Company channel packages, folks will

flock to subscriptions for Internet Content packages.14. RIAA proposes surcharge on network access (like Canada tax on blank CDs)15. NetFlix conversion to Internet delivery of movies to Tivo or PC,or open source set top box16. ISPs will be in pain17. Last mile (fiber, wireless, .) in metro will be funded by municipal bonds18. Death of TV ads, Death of broadcast TV, Tivo & Tivo like appliances all use the Internet with emergence of

targeted ads based ondemographic profiles of viewer19. Google in charge of 20% of ALL ads (TV, Radio, Billboards, .)20. Ubiquitous wifi in every metro with wifi roaming agreements21. Congestion issues drive selective customer acceptance of partial transit offerings22. IPTV fully embraced by cable cos - VOD - no need for VDR and ala carte video services replace analog

frequency23. Near simultaneous release of movies to the theaters, DVDs for the home, PPV, and Internet download to

meet needs of different demographics.

Source: Discussions at NANOG 37 San Jose