Abilene and Internet2 Engineering UpdateAbilene and Internet2 Engineering Update

Guy Almes <almes@internet2.edu>

Terena Networking Conference 2002

Limerick, Ireland

Guy Almes <almes@internet2.edu>

Terena Networking Conference 2002

Limerick, Ireland

6 May 2002 2

Outline

Abilene Update

Engineering Update» Multicast» IPv6» QoS» End-to-End Measurements» Transport for Bulk Data Flows

6 May 2002 3

Internet2 Engineering Objectives

Provide our universities with superlative networking:» Performance» Functionality» Understanding

Make superlative networking strategic for university research and education

6 May 2002 4

Abilene Update

Current 2.5 Gb/s Abilene Network

Plans for 10 Gb/s Upgrade

6 May 2002 5

Abilene is a Partnership

To build/operate Abilene, Internet2 partners with:

»Cisco Systems (routers, switches, and access)»Juniper Networks (routers)»Nortel Networks (SONET kit)»Qwest Communications (circuits and collocation)»Indiana University (network operations center)»Internet2 Test & Evaluation Centers (ITECs)

–North Carolina–Ohio

6 May 2002 6

Current Abilene Status

IP-over-SONET backbone (2.5 Gb/s)»53 direct connections »4 2.5 Gb/s connections »1 Gigabit Ethernet trial »23 will connect via at least 622 Mb/s by 1Q02»Number of ATM-based connections decreasing

215 participants: universities and labs»All 50 states, District of Columbia, and Puerto Rico»15 regional gigaPoPs support ~70% of participants

Expanded access»50 sponsored participants»23 state education networks (SEGPs)

6 May 2002 7

Abilene international connectivity

Transoceanic R&E bandwidths growing !?»GÉANT: 5 Gb/s between Europe and New York City

Key international exchange points:»StarTap and StarLight: Chicago (GigE)»AmPath: Miami (155 Mb/s ATM)»Pacific Wave: Seattle (GigE)»MAN LAN: New York City (GigE/10GigE planned)»CA*net3/4: Seattle, Chicago, and New York»CUDI: CENIC and Univ Texas El Paso

International transit service»Collaboration with CA*net3 and StarTap

Sacramento

Los Angeles

Washington

Abilene International PeeringSTAR TAP/Star LightAPAN/TransPAC, Ca*net3, CERN, CERnet, FASTnet, GEMnet, IUCC, KOREN/KREONET2, NORDUnet, RNP2, SURFnet, SingAREN, TAnet2

NYCMBELNET, CA*net3,

GEANT*,HEANET,

JANET, NORDUnet

Pacific WaveAARNET, APAN/TransPAC, CA*net3, TANET2

SNVAGEMNET, SINET, SingAREN, WIDE

LOSAUNINET

AMPATHREUNA, RNP2 RETINA,

ANSP, (CRNet)

OC3-OC12

El Paso (UACJ-UT El Paso)CUDI

San Diego (CALREN2)CUDI

* ARNES, CARNET, CESnet, DFN, GRNET, RENATER, RESTENA, SWITCH, HUNGARNET, GARR-B, POL-34, RCST, RedIRIS

09 March 2002

6 May 2002 9

Packetized Raw HDTV

Raw HDTV/IP: single 1.5 Gb/s UDP flowDARPA-funded project of USC/ISIe, Tektronix, and Univ Washington

»6 Jan 2002: Seattle to Washington DC via Abilene»18 hours: no packets lost, 15 resequencing episodes»End-to-end network performance (includes P/NW & MAX)

– Loss: <0.8 ppb (90% c.l.)– Reordering: 5 ppb

»Transcontinental 1-Gb/s TCP requires loss of

– <30 ppb (1.5 KB frames)– <1 ppm (9 KB jumbo)

6 May 2002 10

End-to-End Performance:‘High bandwidth is not enough’

Bulk TCP flows »Current median flow over Abilene: 1.9 Mb/s

–95th percentile: 7.0 Mb/s

6 May 2002 11

Future of Abilene

Internet2/Qwest agreement amended, and extended to Oct-06

Upgrade now underway to shift»from OC-48c (using Nortel OC-192 Sonet)»to 10-Gb/s lambda (unprotected)»x4 increase in core backbone bandwidth

6 May 2002 13

Next generation Abilene router selection

Extensive router specification and testingTests focused on next gen advanced services

»High performance TCP/IP throughput »High performance multicast»IPv6 functionality and throughput»Classification for QoS and measurement

3 router platforms tested and commercial ISPs referencedJuniper T640 platform selected

6 May 2002 14

Deployment timing

Ongoing: Backbone router procurement, detailed deployment planningJuly: Rack assembly (Indiana Univ.)Aug/Sep: New rack deployment Fall: First Wave lambdas commissionedFall meeting demonstration events

»Internet2 Fall Member Meeting (Los Angeles): late Oct.»SC2002 (Baltimore): mid Nov.

2003: Remaining lambdas commissioned

6 May 2002 15

Key next-gen Abilene emphases: I

Native IPv6»Motivations

– Resolving IPv4 address exhaustion issues

– Preserving original End-to-End Architecture model

– International collaboration

– Router and host OS capabilities

»Run native IPv6, concurrent with IPv4»Replicate Abilene’s 1999 multicast deployment strategy»Close collaboration with Internet2 IPv6 Working Group

6 May 2002 16

Key next-gen Abilene emphases: II

Network resiliency»Abilene lambdas will not be ring protected»Increasing use of videoconferencing/VoIP impose tighter restoration requirements (<100 ms)

»Options:–MPLS/TE fast reroute (initially)–IP-based IGP fast convergence (preferable)

6 May 2002 17

Key next-gen Abilene emphases: III

Deeper measurement capabilities»Significant factor in NGA rack design

– 4 dedicated servers at each nodes– Additional provisions for future servers– Local data collection to capture data at times of network

instability

»Enhance active probing – Now: Latency and jitter, loss, reachability (Surveyor)– Regular TCP/UDP throughput tests: ~1 Gbps

• Separate server for E2E performance beacon

»Enhance passive measurement– Now: SNMP (NOC) and traffic matrix/type (Netflow)– Routing (BGP and IGP)– Optical splitter taps on backbone links at select location(s)

6 May 2002 18

Abilene Observatories

Currently a sketch of a program for better support of computer science research1) Improved and accessible data archive

»Need coherent database design »Unify & correlate 4 separate data types

– SNMP, active measurement data, routing, Netflow

2) Provision for direct network measurement and experimentation

»Resources reserved for two additional servers– Power (DC), rack space (2RU), router uplink ports (GigE)

»archive/measurement/experiment

6 May 2002 19

Abilene Upgrade Summary

Backbone upgrade project underway»Partnership with Qwest extended through 2006»Juniper T640 routers selected for backbone»10-Gb/s backbone lambda deployment starts this fall

Advanced service foci»Native, high-performance IPv6»Enhanced, differentiated measurement»Network resiliency

Incremental, non-disruptive transition

6 May 2002 20

Multicast

By 1998,» Routing protocols existed» Deployment of native IP multicast quite rare» Early MBone no longer scalable» Considered key to new conferencing and streaming applications

Current native multicast support» PIM-Sparse, MBGP, and MSDP

Emphases on» Deployment and support for operations» Applications» Working to make it scalable

6 May 2002 21

6 May 2002 22

Current Multicast Emphases

Pressing ahead on Deployment»What are the current inhibitors to progress?

Applications / Content»Make it useful for your campuses»Explore the role of multicast in the future Internet

Improve Scalability»Press deployment of SSM»Explore the role of SSM

6 May 2002 23

Could SSM be Enough?

'Classic' Multicast» Group <g> has global significance» A user creates, joins, sends to g» Others can join, then send to and/or listen to g» MBGP, PIM-SM, MSDP triad

Source Specific Multicast» Group <g> has local significance» A user 's' creates, sends to <s,g>» Others can subscribe to, then listen to <s,g>» No need for MSDP (or allocation of <g> values)

6 May 2002 24

Implications of SSM

Simplify Multicast Routing / Addressing» No need for global class-D address allocation» No need for source discovery

Complicates 'few-to-few' applications» Define all the members of the application-level group» Both a burden and an opportunity

Allows better Security, Scalability

Requires new version of IGMP

6 May 2002 25

IPv6

Clarify motivation for IPv6» End-to-end transparency and global addressability» Supports application innovation, e.g., peer-to-peer

Support deployment and engineering expertise on networks, especially on campus

Anticipate need for first-class support» E.g., 10 Gb/s Abilene upgrade» E.g., Linux, Windows XP

6 May 2002 26

6 May 2002 27

Current IPv6 Emphases

IPv6 Training Workshops»About 8-10 workshops this year»First: in Los Angeles, hosted by CENIC, in February

Get some IPv6 on each campus/gigaPoPPrepare for native peering

»Abilene to gigaPoP»gigaPoP to campus»continue within campuses to key departmental LANs

Explore applications, DNS, operational stability

6 May 2002 28

QoS

Premium Service Retrospective»Inter-AS Premium Service proved too ambitious»Too great a demand on all routers to be able to police and to shape

»Too great a demand on ability of peering networks to coordinate

Implicit claims»Over-provisioning plus removal of non-congestive loss»Adaptive applications

6 May 2002 29

Non-Elevated Services

Scavenger»Less than best effort»Easy to deploy»Applications:

–Massive file transfers–Marking non-performance-sensitive applications

Alternative Best Efforts»Active area of research»Avoid gaming by users, while avoiding need for policing

6 May 2002 30

Localized Elevated Services

Nature of Congestion as a threat»Less on national/international backbones»More at hard-to-upgrade local/metro networks

Decentralized Experimentation

Coordinates with some Scavenger deployment

Active area of study by the working group

6 May 2002 31

The Current Situation

Our universities have access to an infrastructure of considerable capacity» examples of multi-hour 1.6 Gb/s flows with no loss and very

little reordering

End-to-end performance varies widely» but 40 Mb/s flows not always predictable» users don't know what their expectations should be

A well-known mismatch

6 May 2002 32

What are our Aspirations?

Candidate Answer #1:Switched 100BaseT + Well-provisioned Internet2 networking at 80 Mb/s

But user expectations and experiences vary widely

6 May 2002 33

What are our Aspirations?

Candidate Answer #2:Lower user expectations and minimize complaining phone calls

There is a certain appeal I suppose...

6 May 2002 34

What are our Aspirations?

Candidate Answer #3:Raise expectations, encourage aggressive use, deliver on performance/functionality to key constituencies.

Not the easy way, but necessary for success

6 May 2002 35

Threats toEnd to End Performance

Fiber problems» dirty fiber» dim lighting» 'not quite right' connectors

6 May 2002 36

Threats toEnd to End Performance

Fiber problems

Switches» horsepower» full vs half-duplex» head-of-line blocking

6 May 2002 37

Threats toEnd to End Performance

Fiber problems

Switches

Inadvertently stingy provisioning» mostly communication» happens also in international settings

6 May 2002 38

Threats toEnd to End Performance

Fiber problems

Switches

Inadvertently stingy provisioning

Wrong Routing» asymmetric» best use of Internet2» distance

6 May 2002 39

Threats toEnd to End Performance

Fiber problems Switches Inadvertently stingy provisioning Wrong Routing

Host issues» NIC» OS / TCP stack» CPU

6 May 2002 40

Perverse Result

'Users' think the network is congested or that the Internet2 infrastructure cannot help them

'Planners' think the network is underutilized, no further investment needed, or that users don't need high performance networks

6 May 2002 41

Transport Protocol Issues

Improved TCP Implementation»Web100 Project»SACK, Window Scaling»ECN

But, still subject to fundamental limits»Mathis et al. Theoretical result»TCP-throughput = (C x MTU) / (RTT x sqrt(loss))»Prospects for raising MTU, reducing RTT, loss

6 May 2002 42

Need to supplement TCP

Bulk Data Transfers»Absolute need to include congestion control»But include flow control in a more aggressive way

Related Ideas»Transport-level gateways?»Known-Capacity pipes vs Groping for available Capacity

www.internet2.edu