Update on Happenings in Internet2 Network Services

Progress in evolving to a next generation R&E network for US Researchers

Dale Finkelson and Matt ZekauskasNetwork Services, Internet2

April 21, 2020

[ 2 ]

Welcome:

2020 SAACCMeeting

Agenda

Intros / WelcomeNGI UpdateService Model

[ 3 ]

WHAT IS THE INTERNET2 NEXT GENERATION INFRASTRUCTURE

PROGRAM ?

The Next Generation Infrastructure Program is a full set of activities to review and update the

services, value and supporting technology of the Internet2 infrastructure portfolio (and

relationships in the larger ecosystem)

– Includes the services and service models through which the community adopts Internet2 infrastructure services

– Includes a number of infrastructure upgrade projects

– Includes new features, primarily driven by software, automation and systems virtualization to allow the infrastructure to be more readily integrated into the broader campus, regional and cloud environment around us.

[ 4 ][ 4 ]

5 Solutions = R&E Value

• Support the Data Intensive Researcher

• Support Software Driven Infrastructure

• Support Cloud for Research & Administration

• Readily Enable Ecosystem-Wide Solutions

• Reset Economics for Scale

[ 5 ]

How we think about the NGI Work Effort / Where are we now?

• Service Model

• Optical

• Packet

• Orchestration (service)

• Telemetry

Listen Define Ideate Prototype Implement & Show

Procurement & Prototyping

[ 6 ]

Cloud Services & Rapid Private Interconnect (RPI)

• Cloud Connect - national resilient footprint for: • AWS Direct Connect (Hosted Interconnect >500Mbps)

• Support for Jumbo Frames • Google Dedicated Interconnect • Microsoft Azure Express Route

• Automated Cloud Connect Portal • Provides API-based provisioning at Layer2 & Layer3

• RPI – incremental 10G and 100G ports rapidly provided at the 7 peering points

• perfSONAR support for the cloud

[ 7 ]

Internet2 Cloud Connect Stats

Cloud Provider Subscribers (Campus)

Regional Providers

AWS Direct Connect 58 17

Microsoft Express Route 70 15

Google Dedicated Interconnect

9 4

As of 1/22/20There are an additional 75 Subscribers in the Queue

[ 8 ]

Peering & Cloud Capacity Augments I2-PX Peering Capacity

2019 2020

980 Gbps 3110+ Gbps

47.4

80.3

104.

120.

626

5.1

351.

857

5.4

694.

510

99.6

1294

.9 1836

.821

08.9

R² = 0.9941

0

1,000

2,000

3,000

4,000

5,000

6,000

CY 0

8CY

09

CY 1

0CY

11

CY 1

2CY

13

CY 1

4CY

15

CY 1

6CY

17

CY 1

8CY

19

YTD

20CY

21

CY 2

2CY

23

CY 2

4

Internet2 Network Total PetaBytesCarried Per Year (Calendar Year)

Peta Bytes per yearPoly . (Peta Bytes per year)

[ 9 ]

Services Portfolio

What we have done:

• Lifted “caps” on I2PX

• Increased peering capacity

• New Cloud Connect Portal

Features

QoS / 10G support / new GUI

Layer 2 & 3 VPN’s on demand

Jumbo Frames for AWS• Beginning to Introduce I2-RUSH

esports peers

• MANRS Office Hours

What we can do:

• Improve cloud-connect workflow

Pre-provisioning vlans & OESS accounts a key enabler for rapid deployment

Better coordinate training & related materials for operators and users

• Secure Routing of our networks

and members

Finish MANRS across community

Accelerate Routing Registry Activities• More office hours, staff

development in software,

automation, telemetry, security

• Investigate a “last resort” internet

option in platform package

[ 10 ]

INFRASTRUCTURE IMPROVEMENTS

[ 11 ]

What’s new in Protocol? We’re moving towards…

Simple – Scalable – Seamless

Unified Forwarding – Traffic Engineering –

On Demand Next Hop – Failure Protection –

Service Chaining

Standards-Based (MP-BGP) control plane to

advertise MAC or IP addresses & vlan tag

Replaces/Simplified MPLS L2 Tunnels

Opportunities for some future L3 services as well

[ 12 ]

OESS / Cloud Connect Portal Features Additions

• Up to 10G for AWS, Google, Azure

• Jumbo frames for AWS

• Transition to new UI for L2 and L3 completed

• Ready for Trusted Identity integration soon.

• Allow self-provisioning of users within workgroups.

TIER

AuthN

AuthZ

Internet2

Campus

TIER

AuthN

AuthZ

Connector

Telemetry - Community Collaboration

CollectorParser

Kafka

Message buspub/sub

Portal/Dashboards

Streaming

CollectorParser

Streaming Internet2TSDS

Utah / Penn State Prototype – Others strongly encouraged to ”opt in”

[ 14 ][ 14 ]

Open Science Grid / Internet2 Collaboration expands with new 100G science data caches

New 100G backbone nodes coming online:• Kansas City (Thanks GPN!)• Sunnyvale• Chicago• New York• Houston• Amsterdam - soon (Thanks SURF!)

Leveraging the OSG HTC stack and cloud

peering adjacency to enable science.

Albuquerque

Minneapolis

Fargo

Missoula

Ogden

Boise

Reno

Eugene

Seattle

Sacramento

Portland

Sunnyvale

Tucson

San Diego

Mobile

AustinJacksonville

TallahasseeNew Orleans

Baton Rouge

San Antonio

El Paso

Cincinnati

Louisville

Nashville

Tulsa

Indianapolis

Memphis

Jackson

HoustonDallas

Cheyenne

Syracuse

Philadelphia

Raleigh

Charlotte

Las VegasLos Angeles

Salt Lake City

Buffalo

Albany Boston

New York

Atlanta

Ashburn

PittsburghToledo

DetroitCleveland

Chicago

Kansas CityDenverSt. Louis

Washington DCSV1/5

1 Wilshire

InfoMart

Hartford

& 56 Marietta

San Louis Ibsibo

Amsterdam

[ 15 ][ 15 ]

ICECUBE / Open Science Grid

Demonstrate Massive Cloud Burst for

Multi-messenger astronomy at SC’19

Internet2 assisted staging/moving 26 TB of data to providers

On demand for 2 hours: GPU’s equivalent to 90% of the capacity of the largest supercomputer in the world

3 Continents3 Cloud Providers51,000 GPU’s350 petaflops capacity26 TB moved2 Hours

Program Highlights

• Nationwide Upgrade of Optical Kit– Open Line System

– Flex Grid (>100 Ghz channels)

• Upgrading to SMF28 ULTRA Fiber – On most paths / some new huts

• New 400-800G transponders for Internet2 backbone circuits

• Substantially greener profile (space, power)– Approximately $2.75m/yr in savings resulting from “upgrade”

• Implementation underway

NGI Optical Upgrade

New Route13

Pueblo

Albuquergue

El Paso

Pueblo

Albuquergue

El Paso12

12

12

12

12

Cheyenne?

Omaha?

U2

U1

11 1111

11

San Diego

PhoenixTucson

Niland1010

10

10

Nashville

Louisville

Atlanta

Birmingham

Chattanooga

Cincinnati

9

9

9

9

9

9

9

9

San Luis Obispo

SacramentoSunnyvale

Reno

8

8

88

Tionesta

Seattle

Eugene

Portland

Olympia/Lacy

7

7

7

7 7

Jacksonville

TallahasseeBaton Rouge

Houston Hardy

New Orleans

Mobile Pensacola

6

66

6

6

Houston I-45

Austin

San Antonio

Dallas

Sanderson

Memphis

No roll pair

Jackson

Washington

5

5

555

5

5

Los Angeles

Denver

Salt Lake City

Boise

Ogden

Las Vegas

Eureka Junction

Rawlins

4

4

4

4

44

4

4

SyracuseAlbany Boston

New York 111 8th

Charlotte

Ashburn

Raleigh

Washington DC

BaltimorePhiladelphia

Fiber upgraded outside of project

Hartford

Frederick

New York 32 AoA

3

3

3

3

3

33

Toledo

Indianapolis

ChicagoDetroit

Starlight350 Cermak

600W

Buffalo

Pittsburgh

Cleveland

Peoria

2

2

2

2

22

Tulsa

Kansas CitySt. Louis

GoodlandNewton

1 1

1

Fiber Segment Endpoint

Intermediate City (no optical add/drop facilities)

1 Phase Number

Optical Installation Phase Boundary (ROADM or DGFF)or

or

ROADM

DGFF

NGI Packet Update

Packet RFP responses received

Community/Staff Evaluation underway• Protocol Support• Hardware Form Factors/Power• Security/Telemetry/etc• TCO 5/8 Years• Partnership

Testbed and / or field Testing likely

Responses Received

Also: New Secure Management Network being deployed with Optical

Simplify operations, coordination and service activities across Atlantic and Pacific Exchange Points

• Initiative of PacificWave and Internet2 to more closely align operations, capabilities and services at MANLAN, WIX and PacificWave

• Includes dedicated 100G Interconnect and shared backup paths

• Upcoming activities include:– Develop a shared portal– Upgrade perfSONAR

infrastructure

[ 18 ]

US-IX Collaboration

[ 19 ][ 19 ]

Ogden

Boise

Reno

Eugene

Seattle

Sacramento

Portland

Sunnyvale

Tucson

San Diego

Phoenix

Mobile

Austin

Jacksonville

Tallahassee

New Orleans

Baton Rouge

San Antonio

El Paso

Cincinnati

Louisville

Nashville

Tulsa

Indianapolis

Memphis

Jackson

Houston

Dallas

Cheyenne

Syracuse

Philadelphia

Raleigh

Charlotte

Las Vegas

Los Angeles

Salt Lake City

Buffalo

Albany Boston

New York

Atlanta

Ashburn

PittsburghToledo

Detroit

InfoMart

Hartford

San Louis Ibsibo

Baltimore

Providence

Missoula

Fargo

Minneapolis

Eugene

ClevelandChicago

Kansas CityDenver

St. Louis

Washington DC

Internet2 & Partners awarded

“Spirit of Innovation Award” at SC’19

… for the collaboration delivering over

2Tbps of capacity and a “first in R&E”

400G link from Chicago to Denver.

[ 20 ][ 20 ]

THANK YOU!

An enormous community effort across many

collaborations is showing progress and promise!