Advanced Communication Exchange Facilities:

Architecture, Services, and Design, Technologies,

Implementation and Operations (i.e., Creating 21st Century

Communications Services and Technology)

Joe Mambretti, Director, (j-mambretti@northwestern.edu)

International Center for Advanced Internet Research (www.icair.org)

Northwestern University

Director, Metropolitan Research and Education Network (www.mren.org)

Hong Kong Workshop on Next Generation Open Communication

Exchanges for Science Research

University of Hong Kong

January 18, 2012

Creating the Future of Communications

• Close Examination of Emerging and Future Requirements

• New Architecture

• New Services

• New Technology

• Testing With Simulations and Models

• Testing via Network Research Testbeds At Scale!

• Best Window To the Future = Data Intensive Science Research

• 30+ Year History of Communication Innovations Has Been Driven

Primarily By Data Intensive Sciences

• These Sciences Encounter Technology Issues Many Years Before

Other Domains

Sloan Digital Sky

Survey

www.sdss.org

Globus Alliance

www.globus.org

LIGO

www.ligo.org TeraGrid

www.teragrid.org

ALMA: Atacama

Large Millimeter

Array

www.alma.nrao.edu

CAMERA

metagenomics

camera.calit2.net

Comprehensive

Large-Array

Stewardship System

www.class.noaa.gov

DØ (DZero)

www-d0.fnal.gov

ISS: International

Space Station

www.nasa.gov/statio

n

IVOA:

International

Virtual

Observatory

www.ivoa.net

BIRN: Biomedical

Informatics Research

Network

www.nbirn.net

GEON: Geosciences

Network

www.geongrid.org

ANDRILL:

Antarctic

Geological

Drilling

www.andrill.org

GLEON: Global Lake

Ecological

Observatory

Network

www.gleon.org Pacific Rim

Applications and

Grid Middleware

Assembly

www.pragma-

grid.net

CineGrid

www.cinegrid.org Carbon Tracker

www.esrl.noaa.gov/

gmd/ccgg/carbontrack

er

XSEDE

www.xsede.org

LHCONE

www.lhcone.net

WLCG

lcg.web.cern.ch/LCG/publi

c/

OOI-CI

ci.oceanobservatories.org

OSG

www.opensciencegrid.org

SKA

www.skatelescope.o

rg

NG Digital

Sky Survey

ATLAS

Compilation By Maxine Brown

A Brief Overview of the Future of

Communications • Transition From Monolithic, Static Services and Infrastructure To

Ad Hoc Specialized, Customized, and Even Personalized

Resources:

– E.g., Personal Global Networks

– Individualized Communication Services

– Note: Historic Progression From Monolithic To

Individualized Environments (Part of the IT DNA)

• Personal Computer vs Mainframe

• Smart Phone vs Personal Computer

• Intelligent Devices vs Smart Phones

• Etc.

– Advanced Network Testbeds Are Being Designed And

Implemented To Accomplish These Goals, Including By The

National Science Foundation

Invisible Nodes,

Elements,

Hierarchical,

Centrally Controlled,

Fairly Static

Traditional Provider Services:

Invisible, Static Resources,

Centralized Management,

Highly Layered

Distributed Programmable Resources,

Dynamic Services,

Visible & Accessible Resources,

Integrated As Required, Non-Layered

Limited Services, Functionality,

Flexibility, Expandability

Unlimited Services, Functionality,

Flexibility, Expandability

Paradigm Shift – Ubiquitous Services Based on Large Scale

Distributed Facility vs Isolated Services Based on Separate

Component Resources

Releasing the Fully Potential of Digital Technologies

Next Phase = A Global Platform For Customized

Networks, Led By Science Networks

• Migration from Designing, Creating, Implementing,

Operating Private Networks As Individual Projects To

Designing, Creating, Implementing, Operating

Distributed Environment (Platform) Within Which Such

Individual Networks Can Be Readily Created

• Existing Building Blocks. Prototypes, Including the

StarLight National/International Communications

Exchange Facility, Metropolitan Research and

Education Network (MREN, Regional), National Lambda

Rail (NLR, National), Global Lambda Integrated Facility

(GLIF, International), GLORIAD (Global Ring for

International Application Development, International)

Distributed Programmable Network

Platforms

• Programmable Network Platforms = Capabilities for Creating

Instant New and Enhanced Services vs Legacy Multi-Year Schedule

of Design, Development, and Deployment

• Advanced =

– Dynamic vs Static

– Highly Customizable, Including At Edge

– High Level of Abstractions, Including APIs

– Flexible Middleware Processes That Can Be Dynamically Provisioned

– Highly Distributed Processes vs Centralized Command and Control

– Etc

• Programmable =

– All Resource Elements As Objects

– Discoverable/Integratable

– Programmability Extending To Hardware Components

– Rich Semantics for Resource Discovery and Integration

Ad Hoc Specialized Networks

• APN Approach: Create Private Network

– Based On Controllable Resources, e.g., Private Optical

Fiber/Lambdas/L2 VLANs

– Advanced Control Planes/Advanced Control Frameworks

– Advanced Management Planes

• Possible To Obtain Leverage From

– IaaS/NaaS

– PaaS

– SaaS

– OaaS

– XaaS

• Additional Leverage from

– Dynamic Clouds Closely Integrated With Dynamic Networks (Ref TransCloud,

Note Demo At GEC10)

Control Framework Architecture and

Implementations

• Control Frameworks Enable Resources Programming

• Many Approaches To Control Framework Architecture Currently

Being Created By Network Research Communities

• Many More Will Be Created By Research Communities

• One Approach Currently Being Transitioned From Research

Communities To A Standard Through a Joint Project Between the

GLIF and the Open Grid Forum (a Standards Organization)

• Network Services Interface (NSI)

• Based In Part On GLIF GOLE Control Frameworks and an API

(FENIUS)

t

Fenius GLIF

Demonstrations

Global Lambda Grid Workshop

SC10

HK GLIF Technical Workshop

Automated GLIF Open Lambda Exchange

Demonstration at SC11

Other Network Control Frameworks

• Many Other Network Control Frameworks Are

Being Created (About 24 Major Initiatives)

• The US National Science Foundation Has

Established an Initiative That Incorporates 4

Additional Major Frameworks

• The Initiative Is The Global Environment for

network Innovations (GENI)

National Science Foundation’s Global

Environment for Network Innovations (GENI)

• GENI Is Funded By The National Science Foundation’s Directorate

for Computer and Information Science and Engineering (CISE)

• GENI Is a Virtual Laboratory For Exploring Future Internets At

Scale.

• GENI Is Similar To Instruments Used By Other Science Disciplines,

e.g., Astronomers – Telescopes, HEP - Synchrotrons

• GENI Creates Major Opportunities To Understand, Innovate and

Transform Global Networks and Their Interactions with Society.

• GENI Is Dynamic and Adaptive.

• GENI Opens Up New Areas of Research at the Frontiers of Network

Science and Engineering, and Increases the Opportunity for

Significant Socio-Economic Impact.

• GENI Is Both

– A Research Instrument and

– A Prototype of Future Programmable Network Platforms

iGENI: The International GENI

• The iGENI Initiative Is Designing, Developing, Implementing, and Operate a Major New National and International Distributed Infrastructure.

• iGENI Is Placing the “G” in GENI Making GENI Truly Global.

• iGENI Is a Unique Distributed Infrastructure Supporting

Research and Development for Next-Generation Network

Communication Services and Technologies.

• This Infrastructure Has Been Integrated With Current and

Planned GENI Resources, and Is Being Operated for Use by

GENI Researchers Conducting Experiments That Involve

Multiple Aggregates At Multiple Sites.

• iGENI Infrastructure Connects Its Resources With Current GENI

National Backbone Transport Resources, With Current and

Planned GENI Regional Transport Resources, and With

International Research Networks and Projects.

•

iGENI/GCDnet National/International Fabric

SC11 SCinet Research Sandbox OpenFlow Demonstrations

Seattle, Wash November 2011

TransCloud Experiments Alvin AuYoung, Andy Bavier, Jessica Blaine, Jim Chen, Yvonne Coady, Paul Muller, Joe Mambretti,Chris

Matthews, Rick McGeer, Chris Pearson, Alex Snoeren, Fei Yeh, Marco Yuen

Demo: http://tcdemo.dyndns.org/

Example of working in the TransCloud [1] Build trans-continental applications spanning clouds: • Distributed query application based on Hadoop/Pig

• Store archived Network trace data using HDFS • Query data using Pig over Hadoop clusters

[2] Perform distributed query on TransCloud, which currently spans the following sites:

• HP OpenCirrus • Northwestern OpenCloud

• UC San Diego • Kaiserslautern

• Use By Outside Researchers? Yes

• Use Involving Multiple Aggregates? Yes

• Use for Research Experiments? Yes Also Ref: Experiments in High Perf Transport at GEC 7

Digital Media TransCoding Demonstration

• TransCloud: Advanced Distributed

Global Environment Enables Dynamic

Creation of Communication Services,

Including Those Based on Rapid

Migration of Virtual Network

and Cloud Resources

•TransCloud: Set of Protocols,

Standards, Management Software

Enables Interoperation of Distinct

Cloud and Network Resources

•Example: Dynamic Cloud+Dynamic

Network for Digital Media Transcoding

Using Single Platform vs Multiple

Infrastructures

Transcoding

Cloud 1 Transcoding

Cloud 2

Video

Sources

OpenFlow

Switches Transcoding

Cloud 3

JGN-X Network Topology

National Institute for Information

Communication Technology (NICT)

National International Testbed

10 Gbps Connection via

GLIF/TransLight StarLight

CNIC/Chinese Academy of Sciences

iCAIR Signing Ceremony

Another Driver For Programmable Network Platforms With

Advanced Control Frameworks

• Advanced Digital Media

• Many Services Have Requirements Similar To

Large Scale Science

• Visualization Is a Key Application.

• Currently, Many Major Innovations In Advanced

Digital Media

Programmability + High Capacity

• Advanced Programmability Provides A Wide Range of

Capabilities

• As Basic Resources, Network Resource Objects Can Be

Encapsulated and Self-Advertised

• Also, Capacity Is a Key Resource

• Currently, 10 Gbps Is Basic Resource Unit

• In 2012, 100 Gbps Will Be a Basic Programmable

Resource Unit

• 100 Gbps Not Merely As A capability For Aggregation of

Multiple Small Flows, But As a Single Programmable

Resource, e.g., For a Single Stream

StarWave: A Multi-100 Gbps Facility

• StarWave, A New Advanced Multi-100 Gbps Facility and Services

Will Be Implemented Within the StarLight International/National

Communications Exchange Facility

• StarWave Is Being Funded To Provide Services To Support Large

Scale Data Intensive Science Research Initiatives

• Facilities Components Will Include:

• An ITU G. 709 v3 Standards Based Optical Switch for WAN

Services, Supporting Multiple 100 G Connections

• An IEEE 802.3ba Standards Based Client Side Switch,

Supporting Multiple 100 G Connections, Multiple 10 G

Connections

• Multiple Other Components (e.g., Optical Fiber Interfaces,

Measurement Servers, Test Servers

DOE Office of Science ESnet ANI 100 Gbps Testbed

StarLight

Evaluations/Demonstrations of 40-

100 Gbps IPv4/IPv6

Disk-to-Disk File Transfer

Performance Across LANs and

WANs

CA*net/Ciena/StarLight/iCAIR 100 Gbps

Testbed Sept-Oct 2010, Oct 2011

Source: CANARIE

100 Gbps

~850 Miles, 1368 Km

CVS 2010/11 Terrestrial Demo Content & Applications – 100G

Chicago

Ottawa MEN LAB 10

100G Connectivity from Ciena

Ottawa to Starlight Gigapop in

Chicago.

Ability to show simultaneous

data-flows sourced from

major collaborators.

MEN LAB 10 Tile Display

Canadian Brain Imaging

Network

CBRAIN portal & Metadata

Montreal

Selected Ad Hoc Testbeds and Protoypes

• Ad Hoc Network Science Research Testbeds

• New Sevices and Technology Experiments

• Service Prototypes

• New Architecture Protypes

• New Science Network Prototypes

FILE

Ampath

GOLE USP

T-LEX

GOLE

U

Mackenzie

vlan 2712

vlan 2711 C-wave

vlans 2711/2

FILE

Phospherous

StarLight Facility Chicago

Ampath

EU Asia

GreenLight International

• GreenLight explores how researchers can take advantage of data

centers linked by high-speed networking in an era of carbon-thrifty

computing

• Partnered with Canada’s GreenStar network – researching a “follow the

sun/wind” distributed computing environment.

• Recent studies migrating virtual machines to green energy sites

indicate that 100 Gb/s networks are far superior to 10 Gb/s to make this

transparent.

10GE CAVEwave

on the National LambdaRail

UIC

NU

NCSA

UCSD

Summary

• Transition From Monolithic, Static Services and Infrastructure To

Ad Hoc Specialized, Customized, and Even Personalized

Resources:

– Individualized Communication Services

– Part of Historic Progression From Monolithic To Individualized

Environments (Part of the IT DNA)

– Design, Implementation, Operation of Distributed Network

Platforms

– Integrated With Sophisticated Control Frameworks

– Addressed via Advanced Programming Methods

– Provided With Extremely High Capacity

– Capabilities Will Be Centered At Advanced Communication

Exchanges

www.startap.net/starlight

Thanks to the NSF, DOE, DARPA

Universities, National Labs,

International Partners,

and Other Supporters