The TeraPaths TeraPaths Collaboration

Presented byPresented by

Dimitrios Katramatos, BNLDimitrios Katramatos, BNL

2

Outline

Background: the TeraPaths projectBackground: the TeraPaths project Objective

View of the network

System architecture

Collaborative effortCollaborative effort The problem of being distributed…

Testing and deployment strategy

Milestones

What is the next step?What is the next step?

3

Objective

Provide QoS guarantees at the individual data flow level, Provide QoS guarantees at the individual data flow level, between actual end hosts, transparentlybetween actual end hosts, transparently Data flows have varying priority/importance

Video streams Critical data Long duration transfers

Default “best effort” network behavior treats all data flows as equal Capacity is not unlimited

Congestion causes bandwidth and latency variations Performance and service disruption problems, unpredictability

Dynamic flow-based SLAs = schedule network utilizationDynamic flow-based SLAs = schedule network utilization Regulate and classify (prioritize) traffic Select routing (if possible)

4

View of the Network

WAN

ctrl

WAN 1

WAN 2

WAN 3

TeraPaths

Domain ctrl

TeraPaths

RN

RN

TeraPaths

WAN

ctrl

WAN

ctrl

Site A

Site B

Site C

Site D

MPLS tunnelDynamic circuitDomain control

5

TeraPathsTeraPaths Web Services Architecture

Domain Controller

DSM

Web Interface

NDCNDCNDC • • •

Database

protected network

API

local

WAN controllers

• • •

Domain controllers(non-TeraPaths)

WAN serviceclients (proxies)

CLI s/w clientWeb browser

NDC database

Domain service clients (proxies)

Site controller

Site service

hardware

“virtualnetwork

engineer”

remote

6

Collaborative Effort

TeraPaths is a set of domain controllers tied together by TeraPaths is a set of domain controllers tied together by

distributed services modules (DSMs)distributed services modules (DSMs)

DSMs interoperate with WAN domain controllersDSMs interoperate with WAN domain controllers

End site controllers directly configure network devicesEnd site controllers directly configure network devices This can be a problem…

Modifying router configs on the fly using experimental software

requires trust and assurances

Thanks to our collaborators we were able to move from Thanks to our collaborators we were able to move from

proof-of-concept to prototype running currently at 3 sites: proof-of-concept to prototype running currently at 3 sites:

BNL, UMich, and BU - SLAC also participatesBNL, UMich, and BU - SLAC also participates

7

“Do no harm”

BNL testbed (2 X Cisco 6509) not part of production network, i.e. BNL testbed (2 X Cisco 6509) not part of production network, i.e.

breaking it is okbreaking it is ok Try software w/o risk to production network

Use only proven software to configure other site routersUse only proven software to configure other site routers

Bring hosts under TeraPaths control selectivelyBring hosts under TeraPaths control selectively Static part of configuration

Reduce adverse effects of bugs

SecuritySecurity “Circle of trust” for servers running TeraPaths modules

X.509 certificates, accounts, grid membership, etc.

“Virtual network engineer” web service to configure routers

There is always some residual riskThere is always some residual risk

8

Milestones

Proof of concept code demonstrated at SC’05Proof of concept code demonstrated at SC’05

Automatic invocation of OSCARS for MPLS tunnels in early 2006 Automatic invocation of OSCARS for MPLS tunnels in early 2006

Prototype code deployed at BNL and UMich, true end-to-end path Prototype code deployed at BNL and UMich, true end-to-end path

setup using MPLS tunnels through ESnet in summer 2006setup using MPLS tunnels through ESnet in summer 2006

Started looking into dynamic circuits in early 2007Started looking into dynamic circuits in early 2007

Modified code utilizes MPLS tunnels and dynamic circuitsModified code utilizes MPLS tunnels and dynamic circuits

Testbed expansion to BU through NoX and 2nd connection to UMich Testbed expansion to BU through NoX and 2nd connection to UMich

through Merit/MiLRthrough Merit/MiLR

Demo at SC’07: basic circuit utilizationDemo at SC’07: basic circuit utilization

Demo at JTW’08: traffic regulation within circuitsDemo at JTW’08: traffic regulation within circuits

9

MPLS Tunnels vs. Dynamic Circuits

WANborder routerborder router

MPLS tunnel ingress/egress

router

MPLS tunnel ingress/egress

router

WANswitch switch

border routerborder router

vs.

10

VLAN Setup for L2

TeraPaths-controlled“virtual border” router(directs flows w/PBR)

e.g.,1 to X, 2 to Y

WAN Site’sBorderRouter

trunked VLAN pass-thru50 VLAN ids (3550-3599)

3550 X Y 3599interfaces trust DSCP

TeraPaths-controlledhost router

#X

#Y

DSCP-friendly LAN

host 1 host nhost 2 . . .

1 to X

2 to

Ycan be the same device

RegionalProvider’s

Router

11

Current TeraPaths Testbed

US ATLAS T2 sites

BNL

OU

UC/IU UMich BU

SLACESnet

UTA

I2

NLR

NoX

StarLight

UltraLight

MiLR/Merit

L2 (dynamic circuit)

L3 (MPLS tunnel)

L2 and L3

JTW08 weathermap

12

Traffic Regulation (demo)

1

2

2

13

What is the next step?

Side effect of utilizing dynamic circuits: path selectionSide effect of utilizing dynamic circuits: path selection Common ground with Lambda Station

End sites need domain controllers in order to utilize dynamic circuits End sites need domain controllers in order to utilize dynamic circuits

and MPLS tunnels transparently and effectivelyand MPLS tunnels transparently and effectively

We would like to:We would like to: At the very least, interoperate with all types of domain controllers

Maybe develop a common-core domain controller for end sites,

customizable to site requirements through plug-ins?

Deploy domain controllers to all sites willing to participate All sites serviced by ESnet and Internet2 qualify

Setup a “protoduction” [?] testbed encompassing these sites

Eventually, move system into production