modern carrier strategies for te drvishalsharma 2002-11-28

8/2/2019 Modern Carrier Strategies for TE DrVishalSharma 2002-11-28

1/36

Modern Carrier Strategies for

Traffic Engineering

Dr. Vishal SharmaPrincipal ConsultantMetanoia, Inc.Voice: +1 408 394 6321Email: [email protected]: http://www.metanoia-inc.com

Metanoia, Inc.Critical Systems Thinking

Copyright 2002

All Rights Reserved


2/36


Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE

2

Basic Service Provider Goals

The two fundamentaltasks before any service provider:

Deploy a physical topology that meets customers needs

Map customer traffic flows on to the physical topology

Earlier (1990s) the mapping task was uncontrolled!

By-product of shortest-path IGP routing

Often handled by over-provisioning


3/36



3

The Early Years (< 1994-95): RoutedNetwork Topology

IP Router

Network Cloud

IP Router

IP RouterIP Router

RouterInterconnections


4/36



4

The Early Years (< 1994-95): AStacked View

IP Routers

DCS/DXC

TDM overcopper (T1/T3)

FDDI rings(100 Mb/s)

Router

MUX

SDH/SONET

-- Service creation-- Pkt. switching-- Stat muxing

-- Connectivity

-- Speed match I/Fs

-- TDM transport-- Fault isolation-- Restoration


5/36



5

TE in Carrier Networks: TraditionalRouted Core (pre 1994-95)

Prior to advent of ATM ...

... IP metricswere the only means available to control traffic

distribution through IP networks

Approach was ad-hoc

Observe traffic flow through network

Adjust weight of links with load lower/higher than desired

Overprovision network as, needed


6/36



6

Two Paths to TE in IP Networks

With increase in traffic, emergence of ATM, and higher-

speed SONET, two approaches emerged

Use a Layer 2 (ATM) network

Build ATM backbone

Deploy complete PVC mesh,

bypass use of IP metrics

TE at ATM layer

With time, evolve ATM to MPLS-

based backbone

Use only Layer 3 (IP) network

Build SONET infrastructure

Rely on SONET for resilience

Run IP directly on SONET (POS)

Use metrics (systematically) to

control flow of traffic (more on

this later)


7/36



7

Genesis of the ATM-Core

Growth in traffic needed faster backbones (> T3/45 Mb/s) Denser backbones metric manipulation impractical

IP routers lagged: offered only DS3 I/Fs & s/w forwarding

ATM emerged, was designed for WAN from start

In 1994-95 had OC-3, and later OC-12 I/Fs available

Allowed carriers to redesign their networks for high-speeds

As an evolutionary step, SPs moved to a switched ATM core


8/36



8

ATM-based Cores (mid to late 1990s)

Router

ATM

MUX

SDH/SONET

-- Service creation-- Pkt. switching-- Stat muxing-- Connectivity

-- Traffic engg.-- Hardware fwding

-- Speed match I/Fs

-- TDM transport-- Fault isolation-- Restoration


9/36



9

Physical Topology with an ATM Core

POP i+1

POP j

POP N

POP 1

POP 2

POP i

OC-12

OC-3

OC-3

Router

ATM Switch

ATM Core

OC-3 ATMSAR I/F


10/36



10

Logical Topology with an ATM Core

ATM-core (usually) fully owned by SP

Dedicated (usually) to supporting IP backbone

Utilized ATM UBR or ATM VBR-rt/CBR, depending on classes of traffic

B

A C

D

Primary PVC Mesh

Secondary PVC Mesh

PVC 1'

PVC 2'

PVC 4'

PVC 3'

PVC 5'

PVC 1

PVC 2

PVC 3

PVC 4

PVC 5B

AC

D

ATM PVC Layout L3 Logical Topology


11/36



11

Genesis of the IP-over-SONET/SDHApproach

Desire to minimize # of network layers Easier management

Simpler operation

Potentially scalable

Belief that high-speed SONET/SDH I/Fs would become

available with advances in components (vindicated with time)

Dictated partly by how (in time) a carriers network evolved


12/36



12

SONET/SDH-based Cores (mid-to-late 1990s and beyond)

Router

MUX

SDH/SONET

-- Service creation-- Pkt. switching-- Stat muxing-- TE w/ metrics or

MPLS

-- Speed match I/Fs

-- Framing

-- Fault isolation-- Restoration(moving away)

DWDM

-- B/w on existingplant


13/36



13

Physical Topology with SONET/SDH Core

POP i+1

POP j

POP N

POP 1

POP 2

POP i

OC-3/12 or

STM-1/4

Router

SONET/SDH Core

OC-3 SONET/STM-1 SDH I/F

OC-48 BLSR/MS-SPRing

OC-3/13 UPSR/SNCP Ring

Point-to-point SONET/SDH framed link

DXC


14/36



14

Ckt. 1

Ckt. 2

Ckt. 3

Ckt. 4

Ckt. 5

Logical Topology with SONET/SDH Core

SONET/SDH infrastructure (usually) owned by SP

Logical links between POP routers realized over a physical SONET/SDH

circuit going over a fiber path

Parallel logical links (physically disjoint) provisioned b/w each router pair

A

B D

C

B

A C

D

Parallel Logial Links forLoad Sharing

Each logical link provisionedfor 2X the bandwidth

SONET/SDH Circuit LayoutL3 Logical Topology


15/36



15

Global Crossing IP Backbone Network

100,000 route miles 27 countries 250 major cities5 continents200+ POPs

Courtesy: Thomas Telkamp, GBLX

M i I


16/36



16

Global Crossing (GBLX): A Bit of History

First independent global fiber network Launched operations -- March 1997

First segment turned on -- May 1998

Expanded network & svcs. by acquisitions & JVs

Frontier Telecommunications, Sept 1999

Racal Telecom, Nov 1999

Hutchison Global Crossing, Jan 2000

IXNET/IPC, June 2000

International network, worldwide reach

100,000 route miles, 27 countries, 250 major cities

195 POPs (mid 2001)

M t i I


17/36



17

Global Crossing IP Network

OC-48c/STM-16c (2.5Gbps) IP backbone Selected 10Gbps links operational (e.g. Atlantic)

Services offered Internet access & Transit services

IP VPNs -- Layer 3 and Layer 2

MPLS and DiffServ deployed globally

Edge Equipment

Core Equipment

Cisco GSR 12000/12400[12.0(17) SI]

Cisco 7500/7200 ESR, OSR

Juniper M10/20/40

M t i I


18/36



18

Global Crossing:Network Design Philosophy

Ensure there are no bottlenecksin normal state

On handling congestion

Preventvia MPLS-TE

Managevia Diffserv

Over-provisioning

Well traffic engineerednetwork can handle all traffic

Can withstand failure of even the most critical link(s)

Avoid excessive complexity & features

Makes the network unreliable/unstable

M t i I


19/36



19

Global Crossings Approach: Big Picture

WebServer

HR

DRBR

AR

CR

WR

DR

HR BR

AR

CR

WR

DR

HR BR

AR

CR

WR

EthernetSwitch

ModemBank

To other ISPs

To Customers

POP1

POP2

POP3

AR = Access RouterBR = Border Router

CR = Core Router

HR = Hosting Router

WR = WAN Router

DR = DSL Aggregation

OC-3/OC-12

OC-12/OC-48OC-48/OC-192

Metanoia Inc


20/36



20

TE in the US IP Network:Deployment Strategy

Decision to adopt MPLS for traffic engineering & VPNs Y2000: 50+ POPs, 300 routers; Y2002: 200+ POPs

Initially, hierarchical MPLS system 2 levels of LSPs

Later, a flat MPLS LSP full mesh onlybetween core routers

Started w/ 9 regions -- 10-50 LSRs/region 100-2500 LSPs/region Within regions: Routers fully-meshed

Across regions: Core routers fully-meshed

Intra-region traffic ~Mb/s to Gb/s, Inter-region traffic ~ Gb/s

Source [Xiao00]

Metanoia Inc


21/36



21

Design Principles: Statistics Collection

A

B

C

LSP1 = 15 Mb/s

LSP2 = 10 Mb/s

LSP3 = 10 Mb/s

Statistics on individual LSPs can

be used to build matrices

A

B

C

25 Mb/s

25 Mb/s

Using packets, we do not knowtraffic individually to B & C

Metanoia Inc

D i P i i l LSP C l &


22/36



22

Design Principles: LSP Control &Management

B

A

D

D

B

OC-48

OC-192

10% in usebefore new req.

New Request

A to D = 2.2 Gb/s

New LSP takes

longer path

Links utilization ismore balancedManually move traffic away from

potential congestion via ERO

B

A

D

D

B

B

A

D

D

B

OC-192

2 LSPs of 1.2Gb/s each

LSPs split acrossalternate routes

Lowered load, greaterheadroom to grow

Load splittingratio = 0.5 each

OC-48

Adding new LSPs with aconfigured load splitting ratio

Metanoia Inc

Gl b l C i C t LSP


23/36



23

Global Crossings Current LSPLayout and Traffic Routing

Region 1Region 2

Region 3

Region 4

POP1POP3

POP4

POP5POP2

Full LSP Meshin Core

Core LSP betweenWRs in POPs 1 & 5

Source

Destination

Metanoia Inc


24/36



24

Sprint (FON): A Bit of History

A century of evolution ...

1899: Brown Telephone Co., Abilene, KS

1976: United Telephone, Kansas City, MS, 3.5M customers

1984: Began building first, all-digital, US-wide, fiber-opticnetwork

1986: United + GTE merge LD subsidiaries US Sprint1992: United buys GTEs stake, renaming co. to Sprint Corp.

2002: ~$23B revenue, 23M customers, 70 countries, 80,000 employees

110,000+ route miles in the long distance (LD) network

34,000+ in US, 78,000+ in rest of the world

Transport infrastructure common to voice, ATM, & IP network

Provides considerable leverage, as well see later

Metanoia Inc


25/36



25

Sprint (FON): IP Network Timeline

'92 '93 '95 '96 '97 '98 '99 '01 '02

First IXC w/ Internet

svc. on T1 network

All DS3 IP network

DEC FDDI Gigaswitch POP

Work with Cisco for next

router for OC-3 backbone

Service via native IP bbone

OC-12 4F-BLSR deployed

Cisco GSR tested/deployed

GigaPOP bbone

GSR in POP

OC-3 WAN

Deploy OC-48

POS over DWDM

Deploy GSR12016

OC-192 TAT links

OC-192 in Europe

All bbone routers

GSR12416s

Expand to Asia,

South America

Metanoia Inc


26/36



26

SprintLinkTM IP Backbone Network

19+ countries

30+ major intl. cities5 continents(reach S. America as well)

400+ POPs

Courtesy: Jeff ChaltasSprint Public Relations

Represents connectivityonly (not to scale)

110,000+ route miles(common with Sprint LD network)

Metanoia Inc


27/36



27

SprintLinkTM IP Network

Tier-1 Internet backbone Customers: corporations, Tier-2 ISPs, univs., ...

Native IP-over-DWDM using SONET framing

4F-BLSR infrastructure (425 SONET rings in network)

Backbone US: OC-48/STM-16 (2.5 Gb/s) links

Europe: OC-192/STM-64 (10 Gb/s) links

DWDM with 8-40 s/fiber Equipment

Core: Cisco GSR 12000/12416 (bbone), 10720 metro edge router

Edge: Cisco 75xxx series

Optical: Ciena Sentry 4000, Ciena CoreDirector

Metanoia, Inc.


28/36



28

SprintLinkTM IP Design Philosophy

Large networks exhibit arch., design & engg. (ADE) non-linearities not

seen at smaller scales

Even small things can & do cause huge effects (amplification)

More simultaneous events mean greater likelihood of interaction (coupling)

Simplicity Principle: simple n/wks are easier to operate & scale

Complexity prohibits efficient scaling, driving up CAPEX andOPEX!

Confine intelligenceat edges

No state in the network core/backbone

Fastest forwarding of packets in core

Ensure packets encounter minimal queueing

Metanoia, Inc.


29/36



29

SprintLinkTM Deployment StrategyL2 failure detection triggers

switchover before L3 converges

ZA

Parallel links 50% utilizationunder normal state

1

2

3

4

SONET framing forerror detection

LineCard

LineCard

SONET

Overhead

IP Data

Metanoia, Inc.


30/36



30

SprintLinkTM Design Principles

Great value on traffic measurement & monitoring

Use it for

Design, operations, management

Dimensioning, provisioning

SLAs, pricing

Minimizing the extent of complex TE & QoS in the core

Metanoia, Inc.


31/36

eta o a, cCritical Systems Thinking


31

Sprints Approach to Monitoring

AccessRouter

AccessRouter

AccessRouter

BackboneLinks

Peering LinksProbe

BackboneRouter

Customers Customers Customers

Adapted from [Diot99]

Analysis platform located atSprint ATL, Burlingame, CA

Metanoia, Inc.


32/36

,Critical Systems Thinking


32

Sprint Approach to TE

Aim: Thoroughly understand backbone traffic dynamics

Answer questions such as:

Composition of traffic? Origin of traffic?

Between any pair of POPs

What is the traffic demand?

Volume of traffic?

Traffic patterns? (In time? In space?)

How is this demand routed?

How does one design traffic matrics optimally?


33/36

Metanoia, Inc.


34/36

,Critical Systems Thinking


34

A Peek at a Row of a Traffic Matrix

Summary of Data Collected

Adapted from [Bhattacharya02]

Distribution of aggregate accesstraffic across other POPs in the Sprintbackbone

Peer 1

Peer 2

Web 2

Web 1

ISP

To Backbone

Sprint POP under study

Metanoia, Inc.


35/36

Critical Systems Thinking


35

Applications of Traffic Matrices

Traffic engineering

Verify BGP peering

Intra-domain routing

SLA drafting

Customer reports

Metanoia, Inc.


36/36

Critical Systems Thinking

Copyright 2002

Acknowledgements

Thomas Telkamp, Global Crossing

Robert J. Rockell, Jeff Chaltas, Ananth Nagarajan, Sprint

Steve Gordon, Cable and Wireless

Jennifer Rexford, Albert Greenberg, Carsten Lund, AT&T Research

Wai-Sum Lai, AT&T

Fang Wu, NTT America

Arman Maghbouleh, Alan Gous, Cariden Technologies

Yufei Wang, VPI Systems

modern carrier strategies for te drvishalsharma 2002-11-28

Documents