deploying mpls traffic engineering - cisco€¦ · introduction • mpls-te was designed to move...
TRANSCRIPT
1© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Deploying MPLS Traffic Engineering
James Moffat
Consulting Systems Engineer
222© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Agenda
• Prerequisites
• Introduction
• How MPLS-TE Works
• Fast ReRoute
• Design
333© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Prerequisites
• Must know how to configure a router!
…Preferably Cisco…
• Basic knowledge of MPLS forwarding
push, pop, swap, etc.
• Some exposure to MPLS-TE helps
Not much time spent on basic configs
444© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Agenda
• Prerequisites
• Introduction
• How MPLS-TE Works
• Fast ReRoute
• Design
555© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
MPLS Is Key technology for delivery of L2& L3 services
IPServices
IPServices
ATMServices
ATMServices
IP+ATM SwitchIP+ATM Switch
PNNIPNNI MPLSMPLS
IPIP
IPServices
IPServices
OpticalServicesOptical
Services
IP+Optical SwitchIP+Optical Switch
O-UNIO-UNI MPLSMPLS
IPIP
FrameRelay
ATM
FrameRelay
IP+ATM Integration
MPLS VPNs: Scalable Network based VPNs
Traffic Engineering: Optimization forAdditional traffic =>$$
Protection solutionReduction in CAPEX & OPEX
IP+Optical Integration
Layer 2 Integration forA single convergedNetwork Infrastructure
666© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Introduction
• MPLS-TE was designed to move traffic along a path other than the IGP shortest path
Bring ATM/FR traffic engineering abilities to an IP networkAvoid full IGP mesh and O(N2) floodingBandwidth-aware connection setup
• Fast ReRoute (FRR) is emerging as another application of MPLS-TE
O(msec) of packet loss when a link goes downReplace expensive SONET gear with routersCan be used in conjunction with MPLS-TE for primary paths, can also be used standalone
• Diffserv Aware Traffic EngineeringDelivering strict QOS guarantees through the integration of MPLS-TE and advanced QOS techniques
777© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Router F
The Problem with Shortest-Path
• Changing to A->C->D->E won’t help
Router C Router D
Router G
80Mb Traffic
80Mb Traffic
35Mb Drops!
35Mb Drops!Router A
Router B
NodeNode Next-HopNext-Hop CostCostBB 1010BB
FF 3030BB
CC 1010CCDD 2020CCEE 2020BB
GG 3030BB
OC-3
OC-3
DS3
DS3
DS3OC-3
OC-3
• Some links are DS3, some are OC-3
• Router A has 40Mb of traffic for Route F, 40Mb of traffic for Router G
• Massive (44%) packet loss at Router B->Router E!
Router E
888© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Router F
40Mb40Mb
What MPLS-TE Address
• Router A sees all links
• Router A computes paths on properties other than just shortest cost
• No link oversubscribed!
OC-3
OC-3
DS3
DS3
DS3OC-3
Router C
Router E
Router D
Router G
Router A
Router B
40Mb40Mb
NodeNode Next-HopNext-Hop CostCostBB 1010BB
F 30Tunnel 0
CC 1010CCDD 2020CCEE 2020BB
G 30Tunnel 1
OC-3
999© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Fast ReRoute
• FRR: A mechanism to minimize packet loss during a failure
• Pre-provision protection tunnels that carry traffic when a protected resource (link/node) goes down
• Use MPLS-TE to signal the FRR protection tunnels, taking advantage of the fact that MPLS-TE traffic doesn’t have to follow the IGP shortest path
• Can protect MPLS traffic or IP traffic, depends on the type of protection
• See later slides on FRR
101010© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Agenda
• Prerequisites
• Introduction
• How MPLS-TE Works
• Fast ReRoute
• Design
111111© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
How MPLS-TE Works
• Information distribution
• Path calculation
• Path setup
• Forwarding traffic down tunnels
121212© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Information Distribution
• Need to tell the network about per-link resources (mostly available bandwidth)
• This is done using extensions to IGP (OSPF, ISIS)
• EIGRP, RIP not supported for MPLS-TE
EIGRP, RIP will work for other MPLS applications (like VPNs!), just not for TE.
131313© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Information Distribution
• OSPFUses type 10 (opaque area—local) lSAs
See draft-katz-yeung-ospf-traffic
router ospf 1mpls traffic-eng area <x>mpls traffic-eng router-id Loopback0
141414© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Information Distribution
• IS-ISUses Type 22 TLVs
See draft-ietf-isis-traffic
router isis foompls traffic-eng level-1|level-2mpls traffic-eng router-id Loopback0metric-style wide
151515© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
• Modified Dijkstra at tunnel head-end
• Often referred to as CSPF
Constrained SPF
• …or PCALC (path calculation)
161616© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
• What if there’s more than one path that meets the minimum requirements (bandwidth, etc.)?
• PCALC algorithm:Find all paths with the lowest IGP cost
Then pick the path with the highest minimum bandwidth along the path
Then pick the path with the lowest hop count (not IGP cost, but hop count)
Then just pick one path at random
171717© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
“What’s the Shortest Path to All Routers?”
RtrA
RtrB
RtrC
RtrE
RtrD
RtrF
RtrG
• Normal SPF—Find shortest path across all links
• See Perlman (2nd ed), Moy, etc. for explanation of SPF
181818© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
RtrA
• Normal SPF—Find shortest path across all links
• See Perlman (2nd ed), Moy, etc. for explanation of SPF
“What’s the Shortest Path to All Routers?”
191919© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
• Normal SPF—Find shortest path across all links
• See Perlman (2nd ed), Moy, etc. for explanation of SPF
“What’s the Shortest Path to All Routers?”
RtrA
RtrB
RtrC
202020© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
• Normal SPF—Find shortest path across all links
• See Perlman (2nd ed), Moy, etc. for explanation of SPF
“What’s the Shortest Path to All Routers?”
RtrA
RtrB
RtrC RtrD
212121© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
• Normal SPF—Find shortest path across all links
• See Perlman (2nd ed), Moy, etc. for explanation of SPF
“What’s the Shortest Path to All Routers?”
RtrA
RtrB
RtrC
RtrE
RtrD
222222© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
• Normal SPF—Find shortest path across all links
• See Perlman (2nd ed), Moy, etc. for explanation of SPF
“What’s the Shortest Path to All Routers?”
RtrA
RtrB
RtrC
RtrE
RtrD
RtrF
RtrG
232323© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
• Normal SPF—Find shortest path across all links
• See Perlman (2nd ed), Moy, etc. for explanation of SPF
“What’s the Shortest Path to All Routers?”
RtrA
RtrB
RtrC
RtrE
RtrD
RtrF
RtrG
242424© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
• Normal SPF—Find shortest path across all links
• See Perlman (2nd ed), Moy, etc. for explanation of SPF
“What’s the Shortest Path to All Routers?”
RtrA
RtrB
RtrC
RtrE
RtrD
RtrF
RtrG
252525© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
“What’s the Shortest Path to All Routers?”
• Normal SPF—Find shortest path across all links
• See Perlman (2nd ed), Moy, etc. for explanation of SPF
RtrA
RtrB
RtrC
RtrE
RtrD
RtrF
RtrG
262626© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
• Constrained SPF—Find shortest path to a specific node
• Consider more than just link cost!
OC3
OC3
DS3
DS3
DS3
OC3
OC3
“What’s the Shortest Path to Router F with 40Mb Available?”
RtrA
RtrB
RtrC
RtrE
RtrD
RtrF
RtrG
272727© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
RtrA
• Constrained SPF—Find shortest path to a specific node
• Consider more than just link cost!
“What’s the Shortest Path to Router F with 40Mb Available?”
282828© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
OC3
OC3
• Constrained SPF—Find shortest path to a specific node
• Consider more than just link cost!
“What’s the Shortest Path to Router F with 40Mb Available?”
RtrA
RtrB
RtrC
292929© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
OC3
OC3
DS3
• Constrained SPF—Find shortest path to a specific node
• Consider more than just link cost!
“What’s the Shortest Path to Router F with 40Mb Available?”
RtrA
RtrB
RtrC RtrD
303030© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
OC3
OC3
DS3
DS3
• Constrained SPF—Find shortest path to a specific node
• Consider more than just link cost!
“What’s the Shortest Path to Router F with 40Mb Available?”
RtrA
RtrB
RtrC
RtrE
RtrD
313131© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
OC3
OC3
DS3
DS3
OC3
OC3
• Constrained SPF—Find shortest path to a specific node
• Consider more than just link cost!
“What’s the Shortest Path to Router F with 40Mb Available?”
RtrA
RtrB
RtrC
RtrE
RtrD
RtrF
RtrG
323232© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
RtrA
RtrB
RtrC
RtrE
RtrD
RtrF
RtrG
OC3
OC3
DS3
DS3
OC3
OC3
• Constrained SPF—Find shortest path to a specific node
• Consider more than just link cost!
“What’s the Shortest Path to Router F with 40Mb Available?”
333333© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
OC3
DS3OC3
• Constrained SPF—Find shortest path to a specific node
• Consider more than just link cost!
“What’s the Shortest Path to Router F with 40Mb Available?”
RtrA
RtrB
RtrE
RtrF
343434© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
• “But wait! There’s nothing different between the two SPF results!”
• …but…
353535© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
• What about the2nd path?
• Available bandwidth has changed!
OC3
5MB
DS3
DS3OC3
115MB115MB
“What’s the Shortest Path to Router G with 40Mb Available?”
RtrA
RtrB
RtrC
RtrE
RtrD
RtrF
RtrG
363636© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
RtrA
• What about the2nd path?
• Available bandwidth has changed!
“What’s the Shortest Path to Router G with 40Mb Available?”
373737© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
OC3
115MB
• What about the2nd path?
• Available bandwidth has changed!
“What’s the Shortest Path to Router G with 40Mb Available?”
RtrA
RtrB
RtrC
383838© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
OC3
DS3
115MB
• What about the2nd path?
• Available bandwidth has changed!
“What’s the Shortest Path to Router G with 40Mb Available?”
RtrA
RtrB
RtrC RtrD
393939© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
OC3
5MB
DS3
115MB
• What about the2nd path?
• Available bandwidth has changed!
“What’s the Shortest Path to Router G with 40Mb Available?”
RtrA
RtrB
RtrC
RtrE
RtrD
404040© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
OC3
5MB
DS3
115MB
• What about the2nd path?
• Available bandwidth has changed!
“What’s the Shortest Path to Router G with 40Mb Available?”
RtrA
RtrB
RtrC
RtrE
RtrD
414141© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
OC3
DS3
115MB
• What about the2nd path?
• Available bandwidth has changed!
“What’s the Shortest Path to Router G with 40Mb Available?”
RtrA
RtrB
RtrC RtrD
424242© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
OC3
DS3
• What about the2nd path?
• Available bandwidth has changed!
“What’s the Shortest Path to Router G with 40Mb Available?”
RtrA
RtrC RtrD
434343© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
OC3
DS3
DS3
• What about the2nd path?
• Available bandwidth has changed!
“What’s the Shortest Path to Router G with 40Mb Available?”
RtrA
RtrC
RtrE
RtrD
444444© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
OC3
DS3
DS3OC3
115MB
• What about the2nd path?
• Available bandwidth has changed!
“What’s the Shortest Path to Router G with 40Mb Available?”
RtrA
RtrC
RtrE
RtrD
RtrF
RtrG
454545© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
OC3
DS3
DS3OC3
• What about the2nd path?
• Available bandwidth has changed!
“What’s the Shortest Path to Router G with 40Mb Available?”
RtrA
RtrC
RtrE
RtrD
RtrG
464646© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
• End result:Bandwidth used efficiently!
30Tunnel1G
30Tunnel0F
20BE
20CD
10CC
10BB
CostNext-HopNode
OC3
OC3
DS3
DS3
DS3
OC3
OC3
RtrA
RtrB
RtrC
RtrE
RtrD
RtrF
RtrG
474747© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
• What if there’s more than one path that meets the minimum requirements (bandwidth, etc.)?
• PCALC algorithm:Find all paths with the lowest IGP costThen pick the path with the highest minimum bandwidth along the pathThen pick the path with the lowest hop count (not IGP cost, but hop count)Then just pick one path at “random” (take the top path on the TENT list)
484848© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
All Left-side LinksAre {10,100M}
All Right-side LinksAre {5,150M}
{cost,available BW}
RtrA RtrZ
{8,90M}
{8,90M}
{4,90M}
{10,100M}
{8,80M}
What’s the BestPath from A to Z with BW of 20M?
Path Has Cost of 25, Not the
Lowest Cost!
494949© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
{cost,available BW}
RtrA RtrZ
{8,90M}
{8,90M}
{4,90M}
{8,80M}
Path Min BW Is Lower than the
Other Paths!
All Left-side LinksAre {10,100M}
All Right-side LinksAre {5,150M}
What’s the BestPath from A to Z with BW of 20M?
505050© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
{cost,available BW}
RtrA RtrZ
{8,90M}
{8,90M}
{4,90M}
Hop Count Is 5, Other Paths
Are 4!
All Left-side LinksAre {10,100M}
All Right-side LinksAre {5,150M}
What’s the BestPath from A to Z with BW of 20M?
515151© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
{cost,available BW}
RtrA RtrZ
{8,90M}
Pick a Path at Random!
{8,90M}All Left-side LinksAre {10,100M}
All Right-side LinksAre {5,150M}
What’s the BestPath from A to Z with BW of 20M?
525252© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Calculation
{cost,available BW}
RtrA RtrZ
{8,90M}
All Left-side LinksAre {10,100M}
All Right-side LinksAre {5,150M}
What’s the BestPath from A to Z with BW of 20M?
535353© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Setup
• Cisco MPLS-TE uses RSVP• RFC2205 (base RSVP), RFC 3209 (TE extensions
for RSVP)• No CR-LDP; no plans for CR-LDP
• Once the path is calculated, it is handed to RSVP• RSVP uses PATH and RESV messages to request
an LSP along the calculated path
545454© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Router F
Path Setup
• PATH message: “Can I have 40Mb along this path?”
• RESV message: “Yes, and here’s the label to use”
• LFIB is set up along each hop
Router B
Router C
Router E
Router D
Router G
Router A
= PATH Messages= RESV Messages
555555© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Router F
Path Setup
• PATH message: “Can I have 40Mb along this path?”
• RESV message: “Yes, and here’s the label to use”
• LFIB is set up along each hop
Router B
Router C
Router E
Router D
Router G
Router A
= PATH Messages= RESV Messages
IMPLICITNULL23
39
15
565656© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Setup
• Errors along the way will trigger RSVP errors
• May also trigger re-flooding of TE information if appropriate
575757© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Forwarding Traffic Down a Tunnel
• There are four ways traffic can be forwarded down a TE tunnel
Static routes
Policy routing
Auto-route
Forwarding-adjacency
• With all but PBR, MPLS-TE gets you unequal cost load balancing
585858© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Static Routing
RtrA(config)#ip route H.H.H.H 255.255.255.255 Tunnel1
Router FRouter H
Router B
Router C
Router E
Router D
Router G
Router A
Router 1
Tunnel1
595959© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Tunnel1
Static Routing
• Router H is known via the tunnel
• Router G is not routed to over the tunnel, even though it’s the tunnel tail!
Router FRouter H
Router B
Router C
Router E
Router D
Router G
Router A
Router 1
NodeNode Next-HopNext-Hop CostCostBB 1010BB
FF 3030BB
CC 1010CCDD 2020CCEE 2020BB
GG 3030BBHH 4040Tunnel 1Tunnel 1II 4040BB
606060© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Policy Routing
RtrA(config-if)#ip policy route-map set-tunnel
RtrA(config)#route-map set-tunnel
RtrA(config-route-map)#match ip address 101
RtrA(config-route-map)#set interface Tunnel1
Router FRouter H
Router B
Router C
Router E
Router D
Router G
Router A
Router 1
Tunnel1
616161© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Policy Routing
• Routing table isn’t affected by policy routing
• Need (12.0(16)ST or 12.2T) or higher for ‘set interface tunnel’ to work (CSCdp54178)
Router FRouter H
Router B
Router C
Router E
Router D
Router G
Router A
Router 1
NodeNode Next-HopNext-Hop CostCostBB 1010BB
FF 3030BB
CC 1010CCDD 2020CCEE 2020BB
GG 3030BBHH 4040BBII 4040BB
Tunnel1Tunnel1
626262© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Auto-Route
• Auto-route = “Use the tunnel as a directly connected link for SPF purposes”
• This is not the CSPF (for path determination), but the regular IGP SPF (route determination)
• Behavior is intuitive, operation can be confusing
636363© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Auto-Route
This Is the Physical Topology
Router FRouter H
Router B
Router C
Router E
Router D
Router G
Router A
Router I
646464© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Auto-Route
• This is Router A’s logical topology
• By default, other routers don’t see the tunnel!
Tunnel1
Router FRouter H
Router B
Router C
Router E
Router D
Router G
Router A
Router I
656565© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Auto-Route
• Router A’s routing table, built via auto-route
• Everything “behind” the tunnel is routed via the tunnel
Tunnel1
Router FRouter H
Router B
Router C
Router E
Router D
Router G
Router A
Router I
NodeNode Next-HopNext-Hop CostCostBB 1010BB
FF 3030BB
CC 1010CCDD 2020CCEE 2020BB
GG 3030Tunnel 1Tunnel 1HH 4040Tunnel 1Tunnel 1II 4040Tunnel 1Tunnel 1
666666© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Auto-Route
• How does autoroute avoid loops?
Process SPF with physical links, replaceOIF with tunnel for tail and all subsequent neighbors
676767© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Forwarding Adjacency
• Autoroute metric change is purely local to the headend
• This makes MPLS TE different from TE with ATM
In ATM TE, the TE link (PVC) has its cost and neighbor advertised into the network
In MPLS TE, no such thing is done—Until FA
686868© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
ATM Model
• Cost of ATM links (blue) is unknown to routers• A sees two links in IGP—E->H and B->D• A can load-share between B and E
A I
E
BC
D
F GH
696969© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Before FA
• All links have cost of 10
• A’s shortest path to I is A->B->C->D->I
• A doesn’t see TE tunnels on {E,B}, alternate path never gets used!
• Changing link costs is undesirable, can have strangeadverse effects
A I
E
B C D
F GH
707070© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
F-A Advertises TE Tunnels in the IGP
• With forwarding-adjacency, A can see the TE tunnels as links
• A can then send traffic across both paths
• This is desirable in some topologies (looks just like ATM did, same methodologies can be applied)
A I
E
B C D
F GH
717171© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
F-A Issues
• In order for A to use F-A links, they need to be the best cost IGP path
Otherwise the physical topo gets used
• F-A configured withtunnel mpls traffic-eng forwarding-adjacencyisis metric <x> level-<y>
727272© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
F-A Issues
• Only ISIS supports F-A
OSPF support coming RSN
• F-A must be bidirectional
• IGP adjacency still not run over TE tunnel
• F-A cost should probably be lower than lowest possible IGP path from head to tail, otherwise it might not always get used
737373© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Unequal Cost Load Balancing
• IP routing has equal-cost load balancing, but not unequal cost*
• Unequal cost load balancing difficult to do while guaranteeing a loop-free topology
*EIGRP Has ‘Variance’, but That’s Not as Flexible
747474© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Unequal Cost Load Balancing
• Since MPLS doesn’t forward based on IP header, permanent IGP routing loops don’t happen with unequal cost
• 16 hash buckets for next-hop, shared in rough proportion to configured tunnel bandwidth or load-share value
757575© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Unequal Cost: Example 1
Router A Router E
Router F
Router G
gsr1#show ip route 192.168.1.8Routing entry for 192.168.1.8/32
Known via "isis", distance 115, metric 83, type level-2Redistributing via isisLast update from 192.168.1.8 on Tunnel0, 00:00:21 agoRouting Descriptor Blocks:* 192.168.1.8, from 192.168.1.8, via Tunnel0
Route metric is 83, traffic share count is 2192.168.1.8, from 192.168.1.8, via Tunnel1
Route metric is 83, traffic share count is 1
40MB
20MB
767676© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Unequal Cost: Example 1
Note That the Load Distribution Is 11:5—Very Close to 2:1, but Not Quite!
gsr1#sh ip cef 192.168.1.8 internal………Load distribution: 0 1 0 1 0 1 0 1 0 1 0 0 0 0 0 0 (refcount 1)Hash OK Interface Address Packets Tags imposed1 Y Tunnel0 point2point 0 {23}2 Y Tunnel1 point2point 0 {34}
………
Router A 40MB
20MBRouter G
Router E
Router F
777777© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Unequal Cost: Example 2
Q: How Does 100:10:1 Fit into a 16-Deep Hash?
gsr1#sh ip rou 192.168.1.8Routing entry for 192.168.1.8/32Known via "isis", distance 115, metric 83, type level-2Redistributing via isisLast update from 192.168.1.8 on Tunnel2, 00:00:08 agoRouting Descriptor Blocks:* 192.168.1.8, from 192.168.1.8, via Tunnel0
Route metric is 83, traffic share count is 100192.168.1.8, from 192.168.1.8, via Tunnel1Route metric is 83, traffic share count is 10
192.168.1.8, from 192.168.1.8, via Tunnel2Route metric is 83, traffic share count is 1
100MB10MB1MB
Router A
Router G
Router E
Router F
787878© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Unequal Cost: Example 2
A: Any Way It Wants to! 15:1, 14:2, 13:2:1, It Depends on the Order the Tunnels Come Up
Deployment Guideline: Don’t Use Tunnel Metrics That Don’t Reduce to 16 Buckets!
gsr1#sh ip cef 192.168.1.8 internal………Load distribution: 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 (refcount 1)
Hash OK Interface Address Packets Tags imposed1 Y Tunnel0 point2point 0 {36}2 Y Tunnel1 point2point 0 {37}
………
100MB10MB1MB
Router A
Router G
Router E
Router F
797979© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Forwarding Traffic down a Tunnel
• You can use any combination of auto-route, forwarding-adjacency, static routes, or PBR
• …but simple is better unless you have a good reason
• Recommendation: autoroute, forwarding-adjacency, or statics to BGP next-hops, depending on your needs
808080© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Agenda
• Prerequisites
• Introduction
• How MPLS-TE Works
• Fast ReRoute
• Design
818181© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Fast ReRoute
• FRR: A mechanism to minimize packet loss during a failure
• Pre-provision protection tunnels that carrytraffic when a protected resource (link/node) goes down
• Use MPLS-TE to signal the FRR protection tunnels, taking advantage of the fact thatMPLS-TE traffic doesn’t have to follow the IGP shortest path
• Can protect MPLS traffic or IP traffic, depends on the type of protection
828282© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Fast Reroute
• In an IP network, a link failure causes several seconds of outage
Link Failure DetectionLink Failure Detection
Information PropagationInformation Propagation
Route RecalculationRoute Recalculation
ThingThing
IGP Timers, NetworkSize, Collective
Router Load
IGP Timers, NetworkSize, Collective
Router Load
Media- and Platform-specific
Media- and Platform-specific ~µsecs (POS + APS)~µsecs (POS + APS)
~5–30 sec~5–30 sec
LSDB Size, CPU Load LSDB Size, CPU Load ~1–3 sec~1–3 sec
DependencyDependency TimeTime
838383© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Fast Reroute
• In an MPLS network, there’s more work to be done, so a (slightly) longer outage happens
Link Failure DetectionLink Failure Detection
Route RecalculationRoute Recalculation
ThingThing
~Usecs (POS + APS)~Usecs (POS + APS)
~5–30 sec~5–30 sec
LSDB Size, CPU Load LSDB Size, CPU Load ~1–3 sec~1–3 sec
DependencyDependency TimeTime
New LSP SetupNew LSP SetupNetwork Size,
CPU Load Network Size,
CPU Load ~5–10 sec~5–10 sec
Information PropagationInformation Propagation
IGP Timers, NetworkSize, Collective
Router Load
IGP Timers, NetworkSize, Collective
Router Load
Media- and Platform-specific
Media- and Platform-specific
848484© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Three Kinds of Fast Reroute
• Link protection
Implemented today
• Node protection
Implemented today
• Path protection
On development radar
858585© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Link Protection
• TE Tunnel A -> B -> D -> E
Router DRouter B
Router C
Router ERouter A10 34 POP
868686© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Link Protection
• B has a pre-provisioned backup tunnel to the other end of the protected link (Router D)
• B relies on the fact that D is using globallabel space
Router D
Router C
Router A Router B Router E10 34 POP
27 POP
PLR MPNHOP
NHOPBackupTunnel
ProtectedLink
878787© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Link Protection
• B -> D link fails, A -> E tunnel is encapsulated in B -> D tunnel
• Backup tunnel is used until A can re-compute tunnel path as A -> B -> C -> D -> E (10–30 seconds or so)
Router C
Router DRouter A Router B Router E10 POP
27, 34 34
888888© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Link Protection
• On tunnel head-end: tunnel mpls traffic-eng fast-reroute
• On protected link:mpls traffic-eng backup-path <backup-tunnel>
Router DRouter B Router ERouter ERouter A
898989© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Node Protection
• Router A has a tunnel A -> B -> D -> E -> F
• Router B has a protect tunnel B -> C -> E -> D
Router D Router FRouter B Router ERouter A
909090© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Node Protection
• Link protection is OK if the B -> D link goes down
• What if Router D goes away?
Router D Router FRouter B Router ERouter A
919191© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Node Protection
• Solution: Protect tunnel to the hop past the protected link
Router D Router FRouter B Router ERouter A10 34 POP
27 POP
22
PLR MPNNHOP
NNHOPBackupTunnel
NHOP
ProtectedNode
929292© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Node Protection
• Node protection still has the same convergence properties as link protection
• Deciding where to place your backup tunnels is a much harder to problem to solve large-scale
• For small-scale protection, link may be better
• Cisco has developed tools to solve these hard problems for you (Tunnel Builder Pro)
939393© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Protection
• Path protection: Multiple tunnels from TE head to tail, across diverse paths
Router D Router FRouter B Router ERouter A
949494© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Protection
• Path protection: Least scalable, most resource-consuming, slowest convergence of all 3 protection schemes
• Path protection is useful in two places:
1. When you have more links than tunnels
2. When you need to protect links not using global label space
959595© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Path Protection
Head-end Switch-overto Protect LSP
Head-end Switch-overto Protect LSP
Network Size, CPU Load
Network Size, CPU Load
~Msec~Msec
Path vs. Local Protection
Local (Link/Node) Protection
Link Failure DetectionLink Failure Detection
Local Switch-over toProtect Tunnel
Local Switch-over toProtect Tunnel
ThingThing
RP-> Communication Time
RP-> Communication Time
Media- and Platform-specific
Media- and Platform-specific ~Usecs (POS + APS)~Usecs (POS + APS)
~Few msec or Less~Few msec or Less
DependencyDependency TimeTime
Link Failure DetectionLink Failure Detection
ThingThing
Media- and Platform-specific
Media- and Platform-specific ~Usecs (POS + APS)~Usecs (POS + APS)
DependencyDependency TimeTime
Information PropagationInformation Propagation
IGP Timers, NetworkSize, Collective
Router Load
IGP Timers, NetworkSize, Collective
Router Load~5–30+ sec~5–30+ sec
969696© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
APS Protection
IGP Reconvergeson New Link
IGP Reconvergeson New Link
IGP Timers,IGP Size, CPU Load, Etc.
IGP Timers,IGP Size, CPU Load, Etc.
Seconds or LessSeconds or Less
Local Protection vs. APS Protection
Local (Link/Node) Protection
Link Failure DetectionLink Failure Detection
Local Switch-over toProtect Tunnel
Local Switch-over toProtect Tunnel
ThingThing
RP-> Communication Time
RP-> Communication Time
Media- and Platform-specific
Media- and Platform-specific ~Usecs (POS + APS)~Usecs (POS + APS)
~Few msec or Less~Few msec or Less
DependencyDependency TimeTime
Link Failure DetectionLink Failure Detection
ThingThing
Media-and Platform-specific
Media-and Platform-specific ~Usecs (POS + APS)~Usecs (POS + APS)
DependencyDependency TimeTime
APS/MSP CutoverAPS/MSP Cutover Generally a Fixed TimeGenerally a Fixed Time <50ms, per spec<50ms, per spec
979797© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Agenda
• Prerequisites
• Introduction
• How MPLS-TE Works
• Fast ReRoute
• Design
989898© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Design
• Two ways to deploy MPLS-TE
As needed to clear up congestion—Tactical
Full mesh between a set of routers—Strategic
• Strategic can be online or offline path calculation
• Both methods are valid, both have their pros and cons
999999© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Tactical
• All links are OC12
• A has consistent ±700MB to send to C
• ~100MB constantly dropped!
Case Study: A Large US ISP
Router A
Router B
Router D Router E
Router C
100100100© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Tactical
• Solution: Multiple tunnels, unequal cost load sharing!
• Tunnels with bandwidth in 3:1 (12:4) ratio
• 25% of traffic sent the long way
• 75% sent the short way
• No out-of-order packet issues—CEF’s normal per-flow hashing is used!
Router A
Router B
Router D Router E
Router C
101101101© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Tactical
• From Router A’s perspective, topology is:
Router A
Router B
Router D Router E
Router C
102102102© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Tactical
• As needed—Easy, quick, but hard to track over time
• Easy to forget why a tunnel is in place
• Inter-node BW requirements may change, tunnels may be working around issues that no longer exist
• Link protection pretty straightforward, node protection harder to track
103103103© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Strategic
• Put a full mesh of TE tunnels between routers
• Initially deploy tunnels with 0 bandwidth
• Watch tunnel interface statistics, see how much bandwidth you are using between router pairs
Tunnels are interfaces—Use IF-MIB!
Make sure that Σtunnel <= Σnetwork BW
104104104© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Strategic
• Some folks deploy full mesh just to get router-to-router (pop-to-pop) traffic matrix
• Largest TE network ~80 routers full mesh (~6400 tunnels)
• As tunnel bandwidth is changed, tunnels will find the best path across your network
105105105© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Strategic
• Physical topology is:
Router A
Router B
Router D Router E
Router C
106106106© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Strategic• Logical topology is*
*Each link is actually 2 unidirectional tunnels
• Total of 20 tunnels in this networkRouter A
Router B
Router D Router E
Router C
107107107© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Strategic
• Things to remember with full mesh
N routers, N*(N-1) tunnels
Routing protocols not run over TE tunnels—Unlike an ATM/FR full mesh!
Tunnels are unidirectional—This is a good thing
…Can have different bandwidth reservations in two different directions
108108108© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Strategic
• Best practices for full mesh:
Periodically re-optimize tunnels based on need (just like an ATM network)
TE was designed to be a combination of online (router-based) and offline (NMS) calculation
Node protection more practical in a full-mesh, offline-generate TE topography
109109109© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Recommended Reading
• Traffic Engineering with MPLS
ISBN: 1-58705-031-5
110110110© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
111111111© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Scalability
• Tests were done on a GSR
• RSP4, RSP8, VXR300, VXR400 will be similar
• 10,000 tunnels come up in 3-5 minutes
How Many Tunnels on a Router?
Number of Head-End
Tunnels
Number of Head-End
Tunnels
Number ofTunnel TailsNumber of
Tunnel TailsNumber
of Mid-PointsNumber
of Mid-PointsCodeCode
12.0ST12.0ST 600600 10,00010,000 5,0005,000
112112112© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Scalability
• Largest TE network today = 80 routers, ~6400 tunnels full mesh
• 12.0ST—600 head-ends, 360,000 tunnels full mesh with 10,000 tunnels per midpoint
• 600 = 80*7.50
Or (360,000=6400*56) if you’re in marketing
• Bottom line: MPLS-TE is not a gating factor in scaling most networks!
113113113© 2002, Cisco Systems, Inc. All rights reserved.
RST-2515465_05_2002_c1
Scalability
• Or just search CCO for “Scalability Enhancements for MPLS Traffic Engineering”
http://www.cisco.com/univercd/cc/td/doc/product/software/ios120/120newft/120limit/120st/120st14/scalable.htm