ospf to isisvijay/work/ppt/oi.pdf · 2003. 10. 26. · vijay gill jon mitchell [email protected]...
TRANSCRIPT
Notes
"But in our enthusiasm, we could notresist a radical overhaul of thesystem, in which all of its majorweaknesses have been exposed,analyzed, and replaced with newweaknesses."
-Bruce Leverett
Why
• Features– Convergence
• Security
• Simplicitybut to learn ISIS you have to know the secret
handshake and be a *%##%ing 33rd levelmason
-Chance Whaley
Security
• http://www.nanog.org/mtg-0006/katz.html
• Packet bombs
• Wasn’t as big of a deal for AOL– We have packet filters on most line cards
• Most is not ALL
• Runs directly on L2– Harder to spoof or attack
Simplicity
• Found out we didn’t need areas– Added complexity
• Configuration
• Typos
• Slowed it down– DV
– Flat area easy to configure and maintain• Stupid, but no stupider (apologies to Einstein)
State of the Art
POP3
BB1 BB2
L0: x.y.z.n
N.N
.N.m
/31 N.N .N.o/31
OSPF AREA 0
OSPF AREA X
ATDN OSPF
POP1
P6/0
P4/0/0
P6/1
P5/0/0
P2/0P2/0
POP2
P1/0P0/0 P1/0P0/0
P0/0 P1/0 P6/0P0/0 P1/0
L0: x.y.z.m
L0: A.B.C.DBlock: X.Y.A.B/28
P6/2to
bb2-ZZZ
to bb2 -XX X
BB Sample Config
router ospf 1log-adjacency-changesarea 0 authenticationarea x authentication
passive-interface Loopback0network A.B.C.0 0.0.3.255 area 0network A.B.D.0 0.0.1.255 area Xnetwork A.B.C.0 0.0.7.255 area X
maximum-paths 6
area X range A.B.C.x 255.255.255.240area X range A.B.C.y 255.255.255.240
Note: Area X is the BGP cluster-ID ofthe site
POP Sample Config
router ospf 1log-adjacency-changesarea X authenticationredistribute connected subnetspassive-interface Loopback0network A.B.C.0 0.0.1.255 area X….Maximum-paths 6
X. Y.A.B/31
N.N.M.Y/31P6/2
A.B.C
.E/3
1to p
op1-YYY
P4/0
A. B.C
.D/30
L0: A.B.C.EBlock: X.Y.A.C/28
L0: A.B.C.FBlock: X.Y.A.D/28
Strategy
Ships in night– Run parallel– Verify routes
• Raise OSPF admin distance• Verify network after change• Remove OSPF
The plan is in the works, but we have not activated the implementation phase.-Frank Caddeo
Main Backbone Nodes
bb2-mtc
bb2-dtc
bb2-dcl
bb2-nyc
bb1-dtc
bb1-frabb1-nye
bb2-spo
bb1-nyc
bb1-mtc
bb1-dcl
bb1-frr
bb2-frabb2-nye
bb1-spo
bb2-frr
bb1-loh
bb2-loh
bb1-tkn
bb2-sun
America OnlineInternet Operations
12100 Sunrise Valley Drive, Reston, VA 20191
Date:
Revision:
Drawn:
October 17, 2003
6.2
tdo
Architect:
AOL Proprietary and Confidential
Updated: tdo
bb2-sje
bb2-ash
bb1-ash
bb1-den bb2-den
bb1-new
bb2-new
bb1-alb
bb2-alb
bb1-hon
bb2-hon
bb2-seabb1-sea
bb1-kcybb2-kcy bb1-ch1
bb2-chi
bb1-sun
bb2-ntc
bb1-ntc
bb1-sje
bb2-tkn
bb2-las
bb1-las
bb2-phobb1-pho
bb1-col
bb2-col
bb1-hou
bb2-hou
bb1-tbybb2-tby
bb1-atm
bb2-atm
bb2-cha
bb1-cha
bb2-vie
bb1-vie
bb2-rtc
bb1-rtc
2x 48
2x48
2x 48
2x48
bb1-rtl
bb2-rtl
bb1-prs
bb2-prs
bb1-cin bb2-cin
bb1-sjg
bb2-sjg
bb2-dls
bb1-dls
Out of Band
“OOB is the saving throw when you @#$%up”
-RS
• Verified OOB reachability to all POPsbeforehand
IS-IS Migration Prep
Pre-Migration– Load IS-IS configuration built with scripts on RTL
routers• Non Customer PoP
– Develop/test scripts to check IS-IS neighborrelationships and route consistency
Migration Week
• Load IS-IS configuration
• Verify IS-IS neighbor relationships
• Verify LSPs in IS-IS database
• Change OSPF administrative distance to 254
– On some edge routers
Some mornings, it's just not worth chewing through the leather straps.-Emo Phillips
Migration Week (cont)
• Compare IS-IS and OSPF routes on pair of pop routers
• IS-IS vs. OSPF cost check on all interfaces in network
• Change OSPF administrative distance of all remainingrouters to 254
• Verify no OSPF routes in fowarding table
• Basic network reachability
– Ping all routers
– Check connectivity to some external sites• Standard NOC monitoring
Your rules are really beginning to annoy me-Snake Plissken
Post-Migration
• No verification– Verification done as part of migration
• Run a script to remove the OSPF configurationfrom all ATDN routers
We had more than enough genuine headaches as it was, and trivial aestheticconcerns weren't even close to making it onto our agenda.
-Geoff Miller
Current Setup
S1/0/0:0 S1/1/0:0
POP2
BB1 BB2
To bb2-den P7/0
to bb1-chi p6/0
POP1 POP3
P0/0
P0/0
P0/0P0/1 P0/1
P1/0P1/0
P1/0P1/0P7/0 P7/0
P0/0
P8/0P8/0
P5/0/0 P8/0/0
to bb2-dal P6/0
P6/0
CustomerAS: Blah
Low Speed CustomerAS: Blah
P3/0P3/0 P3/1 P3/1
PeerAS: Blah
P0/2
503 503 503 505 503 505
1
1010
10
# IS-ISMETRIC
OC-192
OC-48
OC-12
OC-3
DS1
GSR 12410GSR 12410
GSR 12410 GSR 12410 7513
Config Bits• !• interface Loopback0• isis metric 1 level-2• !• interface POS5/0• description P5/0: bb1-nye-P5-0-pop1-nye-P5-0 (66.p.x.y/31 direct-cabled)(T=pbNYE)• ip router isis• isis metric 503 level-2• isis password ISISPASSWORD(hint, this isn’t the real password) level-2• !• router isis• passive-interface Loopback0• maximum-paths 6• net 39.752f.0100.0014.0000.5000.1668.router.id.inIPv4.00
• is-type level-2-only !Why Level 2?• domain-password this-isn’t-the-real-password-either
• metric-style wide !• external overload signalling ! Ensure that IS-IS will tear down
adjacencies when dCEF is disabled on an interface
• set-overload-bit on-startup wait-for-bgp ! Avoid placingrouter on IGP SPF before bgp
• max-lsp-lifetime 65535• lsp-refresh-interval 65000
• no hello padding ! Hello padding to mtu is deprecated• log-adjacency-changes all• !
Design
• All connected interfaces are redistributed intoBGP
• IS-IS will be preferred
• Redistribution into BGP chosen to reduce thenumber of links in the SPF– Is it an issue in practice
• Not really
cluelessness leads to flapping... flapping leads todampening... dampening leads to suffering
-RS
Cost and RR Design
• Backbone links– Used OSPF metrics– BB-POP Interconnects
• OSPF metric + 500• Avoids Inversion on BB-BB link failure• Mirrors OSPF w/ Areas behavior
– MED oscillation issue• Full mesh of POP routers• No client-to-client reflection• Cost (InterPOP) > cost differences IntraPOP
• New cost out procedure– add 10000 to the interface
Timeline
LoadISIS config
Day
Tim
e
Verify routes
SwitchDistance
ConfirmReach
RemoveOSPF
Loading ISIS Config
• Non Disruptive
• Config was loaded in a three hour window,Monday 6-9 am
• Script (OSPF) -> IS-IS
• Output was copied to each router
• No IS-IS routes in use
If you can't remember, then the claymore is pointed at you
Route Verification
• Compare IS-IS neighbor topology with OSPF– show clns neighbor
– show ip ospf neighbor
• Check IS-IS database on all routers– Ensure all other routers LSP’s installed in IS-IS
database (sh isis database)
Route Verification
• On selected edge routers– Change OSPF admin distance to 254
– Verify traffic to peers
– Compare IS-IS and the OSPF routes• All routes in the network are correctly in IS-IS?
• Go or No Go
Great ideas, in theory, should not be hampered bypesky reality
-Dys
MED
• Changing metric affects MEDs– New metric in the BGP one minute after distance
change– Ratchet down
• Does not propagate for another 10 minutes
– One Large Peer – LP• Listened to MEDs• Not enough capacity to fit all of traffic in one circuit
– All routers connected to LP• Migrated at roughly the same time• Manually cleared soft out after the metric advertisement
updated
The Big One
• Flip Admin Distance– IS-IS routes are preferred
• Current network metrics are consistent with config files?
• Slow Start– Manually change admin distance to 254 on more edge POPS
• Go No-Go?
• Script to flip the rest– From the edge to the center (with respect to ops2)
– In order - LP, europe, asia, brazil, us-pop, us-bb, and dc
• External routes in OSPF now in iBGP
Routing
• Convergence time for the installation– <1 second
• No CEF updates– Costs changed but PATHS didn’t
• All production traffic is routed to Edgerouter loopbacks (n-h-s)
• Rollback• Remove admin distance command• Pre-written script
This thing severely violated the Rule of Complexity as applied to the problem. The Rule ofComplexity states that if an answer seems too complicated to be the right answer, it is the
wrong answer.-Steve Cutchen
Removal of OSPF configuration
• After burn in– 0300 EDT
– OSPF configuration removed• Non-disruptive change
– Old OSPF configs archived via RANCID
We are jolly green giants, walking the earth with routers.-Christopher Morgan (after no router ospf 10 at MFN)
Subject: From the install fileDate: 6/25/2003To: [email protected]: John
Network Install Doc for Non-Bounce June 25, 2003General Maintenance (times noted with attribution):
c) Switching ATDN backbone from OSPF to ISIS as the igp. 0300 Expected Impact: None
Dog will hunt/vijay
Line of Truth
Traffic
Questions?
You thinking about smoking off the MPLS hookah?-Brook Bailey
There is a difference between making something foolproof andreducing the number of fools
-Bill Barns