troubleshooting · • routers establish a tcp session ... incorrect configuration • peers are...
TRANSCRIPT
1NANOG 33
Troubleshooting BGPPhilip Smith <[email protected]>
NANOG 33Las Vegas
30 January – 1 February 2005
222NANOG 33
Presentation Slides
• Available onftp://ftp-eng.cisco.com
/pfs/seminars/NANOG33-BGP-Troubleshooting.pdf
333NANOG 33
Assumptions
• Presentation assumes working knowledge of BGP
Beginner and Intermediate experience of protocol
• Knowledge of Cisco CLIHopefully you can translate concepts into your own router CLI
• If in any doubt, please ask!
444NANOG 33
Agenda
• Fundamentals of Troubleshooting
• Local Configuration Problems
• Internet Reachability Problems
555NANOG 33
Fundamentals:Problem Areas
• First step is to recognise what usually causes problems
• Possible Problem Areas:Misconfiguration
Configuration errors caused by bad documentation, misunderstanding of concepts, poor communication between colleagues or departments
Human errorTypos, using wrong commands, accidents, poorly planned maintenance activities
666NANOG 33
Fundamentals:Problem Areas
• More Possible Problem Areas:“feature behaviour”
Or – “it used to do this with Release X.Y(a) but Release X.Y(b) does that”
Interoperability issuesDifferences in interpretation of RFC1771 and its developments
Those beyond your controlUpstream ISP or peers make a change which has an unforeseen impact on your network
777NANOG 33
Fundamentals:Working on Solutions
• Next step is to try and fix the problemAnd this is not about diving into network and trying random commands on random routers, just to “see what difference this makes”
• The best procedure for “unfamiliar problems” is to
Start at one place,
Deal with one symptom, and learn more about it
888NANOG 33
Fundamentals:Working on Solutions
• Remember! Troubleshooting is about:Not panicking
Creating a checklist
Working to that checklist
Starting at the bottom and working up
999NANOG 33
Fundamentals:Checklists
• This presentation will have references in the later stages to checklists
They are the best way to work to a solution
They are what many NOC staff follow when diagnosing and solving network problems
It may seem daft to start with simple tests when the problem looks complex
But quite often the apparently complex can be solved quite easily
101010NANOG 33
Fundamentals:Tools
• Use system and network logs as an aid• Record keeping:
Good and detailed system logsLast known good configurationHistory trail of working configurations and all intermediate changesRecord of commands entered on routers and other network devices
111111NANOG 33
Fundamentals:Tools
• Familiarise yourself with the router’s tools:
Is logging of the BGP process enabled?(And is it captured/recorded off the router?)
Are you familiar with the BGP debug process and commands (if available)
Check vendor documentation before switching on full BGP debugging – you might get fewer surprises
121212NANOG 33
Fundamentals:Tools
• Traffic and traffic flow measurement in the network
Unexplained change in traffic levels on an interface, a connection, a peering,…
Correlation of customer feedback on network or connectivity issues…
131313NANOG 33
Agenda
• Fundamentals
• Local Configuration Problems
• Internet Reachability Problems
141414NANOG 33
Local Configuration Problems
• Peer Establishment
• Missing Routes
• Inconsistent Route Selection
• Loops and Convergence Issues
151515NANOG 33
Peer Establishment
• Routers establish a TCP sessionPort 179 – Permit in interface filters
IP connectivity (route from IGP)
• OPEN messages are exchangedPeering addresses must match the TCP session
Local AS configuration parameters
161616NANOG 33
Common Problems
• Sessions are not establishedNo IP reachability
Incorrect configuration
• Peers are flappingLayer 2 problems
171717NANOG 33
Peer Establishment:Diagram
R2#sh run | begin ^router bgp
router bgp 1
bgp log-neighbor-changes
neighbor 1.1.1.1 remote-as 1
neighbor 3.3.3.3 remote-as 2
AS 1AS 1
AS 2
R1R1iBGPiBGP
eBGP
1.1.1.11.1.1.1 2.2.2.22.2.2.2
3.3.3.3??
?
R2R2
R3R3
181818NANOG 33
R2#show ip bgp summary
BGP router identifier 2.2.2.2, local AS number 1
BGP table version is 1, main routing table version 1
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State
1.1.1.1 4 1 0 0 0 0 0 never Active
3.3.3.3 4 2 0 0 0 0 0 never Idle
Peer Establishment:Symptoms
• Both peers are having problemsState may change between Active, Idle and Connect
191919NANOG 33
Peer Establishment
• Is the Local AS configured correctly?
• Is the remote-as assigned correctly?
• Verify with your diagram or other documentation!
R2#router bgp 1neighbor 1.1.1.1 remote-as 1neighbor 3.3.3.3 remote-as 2
Local AS
eBGP Peer
iBGP Peer
202020NANOG 33
Peer Establishment:iBGP
• Assume that IP connectivity has been checked• Check TCP to find out what connections we are accepting
R2#show tcp brief allTCB Local Address Foreign Address (state)005F2934 *.179 3.3.3.3.* LISTEN0063F3D4 *.179 1.1.1.1.* LISTEN
We Are Listening for TCP Connections for Port 179 for the Configured Peering Addresses Only!
R2#debug ip tcp transactionsTCP special event debugging is onR2#TCP: sending RST, seq 0, ack 2500483296TCP: sent RST to 4.4.4.4:26385 from 2.2.2.2:179
Remote Is Trying to Open the Session from 4.4.4.4 Address…
212121NANOG 33
Peer Establishment:iBGP
What about Us?R2#debug ip bgp BGP debugging is onR2#BGP: 1.1.1.1 open active, local address 4.4.4.5BGP: 1.1.1.1 open failed: Connection refused by remote host
We Are Trying to Open the Session from 4.4.4.5 Address…R2#sh ip route 1.1.1.1Routing entry for 1.1.1.1/32Known via "static", distance 1, metric 0 (connected)* directly connected, via Serial1
Route metric is 0, traffic share count is 1
R2#show ip interface brief | include Serial1Serial1 4.4.4.5 YES manual up up
222222NANOG 33
Peer Establishment:iBGP
• Source address is the outgoing interface towards the destination but peering in this case is using loopback interfaces!
• Force both routers to source from the correct interface
• Use “update-source” to specify the loopback when loopback peering
R2#router bgp 1neighbor 1.1.1.1 remote-as 1neighbor 1.1.1.1 update-source Loopback0neighbor 3.3.3.3 remote-as 2neighbor 3.3.3.3 update-source Loopback0
232323NANOG 33
Peer Establishment:iBGP – Summary
• Assume that IP connectivity has been checkedIncluding IGP reachability between peers
• Check TCP to find out what connections we are acceptingCheck the ports and source/destination addressesDo they match the configuration?
• Common problem:iBGP is run between loopback interfaces on router (for stability), but the configuration is missing from the router ⇒iBGP fails to establishRemember that source address is the IP address of the outgoing interface unless otherwise specified
242424NANOG 33
Peer Establishment:Diagram
• R1 is established now
• The eBGP session is still having trouble!
AS 1AS 1
AS 2
R1R1iBGPiBGP
eBGP
1.1.1.11.1.1.1 2.2.2.22.2.2.2
3.3.3.3
?
R2R2
R3R3
252525NANOG 33
R2#ping 3.3.3.3Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 3.3.3.3, timeout is 2 seconds:!!!!!Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/8 ms
Peer Establishment:eBGP
• Trying to load-balance over multiple links to the eBGP peer
• Verify IP connectivityCheck the routing table
Use ping/trace to verify two way reachability
• Routing towards destination correct, but…
262626NANOG 33
Peer Establishment:eBGP
• Use extended pings to test loopback to loopback connectivity
• R3 does not have a route to our loopback, 2.2.2.2
R2#ping ipTarget IP address: 3.3.3.3Extended commands [n]: ySource address or interface: 2.2.2.2Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 3.3.3.3, timeout is 2 seconds:.....Success rate is 0 percent (0/5)
272727NANOG 33
Peer Establishment:eBGP
R2#sh ip bgp neigh 3.3.3.3BGP neighbor is 3.3.3.3, remote AS 2, external linkBGP version 4, remote router ID 0.0.0.0BGP state = IdleLast read 00:00:04, hold time is 180, keepalive interval is 60 secondsReceived 0 messages, 0 notifications, 0 in queueSent 0 messages, 0 notifications, 0 in queueRoute refresh request: received 0, sent 0Default minimum time between advertisement runs is 30 secondsFor address family: IPv4 UnicastBGP table version 1, neighbor version 0Index 2, Offset 0, Mask 0x40 accepted prefixes consume 0 bytesPrefix advertised 0, suppressed 0, withdrawn 0Connections established 0; dropped 0Last reset neverExternal BGP neighbor not directly connected.No active TCP connection
• Assume R3 added a route to 2.2.2.2• Still having problems…
282828NANOG 33
Peer Establishment:eBGP
• eBGP peers are normally directly connectedBy default, TTL is set to 1 for eBGP peers
If not directly connected, specify ebgp-multihop
• At this point, the session should come up
R2#router bgp 1neighbor 3.3.3.3 remote-as 2neighbor 3.3.3.3 ebgp-multihop 2neighbor 3.3.3.3 update-source Loopback0
292929NANOG 33
Peer Establishment:eBGP
• Still having trouble!Connectivity issues have already been checked and corrected
R2#show ip bgp summary BGP router identifier 2.2.2.2, local AS number 1
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State3.3.3.3 4 2 10 26 0 0 0 never Active
303030NANOG 33
Peer Establishment:eBGP
• If an error is detected, a notification is sent and the sessionis closed
• R3 is configured incorrectlyHas “neighbor 2.2.2.2 remote-as 10”Should have “neighbor 2.2.2.2 remote-as 1”
• After R3 makes this correction the session should come up
R2#debug ip bgp events14:06:37: BGP: 3.3.3.3 open active, local address 2.2.2.214:06:37: BGP: 3.3.3.3 went from Active to OpenSent14:06:37: BGP: 3.3.3.3 sending OPEN, version 414:06:37: BGP: 3.3.3.3 received NOTIFICATION 2/2
(peer in wrong AS) 2 bytes 000114:06:37: BGP: 3.3.3.3 remote close, state CLOSEWAIT14:06:37: BGP: service reset requests14:06:37: BGP: 3.3.3.3 went from OpenSent to Idle14:06:37: BGP: 3.3.3.3 closing
313131NANOG 33
Peer Establishment:eBGP – Summary
• Remember to allow TCP/179 through edge filters
• Be very careful with multihop eBGPCheck IP connectivity (local and remote routing tables)Remember to source updates from loopbackWatch for filters anywhere in the pathTTL must be at least 2 for ebgp-multihop between directly connected neighbours
Use TTL value carefully
access-list 100 permit tcp host 3.3.3.3 eq 179 host 2.2.2.2access-list 100 permit tcp host 3.3.3.3 host 2.2.2.2 eq 179
323232NANOG 33
Peer Establishment:Passwords
• Using passwords on iBGP and eBGP sessionsLink won’t come up
Been through all the previous troubleshooting steps
R2#show ip bgp summary BGP router identifier 2.2.2.2, local AS number 1
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd3.3.3.3 4 2 10 26 0 0 0 never Active
333333NANOG 33
R2#router bgp 1neighbor 3.3.3.3 remote-as 2neighbor 3.3.3.3 ebgp-multihop 2neighbor 3.3.3.3 update-source Loopback0neighbor 3.3.3.3 password 7 05080F1C221C
Peer Establishment:Passwords
• Configuration on R2 looks fine!
• Check the log messages – enable “log-neighbor-changes”%TCP-6-BADAUTH: No MD5 digest from 3.3.3.3:179 to 2.2.2.2:11272%TCP-6-BADAUTH: No MD5 digest from 3.3.3.3:179 to 2.2.2.2:11272%TCP-6-BADAUTH: No MD5 digest from 3.3.3.3:179 to 2.2.2.2:11272
343434NANOG 33
R3#router bgp 2neighbor 2.2.2.2 remote-as 1neighbor 2.2.2.2 ebgp-multihop 2neighbor 2.2.2.2 update-source Loopback0
Peer Establishment:Passwords
• Check configuration on R3Password is missing from the eBGP configuration
• Fix the R3 configurationPeering should now come up!
But it does not
353535NANOG 33
Peer Establishment:Passwords
• Let’s look at the log messages again for any cluesR2#
%TCP-6-BADAUTH: Invalid MD5 digest from 3.3.3.3:11024 to 2.2.2.2:179
%TCP-6-BADAUTH: Invalid MD5 digest from 3.3.3.3:11024 to 2.2.2.2:179
%TCP-6-BADAUTH: Invalid MD5 digest from 3.3.3.3:11024 to 2.2.2.2:179
• We are getting invalid MD5 digest messages – password mismatch!
363636NANOG 33
Peer Establishment:Passwords
• We must have typo’ed the password on one of the peering routers
Fix the password – best to re-enter password on both routers
eBGP session now comes up
%TCP-6-BADAUTH: Invalid MD5 digest from 3.3.3.3:11027 to 2.2.2.2:179%BGP-5-ADJCHANGE: neighbor 3.3.3.3 Up
373737NANOG 33
Peer Establishment:Passwords – Summary
• Common problems:Missing password – needs to be on both endsCut and paste errors – don’t!Typographical & transcription errorsCapitalisation, extra characters, white space…
• Common solutions:Check for symptoms/messages in the logsRe-enter passwords using keyboard, from scratch – don’t cut&paste
383838NANOG 33
Flapping Peer:Common Symptoms
• Symptoms – the eBGP session flaps
• eBGP peering establishes, then drops, re-establishes, then drops,…
AS 2AS 1AS 1
Layer 2
eBGP R2R2R1R1
393939NANOG 33
• Ensure BGP neighbour logging is enabledno logs → no clue what is going on
• R1 and R2 are peering over some 3rd party L2 network
R2#
%BGP-5-ADJCHANGE: neighbor 1.1.1.1 Down BGP Notification sent
%BGP-3-NOTIFICATION: sent to neighbor 1.1.1.1 4/0 (hold time expired) 0 bytes
R2#show ip bgp neighbor 1.1.1.1 | include Last reset
Last reset 00:01:02, due to BGP Notification sent, hold time expired
• We are not receiving keepalives from the other side!
Flapping Peer
404040NANOG 33
R1#show ip bgp summaryBGP router identifier 172.16.175.53, local AS number 1BGP table version is 10167, main routing table version 10167
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd2.2.2.2 4 2 53 284 10167 0 97 00:02:15 0
R1#show ip bgp summary | begin NeighborNeighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd2.2.2.2 4 2 53 284 10167 0 98 00:03:04 0
Flapping Peer
• Hellos are stuck in OutQ behind update packets!• Notice that the MsgSent counter has not moved
• Let’s take a look at our peer!
414141NANOG 33
R1#ping 2.2.2.2Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:!!!!!Success rate is 100 percent (5/5), round-trip min/avg/max = 16/21/24 ms
R1#ping ipTarget IP address: 2.2.2.2Repeat count [5]: Datagram size [100]: 1500Timeout in seconds [2]: Extended commands [n]: Sweep range of sizes [n]: Type escape sequence to abort.Sending 5, 1500-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:.....Success rate is 0 percent (0/5)
Flapping Peer
• Normal pings work but a 1500byte ping fails?
424242NANOG 33
Flapping Peer:Diagnosis and Solution
• DiagnosisKeepalives get lost because they get stuck in the router’s queue behind BGP update packets.
BGP update packets are packed to the size of the MTU –keepalives and BGP OPEN packets are not packed to the size of the MTU ⇒ Path MTU problemsUse ping with different size packets to confirm the above –100byte ping succeeds, 1500byte ping fails = MTU problem somewhere
• SolutionPass the problem to the L2 folks – but be helpful, try and pinpoint using ping where the problem might be in the network
434343NANOG 33
Flapping Peer:Other Common Problems
• Remote router rebooting continually (typical with a 3-5 minute BGP peering cycle time)
• Remote router BGP process unstable, restarting• Traffic Shaping & Rate Limiting parameters• MTU incorrectly set on links, PMTU discovery disabled on
router• For non-ATM/FR links, instability in the L2 point-to-point
circuitsFaulty MUXes, bad connectors, interoperability problems, PPP problems, satellite or radio problems, weather, etc. The list is endless – your L2 folks should know how to solve themFor you, ping is the tool to use
444444NANOG 33
Flapping Peer:Fixed!
• Large packets are ok now
• BGP session is stable!
AS 2AS 1AS 1
Layer 2
eBGP R2R2R1R1
Small Packets
Large Packets
454545NANOG 33
Local Configuration Problems
• Peer Establishment
• Missing Routes
• Inconsistent Route Selection
• Loops and Convergence Issues
464646NANOG 33
Quick Review
• Once the session has been established, UPDATEs are exchanged
All the locally known routes
Only the bestpath is advertised
• Incremental UPDATE messages are exchanged afterwards
474747NANOG 33
Quick Review
• Bestpath received from eBGP peerAdvertise to all peers
• Bestpath received from iBGP peerAdvertise only to eBGP peers
A full iBGP mesh must exist(Unless we are using Route Reflectors)
484848NANOG 33
Missing Routes
• Route Origination
• UPDATE Exchange
• Filtering
• iBGP mesh problems
494949NANOG 33
Missing Routes:Route Origination
• Common problem occurs when putting prefixes into the BGP table
• BGP table is NOT the RIB(RIB = Routing Information Base – a.k.a the Routing Table)BGP table, as with OSPF table, ISIS table, static routes, etc, is used to feed the RIB, and hence the FIBEach routing protocol has a different priority or “distance”
505050NANOG 33
Missing Routes:Route Origination
• To get a prefix into BGP, it must exist in another routing process too, typically:
Static route pointing to customer (for customer routes into your iBGP)
Static route pointing to Null (for aggregates you want to put into your eBGP)
515151NANOG 33
Route Origination:Example I
• Network statementR1# show run | include 200.200.0.0
network 200.200.0.0 mask 255.255.252.0
• BGP is not originating the route???R1# show ip bgp | include 200.200.0.0
R1#
• Do we have the exact route?R1# show ip route 200.200.0.0 255.255.252.0
% Network not in table
525252NANOG 33
• Nail down routes you want to originateip route 200.200.0.0 255.255.252.0 Null0 254
• Check the RIBR1# show ip route 200.200.0.0 255.255.252.0
200.200.0.0/22 is subnetted, 1 subnets
S 200.200.0.0 [1/0] via Null 0
• BGP originates the route!!R1# show ip bgp | include 200.200.0.0
*> 200.200.0.0/22 0.0.0.0 0 32768
Route Origination:Example I
535353NANOG 33
Route Origination:Example II
• Trying to originate an aggregate routeaggregate-address 7.7.0.0 255.255.0.0 summary-only
• The RIB has a component but BGP does not create the aggregate???
R1# show ip route 7.7.0.0 255.255.0.0 longer
7.0.0.0/32 is subnetted, 1 subnets
C 7.7.7.7 [1/0] is directly connected, Loopback 0
R1# show ip bgp | i 7.7.0.0
R1#
545454NANOG 33
Route Origination:Example II
• Remember, to have a BGP aggregate you need aBGP component, not a RIB component
R1# show ip bgp 7.7.0.0 255.255.0.0 longer
R1#
• Once BGP has a component route we originatethe aggregate
network 7.7.7.7 mask 255.255.255.255
R1# show ip bgp 7.7.0.0 255.255.0.0 longer
*> 7.7.0.0/16 0.0.0.0 32768 i
s> 7.7.7.7/32 0.0.0.0 0 32768 i
• s means this component is suppressed due to the “summary-only” argument
555555NANOG 33
Troubleshooting Tips
• BGP Network statement rulesAlways need an exact route (RIB)
• aggregate-address looks in the BGP table, not the RIB
• “show ip route x.x.x.x y.y.y.y longer”Great for finding RIB component routes
• “show ip bgp x.x.x.x y.y.y.y longer”Great for finding BGP component routes
565656NANOG 33
Missing Routes
• Route Origination
• UPDATE Exchange
• Filtering
• iBGP mesh problems
575757NANOG 33
Missing Routes:Update Exchange
• Ah, Route Reflectors…Such a nice solution to help scale iBGP
But why do people insist in breaking the rules all the time?!
• Common issuesClashing router IDs
Clashing cluster IDs
585858NANOG 33
Missing Routes:Example I
• Two RR clusters
• R1 is a RR for R3
• R2 is a RR for R4
• R4 is advertising 7.0.0.0/8
• R2 has the route but R1 and R3 do not?
R1R1 R2R2
R3R3 R4R4
595959NANOG 33
Missing Routes:Example I
• First, did R2 advertise the route to R1?R2# show ip bgp neighbors 1.1.1.1 advertised-routes
BGP table version is 2, local router ID is 2.2.2.2
Network Next Hop Metric LocPrf Weight Path
*>i7.0.0.0 4.4.4.4 0 100 0 i
• Did R1 receive it?R1# show ip bgp neighbors 2.2.2.2 routes
Total number of prefixes 0
606060NANOG 33
Missing Routes:Example I
• Time to debug!!access-list 100 permit ip host 7.0.0.0 host 255.0.0.0
R1# debug ip bgp update 100
• Tell R2 to resend his UPDATEsR2# clear ip bgp 1.1.1.1 out
• R1 shows us something interesting*Mar 1 21:50:12.410: BGP(0): 2.2.2.2 rcv UPDATE w/ attr: nexthop 4.4.4.4, origin i, localpref 100, metric 0, originator 100.1.1.1, clusterlist 2.2.2.2, path , community , extended community
*Mar 1 21:50:12.410: BGP(0): 2.2.2.2 rcv UPDATE about 7.0.0.0/8 -- DENIED due to: ORIGINATOR is us;
• Cannot accept an update with our Router-ID as the ORIGINATOR_ID. Another means of loop detection in BGP
616161NANOG 33
Missing Routes:Example I – Summary
• R1 is not accepting the route when R2 sends it on from its client, R4
R1 and R4 have the same router ID!If R1 sees its own router ID in the originator attribute in any received prefix, it will reject that prefix
This is how a route reflector attempts to avoid routing loops
• Solutiondo NOT set the router ID by hand unless you have a very good reason to do so and have a very good plan for deploymentRouter-ID is usually calculated automatically by router
626262NANOG 33
Missing Routes:Example II
• One RR cluster
• R1 and R2 are RRs
• R3 and R4 are RRCs
• R4 is advertising 7.0.0.0/8
R2 has it
R1 and R3 do notR1#show run | include clusterbgp cluster-id 10R2#show run | include clusterbgp cluster-id 10
R1R1
R3R3
R2R2
R4R4
636363NANOG 33
Missing Routes:Example II
• Same steps as last time!
• Did R2 advertise it to R1?R2# show ip bgp neighbors 1.1.1.1 advertised-routes
BGP table version is 2, local router ID is 2.2.2.2
Origin codes: i - IGP, e - EGP, ? – incomplete
Network Next Hop Metric LocPrf Weight Path
*>i7.0.0.0 4.4.4.4 0 100 0 i
• Did R1 receive it?R1# show ip bgp neighbor 2.2.2.2 routes
Total number of prefixes 0
646464NANOG 33
Missing Routes:Example II
• Time to debug!!access-list 100 permit ip host 7.0.0.0 host 255.0.0.0
R1# debug ip bgp update 100
• Tell R2 to resend his UPDATEsR2# clear ip bgp 1.1.1.1 out
• R1 shows us something interesting*Mar 3 14:28:57.208: BGP(0): 2.2.2.2 rcv UPDATE w/ attr: nexthop 4.4.4.4, origin i, localpref 100, metric 0, originator 4.4.4.4, clusterlist 0.0.0.10, path , community , extended community
*Mar 3 14:28:57.208: BGP(0): 2.2.2.2 rcv UPDATE about 7.0.0.0/8 --DENIED due to: reflected from the same cluster;
• Remember, all RRCs must peer with all RRs in a cluster; allows R4 to send the update directly to R1
656565NANOG 33
• R1 is not accepting the route when R2 sends it onIf R1 sees its own router ID in the cluster-ID attribute in any received prefix, it will reject that prefix
How a route reflector avoids redundant information
• ReasonEarly documentation claimed that RRC redundancy should be achieved by dual route reflectors in the same clusterThis is fine and good, but then ALL clients must peer with both RRs, otherwise examples like this will occur
• SolutionUse overlapping Route Reflector Clusters for redundancy, stay with defaults
Missing Routes:Example II – Summary
666666NANOG 33
Troubleshooting Tips
• “show ip bgp neighbor x.x.x.x advertised”Lets you see a list of NLRI that you sent a peerNote: The attribute values shown are taken from the BGP table; attribute modifications by outbound route-maps will not be shown
• “show ip bgp neighbor x.x.x.x routes”Displays routes x.x.x.x sent to us that made it through our inbound filters
• “show ip bgp neighbor x.x.x.x received”Can only use if “soft-reconfig inbound” is configuredDisplays all routes received from a peer, even those that were denied
676767NANOG 33
Troubleshooting Tips“soft-reconfiguration”
• Ideal for troubleshooting problems with inbound filters and attributes
• “show ip bgp neighbor x.x.x.x routes”alpha#sh ip bgp neigh 192.168.12.1 routes
Network Next Hop Metric LocPrf Weight Path
*>i1.0.0.0 192.168.12.1 0 50 0 i
*>i222.222.0.0/19 192.168.5.1 200 0 3 4 i
• “show ip bgp neighbor x.x.x.x received”alpha#sh ip bgp neigh 192.168.12.1 received-routes
Network Next Hop Metric LocPrf Weight Path
* i1.0.0.0 192.168.12.1 0 100 0 i
* i169.254.0.0 192.168.5.1 0 100 0 3 i
* i222.222.0.0/19 192.168.5.1 100 0 3 4 i
686868NANOG 33
Missing Routes
• Route Origination
• UPDATE Exchange
• Filtering
• iBGP mesh problems
696969NANOG 33
Update Filtering
• Type of filtersPrefix filters
AS_PATH filters
Community filters
Route-maps
• Applied incoming and/or outgoing
707070NANOG 33
Missing RoutesUpdate Filters
• Determine which filters are applied to the BGP session
show ip bgp neighbors x.x.x.x
show run | include neighbor x.x.x.x
• Examine the route and pick out the relevant attributes
show ip bgp x.x.x.x
• Compare the attributes against the filters
717171NANOG 33
Missing RoutesUpdate Filters
• Missing 10.0.0.0/8 in R1 (1.1.1.1)
• Not received from R2 (2.2.2.2)
R1#show ip bgp neigh 2.2.2.2 routes
Total number of prefixes 0
R1R1 R2R2
10.0.0.0/810.0.0.0/8 ???
727272NANOG 33
Missing RoutesUpdate Filters
• R2 originates the route
• Does not advertise it to R1
R2#show ip bgp neigh 1.1.1.1 advertised-routesNetwork Next Hop Metric LocPrf Weight Path
R2#show ip bgp 10.0.0.0BGP routing table entry for 10.0.0.0/8, version 1660Paths: (1 available, best #1)Not advertised to any peerLocal0.0.0.0 from 0.0.0.0 (2.2.2.2)Origin IGP, metric 0, localpref 100, weight 32768, valid,
sourced, local, best
737373NANOG 33
Missing RoutesUpdate Filters
• Time to check filters!• ^ matches the beginning of a line• $ matches the end of a line• ^$ means match any empty AS_PATH• Filter “looks” correct
R2#show run | include neighbor 1.1.1.1neighbor 1.1.1.1 remote-as 3neighbor 1.1.1.1 filter-list 1 out
R2#sh ip as-path 1AS path access list 1permit ^$
747474NANOG 33
R2#show ip bgp filter-list 1
R2#show ip bgp regexp ^$BGP table version is 1661, local router ID is 2.2.2.2Status codes: s suppressed, d damped, h history, * valid,
> best, i - internalOrigin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path*> 10.0.0.0 0.0.0.0 0 32768 i
Missing RoutesUpdate Filters
• Nothing matches the filter-list???
• Re-typing the regexp gives the expected output
757575NANOG 33
R2#show ip bgp regexp ^$
Nothing matches again! Let’s use the up arrow key to see where the cursor stops
R2#show ip bgp regexp ^$ End of Line Is at the Cursor
Missing RoutesUpdate Filters
• Copy and paste the entire regexp line from the configuration
• There is a trailing white space at the end
• It is considered part of the regular expression
767676NANOG 33
R2#clear ip bgp 1.1.1.1 out
R1#show ip bgp 10.0.0.0 % Network not in table
Missing RoutesUpdate Filters
• Force R2 to resend the update after the filter-list correction
• Then check R1 to see if it has the route
• R1 still does not have the route
• Time to check R1’s inbound policy for R2
777777NANOG 33
Missing RoutesUpdate Filters
R1#show run | include neighbor 2.2.2.2neighbor 2.2.2.2 remote-as 12neighbor 2.2.2.2 route-map POLICY in
R1#show route-map POLICYroute-map POLICY, permit, sequence 10
Match clauses:ip address (access-lists): 100 101 as-path (as-path filter): 1
Set clauses:Policy routing matches: 0 packets, 0 bytes
R1#show access-list 100Extended IP access list 100
permit ip host 10.0.0.0 host 255.255.0.0R1#show access-list 101Extended IP access list 101
permit ip 200.1.0 0.0.0.255 host 255.255.255.0R1#show ip as-path 1AS path access list 1
permit ^12$
787878NANOG 33
Missing RoutesUpdate Filters
• Confused? Let’s run some debugsR1#show access-list 99Standard IP access list 99
permit 10.0.0.0
R1#debug ip bgp 2.2.2.2 update 99BGP updates debugging is on for access list 99 for neighbor 2.2.2.2
R1#4d00h: BGP(0): 2.2.2.2 rcvd UPDATE w/ attr: nexthop 2.2.2.2,origin i, metric 0, path 124d00h: BGP(0): 2.2.2.2 rcvd 10.0.0.0/8 -- DENIED due to: route-map;
R1R1 R2R2
10.0.0.0/810.0.0.0/8 ???
797979NANOG 33
Missing RoutesUpdate Filters
R1#sh run | include neighbor 2.2.2.2neighbor 2.2.2.2 remote-as 12neighbor 2.2.2.2 route-map POLICY in
R1#sh route-map POLICYroute-map POLICY, permit, sequence 10
Match clauses:ip address (access-lists): 100 101 as-path (as-path filter): 1
Set clauses:Policy routing matches: 0 packets, 0 bytes
R1#sh access-list 100Extended IP access list 100
permit ip host 10.0.0.0 host 255.255.0.0R1#sh access-list 101Extended IP access list 101
permit ip 200.1.1.0 0.0.0.255 host 255.255.255.0R1#sh ip as-path 1AS path access list 1
permit ^12$
808080NANOG 33
Missing RoutesUpdate Filters
• Wrong mask! Needs to be /8 and the ACL allows a /16 only!Extended IP access list 100
permit ip host 10.0.0.0 host 255.255.0.0
• Should beExtended IP access list 100
permit ip host 10.0.0.0 host 255.0.0.0
• Use prefix-list instead, more difficult to make a mistakeip prefix-list my_filter permit 10.0.0.0/8
• What about ACL 101?Multiple matches on the same line are ORed
Multiple matches on different lines are ANDed
• ACL 101 does not matter because ACL 100 matches which satisfies the OR condition
818181NANOG 33
Update Filtering:Summary
• If you suspect a filtering problem, become familiar with the router tools to find out what BGP filters are applied
• Tip: don’t cut and paste!Many filtering errors and diagnosis problems result from cut and paste buffer problems on the client, the connection, and even the router
828282NANOG 33
Update Filtering:Common Problems
• Typos in regular expressionsExtra characters, missing characters, white space, etcIn regular expressions every character matters, so accuracy is highly important
• Typos in prefix filtersWatch the router CLI, and the filter logic – it may not be as obvious as you think, or as simple as the manual makes outWatch netmask confusion, and 255 profusion – easy to muddle 255 with 0 and 225!
838383NANOG 33
Missing RoutesCommunity Problems
• Missing 10.0.0.0/8 in R1 (1.1.1.1)
• Not received from R2 (2.2.2.2)
R1#show ip bgp neigh 2.2.2.2 routes
Total number of prefixes 0
R1R1 R2R2
10.0.0.0/810.0.0.0/8 ???
848484NANOG 33
R2#show ip bgp 10.0.0.0BGP routing table entry for 10.0.0.0/8, version 1660Paths: (1 available, best #1)Not advertised to any peerLocal0.0.0.0 from 0.0.0.0 (2.2.2.2)Origin IGP, metric 0, localpref 100, weight 32768, valid,
sourced, local, best
Missing RoutesCommunity Problems
• R2 originates the route
• But the community is not setWould be displayed in the “show ip bgp” output
858585NANOG 33
R2#show ip bgp 10.0.0.0BGP routing table entry for 10.0.0.0/8, version 1660Paths: (1 available, best #1)Not advertised to any peerLocal0.0.0.0 from 0.0.0.0 (2.2.2.2)Origin IGP, metric 0, localpref 100, weight 32768,valid, sourced, local, bestCommunity 2:2 1:50
Missing RoutesCommunity Problems
• Fix the configuration so community is setR2#show run | begin bgprouter bgp 2network 10.0.0.0 route-map set-community...route-map set-community permit 10set community 2:2 1:50
868686NANOG 33
Missing RoutesCommunity Problems
• R2 now advertises prefix with community to R1
• But R1 still doesn’t see the prefixR1 insists there is nothing wrong with their configuration
R1#show ip bgp neigh 2.2.2.2 routes
Total number of prefixes 0
• Configuration verified on R2• No filters blocking announcement on R2• So what’s wrong?
878787NANOG 33
Missing RoutesCommunity Problems
• Check R2 configuration again!R2#show run | begin bgprouter bgp 2network 10.0.0.0 route-map set-communityneighbor 1.1.1.1 remote-as 1neighbor 1.1.1.1 prefix-list my-agg outneighbor 1.1.1.1 prefix-list their-agg in!ip prefix-list my-agg permit 10.0.0.0/8ip prefix-list their-agg permit 20.0.0.0/8!route-map set-community permit 10set community 2:2 1:50
• Looks okay – filters okay, route-map okay
• But forgotten “neighbor 1.1.1.1 send-community”Cisco IOS does NOT send communities by default
888888NANOG 33
Missing RoutesCommunity Problems
• R2 now advertises prefix with community to R1
• But R1 still doesn’t see the prefixNothing wrong on R2 now, so turn attention to R1R1#show run | begin bgprouter bgp 1neighbor 2.2.2.2 remote-as 2neighbor 2.2.2.2 route-map R2-in inneighbor 2.2.2.2 route-map R1-out out!ip community-list 1 permit 1:150!route-map R2-in permit 10match community 1set local-preference 150
898989NANOG 33
Missing RoutesCommunity Problems
• Community match on R1 expects 1:150 to be set on prefix
• But R2 is sending 1:50Typo or miscommunication between operations?
• R2 is also using the route-map to filterIf the prefix does not have community 1:150 set, it is dropped – there is no next step in the route-mapWatch the route-map rules in Cisco IOS – they are basically:
if <match> then <set> and exit route-mapelse if <match> then <set> and exit route-mapelse if <match> then <set> etc…
Blank route-map line means match everything, set nothing
909090NANOG 33
R1#show ip bgp neigh 2.2.2.2 routes
Network Next Hop Metric LocPrf Weight Path* 10.0.0.0 2.2.2.2 0 0 2 i
Total number of prefixes 1
Missing RoutesCommunity Problems
• Fix configuration on R2 to set community 1:150 on announcements to R1
• Fix configuration on R1 to also permit prefixes not matching theroute-map – troubleshooting is easier with prefix-filters doing the filtering
R1#show run | begin ^route-maproute-map R2-in permit 10match community 1set local-preference 150route-map R2-in permit 20
919191NANOG 33
Missing RoutesCommunity Problems
• Watch route-mapsRoute-map rules often catch out operators when they are used for filteringAbsence of an appropriate match means the prefix will be discarded
• Remember to configure all routers to send BGP communities
Include it in your default template for iBGPIt should be iBGP default in a Service Provider Network
Remember that it is required to send communities for eBGP too
929292NANOG 33
Missing Routes:Common Community Problems
• Each router implementation has different defaults for when communities are sent
Some don’t send communitiesOthers do for iBGP and not for eBGPOthers do for both iBGP and eBGP peers
• Watch how your implementation handles communities
There may be implicit filtering rules
• Each ISP has different community policiesNever assume that because communities exist that people will use them, or pay attention to the ones you send
939393NANOG 33
Missing Routes:General Problems
• Make and then Stick to simple policy rules:Most router implementations have particular rules for filtering of prefixes, AS-paths, and for manipulating BGP attributesTry not to mix these rules
• Rules for manipulating attributes can also be used for filtering prefixes and ASNs
These can be very powerful, but can also become very confusing
949494NANOG 33
Missing Routes
• Route Origination
• UPDATE Exchange
• Filtering
• iBGP mesh problems
959595NANOG 33
Missing RoutesiBGP Example I
• Symptom: prefixes seen across network, but no connectivity
Prefixes learned from eBGP peer are passed across iBGP mesh
But no connectivity to those prefixes
969696NANOG 33
Missing RoutesiBGP Example I
• R3 customers can reach AS2
• No other customers connected to AS1 or AS3 can reach AS2
AS 1AS 1
AS 3
iBGPiBGP eBGP
1.1.1.11.1.1.1 2.2.2.22.2.2.2
3.3.3.3
4.4.4.4
AA
BB
AS 2
eBGP
R2R2R1R1
R5R5
R4R4R3R3
10.10.0.0/24
979797NANOG 33
Missing RoutesiBGP Example I
• Looking at R3R3#show ip bgp Status codes: * valid, > best, i - internal,
Network Next Hop Metric LocPrf Weight Path*> 3.0.0.0 10.10.10.10 0 2 5 i*> 4.0.0.0 10.10.10.10 0 2 5 i*> 10.10.0.0/24 10.10.10.10 0 2 i*> 10.20.0.0/16 10.10.10.10 0 2 i
R4#show ip bgp Network Next Hop Metric LocPrf Weight Path
* i3.0.0.0 10.10.10.10 100 0 2 5 i* i4.0.0.0 10.10.10.10 100 0 2 5 i* i10.10.0.0/24 10.10.10.10 100 0 2 i* i10.20.0.0/16 10.10.10.10 100 0 2 i
• Looking at R4
989898NANOG 33
Missing Routes:iBGP Example I
• Notice that R3 reports the prefixes learned from AS2
Paths are valid (*) and best (>)
• Notice that R4 reports the prefixes learned from R3
Paths are valid (*) and internal (i)
But no best path
This is the clue…
999999NANOG 33
Missing Routes:iBGP Example I
• Look at the BGP table entry:
• Look at the Routing Table entry
• The next hop?
R4#sh ip bgp 10.10.0.0/24BGP routing table entry for 10.10.0.0/24, version 136Paths: (1 available, no best path)Not advertised to any peer2, (received & used)
10.10.10.10 (inaccessible) from 3.2.1.2 (3.3.3.3)Origin IGP, metric 0, localpref 100, valid, internal
R4#sh ip route 10.10.0.0 255.255.255.0% Network not in table
R4#sh ip route 10.10.10.10% Network not in table
The clues
100100100NANOG 33
Missing Routes:iBGP Example I – Diagnosis
• R4 does not use the 10.10.0.0/24 destination because there is no valid next-hop
• Configuration on R3 has:Either no routing information on how to reach the 10.10.10.10/30 point to point link
By forgetting to put the link into the IGP
Or not excluded external next-hops from the internal network
By forgetting to set itself as the next-hop for all externally learned prefixes on the iBGP session with R4
101101101NANOG 33
Missing Routes:iBGP Example I – Solution
• Make sure that all the BGP NEXT_HOPs are known by the IGP
(whether OSPF/ISIS, static or connected routes)If NEXT_HOP is also in iBGP, ensure the iBGP distance is longer than the IGP distance
—or—
• Don’t carry external NEXT_HOPs in your networkReplace eBGP next_hop with local router address on all the edge BGP routers(Cisco IOS “next-hop-self”)
102102102NANOG 33
Missing RoutesiBGP Example I – Solution
• R3 now includes the missing “next-hop-self” configuration
• Looking at R4 now:
R4#show ip bgp Network Next Hop Metric LocPrf Weight Path
*>i3.0.0.0 3.3.3.3 100 0 2 5 i*>i4.0.0.0 3.3.3.3 100 0 2 5 i*>i10.10.0.0/24 3.3.3.3 100 0 2 i*>i10.20.0.0/16 3.3.3.3 100 0 2 i
103103103NANOG 33
Missing RoutesiBGP Example II
• Symptom: customer complains about patchy Internet access
Can access some, but not all, sites connected to backbone
Can access some, but not all, of the Internet
104104104NANOG 33
Missing RoutesiBGP Example II
• Customer connected to R1 can see AS3, but not AS2
• Also complains about not being able to see sites connected to R5
• No complaints from other customers
AS 1AS 1
AS 3
iBGPiBGP eBGP
1.1.1.11.1.1.1 2.2.2.22.2.2.2
3.3.3.3
4.4.4.4
AA
BB
AS 2
eBGP
R2R2R1R1
R5R5
R4R4R3R3
10.10.0.0/24
105105105NANOG 33
Missing RoutesiBGP Example II
• Diagnosis: This is the classic iBGP mesh problem
The full mesh isn’t complete – how do we know this?
• Customer is connected to R1Can’t see AS2 ⇒ R3 is somehow not passing routing information about AS2 to R1Can’t see R5 ⇒ R5 is somehow not passing routing information about sites connected to R5But can see rest of the Internet ⇒ his prefix is being announced to some places, so not an iBGP origination problem
106106106NANOG 33
Missing RoutesiBGP Example II
• BGP summary shows that the peering with router R1 is down
Up/Down is 3 days 10 hours, yet active
Which means it was last up 3 days and 10 hours ago
So something has broken between R1 and R3
R3#sh ip bgp sum | begin ^NeighNeighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd1.1.1.1 4 1 200 20 32 0 0 3d10h Active2.2.2.2 4 1 210 25 32 0 0 3d16h 154.4.4.4 4 1 213 22 32 0 0 3d16h 125.5.5.5 4 1 215 19 32 0 0 3d16h 010.10.10.10 4 2 2501 2503 32 0 0 3d16h 100R3#
107107107NANOG 33
Missing RoutesiBGP Example II
• Now check configuration on R1
R1#sh conf | b bgprouter bgp 1neighbor iBGP-ipv4-peers peer-groupneighbor iBGP-ipv4-peers remote-as 1neighbor iBGP-ipv4-peers update-source Loopback0neighbor iBGP-ipv4-peers send-communityneighbor iBGP-ipv4-peers prefix-list ibgp-prefixes outneighbor 2.2.2.2 peer-group iBGP-ipv4-peersneighbor 4.4.4.4 peer-group iBGP-ipv4-peersneighbor 5.5.5.5 peer-group iBGP-ipv4-peers
• Where is the peering with R3?
• Restore the missing line, and the iBGP with R3 comes back up
108108108NANOG 33
Missing RoutesiBGP Example II
• BGP summary shows that no prefixes are being heard from R5
This could be due to inbound filters on R3 on the iBGP with R5
But there were no filters in the configuration on R3
This must be due to outbound filters on R5 on the iBGP with R3
R3#sh ip bgp sum | begin ^NeighNeighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd1.1.1.1 4 1 200 20 32 0 0 00:00:50 82.2.2.2 4 1 210 25 32 0 0 3d16h 154.4.4.4 4 1 213 22 32 0 0 3d16h 125.5.5.5 4 1 215 19 32 0 0 3d16h 010.10.10.10 4 2 2501 2503 32 0 0 3d16h 100R3#
109109109NANOG 33
Missing RoutesiBGP Example II
• Now check configuration on R5R5#sh conf | b neighbor 3.3.3.3neighbor 3.3.3.3 remote-as 1neighbor 3.3.3.3 update-source loopback0neighbor 3.3.3.3 prefix-list ebgp-filters outneighbor 4.4.4.4 remote-as 1neighbor 4.4.4.4 update-source loopback0neighbor 4.4.4.4 prefix-list ibgp-filters out!ip prefix-list ebgp-filters permit 20.0.0.0/8ip prefix-list ibgp-filters permit 10.0.0.0/8
• Error in prefix-list in R3 iBGP peeringebgp-filters has been used instead of ibgp-filters
Typo – another advantage of using peer-groups!
110110110NANOG 33
Missing RoutesiBGP Example II
• Fix the prefix-list on R5
• Check the iBGP again on R3Peering with R1 is up
Peering with R5 has prefixes
• Confirm that all is okay with customer
R3#sh ip bgp sum | begin ^NeighNeighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd1.1.1.1 4 1 200 20 32 0 0 00:01:53 82.2.2.2 4 1 210 25 32 0 0 3d16h 154.4.4.4 4 1 213 22 32 0 0 3d16h 125.5.5.5 4 1 215 19 32 0 0 3d16h 610.10.10.10 4 2 2501 2503 32 0 0 3d16h 100R3#
111111111NANOG 33
Troubleshooting Tips
• Watch the iBGP full meshUse peer-groups both for efficiency and to avoid making policy errors within the iBGP meshUse route reflectors to avoid accidentally missing iBGP peers, especially as the mesh grows in size
• Watch the next-hop for external paths
112112112NANOG 33
Local Configuration Problems
• Peer Establishment
• Missing Routes
• Inconsistent Route Selection
• Loops and Convergence Issues
113113113NANOG 33
Inconsistent Route Selection
• Two common problems with route selectionInconsistencyAppearance of an incorrect decision
• RFC 1771 defines the decision algorithm• Every vendor has tweaked the algorithm
http://www.cisco.com/warp/public/459/25.shtml
• Route selection problems can result fromoversights by RFC 1771
114114114NANOG 33
Inconsistent Route Selection:Example I
• RFC says that MED is not always compared• As a result, the ordering of the paths can effect
the decision process• For example, the default in Cisco IOS is to
compare the prefixes in order of arrival (most recent to oldest)
This can result in inconsistent route selectionSymptom is that the best path chosen after each BGP reset is different
115115115NANOG 33
Inconsistent Route Selection:Example I
• Inconsistent route selection may cause problemsRouting loopsConvergence loops—i.e. the protocol continuously sends updates in an attempt to convergeChanges in traffic patterns
• Difficult to catch and troubleshootIn Cisco IOS, the deterministic-med configuration command is used to order paths consistently
Enable in all the routers in the ASThe bestpath is recalculated as soon as the commandis entered
116116116NANOG 33
Symptom I:Diagram
• RouterA will have three paths• MEDs from AS 3 will not be compared to
MEDs from AS 1• RouterA will sometimes select the path from R1 as best and but may
also select the path from R3 as best
AS 3
AS 2
AS 1
RouterA
AS 10AS 1010.0.0.0/810.0.0.0/8
MED 20MED 30
MED 0
R2R3
R1
117117117NANOG 33
Inconsistent Route Selection:Example I
• Initial StatePath 1 beats Path 2 – Lower MED
Path 3 beats Path 1 – Lower Router-ID
RouterA#sh ip bgp 10.0.0.0BGP routing table entry for 10.0.0.0/8, version 40Paths: (3 available, best #3, advertised over iBGP, eBGP)
3 102.2.2.2 from 2.2.2.2
Origin IGP, metric 20, localpref 100, valid, internal3 10
3.3.3.3 from 3.3.3.3Origin IGP, metric 30, valid, external
1 101.1.1.1 from 1.1.1.1
Origin IGP, metric 0, localpref 100, valid, internal, best
118118118NANOG 33
Inconsistent Route Selection:Example I
• 1.1.1.1 bounced so the paths are re-orderedPath 1 beats Path 2 – Lower Router-ID
Path 3 beats Path 1 – External vs Internal
RouterA#sh ip bgp 10.0.0.0BGP routing table entry for 10.0.0.0/8, version 40Paths: (3 available, best #3, advertised over iBGP, eBGP)
1 101.1.1.1 from 1.1.1.1
Origin IGP, metric 0, localpref 100, valid, internal3 10
2.2.2.2 from 2.2.2.2Origin IGP, metric 20, localpref 100, valid, internal
3 103.3.3.3 from 3.3.3.3
Origin IGP, metric 30, valid, external, best
119119119NANOG 33
Deterministic MED:Operation
• The paths are ordered by Neighbour AS
• The bestpath for each Neighbour AS group is selected
• The overall bestpath results from comparing the winners from each group
• The bestpath will be consistent because paths will be placed in a deterministic order
120120120NANOG 33
RouterA#sh ip bgp 10.0.0.0BGP routing table entry for 10.0.0.0/8, version 40Paths: (3 available, best #1, advertised over iBGP, eBGP)
1 101.1.1.1 from 1.1.1.1
Origin IGP, metric 0, localpref 100, valid, internal, best3 10
2.2.2.2 from 2.2.2.2Origin IGP, metric 20, localpref 100, valid, internal
3 103.3.3.3 from 3.3.3.3
Origin IGP, metric 30, valid, external
Deterministic MED:Result
• Path 1 is best for AS 1
• Path 2 beats Path 3 for AS 3 – Lower MED
• Path 1 beats Path 2 – Lower Router-ID
121121121NANOG 33
Deterministic MED:Summary
• Always use “bgp deterministic-med”• Need to enable throughout entire network at
roughly the same time• If only enabled on a portion of the network
routing loops and/or convergence problems may become more severe
• As a result, default behaviour cannot be changed so the knob must be configured by the user
122122122NANOG 33
Inconsistent Route Selection:Solution – Diagram
• RouterA will have three paths
• RouterA will consistently select the path from R1 as best!
AS 3
AS 2
AS 1
RouterA
AS 10AS 1010.0.0.0/810.0.0.0/8
MED 20MED 30
MED 0
R2R3
R1
123123123NANOG 33
R3#show ip bgp 7.0.0.0BGP routing table entry for 7.0.0.0/8, version 15
10 1001.1.1.1 from 1.1.1.1
Origin IGP, metric 0, localpref 100, valid, external20 100
2.2.2.2 from 2.2.2.2Origin IGP, metric 0, localpref 100, valid, external, best
R3R3
AS 10AS 10 AS 20
R1R1
Inconsistent Route Selection:Example II
• The bestpath changes every time the peering is reset
R2R2
124124124NANOG 33
R3#show ip bgp 7.0.0.0 BGP routing table entry for 7.0.0.0/8, version 17Paths: (2 available, best #2)
Not advertised to any peer20 100
2.2.2.2 from 2.2.2.2Origin IGP, metric 0, localpref 100, valid, external
10 1001.1.1.1 from 1.1.1.1
Origin IGP, metric 0, localpref 100, valid, external, best
Inconsistent Route Selection:Example II
• The “oldest” external is the bestpathAll other attributes are the same
Stability enhancement!!—CSCdk12061—Integrated in 12.0(1)
• “bgp bestpath compare-router-id” will disable this enhancement—CSCdr47086—Integrated in 12.0(11)S and 12.1(3)
125125125NANOG 33
R1#sh ip bgp 11.0.0.0BGP routing table entry for 11.0.0.0/8, version 10
1001.1.1.1 from 1.1.1.1
Origin IGP, localpref 120, valid, internal100
2.2.2.2 from 2.2.2.2Origin IGP, metric 0, localpref 100, valid, external, best
Inconsistent Route Selection:Example III
• Path 1 has higher localpref but path 2is better???
• This appears to be incorrect…
126126126NANOG 33
Inconsistent Route Selection:Example III
• Path is from an internal peer which means the path must be synchronized by default
• Check to see if sync is on or offR1# show run | include sync
R1#
• Sync is still enabled, check for IGP path:R1# show ip route 11.0.0.0
% Network not in table
• CSCdr90728 “BGP: Paths are not marked as not synchronized”—Fixed in 12.1(4)
• Path 1 is not synchronized• Router made the correct choice
127127127NANOG 33
Inconsistent Path Selection
• Summary:RFC1771 isn’t prefect when it comes to path selection –years of operational experience have shown this
Vendors and ISPs have worked to put in stability enhancements
But these can lead to interesting problems
And of course some defaults linger much longer than they ought to – so never assume that an out of the box default configuration will be perfect for your network
128128128NANOG 33
Local Configuration Problems
• Peer Establishment
• Missing Routes
• Inconsistent Route Selection
• Loops and Convergence Issues
129129129NANOG 33
• One of the most common problems• Main symptom is that traffic exiting the network
oscillates every minute between two exit pointsThis is almost always caused by the BGP NEXT_HOP being known only by BGPCommon problem in ISP networks – but if you have never seen it before, it can be a nightmare to debug and fix
• Other symptom is high CPU utilisation for the BGP router process
Route Oscillation
130130130NANOG 33
AS 3
AS 12AS 4AS 4
R1R1
R2R2
R3R3
Route Oscillation:Diagram
• R3 prefers routes via AS 4 one minute• BGP scanner runs then R3 prefers routes via AS 12• The entire table oscillates every 60 seconds
142.108.10.2
131131131NANOG 33
R3#show ip bgp summaryBGP router identifier 3.3.3.3, local AS number 3BGP table version is 502, main routing table version 502267 network entries and 272 paths using 34623 bytes of memory
R3#sh ip route summary | begin bgpbgp 3 4 6 520 1400
External: 0 Internal: 10 Local: 0internal 5 5800Total 10 263 13936 43320
Route Oscillation:Diagnosis
• Watch for:Table version number incrementing rapidly
Number of networks/paths or external/internal routes changing
132132132NANOG 33
R3#show ip route 156.1.0.0Routing entry for 156.1.0.0/16
Known via "bgp 3", distance 200, metric 0Routing Descriptor Blocks:* 1.1.1.1, from 1.1.1.1, 00:00:53 ago
Route metric is 0, traffic share count is 1AS Hops 2, BGP network version 474
R3#show ip bgp 156.1.0.0BGP routing table entry for 156.1.0.0/16, version 474Paths: (2 available, best #1)
Advertised to non peer-group peers:2.2.2.2
4 121.1.1.1 from 1.1.1.1 (1.1.1.1)Origin IGP, localpref 100, valid, internal, best
12142.108.10.2 (inaccessible) from 2.2.2.2 (2.2.2.2)Origin IGP, metric 0, localpref 100, valid, internal
Route Oscillation:Troubleshooting
• Pick a route from the RIB that has changed within the last minute
• Monitor that route to see if it changes every minute
133133133NANOG 33
R3#sh ip route 156.1.0.0Routing entry for 156.1.0.0/16
Known via "bgp 3", distance 200, metric 0Routing Descriptor Blocks:
* 142.108.10.2, from 2.2.2.2, 00:00:27 agoRoute metric is 0, traffic share count is 1AS Hops 1, BGP network version 478
R3#sh ip bgp 156.1.0.0BGP routing table entry for 156.1.0.0/16, version 478Paths: (2 available, best #2)
Advertised to non peer-group peers:1.1.1.1
4 121.1.1.1 from 1.1.1.1 (1.1.1.1)Origin IGP, localpref 100, valid, internal
12142.108.10.2 from 2.2.2.2 (2.2.2.2)Origin IGP, metric 0, localpref 100, valid, internal, best
Route Oscillation:Troubleshooting
• Check again after bgp_scanner runs• bgp_scanner runs every 60 seconds and validates
reachability to all nexthops
134134134NANOG 33
Route Oscillation:Troubleshooting
• Lets take a closer look at the nexthopR3#show ip route 142.108.10.2Routing entry for 142.108.0.0/16
Known via "bgp 3", distance 200, metric 0Routing Descriptor Blocks:* 142.108.10.2, from 2.2.2.2, 00:00:50 ago
Route metric is 0, traffic share count is 1AS Hops 1, BGP network version 476
R3#show ip bgp 142.108.10.2BGP routing table entry for 142.108.0.0/16, version 476Paths: (2 available, best #2)
Advertised to non peer-group peers:1.1.1.1
4 121.1.1.1 from 1.1.1.1 (1.1.1.1)Origin IGP, localpref 100, valid, internal
12142.108.10.2 from 2.2.2.2 (2.2.2.2)Origin IGP, metric 0, localpref 100, valid, internal, best
135135135NANOG 33
R3#sh debugBGP events debugging is onBGP updates debugging is onIP routing debugging is on
R3#BGP: scanning routing tablesBGP: nettable_walker 142.108.0.0/16 calling revise_routeRT: del 142.108.0.0 via 142.108.10.2, bgp metric [200/0]BGP: revise route installing 142.108.0.0/16 -> 1.1.1.1RT: add 142.108.0.0/16 via 1.1.1.1, bgp metric [200/0]RT: del 156.1.0.0 via 142.108.10.2, bgp metric [200/0]BGP: revise route installing 156.1.0.0/16 -> 1.1.1.1RT: add 156.1.0.0/16 via 1.1.1.1, bgp metric [200/0]
Route Oscillation:Troubleshooting
• BGP nexthop is known via BGP• Illegal recursive lookup• Scanner will notice and install the other path in the RIB
136136136NANOG 33
Route Oscillation:Troubleshooting
• Route to the nexthop is now valid
• Scanner will detect this and re-install the other path
• Routes will oscillate forever
R3#BGP: scanning routing tablesBGP: ip nettable_walker 142.108.0.0/16 calling revise_routeRT: del 142.108.0.0 via 1.1.1.1, bgp metric [200/0]BGP: revise route installing 142.108.0.0/16 -> 142.108.10.2RT: add 142.108.0.0/16 via 142.108.10.2, bgp metric [200/0]BGP: nettable_walker 156.1.0.0/16 calling revise_routeRT: del 156.1.0.0 via 1.1.1.1, bgp metric [200/0]BGP: revise route installing 156.1.0.0/16 -> 142.108.10.2RT: add 156.1.0.0/16 via 142.108.10.2, bgp metric [200/0]
137137137NANOG 33
AS 3
AS 12AS 4AS 4
R1R1
R2R2
R3R3
Route Oscillation:Step by Step
• R3 naturally prefers routes from AS 12• R3 does not have an IGP route to 142.108.10.2 which is the next-hop for
routes learned via AS 12• R3 learns 142.108.0.0/16 via AS 4 so 142.108.10.2 becomes reachable
142.108.10.2
138138138NANOG 33
Route Oscillation:Step by Step
• R3 then prefers the AS 12 route for 142.108.0.0/16 whose next-hop is 142.108.10.2
• This is an illegal recursive lookup
• BGP detects the problem when scanner runs and flags 142.108.10.2 as inaccessible
• Routes through AS 4 are now preferred
• The cycle continues forever…
139139139NANOG 33
Route Oscillation:Solution
• Make sure that all the BGP NEXT_HOPs are known by the IGP
(whether OSPF/ISIS, static or connected routes)If NEXT_HOP is also in iBGP, ensure the iBGP distance is longer than the IGP distance
—or—
• Don’t carry external NEXT_HOPs in your networkReplace eBGP next_hop with local router address on all the edge BGP routers(Cisco IOS “next-hop-self”)
140140140NANOG 33
AS 3
AS 12AS 4AS 4
R1R1
R2R2
R3R3
Route Oscillation:Solution
• R3 now has IGP route to AS 12 next-hop or R2 is using next-hop-self
• R3 now prefers routes via AS 12 all the time
• No more oscillation!!
142.108.10.2
141141141NANOG 33
Troubleshooting Tips
• High CPU utilisation in the BGP process is normally a sign of a convergence problem
• Find a prefix that changes every minute
• Troubleshoot/debug that one prefix
142142142NANOG 33
Troubleshooting Tips
• BGP routing loop?First, check for IGP routing loops to the BGP NEXT_HOPs
• BGP loops are normally caused byNot following physical topology in RR environment
Multipath with confederations
Lack of a full iBGP mesh
• Get the following from each router in the loop pathThe routing table entry
The BGP table entry
The route to the NEXT_HOP
143143143NANOG 33
Convergence Problems
• Route reflector with 250 route reflector clients
• 100k routes
• BGP will not converge
RRRR
144144144NANOG 33
Convergence Problems
RR# show ip bgp summaryNeighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd20.3.1.160 4 100 10 5416 9419 0 0 00:00:12 Closing20.3.1.161 4 100 11 4418 8055 0 335 00:10:34 020.3.1.162 4 100 12 4718 8759 0 128 00:10:34 020.3.1.163 4 100 9 3517 0 1 0 00:00:53 Connect20.3.1.164 4 100 13 4789 8759 0 374 00:10:37 020.3.1.165 4 100 13 3126 0 0 161 00:10:37 020.3.1.166 4 100 9 5019 9645 0 0 00:00:13 Closing20.3.1.167 4 100 9 6209 9218 0 350 00:10:38 0
• Check the log to find out whyRR#show log | i BGP*May 3 15:27:16: %BGP-5-ADJCHANGE: neighbor 20.3.1.118 Down— BGP Notification sent*May 3 15:27:16: %BGP-3-NOTIFICATION: sent to neighbor 20.3.1.118 4/0 (hold time expired) 0 byt*May 3 15:28:10: %BGP-5-ADJCHANGE: neighbor 20.3.1.52 Down— BGP Notification sent*May 3 15:28:10: %BGP-3-NOTIFICATION: sent to neighbor 20.3.1.52 4/0 (hold time expired) 0 byte
• Have been trying to converge for 10 minutes• Peers keep dropping so we never converge?
145145145NANOG 33
Convergence Problems
• We are either missing hellos or our peers are not sending them• Check for interface input drops
RR# show interface gig 2/0 | include input dropsOutput queue 0/40, 0 drops; input queue 0/75, 72390 dropsRR#
• 72k drops will definitely cause a few peers to go down• We are missing hellos because the interface input queue is very small • A rush of TCP Acks from 250 peers can fill 75 spots in a hurry• Increase the size of the queue
RR# show run interface gig 2/0interface GigabitEthernet 2/0ip address 7.7.7.156 255.255.255.0hold-queue 2000 in
146146146NANOG 33
Convergence Problems
• Let’s start over and give BGP another chance
RR# show log | include BGPRR#
RR# show interface gig 2/0 | include input dropsOutput queue 0/40, 0 drops; input queue 0/2000, 0 dropsRR#
• No more interface input drops
• Our peers are stable!!
RR# clear ip bgp *RR#
147147147NANOG 33
Convergence Problems
• BGP converged in 25 minutes
• Still seems like a long time
• What was TCP doing?RR#show tcp stat | begin Sent: Sent: 1666865 Total, 0 urgent packets
763 control packets (including 5 retransmitted)1614856 data packets (818818410 bytes)39992 data packets (13532829 bytes) retransmitted6548 ack only packets (3245 delayed)1 window probe packets, 2641 window update packets
RR#show ip bgp neighbor | include max data segmentDatagrams (max data segment is 536 bytes):
148148148NANOG 33
Convergence Problems
• 1.6 Million packets is high
• 536 is the default MSS (max segment size) for a TCP connection
• Very small considering the amount of data we need to transfer
RR#show ip bgp neighbor | include max data segmentDatagrams (max data segment is 536 bytes):Datagrams (max data segment is 536 bytes):
RR#show run | include tcpip tcp path-mtu-discoveryRR#
• Enable path mtu discovery
• Sets MSS to max possible value
149149149NANOG 33
Convergence Problems
RR# clear ip bgp *RR#
• Restart the test one more time
RR#show ip bgp neighbor | include max data segmentDatagrams (max data segment is 1460 bytes):Datagrams (max data segment is 1460 bytes):
• MSS looks a lot better
150150150NANOG 33
Convergence Problems
RR# show tcp stat | begin Sent: Sent: 615415 Total, 0 urgent packets
0 control packets (including 0 retransmitted)602587 data packets (818797102 bytes)9609 data packets (7053551 bytes) retransmitted2603 ack only packets (1757 delayed)0 window probe packets, 355 window update packets
• TCP sent 1 million fewer packets
• Path MTU discovery helps reduce overhead by sending more data per packet
• BGP converged in 15 minutes!
• More respectable time for 250 peers and 100k routes
151151151NANOG 33
Summary/Tips
• Use ACLs when enabling debug commands
• Ensure that BGP logging is switched on• Ensure that deterministic MED’s are
enabled• If the entire table is having problem pick
one prefix and troubleshoot it
152152152NANOG 33
Agenda
• Fundamentals
• Local Configuration Problems
• Internet Reachability Problems
153153153NANOG 33
Internet Reachability Problems
• BGP Attribute ConfusionTo Control Traffic in → Send MEDs and AS-PATH prepends on outbound announcements
To Control Traffic out → Attach local-preference to inbound announcements
• Troubleshooting of multihoming and transit is often hampered because the relationship between routing information flow and traffic flow is forgotten
154154154NANOG 33
Internet Reachability ProblemsBGP Path Selection Process
• Each vendor has “tweaked” the path selection process
Know it for your router equipment – saves time later
Especially applies with networks with more than one BGP implementation present
Best policy is to use supplied “knobs” to ensure consistency – and avoid steps in the process which can lead to inconsistency
155155155NANOG 33
Internet Reachability ProblemsMED Confusion
• Default MED on Cisco IOS is ZEROIt may not be this on your router, or your peer’s router
• Best not to rely on MEDs for multihoming on multiple links to upstream
Their default might be 2^32-1 resulting in your hoped for best path being their worst path“Workaround”, i.e. current good practice, is to use communities rather than MEDs
156156156NANOG 33
Internet Reachability ProblemsCommunity Confusion I
• Set community in a route-map does just that – it overwrites any other community set on the prefix
Use additive keyword to add community to existing list
• Use Internet format for community (AS:xx) not the 32-bit IETF format
32-bit format is hard for humans to comprehendWhereas AS:xx format is more intuitive/recognisable
157157157NANOG 33
Internet Reachability ProblemsCommunity Confusion II
• Cisco IOS never sends community by defaultSome implementations send community by default for iBGP peeringsSome implementations also send community by default for eBGP peerings
• Never assume that your neighbouring AS will honour your no-export community – ask first!
If you leak iBGP prefixes to your upstream for loadsharing purposes, this could result in your iBGP prefixes leaking to the Internet
158158158NANOG 33
Internet Reachability ProblemsAS-PATH prepending
• 20 prepends will not lessen the priority of your path any more than 10 prepends will – check it out at a Looking Glass
The Internet is on average only 5 ASes deep, maximum AS prepend most ISPs have to use is around this too
Know you BGP path selection algorithm
• Some ISPs use bgp maxas-path 15 to drop prefixes with AS-paths longer than 15 ASNs
159159159NANOG 33
Internet Reachability ProblemsPrivate ASNs
• Private ASes should not ever appear in the Internet
• Cisco IOS remove-private-AS command does not remove every instance of a private AS
e.g. won’t remove private AS appearing in the middle of a path surrounded by public ASNswww.cisco.com/warp/public/459/32.html
• Apparent non-removal of private-ASNs may not be a bug, but a configuration error somewhere else
160160160NANOG 33
Troubleshooting ConnectivityExample I
• Symptom: AS1 announces 192.168.1.0/24 to AS2 but AS3 cannot see the network
AS 3AS 1AS 1
R3R3R1R1
R2R2
AS 2
192.168.1.0/24
161161161NANOG 33
Troubleshooting ConnectivityExample I
• Checklist:AS1 announces, but does AS2 see it?
Does AS2 see it over entire network?
We are checking eBGP filters on R1 and R2. Remember that R2 access will require cooperation and assistance from your peer
We are checking iBGP across AS2’s network (unneeded step in this case, but usually the next consideration). Quite often iBGP is misconfigured, lack of full mesh, problems with RRs, etc.
162162162NANOG 33
Troubleshooting ConnectivityExample I
• Checklist:Does AS2 send it to AS3?
Does AS3 see all of AS2’s originated prefixes?
We are checking eBGP configuration on R2. There may be a configuration error with as-path filters, or prefix-lists, or communities such that only local prefixes get out
We are checking eBGP configuration on R3. Maybe AS3 does not know to expect prefixes from AS1 in the peering with AS2, or maybe it has similar errors in as-path or prefix or community filters
163163163NANOG 33
Troubleshooting ConnectivityExample I
• Troubleshooting connectivity beyond immediate peers is much harder
Relies on your peer to assist you – they have the relationship with their BGP peers, not you
Quite often connectivity problems are due to the private business relationship between the two neighbouring ASNs
164164164NANOG 33
Troubleshooting ConnectivityExample II
• Symptom: AS1 announces 203.51.206.0/24 to its upstreams but AS3 cannot see the network
AS 3AS 1AS 1
R3R3R1R1
203.51.206.0
The Internet
165165165NANOG 33
Troubleshooting ConnectivityExample II
• Checklist:AS1 announces, but do its upstreams see it?
Is the prefix visible anywhere on the Internet?
We are checking eBGP filters on R1 and upstreams. Remember that upstreams will need to be able to help you with this
We are checking if the upstreams are announcing the network to anywhere on the Internet. See next slides on how to do this.
166166166NANOG 33
Troubleshooting ConnectivityExample II
• Help is at hand – the Looking Glass• Many networks around the globe run Looking
GlassesThese let you see the BGP table and often run simple ping or traceroutes from their siteswww.traceroute.org for IPv4www.traceroute6.org for IPv6
• Next slides have some examples of a typical looking glass in action
167167167NANOG 33
168168168NANOG 33
169169169NANOG 33
Troubleshooting ConnectivityExample II
• Hmmm….
• Looking Glass can see 202.173.144.0/21This includes 202.173.147.0/24
So the problem must be with AS3, or AS3’s upstream
• A traceroute confirms the connectivity
170170170NANOG 33
171171171NANOG 33
Troubleshooting ConnectivityExample II
• Help is at hand – RouteViews• The RouteViews router has BGP feeds from
around 60 peerswww.routeviews.org explains the projectGives access to a real router, and allows any provider to find out how their prefixes are seen in various parts of the InternetComplements the Looking Glass facilities
• Anyway, back to our problem…
172172172NANOG 33
Troubleshooting ConnectivityExample II
• Checklist:Does AS3’s upstream send it to AS3?
Does AS3 see any of AS1’s originated prefixes?
We are checking eBGP configuration on AS3’s upstream. There may be a configuration error with as-path filters, or prefix-lists, or communities such that only local prefixes get out. This needs AS3’s assistance.
We are checking eBGP configuration on R3. Maybe AS3 does not know to expect the prefix from AS1 in the peering with its upstream, or maybe it has some errors in as-path or prefix or community filters
173173173NANOG 33
Troubleshooting ConnectivityExample II
• Troubleshooting across the Internet is harderBut tools are available
• Looking Glasses, offering traceroute, ping and BGP status are available all over the globe
Most connectivity problems seem to be found at the edge of the network, rarely in the transit core
Problems with the transit core are usually intermittent and short term in nature
174174174NANOG 33
Troubleshooting ConnectivityExample III
• Symptom: AS1 is trying to loadshare between its upstreams, but has trouble getting traffic through the AS2 link
AS 3AS 2AS 2
R2R2
The Internet
R1R1
AS 1
R3R3
175175175NANOG 33
Troubleshooting ConnectivityExample III
• Checklist:What does “trouble” mean?
• Is outbound traffic loadsharing okay?Can usually fix this with selectively rejecting prefixes, and using local preferenceGenerally easy to fix, local problem, simple application of policy
• Is inbound traffic loadsharing okay?Errummm, bigger problem if notNeed to do some troubleshooting if configuration with communities, AS-PATH prepends, MEDs and selective leaking of subprefixes don’t seem to help
176176176NANOG 33
Troubleshooting ConnectivityExample III
• Checklist:AS1 announces, but does AS2 see it?
Does AS2 see it over entire network?
We are checking eBGP filters on R1 and R2. Remember that R2 access will require cooperation and assistance from your peer
We are checking iBGP across AS2’s network. Quite often iBGP is misconfigured, lack of full mesh, problems with RRs, etc.
177177177NANOG 33
Troubleshooting ConnectivityExample III
• Checklist:Does AS2 send it to its upstream?
Does the Internet see all of AS2’s originated prefixes?
We are checking eBGP configuration on R2. There may be a configuration error with as-path filters, or prefix-lists, or communities such that only local prefixes get out
We are checking eBGP configuration on other Internet routers. This means using looking glasses. And trying to find one as close to AS2 as possible.
178178178NANOG 33
Troubleshooting ConnectivityExample III
• Checklist:Repeat all of the above for AS3
• Stopping here and resorting to a huge prepend towards AS3 won’t solve the problem
• There are many common problems – listed on next slide
And tools to help decipher the problem
179179179NANOG 33
Troubleshooting ConnectivityExample III
• No inbound traffic from AS2AS2 is not seeing AS1’s prefix, or is blocking it in inbound filters
• A trickle of inbound trafficSwitch on NetFlow (if the router has it) and check the origin of the trafficIf it is just from AS2’s network blocks, then is AS2 announcing the prefix to its upstreams?If they claim they are, ask them to ask their upstream for a “show ip bgp” output – or use a Looking Glass to check
180180180NANOG 33
Troubleshooting ConnectivityExample III
• A light flow of traffic from AS2, but 50% less than from AS3
Looking Glass comes to the rescueLG will let you see what AS2, or AS2’s upstreams are announcingAS1 may choose this as primary path, but AS2 relationship with their upstream may decide otherwise
NetFlow comes to the rescueAllows AS1 to see what the origins are, and with the LG, helps AS1 to find where the prefix filtering culprit might be
181181181NANOG 33
Troubleshooting ConnectivityExample IV
• Symptom: AS1 is loadsharing between its upstreams, but the traffic load swings randomly between AS2 and AS3
AS 3AS 2AS 2
R2R2
The Internet
R1R1
AS 1
R3R3
182182182NANOG 33
Troubleshooting ConnectivityExample IV
• Checklist:Assume AS1 has done everything in this tutorial so far
L2 problem? Route Flap Damping?
All the configurations look fine, the Looking Glass outputs look fine, life is wonderful… Apart from those annoying traffic swings every hour or so
Since BGP is configured fine, and the net has been stable for so long, can only be an L2 problem, or Route Flap Damping side-effect
183183183NANOG 33
Troubleshooting ConnectivityExample IV
• L2 – upstream somewhere has poor connectivity between themselves and the rest of the Internet
Only real solution is to impress upon upstream that this isn’t good enough, and get them to fix it
Or change upstreams
184184184NANOG 33
Troubleshooting ConnectivityExample IV
• Route Flap DampingMany ISPs implement route flap dampingMany ISPs simply use the vendor defaultsVendor defaults are generally far too severeThere is real concern that the “more lenient” RIPE-229 values are too severeOpinion is growing that flap damping does more harm than good
e.g. www.cs.berkeley.edu/~zmao/Papers/sig02.pdf
• Again Looking Glasses come to the operator’s assistance
185185185NANOG 33
186186186NANOG 33
Troubleshooting ConnectivityExample IV
• Several Looking Glasses allow the operators to check the flap or damped status of their announcements
Many oscillating connectivity issues are usually caused by L2 problemsRoute flap damping will cause connectivity to persist via alternative paths even though primary paths have been restoredQuite often, the exponential back off of the flap damping timer will give rise to bizarre routing
Common symptom is that bizarre routing will often clear away by itself
187187187NANOG 33
Troubleshooting Summary
• Most troubleshooting is about:• Experience
Recognising the common problems
• Not panicking• Logical approach
Check configuration firstCheck locally first before blaming the peerTroubleshoot layer 1, then layer 2, then layer 3, etc
188188188NANOG 33
Troubleshooting Summary
• Most troubleshooting is about:
• Using the available toolsThe debugging tools on the router hardware
Internet Looking Glasses
Colleagues and their knowledge
Public mailing lists where appropriate
189189189NANOG 33
Closing Comments
• Presentation has covered the most common troubleshooting techniques used by ISPs today
• Once these have been mastered, more complex or arcane problems are easier to solve
• Feedback and input for future improvements is encouraged and very welcome
190190190NANOG 33
Presentation Slides
• Available onftp://ftp-eng.cisco.com
/pfs/seminars/NANOG33-BGP-Troubleshooting.pdf
191NANOG 33
Troubleshooting BGPThe End! ☺