![Page 1: BGP trends of an AS Looking under the hood and diagnosing the noise. Stephan Millet Network Engineer Telstra](https://reader036.vdocuments.us/reader036/viewer/2022062515/56649c9c5503460f9495b30c/html5/thumbnails/1.jpg)
BGP trends of an AS
Looking under the hood and diagnosing the noise.
Stephan MilletNetwork Engineer
Telstra
![Page 2: BGP trends of an AS Looking under the hood and diagnosing the noise. Stephan Millet Network Engineer Telstra](https://reader036.vdocuments.us/reader036/viewer/2022062515/56649c9c5503460f9495b30c/html5/thumbnails/2.jpg)
Intro• Informational Presentation• 16 month data collection of the BGP activity of AS1221
– Sept 2004 to Jan 2006
• Analysis of the BGP behavior within the AS
Background on AS1221• Only run IP (no other protocol mixes)• Topology hasn’t changes in 3 years• Geographically confined to Australia
– Only one transit provider
• All Cisco (consistency in the BGP algorithm(?))• Core acts as RR for all access routers
– All eBGP is conducted on access– No dampening on transit paths
![Page 3: BGP trends of an AS Looking under the hood and diagnosing the noise. Stephan Millet Network Engineer Telstra](https://reader036.vdocuments.us/reader036/viewer/2022062515/56649c9c5503460f9495b30c/html5/thumbnails/3.jpg)
n-1 BGP topologySydney
Melbourne
Brisbane
Adelaide
Canberra
Perth
Where all routers have an iBGP relationship with all other routers
![Page 4: BGP trends of an AS Looking under the hood and diagnosing the noise. Stephan Millet Network Engineer Telstra](https://reader036.vdocuments.us/reader036/viewer/2022062515/56649c9c5503460f9495b30c/html5/thumbnails/4.jpg)
Add a collector to the n-1 BGP topologySydney
Melbourne
Brisbane
Adelaide
Canberra
Perth
+
![Page 5: BGP trends of an AS Looking under the hood and diagnosing the noise. Stephan Millet Network Engineer Telstra](https://reader036.vdocuments.us/reader036/viewer/2022062515/56649c9c5503460f9495b30c/html5/thumbnails/5.jpg)
Turn on logging and wait 18 months
+
Current configuration:!hostname bgp-logger!log file /data/bgp-mesh/mesh-loglog trap informationallog record-priority!debug bgp updates!router bgp 1221<snip>
BGP: 192.168.1.1 rcvd UPDATE w/ attr: nexthop 203.62.252.59, origin ?,localpref 100, community 1221:500, originator 10.10.1.1, clusterlist 0.0.0.150BGP: 192.168.1.1 rcvd 203.36.198.80/29BGP: 192.168.1.1 rcvd 165.228.83.145/32BGP: 172.16.1.5 rcvd UPDATE w/ attr: nexthop 0.0.0.0, origin i, localpref 0BGP: 172.16.1.5 rcvd UPDATE about 58.163.130.105/32 -- withdrawnBGP: 172.16.1.5 rcvd UPDATE about 144.135.252.150/32 --withdrawnBGP: 172.16.1.5 rcvd UPDATE about 149.135.62.217/32 -- withdrawn
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
1 19 37 55 73 91 109 127 145 163 181 199 217 235 253 271 289 307 325 343 361 379 397 415 433
Time (days)
Up
da
tes
in M
illio
ns
BGP Updates
Prefix Updates
Flaps
Prefixes that updated
iBGP
eBGP
BGP Table
Linear (Prefix Updates)
Every 24hrs process the data and add an extra data point
![Page 6: BGP trends of an AS Looking under the hood and diagnosing the noise. Stephan Millet Network Engineer Telstra](https://reader036.vdocuments.us/reader036/viewer/2022062515/56649c9c5503460f9495b30c/html5/thumbnails/6.jpg)
What the initial data shows
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
05/09/04 05/11/04 05/01/05 05/03/05 05/05/05 05/07/05 05/09/05 05/11/05
Da
ily
up
da
te e
ve
nts
in
millio
ns
BGP Updates
Prefix Updates
Flaps
Prefixes that updated
iBGP
eBGP
BGP Table
Linear (Prefix Updates)
![Page 7: BGP trends of an AS Looking under the hood and diagnosing the noise. Stephan Millet Network Engineer Telstra](https://reader036.vdocuments.us/reader036/viewer/2022062515/56649c9c5503460f9495b30c/html5/thumbnails/7.jpg)
Analysis of initial data set.or what happened in 16 months
• 30% increase in AS1221’s BGP table– 176k to 228k prefixes
• A doubling or greater in all other attributes– Prefix updates from 600k to 1.2M per day– BGP updates from 200k to 550k per day
• A small amount of prefixes are creating a high portion noise– See ‘noisy 100’ later in presentation– ~10% - 15% of the prefixes in the BGP table will generate an update on
a daily basis.• eBGP prefixes are noisier that iBGP prefixes
– Though not for much longer• In 2004 eBGP to iBGP ratio was 4:1• Now eBGP to iBGP ratio is passing 2:1
• The really big spikes.– Operational work on one or more cores e.g. IOS upgrade– The rebooting of the mesh-logger
![Page 8: BGP trends of an AS Looking under the hood and diagnosing the noise. Stephan Millet Network Engineer Telstra](https://reader036.vdocuments.us/reader036/viewer/2022062515/56649c9c5503460f9495b30c/html5/thumbnails/8.jpg)
Looking at the trends.
Raw daily data views
Prefix updates are the new black
pink
![Page 9: BGP trends of an AS Looking under the hood and diagnosing the noise. Stephan Millet Network Engineer Telstra](https://reader036.vdocuments.us/reader036/viewer/2022062515/56649c9c5503460f9495b30c/html5/thumbnails/9.jpg)
Updates and Prefixes
0
100000
200000
300000
400000
500000
600000
700000
800000
900000
01/12/2004 01/02/2005 01/04/2005 01/06/2005 01/08/2005 01/10/2005 01/12/2005
Da
ily B
GP
me
ss
ag
es
0
1
2
3
4
5
6
7
8
9
Pro
po
rtio
n o
f p
refi
xe
s p
er
BG
P m
es
sa
ge
Daily BGP Messages
Avg Prefixes per BGP message
Linear (Daily BGP Messages)
Linear (Avg Prefixes per BGP message)
![Page 10: BGP trends of an AS Looking under the hood and diagnosing the noise. Stephan Millet Network Engineer Telstra](https://reader036.vdocuments.us/reader036/viewer/2022062515/56649c9c5503460f9495b30c/html5/thumbnails/10.jpg)
More going than coming ?
0
0.5
1
1.5
2
2.5
3
3.5
4
01/12/04 01/02/05 01/04/05 01/06/05 01/08/05 01/10/05 01/12/05
Da
ily p
refi
x u
pd
ate
s in
mill
ion
s
Prefix additions
Prefix withdrawls
Linear (Prefix withdrawls)
![Page 11: BGP trends of an AS Looking under the hood and diagnosing the noise. Stephan Millet Network Engineer Telstra](https://reader036.vdocuments.us/reader036/viewer/2022062515/56649c9c5503460f9495b30c/html5/thumbnails/11.jpg)
‘table size’ to ‘prefix updates ratio’
0
50000
100000
150000
200000
250000
300000
350000
05/0
9/0
4
05/1
1/0
4
05/0
1/0
5
05/0
3/0
5
05/0
5/0
5
05/0
7/0
5
05/0
9/0
5
05/1
1/0
5
05/0
1/0
6
05/0
3/0
6
05/0
5/0
6
05/0
7/0
6
05/0
9/0
6
05/1
1/0
6
05/0
1/0
7
05/0
3/0
7
05/0
5/0
7
05/0
7/0
7
05/0
9/0
7
05/1
1/0
7
05/0
1/0
8
Siz
e o
f B
GP
tab
le
0
3
6
9
12
15
18
21
Rati
o o
f d
aily p
refi
x u
pd
ate
s a
s a
pro
po
rtio
n o
f th
e B
GP
tab
le
AS1221 BGP Table sizeRatio of prefix updates to tableLinear (AS1221 BGP Table size)Linear (Ratio of prefix updates to table)
![Page 12: BGP trends of an AS Looking under the hood and diagnosing the noise. Stephan Millet Network Engineer Telstra](https://reader036.vdocuments.us/reader036/viewer/2022062515/56649c9c5503460f9495b30c/html5/thumbnails/12.jpg)
• By Jan 2006 the table to prefix noise ratio increased from 3:1 to 5:1– Today for every prefix in the BGP table expect
5 prefix updates– Heading towards 10:1 by 2008
• With continued BGP table growth, expect 3.0M prefix updates per day.
– How much of this is an artifact of the statistical technique (least squares best fit) and how much is a basic BGP artifact ?
‘table size’ to ‘prefix updates’ ratio continued..
![Page 13: BGP trends of an AS Looking under the hood and diagnosing the noise. Stephan Millet Network Engineer Telstra](https://reader036.vdocuments.us/reader036/viewer/2022062515/56649c9c5503460f9495b30c/html5/thumbnails/13.jpg)
Forecast: number of prefix updates
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
05
/09
/04
05
/11
/04
05
/01
/05
05
/03
/05
05
/05
/05
05
/07
/05
05
/09
/05
05
/11
/05
05
/01
/06
05
/03
/06
05
/05
/06
05
/07
/06
05
/09
/06
05
/11
/06
05
/01
/07
05
/03
/07
05
/05
/07
05
/07
/07
05
/09
/07
05
/11
/07
05
/01
/08
Da
ily p
refi
x u
pd
ate
s in
mill
ion
s
prefix updates
Poly. (prefix updates)
Linear (prefix updates)
![Page 14: BGP trends of an AS Looking under the hood and diagnosing the noise. Stephan Millet Network Engineer Telstra](https://reader036.vdocuments.us/reader036/viewer/2022062515/56649c9c5503460f9495b30c/html5/thumbnails/14.jpg)
Forecast prefix updates continued...
• Does it really matter ?– Hasn’t been a problem to date.
• Traffic is low.. ~1.5kbps
• What about the CPU and Memory on the RP ?
![Page 15: BGP trends of an AS Looking under the hood and diagnosing the noise. Stephan Millet Network Engineer Telstra](https://reader036.vdocuments.us/reader036/viewer/2022062515/56649c9c5503460f9495b30c/html5/thumbnails/15.jpg)
What’s the CPU doing ?
0
10
20
30
40
50
60
70
80
90
100
CP
U U
tili
sa
tio
n
0
50000
100000
150000
200000
250000
Tab
le s
ize
5 Min CPU
1 Min CPU
Routing table
Nov 2003 Nov 2004 Mar 2006 Mar 2008April 2004
GRP-A/256MbGRP-B/512Mb PRP-2/1Gb CRS-1/4Gb ??
![Page 16: BGP trends of an AS Looking under the hood and diagnosing the noise. Stephan Millet Network Engineer Telstra](https://reader036.vdocuments.us/reader036/viewer/2022062515/56649c9c5503460f9495b30c/html5/thumbnails/16.jpg)
What the memory is doing ?
GRP-B to PRP2 upgrade
0
50
100
150
200
250
300
350
04/08/04 04/10/04 04/12/04 04/02/05 04/04/05 04/06/05 04/08/05 04/10/05
Mb
Me
mo
ry u
se
d
0
50000
100000
150000
200000
250000
300000
350000
BG
P t
ab
le s
ize
Memory Usage
BGP Table
![Page 17: BGP trends of an AS Looking under the hood and diagnosing the noise. Stephan Millet Network Engineer Telstra](https://reader036.vdocuments.us/reader036/viewer/2022062515/56649c9c5503460f9495b30c/html5/thumbnails/17.jpg)
Are we OK ?
• Growth, growth and more growth
• 2009 onwards may be the end of a PRP2.– Will probably run at 100% (1Min average)– What happens when the CPU receives
updates faster than it can process them ?– AS’s flapping due to CPU issues will
exacerbate the issue.
![Page 18: BGP trends of an AS Looking under the hood and diagnosing the noise. Stephan Millet Network Engineer Telstra](https://reader036.vdocuments.us/reader036/viewer/2022062515/56649c9c5503460f9495b30c/html5/thumbnails/18.jpg)
Who are the culprits?
Who’s been naughty and who’s been nice.
![Page 19: BGP trends of an AS Looking under the hood and diagnosing the noise. Stephan Millet Network Engineer Telstra](https://reader036.vdocuments.us/reader036/viewer/2022062515/56649c9c5503460f9495b30c/html5/thumbnails/19.jpg)
Noisiest 100 Origin AS’s**
** Includes AS1221 as origin
0
50000
100000
150000
200000
250000
300000
350000
400000
01/12/04 01/02/05 01/04/05 01/06/05 01/08/05 01/10/05 01/12/05
To
p 1
00
Ori
gin
AS
's d
aily
up
da
tes
0.00
12.00
24.00
36.00
48.00
60.00
72.00
84.00
96.00
To
p 1
00
AS
's a
s a
% o
f to
tal d
aily
BG
P u
pd
ate
s
Daily top 100 origin AS updates
% of total daily BGP updates
![Page 20: BGP trends of an AS Looking under the hood and diagnosing the noise. Stephan Millet Network Engineer Telstra](https://reader036.vdocuments.us/reader036/viewer/2022062515/56649c9c5503460f9495b30c/html5/thumbnails/20.jpg)
One AS to rule them all
0
2000
4000
6000
8000
10000
12000
14000
01/12/04 01/02/05 01/04/05 01/06/05 01/08/05 01/10/05 01/12/05
Da
ily B
GP
up
da
tes
0
1
2
3
4
5
6
7
Pe
rce
nta
ge
of
Da
ily u
pd
ate
s
Daily AS9121 Updates
% of daily BGP updates
![Page 21: BGP trends of an AS Looking under the hood and diagnosing the noise. Stephan Millet Network Engineer Telstra](https://reader036.vdocuments.us/reader036/viewer/2022062515/56649c9c5503460f9495b30c/html5/thumbnails/21.jpg)
Info on AS9121
• Turk Telekom• Originate ~160 prefixes
– Snapshot on Jan 30 2006• Varying number of prefixes have an ‘origin’ tag of EGP
– Using really old software or munging routing policy ?– These prefixes seem to oscillate at will
#show ip route 85.100.117.0Routing entry for 85.100.117.0/24 Known via "bgp 1221", distance 200, metric 0 Tag 4637, type internal* 203.50.13.38, from 203.62.252.26, 00:00:56 ago
show ip bgp 85.100.117.0BGP routing table entry for 85.100.117.0/24, version 114434062Paths: (0 available, no best path)
![Page 22: BGP trends of an AS Looking under the hood and diagnosing the noise. Stephan Millet Network Engineer Telstra](https://reader036.vdocuments.us/reader036/viewer/2022062515/56649c9c5503460f9495b30c/html5/thumbnails/22.jpg)
Noisiest 100 Prefixes**
0
100000
200000
300000
400000
500000
600000
01/12/04 01/02/05 01/04/05 01/06/05 01/08/05 01/10/05 01/12/05
Dai
ly t
op
100
pre
fix
up
dat
es
0.00
6.00
12.00
18.00
24.00
30.00
36.00
To
p 1
00 P
refi
xes
as a
% o
f to
tal d
aily
pre
fix
up
dat
es
Top 100 daily prefixes
Daily % of total prefixes
** Includes AS1221 as origin
![Page 23: BGP trends of an AS Looking under the hood and diagnosing the noise. Stephan Millet Network Engineer Telstra](https://reader036.vdocuments.us/reader036/viewer/2022062515/56649c9c5503460f9495b30c/html5/thumbnails/23.jpg)
One prefix to rule them all
0
2000
4000
6000
8000
10000
12000
14000
12/12/04 12/02/05 12/04/05 12/06/05 12/08/05 12/10/05 12/12/05
Da
ily P
refi
x U
pd
ate
s
0
0.5
1
1.5
2
2.5
3
3.5
Pe
rce
nta
ge
of
Da
ily U
pd
ate
s
Daily 61.4.0.0/19 updates
% of daily updates
![Page 24: BGP trends of an AS Looking under the hood and diagnosing the noise. Stephan Millet Network Engineer Telstra](https://reader036.vdocuments.us/reader036/viewer/2022062515/56649c9c5503460f9495b30c/html5/thumbnails/24.jpg)
What can we do ?
• Not run DFZ’s
• Bigger processors, good for those that can afford it.– However come 2009, those that can’t need
alternate steps or issue gets worse for everyone.
• Limit updates ?– Turn on Flap Dampening ?
![Page 25: BGP trends of an AS Looking under the hood and diagnosing the noise. Stephan Millet Network Engineer Telstra](https://reader036.vdocuments.us/reader036/viewer/2022062515/56649c9c5503460f9495b30c/html5/thumbnails/25.jpg)
Questions ?