the temporal and topological characteristics of bgp path changes

2003/11/05 1

The Temporal and Topological Characteristics of BGP Path Changes

Di-Fa Chang Ramesh Govindan John Heidemann

USC/Information Sciences Institute

Presented by Ram Sudhir Nandyala

February 3, 2005

2003/11/05 2

Goal: To understand the changesof inter-domain routing

?

2003/11/05 3

How to Know? Specifically,

How frequently routing events occur? Where do they take place? How many routes do they change?

However, we have only sparse monitoring infrastructure to answer these questions.

2003/11/05 4

Existing Public Monitoring Infrastructure

Looking glasses (provided by ISPs) Active probing per user request. Provide router-level routing information. The monitored space is small.

BGP vantage points (connected to RouteViews and RIPE RRC) Passive monitoring. Provide AS-level routing information. The monitored space is relatively broad.

We start with the analysis of RouteViews datasets.

2003/11/05 5

Challenges To establish the relation between

Path changes observed by vantage points and Routing events taking place in some AS peerings.

Constraints: The routing events are unknown. The routing messages don’t contain information

about the events. The propagation timing of routing messages is

unknown.

2003/11/05 6

Example: Network Topology

Origin ASfor prefix p1

Vantage Points

x

e

a b c

o

yd

2003/11/05 7

Example: Original Paths

Path x -> o: (x, a, b, c, o)Path y -> o: (y, b, c, o)

Vantage Points

x

e

a b c

o

yd Origin AS

for prefix p1

2003/11/05 8

Example: Routing Event

Path x -> o: (x, a, d, e, c, o)Path y -> o: (y, d, e, c, o)

Vantage Points

x

e

a b c

o

yd Origin AS

for prefix p1

2003/11/05 9

Example: Observed Path Changes

Vantage Points

x

e

a b c

o

yd

VP in AS-x observes Time: 0 Path: (x, a, b, c, o) initial path

Origin ASfor prefix p1

2003/11/05 10

Example: Observed Path Changes

Vantage Points

x

e

a b c

o

yd

VP in AS-x observes Time: 0 Path: (x, a, b, c, o) Time: 10 Path:

withdrawn

initial path

Origin ASfor prefix p1

2003/11/05 11

Example: Observed Path Changes

Vantage Points

x

e

a b c

o

yd

VP in AS-x observes Time: 0 Path: (x, a, b, c, o) Time: 10 Path:

withdrawn Time: 35 Path: (x, a, d,

e, c, o)

initial path

converged path

Origin ASfor prefix p1

2003/11/05 12

Vantage Points

x

e

a b c

o

yd

Example: Observed Path Changes

VP in AS-y observes Time: 0 Path: (y, b, c, o)initial path

Origin ASfor prefix p1

2003/11/05 13

Example: Observed Path Changes

Vantage Points

x

e

a b c

o

yd

VP in AS-y observes Time: 0 Path: (y, b, c, o) Time: 11 Path: (y, d, b, c,

o)

initial path transient path

Origin ASfor prefix p1

2003/11/05 14

Example: Observed Path Changes

Vantage Points

x

e

a b c

o

yd

VP in AS-y observes Time: 0 Path: (y, b, c, o) Time: 11 Path: (y, d, b, c,

o) Time: 30 Path: (y, d, e, c,

o)

initial path transient path

converged path

Origin ASfor prefix p1

2003/11/05 15

Example: Observed Path Changes

Vantage Points

x

e

a b c

o

yd

VP in AS-x observes Time: 0 Path: (x, a, b, c, o) Time: 10 Path:

withdrawn Time: 35 Path: (x, a, d,

e, c, o)

VP in AS-y observes Time: 0 Path: (y, b, c, o) Time: 11 Path: (y, d, b, c,

o) Time: 30 Path: (y, d, e, c,

o)

Origin ASfor prefix p1

2003/11/05 16

Example: Original Paths

Path x -> o: (x, a, b, c)Path y -> o: (y, b, c)

Vantage Points

x

e

a b c

yd

Origin ASfor prefix p2

2003/11/05 17

Example: Routing Event

Path x -> o: (x, a, d, e, c)Path y -> o: (y, d, e, c)

Vantage Points

x

e

a b c

yd

Origin ASfor prefix p2

2003/11/05 18

Example: Observed Path Changes

Vantage Points

x

e

a b c

yd

VP in AS-x observes Time: 0 Path: (x, a, b, c) Time: 30 Path: (x, a, d,

e, c)

VP in AS-y observes Time: 0 Path: (y, b, c) Time: 11 Path: (y, d, b,

c) Time: 29 Path:

withdrawn Time: 50 Path: (y, d, e, c)

Origin ASfor prefix p2

2003/11/05 19

Example: Observed Path Changes VP in AS-x observes

Time Prefix Path

10 p1 withdrawn

30 p2 (x, a, d, e, c)

35 p1 (x, a, d, e, c, o)

VP in AS-y observesTime Prefix Path

11 p1 (y, d, b, c, o)

11 p2 (y, d, b, c)

29 p2 withdrawn

30 p1 (y, d, e, c, o)

50 p2 (y, d, e, c)

2003/11/05 20

Example: Observed Path Changes VP in AS-x observes

Time Prefix Path

4 p3 (x, a)

10 p1 withdrawn

25 p3 withdrawn

25 p4 (x, a, d)

26 p3 (x, a)

27 p5 withdrawn

28 p6 (x, a, d)

30 p2 (x, a, d, e, c)

32 p7 (x, a, d, e)

33 p6 withdrawn

35 p1 (x, a, d, e, c, o)

42 p5 (x, a, b)

42 p8 (x, a, d)

45 p7 withdrawn

47 p8 (x, a, b, d)

50 p5 withdrawn

VP in AS-y observesTime Prefix Path

6 p8 withdrawn

8 p3 (y, b, a)

10 p4 withdrawn

11 p1 (y, d, b, c, o)

11 p2 (y, d, b, c)

25 p5 withdrawn

28 p6 (y, d)

29 p2 withdrawn

30 p1 (y, d, e, c, o)

31 p8 (y, b, d)

32 p3 (y, d, a)

48 p4 (y, d)

48 p7 withdrawn

49 p6 withdrawn

49 p5 (y, b)

50 p2 (y, d, e, c)

2003/11/05 21

Message Correlation Problem

How to find the routing messages triggered by the same events in the stream of BGP updates?

Difficulties: Number of transient path changes is unknown. Number of prefixes affected is unknown. Number of events happened is unknown. Number of observing vantage points is unknown.

2003/11/05 22

Peering Location Problem

How can we locate the candidate peerings where the event takes place?

2003/11/05 23

Peering Location Problem

How can we locate the candidate peerings where the event takes place?

VP in AS-x observesTime Prefix Path

0 p1 (x, a, b, c, o)

10 p1 withdrawn

35 p1 (x, a, d, e, c, o)

VP in AS-y observesTime Prefix Path

0 p1 (y, b, c, o)

11 p1 (y, d, b, c, o)

30 p1 (y, d, e, c, o)

2003/11/05 24

Peering Location Problem

How can we locate the candidate peerings where the event takes place?

VP in AS-x observesTime Prefix Path

0 p1 (x, a, b, c, o)

10 p1 withdrawn

35 p1 (x, a, d, e, c, o)

VP in AS-y observesTime Prefix Path

0 p1 (y, b, c, o)

11 p1 (y, d, b, c, o)

30 p1 (y, d, e, c, o)

x

e

a b c

o

yd

Example: (d, e) High local_pref Recover from previous

failure

2003/11/05 25

Which Peerings Can Cause The Path Change?

x

e

a b c

o

y d

x

e

a b c

o

y d

x

e

a b c

o

y d

x

e

a b c

o

y d

x

e

a b c

o

y d

x

e

a b c

o

y d

x

e

a b c

o

yd

x

e

a c

o

y d

b

2003/11/05 26

Peering Locating Problem (cont.)

Hint: The candidate peerings include those peerings that appears only in the original paths or the new paths.

2003/11/05 27

Peering Locating Problem (cont.)

Hint: The candidate peerings include those peerings that appears only in the original paths or the new paths.

VP in AS-x observesTime Prefix Path

0 p1 (x, a, b, c, o)

10 p1 withdrawn

35 p1 (x, a, d, e, c, o)

VP in AS-y observesTime Prefix Path

0 p1 (y, b, c, o)

11 p1 (y, d, b, c, o)

30 p1 (y, d, e, c, o)

x a b c

e

o

y d

2003/11/05 28

Methodology

Cluster the BGP updates triggered by the same routing event into one group based on: Temporal Relation: the receiving time. Topological Relation: prefix, peerings of the path.

Two-step clustering: Prefix-based: for message correlation problem. Peering-based: for peering location problem.

2003/11/05 29

Prefix-based Clusters A sequence of path advertisements for the same

prefix that are sent by the same vantage point and closely spaced in time.

2003/11/05 30

Prefix-based Clusters A sequence of path advertisements for the same

prefix that are sent by the same vantage point and closely spaced in time.

VP in AS-x observesTime Prefix Path4 p3 (x, a)10 p1

withdrawn25 p3

withdrawn25 p4 (x, a,

d)26 p3 (x, a)27 p5

withdrawn28 p6 (x, a,

d)30 p2 (x, a,

d, e, c)32 p7 (x, a,

d, e)33 p6

withdrawn35 p1 (x, a,

d, e, c, o)42 p5 (x, a,

b)42 p8 (x, a,

d)45 p7

withdrawn47 p8 (x, a,

b, d)50 p5

withdrawn

VP in AS-y observesTime Prefix Path6 p8

withdrawn8 p3 (y, b,

a)10 p4

withdrawn11 p1 (y, d,

b, c, o)11 p2 (y, d,

b, c)25 p5

withdrawn28 p6 (y, d)29 p2

withdrawn30 p1 (y, d,

e, c, o)31 p8 (y, b,

d)32 p3 (y, d,

a)48 p4 (y, d)48 p7

withdrawn49 p6

withdrawn49 p5 (y, b)50 p2 (y, d,

e, c)

2003/11/05 31

Prefix-based Clusters A sequence of path advertisements for the same

prefix that are sent by the same vantage point and closely spaced in time.

VP in AS-x observesTime Prefix Path4 p3 (x, a)10 p1

withdrawn25 p3

withdrawn25 p4 (x, a,

d)26 p3 (x, a)27 p5

withdrawn28 p6 (x, a,

d)30 p2 (x, a,

d, e, c)32 p7 (x, a,

d, e)33 p6

withdrawn35 p1 (x, a,

d, e, c, o)42 p5 (x, a,

b)42 p8 (x, a,

d)45 p7

withdrawn47 p8 (x, a,

b, d)50 p5

withdrawn

VP in AS-y observesTime Prefix Path6 p8

withdrawn8 p3 (y, b,

a)10 p4

withdrawn11 p1 (y, d,

b, c, o)11 p2 (y, d,

b, c)25 p5

withdrawn28 p6 (y, d)29 p2

withdrawn30 p1 (y, d,

e, c, o)31 p8 (y, b,

d)32 p3 (y, d,

a)48 p4 (y, d)48 p7

withdrawn49 p6

withdrawn49 p5 (y, b)50 p2 (y, d,

e, c)

<= T1

> T1

2003/11/05 32

Prefix-based Clusters A sequence of path advertisements for the same

prefix that are sent by the same vantage point and closely spaced in time.

VP in AS-x observesTime Prefix Path4 p3 (x, a)10 p1

withdrawn25 p3

withdrawn25 p4 (x, a,

d)26 p3 (x, a)27 p5

withdrawn28 p6 (x, a,

d)30 p2 (x, a,

d, e, c)32 p7 (x, a,

d, e)33 p6

withdrawn35 p1 (x, a,

d, e, c, o)42 p5 (x, a,

b)42 p8 (x, a,

d)45 p7

withdrawn47 p8 (x, a,

b, d)50 p5

withdrawn

VP in AS-y observesTime Prefix Path6 p8

withdrawn8 p3 (y, b,

a)10 p4

withdrawn11 p1 (y, d,

b, c, o)11 p2 (y, d,

b, c)25 p5

withdrawn28 p6 (y, d)29 p2

withdrawn30 p1 (y, d,

e, c, o)31 p8 (y, b,

d)32 p3 (y, d,

a)48 p4 (y, d)48 p7

withdrawn49 p6

withdrawn49 p5 (y, b)50 p2 (y, d,

e, c)

<= T1

> T1

converged path

2003/11/05 33

Prefix-based Clusters A sequence of path advertisements for the same

prefix that are sent by the same vantage point and closely spaced in time.

VP in AS-x observesTime Prefix Path4 p3 (x, a)10 p1

withdrawn25 p3

withdrawn25 p4 (x, a,

d)26 p3 (x, a)27 p5

withdrawn28 p6 (x, a,

d)30 p2 (x, a,

d, e, c)32 p7 (x, a,

d, e)33 p6

withdrawn35 p1 (x, a,

d, e, c, o)42 p5 (x, a,

b)42 p8 (x, a,

d)45 p7

withdrawn47 p8 (x, a,

b, d)50 p5

withdrawn

VP in AS-y observesTime Prefix Path6 p8

withdrawn8 p3 (y, b,

a)10 p4

withdrawn11 p1 (y, d,

b, c, o)11 p2 (y, d,

b, c)25 p5

withdrawn28 p6 (y, d)29 p2

withdrawn30 p1 (y, d,

e, c, o)31 p8 (y, b,

d)32 p3 (y, d,

a)48 p4 (y, d)48 p7

withdrawn49 p6

withdrawn49 p5 (y, b)50 p2 (y, d,

e, c)

10 (ab,bc,ad,de,ec)

cluster

2003/11/05 34

Prefix-based Clusters A sequence of path advertisements for the same

prefix that are sent by the same vantage point and closely spaced in time.

VP in AS-x observesTime Prefix Path4 p3 (x, a)10 p1

withdrawn25 p3

withdrawn25 p4 (x, a,

d)26 p3 (x, a)27 p5

withdrawn28 p6 (x, a,

d)30 p2 (x, a,

d, e, c)32 p7 (x, a,

d, e)33 p6

withdrawn35 p1 (x, a,

d, e, c, o)42 p5 (x, a,

b)42 p8 (x, a,

d)45 p7

withdrawn47 p8 (x, a,

b, d)50 p5

withdrawn

VP in AS-y observesTime Prefix Path6 p8

withdrawn8 p3 (y, b,

a)10 p4

withdrawn11 p1 (y, d,

b, c, o)11 p2 (y, d,

b, c)25 p5

withdrawn28 p6 (y, d)29 p2

withdrawn30 p1 (y, d,

e, c, o)31 p8 (y, b,

d)32 p3 (y, d,

a)48 p4 (y, d)48 p7

withdrawn49 p6

withdrawn49 p5 (y, b)50 p2 (y, d,

e, c)

10 (ab,bc,ad,de,ec)

cluster

2003/11/05 35

Prefix-based Clusters A sequence of path advertisements for the same

prefix that are sent by the same vantage point and closely spaced in time.

VP in AS-x observesTime Prefix Path4 p3 (x, a)10 p1

withdrawn25 p3

withdrawn25 p4 (x, a,

d)26 p3 (x, a)27 p5

withdrawn28 p6 (x, a,

d)30 p2 (x, a,

d, e, c)32 p7 (x, a,

d, e)33 p6

withdrawn35 p1 (x, a,

d, e, c, o)42 p5 (x, a,

b)42 p8 (x, a,

d)45 p7

withdrawn47 p8 (x, a,

b, d)50 p5

withdrawn

VP in AS-y observesTime Prefix Path6 p8

withdrawn8 p3 (y, b,

a)10 p4

withdrawn11 p1 (y, d,

b, c, o)11 p2 (y, d,

b, c)25 p5

withdrawn28 p6 (y, d)29 p2

withdrawn30 p1 (y, d,

e, c, o)31 p8 (y, b,

d)32 p3 (y, d,

a)48 p4 (y, d)48 p7

withdrawn49 p6

withdrawn49 p5 (y, b)50 p2 (y, d,

e, c)

4 (xa)10 (ab,bc,ad,

de,ec)25 (xa)27 (xa)28 (xa)30 (ab,bc,ad,

de,ec)32 (xa)42 (xa)

6 (bd)8 (ba)10 (yd)11 (yb,bc,yd,

de,ec)11 (yb,bc,yd,

de,ec)25 (yb)28 (yd)48 (yd)

2003/11/05 36

Peering-based Clusters A sequence of prefix-based clusters that are closely

spaced in time and have at least one candidate peerings.

4 (xa)10 (ab,bc,ad,

de,ec)25 (xa)27 (xa)28 (xa,ad)30 (ab,bc,ad,

de,ec)32 (xa)42 (xa)

6 (bd)8 (ba)10 (yd)11 (yb,bc,yd,

de,ec)11 (yb,bc,yd,

de,ec)25 (yb)28 (yd)48 (yd)

2003/11/05 37

Peering-based Clusters A sequence of prefix-based clusters that are closely

spaced in time and have at least one candidate peerings.

4 (xa)10 (ab,bc,ad,

de,ec)25 (xa)27 (xa)28 (xa,ad)30 (ab,bc,ad,

de,ec)32 (xa)42 (xa)

6 (bd)8 (ba)10 (yd)11 (yb,bc,yd,

de,ec)11 (yb,bc,yd,

de,ec)25 (yb)28 (yd)48 (yd)

4 (xa)6 (bd)8 (ba)10 (ab,bc,ad,de,ec)11 (yb,bc,yd,de,ec)11 (yb,bc,yd.de,ec)25 (yb)25 (xa)27 (xa)28 (yd)28 (xa,ad)30 (ab,bc,ad,de,ec)32 (xa)42 (xa)48 (yd)

2003/11/05 38

4 (xa)6 (bd)8 (ba)10 (ab,bc,ad,de,ec)11 (yb,bc,yd,de,ec)11 (yb,bc,yd.de,ec)25 (yb)25 (xa)27 (xa)28 (yd)28 (xa,ad)30 (ab,bc,ad,de,ec)32 (xa)42 (xa)48 (yd)

Peering-based Clusters A sequence of prefix-based clusters that are closely

spaced in time and have at least one candidate peerings.

<= T2

> T2

2003/11/05 39

4 (xa)6 (bd)8 (ba)10 (ab,bc,ad,de,ec)11 (yb,bc,yd,de,ec)11 (yb,bc,yd.de,ec)25 (yb)25 (xa)27 (xa)28 (yd)28 (xa,ad)30 (ab,bc,ad,de,ec)32 (xa)42 (xa)48 (yd)

Peering-based Clusters A sequence of prefix-based clusters that are closely

spaced in time and have at least one candidate peerings.

10 (bc,de,ec)

2003/11/05 40

4 (xa)6 (bd)8 (ba)10 (ab,bc,ad,de,ec)11 (yb,bc,yd,de,ec)11 (yb,bc,yd.de,ec)25 (yb)25 (xa)27 (xa)28 (yd)28 (xa,ad)30 (ab,bc,ad,de,ec)32 (xa)42 (xa)48 (yd)

Peering-based Clusters A sequence of prefix-based clusters that are closely

spaced in time and have at least one candidate peerings.

10 (bc,de,ec)

Event occurs at time-10.

3 candidate peerings

2003/11/05 41

Experimental Data

Dataset: BGP updates from RouteViews from July 2002 to June 2003. 31 vantage points in 24 ASes. 1.68 billion path advertisements. 62%: AS_PATH changes.

Experiments Prefix-based clustering: T1 = 60, 120, 240 seconds Peering-based clustering: T2 = 30, 60, 90 secondsResults are similar for T1 , T2 we studied.

2003/11/05 42

How Many Path Changes Per Event?

Prefix-based clustering: ~50% of clusters have only one path advertisement.MinRouteAdver rating limiting is effective.

Peering-based clustering: median number of path advertisements is 8-9.Not all vantage points see each event.

2003/11/05 43

What Is The Impact of Event?

There are ~4,000 peering-based clusters per hour. Median number of prefixes in a cluster is 2.

Most events have only local impact.But a few events can affect tens of thousands of

prefixes.

2003/11/05 44

Where Events Happen?

47% of clusters are caused by routing events in transit peerings. These clusters consist of at least 50% of all path advertisements.Question: Is Internet core more unstable than edges?

Near-far problem: vantage points receives more routing messages when events occur in near peerings than far peerings.

2003/11/05 45

Summary

Summary MinRouteAdver rate limiting mechanism is

effective. Most events have only local impact. Open question: does Internet core change more

frequently than edges?

Future work Parameter evaluation Validation

2003/11/05 46

Question?