Measuring the Autonomous System Measuring the Autonomous System Path Through the InternetPath Through the Internet

Jennifer Rexford

Internet and Networking SystemsAT&T Labs - Research; Florham Park, NJ

http://www.research.att.com/~jrex

Joint work with Z. Morley Mao, David Johnson, Jia Wang, and Randy Katz

IP Forwarding PathIP Forwarding Path

Path packets traverse through the Internet

Why important? Characterize end-to-end network

paths

Discover the router-level Internet topology

Detect and diagnose reachability problems

IP traffic

Internet

sourcedestination

Traceroute: Measuring the Forwarding PathTraceroute: Measuring the Forwarding Path

Time-To-Live field in IP packet header– Source sends a packet with a TTL of n

– Each router along the path decrements the TTL

– “TTL exceeded” sent when TTL reaches 0

Traceroute tool exploits this TTL behavior

source destination

TTL=1

Time exceeded

TTL=2

Send packets with TTL=1, 2, 3, … and record source of “time exceeded” message

Example Traceroute Output (Berkeley to CNN)Example Traceroute Output (Berkeley to CNN)

1 169.229.62.1

2 169.229.59.225

3 128.32.255.169

4 128.32.0.249

5 128.32.0.66

6 209.247.159.109

7 *

8 64.159.1.46

9 209.247.9.170

10 66.185.138.33

11 *

12 66.185.136.17

13 64.236.16.52

Hop number, IP address, DNS nameinr-daedalus-0.CS.Berkeley.EDU

soda-cr-1-1-soda-br-6-2

vlan242.inr-202-doecev.Berkeley.EDU

gigE6-0-0.inr-666-doecev.Berkeley.EDU

qsv-juniper--ucb-gw.calren2.net

POS1-0.hsipaccess1.SanJose1.Level3.net

?

?

pos8-0.hsa2.Atlanta2.Level3.net

pop2-atm-P0-2.atdn.net

?

pop1-atl-P4-0.atdn.net

www4.cnn.com

No responsefrom router

No name resolution

AS A

AS BAS C

AS DAutonomous System (AS)

Autonomous System Forwarding PathAutonomous System Forwarding Path

Example: Pinpoint forwarding loop & responsible AS

IP trafficInternet

sourcedestination

Border Gateway Protocol (BGP)Border Gateway Protocol (BGP)

BGP path may differ from forwarding AS path– Routing loops and deflections

– Route aggregation and filtering

– BGP misconfiguration

AS A AS B AS Cprefix d

Signaling path: control traffic

d: path=[C]

Forwarding path: data traffic

d: path=[BC]Origin AS

Map Traceroute Hops to ASesMap Traceroute Hops to ASes

1 169.229.62.1

2 169.229.59.225

3 128.32.255.169

4 128.32.0.249

5 128.32.0.66

6 209.247.159.109

7 *

8 64.159.1.46

9 209.247.9.170

10 66.185.138.33

11 *

12 66.185.136.17

13 64.236.16.52

Traceroute output: (hop number, IP)AS25

AS25

AS25

AS25

AS11423

AS3356

AS3356

AS3356

AS3356

AS1668

AS1668

AS1668

AS5662

Berkeley

CNN

Calren

Level3

AOL

Need accurate IP-to-AS mappings(for network equipment).

Candidate Ways to Get IP-to-AS MappingCandidate Ways to Get IP-to-AS Mapping

Routing address registry– Voluntary public registry such as whois.radb.net

– Used by prtraceroute and “NANOG traceroute”

– Incomplete and quite out-of-date» Mergers, acquisitions, delegation to customers

Origin AS in BGP paths– Public BGP routing tables such as RouteViews

– Used to translate traceroute data to an AS graph

– Incomplete and inaccurate… but usually right» Multiple Origin ASes (MOAS), no mapping, wrong mapping

Refining Initial IP-to-AS MappingRefining Initial IP-to-AS Mapping

Start with initial IP-to-AS mapping– Mapping from BGP tables is usually correct

– Good starting point for computing the mapping

Collect many BGP and traceroute paths– Signaling and forwarding AS path usually match

– Good way to identify mistakes in IP-to-AS map

Successively refine the IP-to-AS mapping– Find add/change/delete that makes big difference

– Base these “edits” on operational realities

Extra AS due to Internet eXchange PointsExtra AS due to Internet eXchange Points

IXP: shared place where providers meet– E.g., Mae-East, Mae-West, PAIX

– Large number of fan-in and fan-out ASes

A

B

C

D

E

F

G

Traceroute AS path BGP AS path

Physical topology and BGP session graph do not always match.

B

C

F

G

A E

Extra AS due to Sibling ASesExtra AS due to Sibling ASes

Sibling: organizations with multiple ASes:– E.g., Sprint AS 1239 and AS 1791

– AS numbers equipment with addresses of another

Traceroute AS path BGP AS path

A

B

C

D

E

F

G

H

A

B

C

D

E

F

G

Sibling ASes “belong together” as if they were one AS.

Weird Paths Due to Unannounced AddressesWeird Paths Due to Unannounced Addresses

A B

C

A C

A C A C

B A C B C

C does not announce part ofits address space in BGP

(e.g., 12.1.2.0/24)

12.0.0.0/8

Fix the IP-to-AS map to associate 12.1.2.0/24 with C

Reasons BGP and Traceroute Paths May DifferReasons BGP and Traceroute Paths May Differ

IP-to-AS mapping is inaccurate (fix these!)– Internet eXchange Points (IXPs)

– Sibling ASes owned by the same institution

– Unannounced infrastructure addressesForwarding and signaling paths differ (study these!)

– Forwarding loops and deflections

– Route aggregation and filteringTraceroute inaccuracies (don’t overreact to these!)

– Forwarding path changing during measurement

– Address assignment to border links between ASes

– Outgoing link identified in “time exceeded” message

Optimization FrameworkOptimization Framework

Start with initial IP-to-AS map A(x)– IP address x maps to A(x), a set of ASes

Iterative refinement– Apply A(x) to the hops in each traceroute path

– Compare the traceroute hops to the BGP AS path

– Compute mismatch statistics for each entry x

– Modify A(x) depending on a small set of rules

Terminate when no further modifications

Matching Function and Unavoidable ErrorMatching Function and Unavoidable Error

Matching function m for BGP/traceroute pair– Traceroute path: t1, t2, …, tn of n IP addresses

– BGP path: b1, b2, …, bl of l AS numbers

– Matching: associate IP hop ti with AS hop bm(i)

Find the matching m that minimizes error– Number of traceroute hops with bm(i) not in A(ti)

– Dynamic programming algorithm to find best m

t: 7 13 6 5 8 3 10 2

b: A B C

Rules for Modifying the IP-to-AS MappingRules for Modifying the IP-to-AS Mapping

Computing match statistics across paths– Focusing on path pairs with at most two errors

Example rules– Create a mapping: A(x) is null

» Assign to the AS y that appears in the most matchings

– Replace a mapping: A(x) has one entry» If an AS y not in A(x) accounts for > 55% of matchings

– Delete from a mapping: A(x) has multiple entries» If an AS y in A(x) accounts for < 10% of matchings

Algorithm converges in less than ten iterations

Measurement Data: Eight Vantage PointsMeasurement Data: Eight Vantage Points

Organization Location Upstream Provider

AT&T Research NJ, US UUNET, AT&T

UC Berkeley CA, US Qwest, Level3, Internet 2

PSG home network WA, US Sprint, Verio

Univ of Washington WA, US Verio, Cable&Wireless

ArosNet UT, US UUNET

Nortel ON, Canada AT&T Canada

Vineyard.NET MA, US UUNET, Sprint, Level3

Peak Web Hosting CA, US Level 3, Global Crossing, Teleglobe

Sweep the routable IP address space– ~200,000 IP addresses

– 160,000 prefixes

– 15,000 destination ASes

Initial Analysis of BGP and Traceroute PathsInitial Analysis of BGP and Traceroute Paths

Traceroute paths: initial mapping A from BGP– Unmapped hops: match no ASes (1-3% of paths)

– MOAS hops: match any AS in the set (10-13% of paths)

– “*” hops: match any AS (7-9% of paths)

BGP paths: discard 1% of prefixes with AS paths – Routing changes based on BGP updates

– Private AS numbers (e.g., 65100)

– Empty AS paths (local destinations)

– Apparent AS-level loops from misconfiguration

– AS_SET instead of AS sequence

Comparison of IP-to-AS MappingsComparison of IP-to-AS Mappings

Whois: unmapped hops cause half of mismatchesBGP tables: mostly match, as our algorithm assumesRefined mapping: change 2.9% of original mapping

– Robust to reducing # of probes and introducing noise

Whois BGPorigins

Refined mapping

Match 47% 85% 95%

Mismatch 53% 15% 5%

Ratio 0.88 5.8 18

Comparing BGP and Traceroute AS paths for various IP-to-AS mappings

Validating the Changes to the MappingValidating the Changes to the Mapping

AT&T’s tier-1 network (AS 7018)– Dump of configuration state from each of the routers

– Explains 45 of 54 changes involving AS 7018» E.g., customer numbered from AT&T addresses» E.g., Internet exchange point where AT&T connects

Whois query on prefix or AS– Look for “exchange point” or “Internet exchange”

» Explains 24 of the changes to the mappings

– Look for ASes with similar names (Sprintlink vs. Sprintlink3)» Explains many of the changes to the mappings

List of known Internet eXchange Points– Explains 24 of the MOAS inferences

– Total of 38 IXPs contributed to mapping changes

Exploring the Remaining MismatchesExploring the Remaining Mismatches

Route aggregation

– Traceroute AS path longer in 20% of mismatches

– Different paths for destinations in same prefix

Interface numbering at AS boundaries

– Boundary links numbered from one AS

– Verified cases where AT&T (AS 7018) is involved

BGP path: B CTraceroute path: B C DB CC

DD

EE

B CB D DBGP path: B C DTraceroute path: B D

ContributionsContributions

Problem formulation– AS-level traceroute tool for troubleshooting

– Compute an accurate IP-to-AS mappingOptimization approach

– Compute matchings using dynamic programming

– Improve mapping through iterative refinementMeasurement methodology

– Traceroute and BGP paths from many locationsValidation of our results

– Changes to the IP-to-AS mappings

– Remaining mismatches between traceroute and BGP

Future Work on AS TracerouteFuture Work on AS Traceroute

Lower measurement overhead– Avoid traceroute probes that would discover similar paths

– Work with BGP routing tables rather than live feeds Limiting the effects of traceroute inaccuracies

– Catch routing changes through repeat experiments

– Use router-level graphs to detect AS boundaries

– Detect routers using outgoing link in “time exceeded” Public AS traceroute tool

– Periodic data collection and computation of IP-to-AS mapping

– Software to apply mapping to traceroute output Network troubleshooting

– Analyze valid differences between forwarding and signaling paths

– Use the AS traceroute tool to detect and characterize anomalies