512K IPv4 Routes: The New Normal

Jim Cowie

Chief Scientist, Dyn Research

NANOG 62, Baltimore

7 October 2014

Pg. 2

Up And To The Right: IPv4 Routing Table Sizes

2009 2010 2011 2012 2013 2014…

512K

Pg. 3

Up And To The Right: IPv4 Routing Table Sizes

2009 2010 2011 2012 2013 2014…

512K

269K +30K

302K +33K

334K +32K

382K +48K

431K +48K

471K +40K

Pg. 4

Up And To The Right: IPv4 Routing Table Sizes

2009 2010 2011 2012 2013 2014…

512K

269K +30K

302K +33K

334K +32K

382K +48K

431K +48K

471K +40K

519K? +48K?

2015?

Pg. 5

“512K day” …. November 2014 (predicted) Let’s Take a Closer Look

Up And To The Right: IPv4 Routing Table Sizes

2009 2010 2011 2012 2013 2014…

Pg. 6

“Consensus” is Broadening

2013 2014

Today’s consensus estimate: 507K (October 6th 2014) 61% of our peers believe it’s higher than consensus 23% believe it’s higher than 512K already

512K

Today: 507K

Pg. 7

2013 2014

512K

Today: 507K

Invisible at 1hr resolution. This was a “normal intraday fluctuation” …Except that we’re so close to 512K. Global table pushed across the line….

“512K Day” Panic: 12 August 2014 (7:49:30 UTC)

Pg. 8

One-second BGP announce/withdraw rates (logscale)

06h00-09h00 UTC 12 August 2014

•  Advertisement rate peaks first •  Withdrawal rate briefly rivals advertisement rate

Pg. 9

Affected prefixes may represent equipment suffering from TCAM exhaustion

•  Consensus routing table size was roughly 500,000 that morning

•  VZ (AS701) deaggregated almost 30,000 /24s at 7:48:38 UTC

•  Consensus routing table size hits 528,000 for 10 minutes

•  528K > 512K; All hell breaks loose

•  Aggregates withdrawn, table reverts to safely less than 512K

Pg. 10

One-second BGP withdrawal rates, logscale

Most of the withdrawals in the hour are AS701 deaggregates… But 24,280 other prefixes are affected as well.

Rate climbs 60x in 10m

Pg. 11

Affected Prefixes, By Country

It’s not your circulatory system

2479

4494

1014

571

492 389

1084

860

727

398

595 Relatively significant impacts in Iran, Thailand, Indonesia, Argentina, Russia

Pg. 12

Affected routers in critical places

Russian provider Vimpelcom’s AS21332 suffers some significant impairment during the event, as seen through the lens of traceroute completion failure.

Pg. 13

Affected routers in critical places

Bayan (PH) loses its Pacnet transit.. twice

Pg. 14

Conclusions

•  Affected countries/networks are likely to suffer again as consensus table reaches 512K organically, “for real”

•  We project that this will begin occuring broadly later this month, with consensus 512K emerging by mid-November

•  Hopefully this brief event in August was enough to encourage people to upgrade and/or reallocate TCAM appropriately…..

Pg. 15

Thank you! Jim Cowie @jimcowie

Dyn Research http://research.dyn.com