engineering status report

Mark Kosters

Engineering Status Report

Engineering Theme

• 2011 success was aided by contractors

• Lots of work yet to do (but a great deal now done)

• An age for new engineers

Staffing• Operations

– 7 people (one position open)

• Development – 12 people

• 5 Developers + Manager (One position open)

• 6 contractors (down 1 from ARIN XXVII)

• Quality Assurance

– 8 people

• 3 QA + Manager

• 4 contractors (down 1 from ARIN XXVII)

• Requirements/Project Management – 1 person

• Management– Me

Operations• Upgrading end-of-life equipment

• Rolling out Anycast

• Consolidating Colo space in Equinix based on RPKI security needs

• Maintaining the various environments we have running (production/ot&e/dev/qa/staging)

• Consolidating equipment in east coast colo

Whois/Whois-RWS Traffic Loads• At ARIN XXV – 50% of the queries are self-referential (i.e.

source ip 192.168.2.5 asking for 192.168.2.5)

– Most are singleton queries– Was increasing over the last year

– Started noticing decrease after ARIN XXV

Whois-RWS Traffic Loads• At ARIN XXVI

– Saw a rise in traffic day after Google announced OpenID collaboration with Yahoo in September

– Traffic spiked 300%

– Top ten sites being login sites for various providers – Yahoo, AOL, and Facebook

– Approximately 5600 queries per second during the height of the day

Whois-RWS Statistics- Uptick

Whois-RWS Traffic Loads

• At ARIN XXVII– Loads disappeared soon after ARIN XXVI– Running “normally” now at 2000 queries

per second

• So what about now…

Whois-RWS Traffic Loads

• ARIN XXVIII (today)– Still running “normally” now at 2000

queries per second

Whois-RWS Statistics

Months

Qu

eri

es P

er

Secon

d

Queries

0.00

500.00

1000.00

1500.00

2000.00

2500.00

3000.00

3500.00

4000.00

4500.00

Whois-RWS – V6

2009-01

2009-02

2009-03

2009-04

2009-05

2009-06

2009-07

2009-08

2009-09

2009-10

2009-11

2009-12

2010-01

2010-02

2010-03

2010-04

2010-05

2010-06

2010-07

2010-08

2010-09

2010-10

2010-11

2010-12

2011-01

2011-02

2011-03

2011-04

2011-05

2011-06

2011-07

2011-08

2011-090

1000000

2000000

3000000

4000000

5000000

6000000

7000000

8000000

9000000

Tota

l Per

Mon

th

Whois-RWSSince April, 2011

Apr May Jun Jul Aug Sep

0200,000,000400,000,000600,000,000800,000,000

1,000,000,0001,200,000,0001,400,000,0001,600,000,0001,800,000,0002,000,000,000

WebRESTPort 43Q

ueri

es

in-addr.arpa Transition• in-addr.arpa generation moved from ARIN to ICANN

on 2/16/11

• in-addr.arpa moved from root servers to RIR/ICANN managed servers

• Servers moved off root in increments from 2/21/11 until 3/7/11

• in-addr.arpa is now signed

• Provisioned DSs to ICANN for /8’s under ARIN’s control by 5/1/11

• Same goes for v6 reverse (ip6.arpa)

• No longer any need for trust anchors

Traffic from a.in-addr-servers.arpa

Development/QA• Improvements to existing systems

• Three ARIN Online releases since ARIN XXVII

– Billing Integration

– Partial Implementation of 2011-3

– Implementation of 2010-14

– Delegations via REST

– Assorted ACSP requests finished

• IRR deployed on Sept 29

• RPKI development keeps going

How is ARIN Online Used?

• 44,882 accounts activated in Q3 of 2011

2008

2009

2010

2011

Active Usage of ARIN Online

0 1 2 - 5 6 - 10 11 - 15 > 150

5000

10000

15000

20000

25000

Logins

# of

Use

rs

Times Logged In

Management of POCsSince April, 2011

4307

21426

Create

ARIN OnlineTemplates

Modify

4689

Includes POCs created via SWiPs

Management of OrgsSince April, 2011

1507

22138

Create

ARIN OnlineTemplates

3263

16

Modify

Includes Orgs created via SWiPs

ARIN Online 87

Templates 12

Nobody likes to delete their Org records.

Net Record ManagementSince April, 2011

Modify335

394

781

507

Requests

v6 ISPv6 End Userv4 ISPv4 End User

Similar toOrg Modify

All requests made via ARIN Online

2844

20

Reg-RWS (RESTful Provisioning)Since April, 2011

REST

Templates

0 50000 100000 150000 200000 250000

Transactions

AprilM

ayJu

neJu

ly

August

September

0100200300400500

POC CreateOrg CreateCustomer Createv6 Simple Createv6 Simple Removev4 Simple Createv4 Simple Remove

Whois-RWS Releases• Whois-RWS Improvements

– Improved CIDR performance

– CIDR bug fixes

– Removed the errant parent info on IP Address queries Oct 2

Whois-RWS Port 80 User Agents

JavaPerlPythonPHPColdFusionGoogle AppEngineWinAPIWgetInternet ExplorerFirefox

IRR Usage 2009-2Q/2011

2009 2010 2Q/20110

100020003000

Insertions/Changes

0

100

200

300

400

Active Maintainers

In YearSince 2009

442 Insertions by one maintainer

Value-Added Trust Services

• Core services available to all regardless of RSA/LRSA status–Whois– Regular reverse dns services

• Value-Added Services – must have a RSA/LRSA in place– DNSSEC– RPKI

RPKI

• Huge challenge developing against IBM HSM– Poorly documented– Bad failure modes

• Lost the main developer• Estimated to have first part in production in

2012• Protocol is mature – much documentation to

do yet (Global Trust Anchor – NRO and ICANN )

Upcoming Challenges• Community Desires/Policy

– 9 ACSPs – Complete Implementation of 2011-3– Transfers a part of ARIN Online

• Identified Efficiencies– Improved Payments– Voting Integration

• Internal Projects– Limiting liability – Reducing inefficiency

• Thought leadership– Whois-RWS– RESTful Registration

Questions?