web 2.0 presentation sept 2008
DESCRIPTION
TRANSCRIPT
About Me
• With Dyn Inc since 2001• Helped scale the operations from 100k users to 2.5M
users, over 100k domain names, 8 global datacenters– O(B) queries per day– O(M) updates processed per day
• Previously worked for a New Hampshire telco, deploying DSL to residences
What is DNS?
192.168.15.102Comcast User
199.192.8.2(cnn.com server)
www.cnn.com?
DNS Server
199.192.8.2
DNS Query
Web Page Delivery
What can Dyn Inc do?
• DynDNS – Consumer Dynamic DNS Service– Use our domain or register your own domain– E-mail services, SSL certificates, VPS hosting
• Dynect – Enterprise DNS Service– Built for businesses whose web site presence is critical,
failover, load balancing, global traffic management– Global anycast network with 8 locations online
DynDNS – our talk’s focus
• We started the service in 1998 so we could remotely access our PCs on Dynamic IP addresses.
• Instead of knowing what our current IP address was, we just remember a hostname (tomdaly.dyndns.org) and the DynDNS system keeps it updated with the right IP.
What can you do with DynDNS?
• Share web pages, photos, legal music, downloads to your friends
• Access remote desktop, VNC, your security system, cameras, thermostats
• Laptop lo-jack• Gather data from remote weather stations• Update Amber Alert signs
Starting Small - 1998
• Website Form updates BIND• CGI script and text file DB• Functions:
– Log in, log out, get a hostname, set an IP
• Open Source Software:– Perl with Net::DNS module– Apache 1.2 with CGI
Next
• Designed /nic/update– RESTful update interface– /nic/update?hostname=test.dyndns.org&myip=1.2.3.4
• Does the same job as the website CGI• 5000 users on a 56K dialup modem
CGI got slow - 1999
• Multiple users didn’t work – locking problem• Solutions:
– MySQL 3.23– mod_perl– Apache::Registry
• Problems:– CPU– RAM
• Basic colo in Ithaca and 20,000 users
Users demand more - 2001
• Users have: “username.dyndns.org”• Users want: “www.username.com”• Problems:
– More metadata to store– More interaction with BIND (setting up zones)
• Solution:– Lots of shell scripting glue– High-end collocation facility, 5 servers and a switch
Infrastructure Layout
Internet
Firewall
Switch
DNS/Web DNS/Web DNS/Web Database
Basic Scaling
• DNS round robin for web servers– www.dyndns.org 300 A 10.0.0.11– www.dyndns.org 300 A 10.0.0.12– www.dyndns.org 300 A 10.0.0.13
• frontend and backend Apache servers– Frontend to spoon feed slow web clients– Backend to pool MySQL database connections
• Tweak my.cnf for MySQL
Redundancy
• DNS cannot go down, ever.• Ithaca colo for secondary DNS service• Managed servers for secondary DNS
– Consider companies like:• Rackspace• Hurricane Electric• Server Beach• The Planet• Many many others
Continued Growth - 2002
• Rewrote the CGI running members.dyndns.org• Used a perl based templating system• Created chunks of reusable code• Very hard to maintain, hard to involve new
developers, hard to debug• Website – migrated over to PHP
– Mistake for us, spread our competencies out
Rewrite - 2003
• Intergraded www.dyndns.org and members.dyndns.org into one site
• Database and business logic abstraction layer– Perl– Heavy lifting goes here.
• HTML::Mason – embed perl code in HTML• Great for synchronous job processing
Scaling and Redundancy
• MySQL Replication– Problem: Long running queries block– Solution: Replicate data, do queries there
• Load Balancing– Problem: Web servers break– Solution: Buy a load balancer ($$$)– Today: Use Dynect (more later)
• Other single points of failure– Firewalls, Dual PSU in DB server, A+B Power
More Scaling Tips
• Performance is slow:– Disk vs. CPU vs. Network bound?– Use tools: ps, top, netstat, vmstat– man(3) pages are extremely helpful here
• Know your hardware:– RAID Levels – different performance benefits– 32-bit CPU vs. 64-bit CPU– RAM, RAM, RAM– Fast Ethernet? Gigabit Ethernet?
Monitoring
• Is your site up? Use Nagios• Is it fast? Use Smokeping• Look at the internals: Cacti, Munin• Know where your weak spots are
– Logging – what are people doing?– MySQL slow query log – Is the DB suffering?– Exception handlers – e-mail to yourself– Profiling – print time(); stuff(); print time();
Nagios
Smokeping
Cacti
Network is down?
• Console servers• POTS lines and modems
• Views– Outside to inside– Inside to outside– Third party monitoring
DDoS
• We’ve had a lot of ‘em• Stay calm, get your ISP involved right away• Know your network• tcpdump on firewalls• Block destination to recover your network• Block sources at your ISP to make it stop• Don’t make yourself a target
2003 through 2006
• Colocation– Dedicated racks, no more managed servers– Added Chicago, Palo Alto, Hong Kong
• New hardware– BGP routing, multiple carriers everywhere– High end routing gear, switches
• Add MailHop• Add DNS SLA
Scaling /nic/update
• 2.5MM update clients hitting this API.• Apache/mod_perl/mysql• SQLThrottle – distribute R and RW DB connections to
different servers• Same DB queries – use memcached
memcached rocks!
• Memory cache for database calls
• Update, insert, delete – invalidate the cache• Authentication caching, hammers• Consider UDFs
In-house vs. Outsourced Development
• When should you outsource?– Don’t have the talent or knowledge– Need more human staff
• When shouldn’t you outsource?– Complex needs or variables– Takes longer to spec out than just do yourself
Cloud Computing
• We’re not using it yet, but we’re looking at it– On demand usage
• Crunch logs• Analyze statistics• Occasional jobs
• We won’t use the cloud for services– We need control of everything– Not guaranteed for speed or stability
Dynect
• Enterprise DNS service for business customers– Multi User Support– Anycast DNS Network (8 facilities globally)– Logging and Reporting– Failover– Load Balancing– /nic/update and SOAP APIs
Dynect (con’t)
• Rewrote our business logic and database abstraction to be more coherent
• Web code never makes decisions, ever, all in the BLL• Huge amounts of unit and system testing every night• Uses MySQL UDFs to talk to BIND, rather than Perl –
less work for the developers ongoing
Anycast
• Distributes the same IP address globally for DNS queries
• Queries come to local servers, not randomly picked• Speeds DNS query latency from ~50 – 300ms to a
guaranteed 100ms, typ. 50ms• You have to know BGP, get your ISPs to work with
you, much secret sauce• Completely redundant, hard to monitor
Summary
• Start small, grow smart, get big• Get what you need, don’t over build out• Invest into open source software• Determine the right infrastructure:
– Hosted, Managed, Your Own, Cloud
• Plan for scaling (scaling != more boxes)• Listen to what your user base wants
Thank You
Q & A