Scaling an early stage startupby Mark Maunder <mark@feedjit.com>

Why does performance and scaling quickly matter?

• Slow performance could cost you 20% of your revenue according to Google.

• Any reduction in hosting costs goes directly to your bottom line as profit or can accelerate growth.

• In a viral business, slow performance can damage your viral growth.

My first missteps

• Misconfiguration. Web server and DB configured to grab too much RAM.

• As traffic builds, the server swaps and slows down drastically.

• Easy to fix – just a quick config change on web server and/or DB.

Traffic at this stage

• 2 Widgets per second

• 10 HTTP requests per second.

• 1 Widget = 1 Pageview

• We serve as many pages as our users do, combined.

Keepalive – Good for clients, bad for servers.

• As http requests increased to 10 per second, I ran out of server threads to handle connections.

• Keepalive was on and Keepalive Timeout was set to 300.

• Turned Keepalive off.

Traffic at this stage

• 4 Widgets per second

• 20 HTTP requests per second

Cache as much DB data as possible• I used Perl’s Cache::FileCache to cache either DB

data or rendered HTML on disk.• MemCacheD, developed for LiveJournal, caches

across servers. • YMMV – How dynamic is your data?

MySQL not fast enough

• High number of writes & deletes on a large single table caused severe slowness.

• Writes blow away the query cache.• MySQL doesn’t support a large number of small tables (over

10,000).• MySQL is memory hungry if you want to cache large

indexes. • I maxed out at about 200 concurrent read/write queries per

second with over 1 million records (and that’s not large enough).

Perl’s Tie::File to the early rescue

• Tie::File is a very simple flat-file API.

• Lots of files/tables.• Faster – 500 to 1000

concurrent read/writes per second.

• Prepending requires reading and rewriting the whole file.

BerkeleyDB is very very fast!

• I’m also experimenting with BerkeleyDB for some small intensive tasks.

• Data From Oracle who owns BDB: Just over 90,000 transactional writes per second.

• Over 1 Million non-transactional writes per second in memory.

• Oracle’s machine: Linux on an AMD Athlon™ 64 processor 3200+ at 1GHz system with 1GB of RAM. 7200RPM Drive with 8MB cache RAM.

Source: http://www.oracle.com/technology/products/berkeley-db/pdf/berkeley-db-perf.pdf

Traffic at this stage• 7 Widgets per second

• 35 HTTP requests per second

Created a separate image and CSS server

• Enabled Keepalive on the Image server to be nice to clients.

• Static content requires very little memory per thread/process.

• Kept Keepalive off on the App server to reduce memory.

• Added benefit of higher browser concurrency with 2 hostnames.

Source: http://www.die.net/musings/page_load_time/

Now using Home Grown Fixed Length Records

• A lot like ISAM or MyISAM• Fixed length records mean we

seek directly to the data. No more file slurping.

• Sequential records mean sequential reads which are fast.

• Still using file level locking.• Benchmarked at 20,000+

concurrent reads/writes/deletes.

Traffic at this stage

• 12 Widgets per second

• 50 to 60 HTTP requests per second

• Load average spiking to 12 or more about 3 times per day for unknown reason.

Blocking Content Thieves

• Content thieves were aggressively crawling our site on pages that are CPU intensive.

• Robots.txt is irrelevant. • Reverse DNS lookup with

‘dig –x’• Firewall the &^%$@’s with

‘iptables’

Moved to httpd.prefork• Httpd.worker consumes more memory than

prefork because worker doesn’t share memory.• Tuning the number of Perl interpreters vs number

of threads didn’t improve things.• Prefork with no keepalive on the app server uses

less RAM and works well – for Mod_Perl.

The amazing Linux Filesystem Cache

• Linux uses spare memory to cache files on disk.

• Lots of spare memory == Much faster I/O.

• Prefork freed lots of memory. 1.3 Gigs out of 2 Gigs is used as cache.

• I’ve noticed a roughly 20% performance increase since using it.

Tools• httperf for benchmarking your server• Websitepulse.com for perf monitoring.

Summary

• Make content as static as possible.• Cache as much of your dynamic

content as possible.• Separate serving app requests and

serving static content.• Don’t underestimate the speed of

lightweight file access API’s .• Only serve real users and search

engines you care about.