scaling facebook apps - karl bunyan november 28th

S c a lin g a n a p p lic a t io n

F a c e b o o k D e v e lo p e r ’s G a r a g eN o v e m b e r 2 8 t h 2 0 0 7

Scaling an application

What I’m covering

• Facebook specific issues

• A couple of problems solved• A couple of hacks

• Other issues around scaling

Scaling an application

What I’m not covering:

• Writing good code

• Hosting and hardware• Server optimising

Scaling an application

About Exponetic

• Web application developers

• Based in London• 12 members of staff

• Founded in 2002

• 9 Facebook applications live

• 5 currently in development

The application

Two approaches

• A useful app that’s tricky to scale

• A trivial app that uses no resources

The application

Two approaches

• A trivial app that uses no resources– Which Vegetable Are You?

– 155k users

• A useful app that’s tricky to scale– Six Degrees of Separation– 205k users

The application

Six degrees of separation

• Started as an R&D project: 1.5 days work

• Designed to test the end-to-end process of releasing an application

• We didn’t expect it to go anywhere

• Built in PHP on Windows, with SQL Server (to make use of views and triggers)

The application

Six degrees of separation

Original functionality consisted of:

• Calculate how many degrees of separation from the user to me

• Give them some other numbers

The application

What kind of scale?

Active users:

• Small: < 200

• Smallish: < 1,000• Bigish: < 5,000

• Big: < 25,000 We are here

• Huge: …

What kind of scale?

• Currently 205k users

• 87 users after 2 days

• 2000 users after 1 week• 9700 users after 2 weeks

• 128k users after 2 months

What kind of scale?

0

20000

40000

60000

80000

100000

120000

140000

160000

29/08

/200

7

05/09

/200

7

12/09

/200

7

19/09

/200

7

26/09

/200

7

03/10

/200

7

10/10

/200

7

17/10

/200

7

24/10

/200

7

Installs over time

How to get scale

Two different approaches to application design...

How to get scale

Six Degrees:

• Nothing spammy

• No feeds or notifications used• The sign-up process is:

– User sign-up– Shown results– User is encouraged to invite friends “in order

to connect better”

How to get scale

Which Vegetable:

• Give the user a reward for sending invites:– “Invite your friends to find out which

vegetables they are”

• App promoted on “Six Degrees” for 2 days

• Viral spread thereafter

Overview of the code

How “Six Degrees” is put together

Overview of the code

Version 1:

• A handful of FBML pages

• A lot of database triggers• User profiles updated in real time

Week 1 – 2,000 users

• A few bugs to fix

• Scaling is a minimal issue

• Time to add a new feature:– Show connections back

• Moaning users: “I guess you all are in a "repair stage", and the application is unusable at this point? Please notify us when it's been fixed. Thank you!”

Beginning week 2

• Starting to slow down

• “A few kinks”

• Time to streamline the API calls• … and the database

• Use fb:ref to minimise profile updates

• Use an iframe to perform background tasks

Hack #1: Iframe tasks

• FBML is useful, but Facebook doesn’t wait around...

• ...whereas the API may make you wait• Put tasks that may take time into an iframe

• Pass parameters through the fb:iframe src attribute e.g. <fb:iframe src=“find.php?user1=12345&user2=23456" />

The application

[Iframe]

Week 2 – 10,000 users

• It’s getting busy

• New features have new demands

• Almost every calculation is in an iframe• Queuing of profile updates

• “select id from users with (nolock)”

• More RAM for the server

• The “Calculating…” script

Hack #2: Calculating…

Hack #2: Calculating…

• One iframe:– contains a dynamic ‘calculating’ message– and another iframe

• which does the calculation• and tells its parent when it’s finished

• All done through DHTML

Week 4 – 100,000 users

• Separate server for the database– Dual core– 4 gig ram

• Remove the triggers and run “cleanup” scripts

Week 12 – 200,000 users

• Outgrowing the database server

• Queries are often taking minutes

• 20,000 page views/day, but performance is poor

Week 12 – 200,000 users

More user comments:

“It used to work and it was brilliant. The other day, I was showing it to my friends and it didn't work. I tried the same names that gave results before and even they didn't work. Sometimes it does not find the person, sometimes it times out and etc...Make it work, plz!”

Week 12 – 200,000 users

Next steps:

• The concept was never intended to scale

• Needs revision and rethinking

I’d rather have to change the application once I have the users.

How to scale?

How to scale?

• Design for scale?– Don’t know what’s going to be popular– It’s not a bad problem to solve

• Give the user something on the screen

• Limit API calls in FBML

• Use queuing for non-critical updates• Use the “about” wall to communicate

How to scale?

All condensed into one simple PHP function:

How to scale?

if($app->isScalable())

{

$server++;

}else{

break;

}

S c a lin g a n a p p lic a t io nT h a n k s

F a c e b o o k D e v e lo p e r ’s G a r a g eN o v e m b e r 2 8 t h 2 0 0 7