rapidly building farmville: how we built and scaled a #1 facebook game in 5 weeks (gdc 2010)
DESCRIPTION
Amitt Mahajan discusses the techniques used in the development of the hit social game FarmVille. This presentation covers the processes used by the FarmVille development team to quickly release the game and then scale it to support millions of daily players.(Originally presented at GDC 2010)TRANSCRIPT
![Page 1: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/1.jpg)
Rapidly Developing FarmVilleHow we built and scaled a
#1 Facebook game in 5 weeks
Amitt MahajanLead Developer – FarmVille
March 9, 2010
![Page 2: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/2.jpg)
FarmVille
![Page 3: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/3.jpg)
FarmVille
6 developers (PHP/Flash)
2 artists
3 producers/designers
5 week development cycle
![Page 4: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/4.jpg)
FarmVille
18K users/day after 24 hrs
1M users/day after 4 days
At Launch
![Page 5: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/5.jpg)
FarmVille
110M+ Installs
31M Players/Day
Today
![Page 6: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/6.jpg)
My Goals
• Reduce the amount of time it takes you to develop your games
• Give you a blueprint for how to reliably scale your games to millions without your servers falling over
![Page 7: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/7.jpg)
What slows down developers?
• Other developers• Design / Art• Production (Copy)• Lack of knowledge
![Page 8: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/8.jpg)
Increasing Developer Efficiency
• All developers know both
• Design DOESN’T rule all• Developers co-own features with designers
Flash (Client)PHP (Server)
![Page 9: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/9.jpg)
Data Driven Design
• Shoot for a content pipeline that doesn’t need a developer
• Data driven can be as easy as a designer-editable XML file
No dev needed!
![Page 10: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/10.jpg)
String Tables
• A string table is an external file that holds strings for the app
• Best practice to prepare for localization• Developers are not blocked by production• Allows quick response to Facebook TOS
changes
![Page 11: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/11.jpg)
Abstracted Network Layer
Goal Feature developers get client/server communication and serialization for free
![Page 12: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/12.jpg)
Abstracted Network Layer
AMF Protocol (RPC)
Flash PHP
Network Layer
Validation Checks
Action Batching
Networked Actions
Feature A Feature B
Network Layer
Validation Checks
Result Batching
Feature A Service
Feature B Service
Action Dispatcher
![Page 13: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/13.jpg)
Client
Network Layer Batching
• No need to do the work twice• Reduced server load in FarmVille by 50%!
Plow
Plant
Plow
PlantPlow
Only One Request
ServeronPlow()
onPlow()
onPlant()
…
![Page 14: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/14.jpg)
Network Layer Validation
• Solves problem of unfiltered input• Ensures data is received by server in-order it
was sent
![Page 15: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/15.jpg)
Social Network Wrapper
• Single place to perform Facebook API calls• Working with a highly dynamic API can be
difficult• Abstracting FB calls makes them easier to
integrate• Allows for cross-platform games
![Page 16: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/16.jpg)
Continuous Deployment
• Build latest version of source-repo• Deploy to test “auto app” on Facebook to
surface production issues early
![Page 17: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/17.jpg)
Continuous Deployment
FarmVille QA Process
QuickSmoke
Full TestPass
Auto app Staging Production
![Page 18: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/18.jpg)
FarmVille Traffic Growth
1 Million DAUsevery week for 20 weeks
source: developeranalytics.com
![Page 19: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/19.jpg)
Scalable Server Architecture
• We had no choice but to scale on the cloud• Every part of the server architecture scales
horizontally• No single points of failure• Take the DB out of the equation
![Page 20: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/20.jpg)
Memcache Pool
Auto-scaling web array (PHP)
Web Server #1 Web Server #M
User Data & Updates
…
DataReads
Lazy Writes
DB Layer
Round Robin DNS
Load Balancer #1 Load Balancer #N
…
… DB-#KSlave
DB-#1Slave
DB-#KMaster
DB-#1Master
…
![Page 21: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/21.jpg)
Web Server #1 Web Server #M
…
…
All you need to implement is…
…everything else is off-the-shelf components!
![Page 22: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/22.jpg)
Reducing Load-times
• Users want responsive pages• Show something immediately
• Player’s won’t wait for your app to load• Stream non-critical content
![Page 23: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/23.jpg)
Avoid Remote Calls• Remote server calls are slow and unreliable
• Aim for no remote calls during load• Embed data into Javascript
Iframe
Inline-JS here
Flash ClientEmbed
![Page 24: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/24.jpg)
Caching Slow Calls
• Build in Facebook API caching within the social network wrapper
• Write a DB wrapper to generate and cache SQL
![Page 25: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/25.jpg)
FarmVille Page Profiler
Goal Catch and eliminate all un-cached Facebook and DB calls
![Page 26: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/26.jpg)
Fault Tolerance
• Servers do go down randomly• Build redundancy on all levels of server
architecture• Facebook is a dependency also
• Aim for DB-less & Facebook-less modes• “Defcon”-style error management
![Page 27: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/27.jpg)
Runtime Config
• How do we progressively keep the game running?• Create all features with kill-switches• Create a web dashboard to allow non-
technical folks to help out
![Page 28: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/28.jpg)
Is it still running?
• Notify your team when things break using server monitoring (Nagios/Munin)
• What to watch• Server load/traffic graphs• Memcache evictions
![Page 29: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/29.jpg)
Stats Tracking
• How do we know what users are seeing?• Have the client send back statistics
• Number of errors• Load-times
• Take a metrics driven approach to error handling
![Page 30: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/30.jpg)
Before Launching
• Get confidence that your stuff actually works• Perform load testing
• Social games are a marathon not a sprint• Sleep before launching!
![Page 31: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/31.jpg)
Questions?
![Page 32: Rapidly Building FarmVille: How we built and scaled a #1 Facebook game in 5 weeks (GDC 2010)](https://reader038.vdocuments.us/reader038/viewer/2022103110/54b613674a79598d038b45d8/html5/thumbnails/32.jpg)
Thank You!