![Page 1: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/1.jpg)
GAMES FOR THE MASSESSCALING RAILS TO THE EXTREME
EuRuKo 2011 in Berlin
![Page 2: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/2.jpg)
Who is that guy?
Jesper Richter-‐Reichhelm
@jrirei
Berlin, Germany
Head of Engineering @ wooga
![Page 3: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/3.jpg)
Wooga develops social games
![Page 4: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/4.jpg)
Wooga has dedicated game teams
PHPSQLCloud
RubySQL + Redis
Cloud
RubyRedis
Bare Metal
![Page 5: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/5.jpg)
Wooga has dedicated game teams
PHPSQLCloud
RubySQL + Redis
Cloud
RubyRedis
Bare Metal
![Page 6: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/6.jpg)
Both games are similar in their setup
Flash client Ruby backend
![Page 7: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/7.jpg)
Both games have similar problems
High traffic
• 200+ million h=p requests / day
High DB throughput
• 100.000 DB operaDons / second
Hardest problems to solve
• DB migraDons, DB failover, DB performance, ...
![Page 8: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/8.jpg)
What’s best?
Big cloud vs. bare metal
Hybrid (SQL + Redis) vs. Redis only
In-‐memory vs. DB hierarchy
Conclusion: So what’s best?
![Page 9: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/9.jpg)
Big cloud vs. bare metal
Big cloud: Amazon EC2
• new server in 5 minutes
• automated deployment
• load based setup
![Page 10: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/10.jpg)
Big cloud vs. bare metal
Big cloud: Amazon EC2
• new server in 5 minutes
• automated deployment
• load based setup
Bare metal: Hetzner
• new server in 2-‐14 days
• automated deployment
![Page 11: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/11.jpg)
What kind of app server are available?
XL CPU instance
• 8 virtual cores
• 8 GB RAM
![Page 12: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/12.jpg)
Bare metal servers are more powerful
XL CPU instance
• 8 virtual cores
• 8 GB RAM
EQ10 server
• 12 “processing threads”
• 24 GB RAM
![Page 13: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/13.jpg)
Bare metal servers are more powerful
XL CPU instance
• 8 virtual cores
• 8 GB RAM
=> 390,-‐ € / month
EQ10 server
• 12 “processing threads”
• 24 GB RAM
=> 200,-‐ € / month
![Page 14: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/14.jpg)
And how is the network?
min ping: 0.2 -‐ 0.3 ms min ping: 0.2 ms
![Page 15: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/15.jpg)
Want a fast network? Get your own!
min ping: 0.2 -‐ 0.3 ms
max ping: 7.8 -‐ 59 ms
avg ping: 0.8 -‐ 4.4 ms
min ping: 0.2 ms
max ping: 0.3 ms
avg ping: 0.2 ms
![Page 16: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/16.jpg)
Want a fast network? Get your own!
min ping: 0.2 -‐ 0.3 ms
max ping: 7.8 -‐ 59 ms
avg ping: 0.8 -‐ 4.4 ms
22 DB calls per hfp calls
min ping: 0.2 ms
max ping: 0.3 ms
avg ping: 0.2 ms
saved 13 -‐ 92 ms latency
![Page 17: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/17.jpg)
What’s befer? Depends on your needs!
Big cloud is great if you
• don’t know your load
• scale up / down quickly
![Page 18: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/18.jpg)
What’s befer? Depends on your needs!
Big cloud is great if you
• don’t know your load
• scale up / down quickly
Bare metal is great if you
• want to save money
• need stable performance
![Page 19: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/19.jpg)
What’s befer? Depends on your needs!
Big cloud is great if you
• don’t know your load
• scale up / down quickly
Say: “Don’t worry”
Bare metal is great if you
• want to save money
• need stable performance
Say: “Think ahead”
![Page 20: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/20.jpg)
What’s best?
Big cloud vs. bare metal
Hybrid (SQL + Redis) vs. Redis only
In-‐memory vs. DB hierarchy
Conclusion: So what’s best?
![Page 21: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/21.jpg)
Hybrid (SQL + Redis) vs. Redis
Hybrid (SQL + Redis)
• started with MySQL
• hit a wall due to IO limits
• introduced Redis later
Redis only
![Page 22: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/22.jpg)
If MySQL is a truck, ...
Fast enough for reads
Can store on disk
Robust replicakon
h=p://www.flickr.com/photos/erix/245657047/
![Page 23: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/23.jpg)
If MySQL is a truck, Redis is a Ferrari
Fast enough for reads
Can store on disk
Robust replicakon
Super fast reads/writes
Out of memory => dead
Fragile replicakon
h=p://www.flickr.com/photos/erix/245657047/
![Page 24: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/24.jpg)
Hybrid (SQL + Redis) vs. Redis
Hybrid (SQL + Redis)
• started with MySQL
• hit a wall due to IO limits
• introduced Redis later
Redis only
![Page 25: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/25.jpg)
Hybrid (SQL + Redis) vs. Redis
Hybrid (SQL + Redis)
• started with MySQL
• hit a wall due to IO limits
• introduced Redis later
MySQL has its hard limits
Redis only
Works fine if done right
![Page 26: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/26.jpg)
How many tables to store a user in?
22 MySQL tables
7 Redis hashes
![Page 27: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/27.jpg)
How many tables to store a user in?
22 MySQL tables
7 Redis hashes
1 Redis Hash
![Page 28: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/28.jpg)
How many tables to store a user in?
22 MySQL tables
7 Redis hashes
Easy to implement
Faster for small changes
1 Redis Hash
Less top level keys
Export and import is easy
![Page 29: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/29.jpg)
What’s best?
Big cloud vs. bare metal
Hybrid (SQL + Redis) vs. Redis only
In-‐memory vs. DB hierarchy
Conclusion: So what’s best?
![Page 30: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/30.jpg)
Redis storage type
Redis works in-‐memory
Dumps slow down master
• Dumping on slaves only
![Page 31: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/31.jpg)
Redis storage type
Redis works in-‐memory
Dumps slow down master
• Dumping on slaves only
Redis used vm
Dumps to disk impossible
• Own system to store JSON files to disk
![Page 32: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/32.jpg)
Redis storage type
Redis works in-‐memory
Dumps slow down master
• Dumping on slaves only
Redis used vm
Dumps to disk impossible
• Own system to store JSON files to disk
Redis became a cache only
![Page 33: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/33.jpg)
Redis storage type
Redis works in-‐memory
Dumps slow down master
• Dumping on slaves only
Redis now uses diskstore
![Page 34: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/34.jpg)
Redis storage type
Redis works in-‐memory
Dumps slow down master
• Dumping on slaves only
Redis now uses diskstore
• Experimental branch
• Stores async. to disk
![Page 35: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/35.jpg)
Redis storage type
Redis works in-‐memory
Dumps slow down master
• Dumping on slaves only
Redis now uses diskstore
• Experimental branch
• Stores async. to disk
Redis became a DB again
![Page 36: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/36.jpg)
There are many inackve users
hourly
daily
monthly
lifetime
0M 2M 4M 6M
6,0M
3,0M
0,4M
0,1Mhourly
daily
monthly
lifetime
0M 8M 16M 24M
20,8M
8,0M
1,3M
0,3M
![Page 37: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/37.jpg)
Not all users play at the same day
In-‐memory
100% of users in RAM
DB hierarchy
• Redis in-‐memory
• Redis diskstore on disk
3% of users in RAM
![Page 38: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/38.jpg)
What’s best?
Big cloud vs. bare metal
Hybrid (SQL + Redis) vs. Redis only
In-‐memory vs. DB hierarchy
Conclusion: So what’s best?
![Page 39: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/39.jpg)
So, what didn’t work?
Big cloud
Hybrid (SQL + Redis)
In-‐memory “only”
Bare metal
Redis only
DB hierarchy
![Page 40: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/40.jpg)
So, what didn’t work?
Big cloud
Hybrid (SQL + Redis)
In-‐memory “only”
Bare metal
Redis only
DB hierarchy
![Page 41: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/41.jpg)
Let’s dig a bit deeper
1,300,000 daily users
5000 hfp calls/s peak
100 ms response kme
35-‐70 servers
400,000 daily users
2500 hfp calls/s peak
10 ms response kme
11 servers
![Page 42: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/42.jpg)
Hoskng costs / DAU compared
HosDng
5x higher
![Page 43: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/43.jpg)
There are 2 main reasons for higher cost
Databases Other
DB: 20x higher
Other: 3x higher
![Page 44: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/44.jpg)
There are 2 main reasons for higher cost
Databases Other
DB: 20x higher(both RAM and cost)
Other: 3x higher
No DB hierarchy!
![Page 45: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/45.jpg)
So, what didn’t work?
Big cloud
Hybrid (SQL + Redis)
In-‐memory “only”
Bare metal
Redis only
DB hierarchy
![Page 46: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/46.jpg)
So, what didn’t work?
Big cloud
Hybrid (SQL + Redis)
In-‐memory “only”
Bare metal
Redis only
DB hierarchy
depends ...
![Page 47: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/47.jpg)
So, what didn’t work?
Big cloud
Hybrid (SQL + Redis)
In-‐memory “only”
Bare metal
Redis only
DB hierarchy
depends ...
never
again!
![Page 48: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/48.jpg)
So, what didn’t work?
Big cloud
Hybrid (SQL + Redis)
In-‐memory “only”
Bare metal
Redis only
DB hierarchy
depends ...
never, ever,
ever again!
![Page 49: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/49.jpg)
Next game will do things befer
Start in cloud and then go metal?
Redis only
DB hierarchy
![Page 50: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/50.jpg)
BTW, wooga is hiring!
Q & A
slideshare.net/wooga
Jesper Richter-‐Reichhelm@jrirei
wooga.com/jobs
![Page 51: Games for the Masses: Scaling Rails to the Extreme](https://reader034.vdocuments.us/reader034/viewer/2022042714/54b79e664a795997768b45a3/html5/thumbnails/51.jpg)
BTW, wooga is hiring!
Thank you
slideshare.net/wooga
Jesper Richter-‐Reichhelm@jrirei
wooga.com/jobs