architecture evolution at wooga (aws cloud computing for developers,)

ARCHITECTURE  EVOLUTIONAT  WOOGA

How  does  your  company  learn  new  things?

Jesper  Richter-­‐Reichhelm,  @jrirei

Our  games  all  look  the  same

Flash  client Backend

But  the  scale  is  interesFng

14  billion  requests  /  month

But  the  scale  is  interesFng

14  billion  requests  /  month

But  the  scale  is  interesFng

14  billion  requests  /  month

>100,000  DB  operaFons  /  second

But  the  scale  is  interesFng

14  billion  requests  /  month

>100,000  DB  operaFons  /  second

>50,000  DB  updates  /  second

Architecture  EvoluFon  at  Wooga

The  Start

The  Next  Step

Best  of  Two  Worlds

Oct  2009:  Monster  World  based  on  Ruby

Good  code  quality

Easy  to  understandEasy  to  testEasy  to  refactor

Oct  2009:  Monster  World  based  on  Ruby

Good  code  quality

Easy  to  understandEasy  to  testEasy  to  refactor

Oct  2009:  Monster  World  based  on  Ruby

Good  code  quality

Easy  to  understandEasy  to  testEasy  to  refactor

A  basic  setup  using  sharding  worked  fine

app app app

lb

MySQL

MySQL

slave slave

A  basic  setup  using  sharding  worked  fine

app app appapp app

lb

MySQL

MySQL

slave slave

A  basic  setup  using  sharding  worked  fine

app app appapp appapp app app app

lb

MySQL

MySQL

slave slave

250K  daily  users

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

Life  was  good

250K  daily  users

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

Life  was  good NO  MORE

Early  sharding  hell:  8  master  and  8  slaves

app app app app appapp app

app app app app app app app app app

appapp

lb

MySQL

MySQL

slaveslave

Early  sharding  hell:  8  master  and  8  slaves

app app app app appapp app

app app app app app app app app app

appapp

lb

MySQL

MySQL

slave

MySQL

MySQL

slaveslave slave

Early  sharding  hell:  8  master  and  8  slaves

app app app app appapp app

app app app app app app app app app

appapp

lb

MySQL

MySQL

slave

MySQL

MySQL

MySQL

MySQL

slave slaveslave slave

MySQL

MySQL

slaveslave slave

At  500K  daily  users  we  were  at  a  dead  end

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

OUCH!

http://www.flickr.com/photos/billue_the_bear/

Big  and  staFc  data  in  MySQL,  rest  goes  to  Redis

MySQL

Big  and  staFc  data  in  MySQL,  rest  goes  to  Redis

MySQL Redis

Big  and  staFc  data  in  MySQL,  rest  goes  to  Redis

256  GB  data

10%  writes

MySQL Redis

Big  and  staFc  data  in  MySQL,  rest  goes  to  Redis

60  GB  data

50%  writes

256  GB  data

10%  writes

MySQL Redis

Redis  saved  the  day

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

Redis  saved  the  day

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

We  now  have  more  than  2  million  users  /  day

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

We  now  have  more  than  2  million  users  /  day

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

AWS  servicedisrupFonin  Ireland

10  single-­‐points-­‐of-­‐failure  -­‐  no  fun  at  all!

app app app app app app app app app app app appapp

app app app app app app app app app app app appapp

app app app app app app app app app app app appapp

lb lb

redis redisMySQL

MySQL

slave slaveslave slave

10  single-­‐points-­‐of-­‐failure  -­‐  no  fun  at  all!

app app app app app app app app app app app appapp

app app app app app app app app app app app appapp

app app app app app app app app app app app appapp

lb lb

redis redisMySQL

MySQL

slave slave

MySQL

slave

MySQL

MySQL

slave slave slave slave

10  single-­‐points-­‐of-­‐failure  -­‐  no  fun  at  all!

app app app app app app app app app app app appapp

app app app app app app app app app app app appapp

app app app app app app app app app app app appapp

lb lb

redis redisMySQL

MySQL

slave slave

redis redis

slave slave

redis

slave

MySQL

slave

MySQL

MySQL

slave slave slave slave

OUCH!

http://www.flickr.com/photos/wolfsavard/

Architecture  EvoluFon  at  Wooga

The  Start:  Ruby  +  AutomaFon

The  Next  Step

Best  of  Two  Worlds

Stateless  servers  and  DBs

Server Database

Stateless  servers  and  DBs

Server Database

Stateless  servers  and  DBs

Server Database

Stateless  servers  and  DBs

Server Database

Stateless  servers  and  DBs

Server Database

Stateless  servers  and  DBs

Server Database

“Stateless  applica,on  servers  guarantee  one  thing:

“Stateless  applica,on  servers  guarantee  one  thing:

The  data  is  never

where  you  need  it!”Paolo  Negri,  Developer  @  Wooga

Stateful  servers  and  DBs

Server Database

Stateful  servers  and  DBs

Server Database

Stateful  servers  and  DBs

Server Database

Stateful  servers  and  DBs

Server Database

One  Game  Session

Stateful  servers  and  DBs

Server Database

One  Game  Session

Oct  2010:  Magic  Land  based  on  stateful  server

If  DBs  are  the  problem

Don’t  use  themStore  state  in  serverNeed  to  be  robust

Oct  2010:  Magic  Land  based  on  stateful  server

If  DBs  are  the  problem

Don’t  use  themStore  state  in  serverNeed  to  be  robust

Oct  2010:  Magic  Land  based  on  stateful  server

If  DBs  are  the  problem

Don’t  use  themStore  state  in  serverNeed  to  be  robust

Stateful  servers  are  not  as  hard  as  you  think

session

Stateful  servers  are  not  as  hard  as  you  think

sessionsessionsessionsession

Stateful  servers  are  not  as  hard  as  you  think

Server

sessionsessionsessionsession

Stateful  servers  are  not  as  hard  as  you  think

Server

sessionsessionsessionsession

S3

Stateful  servers  are  not  as  hard  as  you  think

Server

sessionsessionsessionsession

S3

Stateful  servers  are  not  as  hard  as  you  think

Server

sessionsessionsessionsession

S3

Stateful  servers  are  not  as  hard  as  you  think

Server

sessionsessionsessionsession

S3

Stateful  servers  are  not  as  hard  as  you  think

Server

sessionsessionsessionsession

S3

Stateful  servers  are  not  as  hard  as  you  think

Server

sessionsessionsessionsession

S3

Stateful  servers  are  not  as  hard  as  you  think

Server

sessionsessionsessionsession

Server

sessionsessionsessionsession

Server

sessionsessionsessionsession

S3

With  stateful  server  the  DB  is  less  used

0

7,500

15,000

22,500

30,000

database  operations  /  sec

Ruby  Stateless Erlang  Stateful

With  stateful  server  the  DB  is  less  used

0

7,500

15,000

22,500

30,000

database  operations  /  sec

Ruby  Stateless Erlang  Stateful

700

Deploying  with  a  stateful  server

In  order  to  bring  up  a  new  version

Deploying  with  a  stateful  server

In  order  to  bring  up  a  new  version

Just  deploy  it

Hot  code  replacement  is  great!

There  are  even  more  advantages

Faster  than  Ruby  (5,000  rps  /  node)-­‐ CPU  bound

There  are  even  more  advantages

Faster  than  Ruby  (5,000  rps  /  node)-­‐ CPU  bound

Very  few  SPOFs-­‐ ...  and  those  are  easy  to  recover

There  are  even  more  advantages

Faster  than  Ruby  (5,000  rps  /  node)-­‐ CPU  bound

Very  few  SPOFs-­‐ ...  and  those  are  easy  to  recover

TransacFonal  logic-­‐ Invariants  instead  of  explicit  error  handling

NICE!

http://www.flickr.com/photos/aigle_dore/

Architecture  EvoluFon  at  Wooga

The  Start:  Ruby  +  AutomaFon

The  Next  Step:  Erlang  +  S3

Company  Values

We’ve  learned  to  value

A  good  value  system

Small  teams                              over   Big  teams

We’ve  learned  to  value

A  good  value  system

Small  teams                              over   Big  teams

We’ve  learned  to  value

CollaboraFon                        over Compe??on

A  good  value  system

Small  teams                              over   Big  teams

We’ve  learned  to  value

Generalists                                over Specialists

CollaboraFon                        over Compe??on

A  good  value  system

Small  teams                              over   Big  teams

We’ve  learned  to  value

Generalists                                over Specialists

Effort  reducFon                over Cost  reduc?on

CollaboraFon                        over Compe??on

A  good  value  system

Small  teams                              over   Big  teams

We’ve  learned  to  value

Generalists                                over Specialists

InnovaFon                                  over Risk  mi?ga?on

Effort  reducFon                over Cost  reduc?on

CollaboraFon                        over Compe??on

A  good  value  system

It works!

It works!

Be fast, be bold!

QuesFons?

Jesper  Richter-­‐Reichhelm@jrirei

slideshare.net/woogawooga.com/jobs