sdec2011 using couchbase for social game scaling and speed

2

Chiyoung Seo, Couchbase Inc. Matt Ingenthron, Couchbase Inc.

USING COUCHBASE FOR SOCIAL GAME SCALING AND SPEED

3

•  Introduc-on •  What is Couchbase Server?

–  Simple, Fast, Elas-c –  Technology Overview (Architecture, data flow, rebalancing)

•  Tribal Crossing Inc: Animal Party –  Challenges before Couchbase

•  Original Architecture –  Why Couchbase?

•  Simplicity •  Performance •  Flexibility

–  Deploying Couchbase •  New Architecture •  EC2 •  Data Model •  Accessing data in Couchbase

•  Product Roadmap •  Q&A

Agenda

4

•  Membase and CouchOne have merged to form Couchbase Inc. (headquartered in Silicon Valley)

•  Team –  Brings together the creators and core contributors of Memcached, Membase and CouchDB technologies

–  Doubles technical team size, accelerates roadmaps by over a year •  Products

–  Couchbase Server (Formerly Membase) –  Couchbase Single Server –  Mobile Couchbase (iPhone and Android)

•  Technology –  Most mature, reliable and widely deployed NoSQL technologies –  Fully featured, open source document datastore –  First complete, end-‐to-‐end NoSQL database product

Couchbase Inc.

5

Modern Interactive Web Application Architecture

Application Scales Out Just add more commodity web servers

Database Scales Up Get a bigger, more complex server

www.facebook.com/animalparty

Web Servers

Rela-onal Database

Load Balancer

-‐ Expensive and disrup-ve sharding -‐ Doesn’t perform at Web Scale

6

Couchbase Server is a distributed database

Couchbase Servers

Web applica-on server

Applica-on user

Couchbase Web Console

7

Couchbase data layer scales like application logic tier Data layer now scales with linear cost and constant performance.

Application Scales Out Just add more commodity web servers

Database Scales Out Just add more commodity data servers

Scaling out flattens the cost and performance curves.

Couchbase Servers

www.facebook.com/animalparty

Web Servers Load Balancer

Horizontally scalable, schema-‐less, auto-‐sharding, high-‐performance at Web Scale

8

Couchbase Server is Simple, Fast, Elas-c

•  Five minutes or less to a working cluster –  Downloads for Windows, Linux and OSX –  Start with a single node –  One bucon press joins nodes to a cluster

•  Easy to develop against –  Just SET and GET – no schema required –  Drop it in. 10,000+ exis-ng applica-ons

already “speak Couchbase” (via memcached) –  Prac-cally every language and applica-on

framework is supported, out of the box

•  Easy to manage –  One-‐click failover and cluster rebalancing –  Graphical and programma-c interfaces –  Configurable aler-ng

9


•  Predictable –  “Never keep an applica-on wai-ng” –  Quasi-‐determinis-c latency and throughput

•  Low latency –  Built-‐in Memcached technology –  Auto-‐migra-on of hot data to lowest latency

storage technology (RAM, SSD, Disk) –  Selectable write behavior – asynchronous,

synchronous (on replica-on, persistence)

•  High throughput –  Mul--‐threaded –  Low lock conten-on –  Asynchronous wherever possible –  Automa-c write de-‐duplica-on

10


•  Zero-‐down-me elas-city –  Spread I/O and data across commodity

servers (or VMs) –  Consistent performance with linear cost –  Dynamic rebalancing of a live cluster

•  All nodes are created equal –  No special case nodes –  Clone to grow

•  Extensible –  Change feeds –  Real-‐-me map-‐reduce –  RESTful interface for management

Couchbase Web Console

11

Proven at Small, and Extra Large Scale

•  Leading cloud service (PAAS) provider

•  Over 150,000 hosted applications •  Couchbase Server serving over

6,200 Heroku customers

•  Social game leader – FarmVille, Mafia Wars, Empires and Allies, Café World, FishVille

•  Over 230 million monthly users •  Couchbase Server is the primary

database behind key Zynga properties

12

Customers and Partners

Customers (par-al lis-ng) Partners

13

moxi

11211 11210

memcached protocol listener/sender

Couchbase Storage Engine

engine interface

memcapable 1.0 memcapable 2.0

21100 – 21199 4369 8091

hcp RE

ST m

anagem

ent A

PI/W

eb UI

Heartbeat

Process m

onito

r

Glob

al singleton supe

rviso

r

Confi

gura-o

n manager

on each node

Erlang/OTP

Rebalance orchestrator

Nod

e he

alth m

onito

r

one per cluster

vBucket state and

replica-

on m

anager

HTTP distributed erlang erlang port mapper

Data Manager Cluster Manager

Couchbase Server Architecture

14

moxi

11211 11210


engine interface

memcapable 1.0 memcapable 2.0

21100 – 21199 4369 8091

hcp RE

ST m

anagem

ent A

PI/W

eb UI

Heartbeat

Process m

onito

r

Glob

al singleton supe

rviso

r

Confi

gura-o

n manager

on each node

Erlang/OTP

Rebalance orchestrator

Nod

e he

alth m

onito

r

one per cluster

vBucket state and

replica-

on m

anager

HTTP distributed erlang erlang port mapper

Couchbase Server Architecture


15

Couchbase “write” Data Flow – applica-on view

User ac-on results in the need to change the VALUE of KEY

Applica-on updates key’s VALUE, performs SET opera-on

Couchbase client hashes KEY, iden-fies KEY’s master server SET request sent over

network to master server

Couchbase replicates KEY-‐VALUE pair, caches it in memory and stores it to disk

1

2

3 4

5

16

Couchbase Data Flow – under the hood

Listener-‐Sender

DiskDisk Disk

RAM*

mem

base storage e

ngine

SSDSSD SSD

Listener-‐Sender

DiskDisk Disk

RAM*

mem

base storage e

ngine

SSDSSD SSD

SET request arrives at KEY’s master server

Listener-‐Sender

Master server for KEY Replica Server 2 for KEY Replica Server 1 for KEY

2 2

1 SET acknowledgement returned to applica-on 3

Disk Disk Disk

RAM*

Couchb

ase storage en

gine

SSD SSD SSD

2

4

17

Elas-city -‐ Rebalancing

vBucket 1 vBucket 2 vBucket 3


Node 1 Node 2 Node 3

vBucket 1 vBucket 2

vBucket 3 vBucket 4

vBucket 5 vBucket 6

vBucket 7 vBucket 8

vBucket 9 vBucket 10

vBucket 11 vBucket 12

Before •  Adding Node 3 •  Node 3 is in pending state •  Clients talk to Node 1,2 only

AOer •  Node 3 is balanced •  Clients are reconfigured to talk to

Node 3

During •  Rebalancing orchestrator recalculates

the vBucket map (including replicas) •  Migrate vBuckets to the new server •  Finalize migra-on



Pending state





Rebalancing

vBucket migrator vBucket migrator

Client

18

Data buckets are secure Couchbase “slices”

Couchbase data servers

In the data center

Web applica-on server

Applica-on user

On the administrator console

Bucket 1

Bucket 2

Aggregate Cluster Memory and Disk Capacity

19

•  Support large-‐scale analy-cs on applica-on data by streaming data from Couchbase to Hadoop –  Real-‐-me integra-on using Flume –  Batch integra-on using Sqoop

•  Examples –  Various game sta-s-cs (e.g., monthly / daily / hourly rankings) –  Analyze game pacerns from users to enhance various game metrics

Couchbase and Hadoop Integra-on


engine interface


TAP

Flume

Sqoop

20








Agenda

21

Common steps on scaling up database:

●  Tune queries (indexing, explain query)

●  Denormaliza-on

●  Cache data (APC / Memcache)

●  Tune MySQL configura-on

●  Replica-on (read slaves)

Where do we go from here to prepare for the scale of a successful social game?

Tribal Crossing: Challenges

22

●  Write-‐heavy requests –  Caching does not help –  MySQL / InnoDB limita-on (Percona)

●  Need to scale dras-cally over night –  My Polls – 100 to 1m users over a weekend

●  Small team, no dedicated sysadmin –  Focus on what we do best – making games

●  Keeping cost down

Tribal Crossing: Challenges

23

●  MySQL with master-‐to-‐master replica-on and sharding

–  Complex to setup, high administra-on cost –  Requires applica-on level changes

●  Cassandra –  High write, but low read throughput –  Live cluster reconfigura-on and rebalance is quite complicated –  Eventual consistency gives too much burden to applica-on

developers

●  MongoDB –  High read/write, but unpredictable latency –  Live cluster rebalance for exis-ng nodes only –  Eventual consistency with slave nodes

Tribal Crossing: “Old” Architecture and Op-ons

24

●  SPEED, SPEED, SPEED

●  Immediate consistency

●  Interface is dead simple to use –  We are already using Memcache

●  Low sysadmin overhead

●  Schema-‐less data store

●  Used and Proven by big guys like Zynga

●  … and lastly, because Tribal CAN –  Bigger firms with legacy code base = hard to adapt –  Small team = ability to get on the cuung edge

Tribal Crossing: Why Couchbase Server?

25

●  But, there are some different challenges in using Couchbase (currently 1.7) to handle the game data:

–  No easy way to query data –  No transac-on / rollback

➔  Couchbase Server 2.0 resolves them by using CouchDB as the underlying database engine

●  Can this work for an online game? –  Break out of the old ORM / rela-onal paradigm! –  We are not handling bank transac-ons

Tribal Crossing: New Challenges With Couchbase

26

Couchbase Cluster

Web Server

Tribal Crossing: Deploying Couchbase in EC2

●  Basic produc-on environment setup

●  Dev/Stage environment – feel free to install Couchbase on your web server

Apache

Couchbase Couchbase

DNS Entry

Client-side Moxi

Cluster Mgmt. Requests

…

27


●  Amazon Linux AMI, 64-‐bit, EBS backed instance

●  Setup swap space ●  Install Couchbase’s Membase Server 1.7

●  Access web console http://<hostname>:8091

●  Start the new cluster with a single node

●  Add the other nodes to the cluster and rebalance

Couchbase Cluster

Web Server

Apache

Couchbase

DNS Entry

Client-side Moxi


… Couchbase

28


Moxi figures out which node in the cluster holds data for a given key.

●  On each web server, install Moxi proxy

●  Start Moxi by poin-ng it to the DNS entry you created

●  Web apps connect to Moxi that is running locally memcache->addServer(‘localhost’, 11211);

Couchbase Cluster

Web Server

Apache

Couchbase Couchbase

DNS Entry

Client-side Moxi


…

29

Use case -‐ simple farming game:

●  A player can have a variety of plants on their farm.

●  A player can add or remove plants from their farm.

●  A Player can see what plants are on another player's farm.

Tribal Crossing: Represen-ng Game Data in Couchbase

30

RepresenUng Objects

●  Simply treat an object as an associa-ve array

●  Determine the key for an object using the class name (or type) of the object and an unique ID

RepresenUng Object Lists

●  Denormaliza-on

●  Save a comma separated list or an array of object IDs


31

Player Object Key: 'Player1' Array ( [Id] => 1 [Name] => Shawn )


Plant Object Key: 'Plant201' Array ( [Id] => 201 [Player_Id] => 1 [Name] => Starflower ) PlayerPlant List

Key: 'Player1_PlantList' Array ( [0] => 201 [1] => 202 [2] => 204 )

32

●  No need to “ALTER TABLE”

●  Add new “fields” all objects at any -me –  Specify default value for missing fields –  Increased development speed

●  Using JSON for data objects though, owing to the ability to query on arbitrary fields in Couchbase 2.0

Tribal Crossing: Schema-‐less Game Data

33

Get all plants belong to a given player Request: GET /player/1/farm $plant_ids = couchbase->get('Player1_PlantList'); $response = array(); foreach ($plant_ids as $plant_id) { $plant = couchbase->get('Plant' . $plant_id); $response[] = $plant; } echo json_encode($response);

Tribal Crossing: Accessing Game Data in Couchbase

34

Give a player a new plant // Create the new plant $new_plant = array ( 'id' => 100, 'name' => 'Mushroom' ); $couchbase->set('Plant100', $new_plant); // Update the player plant list $plant_ids = $couchbase->get('Player1_PlantList'); $plant_ids[] = $new_plant['id']; $couchbase->set('Player1_PlantList', $plant_ids);

Tribal Crossing: Modifying Game Data in Couchbase

35

Concurrency issue can occur when mul-ple requests are working with the same piece of data. Solu-on: ●  CAS (check-‐and-‐set)

–  Client can know if someone else has modified the data while you are trying to update

–  Implement op-mis-c concurrency control

●  Locking (try/wait cycle) –  GETL (get with lock + -meout)

opera-ons –  Pessimis-c concurrency control

Tribal Crossing: Concurrency

36

●  Record object rela-onships both ways –  Example: Plots and Plants

●  Plot object stores id of the plant that it hosts ●  Plant object stores id of the plot that it grows on

–  Resolu-on in case of mismatch

●  Don't sweat the extra calls to load data in a one-‐to-‐many rela-onship

–  Use mul3Get –  We can s-ll cache aggregated results in a Memcache bucket if needed

Tribal Crossing: Data Rela-onship

37

Web Server

First migrated large or slow performing tables and frequently updated fields from MySQL to Couchbase

Tribal Crossing: Migra-ng to Couchbase Servers


engine interface


TAP

TAP Client

Apache + PHP

Client-‐side Moxi

Repor-ng Applica-ons

MySQL

38

Tribal Crossing: Deployment

39

Tribal Crossing: Deployment

40

•  Significantly reduced the cost incurred by scaling up database servers and managing them.

•  Achieved significant improvements in various performance metrics (e.g., read, write, latency, etc.)

•  Allowed them to focus more on game development and op-mizing key metrics

•  Plan to use real-‐-me MapReduce, querying, and indexing abili-es provided by the upcoming Elas-c Couchbase 2.0

Tribal Crossing: Conclusion

41








Agenda

42

•  Mobile to cloud data synchroniza-on •  Cross data center replica-on

Product Roadmap: Couchbase Server 2.0

…

Couchbase Single Server

US West Coast Data Center

Couchbase Server

…

Couchbase Single Server

US East Coast Data Center

Couchbase Server

CouchSync

CouchSync CouchSync

CouchSync

…… …

… …

CouchSync

43

•  Replace Sqlite-‐based storage engine with CouchDB •  Support indexing and querying on values •  Integrate real-‐-me MapReduce into Couchbase server •  SDK for Couchbase server

Product Roadmap: Couchbase Server 2.0

The world’s leading caching and clustering technology

The most reliable and full-‐featured document database

The fastest, most complete and most reliable database on the

planet

Membase Server 1.7 CouchDB 1.1 Couchbase Server 2.0

44

•  Community Edi-on –  Open source build –  Free forum support

•  Enterprise Edi-on –  Free for non-‐produc-on use –  Cer-fied, QA tested version of open source –  Case tracking and guaranteed SLA for produc-on environments

•  Partner in Korea –  N2M Inc. (hcp://www.n2m.co.kr)

Couchbase Product Download

45

Q&A Matt Ingenthron, Couchbase Inc.

([email protected], @ingenthr) Chiyoung Seo, Couchbase Inc.

([email protected], @chiyoungseo)