couchconf-bangalore-couchbase 2.0-in-production
TRANSCRIPT
1
Couchbase Server 2.0 in Production
Frank Weigel, Director Product Management, Couchbase Inc.
2
Couchbase Server 2.0: Overview
• Membase + CouchDB• Managed memory caching layer = super high performance• Clustering and online data redistribution (Rebalancing)• Indexing and Querying via JSON Map-Reduce• New SDK’s and client libraries
• Developer Preview available today!• http://www.couchbase.com/downloads
=Simple. Fast. Elastic.
3
www.facebook.com/farm_town_wars
Web Servers
Load Balancer
Couchbase Servers
Let’s build a social game…
4
COUCHBASE SERVER 2.0 IN PRODUCTION: DEV/OPS LIFECYCLE
Initial Setup
View Development Sizing
Client setup
Monitor Grow
Backup/Restore
Upgrade
Deploy
MaintainFailures
Test
5
Couchbase Server 2.0 in Production: Initial Setup
Extremely easy to get up and running:• RPM/deb/OSX/exe installation• Simple Web UI and setup wizard
6
Couchbase Server 2.0 in Production: Data and Indexes
• Data goes “in and out” via the Memcached protocol
• Queries/indexes are created and accessed via an HTTP protocol
• Couchbase-provided SDK’s expose a single API to the developer and abstract the actual traffic
7
Farm Town Wars App Code
Applic
ati
on s
erv
er
Couch
base
Serv
er
Coucbase Javaclient library
Query API
Couchbase
11210
(“smart”) library
Farm Town Wars App Code
Applic
ati
on s
erv
er
Couch
base
Serv
er
Couchbase PHP Client LIbrary
Moxi (Couchbase proxy)
Query API
Couchbase
11210
Client-side Moxi
OR59845984
Couchbase Server 2.0 in production: Client-side Deployment
8
Couchbase Server 2.0 in Production: Data and Indexes
Example JSON values:
User_id:{ “farms”: [“one_farm”, “two_farm”] “animals”: [{ “pig”:”pink”, “sheep”:”black”, “chicken”:”yellow” }] “enemies”: [“you”, “U2”]}
Example view:
function (doc) { for(var i in doc.animals) if(doc.animals[i].sheep=="black") emit(doc._id)}+
=List of black sheep owners
9
Couchbase Server 2.0 in Production: Data and Indexes
• Indexes/views are based on incremental map-reduce:• Indexes are updated with incremental changes (not
batch)
• View processing is done per-node:• Parallel processing on subset of data• Couchbase provides scatter-gather aggregation
10
Couchbase Server 2.0 in Production: Data and Indexes
• Views are “developed” off of a random (or specific) sampling of the overall dataset and then deployed• Faster• Less load on system
• Updated views can be applied without rescanning entire dataset
11
COUCHBASE SERVER 2.0 IN PRODUCTION: DEV/OPS LIFECYCLE
Sizing
Client setup
Deploy
12
Couchbase Server 2.0 in production: Sizing
Sizing Question: How many nodes do I need?
Considerations:RAMDiskNetworkData Safety
13
Couchbase Server 2.0 in production: Sizing
Sizing Question: How many nodes do I need?
Considerations:RAM• Metadata• Working set• Buffer/overhead
Active+Replica
14
Couchbase Server 2.0 in production: Sizing
500,000 keys to begin with: -20 bytes in length -average value size of 2k -metadata of about 150 bytes per key
~1Gb to store active data, an extra 1Gb to store replica data
Adding in some headroom: Give 3GB RAM to Couchbase to start and grow with
=
15
Couchbase Server 2.0 in production: Sizing
Sizing Question: How many nodes do I need?
Considerations:Disk:• Sustained write rate• Index generation (space and IO)• Append-only B-Tree
16
Couchbase Server 2.0 in production: Sizing
Sizing Question: How many nodes do I need?
Considerations:Network:• Client traffic• Replication• Rebalancing
17
Couchbase Server 2.0 in production: Sizing
Sizing Question: How many nodes do I need?
Considerations:Data Safety
1 node = single point of failure (bad)2 nodes = 1 replica copy (better)3+ nodes = 1 replica copy AND data/load distribution
18
Couchbase Server 2.0 in production: Sizing
Sizing Question: How many nodes do I need?
Considerations:RAMDisk NetworkData Safety
19
COUCHBASE SERVER 2.0 IN PRODUCTION: DEV/OPS LIFECYCLE
Monitor Grow
Backup/Restore
UpgradeMaintainFailures
20
Couchbase Server 2.0 in production: Monitoring
IMMENSE amount of information available
-Real-time traffic graphs-REST API accessible-Per bucket, per node and aggregate statistics-Application and inter-node traffic-RAM <-> Disk-Inter-system timing
21
22
Couchbase Server 2.0 in production: Growth
• Going from 1 hundred users to 2 million users…– RAM usage is growing:
• Ejecting data to and fetching data from disk• Resident item ratios decreasing – might impact failover
– Disk write queue grows higher than usual
• Need to add a few more nodes...• Now we have more RAM and more disk throughput
without any downtime
23
Couchbase Server 2.0 in production: General Maintenance
• Persistence is CouchDB:– Append-only B-tree– Reliability and data integrity– Constantly growing on-disk files
• Compaction is the answer:– Automatic– “Fragmentation” data– Scheduled compaction– Per-vbucket database (saves disk space and time)
24
• Backup/Restore:– Backup is as simple as a file-level copy (thanks CouchDB!)– Server will automatically “warmup” from disk files upon reboot
• Upgrade:– Add nodes of new version, rebalance…– Remove nodes of old version, rebalance…– Done!– No disruption
Couchbase Server 2.0 in production: General Maintenance
25
Couchbase Server 2.0 in production: Failures
• Failures happen:– Hardware– Network– Bugs
• Failover to replica data for immediate access
• Remove and rebalance “malfunctioning” node
(Demo)
26
COUCHBASE SERVER 2.0 IN PRODUCTION: DEV/OPS LIFECYCLE
Initial Setup
View Development Sizing
Client setup
Monitor Grow
Backup/Restore
Upgrade
Deploy
MaintainFailures
Test
