Justin Michaelsjustin@couchbase.com

Solution Architect

Couchbase 101 Server Fundamentals

Evolution of Couchbase

Founders were key contributors to memcached

Evolved into Membase, a distributed key-value cache

Original contributors to CouchDB re-wrote storage engine to create Couchbase Server

Couchbase Server is a cache and persistence engine in a single easy to scale distributed database.

Couchbase Server Core Principles

Easy Scalability

Consistent High Performance

Always On 24x365

Grow cluster without application changes, without downtime with a single click

Consistent sub-millisecond read and write response times

with consistent high throughput

No downtime for software upgrades, hardware maintenance, etc.

Flexible Data Model

JSON Anywhere document model with no fixed schema.

INSTALL SERVER

Install on Amazon via AMI

Download Couchbasewww.couchbase.com/download

1. Run what you downloaded

2. Installs to /opt/couchbase

3. Access <ip>:8091 via http

*CLI and scripted deployments available*

Let’s see a deployment

ARCHITECTURE

Couchbase Node

Hea

rtb

eat

Pro

cess

mo

nit

or

Glo

bal

sin

glet

on

su

per

viso

r

Co

nfi

gura

tio

n m

anag

er

httpR

EST

man

age

me

nt

AP

I/W

eb

UI

HTTP

8091Erlang port mapper

4369Distributed Erlang

21100 - 21199

Erlang/OTP

storage interface

Couchbase EP Engine

11210Memcapable 2.0

Moxi

11211Memcapable 1.0

Memcached

New Persistence Layer

8092Query API

Qu

ery

En

gin

e

Data Manager Cluster ManagerServer/Cluster

Management & Communication

(Erlang)

RAM Cache, Indexing & Persistence

Management(C)

Couchbase Node

New Persistence Layer

storage interface

Couchbase EP Engine

11210Memcapable 2.0

Moxi

11211Memcapable 1.0

Object-level Cachememcached

Disk Persistencecouchstore

8092Query API

Qu

ery

En

gin

e

HTTP

8091Erlang port mapper

4369Distributed Erlang

21100 - 21199

Hea

rtb

eat

Pro

cess

mo

nit

or

Co

nfi

gura

tio

n m

anag

er

http

RES

T m

anag

em

en

t A

PI/

We

b U

I

Erlang/OTP

Data Manager

Cluster Manager

Couchbase Data• Key is any string up to 250 Bytes

• Similar to primary key

• Must be unique within a bucket

• ID based document lookup is extremely fast

• Value can be up to 20mb

• Simple Datatypes: strings, numbers, datetime, boolean, binary

• Complex Datatypes: dictionaries/hashes, arrays/lists, can be stored in JSON format

• JSON: Automatically checked by the server

• BLOB: Array of bytes

• Cluster Aware Clients (Smartclient SDK)

PythonRuby

Couchbase Data

meta{

“id”: “u::tesla”,“rev”: “1-0002bce0000000000”,“flags”: 0,“expiration”: 0,“type”: “json”

}

document{

“sellerid”: 123456,“type”: “car”,“style”: “sedan”,“year”: 2013,“trim”: “performance”,“model”: “s”

}

Server stores metadata with each key/value pair

(document)

Unique and Kept in RAM

Document Value

Most Recent In RAM And Persisted To Disk

CLUSTER MECHANICS

33 2

Write (‘set’) Operation2

Managed Cache

Dis

k Q

ueu

e

Disk

Replication Queue

App Server

Couchbase Server Node

Doc 1Doc 1

Doc 1

To other nodeManaged Cache

Node

GET

Do

c 1

33 2

Read (‘get’) Operation2

Dis

k Q

ueu

e

Replication Queue

App Server

Couchbase Server Node

Doc 1

Doc 1Doc 1

Managed Cache

Disk

To other nodeManaged Cache

Node

33 2

Cache Ejection2

Dis

k Q

ueu

e

Replication Queue

App Server

Couchbase Server Node

Doc 1

Doc 6Doc 5Doc 4Doc 3Doc 2

Doc 1

Doc 6 Doc 5 Doc 4 Doc 3 Doc 2

Managed Cache

Disk

To other nodeManaged Cache

Node

33 2

Cache Miss2

Dis

k Q

ueu

e

Replication Queue

App Server

Couchbase Server Node

Doc 1

Doc 3Doc 5 Doc 2Doc 4

Doc 6 Doc 5 Doc 4 Doc 3 Doc 2

Doc 4

GET

Do

c 1

Doc 1

Doc 1

Managed Cache

Disk

To other nodeManaged Cache

Node

Cluster MechanicsFailover

User Configured Replica Count = 1

SERVER 4

Replica

Active

App Server 1

COUCHBASE Client Library

CLUSTER MAP

COUCHBASE Client Library

CLUSTER MAP

App Server 2

Couchbase Server Cluster

Active

SERVER 1

Doc 5

Doc 2

Doc 9Doc

Doc

Doc

Replica

Doc 4

Doc 1

Doc

Doc

Active

SERVER 2

Doc 4

Doc 7 Doc 8

Doc

Doc

Replica

Doc 6

Doc 3 Doc 2

Doc

Doc

Active

SERVER 3

Doc 1

Doc 3

Doc

Doc

Replica

Doc 7

Doc 9

Doc 5Doc

Doc

Doc

• App servers accessing docs. For this example only a small subset of the documents in 1024 partitions are shown.

• Requests to Server 3 begin to fail

• The Cluster detects the failure and initiates a failover if autofailover is enabled. –Promotes replicas of object to

active from Memory. The replica vBuckets on Disk are also promoted to active.

–Updates cluster map and smart client is immediately aware.

• Requests for docs now go to appropriate server

• Typically a rebalance would follow, but is not required.

• Rebalance is an online operation

Doc 1 Doc 3

Doc

Client Interaction with Couchbase

or vBucket

Client Interaction with Couchbase

CRC32("tesla") => Partition[0..1023] {962}ClusterMap[P(962)] => [x.x.x.x] => IP of Server Responsible for Partition 962

or vBucket

Add CapacityOnline Operation

or vBucket

CRC32("tesla") => Partition[0..1023] {962}ClusterMap[P(962)] => [x.x.x.x] => IP of Server Responsible for Partition 962

Add CapacityTwo More Nodes

or vBucket

All Metadata for All Documents

(64 bytes + Key Length)

Document Values(NRU Ejected if RAM Quota

Used > 90%)

Also Leave RAM For OS:

[Filesystem Cache, Views and Data Center Replication]

Document Indexing

Monitoring

XDCR

Recommended:

minimum 4 Cores+ 1 core per design document

+ 1 core per replicated bucket across data centers

(XDCR)

Persisted Documents

All Indexes for Design Documents/Views

Append-Only Disk Format& Compaction

Performance:

Multiple EBS Volumes High IOPS Raid 0 on Amazon

RAM, CPU and IO Guidelines

RAM CPU Disk IO

MONITORING

http://blog.couchbase.com/monitoring-couchbase-cluster

Couchbase Monitoring

DEMO

Let’s grow an online cluster and monitor progress