webinar - getting started with couchbase app development
TRANSCRIPT
Technical Evangelist
twi0er: @tgrall email: [email protected]
Tugdual “Tug” Grall
Ge;ng Started with Couchbase (Europe)
Technical Evangelist
twi0er: @scalabl3 email: [email protected]
Jasdeep Jaitla
Ge;ng Started with Couchbase (US)
RDBMS vs Couchbase
RDBMS vs Couchbase
• It's Different but Familiar!
• Think in New Ways!
• Simpler in many cases!
• New Pa0erns! • Faster/Lighter/Agile! • "Unlearn what you have Learned!"
Remember: Data is Data; databases are just opYmizing how to store and retrieve it. Objects are Classes with Data and Methods.
RDBMS OrganizaYon
• RDBMS organizes data as tables
-‐ Tables represent data in rows, n columns of m rows
-‐ Table rows have a specific schema, each column as a staYc type
-‐ Simple Datatypes: strings, numbers, dateYmes, booleans, can be represented by columns in a single table
-‐ Complex Datatypes: dicYonaries/hashes, arrays/lists can not be represented in a single table [Impedence Mismatch]
• All rows have idenYcal schema, schema changes require taking database offline
• Reading/WriYng/TransacYons require mutex and locking
Couchbase OrganizaYon
• Couchbase operates like a Key-‐Value Document Store
-‐ Simple Datatypes: strings, numbers, dateYme, boolean, and binary data can be stored; they are stored as Base64 encoded strings
-‐ Complex Datatypes: dicYonaries/hashes, arrays/lists, can be stored in JSON format (simple lists can be string based with delimiter)
-‐ JSON is a special class of string with a specific format for encoding simple and complex data structures
• Schema is unenforced and implicit, schema changes are programmaYc, done online, and can vary from Document to Document
Complex Datatypes
• Can represent both simple and complex data types in JSON data structures
• Can modify schema on the fly, and Documents of a specific "type" can vary in schema
• "Type" is arbitrary, it's a programming strategy, there are no actual "types", but it's typical to embed the class name as a "doctype" json key
Couchbase
{!"doctype": "User","name": "John Doe",!"email": "[email protected]",!"age": 38,!"gender_male": true,!"created_at": "2013-09-20 23:59:59",!"items_viewed": [!
"12345", "23456", 34567"!],!"preferences": {!
"email_notifications": true,!"sms_notifications": false!
},!"authored": [!
{ "title": "Couchbase Models",!"price": 49.95 }]}!
Architecture
Install Couchbase
• Install Couchbase by following InstrucYons • Ajer installaYon, open your browser to h0p://localhost:8091/ • Finish setup step prompts, now you have a running Couchbase server with a default bucket
www.couchbase.com/download
Disk Que
ue
Disk
ReplicaYon Queue
Couchbase Server Node
To other node
Managed Cache
App Server
Doc 1 Doc 1
Doc 1
Write OperaYon
Disk Que
ue
Disk
ReplicaYon Queue
Couchbase Server Node
To other node
Managed Cache
App Server
Doc 1’
Doc 1
Doc 1’ Doc 1
Doc 1’
.
Update OperaYon
Disk Que
ue
Disk
ReplicaYon Queue
Couchbase Server Node
To other node
Managed Cache
GET
Doc 1
2
App Server
Doc 1
Doc 1 Doc 1
Read OperaYon
SDK's
Setup SDK
• Each supported SDK page has instrucYons for setup • PHP, Ruby, NodeJS and Python clients are wrappers around libcouchbase C library, so libcouchbase must be installed first
• For other community clients, click on All Clients on lej nav, scroll down the page and you can see clients for Go, Erlang, Clojure, TCL and Perl.
www.couchbase.com/communiYes
Make a ConnecYon
require 'rubygems'!require 'couchbase'! !cb = Couchbase.connect(!
:bucket => "default",!:hostname => "localhost")!!
data = { jsonkey: "value", created_at: Time.now }!!cb("mydata", data)!puts cb.get("mydata")!
#!/usr/bin/env python!from couchbase import Couchbase!from pprint import pprint!from datetime import datetime!!cb = Couchbase.connect(bucket='default')!!data = { "jsonkey": "value", "created_at": datetime.now() }!!cb.add('mydata', data)!result = cb.get('mydata')!pprint(result.value, indent=4)!
RUBY
PYTHON
Make a ConnecYon
import com.couchbase.client.CouchbaseClient;!import java.net.URI;!import java.util.*;!import com.google.gson.Gson;import com.google.gson.GsonBuilder;!public class HelloWorld {! !public static void main(String[] args) throws Exception {! !List<URI> hosts = Arrays.asList(!new URI("http://127.0.0.1:8091/pools")!);!!CouchbaseClient cb = new CouchbaseClient(hosts, "default", "");!Gson json = new Gson();!!Hashtable data = new Hashtable();!data.put("jsonkey", "value");!data.put("created_at", new Date());!!cb.add("mydata", json.toJson(data))!System.out.println(cb.get("mydata"));! !cb.shutdown();!}!}!
JAVA
Make a ConnecYon
var Couchbase = require('couchbase');!var cb = new Couchbase.Connection({bucket: "default"}, function(err) { });!var data = { jsonkey: "value", created_at: new Date().toString() };!cb.add("mydata", data);!!console.log(cb.get("mydata"));!
!
NODEJS
<?php!// adjust these parameters to match your installation!$cb = new Couchbase("127.0.0.1:8091", "", "", "default");!!$data = array("jsonkey" => "value", ! "created_at" => date("Y-m-d H:i:s"));!!$cb->add("mydata",json_encode($data));!var_dump($cb->get("mydata"));!!?>!
!
PHP
Fundamentals
• Couchbase is structured as a Key-‐Value store, therefore All Data has a Key and a Value
• Keys can be any string up to 250 characters long • Keys are unique, within a database (bucket), there can only be one instance of a key
• Keys are completely in the control of the applicaYon developer, there is no internal mechanism for key generaYon
• Values can be JSON, strings, numbers, binary blobs, or a special posiYve atomic counter (unsigned integer)
• Values can be up to 20MB in size
• get (key) – Retrieve a document
• set (key, value) – Store a document, overwrites if exists
• add (key, value) – Store a document, error/excepYon if exists
• replace (key, value) – Store a document, error/excepYon if doesn’t exist
• cas (key, value, cas) – Compare and swap, mutate document only if it hasn’t changed while execuYng this operaYon
Store & Retrieve OperaYons
Atomic Counter OperaYons These operaYons are always executed in order atomically. • incr (key) – Increase an atomic counter value, default by 1
• cb.incr(“my_counter”) # now it’s 2
• decr (key) – Decrease an atomic counter value, default by 1
• cb.decr(“my_counter”) # now it’s 1
Objects Serialized to JSON and Back User Object
string uid
string firstname
string lastname
int age
array favorite_colors
string email
u::[email protected] { “uid”: 123456, “firstname”: “john”, “lastname”: “doe”, “age”: 22, “favorite_colors”: [“blue”, “black”], “email”: “[email protected]” }
User Object
string uid
string firstname
string lastname
int age
array favorite_colors
string email
u::[email protected] { “uid”: 123456, “firstname”: “John”, “lastname”: “Doe”, “age”: 22, “favorite_colors”: [“blue”, “black”], “email”: “[email protected]” }
add()
get()
Basic Key Pa0erns
Basic Keying
• Use a Unique value for key (email, username, sku, isbn, etc.)
-‐ Users • u::[email protected]
• u::jdoetw -‐ Products
• p::978-‐0321573513 [isbn] • Predictable Keys can follow Key-‐Value pa0erns (Users typically can be done this way and are the most numerous items)
• Unpredictable Keys (GUID, UUID, etc.) require Views (Map-‐Reduce Indexes) to find their documents
Counter-‐ID
ApplicaYon
incr("counter-‐key")
add("key" + counter_val, data)
ApplicaYon
count = get("counter-‐key")
mulY-‐get(keys[])
Data CreaYon
Iterate Through CollecYon
Counter-‐ID
• Similar to IDENTITY column in RDBMS
• CreaYng New Document is a pair of operaYons, INCR and ADD
-‐ IniYalize one Key as an Atomic Counter (I do at App Start)
-‐ Increment Counter and save new value
✴ id = client.incr("blog::couchbase_dev::comment_count")
-‐ Use the id as component of key for new document
✴ client.add(""blog::couchbase_dev::c"::" + id, self.to_json)
Lookup Pa0ern
ApplicaYon
add("[email protected]", "u::550e8400-‐e29b-‐41d4-‐a716")
add("u::550e8400-‐e29b-‐41d4-‐a716", data)
add("jdoetw", "u::550e8400-‐e29b-‐41d4-‐a716")
ApplicaYon
key = get("[email protected]")
get(key)
Data CreaYon
Data Retrieval
Lookup Pa0ern
• Create simple document that has referenYal data (Key) to primary document
-‐ Primary Document u::a2bf2-‐23317-‐2302
-‐ Lookup Document: u::[email protected] { u::a2bf2-‐23317-‐2302 }
• Lookup Documents aren't JSON, they should just be the Key as a string so you skip JSON parsing
• Requires Two GET operaYons, first GET Lookup, then GET primary Document
-‐ key = client.get("u::[email protected]")
-‐ doc = client.get(key)
Combine Counter-‐ID and Lookup
ApplicaYon
add("[email protected]", id)
add("u::" + id, data)
add("jdoetw", id)
ApplicaYon
key = get("[email protected]")
get(key)
Data CreaYon
Data Retrieval
id = incr("user::count")
Combine Counter-‐ID and Lookup
Pro's
• Binary OperaYons, overall faster than large volume of View Queries
• EssenYally creates several ways to find a single document
• Is always consistent, just like all other Binary operaYons
Con's
• Increases Number of Documents, therefore Metadata usage in RAM
-‐ But this is generally a non-‐issue for most people
Views
Map() FuncYon => Index
function(doc, meta) { emit(doc.username, doc.email)
}
Every Document passes through View Map() functions
Map
Single Element Keys (Text Key)
function(doc, meta) { emit(doc.email, doc.points)
} text key
Map
doc.email doc.points
[email protected] 1000
[email protected] 1200
Compound Keys (Array)
function(doc, meta) { emit(dateToArray(doc.timestamp), 1)
} array key
Array Based Index Keys get sorted as Strings, but can be grouped by array elements
Map
dateToArray(doc.Ymestamp) value
[2012,10,9,18,45] 1
[2012,9,26,11,15] 1
[2012,8,13,2,12] 1
Most Common Query’s Are Ranges
doc.email meta.id
[email protected] u::1
[email protected] u::7
[email protected] u::2
[email protected] u::5
[email protected] u::6
[email protected] u::4
[email protected] u::3
?startkey=”b1” & endkey=”zZ”
Pulls the Index-‐Keys between UTF-‐8 Range specified by the startkey and endkey.
?startkey=”bz” & endkey=”zn”
Pulls the Index-‐Keys between UTF-‐8 Range specified by the startkey and endkey.
Index-‐Key Matching
doc.email meta.id
[email protected] u::1
[email protected] u::7
[email protected] u::2
[email protected] u::5
[email protected] u::6
[email protected] u::4
[email protected] u::3
?key=”[email protected]”
Match a Single Index-‐Key
Index-‐Key Set Matches
doc.email meta.id
[email protected] u::1
[email protected] u::7
[email protected] u::2
[email protected] u::5
[email protected] u::6
[email protected] u::4
[email protected] u::3
?keys=[“[email protected]”, “[email protected]”]
Query MulYple in the Set (Array NotaYon)
Key-‐Value Pa0erns vs Views
• Key Pa0erns only use Binary OperaYons • Documents are consistent, CAS operaYons can be used for race condiYons on parYcular documents
• Do not have a Yme gap, therefore can be faster
• Even as data size gets larger, doesn't increase latency
Key-‐Value Pa0erns
Key-‐Value Pa0erns vs Views
• As Documents are created/mutated, they are persisted
• Views (Indexes) do batch indexing for persisted Documents, incrementally from last batch process
• The process of persistence + indexing == Yme gap between CRUD OperaYons and showing in View query ==> Eventually Consistent
Views
Quick Demo
Q&A