three things you need to know about data modeling: couchbase connect 2015
TRANSCRIPT
THREE (OR MORE) THINGS YOU NEED TO KNOW ABOUT DATA MODELINGMatthew Revell, Developer Advocate, Couchbase
WE’RE STILL LEARNING
©2015 Couchbase Inc. 3
©2015 Couchbase Inc. 4
Three query methods
Application layer computation Off-load computation
Predictable queries
Key-value: pre-computed answers Views
Ad-hoc queriesN1QL and key-value with manual indexes
N1QL and views
KEY-VALUE
©2015 Couchbase Inc. 6
Key-value
Pre-compute answers asynchronously Store object state Choose when to embed data and when to refer
to it Design your keys well
PRE-COMPUTED ANSWERS
©2015 Couchbase Inc. 8
We’re used to asking questions
©2015 Couchbase Inc. 9
Key-value look-ups give us a library of answers
42
©2015 Couchbase Inc. 10
Answer oriented databases
http://martinfowler.com/bliki/AggregateOrientedDatabase.html
©2015 Couchbase Inc. 11
Airline bookings
EMBED, OR REFER?
How much should we denormalise?
©2015 Couchbase Inc. 13
An ecommerce order
©2015 Couchbase Inc. 14
The same order in a document
©2014 Couchbase, Inc. ©2015 Couchbase Inc. 15
Embedded v referred
15
©2015 Couchbase Inc. 16
When to embed
You should embed data when: Speed trumps all else
Slow moving data
No duplication
Application layer can keep multiple copies of same data in sync
©2015 Couchbase Inc. 17
When to refer
You should refer to data when: Consistency is a priority
The data has large growth potential
KEY DESIGN
©2015 Couchbase Inc. 19
Three ways to build a key
Key design is as important as document design. There are three broad types of key:
Human readable/deterministic: e.g. an email address
Computer generated/random: e.g. UUID
Compound: e.g. UUID with a deterministic portion
©2015 Couchbase Inc. 20
Human readable/deterministic
public class user {
private String name;private String email;private String streetAddress;private String city;private String country;private String postCode;private String telephone;private Array orders;private Array
productsViewed;}
{ "name": "Matthew Revell", "address": "11-21 Paul Street", "city": "London", "postCode": "EC2A 4JU", "telephone": "44-20-3837-9130", "orders": [ 1, 9, 698, 32 ], “productsViewed”: [8, 33, 99, 100] }
Key: [email protected]
©2015 Couchbase Inc. 21
Random/computer genereated
{ "name": "Matthew Revell", "email": "[email protected]", "address": "11-21 Paul Street", "city": "London", "postCode": "EC2A 4JU", "telephone": "44-20-3837-9130", "orders": [ 1, 9, 698, 32 ], “productsViewed”: [8, 33, 99, 100] }
Key: 1001
©2014 Couchbase, Inc. ©2015 Couchbase Inc. 22
Using counters to generate keys
22
Application
user_id = incr(“counter_key")
add(user_id, data)
Creating a new user
add(email_address, user_id)
Application
key = get("[email protected]")
get(key)
Finding user by email address
©2015 Couchbase Inc. 23
Multiple look-up documents
u::count
1001
u::1001
{ "name": “Matthew Revell",
"facebook_id": 16172910,
"email": “[email protected]”,
“password”: ab02d#Jf02K
"created_at": "5/1/2012 2:30am",
“facebook_access_token”: xox0v2dje20,
“twitter_access_token”: 20jffieieaaixixj }
fb::16172910
1001
nflx::2939202
1001
twtr::2920283830
1001
1001
1001
uname::mrevell
1001
©2015 Couchbase Inc. 24
Compound keys
Compound keys are look-up documents with a predictable name.
It’s a continuation of the embedded versus referred data discussion.
©2015 Couchbase Inc. 25
Compound keys: example
u::1001
{ "name": "Matthew Revell", "email": "[email protected]", "address": "11-21 Paul Street", "city": "London", "postCode": "EC2A 4JU", "telephone": "44-20-3837-9130", "orders": [ 1, 9, 698, 32 ], “productsViewed”: [8, 33, 99, 100]
}
©2015 Couchbase Inc. 26
Compound keys: example
u::1001
{ "name": "Matthew Revell", "email": "[email protected]", "address": "11-21 Paul Street", "city": "London", "postCode": "EC2A 4JU", "telephone": "44-20-3837-9130", "orders": [ 1, 9, 698, 32 ]}
u::1001::productsviewed
{"productsList": [
8, 33, 99, 100]
}
©2015 Couchbase Inc. 27
Compound keys: example
u::1001
{ "name": "Matthew Revell", "email": "[email protected]", "address": "11-21 Paul Street", "city": "London", "postCode": "EC2A 4JU", "telephone": "44-20-3837-9130", "orders": [ 1, 9, 698, 32 ]}
u::1001::productsviewed
{"productsList": [
8, 33, 99, 100]
}
p::8
{
id": 1,"name": "T-shirt","description": "Red Couchbase shirt","quantityInStock": 99,"image": "tshirt.jpg”
}
©2015 Couchbase Inc. 28
Compound keys: example
u::1001
{ "name": "Matthew Revell", "email": "[email protected]", "address": "11-21 Paul Street", "city": "London", "postCode": "EC2A 4JU", "telephone": "44-20-3837-9130", "orders": [ 1, 9, 698, 32 ]}
u::1001::productsviewed
{"productsList": [
8, 33, 99, 100]
}
p::8
{
id": 1,"name": "T-shirt","description": "Red Couchbase shirt","quantityInStock": 99
}
p::8::img
“http://someurl.com/tshirt.jpg”
N1QL AND VIEWS!
The world beyond key-value
©2015 Couchbase Inc. 30
©2015 Couchbase Inc. 31
N1QL and Views
N1QL Views
Ad-hoc querying Predictable queries
Nested JSON in and unnested JSON out
Number crunching
Large growth clustersMulti-dimensional/geospatial queries
©2015 Couchbase Inc. 32
N1QL
Indexes Schema and documents types Keyspaces
©2015 Couchbase Inc. 33
Indexes
You always need at least one index PRIMARY index gives you all of the keys in the
bucket Secondary indexes enable ad-hoc querying at
speed
©2015 Couchbase Inc. 34
Indexes
GSI
versusVIEWS
©2015 Couchbase Inc. 35
Indexes
NEW Global Secondary Indexes Views
N1QL query to create a new indexJavascript map/reduce
Option to run dedicated indexing and query nodes
Queries always run on database nodes
ForestDB backed Couchstore backed
Supports multi-dimensional/geospatial queries
©2015 Couchbase Inc. 36
Maintaining schema and working with keyspaces JOINs work on primary keys and secondary keys They JOIN across and within keyspaces (for now, that
means buckets)
Airlines:
airline_24 ← Key (“primary key”){ "id": "24", "type": "airline", "name": "American Airlines", "iata": "AA", "icao": "AAL", "callsign": "AMERICAN", "country": "United States", "active": "Y"}
Routes:
route_5966 ← Key { "id": "5966", "type": "route", "airline": "AA", "airlineid": "airline_24", ← This is the foreign key "sourceairport": "MCO", "destinationairport": "SEA", "stops": "0", "equipment": "737", "schedule": [... ]}
©2015 Couchbase Inc. 37
Offloading computation to N1QL
N1QL allows you to choose which elements are returned at the top level. This is incredibly useful as it reduces the need to write complex parsing logic: Within the application. In Client side front end frameworks
(Angular/Backbone etc)SELECT a.name, s.flight, s.utc, r.sourceairport, r.destinationairport, r.equipment FROM `travel-sample` r UNNEST r.schedule s JOIN `travel-sample` a ON KEYS r.airlineid WHERE r.sourceairport='LHR' AND r.destinationairport='LAX' AND s.day=2 ORDER BY a.name
QUESTIONS!
OTHER SESSIONS!
IRFC: A design pattern for next gen business applicationsSteve Yen, 4.30pm ThursdayDev track
THANKS!
Matthew [email protected]@matthewrevell
Get Started with Couchbase Server 4.0: www.couchbase.com/beta
Get Trained on Couchbase: training.couchbase.com