ccb12 app development with indexes, queries and geo

111

Developing with Views:See Inside the Data

Matt IngenthronDirector, Developer Solutions

2

What we’ll talk about

• Lifecycle of a view• Index definition, build, and query phase• Consistency options (async by default)• Emergent Schema - Views and Documents• Patterns:

• Secondary index• Basic aggregations (avg ratings by brewery)• Time-based analytics with group_level• Leaderboard• Schema Evolution

3

VIEW LIFECYCLE:DEFINE - BUILD - QUERY

3

4

View Definition (in JavaScript)

like:CREATE INDEX city ON brewery city;

4

5

Distributed Index Build Phase

5

• Optimized for lookups, in-order access and aggregations• All view reads from disk (different performance profile)• View builds against every document on every node

– This is why you should group them in a design document• Automatically kept up to date

Doc 4

Doc 2

Doc 5

SERVER 1

Doc 6

Doc 4

SERVER 2

Doc 7

Doc 1

SERVER 3

Doc 3

Doc 9

Doc 7

Doc 8 Doc 6

Doc 3

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

Doc 9

Doc 5

DOC

DOC

DOC

Doc 1

Doc 8 Doc 2

Replica Docs Replica Docs Replica Docs

Active Docs Active Docs Active Docs

6

Dynamic Range Queries with Optional Aggregation

• Efficiently fetch an row or group of related rows.• Queries use cached values from B-tree inner nodes when possible• Take advantage of in-order tree traversal with group_level queries

Doc 4

Doc 2

Doc 5

SERVER 1

Doc 6

Doc 4

SERVER 2

Doc 7

Doc 1

SERVER 3

Doc 3

Doc 9

Doc 7

Doc 8 Doc 6

Doc 3

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

Doc 9

Doc 5

DOC

DOC

DOC

Doc 1

Doc 8 Doc 2

Replica Docs Replica Docs Replica Docs

Active Docs Active Docs Active Docs

?startkey=“J”&endkey=“K”{“rows”:[{“key”:“Juneau”,“value”:null}]}

7

Queries run against stale indexes by default

• stale=update_after (default if nothing is specified)– always get fastest response– can take two queries to read your own writes

• stale=ok– auto update will trigger eventually– might not see your own writes for a few minutes– least frequent updates -> least resource impact

• stale=false– Use with Persistence observe if data needs to be included in

view results– BUT aware of delay it adds, only use when really required

8

Development vs. Production Views

• Development views index a subset of the data.

• Publishing a view builds the index across the entire cluster.

• Queries on production views are scattered to all cluster members and results are gathered and returned to the client.

9

EMERGENT SCHEMA

9

10

Emergent Schema

• Falls out of your key-value usage• Helps to know what's efficient• Mostly you can relax

JSON.orgGithub API

Twitter API

"Capture the user's intent"

11

QUERY PATTERN:FIND BY ATTRIBUTE

11

12

Find documents by a specific attribute

• Lets find beers by brewery_id!

13

The index definition

14

The result set: beers keyed by brewery_id

15

QUERY PATTERN:BASIC AGGREGATIONS

15

15

16

Use a built-in reduce function with a group query

• Lets find average abv for each brewery!

17

We are reducing doc.abv with _stats

18

Group reduce (reduce by unique key)

18

19

QUERY PATTERN:TIME-BASED ROLLUPS

19

19

20

Find patterns in beer comments by time

{   "type": "comment",   "about_id": "beer_Enlightened_Black_Ale",   "user_id": 525,   "text": "tastes like college!",   "updated": "2010-07-22 20:00:20"}{   "id": "f1e62"}

timestamp

21

Query with group_level=2 to get monthly rollups

22

dateToArray() is your friend

dateTo

Array

()

• String or Integer based timestamps• Output optimized for group_level

queries• array of JSON numbers:

[2012,9,21,11,30,44]

23

group_level=2 results

23

• Monthly rollup• Sorted by time—sort the query results in your

application if you want to rank by value—no chained map-reduce

24

group_level=3 - daily results - great for graphing

• Daily, hourly, minute or second rollup all possible with the same index.

• http://crate.im/posts/couchbase-views-reddit-data/

25

QUERY PATTERN:LEADERBOARD

25

26

Aggregate value stored in a document

• Lets find the top-rated beers!{   "brewery": "New Belgium Brewing",   "name": "1554 Enlightened Black Ale",   "abv": 5.5,   "description": "Born of a flood...",   "category": "Belgian and French Ale",   "style": "Other Belgian-Style Ales",   "updated": "2010-07-22 20:00:20", “ratings” : { “ingenthr” : 5, “jchris” : 4, “scalabl3” : 5, “damienkatz” : 1 }, “comments” : [ “f1e62”, “6ad8c” ]}

ratings

27

Sort each beer by its average rating

• Lets find the top-rated beers!

27

average

28

WHAT NOT TO WRITE

28

29

Most common mistakes

• Reduces that don’t reduce• Trying to do too many things with one view• Emitting too much data into a view value• Expecting view query performance to be as fast as

get/set• Recursive queries require application code.

30

GEOGRAPHIC INDEX

30

31

Experimental Status

• Not yet using Superstar trees • (only fast on large clusters)

• Optimized for bulk loading

32

FULL TEXT INDEX

32

33

Elastic Search Adapter

• Elastic Search is good for ad-hoc queries and faceted browsing• Our adapter is aware of changing Couchbase topology• Indexed by Elastic Search after stored to disk in Couchbase

ElasticSearch

34

QUESTIONS?

34

3535

35

Views Under The Hood

J Chris AndersonArchitect

THIS TALK IS NOT WRITTEN YETmaybe combine with Dustin’s internals talk about vbucket handoff

36

What we’ll talk about

36

• Key areas/topics discussed

37

Dynamic Time Range Queries

37

37

38

The B-tree Index• Helps to know what's efficient• Superstar

38

http://damienkatz.net/2012/05/stabilizing_couchbase_server_2.html

39

Logical View B-tree• Incremental reduce values are stored in the tree

39

REDUCES

40

Logical View B-tree• Incremental reduce values are stored in the tree

40

7 5 5 3 2 3

25 REDUCES

41

Reduce!• Incremental reduce values are stored in the tree

41

7 5 5 3 2 3

25_count

function(keys, values) { return keys ? values.length : sum(values);}

42

Dynamic Queries• You can query that tree dynamically• Lots of the patterns are about pulling value from this data structure

42

25

7 5 5 3 2 3 { }?startkey=“abba”&endkey=“robot”{“value”:19}

_count

function(keys, values) { return keys ? values.length : sum(values);}

43

Dynamic Queries• Queries use cached values from B-tree inner nodes when possible• Take advantage of in-order tree traversal with group_level queries

43

25{7 5 5 3 2 3 {

{ }?startkey=“abba”&endkey=“robot”{“value”:19}

(7 5 5 2)19

_count

function(keys, values) { return keys ? values.length : sum(values);}

44

Respect Reduce! (anti-pattern)• Incremental reduce values are stored in the tree

44

function(keys, values) { return values;}

DO NOT DO THIS!

IT DOESN’T reduce

[“ace”, “argh!”,“asphalt”]s[“front”, “garage”,“hibernate”]s[“pluto”, “nectar”,“mirage”]s

[“ace”, “argh!”,“asphalt”, “front”, “garage”,“hibernate”]

45

Just use the Map

• If you think you need “the identity reduce”—just use the map.

45

[“ace”, “argh!”,“asphalt”, “front”, “garage”,“hibernate”]USE THE MAP

46

Lookup via key-range• Find tables during yesterdays lunch shift• Find shifts owned by which manager

46

7 5 5 3 2 3

25

?startkey=“abba”&endkey=“robot”{“value”:19}

47

Schema evolution

47

47

48

Application and Views

48

• Interactive schema fully controlled by application• If your code can handle it, the database can• Learn to write views defensively

49

Incremental schema evolution

49

• Use a view to decide which documents need work• Make your workers idempotent• Once all your data is cleaned up, and old clients are no

longer writing the old format• The cleanup view is obsolete, so is any app code for

dealing with the old case• You've evolved!