introduction to rest & couchdb

Introduction toREST & CouchDB

Benjamin Erb | 28.01.2011 | Universität Ulm Web Engineering WS10/11

Introduction to REST & CouchDB | Web Engineering WS10/11 | Benjamin Erb | 28.01.2011

Introduction

I Benjamin ErbI http://twitter.com/b_erb

I http://github.com/berb

I http://www.benjamin-erb.de

I student of computer sciene & media informaticsI Ulm University, Germany

http://twitter.com/b_erb

http://github.com/berb

http://www.benjamin-erb.de

Part I: REST


Programming for Distributed Systems

I communication between computer programsI distributed (different processes . . . different machines)I popular approaches

I RPC (remote procedure call)I RMI (OOP-ish RPC)I Messaging (SOAP, MQs)I . . .


Representational State Transfer (REST)

architectural style for distributed hypermediasystems


The Idea behind REST

WWW

REST



WWW

I distributed hypermedia systemI works on “Internet scale”I loose couplingI human users

REST



WWW

REST

I architectural styleI distributed systemsI embracing the ideas behind the webI non-human users (applications)



WWW

REST

Please note: We’ll only focus on applying the REST architecturalstyle on HTTP during this talk.


The Main Principles of REST

I client-server architectureI statelessnessI uniform interfaceI cacheabilityI layered system


Client-Server Architecture

Request

Response


Statelessness

I server manages resource statesI client manages session states

»[...]each request from client to server must contain all of

the information necessary to understand the request, and

cannot take advantage of any stored context on the server.«

Source: Roy T. Fielding — Architectural Styles and the Design of Network-based Software Architectures


Statelessness — Examples

http://google.com/search?q=uni+ulm&start=10

http://example/q?SID=61AUOh6W1TA&page=next




I search query for “uni ulm”I second result page







I server maintains sessions for each search processI SID contains session variableI server must check state of session (i.e. search term)I apply new action (i.e. next page)



X http://google.com/search?q=uni+ulm&start=10


× http://example/q?SID=61AUOh6W1TA&page=next

I server maintains sessions for each search processI SID contains session variableI server must check state of session (i.e. search term)I apply new action (i.e. next page)


Uniform Interface

I set of distinct methodsI generic operationsI methods of interface ∼ verbsI cf. component interface design pattern


HTTP Methods

safe idempotent cachable1

HEAD X X XGET X X XPUT X

DELETE XPOST

1possible


CRUD Verbs for ResourcesC createPUT create resource; client knows URI a-priori

POST create resource; server chooses URI of new resourceR readGET retrieve resource

U updatePUT change resource

D deleteDELETE remove resource


Constraints due to the Uniform Interface

I identification of resourcesI manipulation of resources through representationsI self-descriptive messagesI hypermedia as the engine of application state


Identification of Resources

I “nouns” of the applicationI fine-grained entitiesI addressable through URIs

ExamplesI http://portal/user/4711

I http://shop/customer/23/order/8

I http://blog/post/the-post-title/comments

I http://search/query?term=foo


Manipulation of Resources through Representations

GET /resource/4711

PUT /resource/4711

200 OK

200 OK


Self-descriptive Messages

I messages contain representations of resourcesI descriptitve data available in HTTP headersI also useful for intermediate proxies


Self-descriptive Messages (2)I popular representations for RESTful applications

I XMLI JSONI HTML

I representations can be negotiatedI HTTP Request Headers: Accept, Accept-Language,Accept-Encoding

I HTTP Response Headers: Content-Type,Content-Language, Content-Encoding

I HTTP Responses: 300 Multiple Choices, 406 NotAcceptable


Hypermedia as the Engine of Application State

Linking resources

GET /users

GET /user/23

200 OK

200 OK

/users{"users" : [ { "id": 23, "link" : "/user/23" }, { "id": 42, "link" : "/user/42" }]}

/user/23{"user" : { "id" : 23, "name" : "foobar", "details" { ... }}}


Hypermedia as the Engine of Application State (2)

Creating resources through factory resources

POST /users

201 Created

POST /users HTTP/1.1Host: foo.x...Content-Type: application/json

{"user" : { "name" : "qwerty", "details" { ... }}}

HTTP/1.1 201 CreatedLocation: http://foo.x/user/12...


REST Triangle

NounsURIshttp://host/user/4711

RepresentationsContent-TypesXML, JSON, HTML,…

VerbsHTTP MethodsGET, PUT,…

Use Case

RESTful API


REST: Further Readings

Dissertation of Roy Thomas Fielding (Original source)I http://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm (Chapter 5)

BooksI Richardson, Leonard; Ruby, Sam: RESTful Web Services

(O’Reilly)I Stefan Tilkov: REST und HTTP (dpunkt Verlag)

ArticlesI http://tomayko.com/writings/rest-to-my-wife

http://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm

http://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm

http://tomayko.com/writings/rest-to-my-wife

Part II: CouchDB

Jacob Kaplan-Moss (Django Developer)

»Django may be built for the Web, but CouchDB is built

of the Web. I’ve never seen software that so completely

embraces the philosophies behind HTTP.«

NoSQLNot Only SQL


SQL

I relational algebraI database schemesI powerful (ad-hoc) queriesI transactional behaviourI blocking consistencyI ACID

A atomicityC consistencyI isolation

D durability


SQL. . . but

Think ofI large data sets (i.e. Facebook profiles)I many concurrent reads/writes (i.e. Tweets)

oopsI RDBMS are really hard to scale

I reads are hardI writes are even harder


NoSQL

I dedicated database architectureI special purposes: scalability, read/write performance, . . .I BASE

BA basically availableS soft stateE eventually consistent


NoSQL. . . but

I weak(er) consistency modelsI generally less featuresI querying

I no SQL (sic)I ad-hoc queries difficult or even not possibleI i.e. MapReduce instead


CAP Theorem

Consistency

Availability

Partition Tolerance


CAP Theorem (pick two)

Consistency

Availability

Partition Tolerance Eventual

Consistency

Relational Database Management Systems


NoSQL Databases

Document Stores

Wide Column Stores

Graph Databases

Key-Value Stores


NoSQL Databases

Document Stores

I “documents” ∼ structured, but schemaless data setsI Examples: CouchDB, MongoDB

Wide Column Stores

Graph Databases

Key-Value Stores


NoSQL Databases

Document Stores

Wide Column Stores

I sparse tables / column-oriented instead of row-orientedI Examples: Cassandra, HBase

Graph Databases

Key-Value Stores


NoSQL Databases

Document Stores

Wide Column Stores

Graph Databases

I data structured as graphsI Examples: neo4j, FlockDB

Key-Value Stores


NoSQL Databases

Document Stores

Wide Column Stores

Graph Databases

Key-Value Stores

I cf. HashTable / HashMapI Examples: Redis, Amazon Dynamo, Voldemort


CouchDB

I NoSQL databaseI document storeI distributed architectureI easy replication


CouchDB (2)

I written in ErlangI documents stored as JSONI RESTful HTTP APII MapReduce functions (JavaScript) for views


JSON

I JavaScript Object NotationI leightweight data-interchange formatI types

I string, number, boolean, nullI structures

I array, object


JSON — Example1 {2 "boolean" : true,3 "array" : ["foo","bar"],4 "object" : {5 "key" : "value",6 "num" : 1237 }8 }


Documents

I JSONI two required fields

I _idI _rev (versioning/concurrency control)

Example:

1 {2 "_id": "58fb1df368cd2f9fef8e387508000f6d",3 "_rev": "1-ddc8d18e4d30f529608ec65d7e7879f2",4 "name": "cauliflower soup",5 "type": "soup",6 "labels": [7 "organic",8 "veggie"9 ]

10 }


RESTful API: Create a Database

benjamin@cronos:~$ curl -i -X PUT http://localhost:5984/mensaplan

HTTP/1.1 201 Created

Server: CouchDB/1.1.0a7446e91-git (Erlang OTP/R13B)

Location: http://localhost:5984/mensaplan

Date: Thu, 27 Jan 2011 12:55:29 GMT

Content-Type: application/json;charset=utf-8

Content-Length: 12

Cache-Control: must-revalidate

{"ok":true}


RESTful API: Create a Database (2)

Let’s try again. . .benjamin@cronos:~$ curl -i -X PUT http://localhost:5984/mensaplan

HTTP/1.1 412 Precondition Failed


Date: Thu, 27 Jan 2011 12:57:30 GMT


Content-Length: 95


{"error":"file_exists","reason":"The database could not be created, the file already

exists."}


RESTful API: Delete a Database

benjamin@cronos:~$ curl -i -X DELETE http://localhost:5984/mensaplan

HTTP/1.1 200 OK


Date: Thu, 27 Jan 2011 13:01:39 GMT


Content-Length: 12


{"ok":true}


RESTful API: Create a Document

benjamin@cronos:~$ curl -i -X POST -H "Content-Type: application/json" -d ’{"name":"

goulash","type":"main","labels":["pork"]}’ http://localhost:5984/mensaplan



Location: http://localhost:5984/mensaplan/ad8592f32ccd34f0588b86f5a9000fd3

Date: Thu, 27 Jan 2011 13:04:09 GMT


Content-Length: 95


{"ok":true,"id":"ad8592f32ccd34f0588b86f5a9000fd3","rev":"1-5

bb3c72e38c6c7aaa7a2d4fd578478e7"}


RESTful API: Get a Document

benjamin@cronos:~$ curl -i -X GET http://localhost:5984/mensaplan/

ad8592f32ccd34f0588b86f5a9000fd3

HTTP/1.1 200 OK


Etag: "1-5bb3c72e38c6c7aaa7a2d4fd578478e7"

Date: Thu, 27 Jan 2011 13:07:27 GMT


Content-Length: 136


{"_id":"ad8592f32ccd34f0588b86f5a9000fd3","_rev":"1-5bb3c72e38c6c7aaa7a2d4fd578478e7"

,"name":"goulash","type":"main","labels":["pork"]}


RESTful API: Update a Document

benjamin@cronos:~$ curl -i -X PUT -H "Content-Type: application/json" -d ’{"name":"

goulash","type":"main","labels":["pork","organic"]}’ http://localhost:5984/

mensaplan/ad8592f32ccd34f0588b86f5a9000fd3

HTTP/1.1 409 Conflict


Date: Thu, 27 Jan 2011 13:08:53 GMT


Content-Length: 58


{"error":"conflict","reason":"Document update conflict."}


RESTful API: Update a Document (doing it right)

benjamin@cronos:~$ curl -i -X PUT -H "Content-Type: application/json" -d ’{"name":"

goulash","type":"main","labels":["pork","organic"]}’ http://localhost:5984/

mensaplan/ad8592f32ccd34f0588b86f5a9000fd3 -H ’If-Match: "1-5

bb3c72e38c6c7aaa7a2d4fd578478e7"’



Location: http://localhost:5984/mensaplan/ad8592f32ccd34f0588b86f5a9000fd3

Etag: "2-8ebf81b724635a1b625cc1a66e0e5263"

Date: Thu, 27 Jan 2011 13:10:41 GMT


Content-Length: 95


{"ok":true,"id":"ad8592f32ccd34f0588b86f5a9000fd3","rev":"2-8

ebf81b724635a1b625cc1a66e0e5263"}

MapReduce?

http://www.flickr.com/photos/bdesham/2432400623


MapReduce


MapReduce

map


MapReduce

2x

3x

1x

1x

3x

2x

2x

map


MapReduce

2x

3x

1x

1x

3x

2x

2x

map reduce


MapReduce

2x

3x

1x

1x

3x

2x

2x

2x

6x

3x

3x

map reduce

Source: http://browsertoolkit.com/fault-tolerance.png


CouchDB Views with MapReduce

I views ∼ B+ treesI incrementalI tuples of <key, value>I items sorted by keyI complex keys possible (JSON)


View Example: List organic food only

map1 function(doc){2 if(doc.labels){3 for(var idx in doc.labels){4 if(doc.labels[idx] === "organic"){5 emit(doc._id, doc.name);6 }7 }8 }9 }


Querying the View

benjamin@cronos:~$ curl -i http://localhost:5984/mensaplan/_design/couchapp/_view/

organic

HTTP/1.1 200 OK

Transfer-Encoding: chunked


Etag: "7GW4ZJ7D8F2T2OCFL7VJO5AO6"

Date: Thu, 27 Jan 2011 14:26:35 GMT

Content-Type: text/plain;charset=utf-8


{"total_rows":2,"offset":0,"rows":[

{"id":"58fb1df368cd2f9fef8e387508000f6d","key":"58fb1df368cd2f9fef8e387508000f6d","

value":"cauliflower soup"},

{"id":"58fb1df368cd2f9fef8e387508001149","key":"58fb1df368cd2f9fef8e387508001149","

value":"spaghetti napoli"}

]}


View Example: Count dishes in database

map1 function(doc){2 if(doc.type){3 emit(doc.type, 1);4 }5 }

reduce1 function (key, values, rereduce) {2 return sum(values);3 }


Querying the View (without reduce)


dishes?reduce=false

HTTP/1.1 200 OK



Etag: "B3JLG6SIRJKRHCRNIQSEAH6GL"

Date: Thu, 27 Jan 2011 14:15:25 GMT



{"total_rows":3,"offset":0,"rows":[

{"id":"58fb1df368cd2f9fef8e387508001149","key":"main","value":1},

{"id":"58fb1df368cd2f9fef8e3875080012d4","key":"main","value":1},

{"id":"58fb1df368cd2f9fef8e387508000f6d","key":"soup","value":1}

]}


Querying the View


dishes

HTTP/1.1 200 OK




Date: Thu, 27 Jan 2011 14:16:16 GMT



{"rows":[

{"key":null,"value":3}

]}


Querying the View: Grouping


dishes?group=true

HTTP/1.1 200 OK




Date: Thu, 27 Jan 2011 14:17:41 GMT



{"rows":[

{"key":"main","value":2},

{"key":"soup","value":1}

]}


Replication

I unidirectional or bidirectionalI snapshotted or continuous

Watch out!I conflicts possibleI conflict resolution necessaryI Multiversion Concurrency Control


Replication — Example: Bidirectional


Replication — Example: Master Slave

Writes Reads


Replication — Example: Peer-to-Peer


Web Interface: Futon


Other CouchDB Features

_show

_list

_changes

validate_doc_update



_show

I regular documents are JSON objectsI _show functions render other formatsI i.e. HTML page

_list

_changes

validate_doc_update



_show

_list

I regular view results are encoded in JSONI _list functions convert results into custom formatsI i.e. RSS feed

_changes

validate_doc_update



_show

_list

_changes

I change notificationI fires on new document, updates, etc.I HTTP (long) polling, continuous

validate_doc_update



_show

_list

_changes

validate_doc_update

I validation routineI checks if document change is valid


Some Interesting CouchDB Projects

CouchApp

CouchDB Lounge / BigCouch

GeoCouch

CouchDB-Lucene



CouchApp

I standalone web applications with CouchDBI CouchDB serves client-side HTML + JavaScript


GeoCouch

CouchDB-Lucene



CouchApp


I clustering extensionsI proxy-based partitioning/clustering

GeoCouch

CouchDB-Lucene



CouchApp


GeoCouch

I spatial indexes for CouchDBI efficient geo-spatial queries

CouchDB-Lucene



CouchApp


GeoCouch

CouchDB-Lucene

I full text search for CouchDBI separate index based on Lucene (Java)

Live Demo

CouchDB & Futon


CouchDB: Further Readings

BooksI J. Chris Anderson; Jan Lehnardt; Noah Slater — CouchDB:

The Definitive Guide (O’Reilly)I Open Source Book:http://guide.couchdb.org/index.html

I Stefan Edlich, Achim Friedland, Jens Hampe, BenjaminBrauer — NoSQL (HANSER Verlag)

WebsitesI http://nosqltapes.comI http://www.couchone.com/get

http://guide.couchdb.org/index.html

http://nosqltapes.com

http://www.couchone.com/get


Discussion

Thanks for your attention.

Any questions?


License

Slides incorporate contents form. . .I picol.orgI flickr.com/photos/daveaustriaI flickr.com/photos/bdeshamI github.com/oreilly/couchdb-guide

Backup Slides


Distributed Systems – Motivation

single computerI limited resourcesI limited computing powerI limited storageI single point of failure



many computersI more resourcesI more computing powerI more storageI non-functional benefits (i.e. fault-tolerance)



many computersI more resourcesI more computing powerI more storageI non-functional benefits (i.e. fault-tolerance)

but alsoI more complex systemsI more sources of trouble


Distributed Systems – Different Approaches

environmentsI distributed operating systemsI middleware systemsI standalone applicationsI . . .

interaction modelsI shared memoryI request/replyI message-basedI . . .


Distributed Systems – Some Requirements

I identifiers for resources (what?)I location/naming services (where?)I communication services/protocols (how?)I software support (how to use?)


Historical Approach: RPC

Remote Procedure Calls

add(1,2)

return 3

1+2=3A B

add(1,2);

I Examples: Sun RPC (NFS)


CouchDB: MVCC / Conflicts

Source: CouchDB — The Definitive Guide


CouchDB: Append-only B+ Tree

Source: CouchDB — The Definitive Guide

introduction to rest & couchdb

Technology

introduction benjamin

rest architecturalstyle

shopcustomer23order8

portaluser4711 http

comberb http

statelessness examples

result page http

uris examples http