cloudstock 2010

8/8/2019 Cloudstock 2010

1/77

Alvin [email protected]


2/77

Topics

OverviewDocument Design

Modeling the real worldReplication & ShardingDeveloping with MongoDBDeployment


3/77

Drinking from the fire hose


4/77

Part OneMongoDB Overview


5/77

MongoDB is the leading databasefor cloud deployment

web 2.0 companies started out using thisbut now:- enterprises- financial industries

3 Reason

- Performance- Large number of readers / writers- Large data volume- Agility (ease of development)


6/77

non-relational,next-generationoperationaldatastoresanddatabases

NoSQL Really

Means:


7/77

RDBMS(Oracle,MySQL)

past : one-size-fits-all


8/77

RDBMS(Oracle,MySQL)

New Gen.OLAP

(vertica,aster,greenplum)

present : business intelligence and analytics is now its own segment.


9/77

RDBMS(Oracle,MySQL)

New Gen.OLAP

(vertica,aster,greenplum)

Non-relationalOperational

Stores(NoSQL)

future

we claim nosql segment will be:

* large

* not fragmented

* platformitize-able


10/77

Philosophy:maximizefeatures-uptothekneeinthecurve,thenstop

depthoffunctionality

scalability&perfor

mance memcached

key/value

RDBMS


11/77

Horizontally ScalableArchitectures

nojoins

nocomplextransactions+


12/77

New Data ModelsImproved ways to develop

nojoins

nocomplextransactions+


13/77

Part TwoData Modeling in MongoDB


14/77

So why model data?

http://www.flickr.com/photos/42304632@N00/493639870/


15/77

A brief history of normalization 1970 E.F.Codd introduces 1st Normal Form (1NF)

1971 E.F.Codd introduces 2nd and 3rd Normal Form (2NF, 3NF)

1974 Codd & Boyce define Boyce/Codd Normal Form (BCNF)

2002 Date, Darween, Lorentzos define 6th Normal Form (6NF)

Goals:

Avoid anomalies when inserting, updating or deleting

Minimize redesign when extending the schema

Make the model informative to users

Avoid bias towards a particular style of query

* source : wikipedia


16/77

The real benefit of relational

Before relational

Data and Logic combined

After relational Separation of concerns Data modeled independent of logic Logic freed from concerns of data design

MongoDB continues this separation


17/77

Relational made normalizeddata look like this


18/77

Document databases makenormalized data look like this


19/77

Terminology

RDBMS MongoDB

Table Collection

Row(s) JSONDocument

Index Index

Join Embedding&Linking

Partition Shard

PartitionKey ShardKey


20/77

Terminology

RDBMS MongoDB

Table Collection

Row(s) JSONDocument

Index Index

Join Embedding&Linking

Partition Shard

PartitionKey ShardKey


21/77

Create a document

Design documents that simply map toyour application

post={author:Herg,date:newDate(),text:DestinationMoon,tags:[comic,adventure]}

>db.post.save(post)


22/77

Secondary index for author

// 1 means ascending, -1 means descending

>db.posts.ensureIndex({author: 1})

>db.posts.find({author: 'Herg'})

{ _id : ObjectId("4c4ba5c0672c685e5e8aabf3"), author : "Herg",

... }

Add and index, find via Index


23/77

Explain a query plan>db.blogs.find({author:'Herg'}).explain()

{

"cursor":"BtreeCursorauthor_1",

"nscanned":1,

"nscannedObjects":1,

"n":1, "millis":5,

"indexBounds":{

"author":[

[

"Herg",

"Herg"

]

]

}


24/77

Query operators

Conditional operators:$ne, $in, $nin, $mod, $all, $size, $exists, $type, ..$lt, $lte, $gt, $gte, $ne,

//findpostswithanytags>db.posts.find({tags:{$exists:true}})


25/77

Query operators



Regular expressions://postswhereauthorstartswithh

>db.posts.find({author:/^h/i})


26/77

Query operators



Regular expressions://postswhereauthorstartswithh

>db.posts.find({author:/^h/i})

Counting://numberofpostswrittenbyHerg

>db.posts.find({author:Herg}).count()


27/77

Part ThreeModeling the real world


28/77

Inheritance


29/77

Single Table Inheritance - RDBMS

shapes tableid type area radius d length width

1 circle 3.14 1

2 square 4 2

3 rect 10 5 2


30/77

Single Table Inheritance

>db.shapes.find(){ _id: ObjectId("..."), type: "circle", area: 3.14, radius: 1}{ _id: ObjectId("..."), type: "square", area: 4, d: 2}{ _id: ObjectId("..."), type: "rect", area: 10, length: 5, width: 2}

// find shapes where radius > 0>db.shapes.find({radius: {$gt: 0}})

// create index>db.shapes.ensureIndex({radius: 1})


31/77

One to ManyOne to Many relationships can specify

degree of association between objects containment life-cycle


32/77

One to Many

- Embedded Array / Array Keys- slice operator to return subset of array- some queries hard

e.g find latest comments across all documents

- Embedded tree- Single document- Natural- Hard to query

- Normalized (2 collections)- most flexible- more queries


33/77

One to Many - patterns

- Embedded Array / Array Keys

- Embedded Array / Array Keys- Embedded tree- Normalized


34/77

Many - Many

Example:- Product can be in many categories

- Category can have many products


35/77

products:{_id:ObjectId("4c4ca23933fb5941681b912e"),

name:"DestinationMoon",

category_ids:[ObjectId("4c4ca25433fb5941681b912f"),

ObjectId("4c4ca25433fb5941681b92af]}

categories:{_id:ObjectId("4c4ca25433fb5941681b912f"),

name:"Adventure",

product_ids:[ObjectId("4c4ca23933fb5941681b912e"),

ObjectId("4c4ca30433fb5941681b9130"),

ObjectId("4c4ca30433fb5941681b913a"]}

//Allcategoriesforagivenproduct>db.categories.find({product_ids:ObjectId

("4c4ca23933fb5941681b912e")})

Many - Many


36/77

products:

{_id:ObjectId("4c4ca23933fb5941681b912e"),

name:"DestinationMoon",category_ids:[ObjectId("4c4ca25433fb5941681b912f"),

ObjectId("4c4ca25433fb5941681b92af]}

categories:

{_id:ObjectId("4c4ca25433fb5941681b912f"),name:"Adventure"}

//Allproductsforagivencategory

>db.products.find({category_ids:ObjectId

("4c4ca25433fb5941681b912f")})

//Allcategoriesforagivenproductproduct=db.products.find(_id:some_id)

>db.categories.find({_id:{$in:product.category_ids}})

Alternative


37/77

Modeling - Sumamry

Ability to model rich data constructions

Relationships (1-1, 1-M, M-M) Trees

Queues, Stacks Simple to change your data design Quickly map your application needs to data needs


38/77

Part ThreeReplication & Sharding


39/77

Scaling

Data size only goes up Operations/sec only go up Vertical scaling is limited

Hard to scale vertically in the cloud Can scale wider than higher

What is scaling?

Well - hopefully for everyone here.


40/77

Read Scalability : Replication

write

read

ReplicaSet1

Primary

Secondary

Secondary


41/77

Basics

MongoDB replication is a bit like RDBMS replication

Asynchronous master/slave at its core

Variations:

Master / slave

Replica Sets


42/77

A cluster of N servers Any (one) node can be primary Consensus election of primary Automatic failover Automatic recovery All writes to primary Reads can be to primary (default) or a secondary

Replica Sets


43/77

Replica Sets Design Concepts

1. Write is durable once avilable on a majority ofmembers

2. Writes may be visible before a cluster widecommit has been completed

3. On a failover, if data has not been replicatedfrom the primary, the data is dropped (see #1).


44/77

Replica Set: Establishing

Member 1

Member 2

Member 3


45/77

Replica Set: Electing primary

Member 1

Member 2PRIMARY

Member 3


46/77

Replica Set: Failure of master

Member 1

Member 2DOWN

Member 3PRIMARY

negotiatenew

master


47/77

Replica Set: Reconfiguring

Member 1

Member 2DOWN

Member 3PRIMARY


48/77

Replica Set: Member recovers

Member 1

Member 2RECOVER-

ING

Member 3PRIMARY


49/77

Replica Set: Active

Member 1

Member 2

Member 3PRIMARY


50/77

Write Scalability: Sharding

write

read

ReplicaSet1

Primary

Secondary

Secondary

ReplicaSet2

Primary

Secondary

Secondary

ReplicaSet3

Primary

Secondary

Secondary

keyrange0..30

keyrange31..60

keyrange61..100

Sh di


51/77

Sharding

Scale horizontally for data size, index size, write andconsistent read scaling

Distribute databases, collections or a objects in a

collection

Auto-balancing, migrations, management happenwith no down time

Replica Sets for inconsistent read scaling

for inconsistent read scaling

Sh di


52/77

Sharding

Choose how you partition data Can convert from single master to sharded systemwith no downtime

Same features as non-sharding single master

Fully consistent

R B d


53/77

Range Based

collection is broken into chunks by range chunks default to 200mb or 100,000 objects

Architecture


54/77

Architecture

client

mongos ...mongos

mongodmongod

mongod mongod

mongod

mongod ...

Shards

mongod

mongod

mongod

Config

Servers

Writes


55/77

Writes

Inserts : require shard key, routed Removes: routed and/or scattered Updates: routed or scattered

Queries


56/77

Queries

By shard key: routed Sorted by shard key: routed in order By non shard key: scatter gather

Sorted by non shard key: distributed merge sort

Part Four


57/77

Part FourDeveloping with MongoDB

Platform and Language support


58/77

Platform and Language support

MongoDB is Implemented in C++ for best performance

Platforms 32/64 bit

Windows Linux, Mac OS-X, FreeBSD, Solaris

Language drivers for

Java Ruby / Ruby-on-Rails C# C / C++ Erlang Python, Perl, JavaScript others...

.. and much more ! ..

ease of development a surprisingly big benefit : faster to code, faster to change, avoid upgrades and scheduled downtime

more predictable performance

fast single server performance -> developer spends less time manually coding around the database

bottom line: usually, developers like it much better after trying

MongoDB features


59/77

MongoDB features

Durability Replication Sharding Connection options


60/77

Durability

What failures do you need to recover from?

Loss of a single database node? Loss of a group of nodes?


61/77

Durability - Master only

Write acknowledgedwhen in memory on

master only


62/77

Durability - Master + Slaves

Write acknowledged whenin memory on master +slave

Will survive failure of asingle node

Durability - Master + Slaves +


63/77

Durability - Master + Slaves +fsync Write acknowledged when inmemory on master + slaves

Pick a majority of nodes

fsync in batches (since itblocking)


64/77

Setting default error checking

//Donotcheckorreporterrorsonwritecom.mongodb.WriteConcern.NONE;

//Usedefaultleveloferrorcheck.Donotsend

//agetLastError(),butraiseexctiononerrorcom.mongodb.WriteConcern.NORMAL;

//SendgetLastError()aftereachwrite.Raisean//exceptiononerrorcom.mongodb.WriteConcern.STRICT;

//Settheconcerndb.setWriteConcern(concern);


65/77

Customized WriteConcern//Waitforthreeserverstoacknowledgewrite

WriteConcernconcern=newWriteConcern(3);

//Waitforthreeservers,witha1000mstimeout

WriteConcernconcern=newWriteConcern(3,1000);

//Waitfor3server,100mstimeoutandfsync//datatodiskWriteConcernconcern=newWriteConcern(3,1000,true);//Settheconcerndb.setWriteConcern(concern);

Using Replication


66/77

Using Replication

slaveOk()

- driver to send read requests to Secondaries- driver will always send writes to Primary

Can be set on-DB.slaveOk()

-Collection.slaveOk()

-find(q).addOption(Bytes.QUERYOPTION_SLAVEOK);

Using sharding


67/77

Using sharding

Before sharding

coll.save(

newBasicDBObjectBuilder(author,Herg).

append(text,DestinationMoon). append(date,newDate());

Queryq=ds.find(Blog.class,author,Herg);

After sharding

Nocodechangerequired!

Connection options


68/77

Connection options

MongoOptionsmo=newMongoOptions();

//Restrictnumberofconnections

mo.connectionsPerHost=MAX_THREADS+5;

//Autoreconnectiononconnectionfailure

mo.autoConnectRetry=true;

Part Five


69/77

Part FiveDeploying MongoDB

Part Five


70/77

a t eDeploying MongoDB

Performance tuning

Sizing O/S Tuning / File System layout Backup

Back p


71/77

Backup

Typically backups are driven from a slave Eliminates impact to client / application trafc to master

Slave delay


72/77

y

Protection against app faults Protection against administration mistakes

O/S Config


73/77

O/S Config

RAM - lots of it

Filesystem EXT4 / XFS Better file allocation & performance

I/O

More disk the better Consider RAID10 or other RAID configs

Monitoring


74/77

Monitoring

Munin, Cacti, Nagios

Primary function:

Measure stats over time Tells you what is going on with

your system

Alerts when threshold reached

Remember me?


75/77

Summary


76/77

MongoDB makes building applications simple

You can focus on what the apps needs to do

MongoDB has built-in

Horizontal scaling (reads and writes) Simplified schema evolution Simplified deployed and operation Best match for development tools and agile processes

download at mongodb org


77/77

@mongodb

conferences,appearances,andmeetupshttp://www.10gen.com/events

http://bit.ly/mongoK

Facebook|Twitter|LinkedInhttp://linkd.in/joinmongo

download at mongodb.org

Were Hiring [email protected]

cloudstock 2010

Documents