mongodb 101 (session one) - percona · mongodb 101 (session one) art van scheppingen senior support...

Confidential

MongoDB 101 (session one)

Art van Scheppingen

Senior Support Engineer - Severalnines AB

Confidential

Who am I?

☐ Senior Support Engineer at Severalnines AB☐ Worked with MySQL for over 16 years☐ Has been in a DBA environment for 6 years☐ Polyglot peristence proponent☐ Organizer of Polyglot Persistence meetups (Serveralnines)

☐ Amsterdam, Berlin, Paris, London, Stockholm and Dublin

Confidential

Who is Severalnines?

☐ Database automation / orchestration☐ Software to deploy, monitor, manage and scale☐ Support for MySQL (all flavours), MongoDB and PostgreSQL☐ Main product: ClusterControl

Confidential

Orchestration System: ClusterControl

☐ ClusterControl☐ http://severalnines.com/getting-started☐ Deploy☐ Monitor☐ Manage☐ Scale

☐ Community edition

http://severalnines.com/getting-started

http://severalnines.com/getting-started

Confidential

MongoDB support

Confidential

Logistics

Confidential

Morning / Afternoon sessions

☐ Morning session - Art van Scheppingen☐ Basics☐ Cluster/Schema Design patterns

☐ Afternoon session - David Murphy / Kim☐ Sharding☐ Engines☐ Common Operations issues

Confidential

Agenda for this morning

☐ Part 1: Basics (9:00 - 10:30)☐ MongoDB Primer☐ Running mongod☐ Mongo command basics☐ CRUD: create, read, update, delete

Confidential

Agenda for this morning

☐ Part 3: Essentials (11:00 - 12:00)☐ Aggregate☐ Import / export data☐ Backup / restore☐ Schema design patterns

Confidential

Prerequisites

☐ MongoDB community locally installed☐ https://www.mongodb.com/download-center

☐ Download the zip code data set:☐ http://media.mongodb.org/zips.json

☐ You have to know and understand JSON data structures

https://www.mongodb.com/download-center

https://www.mongodb.com/download-center

http://media.mongodb.org/zips.json

http://media.mongodb.org/zips.json

Confidential

Prerequisites for this afternoon

☐ Clone the following github repo:☐ https://github.com/dbmurphy/MongoDB32Labs

☐ If you are not running on Linux: run a VM with Linux☐ Recommended install for VMs:

☐ Install VirtualBox ☐ https://www.virtualbox.org/wiki/Downloads

☐ Download Fedora or Ubuntu☐ https://getfedora.org/en/cloud/download/

☐ https://cloud-images.ubuntu.com/xenial/current/

https://github.com/dbmurphy/MongoDB32Labs

https://github.com/dbmurphy/MongoDB32Labs

https://www.virtualbox.org/wiki/Downloads

https://www.virtualbox.org/wiki/Downloads

https://getfedora.org/en/cloud/download/

https://getfedora.org/en/cloud/download/

https://cloud-images.ubuntu.com/xenial/current/

https://cloud-images.ubuntu.com/xenial/current/

Confidential

Windows users

☐ Windows installations require the Windows powershell☐ Set path: $env:Path += ";c:\Program Files\MongoDB\Server\3.2\bin\"☐ The --fork parameter does not work in the Windows binary

☐ Solution: open multiple command / shell windows and leave MongoDB running in the foreground

Confidential

Basics: MongoDB

Confidential

What is MongoDB?

☐ Document data store☐ Not a key/value store!☐ Data stored in JSON

☐ Philosophy☐ Flexibility☐ Scalability☐ Geo distributed☐ Strong consistency

Confidential

What is MongoDB?

☐ Originally data was only stored as BSON on disk (MMAP)☐ BSON is binary JSON☐ MongoDB v3.0 allows other storage engines

☐ No-SQL?☐ Javascript based query language☐ Similar feature set in MongoDB queries

Confidential

What are MongoDB advantages?

☐ Doesn’t require a lot of memory☐ No preallocated buffer pools (except for WiredTiger)☐ Makes use of the filesystem cache to cache data☐ Indexes are loaded in memory

☐ Allows high levels of concurrency☐ Strong consistency☐ Easy for scaling reads

☐ Scale replicaSets

☐ Easy for scaling writes☐ Scale shards by adding replicaSets

Confidential

What are MongoDB disadvantages

☐ It is a different approach to problems☐ Different solutions

☐ Not ACID compliant☐ Atomicity only on collection (MMAP) or document (WiredTiger)☐ But there is transaction-like semantics

Confidential

Terminology

☐ Database☐ Contains collections

☐ Collections☐ Collection of documents (think of a table)

☐ Document☐ BSON document (may contain links to docs in other collections)☐ BSON is a binary representation of JSON☐ Document size is limited to 16MB (megabytes)

☐ Fields☐ The properties of a BSON object (think of columns)

Confidential

Basics: document example

{

"_id" : ObjectId("57e171765ffbf76ca639bd65"),

"foo" : "bar",

"counter" : NumberLong(10010101),

"array" : [

"one",

"two",

"three"

],

"subbranch" : {

"another" : "json",

"object" : "in here"

}

}

Confidential

Basics: fields

☐ Field names are strings☐ Field names may not start with the dollar sign character ($)

☐ Preserved for (matching) functions and operators

☐ Field names may not contain dot characters (.)☐ Dot notation is used to access arrays and embedded

documents

☐ Field names may not contain a null character☐ Field name “_id” is reserved

Confidential

Basics: accessing array and embedded fields

{

"_id" : ObjectId("57e171765ffbf76ca639bd65"),

"foo" : "bar",

"counter" : NumberLong(10010101),

"array" : [

"one",

"two",

"three"

],

"subbranch" : {

"another" : "json",

"object" : "in here"

}

}

"array.2"

"subbranch.object"

Confidential

Basics: single server

Confidential

Basics: Running mongod

☐ Running mongod in the foregroundmongodmongod --port <port> --host <hostname> --dbpath ~/data/db

☐ Running mongod in the background (fork)mongod --fork

Confidential

Basics: Checkpointing

☐ MongoDB checkpoints every 60 seconds (both MMAP and WiredTiger)

☐ Between checkpoints all modifications are written to the journal (every 100ms)

Confidential

Exercise 1: run mongod on your laptop

☐ Create the data directory and run mongod (Linux/Mac):mkdir plam101mongod --dbpath plam101 --logpath mongo-101.log --fork

☐ Create the data directory and run mongod (Windows):mkdir plam101mongod --dbpath plam101 --logpath mongo-101.log

Confidential

Exercise 1: connect to mongod

☐ Verify that mongod is running (Linux/MacOS):ps ax | grep mongodtail mongo-101.logmongo

☐ Verify that mongod is running (Windows):Get-Process mongod

Get-Content -Path mongo-101.log

mongo

Confidential

Exercise 1: directory layout

$ ls -la plam101/total 163848drwxr-xr-x 8 youruser staff 272 Sep 30 11:48 .drwxr-xr-x 5 youruser staff 170 Sep 30 11:48 ..drwxr-xr-x 2 youruser staff 68 Sep 30 11:48 _tmpdrwxr-xr-x 2 youruser staff 68 Sep 30 11:48 journal-rw------- 1 youruser staff 67108864 Sep 30 11:48 local.0-rw------- 1 youruser staff 16777216 Sep 30 11:48 local.ns-rw-r--r-- 1 youruser staff 0 Sep 30 11:48 mongod.lock-rw-r--r-- 1 youruser staff 69 Sep 30 11:48 storage.bson

Confidential

Creating databases

☐ MongoDB databases are created implicitly when changing to a non existent database and inserting data

> use new_database

switched to db new_database

> show databases

local 0.000GB

> db.somecollection.insert({"foo": "bar"}) WriteResult({ "nInserted" : 1 })

> show databases

local 0.000GB

new_database 0.000GB

Confidential

Creating databases: directory layout

$ ls -la plam101/total 163848drwxr-xr-x 8 youruser staff 272 Sep 30 11:48 .drwxr-xr-x 5 youruser staff 170 Sep 30 11:48 ..drwxr-xr-x 2 youruser staff 68 Sep 30 11:48 _tmpdrwxr-xr-x 2 youruser staff 68 Sep 30 11:48 journal-rw------- 1 youruser staff 67108864 Sep 30 11:48 local.0-rw------- 1 youruser staff 16777216 Sep 30 11:48 local.ns-rw-r--r-- 1 youruser staff 0 Sep 30 11:48 mongod.lock-rw-r--r-- 1 youruser staff 69 Sep 30 11:48 storage.bson-rw------- 1 youruser staff 67108864 Sep 30 11:52 new_database.0-rw------- 1 youruser staff 16777216 Sep 30 11:52 new_database.ns

Confidential

Dropping databases

☐ To drop a database you have to change to the database you want to drop, only then you can drop it

> use new_database

switched to db new_database

> db.dropDatabase()

{ "dropped" : "new_database", "ok" : 1 }

Confidential

Basics: replicaSets

Confidential

Basics: replicaSet

Confidential

Basics: replicaSet

☐ Replication is transported through the “oplog”☐ The oplog is a special collection in a replicaSet

☐ All transactions are stored in the oplog (except for local db)☐ Oplog resides inside the local database☐ Limited in size (on disk)☐ Sliding window of transactions☐ Purging of transactions happens via FIFO

☐ Oplog durability is one of the most important metrics☐ Select first and last transaction from the oplog: durability in sec.

☐ All nodes send/receive heartbeats

Confidential

Basics: initial sync

☐ Adding a new secondary:☐ Node gets added to the cluster

☐ Cluster will check how advanced the new secondary is (last executed transaction)

☐ If secondary is too far behind an initial sync is executed☐ Copy document by document

☐ Kickstarting / seeding☐ Make a full (binary) copy of the primary to the secondary☐ Add to the cluster☐ Only last transactions from the oplog will be sent

Confidential

Exercise 2: run a replicaSet

☐ Kill previous daemonkillall mongod

☐ Create the data directories and run mongod:mkdir plam101-rs1 plam101-rs2 plam101-rs3mongod --dbpath plam101-rs1 --logpath mongo-101-rs1.log --port 27001 --replSet myrs

--forkmongod --dbpath plam101-rs2 --logpath mongo-101-rs2.log --port 27002 --replSet myrs

--forkmongod --dbpath plam101-rs3 --logpath mongo-101-rs3.log --port 27003 --replSet myrs

--fork

Confidential

Exercise 2: initiate the replicaSet

☐ Connect to the first nodeMongoconnecting to: 127.0.0.1:27001/test>

☐ Initiate the replicaSet:> rs.initiate(){ "info2" : "no configuration explicitly specified -- making one", "me" : "<yourhost>.local:27001", "ok" : 1}myrs:SECONDARY>myrs:PRIMARY> rs.status()

Confidential

Exercise 2: add new members

☐ Connect to the first node$ mongomyrs:PRIMARY>

☐ Add the other members:myrs:PRIMARY> rs.add("<yourhost>.local:27002"){ "ok" : 1 }myrs:PRIMARY> rs.add("<yourhost>.local:27003"){ "ok" : 1 }

Confidential

Exercise 2: Watch the oplog grow

☐ Run rs.printReplicationInfo()myrs:PRIMARY> rs.printReplicationInfo()

☐ Insert some big rows: (should take a few minutes)myrs:PRIMARY> var doc = {"foo": "bar"}

myrs:PRIMARY> for (i = 0; i < 10000; i++) { doc["foo"] += i; db.inserttest.insert(doc);

}

WriteResult({ "nInserted" : 1 })

☐ Run rs.printReplicationInfo() and spot the differencemyrs:PRIMARY> rs.printReplicationInfo()

Confidential

Exercise 2: Maybe adjust the oplog size?

myrs:PRIMARY> rs.printReplicationInfo()

configured oplog size: 192MB

log length start to end: 183secs (0.05hrs)

oplog first event time: Tue Sep 27 2016 16:19:36 GMT+0200 (CEST)

oplog last event time: Tue Sep 27 2016 16:22:39 GMT+0200 (CEST)

now: Tue Sep 27 2016 16:25:22 GMT+0200 (CEST)

Confidential

Demo: oplog too small, full sync necessary

☐ Stop one secondary☐ Insert some big rows: (should take a few minutes)myrs:PRIMARY> var doc = {"foo": "bar"}

myrs:PRIMARY> for (i = 0; i < 10000; i++) { doc["foo"] += i; db.inserttest.insert(doc);

}


☐ Start secondary again and watch log file

Confidential

Basics: High Availability

Confidential

Basics: node failure

Confidential

Basics: node failure

☐ After a primary is lost (heartbeat timeout) no write operations can happen

☐ Read operations can still happen☐ Remaining nodes start electing a new primary

Confidential

Basics: election voting

Confidential


☐ All remaining nodes vote for a new primary☐ Priority: higher values make a node more eligible to become a

primary☐ Votes: allows a node to vote for a new primary☐ Only nodes with priority and voting power can vote☐ You can set the priority (numeric) per node

☐ You can set the voting (on and off) per node

☐ Up to 7 nodes can vote

Confidential


Confidential

Basics: node recovery

Confidential

Basics: durability

☐ Replication happens, like MySQL replication, asynchronously☐ Eventual consistency

☐ Writeconcern☐ Wait for confirmation from secondary nodes☐ numeric, majority or <tag>

☐ Wait for write to journal

Confidential

Basics: durability

Confidential

Basics: understanding eventual consistency

Confidential

Basics: read from secondary

Confidential

Basics: eventual consistency and secondary

Confidential

Basics: arbiter node

Confidential

Basics: arbiter node

☐ The arbiter node will not store any data☐ The arbiter node will confirm writes☐ The arbiter node will take part in voting for a new primary

Confidential

Demo: node recovery on a replicaSet

Confidential

Demo: node recovery on a replicaSet

☐ 3 node cluster (node 3 is arbiter)☐ Data gets inserted on Primary (node 1)☐ Secondary (node 2) fails☐ Some time passes☐ Primary (node 1) fails and node 2 comes back up☐ Node 2 becomes primary☐ Inserting data into new primary (node 2)☐ Node 1 comes up again and realizes it is no longer primary☐ Fetches oplog from the new primary which is more advanced☐ Node 1 performs a rollback

Confidential

Basics: Sharding

Confidential

Basics: sharded cluster

Confidential

Demo: sharding

Confidential

Running commands in the mongo shell

☐ MongoDB built in commands☐ db.help() is your friend

☐ Collection built in commands☐ db.<collection>.help()

☐ For replicaSets and shards the helpers start with rs and sh☐ rs.help()☐ sh.help()

Confidential

Scripting in the mongo shell

☐ Mongo shell runs Javascript☐ Create variables, functions, etc☐ You can iterate over cursors

cursor = db.collection.find();

while ( cursor.hasNext() ) {

printjson( cursor.next() );

}

cursor = db.collection.find();

while ( cursor.hasNext() ) {

doc = cursor.next(); doc["newfield"] = "something"; db.collection.save(doc);

}

Confidential

Recap

☐ MongoDB basics☐ MongoDB terminology☐ replicaSet and replication☐ Durability and eventual consistency☐ Sharding

☐ Running MongoDB☐ Single server☐ ReplicaSet

Confidential

Basics: CRUD

Confidential

Create: inserting data

☐ As you have seen, inserting data is very simple:db.<collection>.insert(document, { [writeConcern], [ordered] })db.<collection>.insertOne(document, { [writeConcern] }) (3.2)db.<collection>.insertMany(document, { [writeConcern], [ordered] }) (3.2)

☐ Comparable to SQLINSERT INTO <collection> VALUES (document);

☐ Ordered defaults to true

Confidential

Create: inserting data

☐ Every document must have an identifier (_id)☐ If no identifier (_id) has been provided, one will be generated

☐ For example:> db.somecollection.insert({"foo": "bar"})


> db.somecollection.insert({"_id": "1234","foo": "bar"})


> db.somecollection.insert({"_id": "foobar","foo": "bar"})


Confidential

Create: inserting multiple documents

> db.somecollection.insert( [ {"foo": "bar"}, {"_id": "1234","foo": "bar"}, {"_id":

"foobar","foo": "bar"} ] )

BulkWriteResult({

"writeErrors" : [ ],

"writeConcernErrors" : [ ],

"nInserted" : 3,

"nUpserted" : 0,

"nMatched" : 0,

"nModified" : 0,

"nRemoved" : 0,

"upserted" : [ ]

})

Confidential

Insert: insert durability (writeConcern)

☐ Similar to the insert method:db.<collection>.insert(document, { writeconcern: {w: <value>, j: <boolean>, wtimeout: <number>}})

☐ W option☐ Wait for confirmation from other nodes☐ Number, “majority” or <tag>

☐ J option☐ Setting to true will wait for the journal write

☐ Wtimeout option☐ Timeout for the writeconcern

Confidential

Insert: wait for journal write

Confidential

Insert: wait for other node to write

Confidential

Insert: insert durability

☐ Examples:> db.somecollection.insert({"foo": "bar"}, {writeconcern: { w: 1, j: false}} )


> db.somecollection.insert({"foo": "bar"}, {writeconcern: { w: majority, j: true}} )


> db.somecollection.insert({"foo": "bar"}, {writeconcern: { w: 2, j: true, wtimeout:

100}} )


Confidential

Create: inserting multiple documents

☐ As you have seen, inserting data is very simple:db.<collection>.insert(document, [writeConcern], [ordered])

☐ Comparable to SQLINSERT INTO <collection> VALUES (document);

☐ Ordered defaults to true.☐ writeConcern will be explained in the afternoon session

Confidential

Create: exercise

1. Create a new collection named mytest in the test database by inserting the following JSON document:

{ "_id": 1, "name": "mytest1" }

2. Insert a second document in the mytest collection:{ "_id": 2, "name": "mytest2", "testdata": "test1234" }

3. Insert a couple of documents in the mytest collection:[ { "name": "mytest3", "testdata": "test1234" },{ "name": "mytest4", "testdata": "test1234" }{ "name": "mytest5", "testdata": "test1234" } ]

Confidential

Create: exercise answer

> use test

> db.mytest.insert({ "_id": 1, "name": "mytest1" })


> db.mytest.insert({ "_id": 2, "name": "mytest2", "testdata": "test1234" })


> db.mytest.insert( [ { "name": "mytest3", "testdata": "test1234" }, { "name":

"mytest4", "testdata": "test1234" }, { "name": "mytest5", "testdata": "test1234" } ] )

BulkWriteResult({

"nInserted" : 3,

...

})

Confidential

Read: finding your data

☐ The find command will retrieve your data as a cursordb.<collection>.find(query, projection)db.<collection>.findOne(query, projection)var cursor = db.<collection>.find(query, projection)

☐ Query: selection filter using query operators☐ Projection: fields to return from the document☐ SQL equivalent:SELECT projection FROM collection WHERE query

Confidential


☐ Example:> db.somecollection.find({"_id": "1234"}, {"_id": 1, "foo": 1})

{ "_id" : "1234", "foo" : "bar" }

> var cursor = db.somecollection.find({"_id": "1234"}, {"_id": 1, "foo": 1})

> while (cursor.hasNext()) { printjson(cursor.next()); }

{

"_id" : "1234",

"foo" : "bar"

}

Confidential

Read: query operators

☐ Basic query operators are $eq, $gt, $gte, $lt, $lte, $ne, $in, $nin

☐ Logical query operators are $and, $or, $not, $nor☐ Element (array) operators are $exists, $type☐ Other noteworthy operators are $regex, $text, $geoWithin

See also: https://docs.mongodb.com/manual/reference/operator/

Confidential


☐ Example: all documents with sale value less than 100> db.somecollection.find({"sale_value": {"$lt":"100"} })

{ "_id" : "1002", "sale_value" : 75 }

{ "_id" : "1004", "sale_value" : 52 }

{ "_id" : "1008", "sale_value" : 95 }

Confidential

Read: sort and limit

☐ You can sort a result by appending the query with the sort function:

db.somecollection.find().sort({"_id": 1})

db.somecollection.find().sort({"_id": -1})

db.somecollection.find().sort({"_id": 1, "foo": -1})

See also: https://docs.mongodb.com/manual/reference/method/cursor.sort/

☐ Limiting a result is done similarly:db.somecollection.find().limit(10)

See also: https://docs.mongodb.com/manual/reference/method/cursor.limit/

Confidential

Read: exercise

☐ Prior to this exercise, import the zipcodes data set:$ mongoimport -d test -c zipcodes zips.json

1. Find the first document2. Find the last document3. Find the zipcodes with a population greater than 100,000

Confidential

Read: exercise answers

1. Find the first document> db.zipcodes.findOne()

> db.zipcodes.find().limit(1)

{ "_id" : "01001", "city" : "AGAWAM", "loc" : [ -72.622739, 42.070206 ], "pop" : 15338,

"state" : "MA" }

2. Find the last document> db.zipcodes.find().sort({"_id": -1}).limit(1)

{ "_id" : "99950", "city" : "KETCHIKAN", "loc" : [ -133.18479, 55.942471 ], "pop" : 422,

"state" : "AK" }

3. Find the zipcodes with a population greater than 100,000> db.zipcodes.find( { "pop": { "$gt": 100000}} )

Confidential

Update: update method

☐ The update command enables you to update one or many rows

db.<collection>.update(query, update, options)

☐ Query: selection filter using query operators (same as find)☐ Update: modification to apply☐ Options: upsert, multi, and writeConcern☐ SQL equivalent:UPDATE <collection> SET update WHERE query

Confidential

Update: update operators

☐ Most important update operators:☐ $set and $unset will update/remove the specified field(s)☐ $inc and $mul will operate on the value of the field☐ $rename will rename a field

See also: https://docs.mongodb.com/manual/reference/operator/update/

Confidential

Update: the update options

☐ Upsert☐ if document exists: update or else insert a new document

☐ Multi☐ By default only one document gets updated☐ Setting multi to true will update multiple documents at once

New in 3.2:db.<collection>.updateOne(query, update, options)db.<collection>.updateMany(query, update, options)db.<collection>.replaceOne(query, update, options)

Confidential

Update: update durability

☐ Similar to the insert method:db.<collection>.update(query, update, { w: <value>, j: <boolean>, wtimeout: <number>})

☐ W option☐ Number, “majority” or <tag>

☐ J option☐ Setting to true will wait for the journal write

☐ Wtimeout option☐ Timeout for the writeconcern

Confidential

Update: example update

☐ Example:db.somecollection.update( {"_id": ObjectId("57e171765ffbf76ca639bd65")}, { $set: { "foo": "barbar", "array.1": "four" }, $inc: {"counter": 2} })

Confidential

Update: example document update

☐ Example:db.somecollection.update( {"_id": ObjectId("57e171765ffbf76ca639bd65")}, { "replace": "all", "contents": ["with","this","new","document"] })

Confidential

Update: save method

☐ The save method performs either an insert or update command

db.<collection>.save(document, writeConcern)

☐ If no _id field has been provided an insert will be performed.☐ The _id field will be filled with an ObjectID

☐ If an _id field has been provided an update will happen☐ Update will be performed with upsert enabled

Confidential

Update: example save

☐ Example:db.somecollection.save({"foo": "bar"})db.somecollection.save({"_id", "1234","foo": "bar"})

Confidential

Update: example save complex

☐ Example:> var doc=db.somecollection.findOne()> doc{ "_id" : ObjectId("57e16f925ffbf76ca639bd64"), "foo" : "bar" }> doc["counter"]=00> db.somecollection.save(doc)WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })> db.somecollection.findOne()

{

"_id" : ObjectId("57e16f925ffbf76ca639bd64"),

"foo" : "bar",

"counter" : 0

}

Confidential

Update: exercise

1. Increase the population of zipcode 90210 (Beverly Hills) by 12. Iterate over the zipcodes collection and add a new field called

“votes” with a value of 0

Confidential

Update: exercise answers

1. Increase the population of zipcode 90210 (Beverly Hills) by 1> db.zipcodes.update( { "_id": "90210" }, { "$inc": {"pop": 1}})WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })

2. Iterate over the zipcodes collection and add a new field called “votes” with a value of 0

> var cur = db.zipcodes.find()> while (cur.hasNext()) { var zip = cur.next(); zip["votes"] = 0; db.zipcodes.save(zip); }WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 0 })

Confidential

Delete: removing documents

☐ To remove documents you can use the following methods:db.<collection>.remove(query, options) db.<collection>.deleteOne(filter, options) //new in 3.2db.<collection>.deleteMany(filter, options) // new in 3.2

☐ Query and filter are basically the same☐ Pass empty document ( {} ) to match all documents

☐ Options are writeConcern and justOne (only for remove method)

Confidential

Recap

☐ Creating databases / collections☐ Dropping databases / collections☐ CRUD

☐ Creating data☐ Reading data☐ Updating data☐ Deleting data

Confidential

Break: 10:30 - 11:00

Next hour: basic management and design patterns

Confidential

Aggregations

Confidential

CRUD bonus: aggregate

☐ The aggregate method fills the gaps for the find method☐ Creates aggregates of matching documents☐ Aggregate allows multiple pipelines☐ Most important pipelines:

☐ $unwind: Unwinds arrays (e.g. like a JOIN)☐ $group: Groups documents together (like GROUP BY)

☐ $match: Only match certain documents. When used in a second pipeline it acts as a HAVING

☐ $sort: Sorts the result set (like ORDER BY)

Confidential

Aggregate: group accumulators

☐ Most important accumulators:☐ $sum☐ $avg☐ $min☐ $max☐ $push / $addToSet

Confidential

Aggregate: match / group example

> db.somecollection.find()

{ "_id" : ObjectId("57e16f925ffbf76ca639bd64"), "foo" : "bar",

"counter" : 1 }

{ "_id" : "1234", "foo" : "bar" }

{ "_id" : "2345", "foo" : "barbar" }

{ "_id" : "foobar", "foo" : "bar" }

> db.somecollection.aggregate([ { $match: {"foo": "bar"}}, { $group: { "_id": "$foo", "count": {$sum: 1}}}

]){ "_id" : "bar", "count" : 3 }

Confidential

Aggregate: unwind / group example (1)

> db.someothercol.find()

{ "_id" : 1, "name" : "blogpost 1", "tags" : [ "music", "literature" ] }

{ "_id" : 2, "name" : "blogpost 2", "tags" : [ "dogs", "cats", "kittens" ] }

{ "_id" : 3, "name" : "blogpost 3", "tags" : [ "memes" ] }

{ "_id" : 4, "name" : "blogpost 4", "tags" : [ "memes", "kittens" ] }

Confidential


> db.someothercol.aggregate([{$unwind: "$tags"}])

{ "_id" : 1, "name" : "blogpost 1", "tags" : "music" }

{ "_id" : 1, "name" : "blogpost 1", "tags" : "literature" }

{ "_id" : 2, "name" : "blogpost 2", "tags" : "dogs" }

{ "_id" : 2, "name" : "blogpost 2", "tags" : "cats" }

{ "_id" : 2, "name" : "blogpost 2", "tags" : "kittens" }

{ "_id" : 3, "name" : "blogpost 3", "tags" : "memes" }

{ "_id" : 4, "name" : "blogpost 4", "tags" : "memes" }

{ "_id" : 4, "name" : "blogpost 4", "tags" : "kittens"

Confidential


db.someothercol.aggregate([

{$unwind: "$tags"},

{$group: {"_id": "$tags", "count": {$sum: 1}}},

{$sort: {"count": 1}}

])

{ "_id" : "cats", "count" : 1 }

{ "_id" : "dogs", "count" : 1 }

{ "_id" : "literature", "count" : 1 }

{ "_id" : "music", "count" : 1 }

{ "_id" : "memes", "count" : 2 }

{ "_id" : "kittens", "count" : 2 }

Confidential

Aggregate: exercise

1. From the zipcodes collection, calculate the total population per city and sort by total population descending

2. From the zipcodes collection, calculate the average population per zipcode of New York (hint: the _id field is the zipcode)

Confidential

Aggregate: exercise

☐ From the zipcodes collection, calculate the total population per city and sort by total population descending

db.zipcodes.aggregate([ { $group: { "_id": "$city", "total_pop": {$sum: "$pop"}} }, { $sort: { "total_pop": -1} }])

Confidential

Aggregate: exercise

☐ From the zipcodes collection, calculate the average population per zipcode of New York

db.zipcodes.aggregate([ { $match: { "city": "NEW YORK"} }, { $group:{ "_id": "$_id", "average_pop": {$avg: "$pop"}} }])

Confidential

Basic Management

Confidential

Exporting data

☐ Exporting data can be done via mongoexport☐ Format limitations

☐ JSON or CSV☐ BSON rich documents are not supported☐ binData☐ objectId☐ Date

☐ Some tricks are applied when using mongoimport

☐ In general: not a reliable way to make backups!

Confidential

Exporting data to JSON

☐ Example:Mongoexport -d test -c mytest -o mytest.json

☐ Contents should be similar to this:{"_id":1.0,"name":"mytest1"}

{"_id":2.0,"name":"mytest2","testdata":"test1234"}

{"_id":{"$oid":"57ea8420506638730683f57c"},"name":"mytest3","testdata":"test1234"}

{"_id":{"$oid":"57ea8420506638730683f57d"},"name":"mytest4","testdata":"test1234"}

{"_id":{"$oid":"57ea8420506638730683f57e"},"name":"mytest5","testdata":"test1234"}

Confidential

Importing data

☐ Importing data can be done via mongoimport☐ Counterpart of mongoexport

☐ Format limitations☐ JSON or CSV

☐ Example:mongoimport -d test -c mytest mytest.json

Confidential

Types of backups

☐ Logical backups☐ Dump of your data

☐ Physical backups☐ File(system) copy of your data

Confidential

Logical backups

☐ Mongodump☐ MongoDB Backup☐ Mongob

Confidential

Logical backups: mongodump

☐ Mongodump☐ BSON dump of the data☐ BSON files per database / collection☐ Archive

☐ OEM tool☐ Works great but needs some wrapping

Confidential

Logical backups: MongoDB Backup

☐ MongoDB Backup☐ https://www.npmjs.com/package/mongodb-backup☐ Nodejs backup solution☐ CLI and API☐ Can stream backups

https://www.npmjs.com/package/mongodb-backup

https://www.npmjs.com/package/mongodb-backup

Confidential

Logical backups: Mongob

☐ Mongob☐ https://github.com/cmpitg/mongob☐ Python based CLI tool☐ MongoDB instance or bz2 target☐ Can copy data between collections☐ Incremental backups☐ Rate limiting

https://github.com/cmpitg/mongob

https://github.com/cmpitg/mongob

Confidential

Physical backups: Filesystem snapshots

☐ Filesystem snapshots☐ LVM☐ ZFS☐ XFS (xfs_freeze)☐ EBS

Confidential

Physical backups: Strata

☐ MongoRocks Strata☐ https://github.com/facebookgo/rocks-strata☐ Backs up on file level☐ Supports incremental backups☐ Queryable backups

https://github.com/facebookgo/rocks-strata

https://github.com/facebookgo/rocks-strata

Confidential

Restore

☐ To restore from a mongodump

Confidential

Exercise: backup using mongodump

1. Create a backup using mongodump2. Log into MongoDB and drop a collection3. Restore the collection using the dump created earlier

Confidential

Exercise: backup using mongodump

1. Create a backup using mongodump$ mongodump --gzip --archive=dump.gz

2. Log into MongoDB and drop a collection> db.inserttest.drop()

3. Restore the collection using the dump created earlier$ mongorestore --port 27003 -d test -c inserttest --gzip --archive=dump.gz

2016-09-27T19:33:19.574+0200 creating intents for archive

2016-09-27T19:33:19.704+0200 reading metadata for test.inserttest from archive

'dump.gz'

2016-09-27T19:33:19.731+0200 restoring test.inserttest from archive 'dump.gz'

2016-09-27T19:33:40.624+0200 restoring indexes for collection test.inserttest from

metadata

2016-09-27T19:33:40.635+0200 finished restoring test.inserttest (146411 documents)

2016-09-27T19:33:40.635+0200 done

Confidential

Schema design patterns

Confidential

Normalized data

{

_id: "@percona",

name: "Percona Twitter account"

}

{

twitter_id: "@percona",

joined: ISODate("2009-04-02"),

location: "Raleigh, NC 27617"

}

Confidential

Embedded document (One-to-one)

{

_id: "@percona",


info: {


location: "Raleigh, NC 27617"

}

}

Confidential

Embedded document (1 on many)

{

_id: "@percona",


lasttweets: [{

tweet_id: 780892298024456193,

tweettime: ISODate("2016-09-27T15:10:01"),

tweet: "Wed 11am PT @PeterZaitsev will go over highlights from the @Percona open

source software roadmap and time for Q/A http://hubs.ly/H04wLsv0"

},{

tweet_id: 780874621386158080,


tweet: "Problems solved, before they appear! Come to #PerconaLive to get hands on

training and more. http://hubs.ly/H04rr-J0"

}]

}

Confidential

How not to use embedded documents!

{

_id: 780892298024456193,



source software roadmap and time for Q/A http://hubs.ly/H04wLsv0",

twitterhandle: {


info: {


location: "Raleigh, NC 27617",

}

}

}

Confidential

Document references (1 on many)

{

_id: "@percona",


info: { joined: ISODate("2009-04-02"), location: "Raleigh, NC 27617" }

}

{

_id:780892298024456193,




twitterhandle: "@percona"

}

Confidential

Impact of various data models

☐ Document growth☐ Reallocation of the same document impacts performance☐ Writing to the same document often creates hotspots

Confidential


1 2 3 4 5

1 3 4 5

2

2

Confidential


☐ Atomicity☐ No single write operation can change more than one document☐ Writing to multiple documents is not atomic☐ Write all changes to a single document at the same time

Confidential


{

_id: "@percona",


lasttweets: [{

tweet_id: 780892298024456193,



source software roadmap and time for Q/A http://hubs.ly/H04wLsv0"

},{

tweet_id: 780874621386158080,


tweet: "Problems solved, before they appear! Come to #PerconaLive to get hands on

training and more. http://hubs.ly/H04rr-J0"

}]

}

Confidential


☐ Sharding☐ Sharding documents requires a shard key

☐ Choosing the right shard key is the start of your document structure

☐ Choosing the wrong shard key may impact performance

Confidential


☐ Example: which field to use as a shard key?{

_id:780892298024456193,




twitterhandle: "@percona"

}

Confidential


☐ Indexes☐ Every index consumes disk space and memory☐ Each new index has a negative impact on write performance☐ High read-to-write ratio will benefit from indexes☐ High write-to-read ratio will benefit from having less indexes

Confidential


☐ Number of collections☐ Having many collections has no performance penalty

☐ Having many collections will improve performance (concurrency)

☐ MMAPv1 limited in number of namespaces

☐ Large number of (small) documents☐ Can give more random disk access

Confidential

Recap

☐ Aggregations☐ Why you need to know about them

☐ Basic management☐ Import/export data☐ Backup/restore

☐ Schema design patterns

Confidential

Exercise: setting up the env

☐ Setup the environment for this afternoon☐ Clone git repo☐ Create a cluster by running the following command:

./build_process.sh

☐ This should build a sharded cluster using Percona Server MongoDB

Confidential

Lunch: 12:00 - 13:30

See you at the afternoon session !

mongodb 101 (session one) - percona · mongodb 101 (session one) art van scheppingen senior support...

Documents