mongodb training for java software engineers

© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers

MongoDBfor Java Software Engineers

Copyrights © Moshe [email protected]


MongoDBfor Java Software Engineers

Moshe KaplanScale Hacker

http://top-performance.blogspot.comhttp://blogs.microsoft.co.il/vprnd

© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers

About Me: It’s all About

3

Scale


NOSQL. ANSWER A NEEDIntroduction

4


5


The Consumer Revolution

6

http://topyaps.com/wp-content/uploads/2013/03/You-are-the-product.-You-feeling-something.jpg


At the fraction of the cost…

7


8

http://lifehacker.com/5697167/if-youre-not-paying-for-it-youre-the-product


Transportation

9


Moovit

10


The Medical Market Opportunities

11


MediSafe

12


13


Askem

14


Major Enablers: Mobile, Cloud and IT Commoditization

15


The Prime Suspect

16


17

Assumptions…


Where did it Fail?Get an Answer, Fast and Cheap


Where did it Fail?I Just Want “Class Persistency

Storage” and Changing Schema on Demand


Where did it Fail?Be Always Available, Even w/ an Old

Answer


Where did it Fail?Get Me Fast and Good Enough

Answer


Where did it Fail?Data is Too Big, and Storage is $$$

But CPU and Network are Even More

http://www.powerbyte.com/Isilon.html


Software Providers

23


It is all great, but…I Need to Meet Compliance

http://www.vision7.com/app_system/lib/image/content/PCI_compliance.jpg


It is all great, but…I Need a Vendor


It is all great, but…I Need Reporting

http://www.novell.com/communities/node/5851/get-ready-sentinel-61


It is all great, but…I Need Transactions

http://www.novell.com/communities/node/5851/get-ready-sentinel-61


It is all great, but…We Need Training for the Data Analysts

db.article.aggregate(

{ $group : {

_id : "$author",

docsPerAuthor : { $sum : 1 },

viewsPerAuthor : { $sum : "$pageViews" }

}}

);

< SUM(pageViews)

< SUM(1) = N< GROUP BY author


NOSQL MARKETIntroduction

29


When Should I Choose NoSQL?

• Eventually Consistent

• Document Store

• Key Value

30

http://guyharrison.squarespace.com/blog/tag/nosql


Key Value Store

• insert

• get

• multiget

• remove

• truncate

31

<Key, Value>http://wiki.apache.org/cassandra/API


Redis

• Very simple protocol (SMTP like)

• Amazing Performance (60Kqps ops on 1 CPU machine)

• Persistency to disk

• Very little security


Column Family Stores: Key Value Store (with benefits)

• insert

• get

• multiget

• remove

• truncate

33

http://wiki.apache.org/cassandra/API


Cassandra• Simple protocol

• Very Good Performance

• You have indexes (but limited)

• Data Model is a pain

• You need to design you data for queries:“Table per Query”


Document Databasesvar mydoc = {

_id: ObjectId("5099803df3f4948bd2f98391"), name: { first: "Alan", last: "Turing" }, birth: new Date('Jun 23, 1912'), death: new Date('Jun 07, 1954'), contribs: [

"Turing machine", "Turing test", "Turingery"

], views : NumberLong(1250000)

}

35


Database for Software Engineers

Class

Subclass

Document

Subdocument


37

MapReduce

http://blogs.microsoft.co.il/blogs/vprnd


HELLO. MY NAME IS MONGODBIntroduction

38


39

#5 Most Popular DB Engine

http://db-engines.com/en/ranking


Who is Using mongoDB?


Who is Behind mongoDB


Why MongoDB?What? Why?

JSON End to End

No Schema “No DBA”, Just Serialize

Write 10K Inserts/sec on virtual machine

Read Similar to MySQL

HA 10 min to setup a cluster

Sharding Out of the Box

GeoData Great for that

No Schema None: no downtime to create new columns

Buzz Trend is with NoSQL

42


What mongoDB is Made of?

43http://www.10gen.com/products/mongodb


Installation: Give Yourself 5min• Add to /etc/yum.repos.d/10gen.repo

• [10gen]

• name=10gen Repository

• baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64

• gpgcheck=0

• enabled=1

• yum –y install mongo-10gen mongo-10gen-server

• The Packages:

• mongo-10gen: tools

• mongo-10gen-server: mongod and mongos


The Ubuntu Way

• sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv7F0CEB10

echo "deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.0.list

sudo apt-get -y update

sudo apt-get install -y mongodb-org


Installation w/ Authentication• /etc/mongod.conf

• > mongo

• use admin db.createUser( {

user: "siteUserAdmin", pwd: “Pss0rdxxx", roles: [ { role: "userAdminAnyDatabase", db: "admin" } ]

} )


Mastering a New Query Language


Connect to the Database• Connect:

• > mongo

• Show current database:• >> db

• Show Databases• >> show databases;

• Show Collections• >> show collections; or show tables;


Databases Manipulation: Create & Drop• Change Database:

• >> use <database>

• Create Database

• Just switch and create an object…

• Delete Database

• > use mydb;

• > db.dropDatabase();


Collections Manipulation

• Create Collcation

>db.createCollection(collectionName)

• Delete Collection

> db.collectionName.drop()

Or just insert to it


SELECT: No SQL, just ORM…

• Select All

• db.things.find()

• WHERE

• db.posts.find({“comments.email” : ”[email protected]”})

• Pattern Matching

• db.posts.find( {“title” : /mongo/i} )

• Sort

• db.posts.find().sort({email : 1, date : -1});

• Limit

• db.posts.find().limit(3)


NoSQL and Data ModelingWhat is the Difference


Database for Software Engineers

Class

Subclass

Document

Subdocument


Same Terminology

• Database Database

• Table Collection

• Row Document


A Blog Case Study in MySQL

http://www.slideshare.net/nateabele/building-apps-with-mongodb


as a SW Engineer would like it to be…

http://www.slideshare.net/nateabele/building-apps-with-mongodb


Migration from RDBMS to NoSQLHow to do that?


Data Migration• Map the table structure

• Export the data and Import It

• Add Indexes

58

http://igcse-geography-lancaster.wikispaces.com/1.2+MIGRATION


Selected Migration Tool

59


Usage Details> Install ruby

> gem install mongify

… Modify the code to your needs

… Create configuration files

> mongify translation db.config > translation.rb

> mongify process db.config translation.rb

60


Date Functions

• Year(), Month()… function included

• … buy only in the JavaScript engine

• Solution: New fields!• [original field]

• [original field]_[year part]

• [original field]_[month part]

• [original field]_[day part]

• [original field]_[hour part]

61


NO SCHEMA IS A GOOD THING BUT…Schemaless

62


Default Values

• No Schema

• No Default Values

• App Challenge

• Timestamps… No single source of truth

63


Casting and Type Safety

• No Schema

• No …

• App Challenge

64


Auto Numbers• Start using _id{

"_id" : 0,"health" : 1,"stateStr" : "PRIMARY","uptime" : 59917

}

• Counter tables • Dedicated database• 1:1 Mapping• Counter++ using findAndModify

65


ODM Solution

66


Data Analysts

67http://www.designersplayground.com/pr/internet-meme-list/data-analyst-2/


Data Analysts

68

Pentaho

RockMongoMongoVUE RoboMongo

MongoChef


No Joins

• Do in the application

• Leverage the power of NoSQL

69http://www.slideshare.net/nateabele/building-apps-with-mongodb


Limited Resultset

70

• 16MB document size

• GridFS


Bottom Line

• Powerful tool

• Embrace the Challenge

• Schema-less limitations: counters, data types

• Tools for Data Scientists

• Data design

71


Mastering a New Query Language


Connect to the Database• Connect:

• > mongo

• Show current database:• >> db

• Show Databases• >> show databases;

• Show Collections• >> show collections; or show tables;


Databases Manipulation: Create & Drop• Change Database:

• >> use <database>

• Create Database

• Just switch and create an object…

• Delete Database

• > use mydb;

• > db.dropDatabase();


Collections Manipulation

• Create Collcation

>db.createCollection(collectionName)

• Delete Collection

> db.collectionName.drop()

Or just insert to it


SELECT: No SQL, just ORM…

• Select All

• db.things.find()

• WHERE

• db.posts.find({“comments.email” : ”[email protected]”})

• Pattern Matching

• db.posts.find( {“title” : /mongo/i} )

• Sort

• db.posts.find().sort({email : 1, date : -1});

• Limit

• db.posts.find().limit(3)


Specific fields

Select Alldb.users.find(

{ },

{ user_id: 1, status: 1, _id: 0 }

)

1: Show; 0: don’t show


WHERE

• != “A” { $ne: "A" }

• > 25 { $gt: 25 }

• > 25 AND <= 50 { $gt: 25, $lte: 50 }

• Like ‘bc%’ /^bc/

• < 25 OR >= 50 { $or : [ { $lt: 25 }, { $gte : 50 } ] }


Join

• Wrong Place…

• Or Map Reduce


80

db.article.aggregate(

{ $group : {

_id : { author : "$author“, name : “$name” },

docsPerAuthor : { $sum : 1 },

viewsPerAuthor : { $sum : "$pageViews" }

}}

);

GROUP BY

< GROUP BY author, name

< SUM(pageViews)

< SUM(1) = N


81

db.Movie.aggregate([

{$match: {SeriesType : "F", MovieID : {$in : arrMovies}}

},{$project:

{MovieID: "$MovieID", SeriesType: "$SeriesType",Genres: "$Genres"}

},

{$unwind : "$Genres" },

{$group : { _id : "$Genres" , count : { $sum : 1 } } },

{$sort : { count: -1 }}

GROUP BY

WHERE

Keep some fields

Genres is an array

Counting and sorting


Aggregation Framework OperatorsOperator Description

$project Adding/Removing fields

$match WHERE

$redact Changes document based on Doc content/structure

$limit First N documents

$skip Skips N docs

$unwind Turns array into a multiple documents

$group Group

$sort Sort

$geoNear Geo spatial

$out Write Output to collection


83

db.posts.update({“comments.email”: ”[email protected]”},{$set : {“comments.email”: ”[email protected]”}}}

SET age = age + 3• db.users.update(• { status: "A" } ,• { $inc: { age: 3 } },• { multi: true }• )

UPDATE


84

j = { name : "mongo" }

k = { x : 3 }

db.things.insert( j )

db.things.insert( k )

INSERT


85

db.users.remove(

{ status: "D" }

)

DELETE


86

Every operation on a document is atomic

Two Phase Commit implementation is up to you

Atomic Transactions: Single Row


87

Multiple documents at oncedb.foo.update({ status : "A" , $isolated : 1 },{ $inc : { count : 1 } },{ multi: true }

)

Disclaimers:

• Sharding is not supported

• Not all or nothing (no roll back on failure)

Atomic Transactions: $isolated


88

t = db.transactions.findAndModify({ query: { state: "initial“

}, update: {

$set: { state: "pending"

}, $currentDate: { lastModified: true }

}, new: true})

Atomic Transactions: findAndModify


89

If it is about complex transactions.

Simplify the case.

or Consider keeping w/ RDBMS

Atomic Transactions: Bottom Line


90

Failure and order:

• db.collection.initializeOrderedBulkOp()

• db.collection.initializeUnorderedBulkOp()

1000 ops/bulk:var bulk = db.items.initializeUnorderedBulkOp();bulk.insert( { item: "abc123", defaultQty: 100, status: "A", points: 100 } );bulk.insert( { item: "ijk123", defaultQty: 200, status: "A", points: 200 } );bulk.insert( { item: "mop123", defaultQty: 0, status: "P", points: 0 } );bulk.execute();

Bulk Operations


91

Create a new project

Get the Maven configuration for MongoDB Java Driver

• http://mongodb.github.io/mongo-java-driver/

Project Setup


92

List l = new ArrayList();

/**** Insert ****/

// create a document to store key and value

for (int i = 1; i < 1000000; ++i) {

Document document = new Document()

.append("name", "Moshe Kaplan")

.append("age", 36 + i)

.append("createdDate", new Date());

l.add(document);

}table.insertMany(l);

Bulk Ops in Java


93

List<String> continentList = Arrays.asList(new String[]{"Africa", "Europe", "Asia"});

DBObject match = new BasicDBObject("$match", new BasicDBObject("continent.name", new BasicDBObject("$in", continentList)));

DBObject projectFields = new BasicDBObject("continent.name", 1);

projectFields.put("area", 1);

projectFields.put("_id", 0);

DBObject project = new BasicDBObject("$project", projectFields );

DBObject groupFields = new BasicDBObject( "_id", "$continent.name");

groupFields.put("average", new BasicDBObject( "$avg", "$area"));

DBObject group = new BasicDBObject("$group", groupFields);

List agList = new ArrayList();

agList.add(match);

agList.add(project);

agList.add(group);

MongoCursor<Document> cursor = countries.aggregate(agList).iterator();

while (cursor.hasNext()) {

System.out.println(cursor.next());

}

Aggregation Framework in Java


Performance TuningMake a Change


MONGODB TUNING

95


journalCommitInterval = 300:

Write to disk: 2ms <= t <= 300ms

Default 100ms, increase to 300ms to save resources

Disk

The Journal

96

Memory

Journal Data

1 2


RAM Optimization: dataSize + indexSize < RAM

97

OS

Data Index

Journal


PROFILING AND SLOW LOG

98


Profiling Configuration• Enable:

• mongod --profile=1 --slowms=15

• db.setProfilingLevel([level] , [time])

• How much:

• 0 (none) 1 (slow queries only) 2 (all)

• 100ms: default

• Where:

• system.profile collection @ local db

99


Profiling Results Analysis• Last 5 >1ms: show profile• w/o commands:db.system.profile.find( { op: { $ne : 'command' } } ).pretty()• Specific database:db.system.profile.find( { ns : 'mydb.test' } ).pretty()• Slower than:db.system.profile.find( { millis : { $gt : 5 } } ).pretty()• Between dates:db.system.profile.find({ts : {

$gt : new ISODate("2012-12-09T03:00:00Z") ,$lt : new ISODate("2012-12-09T03:40:00Z")

}}).pretty()

100


Explain> db.courses.find().explain();{ "cursor" : "BasicCursor",

"isMultiKey" : false,

"n" : 11, “nscannedObjects" : 11, "nscanned" : 11,"nscannedObjectsAllPlans" : 11, "nscannedAllPlans" : 11,

"scanAndOrder" : false, "indexOnly" : false,

"nYields" : 0,

"nChunkSkips" : 0,

"millis" : 0,

"indexBounds" : {},

"server" : "primary.domain.com:27017"

}

101


INDEXES

102


Index Management• Regular Index

• db.users.createIndex( { user_id: 1 } )

• db.users.ensureIndex( { user_id: 1 } )

• Multiple + DESC Index• db.users.ensureIndex( { user_id: 1, age: -1 } )

• Sub Document Index• db.users.ensureIndex( { address.zipcode: 1 } )

• Unique Index• db.users.ensureIndex( { address.zipcode: 1 } , { unique : true } )

• List Indexes• db.users.getIndexes()

• Drop Indexes• db.users.dropIndex(“indexName”)

103


Known Index Issues

• Bound filter should be the last (in the index as well).

• BitMap Indexes not really working

• You should design your indexes carefully

104


Dex: The Index Analyzer• Installation:

• sudo apt-get -y install python-pipsudo pip install dex

• Running:

• dex [mongodb_uri] (-f <logfile_path> | -p) [<options>]

• dex -w -p -n "testdb.*" mongodb://127.0.0.1/testdb -f /var/log/mongodb/mongod.log


mtools: Visualize and Analyze Logs


Capped Collections• Fixed size collections

• Circular buffers like

• High throughput operations

• Order guarantee

db.createCollection("mycoll", {capped: true, size:100000})db.cappedCollection.find().sort( { $natural: -1 } )

• Case studies:

• Logs

• Cache


TTL• Remove Old Data Automatically

• db.log_events.createIndex( { "createdAt": 1 }, { expireAfterSeconds: 3600 } )

• db.log_events.insert( {"expireAt": new Date('July 22, 2013 14:00:00'),"logEvent": 2,"logMessage": "Success!“

} )


ENVIRONMENT TUNING

109


TTL

• # For SSD only• blockdev --setra 16 /dev/sdb

• blockdev --setra 16 /dev/dm-2

• # For all cluser mongod & mongos• for i in /sys/kernel/mm/*transparent_hugepage/enabled; do echo never > $i; done

• for i in /sys/kernel/mm/*transparent_hugepage/defrag; do echo never > $i; done


STATS &SCHEMA DESIGN

111


Sparse Matrix? I don’t Think so

• mongostat

• > db.stats();

• > db.collectionname.stats();

• Fragmentation if storageSize/size > 2

• db.collectionanme.runCommand(“compact”)

• Padding (wrong design) if paddingFactor > 2

112


High AvailabilityGoing Real Time


(Do Not) Master/Slave


• In mongo.conf

• # Replication Options

• replSet=myReplSet

• > rs.initiate()

• > rs.conf()

• > rs.add(“host:port")

• rs.reconfig()

Replication Set

115


• rs.addArb(“host:port")

• Also:

• Low Priority

• Hidden

• (Weighted) Voting

Arbiter

116


Show Status: rs.status();• {"set" : “myReplSet", "date" : ISODate("2013-02-05T10:23:28Z"),

• "myState" : 1,

• "members" : [

• {

• "_id" : 0, "name" : "primary.example.com:27017",

• "health" : 1, "state" : 1,

• "stateStr" : "PRIMARY","uptime" : 164545,

• "optime" : Timestamp(1359901753000, 1),

• "optimeDate" : ISODate("2013-02-03T14:29:13Z"), "self" : true

• },

• {

• "_id" : 1, "name"


Replica Set Recovery

• Create a new mongod• Either install a plain vanilla

• Or duplicate existing mongod (better)

• Connect to the system• Use the previous machine IP

• Or change configuration to remove old and add new


Read Preferences• primary

• primaryPreferred

• secondary

• secondaryPreferred

• Nearest

mongo.setReadPreference(ReadPreference.nearest());


Write Concern: Unacknowledged


Write Concern: Acknowledged

mongo.setWriteConcern(WriteConcern.NORMAL);


Write Concern: Journaled


Write Concern: Replica Aacknowledged


Sharding and Scale out:Make a big Change

Map Reduce and Aggregation


Secondary Read Enabling


The Strategy : Sharding


MongoDB Implementation


Step 1: Create a Config ReplicaSet

• mkdir /data/configdb

• mongod --configsvr --dbpath /data/configdb --port 27019


Step 2: Install Mongos

• mongos --configdb config01:27019, config02:27019, config03:27019


Step 3: Add Shards

• Connect a mongos

• Add Shard

• sh.addShard( "rs1/mongodb0.example.net:27017" )

• sh.addShard( "mongodb0.example.net:27017" )


Step 4: Enable Sharding

• sh.enableSharding("<database>")


Step 5: Sharding Colleciton• sh.shardCollection("<database>.<collection>", shard-key-pattern)

• sh.shardCollection("records.people", { "zipcode": 1, "name": 1 } )

• Keys:

• High Cardinality to enable split

• Use common query field

• Use Compound indexes for sharding


BACKUP AND MONITORING

133


First Option – Single ServerLogical Backup Physical Backup

Method mongodump Point in time snapshot (using LVM tools) or disk image/copy(using AWS or Azure “external” tools)

Pros Low costs Low costs

Cons • Downtime: Long;• Duration: Long (slow backup since logical data needs to be

extracted);• Performance impact: High (slows the disks and may stuck the

machine on heavy used machines);• Data consistency: Intact;• Differential: Supported;• Sharding: Supported;

• Downtime: OS and/or infrastructure depended;• Duration: Short (faster backup since only data blocks are

copied);

• Performance impact: Unknown (depends on OS and/orinfrastructure);

• Data consistency: Unknown state;• Differential: Infrastructure depended;• Sharding: Unsupported;

134

Sharding: is a type of database partitioning that separates very large databasesthe into smaller, faster, more easily managed parts called data shards. The wordshard means a small part of a whole..


SECOND Option – REPLICA SET

Logical Backup Physical Backup

Method mongodump Stop slave and copy its disk

Pros • Downtime: None (backup is performed using Slave server –Master server is always up);

• Duration: Not significant (backup is performed using Slaveserver);

• Performance impact: None (backup is performed using Slaveserver – Master server is not impacted);

• Data consistency: Intact;• Differential: Supported;• Sharding: Supported;

• Downtime: None• Duration: Not significant

Cons Very high costs – requires two additional servers. A slaveserver of the same type and size as the master server; and asmall arbiter server (used as a secondary verification forMaster server availability tests and “voting”).

• Costs: Requires a dedicated server per replica set

135


THIRD OPTION - MongoDB MMS• Part of the MongoDB Enterprise Edition or as a Cloud Service• The Cloud Service offer

• $50/month/node• $2.5/GB/Month backup.• A valid go to market way of MongoDB

for upsale• MMS Features• Point in time recovery• Daily snapshots• Detailed monitoring• Alerts

136


How to Enable Incremental Backup

• In Backup• Use the --oplog flag when doing mongodump• Dump each hour the local.oplog collection

• In recovery• mongorestore --oplogReplay• applyOps to implement hourly dump

137


mongostat


mongotop


db.serverStatus()


db.stats() and db.collection.stats()


rs.status()


STORAGE ENGINES

143


MMAPv1


MongoDB 3.0 and WiredTiger• MongoDB version 3.0 supports new storage engine

(WiredTiger):• Disk Compression• Heavy write• Document level locking• File per collection

• Server wide selection: • config.yaml• launch w/ --storageEngine = wiredTiger

145


MongoDB Pluggable Architecture


Engines Comparison


YAML Based Configurationstorage:

dbPath: "/var/lib/mongodbwt"

directoryPerDB: true

engine: "wiredTiger"

wiredTiger:

engineConfig:

cacheSizeGB: 16

journalCompressor: zlib

directoryForIndexes: true

collectionConfig:

blockCompressor: zlib

indexConfig:

prefixCompression: true

systemLog:

destination: file

path: "/var/log/mongodb/mongod.log"

logAppend: true

timeStampFormat: iso8601-local

processManagement:

fork: true

pidFilePath: "/var/run/mongodb.pid"

#security:

# keyFile: "/etc/mongo.key"

# authorization: "enabled"

replication:

replSetName: "arp0"


SECURITY

149


Providing Permissions• use admin

db.createUser( { user: "siteUserAdmin", pwd: "password", roles: [ { role: "userAdminAnyDatabase", db: "admin" } ]

} )

• use recordsdb.createUser( {

user: "recordsUserAdmin", pwd: "password", roles: [ { role: "userAdmin", db: "records" } ]

} )


Roles

ReadreadWritedbAdmindbOwneruserAdminclusterAdmin, clusterManager, …backup, restorereadAnyDatabase, readWriteAnyDatabaseroot


Granular Actionsuse admindb.createRole(

role: "manageOpRole",privileges: [

{ resource: { cluster: true }, actions: [ "killop", "inprog" ] },{ resource: { db: "", collection: "" }, actions: [ "killCursors" ] }

],roles: []

})


Thank You !Moshe Kaplan

[email protected]

mongodb training for java software engineers

Technology