managing a mongodb deployment

MANAGING A MONGODB DEPLOYMENT

Tony Tam @fehguy

AREAS OF DISCUSSION

Many drivers to go non-relationalMaybe you already are?SOA?

“Yes we should!” Now what? Implementation details, best practices

Depends on what system you chooseOperational Architecture

Making your deployment “maintainable”Maximizing performance

Software + Hardware tuning

WORDNIK SAYS YES TO MONGODB

And we’re alive to talk about itOur stats

14 M REST API calls yesterday7 ms avg. response3 TB of MongoDB dataMostly all “un-cached” responses

SOFTWARE ARCHITECTURE

Java/Scala/Jetty application serversShare (almost) nothingAll roads in must speak REST

DEPLOYMENT ARCHITECTURE

Wordnik uses physical servers

OPERATIONAL CONSTRAINTS

Fundamental question to answerCan we support this?

You need to considerIf you want it to scaleIf you have a SLA to supportIf you carry the pager

Also important ifYou have customersYou like your job

OPERATIONAL CONSTRAINTS

Core items for WordnikRedundancy with master/slave

Multi-datacenter deployment possible?Design for read-only mode

Control at the driver levelBlocking

When is it OK?Steps around with MongoDB

ConsistencyWhen does it matter?

Scale-outWhat limits our speed/size?

Other?Some features => opportunity

REDUNDANCY

MongoDB supports MS and Replica SetsRS is like MS with auto-failoverDriver intelligenceStill one master, elected by peers

Master

Slave

Slave

Slave

Master

Slave

Slave

Master

Slave

Slave

Slave

REDUNDANCY

Master/Slave fine for us but…Not fine for multi operational datacenters

resync?Your data drives the appropriate solutionCustom tools to support thishttps://github.com/wordnik/wordnik-oss

SCP

Primary Datacenter

Master

Incremental Backup

Files

Hot Datacenter

Master

Replay Util

REDUNDANCY

Multi-datacenter deploymentSome data need not be consistent

Some doesReplication tricks for master-master

Master 1 Master 2

db.documentsdocuments.src == 1

db.documentsdocuments.src == 2

CONTROLLING BLOCKING OPERATIONS

Important to know what blocksSome situations can create global locks

Know what happens with:Adding dataDeleting dataModifying data

…Then toss in 100 reads/secOnly access your DB “the right way”

Enforce by good software designThe right way == “right way for your db”

CONTROLLING BLOCKING OPERATIONS

Wordnik SolutionMongoDB access via service layer

Only allow queries you want runAll others => non-production server,

HadoopSmart driver manager

Slave bias $inc:{page_views:1} => $set{page_views:10038}

Read-only modeHot-remove server from pool

HANDLING EVENTUAL CONSISTENCY

EC makes things fastMay create timing issuesSometimes need current data

val db:DB = getReadWriteConnection()

Avoid this a much as possibleBut if you must…

Write then call “getLastError”Forces commit to DB

RS allows “replicate to n nodes”Forces commit to master, n slaves

SCALE-OUT

What are you scaling?Size of dataSize of indexesOperations

Add slave serversWon’t help if size > index/server capacity

Sharding if done rightAutoshard solutionRoll your own

SCALE-OUT

Selecting a shard keyCan be trickyDepends on what you need

KEEPING IT RUNNING

BackupsExport with “mongodump”No incremental backup solution included

Wordnik tools https://github.com/wordnik/wordnik-oss

WORDNIK OSS TOOLS

Take our tools-They work!!!SnapshotUtil

Selectively snapshot in BSON Index info too!

IncrementalBackupUtilTail the oplog, stream to disk

Only the collections you want! Compress & rotate

RestoreUtilRecover your snapshots

Apply indexes yourself

ReplayUtilApply your Incremental backups

WHAT IF SCENARIOS

One collection gets corrupt?Restore itApply all operations to it

“My top developer dropped a collection!”Restore just that oneApply operations to it until that POT

“We got hacked!”Restore it allApply operations until that POT

WRAP UP: WE LIKE MONGODB

See more @ http://developer.wordnik.com

Questions?