2010 mongo berlin-shardinginternals (1)
TRANSCRIPT
![Page 1: 2010 mongo berlin-shardinginternals (1)](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f73a6b4c905bb178b533b/html5/thumbnails/1.jpg)
Eliot Horowitz@eliothorowitzMongoBerlin
October 4, 2010
Sharding Internals
![Page 2: 2010 mongo berlin-shardinginternals (1)](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f73a6b4c905bb178b533b/html5/thumbnails/2.jpg)
MongoDB Sharding
•Scale horizontally for data size, index size, write and consistent read scaling
•Distribute databases, collections or a objects in a collection
•Auto-balancing, migrations, management happen with no down time
![Page 3: 2010 mongo berlin-shardinginternals (1)](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f73a6b4c905bb178b533b/html5/thumbnails/3.jpg)
•Choose how you partition data
•Can convert from single master to sharded system with no downtime
•Same features as non-sharding single master
•Fully consistent
![Page 4: 2010 mongo berlin-shardinginternals (1)](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f73a6b4c905bb178b533b/html5/thumbnails/4.jpg)
Range Based
•collection is broken into chunks by range
•chunks default to 200mb or 100,000 objects
![Page 5: 2010 mongo berlin-shardinginternals (1)](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f73a6b4c905bb178b533b/html5/thumbnails/5.jpg)
User profiles
•Partition by user_id
•Secondary indexes on location, dates, etc...
•Reads/writes know which shard to hit
![Page 6: 2010 mongo berlin-shardinginternals (1)](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f73a6b4c905bb178b533b/html5/thumbnails/6.jpg)
User Activity Stream
•Shard by user_id
•Loading a user’s stream hits a single shard
•Writes are distributed across all shards
•Can index on activity for deleting
![Page 7: 2010 mongo berlin-shardinginternals (1)](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f73a6b4c905bb178b533b/html5/thumbnails/7.jpg)
Photos
•Can shard by photo_id for best read/write distribution
•Secondary index on tags, date
![Page 8: 2010 mongo berlin-shardinginternals (1)](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f73a6b4c905bb178b533b/html5/thumbnails/8.jpg)
Logging
•date
•machine, date
•logger name
Possible Shard Keys
![Page 9: 2010 mongo berlin-shardinginternals (1)](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f73a6b4c905bb178b533b/html5/thumbnails/9.jpg)
Architecture
client
mongos ...mongos
mongodmongod
mongodddd
mongod
mongod
mongod...
Shards
mongod
mongod
mongod
ConfigServers
![Page 10: 2010 mongo berlin-shardinginternals (1)](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f73a6b4c905bb178b533b/html5/thumbnails/10.jpg)
Config Servers
•3 of them
•changes are made with 2 phase commit
•if any are down, meta data goes read only
•system is online as long as 1/3 is up
![Page 11: 2010 mongo berlin-shardinginternals (1)](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f73a6b4c905bb178b533b/html5/thumbnails/11.jpg)
Shards
•Can be master, master/slave or replica sets
•Replica sets gives sharding + full auto-failover
•Regular mongod processes
![Page 12: 2010 mongo berlin-shardinginternals (1)](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f73a6b4c905bb178b533b/html5/thumbnails/12.jpg)
mongos
•Sharding Router
•Acts just like a mongod to clients
•Can have 1 or as many as you want
•Can run on appserver so no extra network traffic
![Page 13: 2010 mongo berlin-shardinginternals (1)](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f73a6b4c905bb178b533b/html5/thumbnails/13.jpg)
Writes
•Inserts : require shard key, routed
•Removes: routed and/or scattered
•Updates: routed or scattered
![Page 14: 2010 mongo berlin-shardinginternals (1)](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f73a6b4c905bb178b533b/html5/thumbnails/14.jpg)
Queries
•By shard key: routed
•sorted by shard key: routed in order
•by non shard key: scatter gather
•sorted by non shard key: distributed merge sort
![Page 15: 2010 mongo berlin-shardinginternals (1)](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f73a6b4c905bb178b533b/html5/thumbnails/15.jpg)
Operations
•split: breaking a chunk into 2
•migrate: move a chunk from 1 shard to another
•balancing: moving chunks automatically to keep system in balance
![Page 16: 2010 mongo berlin-shardinginternals (1)](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f73a6b4c905bb178b533b/html5/thumbnails/16.jpg)
Setting it Up•Start servers
•add shards: db.runCommand( { addshard : "10.1.1.5" } )
•turn on partitioning: db.runCommand( { enablesharding : "test" }
•shard a collection: db.runCommand( { shardcollection : "test.data" , key : { num : 1 } } )
![Page 17: 2010 mongo berlin-shardinginternals (1)](https://reader033.vdocuments.us/reader033/viewer/2022052822/554f73a6b4c905bb178b533b/html5/thumbnails/17.jpg)
Download MongoDBhttp://www.mongodb.org
and let us know what you think@eliothorowitz @mongodb
10gen is hiring!http://www.10gen.com/jobs