Download - MongoDB Capacity Planning
![Page 1: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/1.jpg)
![Page 3: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/3.jpg)
Capacity Planning
• What is Capacity Planning ?
• Why is it important
• Which resources are affected?
• How to do it?
![Page 4: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/4.jpg)
https://tingbudongchine.files.wordpress.com/2012/08/lemonde1.jpeg
What is Capacity Planning?
![Page 5: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/5.jpg)
Fine Art of …
Requirements
![Page 6: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/6.jpg)
Fine Art of …
Requirements
Resources
![Page 7: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/7.jpg)
Preparing for Launch
• Developers are about to finish final Sprint
• Code is good (so they say )
• You feeling confortable to launch soon
• How to deploy?
![Page 8: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/8.jpg)
Requirements
• Availability
– Uptime requirements: RPO and RTO
• Throughput
– Average read/writes/users
– Peek throughput
– Operations per second ? per day? per month?
• Responsiveness
– What's the acceptable latency?
• Higher during peek time?
RTO=recovery time objective RPO=recovery point objective
![Page 9: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/9.jpg)
Resources
![Page 10: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/10.jpg)
Resources
• CPU
• Storage
• Memory
• Network
![Page 11: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/11.jpg)
Requirements vs Resources
Throughput
Availability
Responsiveness
![Page 12: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/12.jpg)
Resource Usage
• Storage
• IOPS
• Size
• Data & Loading
Patterns
• CPU
• Speed
• Cores
• Memory
• Working Set
• Network
• Latency
• Throughput
![Page 13: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/13.jpg)
Why is that Important?
![Page 14: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/14.jpg)
Why
• Once we launch, we don't want to have avoidable down
time due to poorly selected HW
• As our success grows we want to stay in front of the
demand curve
• We want to meet business and users expectations
• We want to keep our jobs!
• Don't be the "goat"
![Page 15: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/15.jpg)
Under allocation
![Page 16: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/16.jpg)
Over Capacity
![Page 17: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/17.jpg)
Over spending
![Page 18: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/18.jpg)
Important Aspects
• Capacity
– Under
– Over
– Just Right?
• Prediction Models
– User/Load
– OPS/Request
– System Behavior (stress testing anyone?)
• Change Velocity
– Data / Resource-Allocation / Provisioning
– Minimum Viable Product?
– Future Releases / Roadmap
![Page 19: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/19.jpg)
Important Aspects
• When?
– Not too early
– Before is too late!
– Iterative Process
Launch Version 2
![Page 20: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/20.jpg)
Important Aspects
• When?
– Not too early
– Before is too late!
– Iterative Process
Launch Version 2
![Page 21: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/21.jpg)
Important Aspects
• When?
– Not too early
– Before is too late!
– Iterative Process
Launch Version 2
![Page 22: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/22.jpg)
http://www.mandywalker.com.au/wp-content/uploads/2013/07/Wall-with-Tools.jpg
Which resources are affected?
![Page 23: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/23.jpg)
CPU
• Non-indexed Data
• Sorting
• Aggregation
– Map/Reduce
– Aggregation Framework
• Data
– Fields
– Nesting
– Arrays/Embedded-Docs
![Page 24: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/24.jpg)
Network
• Latency
– WriteConcern
– ReadPreference
– Batching
• Throughput
– Update/Write Patterns
– Reads/Queries
![Page 25: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/25.jpg)
Network
• Latency
– W:?
– Nearst
– Bulk Write Operations
• Throughput
– Use $set operator
– Filtering fields on queries
![Page 26: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/26.jpg)
Storage
• Active
• Archival
• Loading Patterns
• Integration (BI/DW)
![Page 27: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/27.jpg)
Storage Capability
Type IOPS
7200 rpm SATA ~ 75 – 100
15000 rpm SAS ~ 175 – 210
http://en.wikipedia.org/wiki/IOPS
![Page 28: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/28.jpg)
Storage Capability
Type IOPS
7200 rpm SATA ~ 75 – 100
15000 rpm SAS ~ 175 – 210
SSD Intel X25-E (SLC) ~ 5000
SSD Intel X25-M G2 (MLC) ~ 8000
http://en.wikipedia.org/wiki/IOPS
![Page 29: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/29.jpg)
Storage Capability
Type IOPS
7200 rpm SATA ~ 75 – 100
15000 rpm SAS ~ 175 – 210
SSD Intel X25-E (SLC) ~ 5000
SSD Intel X25-M G2 (MLC) ~ 8000
Amazon EBS ~ 100
Amazon EBS Provisioned Up to 2000
Amazon EBS Provisioned IOPS (SSD) ~3000
http://en.wikipedia.org/wiki/IOPS
![Page 30: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/30.jpg)
Storage Capability
Type IOPS
7200 rpm SATA ~ 75 – 100
15000 rpm SAS ~ 175 – 210
SSD Intel X25-E (SLC) ~ 5000
SSD Intel X25-M G2 (MLC) ~ 8000
Amazon EBS ~ 100
Amazon EBS Provisioned Up to 2000
Amazon EBS Provisioned IOPS (SSD) ~3000
FusionIO ~135 000
Violin Memory 6000 ~ 1 000 000
http://en.wikipedia.org/wiki/IOPS
Higher IOPS higher the Cost!!!
![Page 31: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/31.jpg)
Storage Considerations
• Work out how much data you need to write per unit of
time!
• Databases will use storage to persist data
– More data = Bigger indexes = More Storage
• MongoDB Stores Information into Documents
• BSON Format
– http://bsonspec.org/
![Page 32: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/32.jpg)
Memory
• Working Set
– Active Data in Memory
– Measured Over Periods
• And other operations
– Sorting
– Aggregation
– Connections
• MongoDB Storage Engine
– VMMAP
– Memory Mapped Files
![Page 33: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/33.jpg)
Memory Mapped Files
![Page 34: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/34.jpg)
Memory Usage
• Data & Indexes memory mapped into virtual address
space
• Data access is paged into RAM
• OS evicts using LRU
• More frequently used pages stay in RAM
![Page 35: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/35.jpg)
http://blogdailyherald.com/wp-content/uploads/2013/05/3879-animated_gif-chuck_norris-dodgeball-thumbs_up.gif
How to do it!
![Page 36: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/36.jpg)
Basic Rules
• Determine your Working Set
• Use good Measuring and Monitoring practices
• Plan ahead but be flexible!
• Iterate
– Review Requirements
– Review Capacity
![Page 37: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/37.jpg)
Working Set
Number of Active Users on
the system at any one time
Number of distinct pages
accessed per second
![Page 38: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/38.jpg)
Working Set
![Page 39: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/39.jpg)
Working Set
4 distinct pages per second
RAM
Disk
![Page 40: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/40.jpg)
Working Set
4 distinct pages per second
RAM
Disk
Worst case 4 disk accesses
![Page 41: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/41.jpg)
Working Set
6 distinct pages per second
RAM
Disk
![Page 42: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/42.jpg)
Working Set
6 distinct pages per second
Disk
![Page 43: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/43.jpg)
Working Set
6 distinct pages per second
Worst case disk access on every op
![Page 44: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/44.jpg)
Memory & Storage
MOPs
PFs
![Page 45: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/45.jpg)
Working Set
• Capacity sizing to hold working set + indexes
• Allow room to grow
• If working set is larger than RAM and you can't
reasonably add more resources
– Shard!
– Lots of little instances vs few big instances
• Think about architecture
– Local disk vs central storage
– How many copies of data do I need for availability
reasons
![Page 46: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/46.jpg)
Measuring & Monitoring
• What to measure
– IOPS
– Page Faults
– Resident Memory (Working Set)
– Connections
– Lock %
• How to measure and monitor
– iostat
– vmstat
– mongostat
– mongopref
– MMS
![Page 47: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/47.jpg)
iostat
![Page 48: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/48.jpg)
vmstat
![Page 49: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/49.jpg)
mongostat
• Quick overview of the status of mongodb nodes
![Page 50: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/50.jpg)
mongoperf
• Utility to check disk I/O performance
![Page 51: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/51.jpg)
mongotop
• Utility to track the time spent reading and writing per
namespace
![Page 52: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/52.jpg)
MMS
![Page 53: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/53.jpg)
MMS
• Comprehensive Tool– Monitoring
– Backup
– Deployment
![Page 54: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/54.jpg)
MMS
• Comprehensive Tool– Monitoring
– Backup
– Deployment
![Page 55: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/55.jpg)
Monitoring
• Key Metrics
– Storage
– Memory
– CPU
– Network
– Application Metrics
![Page 56: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/56.jpg)
Models
• Load / Users
– Response Time / TTFB
• System Performance
– Peak Usage
– Min Usage
![Page 57: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/57.jpg)
Velocity of Change
• Limitations -> takes time
– Data Movement
– Allocation / Provisioning (servers/mem/disk)
• Improvement
– Limit Size of Change
– Increase Frequency
– MEASURE its effect
– Practice
![Page 58: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/58.jpg)
http://www.humanandnatural.com/data/media/178/badan_jaran_desert_oasis_china.jpg
Long story short …
![Page 59: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/59.jpg)
Capacity Planning is …
• Needed
– Involves resource allocation
– Hardware specification and sizing
– Cost!
• Vital
– Translate Requirements and Expectations into Experience and Functionality
• And meeting those
• Requires understanding your application
– Measuring resource needs
– Monitoring
– Iterating
– Repeating process
![Page 60: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/60.jpg)
For More Information
Resource Location
Case Studies mongodb.com/customers
Presentations mongodb.com/presentations
Free Online Training education.mongodb.com
Webinars and Events mongodb.com/events
Documentation docs.mongodb.org
MongoDB Downloads mongodb.com/download
Additional Info [email protected]
![Page 61: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/61.jpg)
http://cl.jroo.me/z3/v/D/C/e/a.baa-Too-many-bicycles-on-the-van.jpg
Questions?
@nleite
![Page 62: MongoDB Capacity Planning](https://reader034.vdocuments.us/reader034/viewer/2022042716/55a6863b1a28abed088b4674/html5/thumbnails/62.jpg)