1404 app dev series - session 8 - monitoring & performance tuning
DESCRIPTION
TRANSCRIPT
![Page 1: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/1.jpg)
Application Development SeriesBack to BasicsMonitoring & Performance Tuning
Daniel Roberts@dmroberts
#MongoDBBasics
![Page 2: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/2.jpg)
2
• Recap from last session
• Monitoring Tools– MongoDB command line and shell
• Key Metrics in MongoDB
• Logs– Log levels– mtools
• Disk saturation
Agenda
![Page 3: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/3.jpg)
3
• Virtual Genius Bar
– Use the chat to post questions
– EMEA Solution Architecture / Support team are on hand
– Make use of them during the sessions!!!
Q & A
![Page 4: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/4.jpg)
Recap from last time….
![Page 5: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/5.jpg)
5
• mongodump & mongorestore
• File system copy
• Files system snapshot
• Don’t use mongoimport & mongoexport!
Backup Tools & Approaches
//mongodump Exampleserver> mongodump -h myhost -d cms -c articles
![Page 6: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/6.jpg)
6
• Creates a .bson file – (and a json metadata
file)– Over network or direct
from file system.
Mongodump
>mongodump –h myhost -d cms -c articlesconnected to: myhost 2014-04-16T12:54:56.758+0100 DATABASE: cms to dump/cms2014-04-16T12:54:56.759+0100 cms.articles to dump/cms/articles.bson2014-04-16T12:54:56.816+0100 7 documents2014-04-16T12:54:56.817+0100 Metadata for cms.articles to dump/cms/articles.metadata.json
![Page 7: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/7.jpg)
7
• Use secondary for backup– Or hidden secondary
• Fsync+Lock
• Balance off – In sharded cluster
Backup Architecture
mongodump
![Page 8: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/8.jpg)
Monitoring Tools
![Page 9: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/9.jpg)
9
• Key MongoDB tools– Mongostat– Mongo shell– Mongo Management Service mms
• Mtools– Logs
• OS– iostat
Tools
![Page 10: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/10.jpg)
10
MMS Architecture
![Page 11: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/11.jpg)
11
MMS Alerts
![Page 12: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/12.jpg)
12
Metrics Collection and Reporting
![Page 13: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/13.jpg)
13
Logs and Profile data
![Page 14: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/14.jpg)
14
• Access to key statistics– Connect to mongod or mongos for sharded cluster stats
– Shows, operations per second, memory usage, page faults, queues and more
Mongostat
>mongostat -h localhost --port 27017connected to: localhost:27017insert query update delete getmore command flushes mapped vsize res faults locked db idx miss % qr|qw ar|aw netIn netOut conn time *0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:24 *0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:25 *0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:26 *0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:27 *0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:28 *0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:29 *0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:30 *0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:31 *0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.1% 0 0|0 0|0 62b 5k 2 09:08:32 *0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:33
![Page 15: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/15.jpg)
15
• Commands for access metrics– db.serverStatus()– workingSet analyzer
Mongo shell
//to access a subset of serverStatus output specify the sub document>db.serverStatus().mem{
"bits" : 64,"resident" : 266,"virtual" : 5920,"supported" : true,"mapped" : 1712,"mappedWithJournal" : 3424
}
![Page 16: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/16.jpg)
16
• Estimates the ‘working set’ of the database– Number of ‘pagesInMemory’ accessed in RAM over
‘overSeconds’ number of seconds.– Default Page is 4k– overSecond – delta between oldest and newest
WorkingSet analyzer
>db.serverStatus({ workingSet : 1 }){
"note" : "thisIsAnEstimate","pagesInMemory" : 723,"computationTimeMicros" : 4601,"overSeconds" : 2311
}
overSeconds decreasing or small, workingSet could be larger than RAM
![Page 17: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/17.jpg)
MongoDB Metrics
![Page 18: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/18.jpg)
18
• Queued readers | writers
• Page faults
• OpCounters
• Background flush process
• Memory usage
• Lock %
• Btree misses
• connections
Key Metrics
Available from :
• MMS
• db.serverStatus()
• mongostat
![Page 19: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/19.jpg)
19
Queued Reader | Writers
• Number of operations waiting for:
• Read Lock
• Write Lock
![Page 20: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/20.jpg)
20
Page Faults
• How often we are going to disk to retrieve data?
• Can be a problem if disk IO is saturated
• Combine with iostat
![Page 21: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/21.jpg)
21
Background Flush Process
• Length of time to flush data back to disk
• Again can point to disk saturation.
![Page 22: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/22.jpg)
22
Memory
• Virtual
• Mapped
• Resident
![Page 23: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/23.jpg)
23
Lock %
• Percentage of time in the global lock
![Page 24: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/24.jpg)
24
btree
• Accesses
• Hits
• Misses
• Bad!
• Points to Indexes not in RAM
![Page 25: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/25.jpg)
25
connections
• Connections to the database
• Growing over time?
• Check pool logic in your application code.
![Page 26: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/26.jpg)
Logs
![Page 27: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/27.jpg)
27
• Increase the log level– See further insight to operation performance
– Index efficiency
– Document moves
– Operation time
Set log level
//Increase log level verbosity from the shell> db.adminCommand( { setParameter:1, logLevel:1 } ){ "was" : 0, "ok" : 1 }>
-v [ --verbose ] be more verbose (include multiple times for more verbosity e.g. -vvvvv)
![Page 28: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/28.jpg)
28
• Pinpoint problems – In this example we can see that there has been a document move.
Log output example
2014-05-02T13:55:02.047+0100 [conn7] update cms.articles query: { _id: ObjectId('532198379fb5ba99a6bd4063') } update: { $inc: { comment_count: 1 }, $push: { comments: { $each: [ { date: new Date(1399035302013), text: "Data locality provides an amazing performance boost over relational" } ], $slice: -10, $sort: { date: 1 } } } } nscanned:1 nscannedObjects:1 nmoved:1 nMatched:1 nModified:1 keyUpdates:0 numYields:0 locks(micros) w:33529 33ms
![Page 29: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/29.jpg)
29
• Log analysis (example syntax)– Show me queries that took more than 1000 ms from 6
am to 6 pm:
• Now, graph those queries:
mtools
prompt> mlogfilter mongodb.log --from 06:00 --to 18:00 --slow 1000 > mongodb-filtered.log
prompt> mplotqueries --logscale mongodb-filtered.log
![Page 30: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/30.jpg)
30
mtools output
![Page 31: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/31.jpg)
Disk saturation
![Page 32: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/32.jpg)
32
• iostat– Disk IO usage
– Use to review % utilisation of disk– High %
• If it’s sustained, you may need to increase disk IO.– Sharding– RAID 1+0– Partitioning on different disk– Provisioned IOPS
OS Tools and Paramenters
prompt> iostat –xmt 1
![Page 33: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/33.jpg)
Summary
![Page 34: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/34.jpg)
34
• Monitor regularly– Set alerts for changes
• Quick check with command line and shell
• View trends with MMS graphs over time
• High levels of utilisation– Increase RAM, disk IO– Scale out– (Having first checked Indexes etc!)
• Database profiler (covered in Indexing session)
Summary
![Page 35: 1404 app dev series - session 8 - monitoring & performance tuning](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b6ea804a7959aa218b465d/html5/thumbnails/35.jpg)