Enteros, Inc.

MongoDB Performance Tuning

Load Testing

NoSQL Now! 2013 Conference San Jose

ron@enteros.com Ron Warshawsky

August 23, 2013

2

MongoDB Overview

Agenda

Overview performance tuning

Overview load testing

3

MongoDB Areas Of Performance Tuning

OS tuning Storage tuning

Database tuning Load Testing

4

MongoDB Areas Of Performance Tuning

• OS Tuning • Follow ulimits per mongo manual

• http://docs.mongodb.org/manual/reference/ulimit/

• Follow production notes per mongo manual • http://docs.mongodb.org/manual/administration/production-notes/ • No atime on fs • No hugepages • Use NTT to sync time between the nodes

• Try “tuned”

• Best when used in staging/load testing environment under

realistic load

• CPU, Network and Storage tuning

5

MongoDB Areas Of Performance Tuning

• Storage Tuning • RAID 10

• Ext4 or XFS

• Delay sharding with better I/O

• SSD + FlashCache – shutterfly tested

• https://github.com/facebook/flashcache/

• Provisioned IOPS on AWS

6

MongoDB Areas Of Performance Tuning

• Storage Tuning • Ext4 + external journal on SSD – good I/O results

• Make sure to test • journal_async_commit is used • http://www.raid6.com.au/posts/fs_ext4_external_journal/

7

MongoDB Areas Of Performance Tuning

Softlayer study/ benchmarking “MongoDB Performance Analysis: Bare Metal v. Virtual” http://blog.softlayer.com/2012/mongodb-performance-analysis-bare-metal-v-

virtual/

Highlights:

• When a working data set is smaller than available memory, query performance increases.

• The number of clients performing queries has an impact on query performance because more data is being actively cached at a rapid rate.

• The addition of a separate Journal Mount volume significantly improves performance. Because the Small (SM) engineered server does not include a secondary mount for Journals, whenever MongoDB began to journal, the disk I/O associated with journaling was disruptive to the query and update operations performed on the Data Mount.

• The best deployments in terms of operations per second, stability and control were the configurations with a RAID10 SSD Data Mount and a RAID1 SSD Journal Mount. These configurations are available in both Medium and Large offerings, and I’d highly recommend them.

8

MongoDB Areas Of Performance Tuning

OS Performance monitoring and analysis tools:

• A) Linux utilities: iostat, vmstat, mpstat, sar, free -tm

• B) Open source monitoring / performance analysis systems: Nagios, Cacti

• http://tag1consulting.com/blog/mongodb-cacti-graphs • https://github.com/mzupan/nagios-plugin-mongodb C) Oracle OS Watcher – Free tool, performance collection and chart

generator, easy to operate, very reliable, covers CPU, processes, top processes, I/O, network, etc

OSWatcher Black Box Analyzer User Guide (Doc ID 461053.1)

9

MongoDB Areas Of Performance Tuning

Database Tuning

• Database configuration • Preallocate space (noprealloc = false)

• journalCommitInterval – range of 2 to 300, default 100. Can be

adjusted to for better performance

• Option to disable services: 1. HTTP Interface using nohttpinterface = true. 2. Scripting Engine using noscripting = true (server-side java

script). 3. REST service using rest = false.

10

MongoDB Areas Of Performance Tuning

Database Tuning

• Database monitoring tools • Mongostat – based on “eBay” “freecon” utility.

• Use “–discover” option for members of replicaSet / shared

cluster

• Displays critical performance metrics: • Flushes/Faults/locked db/idx miss/qr/qw

• Collect 24x7 and save for analysis

11

MongoDB Areas Of Performance Tuning

Database Tuning

• Database monitoring tools • Mongotop

• Per collection I/O time tracking

• Per database lock tracking (--lock option)

12

MongoDB Areas Of Performance Tuning

Database network traffic analysis

• Database network monitoring tools • Mongosniff – sniffing MongoDb requests

• Can capture invalid BSON requests from a network (--objcheck option)

• Wireshark – support for MongoDB protocol (not BSON), great options for network performance and latency analysis

13

MongoDB Areas Of Performance Tuning

Database Tuning

• Collections / Fragmentation

Collections in a MongoDB database can become fragmented. This can be a particularly serious problem if data usage patterns are relatively unstructured. In the long run, this will result in databases taking up more space on disk and in RAM to hold the same amount of data, it will make many database operations noticeably slower, and it can reduce overall query capacity significantly.

compact collection: db.runCommand ( { compact: '<collection>', paddingBytes: 100 } ) Automate compaction: http://blog.parse.com/2013/03/26/always-be-compacting/ ./mongo_compact.rb -d userdata1,userdata2,userdata3

14

MongoDB Areas Of Performance Tuning

Database Tuning

Profiler

db.setProfilingLevel(level, slowms) db.setProfilingLevel(1) – collects slow requests Output: system.profile.op – operation system.profile.ns – namespace system.profile.ntoreturn – documents returned system.profile.nscanned – documents scanned … http://docs.mongodb.org/manual/reference/database-profiler/ Profiler deep dive https://speakerdeck.com/kgorman/mongodb-profiler-deep-dive

15

MongoDB Areas Of Performance Tuning

Database Tuning

• Indexes

• Explain db.collection.find({…}).explain() Watch for scans

• Indexes + Profiler = DEX - suggest what indexesare needed

https://github.com/jwilder/mongodb-tools http://architects.dzone.com/articles/mongodb-performance-tuning-

dex

16

MongoDB Areas Of Performance Tuning

Database Monitoring and performance analysis UI

a) MongoDB monitoring service (MMS)

b) Open Source monitoring software http://docs.mongodb.org/manual/administration/monitoring/

c) Enteros High Load Capture for MongoDB

17

MongoDB Areas Of Performance Tuning

• Replication / Replica set tuning Watch for replication lag Problem: a) Weak secondary b) Write bursts c) Index build d) Secondary locked for backup e) Secondary offline Measure lag: db.printSlaveReplicationInfo() db.printReplicationInfo() Get time difference MMS provides replication lag chart and alerts

18

MongoDB Areas Of Performance Tuning

• Sharding Tuning • Shard size

• Migration of chunks

• Failed migration of chunks

19

MongoDB Load Testing

Load testing

1) Why load testing?

• Validate upgrades

• Validate fixes

• Validate hardware and platform changes (to / from cloud, etc)

• Validate multiples of production load

20

MongoDB Load Testing

2) Options for Load testing

A) Benchmark Load Testing • Mongo-Perf - Create standard TPCC-like load on MongoDB • https://github.com/mongodb/mongo-perf

B) Disk I/O load testing

• Mongoperf

• Iozone http://www.iozone.org/

21

MongoDB Load Testing

C) Real traffic Load Testing

Parse PCAP files and generate production-like load http://docs.mongodb.org/meta-driver/latest/legacy/mongodb-wire-

protocol/

Create custom tool

Enteros Load2Test (pre-GA)

Understand business requirements and it’s impact on production system design

Design for performance, scalability and availability

Implement monitoring and performance analysis across layers

Implement reliable production-like load testing

MongoDB Summary

Enteros, Inc http://www.enteros.com/

Enteros is an innovative software company specializing in Performance Management and Load Testing Software for Production Databases - RDBMS and NOSQL/Big Data Enteros solutions enable IT professionals to identify and remediate performance problems in business-critical databases with unprecedented speed, accuracy and scope. Ron Warshawsky; ron@enteros.com 408-207-8408

MongoDB Summary