Performance Tuning CouchbaseKirk Kirkconnell | Sr. Solutions Engineer | Couchbase
@cbcollect_info
©2014 Couchbase, Inc. 2
A Properly Sized ClusterTuning CouchbaseTuning the Operating SystemAdditional Resources
Outline
A Properly Sized Cluster
Good Performance
Starts with Sizing
©2014 Couchbase, Inc. 4
Size server resources for ~3x your total data set on average. Take into account:
• Replicas• Compaction• Views and Indexes• XDCR• Maintenance
Number of Nodes• Distributes data and utilizes more resources across additional servers• Scale out is the preferred strategy
General Sizing
©2014 Couchbase, Inc. 5
Size IO for your workload• Heavy writes, view queries = lean towards SSDs and eSSDs • Heavy reads, stable working set = HDD is good, SSD is better• Consistent performance is most important• Ensure overhead for maintenance tasks
RAM Sizing• Have sufficient RAM for your working set• Have enough RAM for your OS• XDCR needs more RAM, but depends on number of target clusters• If you use indexes/views, leave enough for OS page cache• Factor in meta data
General Sizing
©2014 Couchbase, Inc. 6
Disk Considerations• Spinning disks• SSD• FusionIO
How Many Cores?• Have at least 4 cores for Couchbase• More buckets, usually means more Cores• One more core per design document• Two more cores per XDCR stream
General RAM Sizing
©2014 Couchbase, Inc. 7
Bandwidth• Throughput• Rebalance• Intra-Cluster Replication• Inter-Cluster Replication (XDCR)• Backups
Latency• 1GbE• 10GbE• Infiniband
A Properly Sized Cluster
©2014 Couchbase, Inc. 8
Replicas• The number of replicas impacts overall performance• For each replica there is more network, disk, RAM, etc. used.• If you need more replicas, you must size for them.
A Properly Sized Cluster
©2014 Couchbase, Inc. 9
Do not over-subscribe resources, specifically:• Disk I/O• Network• CPU
Avoid VM migrations if you can (e.g. vMotion)• Manually manage migrations
Monitor the hypervisor as well as the guest OS if you canUse pass-through devices for disk to guarantee resources
Virtualization Considerations
©2014 Couchbase, Inc. 10
• Think of sizing as an ongoing activity• Monitor your cluster’s performance and incorporate sizing into your
continuous improvement process
Sizing Does Not End
Tuning Couchbase
Some of the Internals
©2014 Couchbase, Inc. 12
• Some like to say Couchbase is schemaless, but more so it is flexible• Prioritize your schema around the “hot path” of the app
• How does your app access data?• Which parts have to perform the fastest?• Replicating around large objects takes more resources obviously• This also contributes to scalability
• Get the document by key as much as possible for best performance• Key naming – you have 256 characters, use them wisely
• For best performance, minimize use of views• If you are using indexes/views, group similar documents in the same bucket
to avoid indexing too much unrelated data
Schema Design Matters
©2014 Couchbase, Inc. 13
In 2.x, tune the number of Read/Write threads per bucket
• Best to do when you create the bucket
In 2.x, tune working balance of reader/writer threads• Default is weighted heavily towards reader threads• The other options weigh more towards writing or 50/50 mix
Tuning Thread and Resource Usage
©2014 Couchbase, Inc. 14
In 3.x, GTP makes manual thread tuning is no longer necessary
Bucket Priorities• Balances I/O priorities of buckets• By default every bucket is set to low• Only kicks in when one or more bucket are set differently• So if all buckets are set to low or all set to high, all get equal resources
Couchbase 3.0
©2014 Couchbase, Inc. 15
Three area to look at• Change number of vBuckets moved concurrently• Turn off index aware rebalancing• Disable view compaction during rebalance
Change before maintenance window. When complete, change settings back.Be conservative or you could end up drowning out your application
Have enough RAM to avoid disk backfills, otherwise you have lots of disk I/O
Only do this if you have the cluster resources and test these first.
Tuning Rebalances
©2014 Couchbase, Inc. 16
Auto-Compaction
©2014 Couchbase, Inc. 17
Look to raise the thresholds to 60% if you need it• Compaction runs less often, thus lowering I/O utilization• Increases disk space requirements. Depending on your workload this
could be dramatically more
• If possible, schedule this for off-peak hours• Also increases disk space requirements
Auto-Compaction
©2014 Couchbase, Inc. 18
• Set the maximum parallel indexers• maxParallelIndexers
• Configure automated index updates• updateInterval• updateMinChanges• replicaUpdateMinChanges
• Page Cache buffers indexes• Set the maximum parallel indexers
• maxParallelIndexers
Views
©2014 Couchbase, Inc. 19
XDCR
Tuning the Operating System
Some advanced stuff
©2014 Couchbase, Inc. 21
• OS Page Cache• Best Practice = 20%. More if you have large sized indexes/views
• Deprioritize swap• vm.swappiness = 0
• Turn off Transparent Huge Pages• Limit page cache dirty bytes
• vm.dirty_bytes• vm.dirty_background_bytes
• Control the rate page cache is flushed• vm.dirty_expire_centisecs• vm.dirty_writeback_centisecs
Memory
©2014 Couchbase, Inc. 22
• File Systems• EXT4• XFS
• Mount Options• EXT4: noatime,barrier=0,data=writeback• XFS: noatime,nobarrier,logbufs=8,logbsize=256k,allocsize=2M
• IO Scheduler• Deadline scheduler• Queue depth increase from 128 to 1024
Disk
©2014 Couchbase, Inc. 23
• Increase backlogs• net.core.netdev_max_backlog• net.core.somaxconn• net.ipv4.tcp_max_syn_backlog• net.ipv4.tcp_fin_backlog
• Increase buffer limits• net.core.rmem_max• net.core.wmem_max• net.ipv4.tcp_mem• net.ipv4.tcp_rmem• net.ipv4.tcp_wmem
Network Resilience
©2014 Couchbase, Inc. 24
• Reduce TCP overhead• net.ipv4.tcp_sack• net.ipv4.tcp_fack
• Reduce connection overheat• net.ipv4.tcp_fin_timeout = decrease• net.ipv4.tcp_tw_reuse = enable
• Find failed connections faster• net.ipv4.tcp_keepalive_intvl
• Enable auto-tuning• net.ipv4.tcp_moderate_rcvbuf• net.ipv4.tcp_window_scaling
Network Efficiency
Evaluation Success
Load Testing Tools
©2014 Couchbase, Inc. 26
• Pillowfight – based on libcouchbase (C SDK)• Roadrunner – based on Java SDK• Cbworkloadgen – Python + libcouchbase
Load Testing Tools
©2014 Couchbase, Inc. 27
Again, advanced tuning only helps if your Couchbase cluster is properly sized!
http://docs.couchbase.com/couchbase-manual-2.5/cb-admin/#sizing-guidelines
http://blog.couchbase.com/how-many-nodes-part-1-introduction-sizing-couchbase-server-20-cluster
Resources
©2014 Couchbase, Inc. 28
With great power comes great responsibility.
Performance tuning is often a trade-off between safety and performance.
Understand and test before you deploy.
Advice #2