getting started with riak in the cloud
DESCRIPTION
Given at RICON 2012TRANSCRIPT
Getting started with Riak in the Cloud
I work with Code!
I work with Data!
Michael Brodhead Ines Sombra@RandomMood@mojinations
Agenda
Application layer
Database layer
A tale of 2 apps
•One brand new (no code written)
•One in production running MySQL
MultipleRegions
MultipleClouds
11
The Librariesriak-client
• simple Ruby access to Riak's HTTP & protocol buffer APIs
The Libraries
Ripple
• Higher level, ORM interface
• ActiveModel based buffer APIs
Simplicity
The Librariesdm-riak
• Already using DataMapper
• Hoping for trivial switchover
The LibrariesRiak Shim
• database.yml
• Bucket naming
• Hides JSON
• 2i
Challenges
•Unit tests
•Data migration
•Deployment
Testing
Testing
Deployment
Use your app logic
How do we get here?
... but we have no hardware
“We need a Riak cluster”
(@mojinations circa 3 months ago)
Let’s do it!
Why Riak Product?
• Linearly Scalable
• Fault-tolerant
•Ops friendly
• Product lacks HA options
Riak @ Engine Yard• We’ll use it first
• No custom Chef
• Provisioning, configuration, monitoring, & backups
Your Choices
• Number of nodes
• Instance type
• Data location
• Backend type
You are trying to create a Riak cluster. Do you need any help?
AWS OpsThe Riak on AWS slide
Instances & I/O
• Riak’s I/O patterns
• Small blobs from many places on disk
• EBS best at bulk reads & writes
Instance types
•m1.large & m1.xlarge
• EBS-optimized
•Consider PIOPS
•High I/O Quadruple XL (hi1.4xlarge) - SSDs!
NEW!
NEW!
NEW!
More instances to compensate for the performance variability of virtualized resources
Prepare Yourself
Cluster Sizes
• 5 nodes or moreBest performance & growth
• Scales linearly
•Don’t be stingy
Ring Size
•Change default value!
•Cannot be reset
• Ring size: 2568/64 vnodes per node
•Don’t go above 512
Provisioning
App
haproxy
MyRiakCluster: node-0
MyRiakCluster: node-1
MyRiakCluster: node-2
MyRiakCluster: node-3
MyRiakCluster: node-4
Riak Cluster
HTTPor
ProtobuffApp
haproxy
App
haproxy
Disable Swap• Riak process pages should
not hit swap
• Swap usage can result in unresponsive server
• Let the kernel kill it
Mount & Scheduler
•mount with noatime flag
•Use deadline scheduler on EBS cat /sys/block/xvdj1/queue/schedulernoop [deadline] cfq
echo deadline > /sys/block/xvdj1/queue/scheduler
Linux Filesystems
• xfs, ext3
• ext4
• barrier = 0
• data = writeback
• ZFS not recommended
Backend choices
•Bitcask*
•help mitigate EBS
• LevelDB
•MemoryCan be
changed and
optimized!
Kernel Optimizations
• /etc/sysctl.conf vm.swappiness = 0net.core.somaxconn= 4000net.ipv4.tcp_timestamps = 0net.ipv4.tcp_window_scaling = 1net.ipv4.tcp_max_syn_backlog = 40000net.ipv4.tcp_sack = 1net.ipv4.tcp_fin_timeout = 15net.ipv4.tcp_keepalive_intvl = 30net.ipv4.tcp_tw_reuse = 1
http://wiki.basho.com/Linux-Performance-Tuning.html
Monitoring
• stats or riak-admin status
•Collectd & Nagios
• Riaknostic
• Riak Control (maybe soon)
Backups
• Backup both the ring and data directories
• Integrate with EY tools
• scp or rsync data directory
• EBS snapshots
Scaling Riak
•Vertical
•Horizontal
• reduces node load
• batch is better
riak-admin cluster replace <old> <new>
Benchmarks
Tests
• 5-node cluster / 5 hours
• 50 concurrent connections
• gets 10 /puts 5 /deletes 1
• Bitcask backend
• Protobuffs protocol
Medium - PIOPS 500
Medium - PIOPS 1K
Medium - Ephemeral
FAIL!
XL - EBS
XL -PIOPS 500
4XL - EBS
4XL - PIOPS 1k
What’s next?