riak at engine yard cloud

Riak in the Cloud

Getting Started with

Big Nerd Ranch

Ines Sombra

@RandomMood

About Me

About Us

Our Agenda

The Problem

The Solution

Our Customers

Engine Yard

About US100% Cloud Based

Critical services MUST be up 24/7/365

CloudsRegions

StacksSupport

HA

DR

http://www.flickr.com/photos/meetrajesh/6972883401

Our Customers

An APPLICATION

App

App

App

DB Master

DB Replica

us-east-1a

us-east-1b

http://www.flickr.com/photos/brandongrasley/8227882239/

Then AWS Crashed

Our App AFTER

App

App

App

DB Master

DB Replica

us-east-1a

us-east-1b

X

!

This sucked, A LOTTime to strengthen our HA/DR Story

Customers and us need to be up all the time!

$

Relational

Document

Key/Value

Column

Data Models Consistency Partitions

Our Research

Are ACID properties really needed?

Consistency

Availability

Relational Model

Impedance Mismatch

Memory data-structure

Impedance Mismatch

We needed a fault tolerant and highly available (clustered)

datastore

Linearly scalable

Fault-tolerant

Finally provide HA/DR solution

OSS, Dynamo-based, distributed K/V store Hello RIAK!

Riak’s Use CasesApplication Type Key Value

Session User/Session id Data

Advertising Campaign id Data

Logs Date Log file

Content title, integer text, json, xml, media..

When is Riak a good fit

When availability is more important than consistency

When data can be modeled as keys/values

When the problem fits

Riak is OPS FRIENDLY

node1.2.1

node1.2.1

node1.2.1

node1.2.1

node1.2.1

Growth

Reduction

Upgrades

Failuresnode1.3.0

X

World’s most famous slide

Riak is SO MUCH MORE

Consistent HashingHinted Handoff

Active Anti EntropyTuneable consistencyN | W | R

OUR APP AFTER

App

App

node1.3.0

node1.3.0

node1.3.0

node1.3.0

node1.3.0

App

We need a Riak Cluster but we have

no hardware

RIAK In the Wild

18th and Harrison, San Francisco, CA

Let’s build A PRODUCT

http://www.flickr.com/photos/worldbank/8261699153/

1

2

3

4

Number of Nodes

Your CHOICESYou are trying

to create a Riak cluster. Do you need any help?

Instance TypeData LocationBackend Type

What we GIVE YOU

Our SUPPORT

X2

What you OPS to know

HOWTO

Prepare Yourself

More instances to compensate for the

performance variability of virtualized resources

^ m1.large & m1.xlarge

EBS-optimized or PIOPs

High I/O Quadruple XL (hi1.4xlarge) - SSDs!

Choosing INSTANCES

5 nodes or moreBest performance & growth

Scales linearly

Don’t be stingy

Cluster Sizes

Change DefaultCANNOT BE RESET

(8-64 vnodes per node)

Ring SIZE

Ring Size at 256

DONT GO OVER 512

Our PROVISIONING

App

haproxy

MyRiakCluster: node-0

MyRiakCluster: node-1

MyRiakCluster: node-2

MyRiakCluster: node-3

MyRiakCluster: node-4

Riak Cluster

HTTPor

ProtobuffApp

haproxy

App

haproxy

Riak Cluster

^ Riak process pages should not hit swap

Swap usage can result in unresponsive server

Let the kernel kill it

Disable SWAP

^ Mount with noatime flag

Use deadline scheduler on EBS

Mount & SCHEDULER

cat /sys/block/xvdj1/queue/schedulernoop [deadline] cfq

echo deadline > /sys/block/xvdj1/queue/scheduler

^ xfs, ext3

ext4 barrier = 0 data = writeback

ZFS not recommended

Linux FILESYSTEMS

^ Backup both the ring and data directories

scp or rsync data dir

EBS snapshots

Backup STRATEGY

Monitoring STRATEGY

Stats Interface

riak-admin status

Collectd

Nagios

^ Vertical

Scaling RIAK

riak-admin cluster replace <old> <new>

^ Horizontalriak-admin cluster add <new>

How we SCALE RIAK

+ PostgreSQLRIAK

Hybrid Solutions FTW

+ FTS Engine*RIAK

+ RedisRIAK

Try OUR RIAK

How difficult IS IT?

A few thoughts

Finally!

Pick the right tool for the job

Apps & Services can have multiple databases

Polyglot PERSISTENCE

On SOAMy super nice e-commerce site

Shopping Cart & Session Data

Completed Orders

Inventory and Item Pricing

Recommendations Engine

Session Storage serviceK/V Store

Order Persistence

serviceDocument

Store

Inventory & Price service

RDBMS

Nodes and relations service

Graph Store

NoSQL Distilled: Fowler & Sadalage

Don’t optimize too quickly

Take time to understand your problem

Complexity increases with more databases

We RECOMMEND

Do KEEP IN MIND

Relational Document Store!=

Different paradigms are different

Give me MOAR!

Whatever

Little Riak Book (https://github.com/coderoshi/little_riak_book/)

docs.basho.comblog.engineyard.comnosql.mypopescu.com

Great RESOURCES

Questions?