riak cs in cloudstack

Simple, Available Cloud Storage For Cloudstack

Overview

On March 27, 2012

Bashoannounced a newproduct called

Riak CS

On September 5, 2012

BASHOjoined Apache Cloudstack

On March 20, 2013

Riak CSbecame open source

Riak CS is...

enterprise cloud storage

RiakS3-compatibility

multi-tenancy

per user reporting

offeri

ngbuilton topof

large object storage

Enabling you to host your own

PUBLICPRIVATE

&CLOUDS

or….Reliable Storage Behind Apps

Basho's Commits

@john_burwell 's contribution:@john_burwell 's contribution:S3-backed secondary storage feature in 4.1.0S3-backed secondary storage feature in 4.1.0Uses S3 to sync secondary storage across zonesUses S3 to sync secondary storage across zones

Long term: (shhhhhh!)Native S3 SupportFederated authentication and authorization

DataPipeblog.datapipe.com/datapipe-cloudstack

“Riak CS provides the high-performance,

distributed datastore

we need to deliver a sound foundation for

our cloud storage needs now

and for many years into the future”

- Ed Laczynski, VP Cloud Strategy, Datapipe.

Yahoo!“Today, Yahoo! leverages Riak CS Enterprise to offer an

S3-compatible public cloud storage service,

as well as dedicated hosting options ...

Yahoo! is highly supportive of open source software

and we view Basho’s (OSS) announcement as

a positive move that will work

to accelerate its ability to innovate

and ultimately strengthen our cloud platform.”

- Shingo Saito, cloud product manager, Yahoo!

About Riak

RiakDynamo-inspired key/value store

Written in Erlang with C/C++

Open source under Apache 2 license

Thousands of production deployments

RiakHigh availability

Low-latency

Horizontal scalability

Fault-tolerance

Ops friendliness

RiakMasterless• No master/slave or different roles• All nodes are equal• Write availability and scalability • All nodes can accept/route requests

RiakNo Sharding• Consistent hashing• Prevents “hot spots”• Lowers operational burden of scale• Data rebalanced automatically

RiakAvailability and Fault-Tolerance • Automatically replicates

data• Read and write data during

hardware failure and network partition

• Hinted handoff

How It Works

Riak CS

RiakStanchion

Riak CS nodefor every node of Riak1

RiakNode

RiakNode

RiakNode

RiakNode

RiakNode

Large Object

Riak CS

S3API

ReportingAPI

Riak CS

S3API

ReportingAPI

Riak CS

S3API

ReportingAPI

Riak CS

S3API

ReportingAPI

Riak CS

S3API

ReportingAPI

1. User uploads an object

1 MB

2. Riak CSbreaks object

into 1 MB chunks

1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB

3. Riak CSstreams chunksto Riak nodes

4. Riak replicatesand stores

chunks

USERSmulti-tenancy:

Riak CS will trackindividual usage/stats

BASIC

CONCEP

TS

access_key secret_key

users identified byusers authenticated by

BUCKETSusers create buckets.

buckets are like folders.

store objects in buckets.

names are globally unique.

BASIC

CONCEP

TS

OBJECTSstored in buckets.

objects are opaque.

store any file type.

BASIC

CONCEP

TS

Features

Riak CS Large Object Support• Started with 5GB / object• Now have multipart upload• Content agnostic

Riak CS S3-Compatible API• Use existing S3 libraries

and tools• RESTful operations• Multipart upload • S3-style ACLs for

object/bucket permissions• S3 authentication scheme

Riak CS Administration and Users• Interface for user creation,

deletion, and credentials • Configure so only admins can

create users

Riak CS New Stuff in Riak 1.3• Multipart upload: parts

between 5MB and 5GB • Support for GET range

queries• Restrict access to buckets

based on source IP

Riak CS

Riak CS Packages• Debian• Ubuntu• FreeBSD• Mac• Red Hat Enterprise• Fedora• SmartOS• Solaris • Source

Operations

statsDTrace

&built-in

support

track access &storage per user

monitor totalcluster ops

inspect ops with DTrace probes

OPERATIONAL STATSexposed via HTTP resource: /riak-cs/stats

block bucket object

GET, PUT, DELETELIST KEYS, CREATE,DELETE, GET/PUT ACL

GET, PUT, DELETEHEAD, GET/PUT ACL

HISTOGRAMS & COUNTERS

TRACK INDIVIDUAL USER’S

THE “USAGE”BUCKET

ACCESS STORAGE

QUERY USAGE STATS

Storage and access statistics tracked on per-user basis, as rollups for slices of time

•Operations, Count, BytesIn, BytesOut, + system and user error•Objects, Bytes

Enterprise

Multi-Datacenter Replication

• For active backups, availability zones, disaster recovery, global traffic

• Real-time or full-sync• 24/7 support• Per-node or storage-based

pricing

basho.com

SIGN UP FOR AN ENTERPRISE DEVELOPER

TRIAL http://docs.basho.com/

Riak LondonA distributed systems

meet/drink up

www.meetup.com/riak-london

github.com/bashotwitter.com/

bashodocs.basho.com

Q&A@_stu_