building complete private clouds with apache cloudstack and riak cs

BUILDING COMPLETE PRIVATE CLOUDS WITH APACHE CLOUDSTACK + RIAK CS

John Burwell(jburwell@basho.com | @john_burwell)

Wednesday, October 23, 13

Who Am I

• Consulting Engineer @ Basho Technologies

• Member, Apache CloudStack PMC

• Ran operations and designed automated provisioning for hybrid analytic/virtualization clouds

• Led architectural design and server-side development of a SaaS physical security platform

Wednesday, October 23, 13

... but I submitted a ticket.

Wednesday, October 23, 13

Shifting Role Of IT

From Cost Center To Business Opportunity

Wednesday, October 23, 13

Wednesday, October 23, 13

Agility

Exploitation Requires Rapid Execution

Infrastructure management must be delegated

Wednesday, October 23, 13

Agility

Exploitation Requires Rapid Execution

Infrastructure management must be delegated

Infrastructure Must Become Boring

Wednesday, October 23, 13

Composable Infrastructure

• Resource Pooling

• Rapid Elasticity

• Self Service

• Metering/Charge Back

Wednesday, October 23, 13

Got It ... Why Go Private?

• Leverage existing infrastructure investment

• Customization and Control

• Regulatory/Security Requirements

• Cost

Wednesday, October 23, 13

Doesn’t vSphere/XCP/XenServer already do that?

Wednesday, October 23, 13

Traditional Workload

Management

Network

HypervisorCluster

HypervisorCluster

Shared����������� ������������������  Storage

• Reliable

• Fault tolerant

• Specialized hardware

• IT managed provisioning

• Static scale out model

Wednesday, October 23, 13

Traditional Workload

Management

Network

HypervisorCluster

HypervisorCluster

Shared����������� ������������������  Storage

• Reliable

• Fault tolerant

• Specialized hardware

• IT managed provisioning

• Static scale out model

Focused on resource consolidation not agility

Wednesday, October 23, 13

Software Is Eating Hardware

• Moore’s Law has pulled fault tolerance up the stack

• Commodity Hardware

• Reduced cost

• Services more workload types

• Dynamic resource allocation

Wednesday, October 23, 13

Software Is Eating Hardware

• Moore’s Law has pulled fault tolerance up the stack

• Commodity Hardware

• Reduced cost

• Services more workload types

• Dynamic resource allocation

Specialized Hardware Reduces Agility

Wednesday, October 23, 13

Cloud Workload• Reliable

• Fault Tolerant

• Commodity Hardware

• Self-Service provisioning

• Control Plane -> Dynamic scale out model

SDN

Rack Rack

Object����������� ������������������  Storage

App App

Wednesday, October 23, 13

Cloud Workload• Reliable

• Fault Tolerant

• Commodity Hardware

• Self-Service provisioning

• Control Plane -> Dynamic scale out model

SDN

Rack Rack

Object����������� ������������������  Storage

App App

Resource optimization increases agility

Wednesday, October 23, 13

TraditionalWorkload

CloudWorkload

Reality

Wednesday, October 23, 13

Complete Private Cloud

+

Compute Services Object Storage Services

Wednesday, October 23, 13

What Is Apache CloudStack?

§Open source software that deploys and manages large networks of virtual machines, as a highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform.

§CloudStack is a Top-Level Project at the Apache Software Foundation.

Wednesday, October 23, 13

What Is Apache CloudStack?

§Open source software that deploys and manages large networks of virtual machines, as a highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform.

§CloudStack is a Top-Level Project at the Apache Software Foundation.

Wednesday, October 23, 13

Apache CloudStack Users

Wednesday, October 23, 13

Anatomy Of CloudStack• Management Server Farm

• Management and provisioning tasks

• Zone

• Collection of pods, network offerings and secondary storage

• Pod

• Collection of clusters in the same failure boundary

• Cluster

• A grouping of hosts and their associated storage

• Hosts

• Servers onto which services will be provisioned

• Primary Storage

• VM disk storage

• Network

• Logical network associated with service offerings

• Secondary Storage

• Template, snapshot and ISO storage

Zone

CloudStack Pod

Cluster

Host

Host

Network

PrimaryStorage

VM

VM

CloudStack Pod

Cluster

SecondaryStorage

Wednesday, October 23, 13

Primary Vs Secondary Storage

• Random access read/write

• Changes often

• Zone Scoped

• Focused on throughput

• Write once/read many

• Append only

• Zone or Region Scoped

• Focused on reliability

Wednesday, October 23, 13

Primary Vs Secondary Storage

• Random access read/write

• Changes often

• Zone Scoped

• Focused on throughput

• Write once/read many

• Append only

• Zone or Region Scoped

• Focused on reliability

Secondary Storage is the lifeboat.

Wednesday, October 23, 13

What Is Riak CS?• Simple, reliable storage of large objects accessed using the S3 or Swift

API

• Objects are stored in buckets with a unique key

• Multi-tenant security model with per object ACLs

• Usage Accounting and Reporting

• Open Source (Apache2 Licensed)

Wednesday, October 23, 13

What Is Riak CS?• Simple, reliable storage of large objects accessed using the S3 or Swift

API

• Objects are stored in buckets with a unique key

• Multi-tenant security model with per object ACLs

• Usage Accounting and Reporting

• Open Source (Apache2 Licensed)

Your Own Private S3

Wednesday, October 23, 13

Anatomy Of Riak CS

• S3/Swift API Proxy + Riak key-value store

• Peer-to-Peer Design: Any node can service any read or accept any write request

• Maintains multiple replicas (default 3) spread across multiple nodes

• Ops Friendly

Wednesday, October 23, 13

Anatomy Of Riak CS

• S3/Swift API Proxy + Riak key-value store

• Peer-to-Peer Design: Any node can service any read or accept any write request

• Maintains multiple replicas (default 3) spread across multiple nodes

• Ops Friendly

Horizontally scaleable storage on commodity hardware

Wednesday, October 23, 13

Riak CS Use Cases

• Secondary Storage for VM snapshot and template data

• Database Backup Storage

• Serving Static Content

• Application Log Storage

Wednesday, October 23, 13

Achieving Commodity Reliability

• Create a zone per power source/backbone network connection

• Establish independent primary storage in each zone

• Utilize intra-datacenter block-level replication between zones

• Distribute Riak CS nodes evenly across zones

Wednesday, October 23, 13

The Complete Private Cloud

Management

Network

HypervisorCluster

HypervisorCluster

Shared����������� ������������������  Storage

SDN

Rack Rack

Riak����������� ������������������  CS

App App

Traditional����������� ������������������  Cluster Cloud����������� ������������������  Workload

Wednesday, October 23, 13

Up And Running With Riak CS

• Learn more @ http://basho.com/riak-cloud-storage

• Grab it @ http://docs.basho.com/riakcs/latest/riakcs-downloads/

• Follow us @basho

• Help from the riak-users@lists.basho.com mailing list or on FreeNode @ #riak

• Contribute @ http://github.com/basho/riak_cs

Wednesday, October 23, 13

Join Our Community

• Learn more @ http://cloudstack.apache.org

• Grab it @ http://cloudstack.apache.org/downloads.html

• Follow us @cloudstack

• Help from the cloudstack-users@cloudstack.apache.org mailing list or on Freenode @ #cloudstack

• Contribute on the cloudstack-dev@cloudstack.apache.org mailing list or on Freenode @ #cloudstack-dev

Wednesday, October 23, 13

Thoughts? Questions?

Wednesday, October 23, 13

Thank you!

Deck http://slideshare.net/JohnBurwell1Follow @ john_burwell

Email jburwell@basho.com

Wednesday, October 23, 13