better, faster, cheaper infrastructure: apache cloudstack and riak cs
DESCRIPTION
Software is eating infrastructure. By pulling reliability and scalability responsibilities up the stack from hardware into software, object stores such as Basho's Riak CS and cloud orchestration platforms such as Apache CloudStack increase the utilization of compute and storage resources by dynamically shifting workloads based on demand. Together, those platforms can saturate compute and storage of 1000s of hosts with strong operational visibility and end-user self-service. This talk will cover the following topics to explore private cloud design principles and best practices: * Why Private Cloud? * Anatomy of a Private Cloud * Building a Apache CloudStack Compute Offering * Large Object Storage using Riak CSTRANSCRIPT
BETTER, FASTER, CHEAPER INFRASTRUCTURE: APACHE
CLOUDSTACK + RIAK CSJohn Burwell
([email protected] | @john_burwell)
Friday, November 8, 13
Who Am I
• Consulting Engineer @ Basho Technologies
• Member, Apache CloudStack PMC
• Ran operations and designed automated provisioning for hybrid analytic/virtualization clouds
• Led architectural design and server-side development of a SaaS physical security platform
Friday, November 8, 13
... but I submitted a ticket.
Friday, November 8, 13
Shifting Role Of IT
From Cost Center To Business Opportunity
Friday, November 8, 13
Shifting Role Of IT
From Cost Center To Business Opportunity
Infrastructure Be Boring
Friday, November 8, 13
Better
• Resilience
• Scalability
• Service
Friday, November 8, 13
Faster
• Resource Provisioning
• Systems
• Failure recovery
Friday, November 8, 13
Cheaper
• Hardware
• Labor
• Hosting
Friday, November 8, 13
Composable Infrastructure
• Resource Pooling
• Rapid Elasticity
• Self Service
• Metering/Charge Back
Friday, November 8, 13
Composable Infrastructure
• Resource Pooling
• Rapid Elasticity
• Self Service
• Metering/Charge Back
Late-binding of hardware roles
Friday, November 8, 13
Got It ... Why Go Private?
• Leverage existing infrastructure investment
• Customization and Control
• Regulatory/Security Requirements
• Cost
Friday, November 8, 13
Doesn’t vSphere/XCP/XenServer already do that?
Friday, November 8, 13
Friday, November 8, 13
Hopefully not ...
Friday, November 8, 13
Friday, November 8, 13
Traditional Workload
Management
Network
HypervisorCluster
HypervisorCluster
Shared����������� ������������������ Storage
• Reliable
• Fault tolerant
• Specialized hardware
• IT managed provisioning
• Static scale out model
Friday, November 8, 13
Traditional Workload
Management
Network
HypervisorCluster
HypervisorCluster
Shared����������� ������������������ Storage
• Reliable
• Fault tolerant
• Specialized hardware
• IT managed provisioning
• Static scale out model
Focused on resource consolidation not agility
Friday, November 8, 13
Software Is Eating Hardware
• Moore’s Law has pulled fault tolerance up the stack
• Commodity Hardware
• Reduced cost
• Services more workload types
• Dynamic resource allocation
Friday, November 8, 13
Software Is Eating Hardware
• Moore’s Law has pulled fault tolerance up the stack
• Commodity Hardware
• Reduced cost
• Services more workload types
• Dynamic resource allocation
Specialized Hardware Reduces Agility
Friday, November 8, 13
Cloud Workload• Reliable
• Fault Tolerant
• Commodity Hardware
• Self-Service provisioning
• Control Plane -> Dynamic scale out model
SDN
Rack Rack
Object����������� ������������������ Storage
App App
Friday, November 8, 13
Cloud Workload• Reliable
• Fault Tolerant
• Commodity Hardware
• Self-Service provisioning
• Control Plane -> Dynamic scale out model
SDN
Rack Rack
Object����������� ������������������ Storage
App App
Resource optimization increases agility
Friday, November 8, 13
TraditionalWorkload
CloudWorkload
Reality
Friday, November 8, 13
Complete Private Cloud
+
Compute Services Object Storage Services
Friday, November 8, 13
What Is Apache CloudStack?
§Open source software that deploys and manages large networks of virtual machines, as a highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform.
§CloudStack is a Top-Level Project at the Apache Software Foundation.
Friday, November 8, 13
What Is Apache CloudStack?
§Open source software that deploys and manages large networks of virtual machines, as a highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform.
§CloudStack is a Top-Level Project at the Apache Software Foundation.
Friday, November 8, 13
~300: Acknowledged production implementations
~43,000: Hosts in a single CloudStack instance
~85: Project committers
Friday, November 8, 13
Apache CloudStack Users
Friday, November 8, 13
Anatomy Of CloudStack• Management Server Farm
• Management and provisioning tasks
• Zone
• Collection of pods, network offerings and secondary storage
• Pod
• Collection of clusters in the same failure boundary
• Cluster
• A grouping of hosts and their associated storage
• Hosts
• Servers onto which services will be provisioned
• Primary Storage
• VM disk storage
• Network
• Logical network associated with service offerings
• Secondary Storage
• Template, snapshot and ISO storage
Zone
CloudStack Pod
Cluster
Host
Host
Network
PrimaryStorage
VM
VM
CloudStack Pod
Cluster
SecondaryStorage
Friday, November 8, 13
Management Server
• Orchestrates infrastructure management
• A single management server instance can manage 10,000 hosts
• For production, run n +1 management servers with at least one instance in each zone
• Cluster MySQL
Friday, November 8, 13
Control And Data Planes
• Automation control of resources (e.g. hypervisors, storage, switches, etc)
• Small, linear bandwidth consumption
• Low latency
• Transfer of data between infrastructure components (e.g. templates, snapshots, etc)
• Large, variable bandwidth consumption
• High throughput
Friday, November 8, 13
Control And Data Plane Scalability
• Separate the control and data plane compute and network traffic
• Data transfer operations offloaded to system virtual machines
• Separate management and storage networks
Friday, November 8, 13
Primary Vs Secondary Storage
• Random access read/write
• Changes often
• Zone Scoped
• Focused on throughput
• Write once/read many
• Append only
• Zone or Region Scoped
• Focused on reliability
Friday, November 8, 13
Primary Vs Secondary Storage
• Random access read/write
• Changes often
• Zone Scoped
• Focused on throughput
• Write once/read many
• Append only
• Zone or Region Scoped
• Focused on reliability
Secondary Storage is the lifeboat.
Friday, November 8, 13
What Is Riak CS?• Simple, reliable storage of large objects accessed using the S3 or Swift
API
• Objects are stored in buckets with a unique key
• Multi-tenant security model with per object ACLs
• Usage Accounting and Reporting
• Open Source (Apache2 Licensed)
Friday, November 8, 13
What Is Riak CS?• Simple, reliable storage of large objects accessed using the S3 or Swift
API
• Objects are stored in buckets with a unique key
• Multi-tenant security model with per object ACLs
• Usage Accounting and Reporting
• Open Source (Apache2 Licensed)
Your Own Private S3
Friday, November 8, 13
Anatomy Of Riak CS
• S3/Swift API Proxy + Riak key-value store
• Peer-to-Peer Design: Any node can service any read or accept any write request
• Maintains multiple replicas (default 3) spread across multiple nodes
• Ops Friendly
Friday, November 8, 13
Anatomy Of Riak CS
• S3/Swift API Proxy + Riak key-value store
• Peer-to-Peer Design: Any node can service any read or accept any write request
• Maintains multiple replicas (default 3) spread across multiple nodes
• Ops Friendly
Horizontally scaleable storage on commodity hardware
Friday, November 8, 13
Riak CS Use Cases
• Secondary Storage for VM snapshot and template data
• Database Backup Storage
• Serving Static Content
• Application Log Storage
Friday, November 8, 13
Achieving Commodity Reliability
• Create a zone per power source/backbone network connection
• Establish independent primary storage in each zone
• Utilize intra-datacenter block-level replication between zones
• Distribute Riak CS nodes evenly across zones
Friday, November 8, 13
The Complete Private Cloud
Management
Network
HypervisorCluster
HypervisorCluster
Shared����������� ������������������ Storage
SDN
Rack Rack
Riak����������� ������������������ CS
App App
Traditional����������� ������������������ Cluster Cloud����������� ������������������ Workload
Friday, November 8, 13
Up And Running With Riak CS
• Learn more @ http://basho.com/riak-cloud-storage
• Grab it @ http://docs.basho.com/riakcs/latest/riakcs-downloads/
• Follow us @basho
• Help from the [email protected] mailing list or on FreeNode @ #riak
• Contribute @ http://github.com/basho/riak_cs
Friday, November 8, 13
Join Our Community
• Learn more @ http://cloudstack.apache.org
• Grab it @ http://cloudstack.apache.org/downloads.html
• Follow us @cloudstack
• Help from the [email protected] mailing list or on Freenode @ #cloudstack
• Contribute on the [email protected] mailing list or on Freenode @ #cloudstack-dev
Friday, November 8, 13
Thoughts? Questions?
Friday, November 8, 13
Thank you!
Deck http://slideshare.net/JohnBurwell1Follow @ john_burwell
Email [email protected]
Friday, November 8, 13