openstack architected like aws (and gcp)
DESCRIPTION
A description of how we built Open Cloud System (OCS), an OpenStack-powered complete cloud operating system. With a focus on AWS and GCE interoperability, we describe why hybrid cloud interoperability matters and how we got there. Anyone can do it and we think you should too.TRANSCRIPT
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution*!* All unlicensed or borrowed works retain their original licenses
OpenStack Architected Like AWS (and GCP)
Randy Bias, Founder & CEO!OpenStack Juno Design Summit!
May 14th, 2014
@randybias
OpenStack’s future is hybrid to public clouds!
WhoOpenStack Foundation Board of Directors!
!
Scale Builder:!100k+ sq ft datacenters @ 14MW!1000s of servers, 100s of switches!
!
Prod. OpenStack pioneer, Cloudscaling:!Part of OpenStack community since July 2010 (launch)!Firsts:!
OpenStack Public Compute Cloud in U.S., Public Storage Cloud in U.S. and KR!!
Top 10 Cloud Computing Pioneer
2
A Leader in Production Deployments*
3
* e.g. Currently actively supported clouds; legacy clouds from PS days are not included (multiply by 2-3x if you care)
supported clouds
total cores
total storage
10+
8,000+
~8.5PB
total servers ~600
# of Fortune 15? 2
Today’s GoalsWhy an AWS-like “elastic” cloud?!
Hybrid Cloud Interoperability!
AWS Flavored OpenStack!
Bring It Home
4
1
2
3
4
Why Elastic Cloud?
Top Clouds Are Elastic
6
Epitomized by AWS/GCE!Scale-out model over scale-up model!Small failure domains!
Foundation for Cloud Native Apps!Apps that “route around failure”!Manage own data replication/redundancy!Automated deployments!
Economics are King!Open source software!Vanity-free commodity hardware!Low-cost, scale-out networking
7
What is an Elastic Cloud?
For more information see the Chicago DevOps presentation at http://tinyurl.com/elasticcloud!
Elastic Cloud is a Mindset Change
8
Attribution: Bill Baker, Distinguished Engineer, Microsoft
bowzer.company.com (pets)
web001.company.com (cattle)
Cloud is the era of the disposable server
vs.
Elastic Clouds Host Cloud Native Apps
9
Legacy Enterprise Model!(inelastic)
Cloud Enterprise Model!(elastic)
99.9%!Applications
99.999%!Infrastructure!
($$$$)
99.999% Applications
99% Infrastructure!
($$)
Responsibility!
Shift
Cloud Native Apps Drive Agility
10
Traditional Applications
2012 89M
2015 139M
+56%
New Scale-Out Applications
2012 5M
2015 32M
+540%
Source: IDC & Gartner via EMC @ http://stevetodd.typepad.com/my_weblog/2013/11/a-mobile-to-storage-analytics-architecture.html
Jevon’s Paradox Accelerates Innovation
11
* SOURCE: http://greenimalist.com/2011/04/the-jevons-paradox/
** SOURCE: http://blogs.shell.com/climatechange/2010/09/the-other-side-of-energy-efficiency/
As the cost of compute approaches zero…⬇ cost of experimentation and ⬆ agility drives innovation
As cost ⬇… consumption ⬆
Why Should the Enterprise Care?
“Seventy percent of the companies that were on the Fortune 1000 list a mere 10 years ago have now vanished – unable to adapt to change … ”!!
! — Forrester analyst Craig Le Clair*
12
* SOURCE: http://blogs.forrester.com/craig_le_clair/13-09-09-make_business_agility_a_key_corporate_attribute_it_could_be_what_saves_you
OpenStack: For ALL Clouds
13
Although … the best fit is elastic cloud
Metering / Billing Platform
Elastic Private / Public Cloud
PaaS Platform
Storage Platform
OpenStack!Components
Cloud App Mgt Framework
Hybrid Cloud Enables Biz AgilityTop public clouds are elastic!Elastic clouds have similar attributes!
essentially they share a “reference architecture”!
Agility is primary value for enterprise!Cloud native apps + DevOps drive agility!
they “route around failures”!
Elastic cloud is ideal platform for these apps!doesn’t matter if it’s private or public!
Businesses need choice -> hybrid cloud
14
1
2
3
4
5
6
Hybrid Cloud Interoperability
Hybrid Cloud Value Proposition
16
Application Portability ~ Independence!Move apps between clouds with impunity!Advanced capabilities such as Cloud-bursting/bridging!Future-proofing and reduction of vendor lock-in!Pick the right cloud for the job!!
Portability Requires Interoperability!API compatibility is a subset of what is required!!
Interoperability Is NOT Easy!(details to follow)
Pub Cloud Interop Matters
17
Private Clouds Must be Interoperable with Public Cloud Leaders
OpenStack’s Future is Hybrid
API Compatibility is Not Enough
18
Mac TruckInfiniti G37!
330 HP!270 lb ft torque!5 Second 0-60
600 HP!2,100 lb ft torque!60 Second 0-60
Different Behavior
Semantics & Architecture
Engine
Transmission
Braking
Suspension
Lighting
Same API
19
Hybrid Cloud Interop Requirements
1
2
3
4
5
6
Matching SLAs & Availability
Performance & QoS Guarantees
Infrastructure Feature Parity
Similar TCO / Economies of Scale
API Compatibility
Behavioral Compatibility
Hybrid Cloud Interop Requirements
20
Shared Requirement Public / Private Cloud Interoperability DetailMatching SLAs and Availability Guarantees
VM uptime (hypervisor availability) API endpoint availability
Storage system availability
Performance & QoS Guarantees
VM vCPU, networking & local storage performance Block storage system performance matching
Object storage aggregate throughput
Infrastructure Feature Parity
Must have a majority of same primary features AWS -> EC2, S3, EBS, VPC, ELB
GCP -> VMs, persistent block/object, virt. networks, etc.
Similar Cost Economics & Economies of Scale
Within ½-2x of wholesale costs/pricing TCO must be similar, particularly for operational costs
API Compatibility Support for public cloud APIs At the least: AWS, GCE, Azure, and OpenStack
Behavioral Compatibility
Implied, non-API behavior is same or similar example 1: VMs start in <30 mins
example 2: floating/elastic IP auto-assignment
1
2
3
4
5
6
Enable a Hybrid-first Cloud Strategy
21
HYBRID CLOUD
PUBLIC CLOUDPRIVATE CLOUD
Interoperability means !APIs, QoS, SLAs, TCO, Feature Parity & Same Behavior
AWS Flavored!OpenStack
AWS Interoperable OpenStack
23
1
2
3
4
5
6
Matching SLAs & Availability
Performance & QoS Guarantees
Infrastructure Feature Parity
Similar TCO / Economies of Scale
API Compatibility
Behavioral Compatibility
Matching SLAs & Availability Guarantees
24
1
HA Pair Failover / Master Election
Service Service
vs.
Load Balancing
Service1 Service2 Service3 ServiceN...
Active Passive
Active Active Active Active
With Load Balancing, the control plane scales as the cloud grows
Delivers a 99.999% Uptime APIs & Control Plane on 99.5% Hardware
25
Performance & QoS Guarantees
Spread First Scheduling
m3.2xl
c3.xl
r3.xl
c3.mm1.l
Gaps = Wasted I/O & Hardware Resources
t1
t1
t1m1.m
Bin Packing Scheduler1) fixed proportions of resources!2) all resources allocated 3) manage oversubscription rates!4) ensures Quality of Service (QoS)!5) easy to do w/ filters & host-aggregates
2AWS/GCP-Style Instance Sizing & Scheduling
Bin Packing Scheduling(by Instance Family)
m3.xl
m3.2xl
m3.xl
m3.2xl
m3.2xl
Matching Infrastructure Feature Parity
26
3
Infrastructure Application!Services
PaaS / !Orchestration
Route53
Applications WorkSpacesApp Marketplace
CloudFrontRDS DynamoDB
ElastiCacheRedshift
SimpleDB
EMR Kinesis Data Pipeline
Elastic Beanstalk
CloudFormation OpsWorks
AppStream CloudSearch
SWFSQS SESSNS
Elastic Transcoder
Core Infrastructure Services
EC2 EBS S3VM Import/Export
Auto Scaling
CloudWatchClassic Networking
VPC Networking
Mgt Console IAMCLI
ELB
Matching Infrastructure Feature Parity
27
3
Core Infrastructure Services
EC2EBS
S3
IAMAuto ScalingCloudWatch
Classic NetworkingVPC Networking
Mgt ConsoleCLI
ELB
Nova
Swift
Heat Auto Scaling
LBaaS
L2/L3 Nova NetworkingNeutron SDN Plugin
Horizon
Cinder
KeystoneCLI
Heat Monitoring
VM Import/Export Glance
#1 Cost = Servers!#2 Cost = Power!#3 Cost = Network!!!!!Use industry standard servers & build a cattle cloud (one NIC, one switch, one PSU, etc) reduces costs significantly!
Similar TCO & Economies of Scale
28
4What is required to meet Amazon’s cost structure?
* SOURCE: http://mvdirona.com/jrh/TalksAndPapers/JamesHamilton_Reinvent20131115.pdf
Servers
Power
NetworkOther
Power Dist & Cooling
Similar TCO & Economies of Scale
29
4
Time
Public Cloud@ 20% Annual Growth"($14.2m over 5 Years)
Yr1 Yr2 Yr3 Yr4 Yr5
$1m
$2m
Cost / Yr
$3m
$4m
Private Cloud@ 20% Annual Growth"($9.2m over 5 Years)
1.9
2.3
2.8
3.3
3.9
2.6
1.21.6
1.82.1
OpenStack Elastic Cloud TCO
35% savings over AWS over 3-5 years
2384 cores"100TB object storage"50TB block storage
API Compatibility
30
OpenStack “Core” Services
5OpenStack already has multiple APIs
Native APIs GCE APIsAWS APIs
Behavioral Compatibility Testing
Implicit non-API behavior crucial to test!We have already been testing it w/ Tempest!
Tempest has a pretty sweet set of AWS tests (see next)!
Some of them are turned off by default!OpenStack default settings don’t pass it seems!
Cloudscaling has created more!Ultimately we need exhaustive Tempest AWS behavioral test suite!
Then GCP, then Azure
31
6
Tempest Tests for AWS Behavior
32
4/29/14, 8:00 AMAWS Compat - Google Sheets
Page 1 of 1https://docs.google.com/a/cloudscaling.com/spreadsheets/d/131mg8Kp3vCyeUadr5gsn2X3fLSx8_TXsKatJaQiWBis/edit#gid=0
|
AWS CompatFile Edit View Insert Format Data Tools Add-ons Help View only
tempest.thirdparty.boto.test_s3_buckets.S3BucketsTest
Comments
Count: 25Sheet1
AWS OCS OpenStack
6
Cloudscaling Tempest Tests for AWS Behavior*
33
4/29/14, 8:03 AMAWS Compat - Google Sheets
Page 1 of 1https://docs.google.com/a/cloudscaling.com/spreadsheets/d/131mg8Kp3vCyeUadr5gsn2X3fLSx8_TXsKatJaQiWBis/edit#gid=0
|
AWS CompatFile Edit View Insert Format Data Tools Add-ons Help View only
tempest.cloudscaling.thirdparty.aws_compat.test_ec2_instance_mysql.InstanceMySQLTest
Comments
Count: 37Sheet1
AWS OCS OpenStack
6
* In process of upstreaming now!
* Plan to work with RefStack on “AWS” flavored reference stack for OpenStack
How Do You Know You’re Done?
34
1
2
3
4
5
6
Same Availability
Same Performance
Similar Services
Equivalent TCO
Same APIs
Behaves Identically
Open Cloud System (OCS) Architecture
35
Available. Performant. Interoperable.
Hybrid Cloud Operating System
Hardware
Compute Network Storage
100% OpenStack Distro
L3!Networking VPC EBS
AWS API OpenStack API GCE API
HA Control Plane
Cloud Operator Console
Metrics
Analytics
Logs
Management
Cloud Orchestration
Agility, DevOps, & AWS + OpenStack
Bringing It Home
37
Business Agility
Cloud Native
Apps
DevOps
enablementElastic Cloud
Architecture
Hybr
id to
Pu
blic
OpenStack provides ideal platform for a hybrid cloud operating system designed in the elastic cloud model