openstack : cloud computing platform tutorial

8/3/2019 OpenStack : Cloud Computing Platform Tutorial

1/55

Community StackerBret Piatt

OpenStack TutorialIEEE CloudCom 2010

Twitter: @bpiatt


2/55

70s 80sMainframe Era

90s-2000sClient Server Era

2010-beyondCloud Era

[Based on a Gartner Study]

Application Platforms Undergoing A Major Shift

2010 IT budgets arent getting cut....but CIOs expect their spend to go further.

#1 Priority is Virtualization#2 is Cloud Computing


3/55

Founded in 1998Publicly traded on NYSE: RAX120,000+ customers

$628m revenue in 2009 across two major businessesDedicated Managed Hosting

Cloud Infrastructure & Apps (Servers, Files, Sites, Email)

Primary focus on customer service ("Fanatical Support")3,000+ employees

9 datacenters in the US, UK and Hong Kong65,000+ physical servers

Overview of Rackspace


4/55

Rackspace Cloud: 3 Products with Solid TractionCompute: Cloud Servers

Virtualized, API-accessible servers with root accessWindows & Linux (many distros)Sold by the hour (CPU/RAM/HDD) with persistent storageLaunched 2009Based on SlicehostXen & XenServer HVs

Storage: Cloud FilesLaunched 2008Object file storev2.0 in May 2010

PaaS: Cloud SitesLaunched 2006Formally MossoCode it & Load it: .Net, PHP, Python apps autoscaled

Source: Guy Rosen (http://www.jackofallclouds.com)


5/55

Open ReST APIs released July 2009 (Creative Commons License)

Included in major API bindings: Libcloud, Simple Cloud, jclouds, -cloudSupported by key cloud vendors and SaaS servicesMarketplace: http://tools.rackspacecloud.com

Active Ecosystem on Rackspace APIs
http://tools.rackspacecloud.com/


6/55

What is OpenStack?Overview of the project


7/55

OpenStack: The Mission

"To produce the ubiquitous Open Sourcecloud computing platform that will meet

the needs ofpublic and private cloudproviders regardless of size, by beingsimple to implement and massively

scalable."


8/55

OpenStack History

Rackspace Decidesto Open SourceCloud Software

March

NASA OpenSources Nebula

Platform

May June July

OpenStack formedb/w Rackspace and

NASA

Inaugural DesignSummit in Austin

20102005

RackspaceCloud

developed


9/55

OpenStack History

OpenStacklaunches with 25+

partners

July

First Austin coderelease with 35+

partners

October November February

First public DesignSummit in San

Antonio

Second Bexar coderelease planned

2011


10/55

OpenStack Founding Principles

Apache 2.0 license (OSI), open development process

Open design process, 2x year public Design Summits

Publicly available open source code repository

Open community processes documented andtransparent

Commitment to drive and adopt open standards

Modular design for deployment flexibility via APIs


11/55

Community with Broad CommercialSupport


12/55

OpenStack Isn't Everything

ConsultantsBusiness Process Automation

Database EngineersOperating SystemTechniciansSystems SecurityProfessionalsNetwork Experts

Servers, Firewalls, Load BalancersOperating SystemsStorageManagement Tools

Virtualization

Data CenterNetworkingPower


13/55

Software to provision virtual machines onstandard hardware at massive scale

Software to reliably store billions of objects

distributed across standard hardware

OpenStack Compute

OpenStackObject Storage

creating open source software to buildpublic and private clouds


14/55

OpenStack Release Schedule

Design Summit:April TBA 2011

Cactus:April 15, 2011

Bexar:February 3, 2011

OpenStack Compute readyfor enterprise private clouddeployments and mid-sizeservice providerdeploymentsEnhanced documentation

Easier to install and deploy

Community gathers toplan for next release,likely Fall 2011

OpenStack Compute ready forlarge service provider scaledeployments

This is the Rackspace-readyrelease; need to communicateRackspace support and plansfor deployment


15/55

Building an OpenStack CloudDatacenter, Hardware, and Process


16/55

Business Prerequisites


17/55

Technical Prerequisites


18/55

Cloud Ready Datacenter Requirements


19/55

Bootstrapping Your Physical NodesBootstrapping the Host Machines


20/55

Building an OpenStack CloudObject Storage


21/55

Zettabyte

1,000 Exabytes1,000,000 Petabytes

All of the data on Earth today(150GB of data per person)


22/55

Zettabyte2% OF THE DATA ON EARTH IN 2020


23/55

If we stored all of the global data as an average enterprise..

..it would take..

..38.5% of the World GDP!

Data Must Be Stored Efficiently

ITEM MONTHLY FIGURES

ENTERPRISE AVGERAGE STORAGE COST $1.98 PER GIGABYTE

WORLD GDP $5.13 TRILLION

COST TO STORE A ZETTABYTE $1.98 TRILLION


24/55

Object Storage Summary


25/55

ReST-based API Data distributed evenlythroughout system

Hardware agnostic: standardhardware, RAID not required

Object Storage Key Features

No centraldatabase

Scalable to multiplepetabytes, billions ofobjects

Account/Container/Object structure(not file system, no nesting) plus

Replication (N copies of accounts,containers, objects)


26/55

System Components

The Ring: Mapping of names to entities (accounts,containers, objects) on disk.

Stores data based on zones, devices, partitions, and replicas

Weights can be used to balance the distribution of partitionsUsed by the Proxy Server for many background processes

Proxy Server: Request routing, exposes the public API

Replication: Keep the system consistent, handle failures

Updaters: Process failed or queued updates

Auditors: Verify integrity of objects, containers, and accounts


27/55

System Components (Cont.)

Account Server: Handles listing of containers, stores as SQLite DB

Container Server: Handles listing of objects, stores as SQLite DB

Object Server: Blob storage server, metadata kept in xattrs, data inbinary format

Recommended to run on XFS

Object location based on hash of name & timestamp


28/55

Software DependenciesObject Storage should work on most Linux platforms with the followingsoftware (main build target for Austin release is Ubuntu 10.04):

Python 2.6

rsync 3.0

And the following python libraries:

Eventlet 0.9.8

WebOb 0.9.8

Setuptools

SimplejsonXattr

Nose

Sphinx


29/55

Evolution of Object StorageArchitecture

Version 1: Central DB(Rackspace Cloud Files 2008)

Version 2: Fully Distributed(OpenStack Object Storage 2010)


30/55

Example Small Scale Deployment


31/55

5 Zones2 Proxies per 25

Storage Nodes10 GigE to Proxies

1 GigE to

Storage Nodes24 x 2TB Drivesper Storage Node

Public Internet

Load Balancers (SW)

Example Large Scale Deployment -- Many Configs Possible

Example OpenStackObject Storage Hardware


32/55

Building an OpenStack CloudCompute


33/55

Asynchronouseventually consistent

communication

ReST-based API

Horizontally andmassively scalable

Hypervisor agnostic:support for Xen ,XenServer, Hyper-V,

KVM, UML and ESX is coming Hardware agnostic:standard hardware, RAID not required

OpenStack Compute Key Features


34/55

API: Receives HTTP requests,converts commands to/from APIformat, and sends requests to cloud

controller

Cloud Controller: Global state ofsystem, talks to LDAP, OpenStackObject Storage, and node/storageworkers through a queue

User Manager

ATAoE / iSCSI

Host Machines: workersthat spawn instances

Glance: HTTP + OpenStack ObjectStorage for server imagesOpenStack Compute


35/55

System ComponentsAPI Server: Interface module for command and control requestsDesigned to be modular to support multiple APIsIn current release: OpenStack API, EC2 Compatibility ModuleApproved blueprint: Open Cloud Computing Interface (OCCI)

Message Queue: Broker to handle interactions between servicesCurrently based on RabbitMQ

Metadata Storage: ORM Layer using SQLAlchemy for datastoreabstraction

In current release: MySQLIn development: PostgreSQL

User Manager: Directory service to store user identitiesIn current release: OpenLDAP, FakeLDAP (with Redis)

Scheduler: Determines the placement of a new resourcerequested via the API

Modular architecture to allow for optimizationBase schedulers included in Austin: Round-robin, Least busy


36/55

System Components (Cont.)Compute Worker: Manage compute hosts through commandsreceived on the Message Queue via the API

Base features: Run, Terminate, Reboot, Attach/DetachVolume, Get Console Output

Network Controller: Manage networking resources on computehosts through commands received on the Message Queue via theAPI

Support for multiple network modelsFixed (Static) IP addressesVLAN zones with NAT

Volume Worker: Interact with iSCSI Targets to manage volumesBase features: Create, Delete, Establish

Image Store: Manage and deploy VM images to host machines


37/55

Hypervisor Independence

Cloud applications should be designed and packaged abstracted fromthe hypervisor, deploy and test for best fit for your workloadManage application definition and workload, not the machine image

Configuration managementAbstract virtual machine definition

Open Virtualization Format


38/55

Network Models

Private VMs on Project VLANs or Public VMs on flat networks


39/55

Network Details

Security Group: Named collection of network access rulesAccess rules specify which incoming network traffic should bedelivered to all VM instances in the groupUsers can modify rules for a group at any time

New rules are automatically enforced for all running

instances and instances launched from then onCloudpipe: Per project VPN tunnel to connect users to the cloudCertificate Authority: Used for Project VPNs and to decryptbundled imagesCloudpipe Image: Based on Linux with OpenVPN

Example OpenStack


40/55

Server GroupsDual Quad CoreRAID 10 Drives1 GigE Public1 GigE Private1 GigE Management

Public Network

Private Network(intra data center)

Management

Example OpenStackCompute Hardware

(other models possible)


41/55

Example innovation: Simcloud


42/55

Questions & Answers

Thank You!

Email: [email protected] Piatt

Twitter: @bpiatt


43/55

Backup ContentAdditional Information


44/55

Project Technical DocumentationOverall: http://wiki.openstack.orgObject Storage (Swift): http://swift.openstack.orgCompute (Nova): http://nova.openstack.org

Project General DocumentationHome Page: http://openstack.orgAnnouncements: http://openstack.org/blog

OpenStack Documentation
http://openstack.org/bloghttp://openstack.org/bloghttp://openstack.org/http://openstack.org/http://nova.openstack.org/http://swift.openstack.org/http://swift.openstack.org/http://wiki.openstack.org/http://wiki.openstack.org/http://openstack.org/bloghttp://openstack.org/http://nova.openstack.org/http://swift.openstack.org/http://wiki.openstack.org/


45/55

OpenStack: Core Open Principles

Open Source: All code will be released under the Apache Licenseallowing the community to use it freely.

Open Design: Every 6 months the development community will hold adesign summit to gather requirements and write specifications for theupcoming release.

Open Development: We will maintain a publicly available source coderepository through the entire development process. This will be hostedon Launchpad, the same community used by 100s of projectsincluding the Ubuntu Linux distribution.

Open Community: Our core goal is to produce a healthy, vibrantdevelopment and user community. Most decisions will be made usinga lazy consensus model. All processes will be documented, open andtransparent.


46/55

Backup ContentBootstrapping a cloud


47/55

Hardware Selection

OpenStack is designed to run on industrystandard hardware, with flexible configurations

Computex86 Server (Hardware Virt. recommended)Storage flexible (Local, SAN, NAS)

Object Storagex86 Server (other architectures possible)Do not deploy with RAID (can use controller for cache)


48/55

Server Vendor Support

Find out how much configuration yourhardware vendor can provide

Basic needsBIOS settings

Network bootIP on IPMI card

Advanced supportHost OS installation

Still get management network IP via DHCP


49/55

Network Device Configuration

Build in a manner that requires minimal changeLay out addressing in a block based modelGo to L3 from the top of rack uplink

Keep configuration simpleMore bandwidth is better than advanced QoSLet the compute host machines create logical zones


50/55

Host Networking

DHCP for the management networkInfinite leasesBase DNS on IP

Ex. nh-pod-a-10-241-61-8.example.orgOpenStack Compute handles IP provisioningfor all guest instances Cloud deployment toolsonly need to setup management IPs


51/55

Host OS Seed Installation

BOOTP / TFTP Simple to configureSecurity must be handled outside of TFTPHost node must be able to reach management

system via broadcast requestTop of rack router can be configured to forward

GPXENot all hardware supports

Better concurrent install capability than TFTP


52/55

Host OS Installation

Building a configuration based on a scriptedinstallation is better than a monolithicgolden image

Preseed for Ubuntu / Debian hostsKickstart for Fedora / CentOS / RHEL hostsYaST for SUSE / SLES hostsRemote bootstrapping for XenServer / Hyper-V hosts

Scripted configuration allows for incrementalupdates with less effort


53/55

Post OS Configuration

Utilize a configuration management solutionPuppet / Chef / Cfengine

Create roles to scale out controller infrastructure

QueueDatabaseController

Automate registration of new host machinesBase the configuration to run on management net IP


54/55

Backup ContentCompute


55/55

Component Architecture Detail

openstack : cloud computing platform tutorial

Documents