cloudstack architecture

CloudStack Architecture

4/29/2012Chiradeep Vittal

Alex Huang

Outline

• Overview of CloudStack • Problem Definition• Feature set overview• System VMs• System Architecture & Context• Component View

•Secure, multi-tenant cloud orchestration platform– Turnkey platform for delivering

IaaS clouds– Hypervisor agnostic– Scalable, secure and open– Open source, open standards– Deploys on premise or as a hosted

solution

•Deliver cloud services faster and cheaper

What is CloudStack?

Build your cloud the way the world’s most successful clouds

are built

CloudStack Supports Multiple Cloud Strategies

Multi-tenantPublic Cloud

• Dedicated resources

• Security & total control

• Internal network• Managed by

Enterprise or 3rd party

• Mix of shared and dedicated resources

• Elastic scaling• Pay as you go• Public internet,

VPN access

Hosted Enterprise Cloud

• Dedicated resources

• Security• SLA bound• 3rd party owned

and operated

Private Clouds Public Clouds

On-premise Enterprise Cloud

Compute

CloudStack Provides On-demand Access to Infrastructure Through a Self-Service Portal

Network Storage

Admin

Users

Org A

Admin

Users

Org BUsers

End User

Admin

Open Flexible Platform

Compute

XenServer VMware KVMOracle VM Bare metal

Hypervisor

Storage

Local Disk iSCSI NFSFiber Channel Swift

Block & Object

Network

Network Type Isolation Load

balancerFirewall VPN

Network & Network Services

Primary Storage Secondary Storage

Problem Definition

• Offer a scalable, flexible, manageable IAAS platform that follows established cloud computing paradigms

• IAAS– Orchestrate physical and virtual resources to offer self-service

infrastructure provisioning and monitoring• Scalable

– 1 -> N hypervisors / VMs / virtual resources– 1 -> N end users

• Flexible– Handle new physical resource types

• Hypervisors, storage, networking

– Add new APIs– Add new services– Add new network models

Problem Definition (contd)

• Manageable– Hide complexity of underlying resources– Rich functional end-user and admin UI– Admin API to automate operations– Easy install, upgrade for small -> large clouds– Simple scaling, automated resilience

• Established Paradigms– EC2 –inspired

• Semantic variations based on cloud provider needs, hypervisor capabilities

End-User Experience

Self-service Portal to Create & Manage VMs

Select Operating System• Windows, Linux

Select Compute Offering• CPU & RAM

Select Disk Offering• Volume Size

Select Network Offering• Network & Services

Create VM

Create Custom Virtual Machines via Service Offerings

Dashboard Provides Overview of Consumed Resources

• Running, Stopped & Total VMs

• Public IPs

• Private networks

• Latest Events

Virtual Machine Management

Users

Start

Stop

Restart

Destroy

VM Operations Console Access

• CPU Utilized

• Network Read

• Network Writes

VM StatusChange

Service Offering

2 CPUs

1 GB RAM

20 GB

20 Mbps

4 CPUs

4 GB RAM

200 GB

100 Mbps

Volume & Snapshot Management

Volume

VM 1Add / Delete

Volumes

Schedule Snapshots

Hourly

Daily

Weekly

MonthlyNow

Create Templates from Volumes

Volume Template

View Snapshot History

….

Network & Network Services

• Create Networks and attach VMs

• Acquire public IP address for NAT & load balancing

• Control traffic to VM using ingress and egress firewall rules

• Set up rules to load balance traffic between VMs

Pod 1

….

Cluster N

Access Layer

Host 2

Cluster 1

CloudStack Deployment Architecture

Host 1

Hypervisor is the basic unit of scale.

Cluster consists of one ore more hosts of same hypervisor

All hosts in cluster have access to shared (primary) storage

Pod is one or more clusters, usually with L2 switches.

Availability Zone has one or more pods, has access to secondary storage.

One or more zones represent cloud

PrimaryStorage

Zone 1

….

L3 core

SecondaryStorage

Pod N

CloudStack

Management

Server

Internet

CloudStack Cloud Architecture

- Do Not Distribute

Zone1

Data Center 1

Cloud

Data Center 2

Zone 3

Zone 2

Data Center 2

Zone 3

Zone 2

Data Center 2

Zone 3

Zone 2

Data Center 2

Zone 3

Zone 2

Data Center 2

Zone 3

Zone 2

Data Center 3

Zone 4 CloudStack Cloud can have one or more Availability Zones (AZ).

Management Server Managing Multiple Zones

- Do Not Distribute

Zone1

Data Center 1

Cloud

Data Center 2

Zone 3

Zone 2

Data Center 2

Zone 3

Zone 2

Data Center 2

Zone 3

Zone 2

Data Center 2

Zone 3

Zone 2

Data Center 2

Zone 3

Zone 2

Data Center 3

Zone 4

Management

Server

Single Management Server can manage multiple zones

Zones can be geographically distributed but low latency links are expected for better performance

Single MS node can manage up to 5K hosts.

Multiple MS nodes can be deployed as cluster for scale or redundancy

Management Server Deployment Architecture

- Do Not Distribute

Management

ServerMySQL

DB

Back UpDB

InfrastructureResources

User API

Admin API

Load Balancer

Management

Server

Management

Server

Management

Server

MySQLDB

InfrastructureResources

User API

Admin API

Single-node Deployment

Multi-node Deployment

MS is stateless. MS can be deployed as physical server or VM

Single MS node can manage up to 10K hosts. Multiple nodes can be deployed for scale or redundancy

Commercial: RHEL 5.4+; FOSS: Ubuntu 10.0.4, Fedora 16

Replication

- Do Not Distribute

Pod 1

Host 2

Cluster 1

Host 1PrimaryStorage

L3 switch

SecondaryStorage

L2 switch

CloudStack Storage

• Configured at Cluster-level. Close to hosts for better performance

• Stores all disk volumes for VMs in a cluster

• Cluster can have one or more primary storages

• Local disk, iSCSI, FC or NFS

Primary Storage

• Configured at Zone-level

• Stores all Templates, ISOs and Snapshots

• Zone can have one or more secondary storages

• NFS, OpenStack Swift

Secondary Storage

• Hosts• Servers onto which services will be provisioned

• Primary Storage• VM storage

• Cluster• A grouping of hosts and their associated storage

• Pod• Collection of clusters

• Network• Logical network associated with service offerings

• Secondary Storage• Template, snapshot and ISO storage

• Zone• Collection of pods, network offerings and secondary

storage

• Management Server Farm• Responsible for all management and provisioning tasks

Core CloudStack Components

Zone

CloudStack Pod

Cluster

Host

HostNetwork

PrimaryStorage

VM

VM

CloudStack Pod

ClusterSecondary

Storage

• Primary Storage• Cluster level storage for VMs• Connected directly to hosts• NFS, iSCSI, FC and Local

• Secondary Storage• Zone level storage for template, ISOs and

snapshots• NFS or OpenStack Swift via CloudStack

System VM

• Templates and ISOs• Imported into CloudStack• Can be private or public

Understanding the Role of Storage and Templates

Zone

Secondary Storage

Pod

Cluster

Host

HostPrimary Storage

Template

1. User Requests Instance

2. Provision Optional Network Services

3. Copy instance template from secondary storage to primary storage on appropriate cluster

4. Create any requested data volumes on primary storage for the cluster

5. Create instance

6. Start instance

Provisioning Process

Zone

Secondary Storage

Pod

Cluster

Host

HostPrimary Storage

VM

Template

XenServer Resource Pool

• Integrates directly with XenServer Pool Master

• Snapshots at host level

• System VM control channel at host level

• Network management is host level

Citrix XenServerCloudStack Manager

XenServer Pool Master Host

XenServer Host

XenServer Host

XenServer Host

XenServer Host

• Integrates with ovs-agent




• Does not use OVM Manager

• All templates must be from Oracle

• CloudStack configures ocfs2 nodes

• Requires “helper” cluster• XenServer, KVM or vSphere

Oracle VMCloudStack Manager

OVM Host

OVS Agent

OVM Host

OVS Agent

OVM Host

OVS Agent

OVM Host

OVS Agent

• Integrates with libvirt using Cloud Agent




• Only RHEL 6, not RHEV• Also supports Ubuntu 10.04

RedHat Enterprise Linux (KVM)

KVM Host

Cloud Agent

Libvirt

KVM Host

Cloud Agent

Libvirt

CloudStack Manager

• Integration through vCenter

• System VM control channel via CloudStack private network

• Snapshot and volume management via Secondary Storage VM

• Networking via vSphere vSwitch

VMware vSphereCloudStack Manager

Data Center

vSphere Cluster

vSphere Host

vSphere Host

vSphere Host

vSphere Cluster

vSphere Host

vSphere HostvCenter

Management Server Interaction with Hypervisors

Management Server

XenServer

ESX

vCenter

KVM

Agent

OVM

Agent

XAPI HTTP

• XS 5.6, 5.6FP1, 5.6 SP2, 6.0

• Incremental Snapshots

• VHD

• NFS, iSCSI, FC & Local disk

• Storage over-provisioning: NFS

• ESX 4.1, 5.0 (coming)

• Full Snapshots

• VMDK

• NFS, iSCSI, FC & Local disk

• Storage over-provisioning: NFS, iSCSI

• RHEL 6.0, 6.1, 6.2 (coming)

• Full Snapshots (not live)

• QCOW2

• NFS, iSCSI & FC

• Storage over-provisioning: NFS

• OVM 2.2

• No Snapshots

• RAW

• NFS & iSCSi

• No storage over-provisioning

Multi-tenancy & Account ManagementCloud

• Domain is a unit of isolation that represents a customer org, business unit or a reseller

• Domain can have arbitrary levels of sub-domains

• A Domain can have one or more accounts

• An Account represents one or more users and is the basic unit of isolation

• Admin can limit resources at the Account or Domain levels

Admin

Org A

Admin

Reseller A

Domain

Domain

Admin

Org C

Sub-Domain

User 1

User 2

Group B

Account

Group A

Account

VMs, IPs, Snapshots…

VMs, IPs, Snapshots…

Resources

Resources

Router

L3 Core Switch

Access Layer

Switches

………… …

Availability Zone

Servers

CloudStack Mgmt Server Cluster

Secondary Storage

Pod 1 Pod 2 Pod 3 Pod N

MySQL

Load Balancer

Operations Admin and Cloud API

Users

Physical Network

…

DB Security Group

WebSecurity Group

Layer 3 cloud networking

… …

Web VM

Web VM

Web VM

Web VM

DB VM

Web VM

DB VM

Web VM

Guest Networks with L3 isolationGuest

1 VM 1

Guest 2 VM 1

Guest 1 VM 2

Guest 2 VM 2

Public Internet

10.1.0.1

Public IP address 65.37.141.1165.37.141.2465.37.141.3665.37.141.80

Guest address 10.1.0.2Guest address 10.1.0.3Guest address 10.1.0.4

Guest address 10.1.16.12

Load Balancer

Guest 2 VM 3

Guest 1 VM 3

Guest 1 VM 4


L3 Core Switch

Pod 1 L2 Switch

Pod 3 L2 Switch

10.1.16.1

…

…10.1.8.1Pod 2 L2 Switch

Hypervisor 1

Hypervisor N

Hypervisor 8

Access Switch(es)

VM Traffic

…

Pod K

CLUSTER 1

…

CLUSTER 4

Core (L3) Network

…

Pod M Pod N

Virtual Networks (L2 isolation)

Hypervisor N+1

Public Traffic

Hypervisor

R

R V

VV

V

HypervisorV V

V

RTenant VMTenant Virtual Router

Guest virtual layer-2 network

Guest 1 VM 1

Guest 1 VM 2

Guest 1 VM 3

Guest 1 VM 4

Public Internet

Public Network

Guest Virtual Network 10.1.1.0/24

Gateway address 10.1.1.1

NATDHCPLoad BalancingVPN

Public IP address 65.37.141.1165.37.141.36

Guest address 10.1.1.2Guest address 10.1.1.3Guest address 10.1.1.4Guest address 10.1.1.5

Guest 1 Virtual Router

Guest 2 VM 1

Guest 2 VM 2

Guest 2 VM 3

Guest Virtual Network 10.1.1.0/24


NATDHCPLoad BalancingVPN


Guest 2 Virtual Router

Public IP address 65.37.141.2465.37.141.80

Layer-2 Guest Virtual Network

Public Network/Internet

Guest Virtual Network 10.1.1.1/8VLAN 100


DHCP, DNSNATLoad BalancingVPN

Public IP 65.37.141.11

10.1.1.1Guest VM 1

10.1.1.3Guest VM 2

10.1.1.4Guest VM 3

10.1.1.5Guest VM 4

CSVirtual Router


Guest Virtual Network 10.1.1.1/8VLAN 100

Private IP10.1.1.112

DHCP, DNS

Public IP 65.37.141.112

10.1.1.1Guest VM 1

10.1.1.3Guest VM 2

10.1.1.4Guest VM 3

10.1.1.5Guest VM 4

NetScalerLoad

Blancer

Private IP10.1.1.111

Public IP 65.37.141.111

Juniper SRX

Firewall

CS Virtual Router provides Network Services External Devices provide Network Services

CSVirtual Router

Layer-3 Guest Network

Public Network65.11.0.0/16

65.11.1.2Guest VM

1

Guest VM 2

Guest VM 3

Guest VM 4


NetScaler

Load Blancer

Network Services Managed Externally Network Services Managed by CS

65.11.1.3

65.11.1.4

65.11.1.5

DHCP, DNS

CSVirtual Router

Security Group 1

Security Group 2

10.1.2.3Guest VM

1

Guest VM 2

Guest VM 3

Guest VM 4

10.2.12.4

10.5.2.99

10.1.2.18

DHCP, DNS

CSVirtual Router

Security Group 1

Security Group 2

EIP, ELB

65.11.1.265.11.1.3

65.11.1.4

L3 switch

• Cloud provider defines the feature set for guest networks

• Toggle features or service levels– Security groups on/off– Load balancer on/off– Load balancer software/hardware– VPN, firewall, port forwarding

• User chooses network offering when creating network

• Enables upgrade between network offerings

• Default offerings built-in– For classic CloudStack networking

Network Offerings

CloudStack System VMs

• System VMs optimize and scale the datapath on behalf of CloudStack– Stateless, can be destroyed and recreated from database state– Highly Available– Communicates with Management Server over management network– Usually have 3 interfaces: control, guest and public

• Console Proxy VM – Provides AJAX-style HTTP-only console viewer– Grabs VNC output from hypervisor– Scales out (more spawned) as load increases– Java-based server Communicates with MS over message bus

• Secondary Storage VM– Provides image (template) management services– Download from HTTP file share or Swift– Copy between zones– Scale out to handle multiple NFS mounts– Java-based server communicates with MS over message bus

CloudStack System VMs

• Virtual Router VM – Provides multiple network services– IPAM (DHCP), DNS, NAT, Source NAT, Firewall, PF, VPN– User-data, Meta-data, SSH keys and password change server– Redundancy via VRRP– MS configures VR over SSH

• Proxied via the hypervisor on XS and KVM

Virtual Router Information (applies to all Sys. VMs)

• Debian 6.0 ("Squeeze"), 2.6.32 kernel with the latest security patches from the Debian security APT repository. No extraneous accounts

• 32-bit for enhanced performance on Xen/VMWare• Only essential software packages are installed. Services such as, printing, ftp, telnet, X, kudzu,

dns, sendmail are not installed.• SSHd only listens on the private/link-local interface. SSH port has been changed to a non-

standard port (3922). SSH logins only using keys (keys are generated at install time and are unique for every customer)

• pvops kernel with Xen paravirt drivers + KVM virtio drivers + VMware tools for optimum performance on all hypervisors. Xen tools inclusion allows performance monitoring

• Template is built from scratch and is not polluted with any old logs or history• Latest versions of haproxy, iptables, ipsec, apache from debian repository ensures improved

security and speed• Latest version of jre from Sun/Oracle ensures improved security and speed

System VM contd

• SSH keys and password are unique to cloud installation

• Code can be patched by restarting system vm– Mounts a special ISO file with latest code at boot– If ISO contents differ, patch and reboot

• Same system vm works on XS, KVM, VMWare– Bootstrap step for the cloud is to install the template

for this system vm• Ready to be re-purposed for other specialized tasks

Service Management (Billing, Metering, Accounts, etc.)Service Management (Billing, Metering, Accounts, etc.)

Resource ManagementResource Management Dynamic Workload ManagementDynamic Workload Management

Availability and SecurityAvailability and Security

BackupBackup Load BalancingLoad Balancing High AvailabilityHigh Availability MonitoringMonitoring

Image LibrariesImage Libraries

Application CatalogApplication Catalog

Custom TemplatesCustom Templates

Operating System ISOsOperating System ISOs

Inte

grati

on A

PIIn

tegr

ation

API

Ope

ratio

n, A

dmin

istra

tion,

M

aint

enan

ce a

nd P

rovi

sion

ing User InterfaceUser Interface

AdministratorAdministrator End UserEnd User ConsoleConsole

Developer APIDeveloper API

AmazonAmazon OpenStackOpenStack CustomCustom

Architecture Components

Virtualization Layer

Compute Network Storage

Interactions

CloudStack

Cloud user{API client (Fog/etc)}

End User UI

Admin UI

MySQL

CloudStackClustered

CloudStackManagement

Server

Domain Admin

UI

CS Admin & End-user API

Cloud user{ec2 API client }

ec2 API

Monitoring CS API vSphere ClusterPrimaryStorage

vcenter

Cluster Mgmt

XS ClusterPrimaryStorage

vCenter API

XAPI

KVM ClusterPrimaryStorageJSON

OVM Cluster PrimaryStorage

XenApi

NetConf

Nitro APIJuniper SRX

Netscaler

Console Proxy VMConsole

Proxy VM

JSON

Cloud user

HTTPSAjax Console

VNC

Sec. StorageVM

NFS Server

NFSSec. Storage

VM

HTTP (Template Download)

HTTP (Template Copy)

HTTP (Swift)

NFS

Router VMRouter VM

Router VM

JSON

{Proxied} SSH

Inside a Management Server

APIServlet

AsyncJob

QueueMgr

CloudStackAPI

ServicesAPI

Commands

Responses

cmd.execute()

Kernel

PluginsPlugins

Plugins

Message Bus

Agent Manager

ResourcesAgentAPI(Cmds)

HypervisorNativeAPIs

LocalOrRemote

NetworkDeviceAPI

MySQL

cloudstack architecture

Technology