Download - Apache CloudStack Architecture by Alex Huang
Apache CloudStackArchitecture
Alex HuangSoftware Architect, Citrix Systems
Deployment Architecture
• Hosts• Servers onto which services will be provisioned
• Primary Storage• VM disk storage
• Cluster• A grouping of hosts and their associated storage
• Pod• Collection of clusters in the same failure boundary
• Network• Logical network associated with service offerings
• Secondary Storage• Template, snapshot and ISO storage
• Zone• Collection of pods, network offerings and
secondary storage
• Management Server Farm• Management and provisioning tasks
Components
Zone
CloudStack Pod
Cluster
Host
HostNetwork
PrimaryStorage
VM
VM
CloudStack Pod
ClusterSecondary
Storage
Pod 1
Host 2
Cluster 1
Host 1
PrimaryStorage
L3 switch
SecondaryStorage
L2 switch
Two Types of Storage
• Stores disk volumes for VMs in a cluster• Configured at Cluster-level. • Close to hosts for better performance• Cluster have at least one primary storage• Requires high IOPs (can be expensive)
Primary Storage
• Stores all Templates, ISOs and Snapshots• Configured at Zone-level• Zone can have one or more secondary
storages• High capacity, low cost commodity
storage
Secondary Storage
Pod 1
….
Cluster N
L2
Host 2
Cluster 1
Deployment Architecture
Host 1
Hypervisor is the basic unit of scale.
Cluster consists of one ore more hosts of same hypervisor
All hosts in cluster have access to shared (primary) storage
Pod is one or more clusters, usually with L2 switches.
Availability Zone has one or more pods, has access to secondary storage.
One or more zones represent cloud
PrimaryStorage
Zone 1
….
L3
SecondaryStorage
Pod N
Management Server Cluster
Internet
Management Server Cluster
Replica
InfrastructureResources
User API
Admin API
Load Balancer
Management Server
Management Server
MySQL
MS is stateless. MS can be deployed as physical server or VM
Single MS node can manage up to 10K hosts. Multiple nodes can be deployed for scale or redundancy
RHEL 5.4+, Ubuntu 10.0.4, Fedora 16
Replication
Managing Complexity
The Three C’s of Complexity
• Control• Choice• Compliance
Compute
Giving Control Brings Complexity
Network Storage
Admin
Users
Org A
Admin
Users
Org BUsers
End User
AdminVM
Ware
XenServer
KVM
NFS
iSCSI
FC
NetScaler
F5
Jun. SRX
Local DiskCisco ASA
Swift
HDFS
Hyper-V
• ACL• Limits• Governance
OracleVM
BareMetal
Guest Virtual Layer-2 NetworkGuest 1
VM 1
Guest 1 VM 2
Guest 1 VM 3
Guest 1 Virtual Network 10.1.1.0/24
Gateway 10.1.1.1
Guest 10.1.1.2
Guest 10.1.1.3
Guest 10.1.1.4
Guest 1 Virtual Router
Guest 2 VM 1
Guest 2 VM 2
Guest 2 VM 3
Guest 2 Virtual Network 10.1.1.0/24
Gateway 10.1.1.1
Guest 10.1.1.2
Guest 10.1.1.3
Guest 10.1.1.4
Guest 2 Virtual Router
Public IP 65.37.141.2465.37.141.80
Public IP 65.37.141.1165.37.141.36
Internet
Multi-tier Network
Private IP10.1.1.112
DHCP, DNSUser-data
Public IP 65.37.141.112
10.1.1.1Web VM
1
10.1.1.3Web VM
2
10.1.1.4Web VM
3
10.1.1.5Web VM
4
NetscalerLoad
Balancer
Private IP10.1.1.111
Public IP 65.37.141.111
Juniper SRX
Firewall
Virtual Router
Virtual Network 10.1.1.0/24VLAN 100
Virtual Network 10.1.2.0/24VLAN 1001
10.1.2.21
10.1.2.18
10.1.2.38
10.1.2.39
10.1.2.31App VM
1 10.1.3.21
Virtual Network 10.1.3.0/24VLAN 141
10.1.2.24App VM
2 10.1.3.45
10.1.3.24 DB VM 1
DHCP, DNS, User-data
DHCP, DNSUser-data,Source-NAT, VPN Public IP
65.37.141.115
Virtual Router
Virtual Router
Unified Multi-tier Network
10.1.1.1Web VM 1
10.1.1.3Web VM 2
10.1.1.4Web VM 3
10.1.1.5Web VM 4
Virtual Network 10.1.1.0/24VLAN 100
Virtual Network 10.1.2.0/24VLAN 1001
10.1.2.31App
VM 1
Virtual Network 10.1.3.0/24VLAN 141
10.1.2.24App
VM 2
10.1.3.24DB VM
1
Virtual Router Customer
Premises
IPSec or SSL site-to-site VPN
Internet
Monitoring VLAN
Virtual Router Services• IPAM• DNS• LB [intra]• S-2-S VPN• Static Routes• ACLs• NAT, PF• FW [ingress & egress]• BGP
Load Balancer
Other Topologies
Guest Virtual Network 10.1.1.0/24VLAN 100
Gateway address 10.1.1.1
10.1.1.1Guest VM 1
10.1.1.3Guest VM 2
10.1.1.4Guest VM 3
10.1.1.5Guest VM 4
Guest Virtual Network 10.1.1.0/24VLAN 100
DHCP, DNSUser-data
10.1.1.1Guest VM 1
10.1.1.3Guest VM 2
10.1.1.4Guest VM 3
10.1.1.5Guest VM 4
No services [Static IPs] Dedicated VLAN with DHCP and DNSUser can request specific IP[s] for NIC
Core switch
Gateway address 10.1.1.1
Core switch
Virtual Router
Other Topologies
Guest Virtual Network 10.1.1.0/24VLAN 100
Gateway address 10.1.1.1
10.1.1.100Guest VM 1
10.1.1.200Guest VM 2
10.1.1.101Guest VM 3
10.1.1.115
Guest VM 4
Guest Virtual Network 10.1.1.0/24VLAN 100
DHCP, DNSUser-data
10.1.1.1Guest VM 1
10.1.1.3Guest VM 2
10.1.1.4Guest VM 3
10.1.1.5Guest VM 4
MPLS Use Case Shared VLAN with DHCP and DNS
CSVirtual Router
Core switch
Gateway address 10.1.1.1
Core switch
MPLS VLAN 100
DHCP, DNSUser-data
CSVirtual Router
…
DB Security Group
WebSecurity Group
Layer 3 Networking (Amazon Style)
… …
Web VM
Web VM
Web VM
Web VM
DB VM
Web VM
DB VM
Web VM
Software Architecture
Management Server
Orchestration Engine- Drives long running VM
operations- Syncs between resources
managed and DB- Generates events
Resource Management
Cluster Management
JobManagement
DB
UI Cloud Portal CLI
Other Clients
Deployment Planning
Network Gurus
Network Elements
Hypervisor Gurus
DatabaseAccess
Alert & EventManagement
Plug
in A
PI
Resource APIHypervisor Resources
Network Resources
Storage Resources
ImageResources
SnapshotResources
REST API
OAM&P API End User API EC2 API Pluggable Service API EngineOther APIs
Security Adapters
Account Management Connectors
ACL & Authentication- Accounts, Domains, and Projects- ACL, limits checking
Services API
Serv
ices
API
Console Proxy Management
Template Access
HA
Usage Calculations
Additional Services
Event BusMessage Bus
Usage Server
Orchestration Engine
• Understands how to orchestrate long running processes (i.e. VM starts, Snapshot copies, Template propagation)
• Well defined process steps• Calls Plugin API to execute functionalities that
it needs
Plugins
• Various ways to add more capability to CloudStack
• Implements clearly defined interfaces• All operations must be idempotent• All calls are at transaction boundaries• Compiles only against the Plugin API module
Anatomy of a Plugin
ServerResource- Optional. Required if Plugin needs to be co-
located with the resource- Implements translation layer to talk to
resource- Communicates with server component via
JSON
Rest API- Optional. Required only if needs to expose
configuration API to admin.
Plug
in A
PI
Data Access Layer
Implementation
• Can be two jars: server component to be deployed on management server and an optional ServerResource component to be deployed co-located with the resource
• Server component can implement multiple Plugin APIs to add its feature
• Can expose its own API through Pluggable Service so administrators can configure the plugin
• As an example, OVS plugin actually implements both NetworkGuru and NetworkElement
Plugin Interfaces Available• NetworkGuru – Implements various network isolation and ip address
technologies• NetworkElement – Facilitate network services on network elements
to support a VM (i.e. DNS, DHCP, LB, VPN, Port Forwarding, etc)• DeploymentPlanner – Different algorithms to place a VM and
volumes.• Investigator – Ways to find out if a host is down or VM is down.• Fencer – Ways to fence off a VM if the state is unknown• UserAuthenticator – Methods of authenticating a user• SecurityChecker – ACL access• HostAllocator – Provides different ways to allocate host• StoragePoolAllocator – Provides different ways to allocate volumes
Separating Data and Control
Data Center 1
Cloud
Data Center 2
Data Center 3
Management
Server
Management Servers control all resources, both virtual and physical
SSVMs deployed to transfer data between zones
CPVMs deployed to transfer VNC console traffic
VR deployed for traffic into public internet
Management Server is never in the data path
SSVM
SSVM
SSVMTransfer of Templates,
ISOs, Snapshots
CPVMCPVM
CPVM
VR
VR
VR
Internet
Kernel
Sequence Flow for VM CreationEnd User Rest API
SecurityCheckers
User VM Mgr
Network Mgr
Storage MgrJob
SchedulingVirtualMachine Mgr
Network Guru
Deploy VM
ACL Checks
Allocate Entity in CS
Allocate VM
Allocate NIC
Allocate Volume
Allocate IP
Schedules Deploy Job
Returns with job id, VM id
Query Job Result
Returns with job status
Sequence Flow for VM CreationJob Threads
Network Element
User VM Mgr
Network Mgr
Storage Mgr
VirtualMachine Mgr
Network Guru
Start VM
Start VM
Prepare Nics
Notify that Nic is about to be started in network
Reserve resources for Nic
Services APIServer
Resources
Start User VM
Agent Calls
Prepare Volumes
Template Mgr
Deployment
Planner
Get a Deployment Plan (Host and StoragePool)
Prepare template on Primary Storage
Agent Calls
Agent Start VM Call
Stores job result
Conclusion
Design Goals for CloudStack
• Design for complexity– Clear interfaces
• Design for scalability– Separate out data path and control paths– Design to maximize the use of database connections
• Design against failure– Provide clear boundaries (process and compilation)– Utilize cloud administrator to give guidance
More Information
27
• http://cloudstack.org
• Apache mailing lists–[email protected] –[email protected]
• Thank you