Download - Issues of OpenStack multi-region mode
HUAWEI TECHNOLOGIES CO., LTD. Page 1Huawei Confidential
Issues ofOpenStack multi-region modeChaoyi Huang ( [email protected] )
Last edited May 28, 2015
HUAWEI TECHNOLOGIES CO., LTD. Page 2Huawei Confidential
Nova
Cinder
OpenStackRegion1
GlanceNeutron
Ceilometer
KeyStone
Horizon
Nova
Cinder
OpenStackRegion2
GlanceNeutron
Ceilometer
API Req. API Req.
Token Req.
OpenStack mutli-region mode: each site will be deployed with independent OpenStack instance (including Nova,Cinder,Neutron, Ceilometer, Glance, Glance sometimes will be shared for multi-site). KeyStone service is shared for same credential to access multi-region, and Horizon functions as the unified portal for the multi-region cloud.
Site1 Site2
HUAWEI TECHNOLOGIES CO., LTD. Page 3Huawei Confidential
Quotas, tenants: Missing global view across regions Global quota issue including quotas for Nova, Cinder, Neutron... Issue for soft/hard quotas based on current usage for all region for this tenant Issue for each of the individual projects managing quotas provide a global view
Issues from production multi-site / multi-region cloud:Vancouver session moderated by Comcast: https://etherpad.openstack.org/p/YVR-ops-multi-siteOPNFV multisite project: https://etherpad.opnfv.org/p/multisite_usecase_collection
vApp 5
vApp 1vApp 2
vApp 6
OpenStackvApp 8
vApp 4vApp 3
vApp 7OpenStack
Nova Quota Cinder Quota Neutron Quota Nova Quota Cinder Quota Neutron Quota
Issue: discrete quota control
HUAWEI TECHNOLOGIES CO., LTD. Page 4Huawei Confidential
Resource utilization across regions: Missing global view across
regions Issue for a user being able to management all his virtual resources from one centralized management
interface, at least to have a summarized view of the total resource capacity and the live utilization of his
virtual resources, for example
Issues from production multi-site / multi-region cloud:Vancouver session moderated by Comcast: https://etherpad.openstack.org/p/YVR-ops-multi-siteOPNFV multisite project: https://etherpad.opnfv.org/p/multisite_usecase_collection
vApp 5
vApp 1vApp 2
vApp 6
OpenStackvApp 8
vApp 4vApp 3
vApp 7OpenStack
VM… Volume… Network, Port… VM… Volume… Network, Port…
Issue: discrete resource utilization
HUAWEI TECHNOLOGIES CO., LTD. Page 5Huawei Confidential
Missing global view for tenant level metering data. Issue for a user being able to access all the logs and indicators produced by multiple
OpenStack instances, in a centralized way
Issues from production multi-site / multi-region cloud:Vancouver session moderated by Comcast: https://etherpad.openstack.org/p/YVR-ops-multi-siteOPNFV multisite project: https://etherpad.opnfv.org/p/multisite_usecase_collection
Ceilometer Ceilometer
Metering…Sample…
Issue: discrete metering data
Metering…Sample…
HUAWEI TECHNOLOGIES CO., LTD. Page 6Huawei Confidential
Neutron: No global view for tenant level IP address space management Issue of clone security groups across regions Issue of establishing tenant level L2/L3 networking across region for application high availability across
OpenStack instances
Issues from production multi-site / multi-region cloud:Vancouver session moderated by Comcast: https://etherpad.openstack.org/p/YVR-ops-multi-siteOPNFV multisite project: https://etherpad.opnfv.org/p/multisite_usecase_collection
Neutron vApp Standby
Neutron
Network CIDR… Network CIDR…
Security Group… Security Group…
APP networking across OpenStack instance…For heartbeat, states replication….vApp
master
HUAWEI TECHNOLOGIES CO., LTD. Page 7Huawei Confidential
Nova: Lack of clone ssh keys from one region to another
Issues from production multi-site / multi-region cloud:Vancouver session moderated by Comcast: https://etherpad.openstack.org/p/YVR-ops-multi-siteOPNFV multisite project: https://etherpad.opnfv.org/p/multisite_usecase_collection
Nova Nova
KeyPair… KeyPair…
HUAWEI TECHNOLOGIES CO., LTD. Page 8Huawei Confidential
Glance: Lack of replicate images across regions
Issues from production multi-site / multi-region cloud:Vancouver session moderated by Comcast: https://etherpad.openstack.org/p/YVR-ops-multi-siteOPNFV multisite project: https://etherpad.opnfv.org/p/multisite_usecase_collection
Glance Glance
Upload image… Upload Imgae…
HUAWEI TECHNOLOGIES CO., LTD. Page 9Huawei Confidential
Cinder: Lack the function like volume replication across regions for Geo-site
Redundancy/Disaster Recovery
Issues from production multi-site cloud:Vancouver session moderated by Comcast: https://etherpad.openstack.org/p/YVR-ops-multi-siteOPNFV multisite project: https://etherpad.opnfv.org/p/multisite_usecase_collection
Cinder CinderVolume
Volume replication for Geo-site Redundancy/Disaster Recovery ….
Volume
HUAWEI TECHNOLOGIES CO., LTD. Page 10Huawei Confidential
Cascading OpenStack
…
Tenant 1 Virtual Resources
Tenant 2 Virtual Resources
Tenant x Virtual Resources
Cascaded OpenStack 2Cascaded OpenStack 1 Cascaded OpenStack y
OpenStack API OpenStack API OpenStack API
OpenStack cascading solution – best matched multisite solution: Integrate multiple OpenStack instances from multi-site into one unified cloud for tenant’s resources distributionhttps://wiki.openstack.org/wiki/OpenStack_cascading_solution
Tenant 2Tenant 1 Tenant x
OpenStack API
• global view of tenant quota• global view of tenant resource utilization• global view of tenant metering data• tenant level global IP address space
management• clone security groups across regions• L2/L3 networking across regions• KeyPair replication• Replicate images• Volume replication• …..
Service Orchestrator x
Service Management
Service Orchestrator 1
…
HUAWEI TECHNOLOGIES CO., LTD. Page 11Huawei Confidential
Cascading OpenStack
…
Tenant 1 Virtual Resources
Tenant 2 Virtual Resources
Tenant x Virtual Resources
Cascaded OpenStack 2Cascaded OpenStack 1 Cascaded OpenStack y
OpenStack API OpenStack APIOpenStack API
Service Orchestrator x
Service Management
Service Orchestrator 1
OpenStack API OpenStack APIOpenStack API
… …
Cloud over clouds by OpenStack cascading: the cascaded OpenStack instance even can be running inside Amazon AWS or Windows Azure, and then being integrated into a cloud through the cascading OpenStack. This is cloud over clouds.
HUAWEI TECHNOLOGIES CO., LTD. Page 12Huawei Confidential
Service Orchestrator x
Service Management
Service Orchestrator 1
CascadingOpenStack(Tenant 1)
…CascadingOpenStack(Tenant 2)
CascadingOpenStack(Tenant x)
…
Tenant 1 Virtual Resources
Tenant 2 Virtual Resources
Tenant x Virtual Resources
Cascaded OpenStack 2Cascaded OpenStack 1 Cascaded OpenStack y
https://tenant1.OpenStack/ https://tenant2.OpenStack/ https://tenantx.OpenStack/
OpenStack API
OpenStack API
OpenStack API
OpenStack APIOpenStack API OpenStack API OpenStack API
OpenStack APIOpenStack API
Fully distributed cascading layer: cascading OpenStack layer can be fully distributed if each small piece of cascading OpenStack only serve limited tenants. Each tenant can still has globe view for his distributed resources. Unlimited scalability can be achieved through this horizontal split.
… …
HUAWEI TECHNOLOGIES CO., LTD. Page 13Huawei Confidential
How to do that: just adding driver/agent/plugin each service to treat underneath OpenStack instance as its own backend. The cascading OpenStack becomes control layer only
Neutron Server
OVSAgent
Cinder API
Rabbit-MQ
Cinder Volume
Cinder Volume
Nova API
RabbitMQ
Nova Compute
Nova Compute
Libvirt Driver
Nova Driver
LVM Driver
Cinder Driver
KVM Nova LVM Cinder
RabbitMQ
OVS
L3Agent
Linux Router
Scheduler Scheduler
Neutron Agent
Neutron
Magic happens by just considering OpenStack as its own backend !
Glance Image1: Loc1: NFS Loc2: Glance1 Loc3:Glance2
Glance1
Glance2
Image1: Loc1: Ceph
Image1: Loc1: Ceph
Ceilometer API
hBase
StoreStore
CeilometerCeilometer
Nova as hypervisor Cinder as block storage Neutron as networking device Glance as Img. Location Ceilometer as store
** Architecture simplified for illustration only
HUAWEI TECHNOLOGIES CO., LTD. Page 14Huawei Confidential
Also solve OpenStack scalability at least 10 times level
OpenStack …
…
1 2 100
1 2 1000
OpenStack
…1 2 1000
OpenStack
…1 2 1000
OpenStack
Max. 100 data centers
Max. 100k physical server
nodes
Max. 1 million VMs
OpenStack API OpenStack API OpenStack API
OpenStack API
Scalability inside one data center,
multi-data centers or multi-sites
Test report: http://www.slideshare.net/JoeHuang7/test-report-for-open-stack-cascading-solution-to-support-1-million-v-ms-in-100-data-centers
HUAWEI TECHNOLOGIES CO., LTD. Page 15Huawei Confidential
More information:
Wiki: https://wiki.openstack.org/wiki/OpenStack_cascading_solutionPoC Source Code: https://github.com/stackforge/tricircle