1 million vms in 100 datacenters by openstack cascading

HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential

1 million VMs in 100 datacenters by OpenStack cascading

Chaoyi Huang ( [email protected] ) Last edited May 18, 2015


• Background – what is OpenStack cascading• Semi-Simulation test for 1 million VMs in 100 datacenters• Evolution of OpenStack Cascading


DC 1

OpenStack(Vendor1 /

Version 2.0)DC3

DC 2

OpenStack(Vendor2 /

Version 2.1)

OpenStack(Vendor3 /

Version 2.1)

OpenStack cascading is “OpenStack orchestrate OpenStacks” solution for multi-site cloud with unified global OpenStack API exposed

OpenStack API

OpenStack

OpenStack cascading solution

OpenStack API

OpenStack API

OpenStack API


Neutron Server

OVSAgent

Cinder API

Rabbit-MQ

Cinder Volume

Cinder Volume

Nova API

RabbitMQ

Nova Compute

Nova Compute

Libvirt Driver

Nova Driver

LVM Driver

Cinder Driver

KVM Nova LVM Cinder

RabbitMQ

OVS

L3Agent

Linux Router

Scheduler Scheduler

Neutron Agent

Neutron

Magic happens by just considering OpenStack as its own backend !

Glance Image1: Loc1: NFS Loc2: Glance1 Loc3:Glance2

Glance1

Glance2

Image1: Loc1: Ceph

Image1: Loc1: Ceph

Ceilometer API

hBase

StoreStore

CeilometerCeilometer

Nova as hypervisor Cinder as block storage Neutron as networking device Glance as Img. Location Ceilometer as store

** Architecture simplified for illustration only


Neutron

ServerCinder API

Rabbit-MQ

Cinder Proxy

Nova API

RabbitMQ

Nova Proxy

Nova Cinder

RabbitMQ

Scheduler Scheduler

Neutron Proxy(L2/L3/LB/VPN/

FW)

Neutron

The driver/agent to OpenStack is called “Proxy”

Glance

Glance1

Glance2

Ceilometer API

Ceilometer-Proxy

CeilometerCeilometer

Replic-Manager

CascadingLayer

(Control layer, no resources)

CascadedLayer

(All VMs/Volume/…resources running

in the cascaded layer )

Components introduced for cascading

*KeyStone is global service shared or federated by cascading and cascaded layer* Heat will use OpenStack API to do orchestration, no cascading required.


Scalability verification

OpenStack …

…

1 2 100

1 2 1000

OpenStack

…1 2 1000

OpenStack

…1 2 1000

OpenStack

Max. 100 data centers

Max. 100k physical server

nodes

Max. 1 million VMs

OpenStack API OpenStack API OpenStack API

OpenStack API

Scalability inside one data center,

multi-data centers or multi-sites

Test report: http://www.slideshare.net/JoeHuang7/test-report-for-open-stack-cascading-solution-to-support-1-million-v-ms-in-100-data-centers

http://www.slideshare.net/JoeHuang7/test-report-for-open-stack-cascading-solution-to-support-1-million-v-ms-in-100-data-centers


Test framework 1. One real cascading OpenStack with 100 simulated cascaded OpenStacks

2. Separated RabbitMQ / MySQL for Each service

3. Active/Standby mode for RabbitMQ / MySQL, not using clustering.

4. Each service with multiple API servers with HAproxy as the load balancer.

5. each cascaded OpenStack managed by one Proxy Node in the cascading OpenStack

6. one cascaded OpenStack configured with one AZ.

7. Most of proxy node and cascaded OpenStack simulator running inside VM

8. Basic assumption: one cascaded OpenStack can manage 1k physical hosts and 10k VMs

Traffic generator: SoapUI/Shell Script


Tested system on limited hardware resources

The semi-simulation test using limited hardware resources (45 servers):Huawei E6000 (27 servers) 2 X Intel(R) Xeon(R) CPU E5645 @ 2.40 GHz

Huawei X6000 (5 servers) 2 X Intel(R) Xeon(R) CPU E5645 @ 2.40 GHz

Huawei RH2288 (12 servers) 2 X Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10 GHz

Huawei RH2285 (1 server) 2 X Intel(R) Xeon(R) CPU E5620 @ 2.40 GHz

Controller nodes and compute nodes use 1GE network interface cards (NICs).

S5352 S5352

E6000

……

X6000 RH2288 RH2285


Load generator

HAProxy Keystone Glance

1 x E6000 1 x RH2285 1 x E6000+ 1 x RH2285

1 x RH2288

Resources allocation

Nova API/nova-conductor

Nova scheduler

Nova RabbitMQ

Nova DB

4 x E6000 +1 x RH2288

2 x E6000 1 x X6000 1 x RH2288

Cinder API CinderRabbitMQ Cinder DB

1 x RH2288 1 x X6000 1 x RH2288

Neutron API Neutron RabbitMQ Neutron DB

5 x E6000 +1 x RH2288

1x X6000 1xRH2288

Proxy Node Cascaded OpenStack Simulator

2 x RH2288 + 2 x E6000+ 2 x X6000 2 x RH2288 + 9 x E6000

1. 4xE6000 Nova-API server not enough, adding one RH2288 during the test, each Nova-API 8 workers, each nova-conductor 20 workers

2. Nova-Scheuduler enhanced to multi-workers. Each one with 18 workers

1. 5xE6000 Neutron-API server not enough, adding one RH2288 during the test

2. Each Neutron server 12 RPC workers and 12 API workers

Cinder is not the bottleneck


Feature tested:

1. Nova Cascading: VM boot, delete, start, stop, reboot, query2. Cinder Cascading: Volume create, delete.3. Neutron Cascading: L2 network (VxLAN), Router (DVR)4. Glance cascading: Glance as the image location5. Shared KeyStone: using PKI token

*Security Group/FWaaS/etc Neutron feature not implemented with cascading


Part of the test casesBackground system: Mixed size tenants with L2/L3 networks scenario:

1. Small-scale tenants (4800): Each tenant has 4 networks and each network provide services for 10 instances. There are altogether 192,000 instances. 100 small-scale tenants across 4 cascaded OpenStacks, others only in one cascaded OpenStacks.

2. Medium-scale tenants (150): Each tenant has 20 networks and each network provide services for 100 instances. There are altogether 300,000 instances. All across 5 cascaded OpenStacks

3. Large-scale tenants (50): Each tenant has 40 networks and each network provide services for 250 instances. There are altogether 500,000 instances. All across 10 cascaded OpenStacks


Part of the test cases

1. Concurrently create 500 virtual machines to networks which already have interface on the router

• Test cases were conducted under the system had been populated with 1 million VMs/Ports• Neutron has the most load pressure. All API requests finished successfully. • If higher concurrency required, more resources needed.



2. Execute 500 concurrently API request for add_ router_ interfaces for networks with instances.




3. Execute 500 concurrently API request for remove _router _interface for networks with instances.



Test Conclusion

Conclusion:500 API request concurrency 1 million VMs / 1 million Ports100 simulated cascaded OpenStack

can be in up to 100 data centerswith 100 k physical hosts

work on the limited hardware resources

If security_group, FWaaS, etc feature cascading implemented, need more resources for Neutron to achieve 500 API request concurrency


Issues found in the test

Refer to the test report: http://www.slideshare.net/JoeHuang7/test-report-for-open-stack-cascading-solution-to-support-1-million-v-ms-in-100-data-centers




Evolve to unlimited scalability

CascadingOpenStack(Tenant 1)

…CascadingOpenStack(Tenant 2)

CascadingOpenStack(Tenant x)

…

Tenant 1 Virtual Resources

Tenant 2 Virtual Resources

Tenant x Virtual Resources

Cascaded OpenStack 2Cascaded OpenStack 1 Cascaded OpenStack y

https://tenant1.OpenStack/ https://tenant2.OpenStack/ https://tenantx.OpenStack/

OpenStack API

OpenStack API

OpenStack API

OpenStack APIOpenStack API OpenStack API OpenStack API

OpenStack APIOpenStack API

OpenStack API

OpenStack API

1. Fully distributed, no central point at all, no scalability bottleneck.2. Unlimited OpenStack instances pool in one cloud or federated clouds. 3. Provide tenant with seamless one OpenStack experience no matter how many OpenStack instances behind


More information:

Wiki: https://wiki.openstack.org/wiki/OpenStack_cascading_solutionPoC Source Code: https://github.com/stackforge/tricircle

https://wiki.openstack.org/wiki/OpenStack_cascading_solution

https://github.com/stackforge/tricircle

1 million vms in 100 datacenters by openstack cascading

Software

openstack cascading

real cascading openstack

huawei technologies

openstack vendor2 version

openstack vendor1 version

openstack vendor3 version

huawei confidential

huawei confidential