Contents1 Overview.............................................................................................................................................3

1.1 Executive Summary.....................................................................................................................3

1.2 Requirements..............................................................................................................................3

1.3 Constraints...................................................................................................................................4

1.4 Assumptions................................................................................................................................4

1.5 Risks.............................................................................................................................................4

1.6 Author’s Note..............................................................................................................................4

1.7 Dedication....................................................................................................................................5

2

Management Cluster...................................................................................................................................6

2.1 Physical Host Design....................................................................................................................6

2.2 vSphere Design............................................................................................................................6

2.3 Storage.........................................................................................................................................6

2.4 Networking..................................................................................................................................6

2.5 Virtual Machines..........................................................................................................................6

3 vSphere Fabric Cluster.........................................................................................................................6

3.1 Nested Host Design......................................................................................................................7

3.2 Storage.........................................................................................................................................7

3.3 Networking..................................................................................................................................7

4 OpenStack Architecture.......................................................................................................................7

4.1 Controller.....................................................................................................................................7

4.1.1 Identify Management: Keystone..........................................................................................7

4.1.2 Image Management: Glance................................................................................................8

4.1.3 Dashboard: Horizon.............................................................................................................8

4.1.4 Compute Manager: Nova.....................................................................................................8

4.2 Network Manager........................................................................................................................8

4.3 ....Cinder Storage

Node .............8

4.4 Compute Node 1 .............8

Virtual Design Master Season 2

Deploying Windows and Linux Images to Heterogeneous Hypervisors with OpenStack

Challenge 3 Submission: Stacking the Odds

Byron Schaller7-29-2014

4.5 Compute Node 2..........................................................................................................................8

5 Networking..........................................................................................................................................8

5.1 Neutron with OVS GRE Overlay...................................................................................................8

5.2 vSphere: The Workaround...........................................................................................................9

5.3 vSphere: The Right Way.............................................................................................................10

6 Image Design.....................................................................................................................................10

6.1 Linux on KVM.............................................................................................................................10

6.2 Windows on KVM......................................................................................................................10

6.3 VMDK Images............................................................................................................................10

6.4 Thoughts on Image Creation and Deployment..........................................................................10

1 Overview1.1 Executive Summary“Turtles. All the way down.

That was the first thing we saw upon arrival at the Madagascar base. It was oddly appropriate given what they had done. This is the story of how the OpenStack Infrastructure as a Service system was brought online that helped us take back Earth.”

- Ken Hui, OpenStacker and Hero of the People

Like a team of archeologists stumbling upon great technology during their explorations, the systems administrators and architects found a vSphere cluster running in the office of an abandoned office park in Antananarivo. While fleeing the chaos, someone forgot to turn out the lights.

This cluster served as the base for the OpenStack IaaS system used to deploy systems used to establish the satellite links required to transfer the repositories needed for further colonization.

To make sure that the systems were online as quick as possible the Supreme Project Stakeholder, the new ruler of Madagascar, demanded that the OpenStack system deploy images to two different hypervisors and that everything be up and functional with network connectivity in 5 days. If they failed, power supplies would run out and they would be doomed.

The Supreme Project Stakeholder also ordered that both Linux and Windows images be deployed to both hypervisors and that all instances be launched through the OpenStack Horizon dashboard.

Given the newly discovered vSphere cluster, the architect have designed a management cluster and nested vSphere Fabric Cluster with shared storage, HA and DRS. The management cluster also hosts the OpenStack nodes. Networking is a combination of OpenStack Neutron with OVS GRE overlay and vSphere distributed switches.

Storage is implemented in several ways. There is the existing datastores on the vSphere cluster, OpenStack Cinder block storage and iSCSI storage provided by a Virtual Storage Appliance (VSA). All of these subsystems server separate and unique purposes.

Much like Alan Moore taught us who watches The Watchmen, the Madagascar team has taught us who virtualizes the virtualized.

1.2 RequirementsIdentifier RequirementR01 Create an infrastructure from scratch using a

VMware cluster including shared storage, HA, and DRS.

R02 The VM images will include one Linux and one Windows instance

R03 In order to remove single points of failure due to the availability of human resources, you must integrate a second hypervisor

R04 Each hypervisor must host both a Linux and Windows instance

R05 Documented video proof must be provided to the moonbase team upon completion (see attached)

1.3 ConstraintsIdentifier ConstraintC01 My system must be powered by OpenStackC02 Deployment of instances will be done by using

the OpenStack Horizon dashboard only.C03 In all the panic of the Zed outbreak, DevStack has

been lost forever and cannot be used.C04 The first hypervisor must be vSphere

1.4 AssumptionsIdentifier AssumptionA01 Any version of OpenStack networking may be

used.A02 The second hypervisor can be anything other

than vSphere.

1.5 RisksIdentifier RiskK01 The stability of the underlying vSphere cluster is

unknown.K02 The connectivity and type of the underlying

physical storage is unknown.K03 The connectivity and type of the underlying

network is unknown.

1.6 Author’s NoteThis is not a standard design document that I would create. This is more an explanation of how I made actual lab work based on the requirements and constraints imposed by the VDM team in 5 days’ time. Not everything here is best practice and in fact some of it I consider to be a hack. But it works.

The documentation on doing most of this does not exist. I spent ~30 hours reading Launchpad.net bug reports, Github Pull Request comments, and lurking in freenode IRC to figure most of this out. This led to several facepalms at 3AM. As a result I think that knowing how I did it would be of more use to the community at large than a hypothetical design.

I also modified the original premise and substituted an existing vSphere cluster for bare metal hardware. This is purely a function of the lab that was available to me for testing. The underlying lab should be considered nothing more than a pool of resources, i.e. there is no wizard behind that curtain.

1.7 DedicationThis work is dedicated to my wife Brie and my kids Jack and Gretchen for tolerating me be a curmudgeon for 5 days while I made this work.

2 Management ClusterThis section details the vSphere management cluster as well as the underlying lab physical hardware.

2.1 Physical Host DesignThe physical host underlying the lab are 2 Proliant BL460c blades is a HP c7000 chassis. There are Flex 10 modules providing uplinks to both HP 8500zl networking switches. HP 3Par storage is connected via Fiber Channel. Each blade has two six core Intel Xeon processors are 64 Gb of RAM.

2.2 vSphere DesignEach of the two hosts are running vSphere 5.5. They are configured in a cluster with HA and fully automated DRS. They are managed by a vCenter 5.5 Virtual Appliance.

2.3 StorageBoth hosts boot from the HP 3Par SAN. There are 4 datastores, one for ISO image storage and 3 for virtual machines. The three datastores dedicated to VMs are collected in a data store cluster.

2.4 NetworkingAll networking on the management cluster is handled by one distributed switch. The switch as 4 port groups and 2 uplinks.

The port groups are for Management , vMotion, the VM Network, and a port group for the Neurton br-int internal bridge.

2.5 Virtual MachinesThe management cluster hosts the following virtual machines:

vCenter Virtual Appliance

Open Filer Virtual Storage Appliance

ESXi Nested Host 1

ESXi Nested Host 2

OpenStack Controller

OpenStack Network Manager

OpenStack Cinder Block Storage Node

Open Stack Compute Node A

Open Stack Compute Node B

3 vSphere Fabric ClusterThe vSphere Fabric Cluster is the fabric target for the creation on OpenStack instances hosted on vSphere. It host only VMs created by OpenStack instances.

3.1 Nested Host DesignThe vSphere Fabric Cluster consists of two virtual ESXi 5.5 hosts both configured with 8 vCPUs and 16 GB of vRAM. They are configured with HA and fully automated DRS set to the default recommendation level.

Both hosts are controlled by the same vCenter Virtual Appliance as the Management network.

3.2 StorageBoth hosts are connected to a single 250 GB iSCSI target LUN served by the Open Filer virtual appliance. The Open Filer VSA is a virtual machine hosted on the management cluster and the storage is backed by the management cluster datastores on 3Par storage.

3.3 NetworkingThe Fabric Cluster networking is provided by a single distributed virtual switch. There are 5 port groups and 4 uplinks.

The port groups are Management, iSCSI, vMotion, the VM Network, and the OpenStack Neutron OVS bri-int Internal Bridge.

The 4 uplink are all trunked to the VM Network on the Management Cluster.

All Port Groups run separate vLANs.

4 OpenStack ArchitectureThe OpenStack system deployed is built with OpenStack Icehouse and deployed on 5 nodes running Ubuntu 14.04 LTS “Trusty Tahr”. Each VM node is configured with 2 vCPS, 8 GB of RAM and a 100 virtual disk.

4.1 ControllerThe controller is the lynch pin of the OpenStack system. It servers the Identity Service, the Image Service, The dashboard, The Compute and Block Storage management services , The MySQL database and the RabbitMQ message service. All services use the MySQL database server and the RabbitMQ service.

4.1.1 Identify Management: KeystoneKeystone is configured for four tenants: Admin, Service, Demo, and VDM.

The Admin tenant contains the super user administrator for the OpenStack system.

The Service tenant contains the users for the keystone, glance, cinder, nova, and neutron services.

The Demo tenant contains one use “demo” and is used to as a dev tenant to test new images and instances.

The VDM tenant is the production tenant of the OpenStack system.

There is one MySQL database for the service named “keystone”.

4.1.2 Image Management: GlanceThe Glance service host 4 images, 2 Windows 7 (one for vSphere and one for KVM) a CirrOS Linus image and an Ubuntu 14.04 image.

Images hosted on vSphere are tagged with the proper VMware driver property tags. The Windows 7 image hosted on KVM is tagged with the VirtIO property tags for the NIC and SCSI drivers.

There is one MySQL database for the service named “glance”.

4.1.3 Dashboard: HorizonThe Horizon Dashboard is hosted on the controller node.

4.1.4 Compute Manager: NovaThere are several Nova services running on the controller node that enable scheduling, api calls, and node management.

The Nova service runs one MySQL database named “nova”.

4.2 Network ManagerThe Network Manager node run the OpenStack Neutron service. Neutron service is configured to use GRE tunnels between nodes as a network overlay to fence tenant networks.

The Neutron service runs one MySQL database named “neutron”.

4.3 Cinder Storage NodeThe Cinder storage node provides block level storage to store instances created on KVM. It uses an unformatted LVM volume group to provide the storage.

The Cinder control service runs on the controller node.

The Cinder service runs one database on the controller node named “cinder”.

4.4 Compute Node 1Compute Node 1 runs the nova-compute service, an OpenvSwitch service, and KVM. All instances that are spawned on KVM run as domains on this node.

The storage for the instance can run either on this node on o the Cinder block storage.

4.5 Compute Node 2Compute Node 2 runs the nova-compute service, an OpenvSwitch service and the VMware vCenter driver. All instances spawned on Compute Node 2 are run as virtual machines on the vSphere Fabric Cluster and stored on the iSCSI datastore.

5 Networking5.1 Neutron with OVS GRE OverlayThe following figure is an example GRE configuration and not representative of this actual implementation.

All instances running on KVM communicate to the network through the Neutron network using Open vSwitch and GRE tunnels.

There are three Neutron networks:

Demo-net

Vdm-net

Ext-net

Each network has a corresponding subnet:

Demo-net: demo-subnet: 10.0.0.0 /24

Vdm-net: vdm-subnet: 10.0.1.0 /24

Ext-net: ext-subnet: 192.168.30.0 /24

There are two routers one for demo and one for vdm. Each has the corresponding subnet attached to an internal interface and ext-net attached to the external interface.

5.2 vSphere: The WorkaroundOut of the box vSphere will not work with Neutron. To enable this functionality VMware NSX is required. NSX was not available in the lab and there was not enough time to setup a second nova-network Flat or VLAN backed network.

To fix this issue and allow for connectivity a DCHP server was added to the subnet the vSphere hosted instances are connected to. This allowed for outside connectivity without the complicity of an addition network system. Given a longer amount of time this would have been handled better.

5.3 vSphere: The Right WayVMware’s NSX product contains a Neutron plugin that would allow for extension of the GRE tunnels through the bri-int port group and providing the same connectivity to the vSphere hosted instances that is provided to the KVM hosed instances.

Due to lack of availability of NSX this was not possible given the time constraints.

Note: for how to setup NSX with OVS and GRE read the posts on scottlowe.org.

6 Image Design6.1 Linux on KVMThe Linux image on KVM is CirrOS 3.2.1 image provided by Launchpad.net. This image was built for OpenStack and works with no issue.

6.2 Windows on KVMTo create a Windows 7 image for KVM use the following steps:

1. Create a blank 10GB qcow2 with the qemu tools.2. Launch an instance in KVM manually with the new qcow2 file, the Windows 7 ISO, and the VirtIO

ISO.3. Connect to image via VNC and install the VirtIO drivers when required.4. When finished bot the image and run sysprep.5. Shutdown the image and import with Glance specifying the VirtIO drivers.6. Launch an instance through the Horizon Dashboard.7. Shut it down.8. Edit the instance XML of the domain to specify the VirtIO drivers.9. Start the Instance with the Horizon Dashboard.10. Run sysprep again.11. Don’t shutdown the image.12. Take a snapshot of the instance in the Horizon Dashboard.13. 13. Destroy the instance and original image. Launch new instances from the Snapshot.

6.3 VMDK ImagesThe key to deploying both Windows and Linux images in vSphere rely on the following:

1. Create your own image. Don’t use converted or extracted vmdk disks.2. Install the OS, patch and install the VMware Tools.3. Shutdown the VM and download the VMDK from the datastore.4. Upload the VMDK to Glance and specify the VMware SCSI driver used with the –property flag.5. Create a new Flavor in the Horizon Dashboard that matches the original VM settings as close as

possible. Especially drive size. 6. Create and instance through the Horizon Dashboard.

6.4 Thoughts on Image Creation and Deployment.Deploying successful Images and Instances was the most difficult part of this POC lab. Not following the above steps will lead to corrupted images and non-booting instances.

Images that hang on boot or during sysprep are almost always driver issues.