turning openstack swift into a vm storage platform
DESCRIPTION
OpenStack Swift is the Object Storage project within OpenStack. Alas, due to technical hurdles (eventual consistency, blocks objects, …) it is impossible to run Virtual Machines directly on Swift. You need a layer in between Swift and the hypervisor which can overcome these hurdles. This is where Open vStorage comes in. Open vStorage is an open-source VM storage router. It is a software layer (called the VM Storage Router) in between Virtual Machines and storage backends. It allows to abstract the backend from the Virtual Machine and creates a uniform, single namespace across multiple hosts. These VM Storage Routers (VSRs) operate like a grid leveraging local flash memory or SSDs and any storage back-end (S3 compatible object store, (distributed) filesystem, NAS) to provide an extremely high performance and reliable storage system. One of the supported Object Stores is OpenStack Swift. Open vStorage is the only solution to turn OpenStack Swift into block storage for Virtual Machines. Through a Cinder Plugin it allows to create and manage volumes directly on top of OpenStack Swift. Combining Open vStorage with OpenStack Swift allows to create a scale-out, performing, VM-centric storage platform which neatly integrates with OpenStack.TRANSCRIPT
OpenStack Meetup: Turning OpenStack Swift into a VM storage platform
Wim Provoost (@wimpers_be)Open vStorage (@openvstorage)
CloudFounders
• CloudFounders (2007)– 120 people in the US, Belgium, Romania and India
• Products– Open vStorage (open source under an Apache 2.0 license)
• Ultra reliable, high performance, open storage layer for virtual machines.
• Compatible with all major hypervisors.• Supports OpenStack
– vRun = Converged Infrastructure• Software that transforms server hardware to converged infrastructure
with unparalleled performance & flexibility.• Software can run on any x86 server or as a virtual machine
(VMware/OpenStack)• Uses Open vStorage as basis• Competes with Nutanix, Simplivity and upcoming EVO:RAIL
Object storage: the future of scale out storage
Object storage is the future for storing massive amounts of data:•PRO
– Build scale-out storage by adding more disk/servers when needed– High reliability by distributing content across nodes– Easy to manage (no knowledge needed about RAID or volumes)– Standard API as all solutions support S3 API, no vendor lock-in– Cost-effictive: use low cost, large capacity SATA disks, increase performance
by adding spindles/gateways
•CON– NO support for Virtual Machine Disk Storage (see next slide)
Object storage challenges as VM storage
• Eventual consistency (the CAP Theorem)
• Latency & performance– VMs require low latency and high performance– Object stores are developed to contain lots of data
(large disks, low performance)– Additional latency as Object Store is on the Local LAN instead of attached to
the host like DAS
• Different Management Paradigms– Object Stores understand Objects <> Hypervisors understand blocks, files
What is needed is a technology whereby Virtual Machines can use object stores instead of a SAN and get the benefits of the low cost
and scale-out capabilities of object stores
What is needed is a technology whereby Virtual Machines can use object stores instead of a SAN and get the benefits of the low cost
and scale-out capabilities of object stores
Traditional OpenStack Setup
NovaInstance
Management
SwiftObject Storage
CinderBlock Storage
GlanceImage store
VM
Provides volume for
Provisions
Stores image in
Stores backups in
Provides image for
Ceph, SAN, NAS, ...
Provides disk space
2 storage platforms?! 2 storage platforms?!
The answer: Open vStorage
NovaInstance
Management
SwiftObject Storage
GlanceImage store
VM
Provides volume for
Provisions
Stores image in
Stores backups in
Provides image for
Open vStorage
CinderBlock Storage
Provides disk space
Converts Object storage into Block storage
What is Open vStorage
Open vStorage is an open source “Storage Router“ and is installed on a host or a cluster of hosts to create a VM-centric, clustered, reliable,
scale out and high performance storage system for virtual machines.
Open vStorage Intro
• +40.000 IOPS per Hypervisor• Made for virtual machines• Unified Namespace• Ultra Reliable• Unlimited Snapshots• Endless Scalable• Lowest Management Cost In Market
SAN / NASSAN / NASS3 compatibleObject Based
Storage
S3 compatibleObject Based
Storage
Hypervisor
Open v Storage
Hypervisor
Open v StorageOpen v Storage
Hypervisor
Open v StorageOpen v Storage
Hypervisor
Open v StorageOpen v Storage
Distributed File Systems
Distributed File Systems
Ultra Scalable Storage For Virtual Machines
KVM
Scale-outVM VM
VM VM
SSDSSDSSDSSD
OpenvStorageOpen
vStorage
KVM
VM VM
VM VM
SSDSSDSSDSSD
OpenvStorageOpen
vStorage
KVM
VM VM
VM VM
SSDSSDSSDSSD
OpenvStorageOpen
vStorage
Virtual File System interface
Solving Eventual Consistency using time based approach
SSD or PCI FlashLBA 1: LBA 1: 4k block 1 4k block 1
LBA 2: LBA 2: 4k block 24k block 2
LBA 3: LBA 3: 4k block 34k block 3
LBA 4: LBA 4: 4k block 44k block 4
LBA 5: LBA 5: 4k block 54k block 5
LBA 1: LBA 1: 4k block 64k block 6
LBA 1: LBA 1: 4k block 74k block 7
LBA 3: LBA 3: 4k block 84k block 8
LBA 6: LBA 6: 4k block 94k block 9
LBA 7: LBA 7: 4k block 104k block 10
LBA 8: LBA 8: 4k block 11 4k block 11
LBA 2: LBA 2: 4k block 124k block 12
LBA 9: LBA 9: 4k block 134k block 13
LBA 10: LBA 10: 4k block 144k block 14
New writes
SCO 1
4k block 1 4k block 1
4k block 24k block 2
4k block 34k block 3
4k block 44k block 4
4k block 54k block 5
4k block 64k block 6
4k block 74k block 7
SCO 2
4k block 8 4k block 8
4k block 94k block 9
4k block 104k block 10
4k block 114k block 11
4k block 124k block 12
4k block 134k block 13
4k block 144k block 14
LBA 5: LBA 5: 4k block 154k block 15
LBA 10: LBA 10: 4k block 164k block 16
4k block 15 4k block 15
4k block 164k block 16
SCO 3
New writes
SCO1
SCO2 Transfer SCOs once they are full (4MB)to the Storage Backend at slow paceTransfer SCOs once they are full (4MB)to the Storage Backend at slow pace
Each write is appended
to the current Storage
Container Object (SCO)
Each write is appended
to the current Storage
Container Object (SCO)
Open vStorage <> distributed file system
VSA 1 VSA 2 VSA 3
Arakoon – (config params, metadata, ...)
vDisk1
vDisk1
vDisk2
vDisk2
InternalBucket
vDisk3
vDisk3
VFS2 VFS3
xml
VOLDRV
VM
VOLDRV
Object Router
FILEDRV
FILEDRV
VOLDRV
Object Router
FILEDRV
KVM1 KVM2 KVM3
VFS1
Object Router
Live Motion – In depth (Phase 1)
VSA 1 VSA 2 VSA 3
Arakoon – (config params, metadata, ...)
vDisk1
vDisk1
vDisk2
vDisk2
InternalBucket
vDisk3
vDisk3
VFS3
vmx
VOLDRV
VOLDRV
Object Router
FILEDRV
FILEDRV
VM
VOLDRV
Object Router
FILEDRV
KVM1 KVM2 KVM3
VFS1
VMLive Motion
Object Router
VFS2
Live Motion – In depth (Phase 2)
VSA 1 VSA 2 VSA 3
Arakoon – (config params, metadata, ...)
vDisk1
vDisk1
vDisk2
vDisk2
InternalBucket
vDisk3
vDisk3
VFS2 VFS3
xml
VOLDRV
VOLDRV
FILEDRV
FILEDRV
VM
VOLDRV
Object Router
FILEDRV
KVM1 KVM2 KVM3
VFS1
VMLive Motion
Object Router Handover Object Router
How does Open vStorage solve the problem
• Open vStorage is a middleware layer in between the hypervisor and the object store. (Converts object storage into block storage)– On the host: location based storage (block storage).– On the backend: time based storage (ideal for objects stores).– Open vStorage turns a volume into a single bucket.
• OpenStack Cinder Plugin for easy integration (snapshots, ...).• Distributed file systems don’t work! Open vStorage is not a distributed
file sysem!– All hosts ‘think’ they see the same virtual file systems.– Volume is ‘live’ on 1 host instead of all hosts.– Only the virtual file system metadata is distributed.
• Caching inside the host fixes impedance mismatch between slow, high latency backend and fast, low latency requirement of Virtual Machines.
The benefits
• Single distributed storage platform (Swift) to maintain. This platform is very performant and almost infinitely scalable.
• Better performance due to caching inside the host where the Nova instance runs.
• VM-centric management on storage level.– Thin-cloning, zero-copy snapshots, ...
• Add storage which isn’t compatible with Cinder.
Supported OpenStack actions
• Volumes– Add / Remove– Create / Delete snapshot– Clone from snapshot– Clone volume from volume– Import image to volume– Export volume to image
• Instances– Live Migration– Evacuate
Deployments
• Our internal environments are now running Open vStorage + swift
• Open vStorage is available in beta– Certified OpenStack Cinder plugin is expected with
Kilo release– Feel free to use in test environments and provide
feedback– Supported production release Q1
Next steps
• The code: https://bitbucket.org/openvstorage/openvstorage/• Download the packages and deploy in your OpenStack
environment• Documentation: http://doc.openvstorage.com• Support:
https://groups.google.com/forum/#!forum/open-vstorage• Follow us on Twitter (@OpenvStorage) or read our blog
(http://blog.openvstorage.com)
OpenStack Paris
We will be there! Look out for the “Cloudfounders” booth.
November, 3-7