a virtual storage environment for ssds and hdds in xen hypervisor
TRANSCRIPT
A Virtual Storage Environment for SSDs
and HDDs in Xen Hypervisor
Outline• Introduction• Motivation• Related Work• System Design and Architecture• Implementation• Experiment• Conclusion
雲端運算的定義
Cloud computing is Internet-based computing, whereby shared resources, software, and information are provided to computers and other devices on demand, like the electricity grid. [wiki]
Cloud computing describes a new supplement, consumption, and delivery model for IT services based on the Internet, and it typically involves over-the-Internet provision of dynamically scalable and often virtualized resources. [wiki]
雲端運算的定義
• “... a pay-per-use model for enabling available, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.”
[National Institute of Standards and Technology (NIST) ]
雲端運算的定義
Introduction• Nowadays, virtualization is a popular technology
to provide an abstract hardware emulation due to the increasing hardware speed and capabilities.
• Advantages of virtualization technology:– Merge physical servers, reduce the space required for
the servers.– Simplify the management of the servers.– Enhance resource utilization.– Conserve energy to reduce carbon emission.
虛擬化為雲端運算的基礎技術
傳統單一主機 (low)
虛擬主機 (high)
低使用率
高使用率
現在的電腦效能相當強大,但卻有非常多的電腦平常只有使用不到20% 的運算資源,甚至更少。但因為種種需求,又無法降低電腦數量,因此把數台電腦“虛擬化”塞進一台功能較強大的實體主機,利用實體主機的運算資源,成了一種受歡迎的方法。
虛擬機的概念
• 虛擬機可允許一台實體主機同時執行多個作業系統。
• 一台強大電腦主機能做 30 台電腦的事,在佔用面積
、耗電量、空調、維護人事成本上,都能有效節約。
• 可在一台實體主機內執行多個虛擬主機,每一台虛擬主機既可以互相用線路連線、又可以獨立運作,互不干涉。
• 可將多台實體主機合併成一台大的資源池(Resource Pool) ,統籌管理、分配在上面的虛擬機。
虛擬機的概念
• 只要實體主機的硬體能力許可,便能靈活地建立或移除多台虛擬機,可保持高度彈性。
• 虛擬主機往往在映像檔內直接執行、存取檔案,使得備份、搬移虛擬機較實體主機容易許多。
• 虛擬化可使實體主機擴增、維修、更換不須關閉虛擬機,只要把裡面的虛擬機搬移到別的實體主機上即可。
• 可使用虛擬機測試不穩定的軟體,再利用快照完全還原整個系統 ( 僅需數秒鐘 ) ,而不須在實體主機測試。
Hardware
Virtualized World Physical World
Traditional x86 Architecture• 傳統是一台主機執行一套應用程式
且整體使用率偏低平均約 8%
Virtualization: • 新一代資訊中心是在單一主機執行多
種應用程式讓整體使用率提高到 80%
何謂虛擬化
Traditional Data Center – Fragmented and Inefficient Utilization
Fragmented and Inefficient IT Resources Server capabilities
wasted
0%
20%
40%
60%
80%
100%
All Servers
Server capabilities reach bottleneck
Num
ber o
f CPU
s re
quire
d
Time
Cost increases as new applications grow
Low utilization rate as too many servers are deployed for peak time
App4
OS
HW
App3
OS
HW
App2
OS
HW
App1
OS
HW
Virtual Data Center Resource Pools
New Virtual Data Center - Elastic Provisioning
Elastic scale up when demands increase
Sleeping
Elastic scale down when demands decrease
Running
Running Running Running Running Sleeping
Sleeping Sleeping Sleeping
Hardware
Virtualized World Physical World
Traditional x86 Architecture• 傳統是一台主機執行一套應用程式
且整體使用率偏低平均約 8%
Virtualization: • 新一代資訊中心是在單一主機執行多
種應用程式讓整體使用率提高到 80%
虛擬機的概念
虛擬化技術的好處• Test before we go:
Virtual machine just like a experiment box. You are able to test your application on different kind of system in same machine. It's easy to destroy, rebuild or backup your whole testing environments.
• Server consolidation: Many small physical servers could be replaced by virtual
machine, runs on virtualization environment provided by one powerful physical server. Although hardware is consolidated, typically OSs are not. Instead, each OS running on a physical server becomes converted to a distinct OS running inside a virtual machine. The large server can "host" many such "guest" virtual machines.
• Increase hardware utilization:To increase the utilization of costly hardware resources such as
cpu, memory or even like storage space.
• Rapid provisioning, dynamic fault tolerance..For example: Big system memory can partition as smaller portions for legacy
system runs within a virtual machine.
Virtualization Cloud Computing
Virtualization can Reduce cost for operation
and maintenance
Increase resource utilization
Enable quick deployment of applications and resources
Virtualization can NOT• Reduce system complexity
• Offer elastic resource provisioning
• Provide distributed computing capabilities
Introduction• Three parts of Xen:
– Xen Hypervisor– Domain 0– Domain U
Introduction• Xen Hyperviosr
– An abstraction layer between the guest domains and the physical hardware.
– Allocating and controlling resources.
Introduction• Domain 0
– A modified Linux kernel and is only one privileged virtual machine.
– Directly access the hardware resources, and contain drivers for the hardware resources.
Introduction• Domain U
– A general virtual machine, since domain U is a unprivileged virtual machine.
– Sending I/O requests through front-end driver, and back-end driver in domain 0 will handle and perform the I/O requests.
Introduction• Characteristics of SSD
– No mechanical latency– Low power consumption– Shock resistance– Provide better performance than HDD, especially for a
lot of random accesses– Smaller storage capacity than HDD
Introduction• I/O performance comparison between SSDs and
HDDs under various request sizes [using IOmeter benchmark]– Intel 320 series 160G– Intel 710 series 100G– Hitachi HDS721010CLA332 1TB
Introduction• Storage device combine SSD and HDD
– Seem SSDs to be a storage device.– Seem SSDs to be a secondary-level cache.
Virtual Storage Virtual Storage
HDDs HDDsSSDs
Mapping TableSSDs
Related Work• [1]Chai-Hao Tsai, Jerry Chou, Yeh-Ching Chung,
“Value-Based Tiering Management on Heterogeneous Block-Level Storage System,” IEEE 4th International Conference on Cloud Computing Technology and Science, 2012. – They formulate resource allocation problem into an ILP
(Integer Linear Programming), and propose data migration algorithms based on quantify the performance of difference access patterns on each type storage devices.
– They regard SSDs as storage space and storage cache of HDD.
Related Work• [2]Youngjae Kim, Aayush Gupta, Bhuvan
Urgaonkar, Piotr Berman, and Anand Sivasubramaniam, “HybridStore A Cost-Efficient, High-Performance Storage System Combining SSDs and HDDs,” 19th Annual IEEE International Symposium on Modelling, Analysis, and Simulation of Computer and Telecommunication Systems, 2011. – They formulate some constraints (such capacity,
performance, and lifetime) as a Mixed Integer Program to decide which storage to store.
– But they do not consider data migration in different storage devices.
Related Work• [3]Hannes Payer, Marco A.A. Sanvido,Zvonimir Z.
Bandic and Christoph M. Kirsch, “Combo Drive: Optimizing Cost and Performance in a Heterogeneous Storage Device,” WISH, 2009. – Combine SSD and HDD into a single heterogeneous
storage device.– Move some files to SSDs/HDDs based on file type.
• SSD:.exe, .dll, .sys, .msi, .cab, .drv, .jar, …• HDD:.jpg, .pdf, .wmv, .wma, .mp3, .doc, .xls, …
– Need modify file systems
HDDSSD
Logical Address
Virtual Address
Logical Address
Low high
Related Work• [4]Shih-Yu Liu, “An Efficient Data Management
Method for SSD-Based Storage Systems,” National Taiwan University of Science and Technology Master Thesis, 2012. – Propose a data allocation method for SSD-based
storage systems and consider append/update data.– Consider speed declining of SSDs under different space
utilization.
Related Work• [5]Chih-Kai Kang, “A Hybrid Storage Access
Framework for Virtual Machines,” National Taiwan University of Science and Technology Master Thesis, 2012.– Propose a hybrid storage framework in Xen hypervisor– Adopt potential value to determine which data should be
placed in SSDs or HDDs.– Only consider running one virtual machine.
Motivation• Virtualization technology
– Virtualization technology has ability to dynamically adjust resources in accordance with the demand of each virtual machine.
– Running multiple virtual machines in only one physical machine require high I/O performance.
– I/O is a serious bottleneck for overall performance. – SSDs can provide better I/O performance but higher cost
than HDDs.
Motivation• When space utilization of SSDs was 75%, write
performance of SSDs began to decline.• Read performance of SSDs do not change
significantly.
A Virtual Storage Environment for SSDs and HDDs in Xen Hypervisor
• Blktap in the kernel space – Can not directly access physical storage devices.– Receives all I/O requests of Domain U.– Transfers the I/O requests to Blktap in the user space of
Domain 0 to perform I/O requests to physical storage devices.
A Virtual Storage Environment for SSDs and HDDs in Xen Hypervisor
• Blktap in the user space – Provide Xen hypervisor an interface to access storage
devices.
A Virtual Storage Environment for SSDs and HDDs in Xen Hypervisor
• I/O Requests from Multiple Virtual Machines to Physical Storage Devices
• Monitor • Play an important role for
dynamic and adaptive adjustment in I/O performance
• Content mapping table• Record which data are
transferred to SSDs or HDDs
Data Distributed Management• Potential value [5]
– Data with smaller potential value is suitable to be placed in HDD.
– Data with bigger potential value has higher priority to be assigned to SSD.
• N : request size• RHDD (N) is the average time of random access whose size
is N in HDDs• RSSD (N) is the average time of random access whose size
is N in SSDs.• SHDD (N) is the average time of sequential access whose
size is N in HDDs• SSSD (N) is the average time of sequential access whose
size is N in SSDs
Potential Value• Maintain chunk info
– Chunk table
Potential Value• Maintain chunk info
Our method HD Size16G 32GB 64GB 128GB
Chunk Size
4k ≒12MB ≒24MB ≒48MB ≒96MB16k ≒3MB ≒6MB ≒12MB ≒24MB64k ≒0.75MB ≒1.5MB ≒3MB ≒6MB
256k ≒192kB ≒384kB ≒0.75MB ≒1.5MB1M ≒48kB ≒96kB ≒192kB ≒384kB
Potential Value• Identify high potential chunks
50
30
10
25
25
10
10
5
10
5
0
5
5
Quota : 10
Quota : 510*50/(50+25+25)
Quota : 35*30/(30+10+10)
Quota : 15*10/(30+10+10)
Data Distributed Management• Budget of SSDs [4]
– Smaller Budgetj means that the declining speed ratio for SSD storagej is low.
– Larger Budgetj means that the bandwidth of SSD storagej has been decreased.
– : Average writing speed for SSD storagej
– : Declining speed ratio for SSD storagej
– : Space utilization ratio for SSD storagej
Implementation• Mainly modify two components in Blktap driver:
– Tap-disk – Block-aio
Implementation• Receive and analyze I/O requests
– Tap-disk is a process which can handle I/O requests and provide an interface for virtual machine management.
– We can modify Tap-disk to analyze the I/O requests and then determine their storage destinations (i.e., the monitor mechanism).
– According to the analysis of I/O requests and SSD capacity utilization, we can transfer appropriate I/O requests to suitable SSDs to improve overall I/O performance.
Component File Function Functionality
Tap-disk Tapdisk-vbd.c tapdisk_vbd_create()tapdisk_vbd_issue_request()counter content mapping table()monitor()
1. Create and maintain a content mapping table2. Create a monitor mechanism3. Analyze I/O requests4. Deliver I/O requests and its locations to Block-aio component
Implementation• Perform I/O requests to physical storage devices
– Block-aio component is a process to handle each virtual machine image such as image open/close and I/O requests to the image.
– Block-aio is a bottom layer of Xen hypervisor to access physical storage devices.
• We modify Block-aio to perform I/O requests to the final storage devices (e.g., SSDs or HDDs).
Component File Function Functionality
Block-aio Block-aio.c tdaio_open()tdaio_complete()tdaio_queue_read()tdaio_queue_write()
1. Maintain virtual machine images in SSDs and HDDs2. Perform I/O requests to SSDs and HDDs3. Migrate data between SSDs and HDDs
Experimental Evaluation
Environment Configuration
Hardware
CPUIntel(R) Core(TM) i7 CPU 950 @
3.07GHz
Hard Disk Drive Hitachi HDS721010CLA332 1TB
Solid-state DriveSSD Intel 320 series 160GB
SSD Intel 710 series 100GB
Memory 8GB
SystemVirtualization Software Xen-4.1.2
Linux Kernel Linux 2.6.32
Domain 0Operating System Ubuntu 11.10
Memory 2GB
Domain U
Operating System Ubuntu 11.10
Memory 128MB
Storage space 32GB
vCPU 1
Experimental Evaluation• I/O traces file
Web search Financial
Read 99% Read 25%
Write 1% Write 75%
Request size 8kB Request size 512B~2kB
Workload Size 16.7GB Workload Size 7.2GB
Total Read 65.8GB Total Read 5.2GB
Total Write 8MB Total Write 28GB
Experimental Evaluation• Web search result
– Two Intel SSDs was about 8.44 times faster than two Hitachi HDD when running one virtual machine.
– When running two or more virtual machines, two Intel SSD was about 10 to 14 times faster than two Hitachi HDD.
Experimental Evaluation• Web search result
– Two Intel SSDs was about 1.38 times faster than one Hitachi HDD and two Intel SSDs when running one/two virtual machines.
– When running three/four virtual machines, one Hitachi HDD and two Intel SSDs are close to two Intel SSDs.
Number of VMs 2 Hitachi HDDs 1 Hitachi HD and 2 Intel SSDs
2 Intel SSDs
1 VM 1 21326 3502 2526
Average Execution Time (s) 21326 3502 2526
2 VM 1 45125 5848 4131VM 2 45709 5477 4146
Average Execution Time (s) 45417 5662.5 4138.5
3 VM 1 71893 6265 5796
VM 2 72639 6320 5809
VM 3 70458 6991 5809
Average Execution Time (s) 71663.33 6525.333 5804.667
4 VM 1 100057 6872 5839
VM 2 100959 7567 6862
VM 3 101053 6792 7346
VM 4 96269 7675 7046
Average Execution Time (s) 99584.5 7226.5 6773.25
1.38
1.36
1.12
1.06
Experimental Evaluation• Financial result
– The average execution time of two Intel SSDs was about 11 times faster than two Hitachi when running one virtual machine.
– When running two or more virtual machines, two Intel SSD was about 19 and 23 times faster than two Hitachi HDD.
Experimental Evaluation• Financial result
– Two Intel SSDs was 1.87 times faster than one Hitachi and two Intel SSDs,
– Running two or more virtual machines the result show two Intel SSDs was 1.4 times faster than one Hitachi and two Intel SSDs.
Number of VMs 2 Hitachi HDDs 1 Hitachi HD and 2 Intel SSDs
2 Intel SSDs
1 VM 1 14274 2387 1272
Average Execution Time (s) 14274 2387 1272
2 VM 1 35342 2635 1852
VM 2 35430 2587 1858
Average Execution Time (s) 35386 2611 1855
3 VM 1 56607 3591 2502
VM 2 58215 3575 2511
VM 3 56013 3577 2504
Average Execution Time (s) 56945 3581 2505.667
4 VM 1 77981 4298 3275
VM 2 82056 4214 3288
VM 3 80735 4285 3260
VM 4 68406 4288 3276
Average Execution Time (s) 77294.5 4271.25 3274.75
1.87
1.40
1.42
1.30
Conclusion• We focuses on the implementation and discussion
on how to revise Xen hypervisor to provide a platform using SSDs and HDDs in virtualization environment, and we also include a data management method for SSDs
• Xen hypervisor will play an important role of receiving and analyzing I/O requests from virtual machines, and performing I/O requests to SSDs and HDDs (i.e., the monitor mechanism and the content mapping table).
• The proposed virtual storage environment using SSDs and HDDs can not only provide better performance than only HDDs but also provide a more economical solution than only SSDs.
Reference• [1]Chai-Hao Tsai, Jerry Chou, Yeh-Ching Chung, “Value-Based
Tiering Management on Heterogeneous Block-Level Storage System,” IEEE 4th International Conference on Cloud Computing Technology and Science, 2012.
• [2]Youngjae Kim, Aayush Gupta, Bhuvan Urgaonkar, Piotr Berman, and Anand Sivasubramaniam, “HybridStore A Cost-Efficient, High-Performance Storage System Combining SSDs and HDDs,” 19th Annual IEEE International Symposium on Modelling, Analysis, and Simulation of Computer and Telecommunication Systems, 2011.
• [3]Hannes Payer, Marco A.A. Sanvido,Zvonimir Z. Bandic and Christoph M. Kirsch, “Combo Drive: Optimizing Cost and Performance in a Heterogeneous Storage Device,” WISH, 2009.
• [4]Shih-Yu Liu, “An Efficient Data Management Method for SSD-Based Storage Systems,” National Taiwan University of Science and Technology, Master Thesis, 2012.
• [5]Chih-Kai Kang, “A Hybrid Storage Access Framework for Virtual Machines,” National Taiwan University of Science and Technology, Master Thesis, 2012.
•