unifiedvirtualmachine migraonoverlongdistance · xvmo%on: unifiedvirtualmachine...
TRANSCRIPT
XvMo%on: Unified Virtual Machine
Migra%on over Long Distance Ali José Mash,zadeh, Min Cai, Gabriel Tarasuk-‐Levin,
Ricardo Koller, Tal Garfinkel, Sreekanth SeBy Stanford University – VMware, Inc.
Live Migra%on
Virtual Machine
Source Host Des,na,on Host Cloud
Migra%on Benefits
• Test to produc,on migra,ons • Disaster Preparedness Tes,ng • Cross-‐Datacenter Load Balancing • Shared-‐Nothing Architecture
Virtual Machine
Migra%on in Prac%ce
Virtual Machine
Source Host Des,na,on Host
Requires: Shared Storage Fast Networking Single Subnet
This is not what we really want
• Migra,ons are limited to machines that: With shared storage, fast networks, and same LAN
• Technological Changes: • Shared nothing applica,on architectures • Network mobility possible: LISP, OTV, VXlan, OpenFlow (SDN)
• Very large virtualized datacenters
• No reason for these limita,ons any longer • Customers have new use cases
XvMo%on
• XvMo,on: First commercially viable WAN migra,on
• Achieve Low Down,me AND Atomic Switchover • Uses Asynchronous IO Mirroring
• Principle: Flow control mechanisms for memory and disk
Customer Scenario
LAN WAN Bandwidth 10 Gbps
(some,mes 40 Gbps) 1 Gbps or less
Latency <1 ms ~100 ms Typical Network Dedicated NIC(s) Shared connec,on
between sites
Example Workload
Applica,on Highly Available Database
Client
HA Timeouts several seconds TCP Timeouts 120 seconds
Down%me (Switchover Time)
Virtual Machine
Running Stopped Running
Goal: Less than 1 Second of Down,me
Source Host Des,na,on Host
Virtual Machine
Atomicity
Virtual Machine
Source Host Des,na,on Host
Goal: Atomic switchover
XvMo%on
• Unifies Memory Migra,on and Storage Migra,on • Tolerates wide area network bandwidth/latency and reliability
• Tolerates heterogeneous storage performance • Down,mes and workload impact comparable to local migra,on
• Atomic switchover
• Deployed in customer metro area networks • Cross con,nent migra,on e.g. Palo Alto to India is prac,cal
Overview
• Architecture Overview • Wide Area Memory Migra,on • Wide Area Storage Migra,on • Evalua,on
Virtual Machine
Unified Live Migra%on
Virtual Machine
Source Host Des,na,on Host
XvMo%on Architecture
IO Mirror
Virtual Machine IO
IO Buffer
Live Migra,on
Stream
s
Stream
s
Source Des4na4on
IO Buffer/Writer
Live Migra,on
TCP
Overview
• Architecture Overview • Wide Area Memory Migra4on • Wide Area Storage Migra,on • Evalua,on
XvMo%on Architecture: Live Migra%on
IO Mirror
Virtual Machine IO
IO Buffer
Live Migra,on
Stream
s
Stream
s
Source Des4na4on
IO Buffer/Writer
Live Migra,on
TCP
Live Migra%on: Total Time vs Down%me
Memory
Source Des,na,on
Live Migra%on: Total Time vs Down%me
Memory
Source Des,na,on
Live Migra%on: Total Time vs Down%me
Memory
Source Des,na,on
Live Migra%on: Total Time vs Down%me
• Itera,ve copy hopefully reduces the working set each itera,on • Depends on Network being faster than Dirty rate
Memory
Source Des,na,on
Current Solu%on: Early Resume/Post-‐copy • Problem: Applica,ons can change memory faster than network bandwidth
• Solu,on: • Stop migra,on copy everything • Resume early if down,me is high
• Des,na,on remote page faults on missing pages
Stun During Page Send (SDPS)
• Problems with Early Resume: • Remote page faults very expensive for high latency networks
• Not atomic: Unsafe for WANs where network hiccups are more likely
• Solu,on: ThroBle VM just enough to keep dirty rate < network rate
SDPS Down%me Results
0
0.5
1
1.5
2
2.5
24-‐users 48-‐users
Down4
me (s)
SDPS OFF SDPS ON
Overview
• Architecture Overview • Wide Area Memory Migra,on • Wide Area Storage Migra4on • Evalua,on
XvMo%on Architecture: IO Mirroring
IO Mirror
Virtual Machine IO
IO Buffer
Live Migra,on
Stream
s
Stream
s
Source Des4na4on
IO Buffer/Writer
Live Migra,on
TCP
Problem: Synchronous mirroring over the WAN slows guest workload Goal: Hide network latency from VM
VM IO
IO
XvMo%on Buffering: Asynchronous IO
TCP Streams Transport Framework
Source Des4na4on
Bulk Disk Copy
Mirror IO
IO Buffer
Streams Transport Framework IO
XvMo%on Buffering: Asynchronous IO
VM TCP
Streams Transport Framework
Source Des4na4on
IO Buffer
Streams Transport Framework
IO IO IO IO
Problem: Workload may be too fast on source for the des;na;on Goal: ThroBle copy process and workload as necessary
IO
XvMo%on Buffering: Conges%on Control
VM TCP
Streams Transport Framework
Source Des4na4on
IO Buffer
Streams Transport Framework
IO
IO IO IO IO IO IO IO IO Back Pressure
Limit per-‐disk OIOs/buffer on des,na,on
Send free buffer space
Slow Disk
IO
XvMo%on Buffering: Conges%on Control
VM TCP
Streams Transport Framework
Source Des4na4on
IO Buffer
Streams Transport Framework
IO
IO IO IO IO IO IO IO IO Back Pressure
Limit per-‐disk OIOs/buffer on des,na,on
Send free buffer space
Slow Disk
Overview
• Architecture Overview • Wide Area Memory Migra,on • Wide Area Storage Migra,on • Evalua4on
Evalua%on
• Migra,on Time: Total end-‐to-‐end ,me • Down,me: Time machine execu,on is suspended for final switchover • Workload Penalty: Average percentage penalty to the VM workload
• OLTP IO Workload (data disk only) • 10 GB system/12 GB data
XvMo%on Down%mes
0
0.2
0.4
0.6
0.8
1
1.2
0 50 100 150 200
Down,
me (s)
Latency (ms)
Take Away: ~1 second down,me independent of distance
XvMo%on Migra%on Time
0
50
100
150
200
250
300
0 50 100 150 200
Total Tim
e (s)
Latency (ms)
Latency
Take Away: small linear ,me increase with distance
California to India Migra%on
• 1 Gbps network with 214 ms RTT
• OLTP: 68 MB/s disk copy – 89 MB/s memory copy • ~11% Workload Penalty from ThroBling
Summary
• XvMo,on frees migra,on from the need for shared storage and fast local networks • Tolerates wide area network bandwidth/latency and reliability
• Tolerates heterogeneous storage performance • Down,mes and workload penalty comparable to local migra,on
• Atomic Switchover
• Enables new use cases – e.g. disaster preparedness, cluster upgrade, shared nothing
• On the path to deployment: • Deployed in customer metro area networks • Cross con,nent migra,on e.g. Palo Alto to India is prac,cal
Ques%ons?