the turtles project: design and implementation of nested virtualization · 2020. 8. 2. · mode,...
TRANSCRIPT
![Page 1: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/1.jpg)
The Turtles Project:Design and Implementation of Nested Virtualization
Muli Ben-Yehuda† Michael D. Day‡ Zvi Dubitzky† Michael Factor†
Nadav Har’El† Abel Gordon† Anthony Liguori‡ Orit Wasserman†
Ben-Ami Yassour†
†IBM Research – Haifa
‡IBM Linux Technology Center
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 1 / 30
![Page 2: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/2.jpg)
What is nested x86 virtualization?
Running multiple unmodifiedhypervisorsWith their associatedunmodified VM’sSimultaneouslyOn the x86 architectureWhich does not supportnesting in hardware. . .. . . but does support a singlelevel of virtualization Hardware
Hypervisor
GuestHypervisor
GuestOS
GuestOSGuestOS
GuestOS
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 2 / 30
![Page 3: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/3.jpg)
Why?
Operating systems are already hypervisors (Windows 7 with XPmode, Linux/KVM)To be able to run other hypervisors in cloudsSecurity (e.g., hypervisor-level rootkits)Co-design of x86 hardware and system softwareTesting, demonstrating, debugging, live migration of hypervisors
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 3 / 30
![Page 4: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/4.jpg)
Why?
Operating systems are already hypervisors (Windows 7 with XPmode, Linux/KVM)To be able to run other hypervisors in cloudsSecurity (e.g., hypervisor-level rootkits)Co-design of x86 hardware and system softwareTesting, demonstrating, debugging, live migration of hypervisors
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 3 / 30
![Page 5: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/5.jpg)
Why?
Operating systems are already hypervisors (Windows 7 with XPmode, Linux/KVM)To be able to run other hypervisors in cloudsSecurity (e.g., hypervisor-level rootkits)Co-design of x86 hardware and system softwareTesting, demonstrating, debugging, live migration of hypervisors
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 3 / 30
![Page 6: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/6.jpg)
Why?
Operating systems are already hypervisors (Windows 7 with XPmode, Linux/KVM)To be able to run other hypervisors in cloudsSecurity (e.g., hypervisor-level rootkits)Co-design of x86 hardware and system softwareTesting, demonstrating, debugging, live migration of hypervisors
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 3 / 30
![Page 7: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/7.jpg)
Why?
Operating systems are already hypervisors (Windows 7 with XPmode, Linux/KVM)To be able to run other hypervisors in cloudsSecurity (e.g., hypervisor-level rootkits)Co-design of x86 hardware and system softwareTesting, demonstrating, debugging, live migration of hypervisors
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 3 / 30
![Page 8: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/8.jpg)
Related work
First models for nested virtualization [PopekGoldberg74,BelpaireHsu75, LauerWyeth73]First implementation in the IBM z/VM; relies on architecturalsupport for nested virtualization (sie)Microkernels meet recursive VMs [FordHibler96]: assumes wecan modify software at all levelsx86 software based approaches (slow!) [Berghmans10]KVM [KivityKamay07] with AMD SVM [RoedelGraf09]Early Xen prototype [He09]Blue Pill rootkit hiding from other hypervisors [Rutkowska06]
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 4 / 30
![Page 9: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/9.jpg)
Related work
First models for nested virtualization [PopekGoldberg74,BelpaireHsu75, LauerWyeth73]First implementation in the IBM z/VM; relies on architecturalsupport for nested virtualization (sie)Microkernels meet recursive VMs [FordHibler96]: assumes wecan modify software at all levelsx86 software based approaches (slow!) [Berghmans10]KVM [KivityKamay07] with AMD SVM [RoedelGraf09]Early Xen prototype [He09]Blue Pill rootkit hiding from other hypervisors [Rutkowska06]
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 4 / 30
![Page 10: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/10.jpg)
Related work
First models for nested virtualization [PopekGoldberg74,BelpaireHsu75, LauerWyeth73]First implementation in the IBM z/VM; relies on architecturalsupport for nested virtualization (sie)Microkernels meet recursive VMs [FordHibler96]: assumes wecan modify software at all levelsx86 software based approaches (slow!) [Berghmans10]KVM [KivityKamay07] with AMD SVM [RoedelGraf09]Early Xen prototype [He09]Blue Pill rootkit hiding from other hypervisors [Rutkowska06]
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 4 / 30
![Page 11: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/11.jpg)
Related work
First models for nested virtualization [PopekGoldberg74,BelpaireHsu75, LauerWyeth73]First implementation in the IBM z/VM; relies on architecturalsupport for nested virtualization (sie)Microkernels meet recursive VMs [FordHibler96]: assumes wecan modify software at all levelsx86 software based approaches (slow!) [Berghmans10]KVM [KivityKamay07] with AMD SVM [RoedelGraf09]Early Xen prototype [He09]Blue Pill rootkit hiding from other hypervisors [Rutkowska06]
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 4 / 30
![Page 12: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/12.jpg)
Related work
First models for nested virtualization [PopekGoldberg74,BelpaireHsu75, LauerWyeth73]First implementation in the IBM z/VM; relies on architecturalsupport for nested virtualization (sie)Microkernels meet recursive VMs [FordHibler96]: assumes wecan modify software at all levelsx86 software based approaches (slow!) [Berghmans10]KVM [KivityKamay07] with AMD SVM [RoedelGraf09]Early Xen prototype [He09]Blue Pill rootkit hiding from other hypervisors [Rutkowska06]
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 4 / 30
![Page 13: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/13.jpg)
Related work
First models for nested virtualization [PopekGoldberg74,BelpaireHsu75, LauerWyeth73]First implementation in the IBM z/VM; relies on architecturalsupport for nested virtualization (sie)Microkernels meet recursive VMs [FordHibler96]: assumes wecan modify software at all levelsx86 software based approaches (slow!) [Berghmans10]KVM [KivityKamay07] with AMD SVM [RoedelGraf09]Early Xen prototype [He09]Blue Pill rootkit hiding from other hypervisors [Rutkowska06]
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 4 / 30
![Page 14: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/14.jpg)
Related work
First models for nested virtualization [PopekGoldberg74,BelpaireHsu75, LauerWyeth73]First implementation in the IBM z/VM; relies on architecturalsupport for nested virtualization (sie)Microkernels meet recursive VMs [FordHibler96]: assumes wecan modify software at all levelsx86 software based approaches (slow!) [Berghmans10]KVM [KivityKamay07] with AMD SVM [RoedelGraf09]Early Xen prototype [He09]Blue Pill rootkit hiding from other hypervisors [Rutkowska06]
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 4 / 30
![Page 15: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/15.jpg)
What is the Turtles project?
Efficient nested virtualization for Intel x86 based on KVMRuns multiple guest hypervisors and VMs: KVM, VMware, Linux,Windows, . . .Code publicly available
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 5 / 30
![Page 16: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/16.jpg)
What is the Turtles project? (cont’)
Nested VMX virtualization for nested CPU virtualizationMulti-dimensional paging for nested MMU virtualizationMulti-level device assignment for nested I/O virtualizationMicro-optimizations to make it go fast
+ + =
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 6 / 30
![Page 17: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/17.jpg)
What is the Turtles project? (cont’)
Nested VMX virtualization for nested CPU virtualizationMulti-dimensional paging for nested MMU virtualizationMulti-level device assignment for nested I/O virtualizationMicro-optimizations to make it go fast
+ + =
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 6 / 30
![Page 18: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/18.jpg)
What is the Turtles project? (cont’)
Nested VMX virtualization for nested CPU virtualizationMulti-dimensional paging for nested MMU virtualizationMulti-level device assignment for nested I/O virtualizationMicro-optimizations to make it go fast
+ + =
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 6 / 30
![Page 19: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/19.jpg)
What is the Turtles project? (cont’)
Nested VMX virtualization for nested CPU virtualizationMulti-dimensional paging for nested MMU virtualizationMulti-level device assignment for nested I/O virtualizationMicro-optimizations to make it go fast
+ + =
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 6 / 30
![Page 20: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/20.jpg)
Theory of nested CPU virtualization
Trap and emulate[PopekGoldberg74]⇒ it’s all about the trapsSingle-level (x86) vs. multi-level (e.g., z/VM)Single level ⇒ one hypervisor, many guestsTurtles approach: L0 multiplexes the hardware between L1 and L2,running both as guests of L0—without either being aware of it(Scheme generalized for n levels; Our focus is n=2)
Hardware
Host Hypervisor
Guest
Hardware
Host Hypervisor
Multiplexed on a single level Multiple logical levels
L0
L1
L2
L1
GuestL2
GuestL2
L0
GuestL2L2
Guest Hypervisor
GuestHypervisor GuestGuest
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 7 / 30
![Page 21: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/21.jpg)
Theory of nested CPU virtualization
Trap and emulate[PopekGoldberg74]⇒ it’s all about the trapsSingle-level (x86) vs. multi-level (e.g., z/VM)Single level ⇒ one hypervisor, many guestsTurtles approach: L0 multiplexes the hardware between L1 and L2,running both as guests of L0—without either being aware of it(Scheme generalized for n levels; Our focus is n=2)
Hardware
Host Hypervisor
Guest
Hardware
Host Hypervisor
Multiplexed on a single level Multiple logical levels
L0
L1
L2
L1
GuestL2
GuestL2
L0
GuestL2L2
Guest Hypervisor
GuestHypervisor GuestGuest
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 7 / 30
![Page 22: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/22.jpg)
Theory of nested CPU virtualization
Trap and emulate[PopekGoldberg74]⇒ it’s all about the trapsSingle-level (x86) vs. multi-level (e.g., z/VM)Single level ⇒ one hypervisor, many guestsTurtles approach: L0 multiplexes the hardware between L1 and L2,running both as guests of L0—without either being aware of it(Scheme generalized for n levels; Our focus is n=2)
Hardware
Host Hypervisor
Guest
Hardware
Host Hypervisor
Multiplexed on a single level Multiple logical levels
L0
L1
L2
L1
GuestL2
GuestL2
L0
GuestL2L2
Guest Hypervisor
GuestHypervisor GuestGuest
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 7 / 30
![Page 23: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/23.jpg)
Theory of nested CPU virtualization
Trap and emulate[PopekGoldberg74]⇒ it’s all about the trapsSingle-level (x86) vs. multi-level (e.g., z/VM)Single level ⇒ one hypervisor, many guestsTurtles approach: L0 multiplexes the hardware between L1 and L2,running both as guests of L0—without either being aware of it(Scheme generalized for n levels; Our focus is n=2)
Hardware
Host Hypervisor
Guest
Hardware
Host Hypervisor
Multiplexed on a single level Multiple logical levels
L0
L1
L2
L1
GuestL2
GuestL2
L0
GuestL2L2
Guest Hypervisor
GuestHypervisor GuestGuest
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 7 / 30
![Page 24: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/24.jpg)
Theory of nested CPU virtualization
Trap and emulate[PopekGoldberg74]⇒ it’s all about the trapsSingle-level (x86) vs. multi-level (e.g., z/VM)Single level ⇒ one hypervisor, many guestsTurtles approach: L0 multiplexes the hardware between L1 and L2,running both as guests of L0—without either being aware of it(Scheme generalized for n levels; Our focus is n=2)
Hardware
Host Hypervisor
Guest
Hardware
Host Hypervisor
Multiplexed on a single level Multiple logical levels
L0
L1
L2
L1
GuestL2
GuestL2
L0
GuestL2L2
Guest Hypervisor
GuestHypervisor GuestGuest
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 7 / 30
![Page 25: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/25.jpg)
Nested VMX virtualization: flow
L0 runs L1 with VMCS0→1
L1 prepares VMCS1→2 andexecutes vmlaunchvmlaunch traps to L0
L0 merges VMCS’s:VMCS0→1 merged withVMCS1→2 is VMCS0→2
L0 launches L2
L2 causes a trapL0 handles trap itself orforwards it to L1
. . .eventually, L0 resumes L2
repeat
Hardware
Host Hypervisor
GuestOS
VMCS
MemoryTables
VMCS
MemoryTables
VMCSMemoryTables
L0
L1 L2
1-2 State
0-2 State0-1 State
GuestHypervisor
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 8 / 30
![Page 26: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/26.jpg)
Nested VMX virtualization: flow
L0 runs L1 with VMCS0→1
L1 prepares VMCS1→2 andexecutes vmlaunchvmlaunch traps to L0
L0 merges VMCS’s:VMCS0→1 merged withVMCS1→2 is VMCS0→2
L0 launches L2
L2 causes a trapL0 handles trap itself orforwards it to L1
. . .eventually, L0 resumes L2
repeat
Hardware
Host Hypervisor
GuestOS
VMCS
MemoryTables
VMCS
MemoryTables
VMCSMemoryTables
L0
L1 L2
1-2 State
0-2 State0-1 State
GuestHypervisor
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 8 / 30
![Page 27: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/27.jpg)
Nested VMX virtualization: flow
L0 runs L1 with VMCS0→1
L1 prepares VMCS1→2 andexecutes vmlaunchvmlaunch traps to L0
L0 merges VMCS’s:VMCS0→1 merged withVMCS1→2 is VMCS0→2
L0 launches L2
L2 causes a trapL0 handles trap itself orforwards it to L1
. . .eventually, L0 resumes L2
repeat
Hardware
Host Hypervisor
GuestOS
VMCS
MemoryTables
VMCS
MemoryTables
VMCSMemoryTables
L0
L1 L2
1-2 State
0-2 State0-1 State
GuestHypervisor
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 8 / 30
![Page 28: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/28.jpg)
Nested VMX virtualization: flow
L0 runs L1 with VMCS0→1
L1 prepares VMCS1→2 andexecutes vmlaunchvmlaunch traps to L0
L0 merges VMCS’s:VMCS0→1 merged withVMCS1→2 is VMCS0→2
L0 launches L2
L2 causes a trapL0 handles trap itself orforwards it to L1
. . .eventually, L0 resumes L2
repeat
Hardware
Host Hypervisor
GuestOS
VMCS
MemoryTables
VMCS
MemoryTables
VMCSMemoryTables
L0
L1 L2
1-2 State
0-2 State0-1 State
GuestHypervisor
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 8 / 30
![Page 29: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/29.jpg)
Nested VMX virtualization: flow
L0 runs L1 with VMCS0→1
L1 prepares VMCS1→2 andexecutes vmlaunchvmlaunch traps to L0
L0 merges VMCS’s:VMCS0→1 merged withVMCS1→2 is VMCS0→2
L0 launches L2
L2 causes a trapL0 handles trap itself orforwards it to L1
. . .eventually, L0 resumes L2
repeat
Hardware
Host Hypervisor
GuestOS
VMCS
MemoryTables
VMCS
MemoryTables
VMCSMemoryTables
L0
L1 L2
1-2 State
0-2 State0-1 State
GuestHypervisor
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 8 / 30
![Page 30: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/30.jpg)
Nested VMX virtualization: flow
L0 runs L1 with VMCS0→1
L1 prepares VMCS1→2 andexecutes vmlaunchvmlaunch traps to L0
L0 merges VMCS’s:VMCS0→1 merged withVMCS1→2 is VMCS0→2
L0 launches L2
L2 causes a trapL0 handles trap itself orforwards it to L1
. . .eventually, L0 resumes L2
repeat
Hardware
Host Hypervisor
GuestOS
VMCS
MemoryTables
VMCS
MemoryTables
VMCSMemoryTables
L0
L1 L2
1-2 State
0-2 State0-1 State
GuestHypervisor
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 8 / 30
![Page 31: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/31.jpg)
Nested VMX virtualization: flow
L0 runs L1 with VMCS0→1
L1 prepares VMCS1→2 andexecutes vmlaunchvmlaunch traps to L0
L0 merges VMCS’s:VMCS0→1 merged withVMCS1→2 is VMCS0→2
L0 launches L2
L2 causes a trapL0 handles trap itself orforwards it to L1
. . .eventually, L0 resumes L2
repeat
Hardware
Host Hypervisor
GuestOS
VMCS
MemoryTables
VMCS
MemoryTables
VMCSMemoryTables
L0
L1 L2
1-2 State
0-2 State0-1 State
GuestHypervisor
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 8 / 30
![Page 32: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/32.jpg)
Nested VMX virtualization: flow
L0 runs L1 with VMCS0→1
L1 prepares VMCS1→2 andexecutes vmlaunchvmlaunch traps to L0
L0 merges VMCS’s:VMCS0→1 merged withVMCS1→2 is VMCS0→2
L0 launches L2
L2 causes a trapL0 handles trap itself orforwards it to L1
. . .eventually, L0 resumes L2
repeat
Hardware
Host Hypervisor
GuestOS
VMCS
MemoryTables
VMCS
MemoryTables
VMCSMemoryTables
L0
L1 L2
1-2 State
0-2 State0-1 State
GuestHypervisor
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 8 / 30
![Page 33: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/33.jpg)
Nested VMX virtualization: flow
L0 runs L1 with VMCS0→1
L1 prepares VMCS1→2 andexecutes vmlaunchvmlaunch traps to L0
L0 merges VMCS’s:VMCS0→1 merged withVMCS1→2 is VMCS0→2
L0 launches L2
L2 causes a trapL0 handles trap itself orforwards it to L1
. . .eventually, L0 resumes L2
repeat
Hardware
Host Hypervisor
GuestOS
VMCS
MemoryTables
VMCS
MemoryTables
VMCSMemoryTables
L0
L1 L2
1-2 State
0-2 State0-1 State
GuestHypervisor
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 8 / 30
![Page 34: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/34.jpg)
Nested VMX virtualization: flow
L0 runs L1 with VMCS0→1
L1 prepares VMCS1→2 andexecutes vmlaunchvmlaunch traps to L0
L0 merges VMCS’s:VMCS0→1 merged withVMCS1→2 is VMCS0→2
L0 launches L2
L2 causes a trapL0 handles trap itself orforwards it to L1
. . .eventually, L0 resumes L2
repeat
Hardware
Host Hypervisor
GuestOS
VMCS
MemoryTables
VMCS
MemoryTables
VMCSMemoryTables
L0
L1 L2
1-2 State
0-2 State0-1 State
GuestHypervisor
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 8 / 30
![Page 35: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/35.jpg)
Exit multiplication makes angry turtle angry
To handle a single L2 exit, L1 does many things: read and writethe VMCS, disable interrupts, . . .Those operations can trap, leading to exit multiplicationExit multiplication: a single L2 exit can cause 40-50 L1 exits!Optimize: make a single exit fast and reduce frequency of exits
……
…
…
L0
L1
L2
L3
Two Levels
Three LevelsSingle Level
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 9 / 30
![Page 36: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/36.jpg)
Exit multiplication makes angry turtle angry
To handle a single L2 exit, L1 does many things: read and writethe VMCS, disable interrupts, . . .Those operations can trap, leading to exit multiplicationExit multiplication: a single L2 exit can cause 40-50 L1 exits!Optimize: make a single exit fast and reduce frequency of exits
……
…
…
L0
L1
L2
L3
Two Levels
Three LevelsSingle Level
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 9 / 30
![Page 37: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/37.jpg)
Exit multiplication makes angry turtle angry
To handle a single L2 exit, L1 does many things: read and writethe VMCS, disable interrupts, . . .Those operations can trap, leading to exit multiplicationExit multiplication: a single L2 exit can cause 40-50 L1 exits!Optimize: make a single exit fast and reduce frequency of exits
……
…
…
L0
L1
L2
L3
Two Levels
Three LevelsSingle Level
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 9 / 30
![Page 38: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/38.jpg)
Exit multiplication makes angry turtle angry
To handle a single L2 exit, L1 does many things: read and writethe VMCS, disable interrupts, . . .Those operations can trap, leading to exit multiplicationExit multiplication: a single L2 exit can cause 40-50 L1 exits!Optimize: make a single exit fast and reduce frequency of exits
……
…
…
L0
L1
L2
L3
Two Levels
Three LevelsSingle Level
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 9 / 30
![Page 39: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/39.jpg)
Introduction to x86 MMU virtualization
x86 does page table walks in hardwareMMU has one currently active hardware page tableBare metal ⇒ only needs one logical translation,(virtual → physical)Virtualization ⇒ needs two logical translations
1 Guest page table: (guest virt→ guest phys)2 Host page table: (guest phys→ host phys)
. . . but MMU only knows to walk a single table!
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 10 / 30
![Page 40: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/40.jpg)
Introduction to x86 MMU virtualization
x86 does page table walks in hardwareMMU has one currently active hardware page tableBare metal ⇒ only needs one logical translation,(virtual → physical)Virtualization ⇒ needs two logical translations
1 Guest page table: (guest virt→ guest phys)2 Host page table: (guest phys→ host phys)
. . . but MMU only knows to walk a single table!
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 10 / 30
![Page 41: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/41.jpg)
Introduction to x86 MMU virtualization
x86 does page table walks in hardwareMMU has one currently active hardware page tableBare metal ⇒ only needs one logical translation,(virtual → physical)Virtualization ⇒ needs two logical translations
1 Guest page table: (guest virt→ guest phys)2 Host page table: (guest phys→ host phys)
. . . but MMU only knows to walk a single table!
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 10 / 30
![Page 42: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/42.jpg)
Introduction to x86 MMU virtualization
x86 does page table walks in hardwareMMU has one currently active hardware page tableBare metal ⇒ only needs one logical translation,(virtual → physical)Virtualization ⇒ needs two logical translations
1 Guest page table: (guest virt→ guest phys)2 Host page table: (guest phys→ host phys)
. . . but MMU only knows to walk a single table!
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 10 / 30
![Page 43: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/43.jpg)
Introduction to x86 MMU virtualization
x86 does page table walks in hardwareMMU has one currently active hardware page tableBare metal ⇒ only needs one logical translation,(virtual → physical)Virtualization ⇒ needs two logical translations
1 Guest page table: (guest virt→ guest phys)2 Host page table: (guest phys→ host phys)
. . . but MMU only knows to walk a single table!
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 10 / 30
![Page 44: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/44.jpg)
Software MMU virtualization: shadow paging
Guest virtual
Guest physical
Host physical
SPT
GV->GP
GP->HP
Two logical translations compressed onto the shadow pagetable [DevineBugnion02]Unmodified guest OS updates its own tableHypervisor traps OS page table updatesHypervisor propagates updates to the hardware tableMMU walks the tableProblem: traps are expensive
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 11 / 30
![Page 45: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/45.jpg)
Software MMU virtualization: shadow paging
Guest virtual
Guest physical
Host physical
SPT
GV->GP
GP->HP
Two logical translations compressed onto the shadow pagetable [DevineBugnion02]Unmodified guest OS updates its own tableHypervisor traps OS page table updatesHypervisor propagates updates to the hardware tableMMU walks the tableProblem: traps are expensive
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 11 / 30
![Page 46: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/46.jpg)
Software MMU virtualization: shadow paging
Guest virtual
Guest physical
Host physical
SPT
GV->GP
GP->HP
Two logical translations compressed onto the shadow pagetable [DevineBugnion02]Unmodified guest OS updates its own tableHypervisor traps OS page table updatesHypervisor propagates updates to the hardware tableMMU walks the tableProblem: traps are expensive
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 11 / 30
![Page 47: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/47.jpg)
Software MMU virtualization: shadow paging
Guest virtual
Guest physical
Host physical
SPT
GV->GP
GP->HP
Two logical translations compressed onto the shadow pagetable [DevineBugnion02]Unmodified guest OS updates its own tableHypervisor traps OS page table updatesHypervisor propagates updates to the hardware tableMMU walks the tableProblem: traps are expensive
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 11 / 30
![Page 48: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/48.jpg)
Software MMU virtualization: shadow paging
Guest virtual
Guest physical
Host physical
SPT
GV->GP
GP->HP
Two logical translations compressed onto the shadow pagetable [DevineBugnion02]Unmodified guest OS updates its own tableHypervisor traps OS page table updatesHypervisor propagates updates to the hardware tableMMU walks the tableProblem: traps are expensive
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 11 / 30
![Page 49: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/49.jpg)
Software MMU virtualization: shadow paging
Guest virtual
Guest physical
Host physical
SPT
GV->GP
GP->HP
Two logical translations compressed onto the shadow pagetable [DevineBugnion02]Unmodified guest OS updates its own tableHypervisor traps OS page table updatesHypervisor propagates updates to the hardware tableMMU walks the tableProblem: traps are expensive
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 11 / 30
![Page 50: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/50.jpg)
Hardware MMU virtualization: Extended Page Tables
Guest virtual
Guest physical
Host physical
GPTGV->GP
GP->HP
EPT
Two-dimensional paging: guest owns GPT, hypervisor ownsEPT [BhargavaSerebrin08]Unmodified guest OS updates GPTHypervisor updates EPT table controlling (guest phys→ hostphys) translationsMMU walks both tables
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 12 / 30
![Page 51: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/51.jpg)
Hardware MMU virtualization: Extended Page Tables
Guest virtual
Guest physical
Host physical
GPTGV->GP
GP->HP
EPT
Two-dimensional paging: guest owns GPT, hypervisor ownsEPT [BhargavaSerebrin08]Unmodified guest OS updates GPTHypervisor updates EPT table controlling (guest phys→ hostphys) translationsMMU walks both tables
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 12 / 30
![Page 52: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/52.jpg)
Hardware MMU virtualization: Extended Page Tables
Guest virtual
Guest physical
Host physical
GPTGV->GP
GP->HP
EPT
Two-dimensional paging: guest owns GPT, hypervisor ownsEPT [BhargavaSerebrin08]Unmodified guest OS updates GPTHypervisor updates EPT table controlling (guest phys→ hostphys) translationsMMU walks both tables
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 12 / 30
![Page 53: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/53.jpg)
Hardware MMU virtualization: Extended Page Tables
Guest virtual
Guest physical
Host physical
GPTGV->GP
GP->HP
EPT
Two-dimensional paging: guest owns GPT, hypervisor ownsEPT [BhargavaSerebrin08]Unmodified guest OS updates GPTHypervisor updates EPT table controlling (guest phys→ hostphys) translationsMMU walks both tables
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 12 / 30
![Page 54: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/54.jpg)
Nested MMU virt. via multi-dimensional paging
Three logical translations: L2 virt → phys, L2 → L1, L1 → L0
Only two tables in hardware with EPT:virt → phys and guest physical→ host physicalL0 compresses three logical translations onto two hardware tables
SPT12
L2 virtual
L2 physical
L1 physical
L0 physical
GPT
L2 virtual
L2 physical
L1 physical
L0 physical
GPT
SPT02
Shadow on top of shadow
SPT12
EPT01
L2 virtual
L2 physical
L1 physical
L0 physical
GPT
EPT02
EPT12
Multi-dimensional paging
Shadow on top of EPT
EPT01
baseline better best
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 13 / 30
![Page 55: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/55.jpg)
Baseline: shadow-on-shadow
SPT12
L2 virtual
L2 physical
L1 physical
L0 physical
GPT
SPT02
Assume no EPT table; all hypervisors use shadow pagingUseful for old machines and as a baselineMaintaining shadow page tables is expensiveCompress: three logical translations⇒ one table in hardware
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 14 / 30
![Page 56: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/56.jpg)
Better: shadow-on-EPT
L2 virtual
L2 physical
L1 physical
L0 physical
SPT12
EPT01
GPT
Instead of one hardware table we have twoCompress: three logical translations⇒ two in hardwareSimple approach: L0 uses EPT, L1 uses shadow paging for L2
Every L2 page fault leads to multiple L1 exits
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 15 / 30
![Page 57: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/57.jpg)
Best: multi-dimensional pagingL2 virtual
L2 physical
L1 physical
L0 physical
GPT
EPT02
EPT12
EPT01
EPT table rarely changes; guest page table changes a lotAgain, compress three logical translations⇒ two in hardwareL0 emulates EPT for L1
L0 uses EPT0→1 and EPT1→2 to construct EPT0→2
End result: a lot less exits!Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 16 / 30
![Page 58: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/58.jpg)
Introduction to I/O virtualization
From the hypervisor’s perspective, what is I/O?
(1) PIO (2) MMIO (3) DMA (4) interruptsDevice emulation [Sugerman01]
GUEST
HOST
1
2
34
deviceemulation
driverdevice
driverdevice
Para-virtualized drivers [Barham03, Russell08]GUEST
HOST
driver
1
23
back−end
virtualdriver
front−end
virtualdevicedriver
Direct device assignment [Levasseur04,Yassour08]GUEST
HOST
devicedriver
Direct assignment best performing optionDirect assignment requires IOMMU for safe DMA bypass
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 17 / 30
![Page 59: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/59.jpg)
Introduction to I/O virtualization
From the hypervisor’s perspective, what is I/O?(1) PIO
(2) MMIO (3) DMA (4) interruptsDevice emulation [Sugerman01]
GUEST
HOST
1
2
34
deviceemulation
driverdevice
driverdevice
Para-virtualized drivers [Barham03, Russell08]GUEST
HOST
driver
1
23
back−end
virtualdriver
front−end
virtualdevicedriver
Direct device assignment [Levasseur04,Yassour08]GUEST
HOST
devicedriver
Direct assignment best performing optionDirect assignment requires IOMMU for safe DMA bypass
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 17 / 30
![Page 60: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/60.jpg)
Introduction to I/O virtualization
From the hypervisor’s perspective, what is I/O?(1) PIO (2) MMIO
(3) DMA (4) interruptsDevice emulation [Sugerman01]
GUEST
HOST
1
2
34
deviceemulation
driverdevice
driverdevice
Para-virtualized drivers [Barham03, Russell08]GUEST
HOST
driver
1
23
back−end
virtualdriver
front−end
virtualdevicedriver
Direct device assignment [Levasseur04,Yassour08]GUEST
HOST
devicedriver
Direct assignment best performing optionDirect assignment requires IOMMU for safe DMA bypass
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 17 / 30
![Page 61: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/61.jpg)
Introduction to I/O virtualization
From the hypervisor’s perspective, what is I/O?(1) PIO (2) MMIO (3) DMA
(4) interruptsDevice emulation [Sugerman01]
GUEST
HOST
1
2
34
deviceemulation
driverdevice
driverdevice
Para-virtualized drivers [Barham03, Russell08]GUEST
HOST
driver
1
23
back−end
virtualdriver
front−end
virtualdevicedriver
Direct device assignment [Levasseur04,Yassour08]GUEST
HOST
devicedriver
Direct assignment best performing optionDirect assignment requires IOMMU for safe DMA bypass
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 17 / 30
![Page 62: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/62.jpg)
Introduction to I/O virtualization
From the hypervisor’s perspective, what is I/O?(1) PIO (2) MMIO (3) DMA (4) interrupts
Device emulation [Sugerman01]GUEST
HOST
1
2
34
deviceemulation
driverdevice
driverdevice
Para-virtualized drivers [Barham03, Russell08]GUEST
HOST
driver
1
23
back−end
virtualdriver
front−end
virtualdevicedriver
Direct device assignment [Levasseur04,Yassour08]GUEST
HOST
devicedriver
Direct assignment best performing optionDirect assignment requires IOMMU for safe DMA bypass
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 17 / 30
![Page 63: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/63.jpg)
Introduction to I/O virtualization
From the hypervisor’s perspective, what is I/O?(1) PIO (2) MMIO (3) DMA (4) interruptsDevice emulation [Sugerman01]
GUEST
HOST
1
2
34
deviceemulation
driverdevice
driverdevice
Para-virtualized drivers [Barham03, Russell08]GUEST
HOST
driver
1
23
back−end
virtualdriver
front−end
virtualdevicedriver
Direct device assignment [Levasseur04,Yassour08]GUEST
HOST
devicedriver
Direct assignment best performing optionDirect assignment requires IOMMU for safe DMA bypass
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 17 / 30
![Page 64: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/64.jpg)
Introduction to I/O virtualization
From the hypervisor’s perspective, what is I/O?(1) PIO (2) MMIO (3) DMA (4) interruptsDevice emulation [Sugerman01]
GUEST
HOST
1
2
34
deviceemulation
driverdevice
driverdevice
Para-virtualized drivers [Barham03, Russell08]GUEST
HOST
driver
1
23
back−end
virtualdriver
front−end
virtualdevicedriver
Direct device assignment [Levasseur04,Yassour08]GUEST
HOST
devicedriver
Direct assignment best performing optionDirect assignment requires IOMMU for safe DMA bypass
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 17 / 30
![Page 65: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/65.jpg)
Introduction to I/O virtualization
From the hypervisor’s perspective, what is I/O?(1) PIO (2) MMIO (3) DMA (4) interruptsDevice emulation [Sugerman01]
GUEST
HOST
1
2
34
deviceemulation
driverdevice
driverdevice
Para-virtualized drivers [Barham03, Russell08]GUEST
HOST
driver
1
23
back−end
virtualdriver
front−end
virtualdevicedriver
Direct device assignment [Levasseur04,Yassour08]GUEST
HOST
devicedriver
Direct assignment best performing optionDirect assignment requires IOMMU for safe DMA bypass
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 17 / 30
![Page 66: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/66.jpg)
Introduction to I/O virtualization
From the hypervisor’s perspective, what is I/O?(1) PIO (2) MMIO (3) DMA (4) interruptsDevice emulation [Sugerman01]
GUEST
HOST
1
2
34
deviceemulation
driverdevice
driverdevice
Para-virtualized drivers [Barham03, Russell08]GUEST
HOST
driver
1
23
back−end
virtualdriver
front−end
virtualdevicedriver
Direct device assignment [Levasseur04,Yassour08]GUEST
HOST
devicedriver
Direct assignment best performing option
Direct assignment requires IOMMU for safe DMA bypass
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 17 / 30
![Page 67: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/67.jpg)
Introduction to I/O virtualization
From the hypervisor’s perspective, what is I/O?(1) PIO (2) MMIO (3) DMA (4) interruptsDevice emulation [Sugerman01]
GUEST
HOST
1
2
34
deviceemulation
driverdevice
driverdevice
Para-virtualized drivers [Barham03, Russell08]GUEST
HOST
driver
1
23
back−end
virtualdriver
front−end
virtualdevicedriver
Direct device assignment [Levasseur04,Yassour08]GUEST
HOST
devicedriver
Direct assignment best performing optionDirect assignment requires IOMMU for safe DMA bypass
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 17 / 30
![Page 68: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/68.jpg)
Multi-level device assignment
With nested 3x3 options for I/O virtualization (L2 ⇔ L1 ⇔ L0)Multi-level device assignment means giving an L2 guest directaccess to L0’s devices, safely bypassing both L0 and L1
L1 hypervisor
physical device
L0 hypervisor
L2 device driver
MMIOs and PIOs
L0 IOMMUL1 IOMMU
Device DMA via platform IOMMU
How? L0 emulates an IOMMU for L1 [Amit10]L0 compresses multiple IOMMU translations onto the singlehardware IOMMU page tableL2 programs the device directlyDevice DMA’s into L2 memory space directly
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 18 / 30
![Page 69: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/69.jpg)
Multi-level device assignment
With nested 3x3 options for I/O virtualization (L2 ⇔ L1 ⇔ L0)Multi-level device assignment means giving an L2 guest directaccess to L0’s devices, safely bypassing both L0 and L1
L1 hypervisor
physical device
L0 hypervisor
L2 device driver
MMIOs and PIOs
L0 IOMMUL1 IOMMU
Device DMA via platform IOMMU
How? L0 emulates an IOMMU for L1 [Amit10]L0 compresses multiple IOMMU translations onto the singlehardware IOMMU page tableL2 programs the device directlyDevice DMA’s into L2 memory space directly
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 18 / 30
![Page 70: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/70.jpg)
Multi-level device assignment
With nested 3x3 options for I/O virtualization (L2 ⇔ L1 ⇔ L0)Multi-level device assignment means giving an L2 guest directaccess to L0’s devices, safely bypassing both L0 and L1
L1 hypervisor
physical device
L0 hypervisor
L2 device driver
MMIOs and PIOs
L0 IOMMUL1 IOMMU
Device DMA via platform IOMMU
How? L0 emulates an IOMMU for L1 [Amit10]L0 compresses multiple IOMMU translations onto the singlehardware IOMMU page tableL2 programs the device directlyDevice DMA’s into L2 memory space directly
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 18 / 30
![Page 71: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/71.jpg)
Multi-level device assignment
With nested 3x3 options for I/O virtualization (L2 ⇔ L1 ⇔ L0)Multi-level device assignment means giving an L2 guest directaccess to L0’s devices, safely bypassing both L0 and L1
L1 hypervisor
physical device
L0 hypervisor
L2 device driver
MMIOs and PIOs
L0 IOMMUL1 IOMMU
Device DMA via platform IOMMU
How? L0 emulates an IOMMU for L1 [Amit10]L0 compresses multiple IOMMU translations onto the singlehardware IOMMU page tableL2 programs the device directlyDevice DMA’s into L2 memory space directly
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 18 / 30
![Page 72: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/72.jpg)
Multi-level device assignment
With nested 3x3 options for I/O virtualization (L2 ⇔ L1 ⇔ L0)Multi-level device assignment means giving an L2 guest directaccess to L0’s devices, safely bypassing both L0 and L1
L1 hypervisor
physical device
L0 hypervisor
L2 device driver
MMIOs and PIOs
L0 IOMMUL1 IOMMU
Device DMA via platform IOMMU
How? L0 emulates an IOMMU for L1 [Amit10]L0 compresses multiple IOMMU translations onto the singlehardware IOMMU page tableL2 programs the device directlyDevice DMA’s into L2 memory space directly
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 18 / 30
![Page 73: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/73.jpg)
Multi-level device assignment
With nested 3x3 options for I/O virtualization (L2 ⇔ L1 ⇔ L0)Multi-level device assignment means giving an L2 guest directaccess to L0’s devices, safely bypassing both L0 and L1
L1 hypervisor
physical device
L0 hypervisor
L2 device driver
MMIOs and PIOs
L0 IOMMUL1 IOMMU
Device DMA via platform IOMMU
How? L0 emulates an IOMMU for L1 [Amit10]L0 compresses multiple IOMMU translations onto the singlehardware IOMMU page tableL2 programs the device directlyDevice DMA’s into L2 memory space directly
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 18 / 30
![Page 74: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/74.jpg)
Micro-optimizations
Goal: reduce world switch overheadsReduce cost of single exit by focus on VMCS merges:
Keep VMCS fields in processor encodingPartial updates instead of whole-sale copyingCopy multiple fields at onceSome optimizations not safe according to spec
Reduce frequency of exits—focus on vmread and vmwriteAvoid the exit multiplier effectLoads/stores vs. architected trapping instructionsBinary patching?
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 19 / 30
![Page 75: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/75.jpg)
Micro-optimizations
Goal: reduce world switch overheadsReduce cost of single exit by focus on VMCS merges:
Keep VMCS fields in processor encodingPartial updates instead of whole-sale copyingCopy multiple fields at onceSome optimizations not safe according to spec
Reduce frequency of exits—focus on vmread and vmwriteAvoid the exit multiplier effectLoads/stores vs. architected trapping instructionsBinary patching?
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 19 / 30
![Page 76: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/76.jpg)
Micro-optimizations
Goal: reduce world switch overheadsReduce cost of single exit by focus on VMCS merges:
Keep VMCS fields in processor encodingPartial updates instead of whole-sale copyingCopy multiple fields at onceSome optimizations not safe according to spec
Reduce frequency of exits—focus on vmread and vmwriteAvoid the exit multiplier effectLoads/stores vs. architected trapping instructionsBinary patching?
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 19 / 30
![Page 77: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/77.jpg)
Windows XP on KVM L1 on KVM L0
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 20 / 30
![Page 78: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/78.jpg)
Linux on VMware L1 on KVM L0
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 21 / 30
![Page 79: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/79.jpg)
Experimental Setup
Running Linux, Windows,KVM, VMware, SMP, . . .Macro workloads:
kernbenchSPECjbbnetperf
Multi-dimensional paging?Multi-level device assignment?KVM as L1 vs. VMware as L1?
See paper for full experimentaldetails and more benchmarksand analysis
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 22 / 30
![Page 80: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/80.jpg)
Macro: SPECjbb and kernbench
kernbenchHost Guest Nested NestedDRW
Run time 324.3 355 406.3 391.5% overhead vs. host - 9.5 25.3 20.7% overhead vs. guest - - 14.5 10.3
SPECjbbHost Guest Nested NestedDRW
Score 90493 83599 77065 78347% degradation vs. host - 7.6 14.8 13.4% degradation vs. guest - - 7.8 6.3
Table: kernbench and SPECjbb results
Exit multiplication effect not as bad as we feared
Direct vmread and vmwrite (DRW) give an immediate boost
Take-away: each level of virtualization adds approximately the sameoverhead!
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 23 / 30
![Page 81: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/81.jpg)
Macro: multi-dimensional paging
Shadow on EPTMulti−dimensional paging
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
kernbench specjbb netperf
Impr
ovem
ent r
atio
Impact of multi-dimensional paging depends on rate of page faultsShadow-on-EPT: every L2 page fault causes L1 multiple exitsMulti-dimensional paging: only EPT violations cause L1 exitsEPT table rarely changes: #(EPT violations) << #(page faults)Multi-dimensional paging huge win for page-fault intensivekernbench
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 24 / 30
![Page 82: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/82.jpg)
Macro: multi-level device assignment
throughput (Mbps)%cpu
0
100
200
300
400
500
600
700
800
900
1,000
nativesingle level guest
emulation
single level guest
virtio
single level guest
direct access
nested guest emulation / emulation
nested guest virtio / emulation
nested guest virtio / virtio
nested guest direct / virtio
nested guest direct / direct
0
20
40
60
80
100
throug
hput (M
bps)
% cpu
Benchmark: netperf TCP_STREAM (transmit)Multi-level device assignment best performing optionBut: native at 20%, multi-level device assignment at 60% (x3!)Interrupts considered harmful, cause exit multiplication
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 25 / 30
![Page 83: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/83.jpg)
Macro: multi-level device assignment (sans interrupts)
100
200
300
400
500
600
700
800
900
1000
16 32 64 128 256 512
Thro
ughp
ut (M
bps)
Message size (netperf -m)
L0 (bare metal)L2 (direct/direct)L2 (direct/virtio)
What if we could deliver device interrupts directly to L2?Only 7% difference between native and nested guest!
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 26 / 30
![Page 84: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/84.jpg)
Micro: synthetic worst case CPUID loop
L1L0cpu mode switch
0
10,000
20,000
30,000
40,000
50,000
60,000
1. Single Level
Guest
2. Nested Guest
3. Nested Guest
optimizations 3.5.1
4. Nested Guest
optimizations 3.5.2
5. Nested Guest
optimizations 3.5.1 & 3.5.2
CPU
Cycle
s
CPUID running in a tight loop is not a real-world workload!Went from 30x worse to “only” 6x worseA nested exit is still expensive—minimize both single exitcost and frequency of exits
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 27 / 30
![Page 85: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/85.jpg)
Conclusions
Efficient nested x86virtualization is challenging butfeasibleA whole new ballpark openingup many excitingapplications—security, cloud,architecture, . . .Current overhead of 6-14%
Negligible for someworkloads, not yet for othersWork in progress—expect atmost 5% eventually
Code is availableWhy Turtles?It’s turtles all the way down
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 28 / 30
![Page 86: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/86.jpg)
Conclusions
Efficient nested x86virtualization is challenging butfeasibleA whole new ballpark openingup many excitingapplications—security, cloud,architecture, . . .Current overhead of 6-14%
Negligible for someworkloads, not yet for othersWork in progress—expect atmost 5% eventually
Code is availableWhy Turtles?It’s turtles all the way down
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 28 / 30
![Page 87: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/87.jpg)
Conclusions
Efficient nested x86virtualization is challenging butfeasibleA whole new ballpark openingup many excitingapplications—security, cloud,architecture, . . .Current overhead of 6-14%
Negligible for someworkloads, not yet for othersWork in progress—expect atmost 5% eventually
Code is availableWhy Turtles?It’s turtles all the way down
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 28 / 30
![Page 88: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/88.jpg)
Conclusions
Efficient nested x86virtualization is challenging butfeasibleA whole new ballpark openingup many excitingapplications—security, cloud,architecture, . . .Current overhead of 6-14%
Negligible for someworkloads, not yet for othersWork in progress—expect atmost 5% eventually
Code is availableWhy Turtles?It’s turtles all the way down
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 28 / 30
![Page 89: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/89.jpg)
Conclusions
Efficient nested x86virtualization is challenging butfeasibleA whole new ballpark openingup many excitingapplications—security, cloud,architecture, . . .Current overhead of 6-14%
Negligible for someworkloads, not yet for othersWork in progress—expect atmost 5% eventually
Code is availableWhy Turtles?It’s turtles all the way down
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 28 / 30
![Page 90: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/90.jpg)
Conclusions
Efficient nested x86virtualization is challenging butfeasibleA whole new ballpark openingup many excitingapplications—security, cloud,architecture, . . .Current overhead of 6-14%
Negligible for someworkloads, not yet for othersWork in progress—expect atmost 5% eventually
Code is availableWhy Turtles?It’s turtles all the way down
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 28 / 30
![Page 91: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/91.jpg)
Questions?
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 29 / 30
![Page 92: The Turtles Project: Design and Implementation of Nested Virtualization · 2020. 8. 2. · mode, Linux/KVM) To be able to run other hypervisors inclouds Security (e.g., hypervisor-level](https://reader036.vdocuments.us/reader036/viewer/2022081411/60ac15af04b0c9657b6b3d89/html5/thumbnails/92.jpg)
IBM Research – Haifa is hiring
© 2010 IBM Corporation
Systems Software Research in Haifa
Contributions to IBM products, cutting-edge research– Virtualization, Cloud Computing, Data Storage
Influencing European cloud research agenda– Consulting to European Commission, working through industrial groups, leading
projects
Papers, patents, image for IBM and Israel
We are Hiring!– Researchers and Engineers
Ben-Yehuda et al. (IBM Research) The Turtles Project: Nested Virtualization Technion Clubnet Oct 2010 30 / 30