bud17-301: zeno's paradox: optimizing kvm/arm
TRANSCRIPT
![Page 1: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/1.jpg)
BUD17-301: KVM/ARM Nested VirtualizationChristoffer Dall
![Page 2: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/2.jpg)
Hardware
Hypervisor
VM
VM
Kernel
App App
Nested Virtualization
Hypervisor
VM
Kernel
App App
VM
Kernel
App App
![Page 3: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/3.jpg)
Hardware
Host Hypervisor
VM
VM
Kernel
App App
Terminology
Guest Hypervisor
Nested VM
Kernel
App App
Nested VM
Kernel
App App
L0
L1
L2
L1
L0
![Page 4: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/4.jpg)
Use Cases
1. IaaS hosting private clouds
2. Test your hypervisor in a VM
3. Debug your hypervisor in a VM
4. Develop hypervisors using a cloud
![Page 5: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/5.jpg)
Theorem 2
“A conventional third generation computer is recursively virtualizable if it is: (a) virtualizable, and (b) a VMM without any timing dependencies can be
constructed for it.”
Formal requirements for virtualizable third generation architectures [Popek and Goldberg ’74]
![Page 6: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/6.jpg)
Recursively Virtualizable
• Only applies to virtualizable architectures
• ARM and x86 are not virtualizable
• Hardware support for virtualization
![Page 7: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/7.jpg)
ARM Virtualization Extensions
Kernel
UserEL0
EL1
HypervisorEL2
![Page 8: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/8.jpg)
VM
ARM Virtualization Extensions
EL0
EL1
EL2 Hypervisor
Kernel
User Space
VM
Kernel
User Space
![Page 9: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/9.jpg)
ARM Nested Virtualization
EL0
EL1
EL2 Host Hypervisor
Kernel
User Space
Kernel
User Space
VirtualEL2 Guest Hypervisor Guest Hypervisor
![Page 10: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/10.jpg)
ARM Nested Virtualization
EL0
EL1
EL2 Host Hypervisor
Kernel
User Space
Kernel
User Space
EL ?? Guest Hypervisor Guest Hypervisor
![Page 11: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/11.jpg)
ARM Nested Virtualization
EL0
EL1
EL2 Host Hypervisor
Kernel
User Space
Kernel
User Space
EL0 Guest Hypervisor Guest Hypervisor
Trap-and-emulate
![Page 12: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/12.jpg)
ARM Nested Virtualization
EL0
EL1
EL2 Host Hypervisor
Kernel
User Space
Kernel
User Space
EL1 Guest Hypervisor Guest Hypervisor
?? -and-emulate
![Page 13: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/13.jpg)
ARMv8.3
• Supports running the guest hypervisor in EL1
• HCR_EL2.NV:
• Traps EL2 operations executed in EL1 to EL2
• Traps eret to EL2
![Page 14: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/14.jpg)
• CPU Virtualization
• Memory Virtualization
• Timer Virtualization
• Interrupt Virtualization
KVM/ARM Nested Virtualization
![Page 15: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/15.jpg)
struct kvm_cpu_context { u64 sys_regs[NR_SYS_REGS]; + u64 el2_regs[NR_EL2_REGS]; }
struct kvm_vcpu_arch { … struct kvm_cpu_context ctxt; }
Nested CPU Virtualization
![Page 16: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/16.jpg)
Host
Linux
AppApp
VM
Kernel
AppApp
KVM
EL0
EL1
EL2Restore EL1 sys_regs
Save EL1 sys_regs
Hypervisor-VM Switch
![Page 17: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/17.jpg)
Host
Linux
AppApp
VM
Kernel
KVM
EL0
EL1
EL2Save/restore EL1 sys_regs
Guest Hypervisor
Save/restore el2_regs
Hypervisor-Hypervisor Switch
![Page 18: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/18.jpg)
• Define mapping of EL2 registers to EL1 registers
• Example: TTBR0_EL2 to TTBR0_EL1
• Example: SCTLR_EL2 adapted to SCTLR_EL1
• Shadow EL1 registers
Emulating EL2 in EL1
![Page 19: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/19.jpg)
&sys_regs
u64 *vcpu->ctxtx.hw_regs
&shadow_sys_regs
PSTATE.mode == EL2PSTATE.mode == EL0/1
Shadow Registers
![Page 20: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/20.jpg)
• Trap to virtual EL2
• “Forward” exceptions
• Emulate virtual exceptions
VM
EL0
EL1
EL2 Host KVM
Kernel
User Space
Guest KVMvEL2
Virtual Exceptions
![Page 21: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/21.jpg)
• Returning from virtual EL2
• Trap eret to EL2 (ARMv8.3)
• Emulate virtual exception return
VM
EL0
EL1
EL2 Host KVM
Kernel
User Space
Guest KVMvEL2
Virtual Exceptions
![Page 22: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/22.jpg)
• CPU Virtualization
• Memory Virtualization
• Timer Virtualization
• Interrupt Virtualization
KVM/ARM Nested Virtualization
![Page 23: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/23.jpg)
Virtual Address (VA)
Physical Address (PA)
Memory Virtualization
![Page 24: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/24.jpg)
Virtual Address (VA)
Physical Address (PA)
Intermediate Physical Address (IPA)
Memory Virtualization
Stage 1: VM kernel
Stage 2: Hypervisor
![Page 25: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/25.jpg)
Virtual Address (VA)
Physical Address (PA)
Intermediate Physical Address (IPA)
Stage 1: Nested VM kernel
Nested Intermediate Physical Address
Stage 2: Host hypervisor
Stage ?: Guest hypervisor
Nested Memory Virtualization
![Page 26: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/26.jpg)
Virtual Address (VA)
Physical Address (PA)
Intermediate Physical Address (IPA)
Nested Memory Virtualization
Stage 1: Nested VM kernel
Stage 2: Host hypervisorShadowStage 2
Page Table
![Page 27: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/27.jpg)
Shadow Stage 2Page Tables
• Translate IPA to PA
• Entries are created by host KVM by walking guest hypervisor stage 2 page tables in software
VM
EL0
EL1
EL2 Host KVM
Kernel
User Space
vEL2 Guest KVM
IPA -> VM PA
VA -> IPA
VM PA -> PA
![Page 28: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/28.jpg)
• CPU Virtualization
• Memory Virtualization
• Timer Virtualization
• Interrupt Virtualization
KVM/ARM Nested Virtualization
![Page 29: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/29.jpg)
• ARM provides a virtual and physical timer in EL1
• EL2 provides a separate EL2 “hyp” timer
• KVM must emulate a VM with EL2 and the hyp timer
Nested Timer Virtualization
![Page 30: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/30.jpg)
• CPU Virtualization
• Memory Virtualization
• Timer Virtualization
• Interrupt Virtualization
KVM/ARM Nested Virtualization
![Page 31: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/31.jpg)
ARM Generic Interrupt Controller (GIC)
GIC
CPU 0
CPU 1
CPUInterface
CPUInterface
Dist.
IRQ
ACK
DeviceInterrupt
Lines
![Page 32: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/32.jpg)
ARM Generic Interrupt Controller (GIC)
GIC
CPU 0
CPU 1
CPUInterface
CPUInterface
Dist.
IRQ
ACK
Virtual CPUInterface
Virtual CPUInterface
VIRQ
ACK
List Registers (LRs)
List Registers (LRs)
![Page 33: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/33.jpg)
VMNested VM
• Deliver both virtual and nested virtual interrupts using the GIC
• Multi-level virtualization using single-level virtualization hardware [Turtles - OSDI ‘10]
Nested Interrupt Virtualization
Host VMM
Kernel
User Space
Guest VMM
Virtual CPUInterface
LRs
![Page 34: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/34.jpg)
• Shadow LRs
• Guest hypervisor traps when attempting to program virtual LRs
• Host hypervisor handles traps by writing to shadow LRs
• Hardware uses shadow LRs when running the nested VM
Nested Interrupt Virtualization
![Page 35: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/35.jpg)
Implementation Status
• RFC v1 on @kvmarm by Jintack Lim (Columbia University)
• CONFIG_KVM_ARM_NESTED_HYP
• vcpu->arch.features & KVM_ARM_VCPU_NESTED_VIRT
![Page 36: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/36.jpg)
To Do• Must expose EL2 registers to user space
• Mostly scattered out over existing files. Should we try to isolate more?
• Hard-coded addresses and interrupt numbers
• Reverse map for shadow stage 2 page tables
• More efficient emulation of TLBI instructions
• Get rid of config option and use command line parameter instead
• Hypercalls from the VM vs. virtual self-hypercalls and PSCI
![Page 37: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/37.jpg)
Questions?and please review the patches…
![Page 38: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/38.jpg)
Backup Slides
![Page 39: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/39.jpg)
KVM/ARM Nested Virtualization
• VHE is fun with nested virtualization
• We don’t set the E2H bit
• The VM thinks it runs in EL2 using VHE so uses EL1 register accesses to access EL2 registers
• But really does run in EL1 and doesn’t need to trap except on a few registers with different bit configuration
CPU Virtualization - VHE
![Page 40: BUD17-301: Zeno's paradox: Optimizing KVM/ARM](https://reader034.vdocuments.us/reader034/viewer/2022042611/58ce8ea51a28ab8c3b8b4dfb/html5/thumbnails/40.jpg)
KVM/ARM Nested Virtualization
• EL2 is separate translation regime
• No ASIDs - cannot alias with EL1 translations
• VMID not used
• Emulating virtual EL2 in EL1
• Separate VMID for virtual EL2
• Always use ASID 0
Memory Virtualization - VMIDs and ASIDs