nested arm virt - kvm · 2017-12-22 · conclusion • we have an implementation of kvm/arm for...
TRANSCRIPT
![Page 1: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/1.jpg)
connect.linaro.org
LEADING COLLABORATION
IN THE ARM ECOSYSTEM
Nested Virtualization on ARMNEVE: Nested Virtualization Extensions
[email protected] [email protected] [email protected], [email protected] [email protected]
Jin Tack Lim Christoffer Dall Shih-Wei Li Jason Nieh Marc Zyngier
![Page 2: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/2.jpg)
Hardware
Hypervisor
VM
VM
Kernel
App App
Nested Virtualization
Hypervisor
VM
Kernel
App App
VM
Kernel
App App
![Page 3: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/3.jpg)
Terminology
Hardware
Host Hypervisor
VM
VM
Kernel
App App
Guest Hypervisor
Nested VM
Kernel
App App
Nested VM
Kernel
App App
![Page 4: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/4.jpg)
Use Cases
1. Run guest operating systems with built-in virtualization.
2. IaaS hosting private clouds
3. Test your hypervisor in a VM
4. Debug your hypervisor in a VM
5. Develop hypervisors using a cloud
![Page 5: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/5.jpg)
VM
ARM Virtualization Extensions
EL0
EL1
EL2 Hypervisor
Kernel
User Space
VM
Kernel
User Space
![Page 6: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/6.jpg)
VM
ARM Nested Virtualization
EL0
EL1
EL2 Host Hypervisor
Kernel
User Space
VM
Kernel
User Space
VirtualEL2 Guest Hypervisor Guest Hypervisor
![Page 7: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/7.jpg)
VM
ARM Nested Virtualization
EL0
EL1
EL2 Host Hypervisor
Kernel
User Space
VM
Kernel
User Space
EL ?? Guest Hypervisor Guest Hypervisor
![Page 8: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/8.jpg)
VM
ARMv8.0 Nested Virtualization
EL0
EL1
EL2 Host Hypervisor
Kernel
User Space
VM
Kernel
User Space
EL0 Guest Hypervisor Guest Hypervisor
Trap-and-emulate
![Page 9: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/9.jpg)
VM
ARMv8.0 Nested Virtualization
EL0
EL1
EL2 Host Hypervisor
Kernel
User Space
VM
Kernel
User Space
EL1 Guest Hypervisor Guest Hypervisor
?? -and-emulate
![Page 10: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/10.jpg)
VM
ARMv8.3 Nested Virtualization
EL0
EL1
EL2 Host Hypervisor
Kernel
User Space
EL1 Guest Hypervisor
Trap -and-emulate
• Gives you software emulation of vEL2 in EL1
• HCR_EL2.NV:
• Traps EL2 operations executed in EL1 to EL2
• Traps eret to EL2
• CurrentEL reports EL2 even in EL1
![Page 11: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/11.jpg)
KVM/ARM Nested Virtualization Implementation
• EL2 Emulation
• Stage 2 MMU Virtualization
• Hyp Timer Virtualization
• Nested Virtual Interrupts
![Page 12: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/12.jpg)
Nested CPU Virtualization
struct kvm_cpu_context { u64 sys_regs[NR_SYS_REGS]; + u64 el2_regs[NR_EL2_REGS]; }
struct kvm_vcpu_arch { … struct kvm_cpu_context ctxt; }
![Page 13: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/13.jpg)
Host
Linux
AppApp
VM
Kernel
AppApp
KVM
EL0
EL1
EL2Restore EL1 sys_regs
Save EL1 sys_regs
Hypervisor-VM Switch
![Page 14: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/14.jpg)
Host
Linux
AppApp
VM
Kernel
KVM
EL0
EL1
EL2Save/restore EL1 sys_regs
Guest Hypervisor
Save/restore el2_regs
Hypervisor-Hypervisor Switch
![Page 15: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/15.jpg)
Emulating EL2 in EL1
• Define mapping of EL2 registers to EL1 registers
• Example: TTBR0_EL2 to TTBR0_EL1
• Example: SCTLR_EL2 adapted to SCTLR_EL1
• Shadow EL1 registers
![Page 16: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/16.jpg)
Nested CPU Virtualization struct kvm_cpu_context { u64 sys_regs[NR_SYS_REGS]; + u64 el2_regs[NR_EL2_REGS]; + u64 shaow_sys_regs[NR_SYS_REGS]; }
struct kvm_vcpu_arch { … struct kvm_cpu_context ctxt; }
![Page 17: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/17.jpg)
&sys_regs
u64 *vcpu->ctxtx.hw_regs
&shadow_sys_regs
PSTATE.mode == EL2PSTATE.mode == EL0/1
Shadow Registers
![Page 18: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/18.jpg)
Virtual Exceptions
• Trap to virtual EL2
• “Forward” exceptions
• Emulate virtual exceptions
VM
EL0
EL1
EL2 Host KVM
Kernel
User Space
Guest KVMvEL2
![Page 19: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/19.jpg)
Virtual Exceptions
• Returning from virtual EL2
• Trap eret to EL2 (ARMv8.3)
• Emulate virtual exception return
VM
EL0
EL1
EL2 Host KVM
Kernel
User Space
Guest KVMvEL2
![Page 20: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/20.jpg)
KVM/ARM Nested Virtualization Implementation
• EL2 Emulation
• Stage 2 MMU Virtualization
• Hyp Timer Virtualization
• Nested Virtual Interrupts
![Page 21: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/21.jpg)
Memory Virtualization
EL0
EL1 Kernel
User Space
Stage 1: VA -> IPA
![Page 22: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/22.jpg)
VM
Memory Virtualization
Host HypervisorEL2
EL0
EL1 Kernel
User Space
Stage 1: VA -> IPA
Stage 2: IPA -> PA
![Page 23: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/23.jpg)
VMNested VM
Memory Virtualization
Host HypervisorEL2
EL0
EL1 Kernel
User Space
Guest Hypervisor
????
Stage 1: VA -> IPA
Stage 2: IPA -> PA
![Page 24: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/24.jpg)
VMNested VM
Memory Virtualization
Host HypervisorEL2
Shadow Stage 2:IPA -> PA
EL0
EL1 Kernel
User Space
Stage 1: VA -> IPA
Guest Hypervisor
Virtual stage 2
![Page 25: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/25.jpg)
KVM/ARM Nested Virtualization Implementation
• EL2 Emulation
• Stage 2 MMU Virtualization
• Hyp Timer Virtualization
• Nested Virtual Interrupts
![Page 26: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/26.jpg)
Nested Timer Virtualization
• ARM provides a virtual and physical timer in EL1
• EL2 provides a separate EL2 “hyp” timer
• Nested KVM/ARM supports a virtual CPU with EL2 and the hyp timer
![Page 27: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/27.jpg)
KVM/ARM Nested Virtualization Implementation
• EL2 Emulation
• Stage 2 MMU Virtualization
• Hyp Timer Virtualization
• Nested Virtual Interrupts
![Page 28: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/28.jpg)
ARM Generic Interrupt Controller (GIC)
GIC
CPU
CPUInterface
CPUInterface
Dist.
IRQ
ACK/EOI
DeviceInterrupt
Lines
![Page 29: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/29.jpg)
ARM Generic Interrupt Controller (GIC)
GIC
CPUInterface
CPUInterface
Dist.
IRQ
ACK/EOI
Virtual CPUInterface
Virtual CPUInterface
VIRQ
ACK/EOI
List Registers (LRs)
List Registers (LRs)
CPU
![Page 30: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/30.jpg)
VMNested VM
Nested Interrupt Virtualization
• Deliver virtual interrupts from the host to the VM
Host VMM
Kernel
User Space
Guest VMM
Virtual CPUInterface
LRs
![Page 31: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/31.jpg)
VMNested VM
Nested Interrupt Virtualization
Host VMM
Kernel
User Space
Guest VMM
Virtual CPUInterface
LRs
• Deliver virtual interrupts from the guest hypervisor to the nested VM
• Shadow list registers
• The nested VM can ACK and EOI virtual interrupts without trapping
![Page 32: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/32.jpg)
Performance Evaluation
• Problem: No ARMv8.3 hardware available.
• Solution: Use ARMv8.0 hardware with the software modification
![Page 33: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/33.jpg)
Emulating v8.3 on v8.0
Host Hypervisor
VM
EL1
EL0
EL2
Guest Hypervisor
Nested VM
OS Kernel
App AppApp
ARMv8.0 Hardware
ParavirtualizationHVC HVCHVCHVC HVC
![Page 34: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/34.jpg)
Hypercall MicroBenchmark
Hypervisor
VM
OS Kernel
App AppApp
EL1
EL0
EL2Hypercall Return
Host Hypervisor
VM
EL1
EL0
EL2
Guest Hypervisor
Nested VM
OS Kernel
App AppApp
Hypercall Return
![Page 35: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/35.jpg)
Hypercall MicroBenchmark
ARMv8.3
VM Nested VM
Cycle counts 2,729 422,720
Ratio to VM 1 155x
![Page 36: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/36.jpg)
Application Benchmarks
01020304050 ARMv8.3 VM
ARMv8.3 Nested
012345
Kernbench
Hackbench
SPECjvm2008TCP RR
TCP STREAM
TCP MAERTSApache
Nginx
MemcachedMySQL
Normalized overhead (lower is better)
![Page 37: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/37.jpg)
Nested VM Exit/Entry on ARM
Host Hypervisor
VM
EL1
EL0
EL2
Guest Hypervisor
Nested VM
OS Kernel
App AppApp
VM EntryVM ExitEL1 Registers
EL2 Registers
> 120 traps
…
![Page 38: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/38.jpg)
NEVE: NEsted Virtualization Extensions for ARM
• Supports unmodified guest hypervisors and OSes
• Improves performance by providing register redirection
![Page 39: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/39.jpg)
Register Classification• VM registers: EL1 registers only affecting the nested VM’s execution
• Hypervisor registers: EL2 registers affecting the hypervisor’s execution
![Page 40: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/40.jpg)
VM Registers
Host Hypervisor
VM
EL1
EL0
EL2
Guest Hypervisor
Nested VM
OS Kernel
App AppApp
VM EntryVM ExitEL1 Registers
…
This is when VM register states are used
![Page 41: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/41.jpg)
VM Registers: Logging to Memory
VM Register
msr x0, TTBR0_EL1
Without NEVE Trap!
Memory
![Page 42: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/42.jpg)
VM Registers: Logging to Memory
msr x0, TTBR0_EL1
TTBR0_EL1
MemoryWith NEVE
VM Register
![Page 43: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/43.jpg)
Hypervisor control registers
Host Hypervisor
EL1
EL2
Guest Hypervisor EL1 Registers
EL2 Registers
• Can’t apply the technique for VM registers
• They have an immediate impact (EL2 system registers)
• Traps are handled by redirecting to EL1 registers in software
![Page 44: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/44.jpg)
Hypervisor control registers• Can’t apply the technique for VM registers
• They have an immediate impact (EL2 system registers)
• Traps are handled by redirecting to EL1 registers in software
• Redirect in hardware instead!
Host Hypervisor
EL1
EL2
Guest Hypervisor EL1 Registers
EL2 Registers
![Page 45: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/45.jpg)
Hypercall MicroBenchmark
ARMv8.3 NEVE
VM Nested VM Nested VM
Cycle counts 2,729 422,720 92,385
Ratio to VM 155x 34x
Trap counts 1 126 15
![Page 46: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/46.jpg)
Application Workloads
Application Description Application Description
Kernbench Kernel compile Netperf TCP_RR Network performance
Hackbench Scheduler stress Netperf TCP STREAM Network performance
SPECjvm2008 Java Runtime Netperf TCP MAERTS Network performance
MySQL Database management Apache Web server stress
Memcached Key-Value store Nginx Web server stress
![Page 47: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/47.jpg)
Experimental Setup• ARM Hardware• APM X-Gene (ARMv8.0)• 8-way SMP• 64 GB RAM
• Software
• KVM on KVM
• v4.10
• Native/VM/Nested VM
• 4-way SMP
• 12 GB RAM
• Virt I/O(VM/nested VM)
• 10 Gb Ethernet
• x86 Hardware• Intel E5-2630 v3 • VMCS Shadowing• 8-way SMP• 128 GB RAM
![Page 48: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/48.jpg)
Application Benchmarks
01020304050 ARMv8.3 VM
ARMv8.3 NestedNEVE Nested
012345
Kernbench
Hackbench
SPECjvm2008TCP RR
TCP STREAM
TCP MAERTSApache
Nginx
MemcachedMySQL
Normalized overhead (lower is better)
![Page 49: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/49.jpg)
Application Benchmarks
01020304050 ARMv8.3 VM
ARMv8.3 NestedNEVE Nestedx86 Nested VM
012345
Kernbench
Hackbench
SPECjvm2008TCP RR
TCP STREAM
TCP MAERTSApache
Nginx
MemcachedMySQL
Normalized overhead (lower is better)
![Page 50: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/50.jpg)
Conclusion• We have an implementation of KVM/ARM for v8.3
• Evaluated nested virtualization performance by emulating ARMv8.3
• Nested virtualization on ARMv8.3 incurs high overhead
• Due to the exit multiplication problem
• NEVE enhances performance significantly by reducing number of traps
• NEVE is used as basis for extended nested virtualization support in ARMv8.4
• NEVE to appear at SOSP later month - read the paper for more details
![Page 51: Nested ARM Virt - KVM · 2017-12-22 · Conclusion • We have an implementation of KVM/ARM for v8.3 • Evaluated nested virtualization performance by emulating ARMv8.3 • Nested](https://reader033.vdocuments.us/reader033/viewer/2022060321/5f0d4db47e708231d439ae3a/html5/thumbnails/51.jpg)
Code• Nested CPU Virtualization patches for ARMv8.3 [RFC v2]:
https://lists.cs.columbia.edu/pipermail/kvmarm/2017-July/026388.html
• Nested Memory Virtualization patches for ARMv8.3 [RFC]:https://lists.cs.columbia.edu/pipermail/kvmarm/2017-October/027286.html
• v8.3 and NEVE Paravirtualization on Linux v4.12-rc1:https://github.com/columbia/nesting-pub
• QEMU Patches:https://github.com/columbia/qemu-pub nested-v2.3.0-model