virtual machine boot-up analysis - polytechnique montréal - virtual machin… · virtual machine...
TRANSCRIPT
![Page 1: Virtual Machine Boot-up Analysis - Polytechnique Montréal - virtual machin… · Virtual Machine Boot-up Analysis Abderrahmane Benbachir May 5, 2017 École Polytechnique de Montréal](https://reader030.vdocuments.us/reader030/viewer/2022041012/5ebf07fb06152502072ae7ca/html5/thumbnails/1.jpg)
Virtual Machine Boot-upAnalysis
Abderrahmane BenbachirMay 5, 2017
École Polytechnique de Montréal
Laboratoire DORSAL
![Page 2: Virtual Machine Boot-up Analysis - Polytechnique Montréal - virtual machin… · Virtual Machine Boot-up Analysis Abderrahmane Benbachir May 5, 2017 École Polytechnique de Montréal](https://reader030.vdocuments.us/reader030/viewer/2022041012/5ebf07fb06152502072ae7ca/html5/thumbnails/2.jpg)
Agenda
IntroductionTracing Through Para-virtualization LayersInvestigating Boot-up IssuesQuestions
POLYTECHNIQUE MONTREAL – Abderrahmane Benbachir (Abder)
![Page 3: Virtual Machine Boot-up Analysis - Polytechnique Montréal - virtual machin… · Virtual Machine Boot-up Analysis Abderrahmane Benbachir May 5, 2017 École Polytechnique de Montréal](https://reader030.vdocuments.us/reader030/viewer/2022041012/5ebf07fb06152502072ae7ca/html5/thumbnails/3.jpg)
Introduction
POLYTECHNIQUE MONTREAL – Abderrahmane Benbachir (Abder)
![Page 4: Virtual Machine Boot-up Analysis - Polytechnique Montréal - virtual machin… · Virtual Machine Boot-up Analysis Abderrahmane Benbachir May 5, 2017 École Polytechnique de Montréal](https://reader030.vdocuments.us/reader030/viewer/2022041012/5ebf07fb06152502072ae7ca/html5/thumbnails/4.jpg)
Boot-up process
Init process
BIOS + MBR + Bootloader
Execute /sbin/init, initramfs root
LTTng, Ftrace, Perf, ...
Ftrace handle this part
No tracer, a few printk
Kernel
CPU, Memory, Bus,IRQ, …
Load built-in Modules
Run threads
Initializations
POLYTECHNIQUE MONTREAL – Abderrahmane Benbachir (Abder)
![Page 5: Virtual Machine Boot-up Analysis - Polytechnique Montréal - virtual machin… · Virtual Machine Boot-up Analysis Abderrahmane Benbachir May 5, 2017 École Polytechnique de Montréal](https://reader030.vdocuments.us/reader030/viewer/2022041012/5ebf07fb06152502072ae7ca/html5/thumbnails/5.jpg)
Tracing through para-virtualization layers
POLYTECHNIQUE MONTREAL – Abderrahmane Benbachir (Abder)
![Page 6: Virtual Machine Boot-up Analysis - Polytechnique Montréal - virtual machin… · Virtual Machine Boot-up Analysis Abderrahmane Benbachir May 5, 2017 École Polytechnique de Montréal](https://reader030.vdocuments.us/reader030/viewer/2022041012/5ebf07fb06152502072ae7ca/html5/thumbnails/6.jpg)
Hypercall
Host Kernel
Qemu
KVM.KO
VM#1
Paravirtualized Guest OS
Direct Execution of Guest Requests
Hardware
Layer notinvolvedH
yper
call
POLYTECHNIQUE MONTREAL – Abderrahmane Benbachir (Abder)
![Page 7: Virtual Machine Boot-up Analysis - Polytechnique Montréal - virtual machin… · Virtual Machine Boot-up Analysis Abderrahmane Benbachir May 5, 2017 École Polytechnique de Montréal](https://reader030.vdocuments.us/reader030/viewer/2022041012/5ebf07fb06152502072ae7ca/html5/thumbnails/7.jpg)
Micro-Benchmarks310 ns : VMCS switch + kvm_exit + kvm_entry
Hypercall Overhead
Mean 340 ns
Median 310 ns
Std deviation 833 ns323 ns
Hardware : Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
(x86_64)
POLYTECHNIQUE MONTREAL – Abderrahmane Benbachir (Abder)
![Page 8: Virtual Machine Boot-up Analysis - Polytechnique Montréal - virtual machin… · Virtual Machine Boot-up Analysis Abderrahmane Benbachir May 5, 2017 École Polytechnique de Montréal](https://reader030.vdocuments.us/reader030/viewer/2022041012/5ebf07fb06152502072ae7ca/html5/thumbnails/8.jpg)
Implementation
ftrace
function_graph
Irqsoff
function
wakeup
hypergraphnop
sched_switch
POLYTECHNIQUE MONTREAL – Abderrahmane Benbachir (Abder)POLYTECHNIQUE MONTREAL – Abderrahmane Benbachir (Abder)
![Page 9: Virtual Machine Boot-up Analysis - Polytechnique Montréal - virtual machin… · Virtual Machine Boot-up Analysis Abderrahmane Benbachir May 5, 2017 École Polytechnique de Montréal](https://reader030.vdocuments.us/reader030/viewer/2022041012/5ebf07fb06152502072ae7ca/html5/thumbnails/9.jpg)
Implementation
• Hook to Ftrace “function graph” probe for entry & exit
• Send function address
• Tracing only the hypervisor
• Use only host Timestamp
-> No synchronization is required
• Resolve addresses using guest kernel symbols
Tracing through Hypercalls
POLYTECHNIQUE MONTREAL – Abderrahmane Benbachir (Abder)
![Page 10: Virtual Machine Boot-up Analysis - Polytechnique Montréal - virtual machin… · Virtual Machine Boot-up Analysis Abderrahmane Benbachir May 5, 2017 École Polytechnique de Montréal](https://reader030.vdocuments.us/reader030/viewer/2022041012/5ebf07fb06152502072ae7ca/html5/thumbnails/10.jpg)
Ftrace : Mcount and hypercall
Instrumentedfunction
mcount
Function entrytracer
Function exittracer
Return hook
Change returnentry to hook
1
2
3
return
4
6
5
hypercall
hypercall
Caller
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
POLYTECHNIQUE MONTREAL – Abderrahmane Benbachir (Abder)
![Page 11: Virtual Machine Boot-up Analysis - Polytechnique Montréal - virtual machin… · Virtual Machine Boot-up Analysis Abderrahmane Benbachir May 5, 2017 École Polytechnique de Montréal](https://reader030.vdocuments.us/reader030/viewer/2022041012/5ebf07fb06152502072ae7ca/html5/thumbnails/11.jpg)
Hypervisor / Host
Depth Overhead Analysis Hypercall
A
B C
hypercall hypercall
kvm_exit kvm_entry
Context Switch Overhead
Function entry Function exit
Hypervisor Overhead
D
E
POLYTECHNIQUE MONTREAL – Abderrahmane Benbachir (Abder)
![Page 12: Virtual Machine Boot-up Analysis - Polytechnique Montréal - virtual machin… · Virtual Machine Boot-up Analysis Abderrahmane Benbachir May 5, 2017 École Polytechnique de Montréal](https://reader030.vdocuments.us/reader030/viewer/2022041012/5ebf07fb06152502072ae7ca/html5/thumbnails/12.jpg)
Investigating Boot-up issues
Kernel Crash/PanicLatencies
POLYTECHNIQUE MONTREAL – Abderrahmane Benbachir (Abder)
![Page 13: Virtual Machine Boot-up Analysis - Polytechnique Montréal - virtual machin… · Virtual Machine Boot-up Analysis Abderrahmane Benbachir May 5, 2017 École Polytechnique de Montréal](https://reader030.vdocuments.us/reader030/viewer/2022041012/5ebf07fb06152502072ae7ca/html5/thumbnails/13.jpg)
Boot-up issues
A
CB
D
E F
POLYTECHNIQUE MONTREAL – Abderrahmane Benbachir (Abder)
![Page 14: Virtual Machine Boot-up Analysis - Polytechnique Montréal - virtual machin… · Virtual Machine Boot-up Analysis Abderrahmane Benbachir May 5, 2017 École Polytechnique de Montréal](https://reader030.vdocuments.us/reader030/viewer/2022041012/5ebf07fb06152502072ae7ca/html5/thumbnails/14.jpg)
Boot-up issues
A
CB
D
E F
hypervisor
Entry
Exit
Hypercall overhead
A
B C
D
E F
Received sequence : A-B-C-D-E-F
POLYTECHNIQUE MONTREAL – Abderrahmane Benbachir (Abder)
![Page 15: Virtual Machine Boot-up Analysis - Polytechnique Montréal - virtual machin… · Virtual Machine Boot-up Analysis Abderrahmane Benbachir May 5, 2017 École Polytechnique de Montréal](https://reader030.vdocuments.us/reader030/viewer/2022041012/5ebf07fb06152502072ae7ca/html5/thumbnails/15.jpg)
Boot-up issues Kernel crash : Oops, panic, … etc.
A
CB
D
E F
hypervisor
Entry
Exit
Hypercall overhead
Received sequence : A-B-C-D-E-FA
B C
D
E F
Call stack
Kernel panic
POLYTECHNIQUE MONTREAL – Abderrahmane Benbachir (Abder)
![Page 16: Virtual Machine Boot-up Analysis - Polytechnique Montréal - virtual machin… · Virtual Machine Boot-up Analysis Abderrahmane Benbachir May 5, 2017 École Polytechnique de Montréal](https://reader030.vdocuments.us/reader030/viewer/2022041012/5ebf07fb06152502072ae7ca/html5/thumbnails/16.jpg)
Boot-up issues
→ Reduce by half the trace events
A
B
hypercall
Function entry Function exit
Trace only exits + duration
C
D
E
→ Better for Performance Analysis
POLYTECHNIQUE MONTREAL – Abderrahmane Benbachir (Abder)
![Page 17: Virtual Machine Boot-up Analysis - Polytechnique Montréal - virtual machin… · Virtual Machine Boot-up Analysis Abderrahmane Benbachir May 5, 2017 École Polytechnique de Montréal](https://reader030.vdocuments.us/reader030/viewer/2022041012/5ebf07fb06152502072ae7ca/html5/thumbnails/17.jpg)
Boot-up issues Post-order tree traversal
A
CB
D
E F
hypervisor
Received sequence : B-E-F-D-C-AA
B C
D
E F
Hypercall overhead
POLYTECHNIQUE MONTREAL – Abderrahmane Benbachir (Abder)
![Page 18: Virtual Machine Boot-up Analysis - Polytechnique Montréal - virtual machin… · Virtual Machine Boot-up Analysis Abderrahmane Benbachir May 5, 2017 École Polytechnique de Montréal](https://reader030.vdocuments.us/reader030/viewer/2022041012/5ebf07fb06152502072ae7ca/html5/thumbnails/18.jpg)
Boot-up issues Kernel crash : Oops, panic, … etc.
A
CB
D
E F
hypervisor
Received sequence : B-E
A
B C
D
E F
Kernel panic
Hypercall overhead
→ Not good for debuggingkernel crach
POLYTECHNIQUE MONTREAL – Abderrahmane Benbachir (Abder)
![Page 19: Virtual Machine Boot-up Analysis - Polytechnique Montréal - virtual machin… · Virtual Machine Boot-up Analysis Abderrahmane Benbachir May 5, 2017 École Polytechnique de Montréal](https://reader030.vdocuments.us/reader030/viewer/2022041012/5ebf07fb06152502072ae7ca/html5/thumbnails/19.jpg)
Visualization
POLYTECHNIQUE MONTREAL – Abderrahmane Benbachir (Abder)
![Page 20: Virtual Machine Boot-up Analysis - Polytechnique Montréal - virtual machin… · Virtual Machine Boot-up Analysis Abderrahmane Benbachir May 5, 2017 École Polytechnique de Montréal](https://reader030.vdocuments.us/reader030/viewer/2022041012/5ebf07fb06152502072ae7ca/html5/thumbnails/20.jpg)
Call graph TraceCompass : Callstack XML analysis
Graph of Guest Kernel space
Thanks to GenevièvePOLYTECHNIQUE MONTREAL – Abderrahmane Benbachir (Abder)
![Page 21: Virtual Machine Boot-up Analysis - Polytechnique Montréal - virtual machin… · Virtual Machine Boot-up Analysis Abderrahmane Benbachir May 5, 2017 École Polytechnique de Montréal](https://reader030.vdocuments.us/reader030/viewer/2022041012/5ebf07fb06152502072ae7ca/html5/thumbnails/21.jpg)
Call graph Console output
POLYTECHNIQUE MONTREAL – Abderrahmane Benbachir (Abder)
![Page 22: Virtual Machine Boot-up Analysis - Polytechnique Montréal - virtual machin… · Virtual Machine Boot-up Analysis Abderrahmane Benbachir May 5, 2017 École Polytechnique de Montréal](https://reader030.vdocuments.us/reader030/viewer/2022041012/5ebf07fb06152502072ae7ca/html5/thumbnails/22.jpg)
Questions
https://github.com/abenbachir
POLYTECHNIQUE MONTREAL – Abderrahmane Benbachir (Abder)
![Page 23: Virtual Machine Boot-up Analysis - Polytechnique Montréal - virtual machin… · Virtual Machine Boot-up Analysis Abderrahmane Benbachir May 5, 2017 École Polytechnique de Montréal](https://reader030.vdocuments.us/reader030/viewer/2022041012/5ebf07fb06152502072ae7ca/html5/thumbnails/23.jpg)
References
Hypercall Implementation : https://gist.github.com/abenbachir/344822b5ba9fc5ac384cdec3f087e018
QEMU Hypertrace Patches: http://patchwork.ozlabs.org/project/qemu-devel/list/?state=&q=Hypertrace&archive
Trampoline: https://www.kernel.org/doc/ols/2009/ols2009-pages-47-54.pdf
Callstack xml analysis: https://gist.github.com/abenbachir/e813790f183945b6f74dc74ecee57c75
POLYTECHNIQUE MONTREAL – Abderrahmane Benbachir (Abder)