continuous integration: jenkins, libvirt & real …m.pdfcontinuous integration: jenkins, libvirt...
TRANSCRIPT
![Page 1: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/1.jpg)
Continuous Integration:Jenkins, Libvirt & Real Hardware
Anna-Maria Gleixner <[email protected]>
Manuel Traut <[email protected]>
ELCE 2017 - Prague
aM
![Page 2: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/2.jpg)
What's next? 1. Overview
2. The challenge: control real HW
3. Solution - r4d
4. Libvirt connection
5. Jenkins
6. Future extensions
![Page 3: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/3.jpg)
1. Overview★ RTL project
★ CI-RT
★ Jenkins
★ Libvirt
![Page 4: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/4.jpg)
RTL project ● Linuxfoundation collaborative project
● first announced in October 2015
● funds Linux Kernel PREEMPT_RT patch set ...
○ mainlining
○ maintenance
○ documentation
(new RT Wiki: https://wiki.linuxfoundation.org/realtime/start )
○ establishing community - required for long term maintenance
○ CI-RT
![Page 5: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/5.jpg)
CI-RT
● PREEMPT_RT Linux kernel tests on different machines (HW & VM)
○ Power control machines
○ Retrieve bootlog even if kernel crashes and doesn't boot into user-space
○ Schedule different tests on a machine
![Page 6: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/6.jpg)
Jenkins
● De facto standard for CI
● Is able to distribute tests on different machines
available Jenkins plugins to control machines via
● docker
● libvirt
![Page 7: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/7.jpg)
Libvirt
● Supports different hypervisors (via drivers)
● Standardized interface to enumerate
and control VMs
● Graphical frontends like 'virt-manager'
and CLI like 'virsh' available
BUT:
Can't interact with real HW
![Page 8: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/8.jpg)
2. The challenge ★ We want to use ....
★ Vision
M
![Page 9: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/9.jpg)
We want to use … libvirt
● Restart node on kernel-crash
● Tests on VM behave like tests on real HW
● Nodes that are not used are powered off
![Page 10: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/10.jpg)
We want to use … existing racks
● 19” industrial rack with 8 slots
● 8 port power control switch
● 8 port serial device server
● network switch
![Page 11: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/11.jpg)
Vision
![Page 12: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/12.jpg)
3. Solution - r4d ★ Architecture
★ Features
★ Usage
![Page 13: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/13.jpg)
Architecture
source: http://www.senanetworks.com/
source: https://gude.info/
![Page 14: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/14.jpg)
Features
● multiple DB backends supported (sqlalchemy)
● Plugin interface to support different power controls and serial device servers
● CLI tool to ...
○ add new racks, power-control, serial device servers and boards
○ move board into another rack/slot
○ switch board on/off
○ dump whole DB or only parts of it
![Page 15: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/15.jpg)
Usage - basic setup
$ r4dcfg --add-rack ci-rt-1 room209
$ r4dcfg --add-power ci-rt-1 pc8210 pc-ci-rt-1.lab.linutronix.de
$ r4dcfg --add-serial ci-rt-1 PS810 ds-ci-rt-1.lab.linutronix.de
$ r4dcfg --add-board ci-rt-1 6 seattle
![Page 16: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/16.jpg)
Usage - list-boards
![Page 17: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/17.jpg)
4. Libvirt connection ★ Architecture
★ Code changes
★ Usage
aM
![Page 18: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/18.jpg)
Architecture
![Page 19: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/19.jpg)
Code changes
![Page 20: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/20.jpg)
Code changes r4d_driver.c
r4dConnectOpen()
r4dConnectClose()
r4dConnectNumOfDefinedDomains()
r4dConnectListDefinedDomains()
r4dDomainCreate()
r4dDomainDestroy()
r4dDomainGetState()
r4dDomainOpenConsole()
open connection to r4dd 'hypervisor'
close connection and drop handle
get number of boards configured in r4d
list all boards configured in r4d
power a board
power off a board
returns current power state of a board
open a serial console stream
![Page 21: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/21.jpg)
Usage
![Page 22: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/22.jpg)
5. Jenkins ★ Code changes
★ Add a new node
★ Control the node
M
![Page 23: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/23.jpg)
Code changes
Entry for hypervisor in libvirt-slave-plugin:
![Page 24: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/24.jpg)
Add a new hypervisor
![Page 25: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/25.jpg)
Add a new node
![Page 26: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/26.jpg)
Add a new node
![Page 27: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/27.jpg)
Control node
![Page 28: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/28.jpg)
![Page 29: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/29.jpg)
![Page 30: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/30.jpg)
Future Extensions ★ Various improvements
★ r4d Testbox
![Page 31: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/31.jpg)
Various improvements
● Soft power off
● Authentication
● Post libvirt and jenkins-libvirt-slave-plugin patches
● ...
![Page 32: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/32.jpg)
r4d Testbox
● Redirect USB serial
● GPIO to control power buttons
● Interface tests for e.g. CAN
● Could serve as an all in one: serial device
server and power control server
aM
![Page 33: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/33.jpg)
References CI-RT
https://ci-rt.linutronix.de
R4D
https://github.com/ci-rt/r4d
https://github.com/ci-rt/libr4d
Libvirt
https://github.com/ci-rt/libvirt-debian
Jenkins
https://github.com/ci-rt/libvirt-slave-plugin
Thanks Benedikt Spranger
![Page 34: Continuous Integration: Jenkins, Libvirt & Real …M.pdfContinuous Integration: Jenkins, Libvirt & Real Hardware ... CI-RT. CI-RT PREEMPT_RT ... //github.com/ci-rt/libvirt-debian Jenkins](https://reader031.vdocuments.us/reader031/viewer/2022021505/5b00b0757f8b9af1148d00f1/html5/thumbnails/34.jpg)
Contact Anna-Maria Gleixner
Manuel Traut
Linutronix GmbH
https://linutronix.de
Bahnhofstraße 3
88690 Uhldingen
Germany