automation beyond testing and embedded system...
TRANSCRIPT
![Page 1: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/1.jpg)
Slide 1 - http://www.pengutronix.de – 2017-10-23
Automation beyond Testing and
Embedded System Validation
Embedded Linux Conference EuropeJan Lübbe <[email protected]>
![Page 2: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/2.jpg)
Slide 2 - http://www.pengutronix.de – 2017-10-23
Some Background
• Embedded Linux integration and development for custom hardware
• Using Linux mainline, mesa, wayland, gstreamer, Qt, chromium, …
Everything changes all the time⇒
⇒ Updates break user-visible features
Kernel and application level testing“solved” with Jenkins & LAVA
![Page 3: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/3.jpg)
Slide 3 - http://www.pengutronix.de – 2017-10-23
A Short Survey
• Who has developed embedded Linux systems?
– … rolled out a major base-system update?
– … updates the base-system at least once a year?
• Who has automated tests for:
– The application?
– … and the kernel (-drivers)?
– … and the update installer?
– … and the rollback mechanism?
• What do you use (in-house, Jenkins, LAVA, …)?
![Page 4: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/4.jpg)
Slide 4 - http://www.pengutronix.de – 2017-10-23
Current State
• Test automation:
– LAVA, Fuego, autotest, avocado, TI VATF, U-Boot test tool, CI-RT R4D, Baylibe Lab in a Box, …
• Development automation:
– scripting via SSH
– expect
• Production automation:
– flash images via robot
– SoC-vendor-sepecific tools (running on windows)
– ad-hoc scripting
![Page 5: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/5.jpg)
Slide 5 - http://www.pengutronix.de – 2017-10-23
Our Wishlist
• Short turnaround times for interactive use during development
• Support reuse for other use cases
• Use the same board for devel and CI
• Complex state transitions (BL→Linux→update & reboot→BL→new Linux)
• Library interface (for use-cases besides testing)
• Control of additional interfaces (SD, buttons, boot mode, logic analyzer, USB, …)
– supported by TI’s VATF for some special cases
• Multiple targets in one test case
– supported by LAVA
![Page 6: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/6.jpg)
Slide 6 - http://www.pengutronix.de – 2017-10-23
NIH Syndrome?
• All tools are shaped by requirements
• Our use-cases != your use-cases
![Page 7: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/7.jpg)
Slide 7 - http://www.pengutronix.de – 2017-10-23
LAVA - Linaro Automated Validation Architecture
• Used by Linaro, Kernel CI and many others
• lavapdu daemon
good web-interface with useful logs automatic health checks
boards must be dedicated to LAVA long turn-around times
see “Introducing the Lab in a Box Concept” tomorrow (http://sched.co/ByYM)
![Page 8: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/8.jpg)
Slide 8 - http://www.pengutronix.de – 2017-10-23
Fuego
• Used by LF CE WG, LTSI (Long Term Support Initiative), AGL, CIP
• Consists of Jenkins + Scripts + Tests (in Docker)
builds test and deploys test binaries hard to setup on an existing Jenkins instance
![Page 9: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/9.jpg)
Slide 9 - http://www.pengutronix.de – 2017-10-23
U-Boot “pytest suite”
• Lives in u-boot/test/py
• Helper functions to build and control U-Boot
expressive test cases using pytest
only for U-Boot (with build support) no library interface or target abstraction
![Page 10: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/10.jpg)
Slide 10 - http://www.pengutronix.de – 2017-10-23
CI-RT R4D
• Power & serial control
• Implemented as libvirt backend
embedded boards controlled similar to VMseasy to use from Jenkins
libvirt interface does not fit more complex use cases difficult to synchronize multi node tests needs custom code fo rinterfaces besides power and serial
see “CI: Jenkins, libvirt and Real Hardware” (http://sched.co/ByYA)
![Page 11: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/11.jpg)
Slide 11 - http://www.pengutronix.de – 2017-10-23
Heiko Schocher’s tbot
• Python tool to control boards and execute test cases
access to remote boards via SSH fexible event collection for reporting
patch and build support plain python code for testcases (instead of pytest)
![Page 12: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/12.jpg)
Slide 12 - http://www.pengutronix.de – 2017-10-23
Project Specific Tools
• Autotest fork by Google for Chromium OS
• Avocado (another Autotest fork) for libvirt testing
• TI’s VATF
directly contain the testsuites very focused on special requirements only for testing
![Page 13: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/13.jpg)
Slide 13 - http://www.pengutronix.de – 2017-10-23
Shortcomings
• Large overhead for running and writing a single test painful to use during iterative development⇒
• Limited control over the target from the individual test
– no reboots during test case
– no easy control over additional IO (buttons, config switches, USB, …)
• Hard to reuse for other use-cases and one-off tools
– git bisect
– …
![Page 14: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/14.jpg)
Slide 14 - http://www.pengutronix.de – 2017-10-23
Goals
• Make automation useful during normal iterative development
– Upload bootloader via USB
– Control distributed equipment
– Easy test loops
• Support the same tests and tools from a CI environment
• Make it easy to extend and embed
• Connect/automate existing tools (LTP, …)
![Page 15: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/15.jpg)
Slide 15 - http://www.pengutronix.de – 2017-10-23
Try Something Else Less
• no integrated build system (unlike Fuego)
– use OE/PTXdist/buildroot instead
• no integrated test runner (unlike LAVA, autotest, many others)
– use pytest and/or custom scripts
• no scheduler (unlike LAVA, Fuego)
– use Jenkins instead or use from shell
• no fixed boot process (all? others)
– full control from client code
• library interface not only for testing⇒
![Page 16: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/16.jpg)
Slide 16 - http://www.pengutronix.de – 2017-10-23
HW/SW Control as a Library
• Embedded system testing should feel like pure SW testing
• Don’t handle control-flow
• Client code should be high-level
– Similar to what I would tell a colleague to do
![Page 17: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/17.jpg)
Slide 17 - http://www.pengutronix.de – 2017-10-23
Labgrid - Architecture
Serial
ConsoleDriver
ShellDriver
BootloaderDriver
CommandProtocolProtocol
Driver
Resource
![Page 18: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/18.jpg)
Slide 18 - http://www.pengutronix.de – 2017-10-23
Architecture – Targets
Serial PDU
ConsoleDriver
PowerDriver
ShellDriver
BootloaderDriver
Fastboot
FastbootDriver
USB-Loader
BootstrapDriver
StrategyTarget API
![Page 19: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/19.jpg)
Slide 19 - http://www.pengutronix.de – 2017-10-23
Architecture - Flexibility
Serial PDU
SerialDriver
PowerDriver
ShellDriver
BootloaderDriver
Fastboot
FastbootDriver
USB-Loader
BootstrapDriver
StrategyTarget API
PIODriver
1W-PIO
![Page 20: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/20.jpg)
Slide 20 - http://www.pengutronix.de – 2017-10-23
Labgrid - Configuration
• YAML
• Describes Targets with
– Resources
– Drivers
• HW/SW-Specific parameters
• The “Environment”
![Page 21: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/21.jpg)
Slide 21 - http://www.pengutronix.de – 2017-10-23
Labgrid - pytest
• Test execution, selection and reporting is provided by pytest
• Fixtures provide access at different levels (command, strategy, target, env)
• pytest (and Python libs) make it easy to prepare test data and analyze results
• Easy to integrate in Jenkins
![Page 22: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/22.jpg)
Slide 22 - http://www.pengutronix.de – 2017-10-23
![Page 23: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/23.jpg)
Slide 23 - http://www.pengutronix.de – 2017-10-23
![Page 24: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/24.jpg)
Slide 24 - http://www.pengutronix.de – 2017-10-23
Labgrid - CLI
• Configure boards from distributed resoures (“board farms”/“labs”)
• Control power, serial, buttons, fastboot, bootloader upload
• Lock/unlock boards
• Use labgrid strategies
• Usable from shells scripts, CI or other automation (such as LAVA)
labgrid-client -p riot1 locklabgrid-client -p riot1 io high bootmodelabgrid-client -p riot1 pw onlabgrid-client -p riot1 bootstrap …/bootload.imglabgrid-client -p riot1 fastboot boot …/kernel.imglabgrid-cleint -p riot1 console
![Page 25: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/25.jpg)
Slide 25 - http://www.pengutronix.de – 2017-10-23
Labgrid – Remote Control
Coordinator
Exporter
Exporter
Exporter
Exports
Exports
Exports
Direct resource access
Client
ClientAccess
Access
![Page 26: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/26.jpg)
Slide 26 - http://www.pengutronix.de – 2017-10-23
Labgrid - Scripting
• Example: Sometimesa ethernet interfacediscards frames insteadof sending them in~1% of boots.
• Loop until error occours
• Manual investigationafter script exits
![Page 27: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/27.jpg)
Slide 27 - http://www.pengutronix.de – 2017-10-23
Labgrid - Autoinstaller
• Each host manages serveral flashing stations
• Uses USB tree topology for configuration
Target Board
USB
USBHub
UART
USBUART
Target Board
UART USB
USBHub
USBUART
Target Board
UART USB
USBHub
USBUART
https://github.com/labgrid-project/labgrid/blob/master/labgrid/autoinstall/main.py
![Page 28: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/28.jpg)
Slide 28 - http://www.pengutronix.de – 2017-10-23
Demo
![Page 29: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/29.jpg)
Slide 29 - http://www.pengutronix.de – 2017-10-23
Currently Working
• Remotly control boards in lab from CLI (console, power, BL upload, fastboot)
• Run pytest against local and remote boards
• Run tests from Jenkins and collect results via Junit-XML
• Ad-Hoc automation: git bisect, reproducing sporadic errors
• Automatic factory installation via USB directly from built BSPw
• Used as a backend for internal QA tools
![Page 30: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/30.jpg)
Slide 30 - http://www.pengutronix.de – 2017-10-23
Next Steps
• Remote target reservation (for use with Jenkins CI)
• Automatic integration tests for RAUC with QEmu in Jenkins
• Improved logging and reports
• Driver priorities (use ResetProtocol instead of PowerProtocol when available)
• Driver preemption (handle unexpected state changes)
![Page 31: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/31.jpg)
Slide 31 - http://www.pengutronix.de – 2017-10-23
Getting Started
• Wait for 0.2.0 release or git clone master
• Setup in Python venv
• Connect board
• Copy and modify one of the examples
• If it breaks: talk to us! ;-)
https://labgrid.readthedocs.io/en/latest/getting_started.html
![Page 32: Automation beyond Testing and Embedded System Validationevents17.linuxfoundation.org/sites/events/files/... · • Control of additional interfaces (SD, buttons, boot mode, logic](https://reader033.vdocuments.us/reader033/viewer/2022041811/5e57a9ebabbaaa7e4478e2e9/html5/thumbnails/32.jpg)
Slide 32 - http://www.pengutronix.de – 2017-10-23
Discussion
[email protected], @shoragan, +JanLübbe-jlu