workshop remote testing environment - osfc · 2018. 9. 22. · introduction firmware developer...
TRANSCRIPT
Remote Testing EnvironmentWorkshop
OSFC 2018
Piotr Król, Michał Żygowski
1 / 41
IntroductionFirmware developer routinesRemote Testing EnvironmentInfrastructureHardware setupAutomation TestsFixing bugsFuture plans
RTE Control API (beta)New RTE revision for RPi
Q&A and RTE giveaway
Agenda
2 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
Piotr Król Michał ŻygowskiFounder & Embedded Systems
ConsultantFirmware Engineer
[email protected]/in/michał-
żygowski-88954416b
@pietrushnic [email protected] linkedin.com/in/krolpiotrfacebook.com/piotr.krol.756859
Introduction
3 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
Narrowing down serious issue requires a lot of repetitive work:
modify & build firmware with potential fixflash affected firmware versionboot systemverify if the bug still exists
Problem with automation and remote development
To get rid of these problems we created Remote Testing Environment.
Firmware developer routines
4 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
Remote Testing Environment
5 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
Today, our Devices Under Test are PC Engines APU platforms. For workshoppurposes we will need:
Additional infrastructure that can be used is Debian machine with iPXE serverfor supplementary boot option.
Infrastructure
6 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
connect RTE Hat with Orange Pi Zero board (~3min)
Hardware setup
7 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
RS232 interface via a null modem cable (~2min)
Hardware setup
8 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
SPI interface via IDC 2x4 pin cable (~3min)
Hardware setup
9 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
APU PWR/RST pins with RTE OC buffers (~5min)
Hardware setup
10 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
DC-DC jack cable (~2min)
Hardware setup
11 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
take the connected setup to the power/network section (~10min):power up RTE and APU platformsconnect Ethernet cable to Orange Pi Zero
Hardware setup
12 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
Generic test automation framework for acceptance test-drivendevelopment (ATDD)Utilizes the keyword-driven testing approachThe core is implemented using Python and runs also on Jython (JVM) andIronPython (.NET)Bug fixes need tests too - Test-Driven Bug Fixing (TDBF)
Automation tests - RobotFramework
13 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
Each test suite should contain 4 sections:
*** Settings ***Library SSHLibraryResource robotframework/variables.robotSuite Setup SSH Connection and Log In ${rte_ip}Suite Teardown Log Out And Close Connections
*** Variables ***${username} root${password} meta-rte
*** Keywords ***Log Out And Close Connections [Documentation] Close all telnet and ssh open connections. Telnet.Close All Connections SSHLibrary.Close All Connections
*** Test Cases ***RTE: 1.1 SSH connection ${ssh_info}= SSHLibrary.Get Connection Should Be Equal As Strings ${ssh_info.host} ${rte_ip}
Automation tests - test suite structure
14 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
First, prepare the environment (~10min):
Automation tests - setup environment
15 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
First, prepare the environment (~10min):
start Debian RTE Workshop VBox and loginpassword: rte-workshop
Automation tests - setup environment
16 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
First, prepare the environment (~10min):
start Debian RTE Workshop VBox and loginpassword: rte-workshop
open terminal (Activities/Terminal)
Automation tests - setup environment
17 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
First, prepare the environment (~10min):
start Debian RTE Workshop VBox and loginpassword: rte-workshop
open terminal (Activities/Terminal)
change directory: cd RTE-Workshop
Automation tests - setup environment
18 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
First, prepare the environment (~10min):
start Debian RTE Workshop VBox and loginpassword: rte-workshop
open terminal (Activities/Terminal)
change directory: cd RTE-Workshop
activate virtualenv: source robot-venv/bin/activate
Automation tests - setup environment
19 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
RTE-Workshop directory tree
20 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
Now we should run script which tests our hardware setup (~5min):
Connection validation - part 1
21 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
Now we should run script which tests our hardware setup (~5min):
robot -v rte_ip:RTE_IP validation.robot
where RTE_IP is your RTE IP address
Connection validation - part 1
22 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
Now we should run script which tests our hardware setup (~5min):
robot -v rte_ip:RTE_IP validation.robot
where RTE_IP is your RTE IP address
Connection validation - part 1
23 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
Connection validation - part 2
24 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
Connection validation - part 3
25 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
Possible failures:ssh timeout - RTE lacks network connectionNoValidConnections - bad IP addressTelnet timeout - check RS232 cable or close established connection
Automation tests - logs
26 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
Possible failures:ssh timeout - RTE lacks network connectionNoValidConnections - bad IP addressTelnet timeout - check RS232 cable or close established connection
After each RobotFramework test suite we can track generated log files:
Try it by yourself by typing: firefox log.html report.html output.xml
Automation tests - logs
27 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
Flashing process can be started by flash-apu.robot script (~5min):
robot -v rte_ip:RTE_IP -v fw_file:PATH_TO_FILE flash-apu.robot
remember to change fw_file firmware path for appropriate APU2/3/4binary (if you are not sure, check the sign of life from previous tests)
Flashing firmware - automated process
28 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
Flashing process can be started by flash-apu.robot script (~5min):
robot -v rte_ip:RTE_IP -v fw_file:PATH_TO_FILE flash-apu.robot
remember to change fw_file firmware path for appropriate APU2/3/4binary (if you are not sure, check the sign of life from previous tests)
Possible failures:
opening "/tmp/coreboot.rom" failed - SSH key not authenticatedNo such file or directory - bad firmware file path
Flashing firmware - automated process
29 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
We have prepared 2 binaries to present the RTE and RobotFramework capabilityto track down unexpected bugs and flash APUs firmware:
Fixing bugs (~10min)
30 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
We have prepared 2 binaries to present the RTE and RobotFramework capabilityto track down unexpected bugs and flash APUs firmware:
I. First, let's try to update our DUT with custom firmware:
Fixing bugs (~10min)
31 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
We have prepared 2 binaries to present the RTE and RobotFramework capabilityto track down unexpected bugs and flash APUs firmware:
I. First, let's try to update our DUT with custom firmware:
run: robot -v rte_ip:RTE_IP -v fw_file:firmware/apu4_custom.rom flash-apu.robot
Fixing bugs (~10min)
32 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
We have prepared 2 binaries to present the RTE and RobotFramework capabilityto track down unexpected bugs and flash APUs firmware:
I. First, let's try to update our DUT with custom firmware:
run: robot -v rte_ip:RTE_IP -v fw_file:firmware/apu4_custom.rom flash-apu.robot
Fixing bugs (~10min)
33 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
We have prepared 2 binaries to present the RTE and RobotFramework capabilityto track down unexpected bugs and flash APUs firmware:
II. Assumes that we made appropriate changes and build the new binary, flashAPU with the newest v4.8.0.3 firmware:
Fixing bugs (~10min)
34 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
We have prepared 2 binaries to present the RTE and RobotFramework capabilityto track down unexpected bugs and flash APUs firmware:
II. Assumes that we made appropriate changes and build the new binary, flashAPU with the newest v4.8.0.3 firmware:
run: robot -v rte_ip:RTE_IP -v fw_file:firmware/apu4_mainline.romflash-apu.robot
Fixing bugs (~10min)
35 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
We have prepared 2 binaries to present the RTE and RobotFramework capabilityto track down unexpected bugs and flash APUs firmware:
II. Assumes that we made appropriate changes and build the new binary, flashAPU with the newest v4.8.0.3 firmware:
run: robot -v rte_ip:RTE_IP -v fw_file:firmware/apu4_mainline.romflash-apu.robot
Fixing bugs (~10min)
36 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
To open rest API service, type: firefox RTE_IP:8000 (~3min)
Our plans - RTE Control API (beta)
37 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
New RTE Hat compatible with 40-pin header RaspberryPi 2/3/Zero
Our plans - New revision for RPi
38 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
New revision during testing phase
Our plans - New revision for RPi
39 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
RTE website & shop
40 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski
Q&A and lottery
Cards received at the entrance will serve as numbers to select the winnerof the lottery. The prizes are three full RTE sets!
Where can I order the RTE for myself? The online store is starting soon, so please submit any orders to:
Please visit our RTE website: 3mdeb.com/rte
41 / 41OSFC 2018
CC BY 4.0 | Piotr Król, Michał Żygowski