andreas rasmusson, göran söderman scania karl meinke ... · and oracle model abstraction observed...
TRANSCRIPT
![Page 1: Andreas Rasmusson, Göran Söderman Scania Karl Meinke ... · and Oracle Model abstraction Observed output o n i n n = 1, 2, … o n i n M n verdict v n Stochastic Equivalence Checker](https://reader036.vdocuments.us/reader036/viewer/2022090604/60586979921650094e64463e/html5/thumbnails/1.jpg)
Title Slide
Virtualized Embedded Systems for Testing and Development
Andreas Rasmusson, Göran Söderman Scania
Karl Meinke , Hojat Khosrowjerdi KTH-CSC
FFI Dnr 2013-05608
![Page 2: Andreas Rasmusson, Göran Söderman Scania Karl Meinke ... · and Oracle Model abstraction Observed output o n i n n = 1, 2, … o n i n M n verdict v n Stochastic Equivalence Checker](https://reader036.vdocuments.us/reader036/viewer/2022090604/60586979921650094e64463e/html5/thumbnails/2.jpg)
One Image
Agenda
• Driving factors, Needs
• Methods & Tools
• Case-studies
• Results & Experiences
2
![Page 3: Andreas Rasmusson, Göran Söderman Scania Karl Meinke ... · and Oracle Model abstraction Observed output o n i n n = 1, 2, … o n i n M n verdict v n Stochastic Equivalence Checker](https://reader036.vdocuments.us/reader036/viewer/2022090604/60586979921650094e64463e/html5/thumbnails/3.jpg)
Title and Content
3
Virtues explores methods for Quality Assurance oflarge, evolving, embedded software systems,with a focus on sw-development and sw-testing
How:1. Test the unmodified, true ECU-binaries2. Generate tests from the concise formal requirements
Needs:1. A test environment independent of source-code/architechture/OS2. Keep test code volume down
![Page 4: Andreas Rasmusson, Göran Söderman Scania Karl Meinke ... · and Oracle Model abstraction Observed output o n i n n = 1, 2, … o n i n M n verdict v n Stochastic Equivalence Checker](https://reader036.vdocuments.us/reader036/viewer/2022090604/60586979921650094e64463e/html5/thumbnails/4.jpg)
Title and Content
• 3 year project – 2014-2017
• FFI Dnr 2013-05608
• Learning-Based Requirements Testing on SIL and VHIL Platforms
• Software-in-the-loop testing Wincomp
• Virtualised hardware-in-the-loop testing QEMU
• Integrated with learning-based test tool LBTest
4
Scania/KTH project VIRTUES
![Page 5: Andreas Rasmusson, Göran Söderman Scania Karl Meinke ... · and Oracle Model abstraction Observed output o n i n n = 1, 2, … o n i n M n verdict v n Stochastic Equivalence Checker](https://reader036.vdocuments.us/reader036/viewer/2022090604/60586979921650094e64463e/html5/thumbnails/5.jpg)
Title and Content
TestTool
PerformanceTool
SecurityTool
EnergyTool
SILEmulator:WinComp
ECUComponent
SIL Tool Chain
ECU SoftwareComponent
![Page 6: Andreas Rasmusson, Göran Söderman Scania Karl Meinke ... · and Oracle Model abstraction Observed output o n i n n = 1, 2, … o n i n M n verdict v n Stochastic Equivalence Checker](https://reader036.vdocuments.us/reader036/viewer/2022090604/60586979921650094e64463e/html5/thumbnails/6.jpg)
Title and Content
6
VHIL Tool Chain
![Page 7: Andreas Rasmusson, Göran Söderman Scania Karl Meinke ... · and Oracle Model abstraction Observed output o n i n n = 1, 2, … o n i n M n verdict v n Stochastic Equivalence Checker](https://reader036.vdocuments.us/reader036/viewer/2022090604/60586979921650094e64463e/html5/thumbnails/7.jpg)
Title and Content
7
LBTest: Architecture
Incremental Learning Algorithm
Model Checker(NuSMV)
TestCaseGeneratorand Oracle
Model abstraction
Observed output
inon n = 1, 2, …
inon
Mn
verdict vn
Stochastic Equivalence
Checker
counterexample in
Multiple, parallellSUT processespossible
Communication wrapper
System Under Test
PLTL User Requirement
![Page 8: Andreas Rasmusson, Göran Söderman Scania Karl Meinke ... · and Oracle Model abstraction Observed output o n i n n = 1, 2, … o n i n M n verdict v n Stochastic Equivalence Checker](https://reader036.vdocuments.us/reader036/viewer/2022090604/60586979921650094e64463e/html5/thumbnails/8.jpg)
Title and Content
8
SIL: Stubbed x86-compiled app
Appl-SW
OS HL
OS LL(Stubbed)
x86
Wrapper
LBTest
ApplReq
![Page 9: Andreas Rasmusson, Göran Söderman Scania Karl Meinke ... · and Oracle Model abstraction Observed output o n i n n = 1, 2, … o n i n M n verdict v n Stochastic Equivalence Checker](https://reader036.vdocuments.us/reader036/viewer/2022090604/60586979921650094e64463e/html5/thumbnails/9.jpg)
Title and Content
9
VHIL: unmodified srec-file (guest-level)
Wrapper
LBTest
ApplReq
ISS (TCG)
srec
Appl-SW
OS HL
OS LL
Hw-modelCPU, MMU, InterruptCtrl,
Timer, MMIO, CAN …
gdb-server
![Page 10: Andreas Rasmusson, Göran Söderman Scania Karl Meinke ... · and Oracle Model abstraction Observed output o n i n n = 1, 2, … o n i n M n verdict v n Stochastic Equivalence Checker](https://reader036.vdocuments.us/reader036/viewer/2022090604/60586979921650094e64463e/html5/thumbnails/10.jpg)
Title and Content
GDB (x86)10
VHIL: unmodified srec-file (host-level)
Wrapper
LBTest
ApplReq
ISS (TCG)
srec
Appl-SW
OS HL
OS LL
Hw-modelCPU, MMU, InterruptCtrl,
Timer, MMIO, CAN …
gdb-server
![Page 11: Andreas Rasmusson, Göran Söderman Scania Karl Meinke ... · and Oracle Model abstraction Observed output o n i n n = 1, 2, … o n i n M n verdict v n Stochastic Equivalence Checker](https://reader036.vdocuments.us/reader036/viewer/2022090604/60586979921650094e64463e/html5/thumbnails/11.jpg)
Title and Content
We considered SIL requirements testing of 3 ECU applications
• remote engine start (ESTA),
• dual circuit steering (DCS),
• fuel level display (FLD).
11
SIL Case Studies
Learning-based Testing for Safety Critical Automotive ApplicationsH. Khosrowjerdi, K. Meinke, A. Rasmusson Proc. Int. Symp. on Model Based Safety Assessment, 2017
![Page 12: Andreas Rasmusson, Göran Söderman Scania Karl Meinke ... · and Oracle Model abstraction Observed output o n i n n = 1, 2, … o n i n M n verdict v n Stochastic Equivalence Checker](https://reader036.vdocuments.us/reader036/viewer/2022090604/60586979921650094e64463e/html5/thumbnails/12.jpg)
Title and Content
Benchmark LBTest against an in-house test suite using mutation testing
Requirements Capture Challenges
“the electric motor had been on while the second sensor had a flow or noflow, since the last engine restart”
(emotor = on & (sensor2 = flow | sensor2 = noflow) Since ignition = restart)
12
Dual Circuit Steering (DCS)
Learning-based testing of automotive ECUsSophia BäckströmMasters project, KTH
![Page 13: Andreas Rasmusson, Göran Söderman Scania Karl Meinke ... · and Oracle Model abstraction Observed output o n i n n = 1, 2, … o n i n M n verdict v n Stochastic Equivalence Checker](https://reader036.vdocuments.us/reader036/viewer/2022090604/60586979921650094e64463e/html5/thumbnails/13.jpg)
Title and Content
• Tested 7 hours 24 minutes
• 60 state model,
• 800 transitions
• 97% convergence
• 5 requirements errors found – but none were safety critical
13
SIL Test Results for DCS
Learning-based testing of automotive ECUsSophia BäckströmMasters project, KTH
![Page 14: Andreas Rasmusson, Göran Söderman Scania Karl Meinke ... · and Oracle Model abstraction Observed output o n i n n = 1, 2, … o n i n M n verdict v n Stochastic Equivalence Checker](https://reader036.vdocuments.us/reader036/viewer/2022090604/60586979921650094e64463e/html5/thumbnails/14.jpg)
Title and Content
We considered VHIL requirements testing of 3 ECU requirements
• bit-flip Checksummed Memory Area,
• stuck-at in Checksummed Memory Area,
• (TBD) stuck-at in MMIO
14
VHIL Case Studies
Virtualized-Fault Injection with Learning-based Requirements TestingH. Khosrowjerdi, K. Meinke, A. RasmussonSubmitted to ICST2018
![Page 15: Andreas Rasmusson, Göran Söderman Scania Karl Meinke ... · and Oracle Model abstraction Observed output o n i n n = 1, 2, … o n i n M n verdict v n Stochastic Equivalence Checker](https://reader036.vdocuments.us/reader036/viewer/2022090604/60586979921650094e64463e/html5/thumbnails/15.jpg)
Title and Content
15
Bit-flip in Checksummed Memory Area
Inject errors via scripted debugger
guest-level, with symbols
Bit-flip
![Page 16: Andreas Rasmusson, Göran Söderman Scania Karl Meinke ... · and Oracle Model abstraction Observed output o n i n n = 1, 2, … o n i n M n verdict v n Stochastic Equivalence Checker](https://reader036.vdocuments.us/reader036/viewer/2022090604/60586979921650094e64463e/html5/thumbnails/16.jpg)
Title and Content
Conclusions
• Successful case studies on real-products
• Both SIL and VHIL testing can be handled by one tool
• Formal requirements improved Scania’s product understanding
• Instruction Set Simulation is useful for ecu-testing.
• Test the realization. It contains more behavior than the source-code.
![Page 17: Andreas Rasmusson, Göran Söderman Scania Karl Meinke ... · and Oracle Model abstraction Observed output o n i n n = 1, 2, … o n i n M n verdict v n Stochastic Equivalence Checker](https://reader036.vdocuments.us/reader036/viewer/2022090604/60586979921650094e64463e/html5/thumbnails/17.jpg)
Last Page - Blue
Andreas RasmussonSenior Engineer | Systems Development | Scania CV ABPhone: +46 (0)8 553 894 64 Mobile: +46 (0)700 831 914Address, 151 87 Södertälje, [email protected]
www.scania.com | www.facebook.com/scaniagroup
![Page 18: Andreas Rasmusson, Göran Söderman Scania Karl Meinke ... · and Oracle Model abstraction Observed output o n i n n = 1, 2, … o n i n M n verdict v n Stochastic Equivalence Checker](https://reader036.vdocuments.us/reader036/viewer/2022090604/60586979921650094e64463e/html5/thumbnails/18.jpg)
Last Page/Chapter Divider
18