university college londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_briand.pdf• issre 2003, leon and...
TRANSCRIPT
![Page 1: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/1.jpg)
.lusoftware verification & validationVVS
Testing of Cyber-Physical Systems:
Diversity-driven Strategies
Lionel Briand
COW 57, London, UK
![Page 2: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/2.jpg)
Cyber-Physical Systems• A system of collaborating computational elements controlling
physical entities
2
![Page 3: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/3.jpg)
Context• Projects on verification of cyber-physical systems, control
systems, autonomous systems, …
• Focus on safety, performance, resource usage …
• Automotive, satellite, energy, manufacturing …
3
![Page 4: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/4.jpg)
Controllers
4
Plant
Controller
Actuator Sensor
Disturbances
Reference Inputs
![Page 5: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/5.jpg)
Decision-Making Components
5
Sensor
Controller
Actuator Decision
Plant
![Page 6: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/6.jpg)
Development Process
6
Functional modeling: • Controllers• Plant• Decision
Continuous and discrete Simulink models
Model simulation and testing
Architecture modelling• Structure• Behavior• Traceability
System engineering modeling (SysML)
Analysis: • Model execution and
testing• Model-based testing• Traceability and
change impact analysis
• ...
(partial) Code generation
Deployed executables on target platform
Hardware (Sensors ...) Analog simulators
Testing (expensive)
Hardware-in-the-Loop Stage
Software-in-the-Loop StageModel-in-the-Loop Stage
![Page 7: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/7.jpg)
Simulink Models - Simulation
• Simulation Models
• heterogeneous
8
Software Plant Model
Network Model
• continuous behavior
• are used for
• algorithm design testing
• comparing design options
![Page 8: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/8.jpg)
Cruise Control: Plant
9
![Page 9: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/9.jpg)
Problem • How do we automatically verify and test CP functional models
(e.g., controller, plant, decision) at MiL?
• What types of requirements / properties do we check?
• Commercial tools:
• Cannot handle continuous operators, floating point models (e.g., SLDV, Reactis)
• Based on model structural coverage: low fault detection
• Can only handle linear systems and specific properties (Linear analysis toolbox)
10
![Page 10: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/10.jpg)
Challenges
• Limited work on verification and testing of controllers and decision-making components in CP systems
• Space of test input signals is extremely large.
• Model execution, especially when involving plant models, is extremely expensive.
11
![Page 11: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/11.jpg)
More Challenges
• Test oracles are not simple Boolean properties and easily known in advance – they involve analyzing changes in value over time (e.g., signal patterns) and assessing levels of risk.
• Simulatable plant model of the physical environment is not always available or fully accurate and precise.
12
![Page 12: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/12.jpg)
Diversity Strategies• Strategy: Maximize diversity of test cases
• Assumption: “the more diverse the test cases the higher their fault revealing capacity”
• Challenge: Define “diversity” in context, pair-wise similarity computation, test selection algorithm
• Examples of early work
• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage
• FSE 2010, Hemmati et al., Similarity-based test selection applied to model-based testing based on state models for control systems, selection with GA
• Full control on test budget: Maximize fault detection for a given test suite size
13
![Page 13: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/13.jpg)
Testing Controllers
14
![Page 14: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/14.jpg)
Controllers are Pervasive
15
![Page 15: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/15.jpg)
• Supercharger bypass flap controllerüFlap position is bounded within
[0..1]üImplemented in MATLAB/Simulinkü34 (sub-)blocks decomposed into 6
abstraction levels
Supercharger
Bypass Flap
Supercharger
Bypass Flap
Flap position = 0 (open) Flap position = 1 (closed)
Simple Example
16
![Page 16: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/16.jpg)
MiL Test Cases
17
ModelSimulation
InputSignals
OutputSignal(s)
S3t
S2t
S1t
S3t
S2t
S1t
Test Case 1
Test Case 2
![Page 17: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/17.jpg)
InitialDesired Value
FinalDesired Value
time time
Desired Value
Actual Value
T/2 T T/2 T
Test Input Test Output
Plant Model
Controller(SUT)
Desired value Error
Actual value
System output+-
MiL Testing of Controllers
18
![Page 18: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/18.jpg)
Requirements and Test ObjectivesIn
itial
Des
ired
(ID)
Desired ValueI (input)Actual Value (output)
Fina
l Des
ired
(FD
)
timeT/2 T
Smoothness
Responsiveness
Stability
20
![Page 19: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/19.jpg)
21
Test Generation Approach
• We formalize controller’s requirements in terms of desired and actual outputs
Smoothness
• We rely on controller’s feedback to automate test oracles
desired valueactual value
< Threshold Desired Value (Setpoint)
Actual Value (feedback)
System Output+
-
Control Signal
Plant(environment)Controller
![Page 20: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/20.jpg)
A Search-Based Test Approach
Initial Desired (ID)
Fina
l Des
ired
(FD
)
Worst Case(s)?
• Search directed by model execution feedback
• Finding worst case inputs
• Possible because of automated oracle (feedback loop)
• Different worst cases for different requirements
• Worst cases may or may not violate requirements
22
![Page 21: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/21.jpg)
Initial Solution
HeatMap Diagram
1. ExplorationList of Critical RegionsDomain
Expert
Worst-Case Scenarios
+Controller-
plant model
Objective Functionsbased on
Requirements 2. Single-State
Search
time
Desired ValueActual Value
0 1 20.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Initial Desired
Final Desired
23
![Page 22: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/22.jpg)
Results
• We found much worse scenarios during MiL testing than our partner had found so far
• These scenarios are also run at the HiL level, where testing is much more expensive: MiL results -> test selection for HiL
• But further research was needed:
• Simulations are expensive
• Configuration parameters
24
![Page 23: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/23.jpg)
Final Solution
+Controller
Model (Simulink)
Worst-Case Scenarios
List of Critical
PartitionsRegressionTree
1.Exploration with Dimensionality
Reduction
2.Search withSurrogate Modeling
Objective Functions
DomainExpert
Visualization of the 8-dimension space
using regression treesDimensionality reduction to identify
the significant variables(Elementary Effect Analysis)
Surrogate modeling to predict the fitness
function and speed up the search (Machine learning)
25
![Page 24: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/24.jpg)
Open Loop Controllers
On
Off
CtrlSig
• Mixed discrete-continuous behavior: Simulink stateflows
• No plant model: Much quicker simulation time
• No feedback loop -> no automated oracle
• The main testing cost is the manual analysis of output signals
• Goal: Minimize test suites
• Challenge: Test selection
• Entirely different approach to testing
respectively. In addition, we adapt the whitebox coverage and theblackbox output diversity selection criteria to Stateflows, and evalu-ate their fault revealing power for continuous behaviours. Coveragecriteria are prevalent in software testing and have been consideredin many studies related to test suite effectiveness in different appli-cation domains [?]. In our work, we consider state and transitioncoverage criteria [?] for Statflows. Our output diversity criterion isbased on the recent output uniqueness criterion [?] that has beenstudied for web applications and has shown to be a useful surro-gate to whitebox selection techniques. We consider this criterionin our work because Stateflows have complex internal structuresconsisting of differential equations, making them less amenable towhitebox techniques, while they have rich time-continuous outputs.
In this paper, we make the following contributions:
• We focus on the problem of testing Stateflows with mixeddiscrete-continuous behaviours. We propose two new testcase selection criteria output stability and output continuitywith the goal of selecting test inputs that are likely to pro-duce continuous outputs exhibiting instability and disconti-nuity failures, respectively.
• We adapt the whitebox coverage and the blackbox outputdiversity selection criteria to Stateflows, and evaluate theirfault revealing power for continuous behaviours. The formeris defined based on traditional state and transition coveragefor state machines, and the latter is defined based on the re-cent output uniqueness criterion [?].
• We evaluate effectiveness of our newly proposed and theadapted selection criteria by applying them to three Stateflowcase study models: two industrial and one public domain.Our results show that RESULT.
Organization of the paper.
2. BACKGROUND AND MOTIVATIONMotivating example. We motivate our work using a simplifiedStateflow from the automotive domain which controls a superchargerclutch and is referred to as the Supercharger Clutch Controller (SCC).Figure 1(a) represents the discrete behaviour of SCC specifyingthat the supercharger clutch can be in two quiescent states [?]: en-gaged or disengaged. Further, the clutch moves from the disen-gaged to the engaged state whenever both the engine speed engspdand the engine coolant temperature tmp respectively fall inside thespecified ranges of [smin..smax] and [tmin..tmax]. The clutchmoves back from the engaged to the disengaged state whenevereither the speed or the temperature falls outside their respectiveranges. The variable ctrlSig in Figure 1(a) indicates the sign andmagnitude of the voltage applied to the DC motor of the clutchto physically move the clutch between engaged and disengagedpositions. Assigning 1.0 to ctrlSig moves the clutch to the en-gaged position, and assigning �1.0 to ctrlSig moves it back tothe disengaged position. To avoid clutter in our figures, we useengageReq to refer to the condition on the Disengaged ! En-gaged transition, and disengageReq to refer to the condition onthe Engaged ! Disengaged transition.
The discrete transition system in Figure 1(a) assumes that theclutch movement takes no time, and further, does not provide anyinsight on the quality of movement of the clutch. Figure 1(b) ex-tends the discrete transition system in Figure 1(a) by adding a timervariable, i.e., time, to explicate the passage of time in the SCCbehaviour. The new transition system in Figure 1(b) includes two
(a) SCC -- Discrete Behaviour
(b) SCC -- Timed Behaviour
EngagedDisengaged
Engaging
(c) Engaging state of SCC -- mixed discrete-continuous behaviour
Disengaging
Disengaged
Engaged
time + +;
[disengageReq]/time := 0
[time
>5]
[time
>5]
time + +;
[(engspd > smin � engspd < smax) � (tmp > tmin � tmp < tmax)]/ctrlSig := 1
[engageReq]/ time := 0
[¬(engspd > smin � engspd < smax) � ¬(tmp > tmin � tmp < tmax)] /ctrlSig := �1
OnMoving OnSlipping
OnCompleted
time + +;ctrlSig := f(time)
Engaging
time + +;ctrlSig := g(time)
time + +;ctrlSig := 1.0
[¬(vehspd = 0) �time > 2]
[(vehspd = 0) �time > 3]
[time > 4]
Figure 1: Supercharge Clutch Controller (SCC) Stateflow.
transient states [?], engaging and disengaging, specifying that mov-ing from the engaged to the disengaged state and vice versa takessix milisec. Since this model is simplified, it does not show han-dling of alterations of the clutch state during the transient states.In addition to adding the time variable, we note that the variablectrlSig, which controls physical movement of the clutch, cannotabruptly jump from 1.0 to �1.0, or vice versa. In order to ensuresafe and smooth movement of the clutch, the variable ctrlSig hasto gradually move between 1.0 and �1.0 and be described as afunction over time, i.e., a signal. To express the evolution of thectrlSig signal over time, we decompose the transient states en-gaging and disengaging into sub-state machines. Figure 1(c) showsthe sub-state machine related to the engaging state. The one relatedto the disengaging state is similar. At beginning (in state OnMov-ing), the function ctrlSig has a steep grade (i.e., function f ) tomove the stationary clutch from the disengaged state and acceler-ate it to reach a certain speed in about two milisec. Afterwards (instate OnSlipping), ctrlSig decreases the speed of clutch basedon the gradual function g until about four milisec. This is to ensurethat the clutch slows down as it gets closer to the crank shaft ofthe car. Finally, at state OnCompleted, ctrlSig reaches value 1.0and remains constant, causing the clutch to get engaged in aboutone milisec. When the car is stationary, i.e., vehspd is 0, the clutchmoves based on the steep grade function f for three milisec, anddoes not have to go to the OnSlipping phase to slow down beforeit reaches the crank shaft at state OnCompleted.Input and Output. The Stateflow inputs and outputs are signals(functions over time). Each input/output signal has a data type,e.g. boolean, enum or float, specifying the range of the signal.For example, Figure 2 shows an example input (dashed line) andoutput (solid line) signals for SCC. The input signal is related toengageReq and is boolean, while the output signal is related to
28
![Page 25: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/25.jpg)
Selection Strategies Based on Search• White-box structural coverage
• State Coverage
• Transition Coverage
• Input signal diversity
• Output signal diversity
• Failure-Based selection criteria
• Domain specific failure patterns
• Output Stability
• Output Continuity
S3t
S3t
29
![Page 26: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/26.jpg)
System Output
InputSignals
Output Signals
Controller Plant(environment)
30
instabilityfailure pattern
discontinuityfailure pattern
output diversity
• We assume test oracles are manual
Test Generation Approach
• We rely on output signals to produce small test suites with high fault-revealing ability
![Page 27: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/27.jpg)
Output Diversity -- Vector-Based
31
Output
TimeOutput Signal 2Output Signal 1
Normalized Euclidian Distance
![Page 28: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/28.jpg)
32
Output Diversity -- Feature-Based
increasing (n) decreasing (n)constant-value (n, v)
signal featuresderivative second derivative
sign-derivative (s, n) extreme-derivatives
1-sided discontinuity
discontinuity
1-sided continuitywith strict local optimum
value
instant-value (v)constant (n)
discontinuitywith strict local optimum
increasing
C
A
B
![Page 29: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/29.jpg)
33
Output Diversity -- Feature-Based
increasing (n) decreasing (n)constant-value (n, v)
signal featuresderivative second derivative
sign-derivative (s, n) extreme-derivatives
1-sided discontinuity
discontinuity
1-sided continuitywith strict local optimum
value
instant-value (v)constant (n)
discontinuitywith strict local optimum
increasing
C
A
B
Similarity: To which extent any part of a signal is similar to a feature
![Page 30: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/30.jpg)
Failure-based Test Generation
34
Instability Discontinuity
0.0 1.0 2.0-1.0
-0.5
0.0
0.5
1.0
Time
Ctr
lSig
Output
• Search: Maximizing the likelihood of presence of specific failure patterns in output signals
• Domain-specific failure patterns elicited from engineers
0.0 1.0 2.0Time
0.0
0.25
0.50
0.75
1.0
Ctr
lSig
Output
![Page 31: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/31.jpg)
Search• Whole test suite generation approach
• Used when objective functions characterize the test suite
• Optimize test objective for a given test suite size (budget for manual oracles)
• Maximize the minimum distances of each output signal vector from the other output signal vectors
• Adaptation of Simulated Annealing
35
![Page 32: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/32.jpg)
Faulty Model Output
36
Correct Model Output
Fault-Revealing Ability
Covers the fault and Covers the fault but is Likely to reveal it is very unlikely to reveal it
![Page 33: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/33.jpg)
Results• The test cases resulting from state/transition coverage
algorithms cover the faulty parts of the models
• However, they fail to generate output signals that are sufficiently distinct from expectations, hence yielding a low fault revealing rate
• Diversity strategies significantly outperforms coverage-based and random testing
• Output-based algorithms are much more effective, both based on diversity and failure patterns
37
![Page 34: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/34.jpg)
Results
• Feature-based diversity fares significantly better than vector-based diversity
• Strategies based on failure patterns find different types of faults than diversity strategies
• Existing commercial tools: Not effective at finding faults, not applicable to entire Simulink models, e.g., Simulink Design Verifier
38
![Page 35: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/35.jpg)
Example Failures
39
![Page 36: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/36.jpg)
Example Failures
40
Instability
Discontinuity
![Page 37: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/37.jpg)
Conclusions• Maximizing output diversity helps identify scenarios
where the discrepancy between the produced and expected signal is large
• Useful when test output signals are analyzed manually
• Simulink models and their outputs are complex
• Helps maximize fault detection within a fixed budget
• Properly defining diversity was a challenge
41
![Page 38: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/38.jpg)
Reflections on Diversity• Useful strategy when no precise guidance, directly
related to the test objectives, is available for the search
• In the general, the key issue is how to define diversity in an optimal way given the objectives
• In practice, how diversity is defined also depends on what information is available at a reasonable cost
• The time complexity of computing diversity is a major cost of the search – it must be accounted for
42
![Page 39: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/39.jpg)
Acknowledgements
• Shiva Nejati
• Reza Matinnejad
• Delphi Automotive Systems, Luxembourg
43
![Page 40: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/40.jpg)
References• R. Matinnejad et al., “MiL Testing of Highly Configurable Continuous Controllers:
Scalable Search Using Surrogate Models”, IEEE/ACM ASE 2014 (Distinguished paper award)
• R. Matinnejad et al., “Effective Test Suites for Mixed Discrete-Continuous StateflowControllers”, ACM ESEC/FSE 2015 (Distinguished paper award)
• R. Matinnejad et al., “Automated Test Suite Generation for Time-continuous Simulink Models“, IEEE/ACM ICSE 2016
• R. Matinnejad et al., “Test Generation and Test Prioritization for Simulink Models with Dynamic Behavior“, under minor revision with IEEE TSE
44
![Page 41: University College Londoncrest.cs.ucl.ac.uk/cow/57/slides/cow57_Briand.pdf• ISSRE 2003, Leon and Podgurski, filtering and prioritizing test cases, relying on code coverage • FSE](https://reader035.vdocuments.us/reader035/viewer/2022081407/5f1edd7c8b072610bb331e58/html5/thumbnails/41.jpg)
.lusoftware verification & validationVVS
Testing of Cyber-Physical Systems:
Diversity-driven Strategies
Lionel Briand
COW 57, London, UK