systematic model based and search basedtesting of …
TRANSCRIPT
![Page 1: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/1.jpg)
Sophia Antipolis, French Riviera20-22 October 2015
SYSTEMATIC MODEL‐BASED AND SEARCH‐BASED TESTING OF CYBER‐PHYSICAL SYSTEMS
Shaukat Ali, PhD, Senior Research ScientistEmail: [email protected] © All rights reserved
![Page 2: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/2.jpg)
OUTLINE OF THE PRESENTATION
• BackgroundCyber‐Physical SystemsSearch‐based Software Engineering
• Results from previous projects• Ongoing Project• Summary, Experiences, and Lessons Learnt
2 © All rights reserved
![Page 3: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/3.jpg)
BACKGROUNDCyber‐Physical Systems (CPS)
© All rights reserved
![Page 4: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/4.jpg)
CYBER‐PHYSICAL SYSTEMS
4 © All rights reserved
CPSs are the new generation of connected embeddedsystems integrating cyber‐technologies, software, andphysical components interacting with each other viainformation and physical interfaces [1].
[1] http://cyberphysicalsystems.org/
Handling Systems: Picture courtesy ULMA Handling Systems, Spain
Geo Sports: Picture courtesy Future Position X, Sweden
![Page 5: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/5.jpg)
Testing Levels for CPS
5 © All rights reserved
Application Level : Events and data coming from the user space, e.g., from applications and human
Infrastructure Level : Events and data coming from, e.g., physical units, network equipment, and cloudinfrastructure
Integration Level : Interactions between Application and Infrastructure levels
![Page 6: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/6.jpg)
SCIENTIFIC AND TECHNICAL CHALLENGES
6
• Heterogeneous, Large‐scale Embedded Systems• Dealing with Novel InteractionsSoftware, Hardware, Communication, Human
• Dealing with Uncertainty• Verifying and validation of extra‐functionalproperties such as performance, robustness, ..
• Autonomous
©All rights reserved
![Page 7: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/7.jpg)
WHY IS IT IMPORTANT TO TEST CPS?
7
• Applications, e.g., Healthcare, Aerospace, Avionics, Oil/gas andMaritime, Industrial Automation, and Tele‐communication
• Current applications > $32.3 trillions. By 2025, > $82 trillions [1].• CPSs must be dependable, i.e., safe, trustworthy, reliable, robust,
...• Improving CPS dependability via systematic and automated
testing
©All rights reserved
[1] Evans, P.C., Annunziata, M.: Pushing the Boundaries of Minds and Machines. General Electric (GE), (2012)
![Page 8: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/8.jpg)
BACKGROUNDSEARCH‐BASED SOFTWARE ENGINEERING
© All rights reserved
![Page 9: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/9.jpg)
9
“Using search techniques to search large searchspaces, guided by a fitness function that capturesproperties of the acceptable software artifacts weseek”[1]
Search‐Based Software Engineering
[1] Borrowed from: SBSE: Introduction, Motivation, Results and Directions. Mark HarmanKeynote at SSBSE, 2014
Search Techniques: Genetic Algorithms, Particle SwarmOptimization, …
Large Search Spaces: Millions or billions of possible solutions tosearch from.Fitness Function: To determine solution x is better than Solutiony.
![Page 10: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/10.jpg)
RESULTS FROM SELECTED PROJECTS ROBUSTNESS TESTING OF VIDEOCONFERENCING SYSTEMS
© All rights reserved
![Page 11: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/11.jpg)
What is Robustness?
11 © All rights reserved
“Robustness is the degree to which a softwarecomponent functions correctly in the presence ofexceptional inputs or stressful environmentalconditions” (IEEE Std 610.12‐1990)
![Page 12: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/12.jpg)
MODEL‐BASED ROBUSTNESS TESTING IS CHALLENGING
• Modeling robustness behavior makes modeling highly complex and redundant
• Automated generation of executable test cases from robustness modelsTargeted to reveal robustness faultsGenerating test dataDefining appropriate test strategies for robustness testing
12 © All rights reserved
![Page 13: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/13.jpg)
CASE STUDY: VIDEOCONFERENCING SYSTEMS (CISCO)
13 © All rights reserved
![Page 14: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/14.jpg)
SOLUTION FOR MODEL‐BASED ROBUSTNESS TESTING
14 © All rights reserved
![Page 15: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/15.jpg)
MODELLING METHODOLOGY
15 © All rights reserved
![Page 16: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/16.jpg)
MODELING FUNCTIONAL BEHAVIOR
16 © All rights reserved
![Page 17: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/17.jpg)
MODELING FUNCTIONAL BEHAVIOR
17 © All rights reserved
![Page 18: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/18.jpg)
MODELING ASPECT CLASS DIAGRAM
18 © All rights reserved
![Page 19: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/19.jpg)
MODELING ASPECT STATE MACHINES
19 © All rights reserved
![Page 20: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/20.jpg)
WOVEN STATE MACHINE
20 © All rights reserved
![Page 21: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/21.jpg)
RESULTS FROM MODELING
21 © All rights reserved
Crosscutting behavior
Using aspects Without aspects Effort Saved (%)
States (Added)
Transition (Added)
Trigger (Added) States (Modified/Added)
Transitions (Modified/Added)
Trigger (Added)
States Transitions Trigger
Updating audio constraints
1 - - 86 (Modified) - - 98% - -
Updating video constraints
1 - 86 (Modified) - - 98% - -
Media quality recovery
3 3 19 20 (Added) 178 1604 - 98% 98%
Network communication
3 3 13 20 (Added) 178 1082 - 98% 98%
Add Guard 2 1 - 0 22 (Modified)
- - 95% -
![Page 22: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/22.jpg)
TEST CASE GENERATION• Constraint solving using search algorithms for Test Data Generation (EsOCL tool)Violates properties of the environment to check robustness of the system against those violationsSearch algorithms such as GA, 1+1 (EA), ..EsOCL’s performance is practically applicable
• Developed a tool TRansformation‐based tool for Uml‐baSedTesting (TRUST)Supports configurable and extensible features such as input models, test models, coverage criteria, test data generation strategies, and test script languages.Applied to ABB Robotics and Cisco case studies
22 © All rights reserved
![Page 23: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/23.jpg)
RESULTS FROM EXISTING PROJECTSPRODUCT LINE TESTING
© All rights reserved
![Page 24: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/24.jpg)
USING FEATURE MODELS TO SELECT AND CONFIGURE BEHAVIORAL MODELS
• Select features in FM_T through the Selection Front‐end• Configure attributes CFM_B through the Configuration Front‐end
24 © All rights reserved
![Page 25: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/25.jpg)
STATE MACHINE VARIABILITY
25 © All rights reserved
CFM_B
<<Restrictions>><<Restrictions>>
BMRepository
<<Trace>>
<<Trace>>
FM_T
![Page 26: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/26.jpg)
STATE MACHINE MODEL ELEMENT VARIABILITY
26 © All rights reserved
CFM_B
<<Restrictions>><<Restrictions>>
BMRepositoryFM_T
<<Trace>>
<<Trace>>
![Page 27: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/27.jpg)
ATTRIBUTE VARIABILITY
27 © All rights reserved
CFM_B
<<Restrictions>><<Restrictions>>
BMRepository
FM_T
<<Trace>>
context Multisite inv: self. MaxNumberOfCalls = self.MaxCalls
![Page 28: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/28.jpg)
ATTRIBUTE VARIABILITY
28 © All rights reserved
CFM_B
<<Restrictions>><<Restrictions>>
BMRepositoryFM_T
<<Trace>>
![Page 29: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/29.jpg)
CONCLUDING REMARKS
• Applied to configure several products in aVideoconferencing Product Line of Cisco.
• The configured models were successfully used togenerate test cases using TRUST.
29 © All rights reserved
![Page 30: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/30.jpg)
RESULTS FROM SELECTED PROJECTS ROBUSTNESS TEST SELECTION IN A MARITIME APPLICATION
© All rights reserved
![Page 31: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/31.jpg)
CONTEXT OF THE PROBLEM
• As part of a project in Certus [1] with Kongsberg Maritime as industrial partner
• Overall optimization objective is to find a set of test cases to “break” a CPS as soon as possible
31 © All rights reserved
Remote Controller UnitRemote Controller Unit
Remote Input/output UnitRemote Input/output Unit Maritime ApplicationsMaritime ApplicationsIntegrated into
……
1. Certus Software Verification and Validation Center, http://certus‐sfi.no/
![Page 32: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/32.jpg)
OPTIMIZATION OBJECTIVES
• Input: Set of test cases focusing on testing software,hardware (mechanical, electronics, ..), interactionsamong them
• CostOverall Execution Time,..
• EffectivenessProbability of Failure, Risk, Safety Level, …
• Fitness Function using Cost and Effectiveness measures
• Existing Implementation of Search Algorithms
32 © All rights reserved
![Page 33: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/33.jpg)
TOOL SUPPORT
33 © All rights reserved
![Page 34: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/34.jpg)
CONCLUDING REMARKS
• Some preliminary results are obtained.• The tool seems to be promising based on initial experiments.
• More case studies are being conducted.
34 © All rights reserved
![Page 35: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/35.jpg)
ONGOING CPS TESTING PROJECT: H2020U‐TEST: Testing Cyber‐Physical Systems under Uncertainty: Systematic, Extensible, and Configurable Model‐based and Search‐based Testing Methodologies
© All rights reserved
![Page 36: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/36.jpg)
OBJECTIVES OF U‐TEST
• Objective: Improve the dependability•Means: Model‐based and Search‐based Testing• Objective will be achieved by:Uncertainty TaxonomyHolistic Modeling and Testing FrameworksStandards
36 © All rights reserved
![Page 37: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/37.jpg)
Case Study Providers: Geo SportsFuture Position X, Sweden
37 © All rights reserved
![Page 38: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/38.jpg)
Case Study Providers: Automated WarehousesULMA Handling Systems, Spain
38 © All rights reserved
![Page 39: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/39.jpg)
CONSORTIUM
39 © All rights reserved
• Research Partners Simula Research Laboratory, Norway Fraunhofer FOKUS, Germany TU Wien, Austria
• Tool Vendors Easy Global Marketing FOKUS!MBT
• Exploitation Ikerlan
• Test Bed Provider Nordic MedTest
• Project Management and Administration Oslo Medtech
![Page 40: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/40.jpg)
SUMMARY OF RESULTSMODELING AND TESTING SOLUTIONS
© All rights reserved
![Page 41: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/41.jpg)
CPS MODELING FOR MBT
41 © All rights reserved
www.u-test.eu
![Page 42: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/42.jpg)
Experiences
42 © All rights reserved
Experiences:• Largely relied on standards• Proposed several profiles for various purposes• Developed our own NL-based and model-based solutions• Developed tools• Intentionally made effort to reduce modeling effort• Evaluated with controlled experiments and industrial case studies• Used existing search algorithms and in rare cases extended
![Page 43: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/43.jpg)
Lessons Learnt
43 © All rights reserved
Lessons Learnt:• Poor capability of integrating methodologies and tools• Difficult to think systematically since the beginning
Uncertainty taxonomy• Always a challenge to evaluate a modeling methodology
Expressiveness, Usability, Applicability, Readability, etc.• Which search algorithm to use in which situation
Large scale experiments to select algorithms for different situations
![Page 44: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/44.jpg)
Acknowledgements• Tao Yue, Shuai Wang, Bran Selic• Man Zhang• Former Colleagues: Lionel Briand, Andrea Arcuri, HadiHemmati, Zohaib Iqbal, Nina Holt
44 © All rights reserved
![Page 45: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/45.jpg)
References• Modeling References S. Ali, T. Yue, L. Briand, and S. Walawege. A Product Line Modeling and Configuration Methodology to Support Model‐
based Testing: An Industrial Case Study, (MODELS), 2012. S. Ali, L. Briand, and H. Hemmati. Modeling Robustness Behavior Using Aspect‐Oriented Modeling to Support Robustness
Testing of Industrial Systems, the Journal of Software and Systems Modeling (SOSYM), Springer, 11(4):633‐670, 2012. S. Ali, T. Yue, and L. Briand. Does Aspect‐Oriented Modeling Help Improve the Readability of UML State Machines?,
Springer Software and System Modeling (SOSYM), 2012. S. Ali, T. Yue, M. Z. Iqbal, and R. K. Panesar‐Walawege. Insights on the Use of OCL in Diverse Industrial Applications, In:
SAM'14 M. Z. Iqbal, S. Ali, T. Yue and L. Briand, Applying UML/MARTE on Industrial Projects: Challenges, Experiences, and
Guidelines, in SoSyM 2014• Testing References S. Ali, M. Z. Iqbal, A. Arcuri, and L. Briand. Solving OCL Constraints for Test Data Generation in Industrial Systems with
Search Techniques, IEEE Transactions on Software Engineering, 2013. S. Ali and H. Hemmati. Model‐based Testing of Video Conferencing Systems: Challenges, Lessons Learnt, and Results, In:
ICST 2014. S. Ali, L. Briand, A. Arcuri, and S. Walawege. An Industrial Application of Robustness Testing using Aspect‐Oriented
Modeling, UML/MARTE, and Search Algorithms, In: Models 2011 S. Wang, S. Ali, T. Yue, M. Liaaen, Using Feature Model to Support Model‐Based Testing of Product Lines: An Industrial
Case Study, In: Proceedings of the 13th International Conference On Quality Software (QSIC), 2013• Others T. Yue, S. Ali, and B. Selic. Cyber‐Physical System Product Line Engineering: Comprehensive Domain Analysis and
Experience Report, in SPLC 2015.
45 © All rights reserved
![Page 46: SYSTEMATIC MODEL BASED AND SEARCH BASEDTESTING OF …](https://reader030.vdocuments.us/reader030/viewer/2022020910/61ffe3a74aaaca0f6511d7f8/html5/thumbnails/46.jpg)
46
Questions
©All rights reserved