![Page 1: TMPA-2017: Technology and Tools for Developing Industrial Software Test Suites Based on Formal Models and Implementing Scalable Testing Process on Supercomputer](https://reader031.vdocuments.us/reader031/viewer/2022030306/58e4ac6c1a28abbb038b59f9/html5/thumbnails/1.jpg)
Pavel Drobintsev, Vsevolod Kotlyarov, Ivan Selin, Alexey Tolstoles, Nikita Voinov
Peter the Great Saint Petersburg Polytechnic University
Saint Petersburg, Russia
Scalable testing process based on formal models
![Page 2: TMPA-2017: Technology and Tools for Developing Industrial Software Test Suites Based on Formal Models and Implementing Scalable Testing Process on Supercomputer](https://reader031.vdocuments.us/reader031/viewer/2022030306/58e4ac6c1a28abbb038b59f9/html5/thumbnails/2.jpg)
Model Oriented Approach
• Multilevel model of application during design process
• Behavioral model is built from requirements
• Model is iteratively specified
• Test cases are generated from this model in an automated way
03.03.2017 2
![Page 3: TMPA-2017: Technology and Tools for Developing Industrial Software Test Suites Based on Formal Models and Implementing Scalable Testing Process on Supercomputer](https://reader031.vdocuments.us/reader031/viewer/2022030306/58e4ac6c1a28abbb038b59f9/html5/thumbnails/3.jpg)
VRS/TAT
• Software verification and testing toolset
• Requirements are formalized in UCM language
• VRS verifier runs reachability and consistency checks
• Iterative process of model modification until all checks are passed
• Generation of symbolic traces
• Concretization (symbolic traces – traces with specific values)
• Adapting tests to run them on real system
• Testing
03.03.2017 3
![Page 4: TMPA-2017: Technology and Tools for Developing Industrial Software Test Suites Based on Formal Models and Implementing Scalable Testing Process on Supercomputer](https://reader031.vdocuments.us/reader031/viewer/2022030306/58e4ac6c1a28abbb038b59f9/html5/thumbnails/4.jpg)
Use Case Maps (UCM)
• High-level graphical language for requirements modelling
• ITU Z.151
• Can be used for creating structured behavioral diagrams of interacting agents
• UCM notation doesn’t provide enough information to generate traces
03.03.2017 4
![Page 5: TMPA-2017: Technology and Tools for Developing Industrial Software Test Suites Based on Formal Models and Implementing Scalable Testing Process on Supercomputer](https://reader031.vdocuments.us/reader031/viewer/2022030306/58e4ac6c1a28abbb038b59f9/html5/thumbnails/5.jpg)
UCM Metadata
• Auxiliary text field linked to a UCM responsibility
• Stores information about variables and signals
• Signal is a way of message transfer between agents
03.03.2017 5
![Page 6: TMPA-2017: Technology and Tools for Developing Industrial Software Test Suites Based on Formal Models and Implementing Scalable Testing Process on Supercomputer](https://reader031.vdocuments.us/reader031/viewer/2022030306/58e4ac6c1a28abbb038b59f9/html5/thumbnails/6.jpg)
VRS Verifier
• During the initial model development:• Reachability and consistency checks
• During testing process:• Generation of symbolic traces based on UCM model control flow
• Concretization (substitution of left, right and middle values from tolerance range)
03.03.2017 6
![Page 7: TMPA-2017: Technology and Tools for Developing Industrial Software Test Suites Based on Formal Models and Implementing Scalable Testing Process on Supercomputer](https://reader031.vdocuments.us/reader031/viewer/2022030306/58e4ac6c1a28abbb038b59f9/html5/thumbnails/7.jpg)
VRS/TAT Levels of Abstraction
03.03.2017 7
Automated
Automated
Not automated
![Page 8: TMPA-2017: Technology and Tools for Developing Industrial Software Test Suites Based on Formal Models and Implementing Scalable Testing Process on Supercomputer](https://reader031.vdocuments.us/reader031/viewer/2022030306/58e4ac6c1a28abbb038b59f9/html5/thumbnails/8.jpg)
Challenges
• Data mapping problem:• Abstract models which need to be extended in order to run against real
system
• Extension may require additional information
• New information must comply with what’s already in the model
• Amount of generated test cases may be too big to run using common methods
03.03.2017 8
![Page 9: TMPA-2017: Technology and Tools for Developing Industrial Software Test Suites Based on Formal Models and Implementing Scalable Testing Process on Supercomputer](https://reader031.vdocuments.us/reader031/viewer/2022030306/58e4ac6c1a28abbb038b59f9/html5/thumbnails/9.jpg)
Data Structures Conversion Problem
• Structure of concrete test signals does not correspond with real system signals
• Manual creation of detailed test scenarios takes too much time and resources
03.03.2017 9
![Page 10: TMPA-2017: Technology and Tools for Developing Industrial Software Test Suites Based on Formal Models and Implementing Scalable Testing Process on Supercomputer](https://reader031.vdocuments.us/reader031/viewer/2022030306/58e4ac6c1a28abbb038b59f9/html5/thumbnails/10.jpg)
Data Structures Conversion. Approach
• Process called “Lowering”
• The name comes from descending on lower levels of abstraction
• In general, “Lowering” can be described as creating processing rules for each signal and application of these rules to concrete scenarios
• An editor was made to restrict user from making incorrect structures
03.03.2017 10
![Page 11: TMPA-2017: Technology and Tools for Developing Industrial Software Test Suites Based on Formal Models and Implementing Scalable Testing Process on Supercomputer](https://reader031.vdocuments.us/reader031/viewer/2022030306/58e4ac6c1a28abbb038b59f9/html5/thumbnails/11.jpg)
Data Structures Conversion. Restrictions
• There are several restrictions to ensure the correctness of test scenarios:• If you separate concrete value into several independent parts, it is prohibited
to change them in a way, when joining them back together will give another result than it was before separation
• Only structures similar to SUT interfaces could be used
• Only constant values from concretization and templates values are allowed
03.03.2017 11
![Page 12: TMPA-2017: Technology and Tools for Developing Industrial Software Test Suites Based on Formal Models and Implementing Scalable Testing Process on Supercomputer](https://reader031.vdocuments.us/reader031/viewer/2022030306/58e4ac6c1a28abbb038b59f9/html5/thumbnails/12.jpg)
Data Structures Conversion. Results
• Before introduction of “lowering”, test mapping was made by hand after each tests generation
• After automating the tests mapping there are only 2 manual steps left in VRS/TAT process:• UCM model development
• Specifying “lowering” rules
• Both of them only needed to be done once
03.03.2017 12
![Page 13: TMPA-2017: Technology and Tools for Developing Industrial Software Test Suites Based on Formal Models and Implementing Scalable Testing Process on Supercomputer](https://reader031.vdocuments.us/reader031/viewer/2022030306/58e4ac6c1a28abbb038b59f9/html5/thumbnails/13.jpg)
UPD: VRS/TAT Levels of Abstraction
03.03.2017 13
Automated
Automated
Automated
Huge amount
![Page 14: TMPA-2017: Technology and Tools for Developing Industrial Software Test Suites Based on Formal Models and Implementing Scalable Testing Process on Supercomputer](https://reader031.vdocuments.us/reader031/viewer/2022030306/58e4ac6c1a28abbb038b59f9/html5/thumbnails/14.jpg)
Execution problem
• Huge test amount
• A lot of time needed for execution
• Costs increase
• Run tools in parallel
03.03.2017 14
![Page 15: TMPA-2017: Technology and Tools for Developing Industrial Software Test Suites Based on Formal Models and Implementing Scalable Testing Process on Supercomputer](https://reader031.vdocuments.us/reader031/viewer/2022030306/58e4ac6c1a28abbb038b59f9/html5/thumbnails/15.jpg)
VRS Guided Search
• VRS has an option to perform guided search
• Guide is a marked sequence on a model, which must be traversed
• All traces that don’t apply the criteria to travel through guide are cut off by VRS during traces generation
• The idea is to set guides on UCM model and launch several VRS instances with corresponding guides
03.03.2017 15
![Page 16: TMPA-2017: Technology and Tools for Developing Industrial Software Test Suites Based on Formal Models and Implementing Scalable Testing Process on Supercomputer](https://reader031.vdocuments.us/reader031/viewer/2022030306/58e4ac6c1a28abbb038b59f9/html5/thumbnails/16.jpg)
VRS Guide
03.03.2017 16
![Page 17: TMPA-2017: Technology and Tools for Developing Industrial Software Test Suites Based on Formal Models and Implementing Scalable Testing Process on Supercomputer](https://reader031.vdocuments.us/reader031/viewer/2022030306/58e4ac6c1a28abbb038b59f9/html5/thumbnails/17.jpg)
Execution process
• User sets guides in the most “thick” places of UCM model
• Several VRS instances run, each on a different compute node with a unique set of guides
• As a result, whole test suite is divided into several parts
• Each part can be processed independently in an isolated container (node, VM, etc.)
03.03.2017 17
![Page 18: TMPA-2017: Technology and Tools for Developing Industrial Software Test Suites Based on Formal Models and Implementing Scalable Testing Process on Supercomputer](https://reader031.vdocuments.us/reader031/viewer/2022030306/58e4ac6c1a28abbb038b59f9/html5/thumbnails/18.jpg)
Scalability
• Theoretically, scalability is linear
• In real life, it is not achievable because of irregular test distribution and different execution complexity of test cases
03.03.2017 18
![Page 19: TMPA-2017: Technology and Tools for Developing Industrial Software Test Suites Based on Formal Models and Implementing Scalable Testing Process on Supercomputer](https://reader031.vdocuments.us/reader031/viewer/2022030306/58e4ac6c1a28abbb038b59f9/html5/thumbnails/19.jpg)
Conclusion
• Tests are generated from the verified model and can be ran against real systems
• Presented approach allows to speed up the testing process
• It speeds up both tests generation and tests execution
• Scalable process, performance increase is limited by the model and/or available computational resources
03.03.2017 19
![Page 20: TMPA-2017: Technology and Tools for Developing Industrial Software Test Suites Based on Formal Models and Implementing Scalable Testing Process on Supercomputer](https://reader031.vdocuments.us/reader031/viewer/2022030306/58e4ac6c1a28abbb038b59f9/html5/thumbnails/20.jpg)
Thanks for your attention!
03.03.2017 20
![Page 21: TMPA-2017: Technology and Tools for Developing Industrial Software Test Suites Based on Formal Models and Implementing Scalable Testing Process on Supercomputer](https://reader031.vdocuments.us/reader031/viewer/2022030306/58e4ac6c1a28abbb038b59f9/html5/thumbnails/21.jpg)
Backup 1. New features in lowering. TDL
• There is an ability to use SUT interfaces files (*.tdl) in Lowering Editor to obtain parameters signal structure
03.03.2017 21
![Page 22: TMPA-2017: Technology and Tools for Developing Industrial Software Test Suites Based on Formal Models and Implementing Scalable Testing Process on Supercomputer](https://reader031.vdocuments.us/reader031/viewer/2022030306/58e4ac6c1a28abbb038b59f9/html5/thumbnails/22.jpg)
Backup 2. Few words on VRS
• VRS works with inner model in basic protocols (Hoare triples)
• Each one has:• Pre-condition
• Process
• Post-condition
03.03.2017 22
![Page 23: TMPA-2017: Technology and Tools for Developing Industrial Software Test Suites Based on Formal Models and Implementing Scalable Testing Process on Supercomputer](https://reader031.vdocuments.us/reader031/viewer/2022030306/58e4ac6c1a28abbb038b59f9/html5/thumbnails/23.jpg)
Backup 2. Few words on VRS
• UCM model is converted into basic protocols model using UCM elements location and UCM metadata (and inverse)
• VRS matches pre and post conditions of different elements and checks tolerance ranges to find all possible traces
• Guided search builds traces from guide forward and backward
03.03.2017 23