comparing model coverage and code coverage in model driven testing: an exploratory study

i

TESTBEDS 2015 – Lincoln, Nebraska – November 10th

Comparing model coverage and code coverage in Model Driven Testing: an exploratory study

Domenico AmalfitanoVincenzo De SimoneAnna Rita FasolinoVincenzo Riccio

i


Context and MotivationsContext

◦Model Driven Testing (MDT)

Motivation◦To investigate the relationship

between model coverage and code coverage of test suites in a MDT approach.

i


MDE as a new approachIn last years Model-Driven

Engineering (MDE) is widely spreading as approach for developing software systems.◦Ever more used in industries.

Automotive, aircraft, railway, etc.◦Particular emphasis on models and

automated code generation.◦Shift from a Code-centric approach

toward a Model-centric approach

i


MDA and MDTSoftware V-lifecycles may be conducted by

following two MDE approaches:◦ Model Driven Architecture (MDA)

The system code is obtained by stepwise refinements of models at different levels of detail

◦ Model Driven Testing (MDT) The test code is obtained by stepwise refinements of

models at different levels of detail

Transformation is one of the key concepts of MDE approaches.◦ Describes how a model defined in a source language

(source model) can be transformed into one or more models in a target language (target model).

i


MDA and MDT At Platform

Independent level there are views of the system that are independent by the implementing and deploying platform.

At Platform Specific level there are system views combining the specifications of the PIM with the details that specify how the system uses a specific type of platform.MDA approach MDT approach

i


Typical approachIn industry, usually, PSMs and PSTs

are developed and then automatically transformed in system code and test code respectively. ◦Thanks to the support of consolidated

commercial tools: PSM System Code: IBM Rhapsody, Sparx EA,

Visual Paradigm, Matlab StateFlow, Mathworks Simulink, etc.

PST Test Code: Conformiq, Simulink V&V, Simulink Design Verifier etc.

i


Typical approach cont’dUML StateMachine models are widely exploited

for modelling the behaviour of the system at Platform Specific levels.

Usually, the transformations from state machines at PST level towards test code are designed to guarantee that specific model coverage criteria are met, such as the coverage of all states, all transitions, all paths, etc.

For early detection purposes, usually, at early stages of the development process the same PSM may be transformed in PST◦ test code is produced to test the system code.

i


Exploratory StudyIn this context we performed an

exploratory study aimed at understanding:◦Which levels of code coverage testing

adequacy can be reached by a test suite generated to guarantee a specific model coverage testing adequacy.

◦What are the factors that may cause the differences between code coverage and model coverage testing adequacy

i


Study Process

i


Metrics Complexity of UML StateMachine:

◦ #States◦ #Transitions◦ AACC: Average Action Cyclomatic

Complexity. Model coverage reached by a test suite:

◦CS%: percentage of covered states◦CT%: percentage of covered transitions

Code coverage obtained by a test suite◦CSTM%: percentage of covered statements◦CB%: percentage of covered branches

i


Study Process ExecutionAs object of study we considered four

different UML StateMachine models at the PSM level.

Generated test suites guaranteed the following test adequacy criteria:◦TA1: coverage of all states◦TA2: coverage of all transitions

SM1 SM2 SM3 SM4

#States 4 4 8 8#Transitions

9 9 12 12

AACC 1 7 1 2.67

i


ResultsSC1

Test Suite CSTM% CB%TA1 53.1 % 25.0 %TA2 69.9 % 35 %

SC2

Test Suite CSTM% CB%TA1 54.3 % 36.7 %TA2 71.3 % 51.7 %

SC3

Test Suite CSTM% CB%TA1 51.3 % 29.2 %TA2 75.1 % 41.7%

SC4


i


Findings All the executions of the test suites

never reached the 100% of the CSTM% neither of the CB% there are differences between model coverage and code coverage

Mainly due to additional code inserted by the tool implementing the transformation rules from PSM towards system code.◦Code related to exception handling or

execution in debug mode.

i


Refined resultsAdditional

code not considered

SC1*

Test Suite CSTM% CB%TA1 74 % 50 %TA2 100 % 83.3 %

SC2*

Test Suite CSTM% CB%TA1 72 % 57.7 %TA2 96.6 % 84.6 %

SC3*

Test Suite CSTM% CB%TA1 67.8 % 42.8 %TA2 100 % 78.6 %

SC4*


i


Further findingsTest suites covering all the transitions

always reached CSTM%=100% for the code generated from models having AACC equals to 1. ◦The CB% values were always lower than100% .

Mainly due to two reasons:◦Test suites covering all transitions on the model

are not able to exercise all branches on the code. If there are guards on the transitions

◦Test suites are not able to exercise all the code related to the actions. If the developer inserts action code having cyclomatic

complexity > 1.

i


Transition coverage does not assure code coverage - Example 1

Code exercised by executing a test suite able to cover the two transitions of the model.

i


Transition coverage does not assure code coverage - Example 2

• The code related to the confirmRegular() action was completely covered since it had a cyclomatic complexity value equals to one.

• The code related to the confirmSpecial() action was not completely exercised since it has cyclomatic complexity equals to four.

i


Conclusions and Future Works

In conclusion an exploratory study in the context of model driven approaches was performed.◦ It allowed us to understand:

there are differences between model coverage and code coverage, and

which are the main factors influencing these differences.As future work we plan to perform an empirical

study◦ involving a meaningful number of UML

StateMachines, ◦ adopting different PSM (and PST) to code

transformation rules, and ◦ considering several technologies and modelling

styles.

i


Thanks for your attention

Questions?

Further Information:http://reverse.dieti.unina.it

@REvERSE_UNINA

[email protected]

https://twitter.com/REvERSE_UNINA

mailto:[email protected]

mailto:[email protected]