risk-based regression testing in a telecommunication system node

Slide titleIn CAPITALS

50 pt

Slide subtitle 32 pt

Risk-based regression testing in a telecommunication system

node

Master’s thesis presentation 14.8.2007

Author: Mika Lehto

Supervisor: Professor Heikki Hämmäinen

Instructor: M.Sc. Pekka Hämäläinen

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt


Text 24 pt

Bullets level 2-520 pt

Ericsson Confidential 2007-07-202

Agenda

Background and objectives Risk-based testing Regression testing Case Study

– Ericsson’s Media Gateway for Mobile Networks (M-MGw)– M-MGw Node Functional Verification– Risk model– Risk analysis and prioritization– Scenarios for the case study– Results

Conclusions Future research


Slide title 40 pt


Text 24 pt



Background

The costs of regression testing comprise 20-30 % of the whole testing costs on average in software development

→ regression testing is often the target of cost reduction activities in order to achieve shorter time-to-market and

lower cost structure of the product The efficiency of regression testing is usually attempted to

improve by using regression testing methods such as regression test selection, test suite reduction, and test case prioritization

One approach that has been studied is called risk-based regression testing but only a little information about the feasibility of risk-based regression testing is currently available


Slide title 40 pt


Text 24 pt



Objectives

The primary objective of this thesis was to study whether it would have been beneficial to use risk-based regression testing in an old M-MGw Node Functional Verfication project

This thesis also analyzed the differences in test case selections between previously used and risk-based test case selections methods

The aim of this thesis was also to demonstrate how risk-based testing approach can be utilized in regression testing


Slide title 40 pt


Text 24 pt



Risk-based testing

The fundamental difference between traditional testing and risk-based testing is that risk-based testing brings formal risk assessment methods (risk identification, risk analysis, and risk prioritization) in testing process

The aim of risk-based testing is to focus the test effort on the most critical areas of the system

Risk-based testing approach can be used for instance to:

– Select the most important test objects and test cases– Evaluate the current level of system quality and risk– Provide input for the decision to continue or stop testing


Slide title 40 pt


Text 24 pt



Regression testing

“Selective retesting of a system or component to verify that modifications have not caused unintended effects and that the system or component still complies with its specified requirements” (IEEE Software glossary)

Regression testing is typically used during iterative software development and to support application maintenance

The biggest challenge in regression testing is the test case selection

In risk-based regression testing, formal risk analysis methods are used to select the most risky test cases for execution


Slide title 40 pt


Text 24 pt



Ericsson’s Media Gateway for Mobile Networks (M-MGw) M-MGw operates in the connectivity layer of the core network and

its fundamental purpose is to connect the core network to other networks such as UTRAN and PSTN

M-MGw also supports media conversion, bearer control and payload processing

M-MGw is remotely controlled by the MSC or SGSN Servers which operate in the network control layer.


Slide title 40 pt


Text 24 pt



M-MGw Node Functional Verification Testing in the NFV is based on requirements (e.g. Specifications

and call flows) NFV is a testing phase in which the M-MGw software is first time

tested in a real node environment Test environment is established of one M-MGw node and all the

external interfaces to this node are simulated


Slide title 40 pt


Text 24 pt



Risk model

The case study was based on a qualitative risk model and a risk was measured as a risk exposure:

Risk Exposure = Probability x Consequence

Risk items were categorized into three classes according to the probability and consequence of a risk: tolerable, undesirable, and intolerable


Slide title 40 pt


Text 24 pt



Risk analysis and prioritization

Risk analysis was performed both for the test objects and test cases using systems experts

– 2 systems experts participated into the risk analysis of test objects and 8 systems experts analyzed the risks related to test cases

Test objects and test cases were prioritized according to risk exposure– In a case of test case prioritization, also other prioritization criteria

(consequence, old test case priority) were needed because several test cases had the same risk exposure


Slide title 40 pt


Text 24 pt



Scenarios

The analysis was based on 5 scenarios in which the amount of executed test cases was changed

In scenario 1, as many test cases were selected for execution per test object as in the original case but the test cases were selected based on risk

In other scenarios, the test effort was focused on the most risky test objects by defining how many percent of the test cases per test object were executed

– Scenario 2: Intolerable 25%, Undesirable 15%, Tolerable 10% – Scenario 3: Intolerable 20%, Undesirable 10%, Tolerable 5%– Scenario 4: Intolerable 15%, Undesirable 10%, Tolerable 0%– Scenario 5: Intolerable 10%, Undesirable 5%, Tolerable 0%


Slide title 40 pt


Text 24 pt



Results – used man hours

Significant savings in used man hours would have been achieved in scenarios 3-5

In the original case, the used man hours were distributed equally to all test objects

In risk-based approach (i.e. scenarios 2-5), approximately 55 % of man hours were consumed to 20 % of test objects

Distribution of used man hours

0 %

10 %

20 %

30 %

40 %

50 %

60 %

70 %

80 %

90 %

100 %

0 % 20 % 40 % 60 % 80 % 100 %

% of test objects

% o

f m

an h

ou

rs Original

Scenario 1

Scenario 2

Scenario 3

Scenario 4

Scenario 5


Slide title 40 pt


Text 24 pt



Results – risk level

Risk level defines how many percent of the total risk is still uncovered

Risk level would have been 10 percentage units lower in the original case if the test cases had been selected based on risk

In the scenarios 3-4, risk level is at the same level than in the original case even though much less man hours were consumed

Risk Levels of Scenarios

0 %

10 %

20 %

30 %

40 %

50 %

60 %

70 %

80 %

90 %

100 %

Original Scenario 1 Scenario 2 Scenario 3 Scenario 4 Scenario 5

Ris

k L

evel

(%

)


Slide title 40 pt


Text 24 pt



Results – detected faults

Only 40 % of the faults were found in scenario 1 in which the test cases were selected based on risk but the test effort was not concentrated on the most risky test objects

By concentrating also the test effort to the most risky test objects, 40-60 % of the faults were found

Deeper analysis to these poor fault-detection rates revealed that the risk analysis of test cases of TO_04 had failed and caused faults to slip through testing

Distribution of detected faults

0 %

10 %

20 %

30 %

40 %

50 %

60 %

70 %

80 %

90 %

100 %

Original Scenario 1 Scenario 2 Scenario 3 Scenario 4 Scenario 5

Per

cen

tag

e o

f th

e d

etec

ted

fau

lts

TO_81

TO_79

TO_04


Slide title 40 pt


Text 24 pt



Results – test case selections

Differencies in test case selections between previously used and risk-based approach was analyzed by determining test case coverage which defines how many percent of test cases executed in the original case was also executed in designed scenarios

In addition, the average risk exposure per test case was determined to each scenario

Test case coverage in proportion to the original case

0 %10 %20 %30 %40 %50 %60 %70 %80 %90 %

100 %

Tes

t ca

se c

ove

rag

e (%

)

Test case coverage 44 % 31 % 23 % 18 % 12 %

Scenario 1 Scenario 2 Scenario 3 Scenario 4 Scenario 5


Slide title 40 pt


Text 24 pt



Conclusions

Risk-based regression testing would not have brought cost savings in used man hours because the costs of the faults that slipped through testing would have eliminated all the gained cost savings

Testing effort was not concentrated on the most risky test object in the original case

It can not be said that testers would have utilized risk-based testing approach when selecting regression test cases in the original case


Slide title 40 pt


Text 24 pt



Future Research

A possible pilot project for risk-based testing in order to get more empirical experience how risk-based testing works in practice

The concept and usage of risk level should be investigated more because risk level could be possible used as an indicator to tell when it is time to stop testing

To develop a framework for assessing the costs of the faults quantitatively


Slide title 40 pt


Text 24 pt



risk-based regression testing in a telecommunication system node

Documents