Iterative Development of a Domain-Specific Fault Classification

An Industrial Case Study

Philip Preissing and Jan Schulte2009-06-19

WEMSE Workshop- 2009-06-19

2/16

Table of Contents

● Introduction▬ RUAG

▬ Context

● Related Work & Quality Characteristics

● Development Process

● Case Study

● Discussion

● Summary

RUAG Aerospace Sweden AB

● Headquarter in Göteborg▬ 360 employees

● Formerly SAAB Aerospace

● Highly reliable Satellite Equipment▬ Computers

▬ Antennas

▬ Microwave Systems

● Projects▬ Ariane 5

▬ Herschel/Planck

▬ Galileo

3/16

WEMSE Workshop- 2009-06-19

4/16

Problems at RUAG

● High dependability & reliability

● ECSS Standards

● Late faults several times more expensive

Software Testing accounts for 60% of Development Time

Master thesis to develop an optimization framework for the Verification & Validation Activities (VAs)

WEMSE Workshop- 2009-06-19

5/16

Goals

● Measure defects

Gain insight into VAs

Fault-slippage between VAs

Overlap in VAs

Fault classification(FC) to group similar types of defects

▬ Analyze problems in the process on a high level

▬ Simplify the measurement process

WEMSE Workshop- 2009-06-19

6/16

Example WEMSE Workshop- 2009-06-19

7/16

Related Work

● Schemes exist▬ Orthogonal Defect Classification (IBM)▬ Standard Classification for SW Anomalies (IEEE)▬ Origins, Types & Modes (HP)▬ FC in use at Ericsson

Classifications need to be adapted (Case studies)

● Development processes▬ Expert opinion▬ Commit comments

Development process needed

WEMSE Workshop- 2009-06-19

8/16

Quality characteristics

● Classes should…

… be at most 5-10

… describe the fault type

… be orthogonal

… be consistent

… be complete

… be applicable to every software artifact

WEMSE Workshop- 2009-06-19

9/16

Development Process

1. Initial Fault Classification

2. Iterative Refinement

1. Selection

2. Classification

3. Comparison & Discussion

3. Quality Review

WEMSE Workshop- 2009-06-19

10/16

Case Study – Initial Fault Classification

Class

Assignment

Understandability

Threading/ Concurrency

Function/Interface

Hardware Interface

Timing/ Performance

Algorithms

Robustness

WEMSE Workshop- 2009-06-19

● Select data source▬ Code Inspection sheets

● Classification of faults▬ Completely sure

▬ Uncertain

▬ Don‘t know

● Analysis

Several iterations

11/16

Case Study – Iterative Refinement

10

21

375

398

0

0

27

37

7

7

20

36

42

15

4

1

42

17

38

32

0 100 200 300 400 500

Assignment

Assignment

Understandability

Understandability

Threading/Concurrency

Threading/Concurrency

Algorithms

Algorithms

Function/Interface

Function/Interface

A1A2

A1A2

A1A2

A1A2

A1A2 Uncertain

Completely

WEMSE Workshop- 2009-06-19

12/16

Case Study – Quality Review

● Agreement Factor (Consistency)

▬ Initial: 0.30

▬ Final: 0.71

● Workshop

▬ Project Manager, Designer, Developer, Tester

▬ Only minor changes

< 0.45 Inadequate agreement

> 0.62 Good agreement

> 0.78 Excellent agreement

WEMSE Workshop- 2009-06-19

Advantages & Drawbacks

+ Analysis of all stages

+ Low developer involvement

- Participants have limited understanding

- Validation first performed at the end

13/16

WEMSE Workshop- 2009-06-19

14/16

Lessons learned

● A posteriori analysis difficult

● Documentation important

● Alternative data sources difficult

▬ Checkin comments

▬ Observation of developers

▬ Interviews

WEMSE Workshop- 2009-06-19

15/16

Summary

● Software Testing very expensive

● FC necessary to analyse & improve process

● Process to develop domain-specific FC

▬ Incorporating existing schemes

▬ Iterative refinement by analyzing all steps

▬ Quality review

● Validation in industrial case study at RUAG

WEMSE Workshop- 2009-06-19

16/16

Thank you very much!

Tack så mycket!Muchas gracias!

Vielen Dank!Muito obrigado!

Mille grazie!

Merci beaucoup !

WEMSE Workshop- 2009-06-19

Moltes gràcies!