ISSRE 2013– Pasadena, CA, USA

Arbi Ghazarian Department of Engineering & Computing Systems

Arizona State University Email: Arbi.Ghazarian@asu.edu

Detection of Missing Requirements Using Base Requirements Pairs

37978-1-4799-2552-0/13/$31.00 ©2013 IEEE 61

2

Incomplete Specifications & Software Defects

A. Ghazarian, A Case Study of Defect Introduction Mechanisms, Proc. of CAiSE’09, Springer LNCS, pp. 156-170, 2009

42.5% of all software defects in enterprise systems are faults of omission (i.e., missing code statements). In 82% of faults of omission, the requirements engineers had not explicitly included the corresponding statements of requirements in the specifications.

3862

3

Current Solution

Requirements Inspection Ad-hoc Check-List Based Scenario-Based

Reliance on Inspector’s experience and domain knowledge Domain-Agnostic (i.e., generic) Non-Systematic

3963

4

Criteria for a New Solution

Domain-Specific

Less reliance on inspector’s experience and domain knowledge More systematic that existing approaches Lightweight approach with high ROI Can be automated

4064

5

Observations and Insights

Often, the existence of one type of functional requirement makes the existence of another relevant type of functional requirement highly probable.

In specifications of systems, certain classes of requirements often appear together, forming patterns of base requirement pairs. A software requirement whose class participates in a known base pair pattern, but whose complementary base-forming requirement is not specified is indicative of a missing requirement.

4165

6

The Rule Derivation Process

1. Classification of atomic statements of FRs obtained from past projects • Grounded-theory-based comparative data analysis

2. Distinguish between the core and non-core requirements classes • Compute the statistical frequency distributions of FRs over the identified classes

3. Analyze and compile a list of relationships among the identified core classes 4. Rule pruning – eliminate the infrequent rules

4266

7

FR Classes and Their Frequency Distributions

Type %

Data Output 26.37

Data Input 19.88

Event Trigger 16.18

Business Logic 11.66

Data Persistence 10.84

User Interface Navigation

4.84

External Call 2.62

Communication 2.30

User Interface 1.97

User Interface Logic 1.64

Data Validation 0.98

External Behavior 0.65

Ghazarian, A., Characterization of Functional Requirements Space: The Law of Requirements Taxonomic Growth, Proceedings of 20th IEEE International Requirements Engineering Conference (RE’12), pp. 241-250, Chicago, IL, USA, September 2012. 4367

8

FR Classes and Their Frequency Distributions

Type %

Data Output 26.37

Data Input 19.88

Event Trigger 16.18

Business Logic 11.66

Data Persistence 10.84

User Interface Navigation

4.84

External Call 2.62

Communication 2.30

User Interface 1.97

User Interface Logic 1.64

Data Validation 0.98

External Behavior 0.65

} High-Frequency FRs

85%

4468

9

FR Classes and Their Frequency Distributions

Type %

Data Output 26.37

Data Input 19.88

Event Trigger 16.18

Business Logic 11.66

Data Persistence 10.84

User Interface Navigation

4.84

External Call 2.62

Communication 2.30

User Interface 1.97

User Interface Logic 1.64

Data Validation 0.98

External Behavior 0.65

} High-Frequency FRs

85%

} Low-Frequency FRs

15% 4569

10

System Make-Up (High- vs Low-Frequency FRs)

System % of High-Frequency FRs

% of Low-Frequency FRs

1 91.13 8.87

2 70.97 29.03

3 71.43 28.57

4 81.13 18.87

5 84.37 15.63

6 100 0

7 90.91 9.09

8 78.43 21.57

9 82.14 17.86

10 87.18 12.82

11 69.52 30.48

12 82.67 17.33

13 83.33 16.67

14 72.81 27.19

15 93.05 6.95

4670

11

Sample Rules

• For every Data Input (DI), there exists one or more corresponding

requirements of type Data Validation (DV) : {DIDV}

• For every DV, there exists one or more corresponding requirements of type Data Output (DO): {DVDO}

• For every DI, there exists one or more corresponding requirement of type Data Persistence (DP): {DIDP}

• For every Use Case (UC), there exists one or more requirement of type DV: {UCDV} • For every UC, there exists one or more requirement of type Business Logic (BL): {UCBL}

4771

12

Evaluation of the Rules

Applied the rules to specifications for 5 industrial software systems (302 requirements)

Defect Detection Rate (DDR): ratio of detected defects to the number of rule applications

• A measure of inspection effort in terms of rule executions Defect Detection Precision (DDP) : ration of detected defects to the number of warnings generated

• Percentage of real warnings vs. false positives

4872

13

Overall Results

DDR: 35% Roughly, the execution of every 3 rules resulted in the detection of a missing requirement.

DDP: 68% Detected 196 missing requirements

4973

14

More Results

DDR: 35% Roughly, the execution of every 3 rules resulted in the detection of a missing requirement.

DDP: 68%

Detected 196 missing requirements (n) UCDV: n = 48, DDR = 94%, DDP = 100% DIDV: n =23, DDR = 32%, DDP = 35% UCDP: n = 23, DDR = 45%, DDP = 92% DI DP: n = 18, DDR = 25%, DDP = 34%

5074

15

More Results

Over 45% of the base pair rules achieved a DDP of 100%

Over 81% of the base pair rules achieved a DDP of 84% and over.

5175

16

The Domino Effect

Chained execution of the rules: The execution of a rule can trigger the execution of another rule, increasing the effectiveness of the approach.

DIDV (n=23)DO (n=23)

The rule-based approach can be used in an iterative fashion.

5276

17

Quality Measurement Using Base-Pair Rules

The base-pair rules approach can be used for in-process measurement of the quality of the requirements.

As missing requirements are detected and added to the specification, the DDR and DDP measures gradually drop in performance. The poor performance of the approach on a specification is indicative of the high quality of the specification (in terms of specification completeness).

5377

18

The Cost of the Approach

The cost of deriving the rules (dominant cost) manual corpus-mining process (grounded-theory-based analysis) The reuse of the rules in subsequent projects will not only offset this cost, but also yield a high ROI.

The cost of applying the rules

5478

19

Contributions

A light-weight rule-based approach to requirements inspection for enterprise systems

A reusable process for generating inspection rules for other software domains

5579

20

Future Work

Deriving and evaluating base-pair requirements rules for other software domains

Scientific Software Smart-house Systems

Automation of the approach

5680