empirical study of software engineering results · empirical study of software engineering results...

Empirical Study of Software Engineering Results

Software Engineering Institute Carnegie Mellon University Pittsburgh, PA 15213 TSP Initiative, SEAP/SDD Sept. 18, 2013

If you are a data lover …

… then this interactive is for you.

If you are a data-lover, don’t be shy …

… because you are in great company.

W. Edwards Deming

“When it comes to the really important

decisions, data trumps intuition every time.”

Jeff Bezos

That’s 12 quadrillion bytes

12 petabytes for data storage

… or 1015 bytes

TSP Symposium Attendee

Introduction

Format of the “Interactive”

Data Provenance

The Data

Next Steps

Topics

Please ask questions!

And … we will be asking you questions!

Looking for Fresh Ideas

Looking for FRESH ideas …

All comments are welcome!

What is the single most important question that you would want to be addressed through the analysis of TSP data?

Before

Introduction

Data Provenance

The Data

Next Steps

Topics

What Is Data Provenance?

Provenance, from the French provenir, "to come from," refers to the chronology of the ownership, custody or location of a historical object.

Data provenance refers to a record trail that accounts for the origin of a piece of data (in a document, repository, or database) together with an explanation of how it got to the present place.

Data provenance assists scientists with tracking their data through all transformations, analyses, and interpretations.

Master Jim at work … prying the data from the tool.

The TSP Data Repository includes two archives of approximately 50,000 files.

• 20-25% of those are supporting documents for a launch or postmortem (e.g., presentations, analysis spreadsheets, lists, surveys, etc.).

• 75-80% of those are TSP team performance data files.

There are more than 50 different file formats.

• At this time, only 22 of the needed file import utilities have been developed.

• Only those that contain TSP cycle or postmortem data were processed.

• About 60% of Archive #1 fit the criteria and have been processed. Archive #2 has not been processed yet.

Tests were conducted to ensure that extracted data represented unique projects.

Analysis Based on Teams’ Composite Data The data is composite team data; each record represents data that has been aggregated from individual team member data.

Team composite data

Individual team member data

To Analysis

Team composite data

Individual team member data

Statistical Analysis

Outlier Analysis

Exploratory analysis yielded some outliers that were removed from some of the analyses.

Introduction

Data Provenance

The Data • Team and Project Characteristics • Product Size • Schedule Performance • Quality Indicators

Topics

Country Source

United States Mexico

Project Start Date

Jul-2012Jul-2010Jul-2008Jul-2006Jul-2004Jul-2002Jul-2000

Project Start Date

Frequency

Project Duration (Calendar Days)

6005004003002001000

Project Duration (Days)

Mean = 119.4 Median = 91.0 n = 113

80644832160

Duration (Weeks)

Frequency

Project Duration (Weeks)

Mean = 16.9 Median = 13.0 n = 113

Team Size

181512963

Team Size

Frequency

Mean = 8.2 Median = 8.0 n = 111

What other types of “team and project characteristics” analyses would you find valuable?

Introduction

Data Provenance

Next Steps

Topics

24020016012080400

Total Size (KLOC)

Frequency

Actual Total Size

Mean = 24.0 Median = 3.8 n = 111

6050403020100

Actual Added & Modified Code (KLOC)

Frequency

Actual Added and Modified Size

Mean = 11.7 Median = 6.6 n = 112

Another View of the Size Data From the data, calculate the

1. log of each value

2. mean (�̅�𝑥) of values from #1

3. standard deviation (𝑠𝑠) of values from #1

4. exponent of values from #2 and #3.

Values of the relative size table:

Very Small

Medium

Very large

�̅�𝑥 − 2𝑠𝑠

�̅�𝑥 − 𝑠𝑠

�̅�𝑥

�̅�𝑥 + 𝑠𝑠

�̅�𝑥 + 2𝑠𝑠 �̅�𝑥 −𝑠𝑠 −2𝑠𝑠 2𝑠𝑠 𝑠𝑠 M S VS L VL

Bin Width = 1 SD

287 1,265 5,571 24,532 108,022

Frequency

Added and Modified Lines of Code

Added & Modified Code – Relative Sizes

VS S M L VL

Relative Size

What would you like to see in terms of analyses associated with product size?

Introduction

Data Provenance

Next Steps

Topics

What do you think is the average number of weekly task hours that teams are able to accomplish?

Mean Team Member Weekly Task Hours

23.620.016.412.89.25.62.0

Mean Team Member Weekly Task Hours

Frequency

Mean = 10.3 Median = 9.0 n = 111

9075604530150

Productivity (LOC/Hr)

Frequency

Productivity

Mean = 10.3 Median = 7.1 n = 112

Let’s Look At Some Scatter Plots

y Using linear regression:

y = mx + b

where: m is the slope b is the y-intercept

r is a measure of the correlation between x values and y values.

Values of r2 ≥ 0.5 indicate a meaningful relationship.

There Are A Few Rules …

Plan Task Hours Vs. Actual Task Hours

R² = 0.8038

0 2000 4000 6000 8000 10000

Actual Task Hours

Plan Task Hoursn=113

Duration: Planned Weeks Vs. Actual Weeks

R² = 0.7401

0 20 40 60 80 100 120

Duration - Actual Weeks

Duration - Planned Weeksn = 113

Actual Task Hours Vs. Added & Modified LOC

R² = 0.3012

10,000

20,000

30,000

40,000

50,000

60,000

70,000

0 2,000 4,000 6,000 8,000 10,000

Actual Added & Modified LOC

Actual Task Hoursn = 113

Team Size vs. Productivity

R² = 0.0297

0 5 10 15 20

Productivity(LOC/Hr)

Team Sizen = 111

Actual Added & Modified LOC Per Staff Week

2000160012008004000

Total LOC Per Staff Week

Frequency

Mean = 204.9 Median = 100.9 n = 109

n = 109 Actual A&M LOC Per Staff Week

Plan Vs. Actual Hours for Completed Parts

R² = 0.952

10,000

0 2,000 4,000 6,000 8,000 10,000

Actual Hours for Completed Parts

Plan Hours for Completed Partsn = 113

Schedule Growth Beyond Baseline

12080400-40-80

Schedule Growth from Baseline (hours)

Frequency

Mean = 2.8 Median = 0.0 n = 113

907560453015

Final Earned Value

Frequency

Final Earned Value

Mean = 85.2 Median = 94.3 n = 110

Are there other types of “schedule performance” analyses that you would like to see?

Introduction

Data Provenance

Next Steps

Topics

Total Defects Injected Per KLOC

480400320240160800

Defects Injected per KLOC

Frequency

Mean = 54.7 Median = 31.5 n = 79

Defect Density – DLD Review

2824201612840

Defects Per KLOC - DLD Review

Frequency

Mean = 5.2 Median = 2.2 n = 106

Defect Density – Code Review

33.628.824.019.214.49.64.80.0

Defects Per KLOC - Code Review

Frequency

Mean = 8.4 Median = 5.2 n = 106

120100806040200

Defects Per KLOC - Inspection

Frequency

Defect Density – Code Inspection

Mean = 6.7 Median = 3.3 n = 106

Defect Density – Unit Test

847260483624120

Defects Per KLOC - Unit Test

Frequency

Mean = 5.0 Median = 3.8 n = 106

Defect Density – Build and Integration Test

16.814.412.09.67.24.82.40.0

Defects Per KLOC - Build and Integration Test

Frequency

Mean = 1.7 Median = 0.7 n = 106 < 0.5

15129630

Defects Per KLOC - System Test

Frequency

Defect Density – System Test

Mean = 1.5 Median = 0.15 n = 106 < 0.2

2824201612840

Defects Per KLOC - DLD Review

Frequency

33.628.824.019.214.49.64.80.0

Defects Per KLOC - Code Review

Frequency

120100806040200

Defects Per KLOC - Inspection

Frequency

847260483624120

Defects Per KLOC - Unit Test

Frequency

15129630

Defects Per KLOC - Build and Integration T

Frequency

15129630

Defects Per KLOC - System Test

Frequency

Defect Density - Summary

Defect Density – Median of Defects Per KLOC

0 1 2 3 4 5 6

System Test

Build/Integration Test

Unit Test

Code Inspection

Code Review

DLD Review

Defects Per KLOC (Median)

Actual UT Defect Density Vs. ST Defect Density

R² = 0.0031

0 5 10 15 20 25 30

System Test Defect Density

Unit Test Defect Densityn=111

The Yield Quality Measure

Defect removal phase

Development phase injects defects Intermediate product with defects

Phase Yield

Phase Yield Defect removal phase

Defect removal phase

The yield of a phase is the percentage of defects removed in that phase.

Phase Yield

Yield: Detailed-Level Design Review

1.00.80.60.40.20.0

Yield - Detailed-Level Design Review

Frequency

Mean = 31.6% Median = 29.3% n = 98

1.00.80.60.40.20.0

Yield - Detailed-Level Design Inspection

Frequency

Yield: Detailed-Level Design Inspection

Mean = 49.3% Median = 50.0% n = 83

0.900.750.600.450.300.150.00

Yield - Code Review

Frequency

Yield: Code Review

Mean = 29.2% Median = 30.1% n = 109

0.900.750.600.450.300.150.00

Yield - Code Inspection

Frequency

Yield: Code Inspection

Mean = 30.3% Median = 26.1% n = 110

1.00.80.60.40.20.0

Yield - Unit Test

Frequency

Yield: Unit Test

Mean = 49.7% Median = 46.2% n = 106

Summary: Median Phase Yields

0.0 10.0 20.0 30.0 40.0 50.0 60.0

Unit Test

Code Inspection

Code Review

DLD Inspection

DLD Revew

Process Yield

1.00.90.80.70.60.50.4

Process Yield

Frequency

Mean = 73.6% Median = 73.7% n = 77

Review of Some Definitions …

In the TSP:

𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨𝑨 𝑪𝑪𝑪𝑪𝑪𝑪 =𝑹𝑹𝑹𝑹𝑹𝑹𝑨𝑨𝑹𝑹𝑹𝑹 & 𝑰𝑰𝑰𝑰𝑨𝑨𝑨𝑨𝑹𝑹𝑰𝑰𝑰𝑰𝑨𝑨𝑰𝑰𝑰𝑰 𝑻𝑻𝑨𝑨𝑻𝑻𝑹𝑹𝑻𝑻𝑰𝑰𝑰𝑰𝑨𝑨𝑨𝑨 𝑫𝑫𝑹𝑹𝑹𝑹𝑹𝑹𝑨𝑨𝑰𝑰𝑨𝑨𝑻𝑻𝑹𝑹𝑰𝑰𝑰𝑰 𝑻𝑻𝑨𝑨𝑻𝑻𝑹𝑹

× 𝟏𝟏𝟏𝟏𝟏𝟏

𝑭𝑭𝑨𝑨𝑨𝑨𝑨𝑨𝑭𝑭𝑨𝑨𝑹𝑹 𝑪𝑪𝑪𝑪𝑪𝑪 =𝑻𝑻𝑹𝑹𝑨𝑨𝑰𝑰 𝑻𝑻𝑨𝑨𝑻𝑻𝑹𝑹

𝑻𝑻𝑰𝑰𝑰𝑰𝑨𝑨𝑨𝑨 𝑫𝑫𝑹𝑹𝑹𝑹𝑹𝑹𝑨𝑨𝑰𝑰𝑨𝑨𝑻𝑻𝑹𝑹𝑰𝑰𝑰𝑰 𝑻𝑻𝑨𝑨𝑻𝑻𝑹𝑹× 𝟏𝟏𝟏𝟏𝟏𝟏

Appraisal Cost of Quality

50403020100

Appraisal Cost of Quality (Percent)

Frequency

Mean = 26.6% Median = 27.0% n = 86

Failure Cost of Quality

75604530150

Failure Cost of Quality (Percent)

Frequency

Mean = 22.0% Median = 16.9% n = 86

R² = 0.0347

0 10 20 30 40 50 60

Defects Per KLOC Removed During Appraisal Phases

Appraisal Cost of Quality (Percent)

Appraisal: COQ vs. Defects Removed

R² = 0.495

0 100 200 300 400 500

Defects Removed in Code Review

Actual Time in Code Review (Hours)

Time in Code Review Vs. Defects Removed

R² = 0.6272

0 100 200 300 400 500 600

Defects Removed

DuringInspection

Actual Time in Inspection (Hours)

Time in Inspection Vs. Defects Removed

R² = 0.3505

0 200 400 600 800 1000 1200

Actual Time in Unit Test (Hours)

Time in Unit Test Vs. Defects Removed

R² = 0.1996

0 100 200 300 400 500 600 700

Defects Found

Actual Time in System Test (Hours)

Time in System Test Vs. Defects Removed

Summary: Time in Phase Vs. Defects Removed

Phase Correlation

Appraisal Code Review 0.50 Inspection 0.63

Test Unit Test 0.35 System Test 0.20

Predictive

Code Review to Code – Actual Time in Phase

0.900.750.600.450.300.150.00

Code Review to Code - Actual Time

Frequency

Mean = 0.3 Median = 0.3 n = 113 > 0.5

4.503.753.002.251.500.750.00

Design to Code - Actual Time

Frequency

Design to Code – Actual Time in Phase

Mean = 0.9 Median = 0.8 n = 113 > 1.0

Design Review to Design – Actual Time in Phase

1.21.00.80.60.40.20.0

Design Review to Design - Actual Time

Frequency

Mean = 0.28 Median = 0.23 n = 113 > 0.5

3.753.002.251.500.750.00

Req. Inspection to Req. - Actual Time

Frequency

Req. Inspection to Req. – Actual Time in Phase

Mean = 0.88 Median = 0.46 n = 19

> 0.25

For “quality indicators,” what additional analyses would you like to see?

What is the single most important question that you would want addressed through the analysis of TSP data?

Topics

Introduction

Data Provenance

Next Steps

• Review your feedback from today and adjust the analysis approach accordingly.

• Extract the data from Process Dashboard tool submitted files.

• Extract and analyze individual team member data.

• Continue with the data analysis. Publish the results.

Contact Information

Mark Kasunic Senior Member of Technical Staff TSP Initiative Telephone: +1 412-268-5863 Email: info@sei.cmu.edu

U.S. Mail Software Engineering Institute Customer Relations 4500 Fifth Avenue Pittsburgh, PA 15213-2612 USA

Web www.sei.cmu.edu www.sei.cmu.edu/contact.cfm

Customer Relations Email: info@sei.cmu.edu Telephone: +1 412-268-5800 SEI Phone: +1 412-268-5800 SEI Fax: +1 412-268-6257

This material is based upon work supported by the Department of Defense under Contract No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center.

Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Department of Defense.

NO WARRANTY

THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.

This material has been approved for public release and unlimited distribution except as restricted below.

Internal use:* Permission to reproduce this material and to prepare derivative works from this material for internal use is granted, provided the copyright and “No Warranty” statements are included with all reproductions and derivative works.

External use:* This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other external and/or commercial use. Requests for permission should be directed to the Software Engineering Institute at permission@sei.cmu.edu.

*These restrictions do not apply to U.S. government entities.

empirical study of software engineering results · empirical study of software engineering results...

Documents

guide to advanced empirical software engineering

selecting empirical methods for software engineering

systematic reviews in software engineering: an empirical...

empirical software engineering introduction & basic...

better empirical science for software engineering icse with...

course empirical software engineeringlaemmel/esecourse/...©...

paper presentation: ``preliminary guidelines for empirical...

empirical evidence in global software engineering: a...

382c empirical studies in software engineering lecture...

the role of empirical study in software...

fast feedback cycles in empirical software engineering...

guidelines for using empirical studies in software ... ·...

winter 2021 software engineering empirical research

empirical software engineering - department of computer...

theories in empirical software engineering

empirical software engineering at microsoft research ·...

annual report...» model driven software development...

empirical software engineering using ultra large...

an empirical analysis of c preprocessor use - software...

empirical software engineering