unclassified schopenhauer's proof for software: pessimistic bias in the nostromo tool (u) dan...

UNCLASSIFIED

UNCLASSIFIED

Schopenhauer's Proof For Software: Pessimistic Bias In the NOSTROMO Tool (U)

Dan StricklandDynetics Program Software Support

[email protected]

Ashley MathisTHAAD Project Office System Software Engineering

[email protected]

UNCLASSIFIED

2UNCLASSIFIED

Overview (U)

• Background

• Pessimism in Code Size

• NOSTROMO Tool

• NOSTROMO Demo

UNCLASSIFIED

3UNCLASSIFIED

UNCLASSIFIED

4UNCLASSIFIED

Background (U)

• In 2003, TPO and Dynetics presented a methodology on using Monte Carlo simulation and uncertainty in COCOMO II to estimate risk in estimates

• In 2004, TPO and Dynetics presented updated methodologies to include Auto-Generated Code and other updates to NOSTROMO

• A new methodology for estimating pessimistic size growth has been incorporated into NOSTROMO

UNCLASSIFIED

5UNCLASSIFIED

Boehm Horn Curve (U)

[Ref (1) - SOFTWARE COST ESTIMATION WITH COCOMO II (P 10) - BOEHM 2000]

• Size (and effort) should converge over time relative to development phase

• During Concept of Operation, estimates can be 50% to 200% of the actual final size

• Some types of software are always on the “bottom” part of the curve – always growing in size

UNCLASSIFIED

6UNCLASSIFIED

Code Size Growth (U)

• Code size estimates (and subsequent cost/schedule estimates) are required long before software requirements are baselined and finalized

• Code estimates grow to any number of factors:

• Requirements Volatility

• Auto-Generated Code

• Poor Estimating Techniques

• Many large, complex programs exhibit constant code size growth outside the boundaries of the “horn curve”

• We define pessimism as extending the boundaries of the expected code size growth to historical limits with only growth potential – “things can only get worse”

UNCLASSIFIED

7UNCLASSIFIED

Three Historical Projects – Size Growth (U)

• Three projects that have completed development or are very near completion

• Graphs represent code size estimates over time for new code and subsequent Effective SLOC

• Graphs show growth realization in new and ESLOC growth over time

New SLOC Growth

0

100

200

300

400

500

600

700

800

Sep-0

2

Dec-0

2

Mar

-03

Jun-

03

Sep-0

3

Dec-0

3

Mar

-04

Jun-

04

Sep-0

4

Dec-0

4

Mar

-05

Jun-

05

Sep-0

5

Dec-0

5

KS

LO

C Project 1

Project 2

Project 3

ESLOC Growth

0

100

200

300

400

500

600

700

800

KE

SL

OC Project 1

Project 2

Project 3

UNCLASSIFIED

8UNCLASSIFIED

Three Historical Projects – Percentage Size Growth (U)

• Projects new code size grew 90-180% from original estimates

• Projects ESLOC size grew 40-120% from original estimates

• Programs exhibited extensive growth after Critical Design Review (CDR) – during Coding and Unit Testing

• Percentage growth is outside of the boundaries expected by the horn curve

Percentage New SLOC Growth

0.0%20.0%40.0%60.0%80.0%

100.0%120.0%140.0%160.0%180.0%200.0%

Sep-02

Dec-02

Mar

-03

Jun-

03

Sep-03

Dec-03

Mar

-04

Jun-

04

Sep-04

Dec-04

Mar

-05

Jun-

05

Sep-05

Dec-05

Project 1

Project 2

Project 3

Percentage ESLOC Growth

0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

120.0%

140.0%

Project 1

Project 2

Project 3

UNCLASSIFIED

9UNCLASSIFIED

Pessimism Curve (U)

• Plot the data points of the three projects in the same manner as the relative size of the horn curve

• Assume that the “ceiling” for Pessimism is the final size x and the “floor” encompasses the most extreme data points

• Where data is unavailable, use the horn curve values and some extrapolation if necessary

SLOC Distrbution

0.2x

PDR CDR TRR FQT

0.4x

0.6x

0.8x

x

Horn PessimismFeasibility 0.25 0.25Plans and Reqs 0.50 0.31Prod Design 0.67 0.36Detail Design 0.80 0.47Develop and Test 0.87 0.78Final Qual Test 1.00 1.00

***

* Value taken from horn curve – no like value in data

** Value extrapolated – average of previous and subsequent phases

UNCLASSIFIED

10UNCLASSIFIED

Smoothed Pessimism Curve (U)

• Mapping the points of the pessimism curve against the horn curve show an exponential curve of greater growth potential

• Using MS_Excel, we added an exponential trendline to smooth the pessimism curve and identify a formula for use (R-squared value = 0.9666)

• Using the smoothed curve, we can calculate the ranges for our pessimism curve as tied to development phase

• We used this new range (inverted) as boundary conditions for size in NOSTROMO

• NOSTROMO uses a triangle distribution between high and low boundaries for size

Relative Size Range - Horn and Pessimistic

y = 0.1707e0.2861x

R2 = 0.96660.00

0.20

0.40

0.60

0.80

1.00

1.20

Feasib

ility

Plans a

nd R

eqs

Prod

Design

Detail

Des

ign

Devel

op a

nd T

est

Final Q

ual T

est

Horn Curve

PrelimPessimisticCurvePessimisticCurve

Horn Pessimism SmoothFeasibility 0.25 0.25 0.23Plans and Reqs 0.50 0.31 0.30Prod Design 0.67 0.36 0.40Detail Design 0.80 0.47 0.54Develop and Test 0.87 0.78 0.71Final Qual Test 1.00 1.00 0.95

UNCLASSIFIED

11UNCLASSIFIED

Original NOSTROMO Concept (U)

• NOSTROMO first takes normal COCOMO II inputs• NOSTROMO accounts for uncertainty in the settings• NOSTROMO uses Monte Carlo to simulate hundreds of COCOMO II estimates using the inputs and uncertainties• NEWT – NOSTROMO Entry Writing Tool – a Delphi polling tool that captures Uncertainty

NEWT

DATAPAGE

REPORTS

NOSTROMO

Notional Obscurity STatistical Risk Observation MOdel

UNCLASSIFIED

12UNCLASSIFIED

New NOSTROMO Concept (U)

NEWT

DATAPAGE

REPORTSNOSTROMO

DEFAULT UNCERTAINTY

COMBINATION

AGC CALCULATOR

NOSTROMO 0.3.15 offers:• multiple methods for inputting Uncertainty• combination of subcomponents• more reporting capability• AGC methodology• Pessimistic SLOC estimation

PESSIMISM

UNCLASSIFIED

13UNCLASSIFIED

Distributions of Uncertainty (U)

Setting n-1 Setting n Setting n+1

Level 1 - Certain

Level 2 - Low Uncertainty - Normal

Level 3 - Medium Uncertainty - Triangle

Level 4 - High Uncertainty - Uniform

NOSTROMO assumes ceiling and floor limits of the highest and lowest default values for each COCOMO II Scale Factor and Cost Driver

NOSTROMO changes the distribution and sets the high and low points based on the level of uncertainty with each factor

UNCLASSIFIED

14UNCLASSIFIED

NOSTROMO Application (U)

• Microsoft Excel 2002 worksheet with attached macros and Visual Basic code

• Currently on version 0.3.15 (as of 10/19/06)

• Does not use outside applications for Monte Carlo simulation (standard random number generation from Visual basic)

• Limited testing performed on a proof-of-concept application

• Suggest limiting Monte Carlo run size to 500-2000 runs

• Generates two pages of output – Data and Charts

• Data page contains all outputs from Monte Carlo simulation, charting data, and histograms

• Charts page contains Confidence Intervals for output of COCOMO II and Putnam models, charts, and histograms

• NOSTROMO – Data Sheet and CM Control Sheet are Developer pages and should not be removed

UNCLASSIFIED

15UNCLASSIFIED

SI Name

Input Rating Uncertainty UncValPREC 2 2

FLEX 2 2

RESL 2 4

TEAM 2 3

PMAT 2 3

NEW CODE 1000 2 2 FALSE

REVL 2 2

ADAPTED CODE 2

% DESIGN MOD 2 2

% CODE MOD 2 2

% TEST MOD 2 2

Assessment and Adaptation 2 2

Software Understanding 2 2

Unfamiliarity with Software 2 2

TIME 2 2

STOR 2 3

PVOL 2 2

RELY 2 2

DATA 2 4

CPLX 2 2

RUSE 2 1

DOCU 2 3

ACAP 2 4

APEX 2 1

PCAP 2 4

PLEX 2 1

LTEX 2 1

PCON 2 1

TOOL 2 3

SITE 2 2

SCED 2 3

Number of Iterations 500

Putnam Technology Rating 2

Run SI

Clear Page

Low Uncertainty

Low Uncertainty

High Uncertainty

Medium Uncertainty

Medium Uncertainty

Plans and Requirements

Low Uncertainty

Low Uncertainty

Low Uncertainty

Low Uncertainty

Low Uncertainty

Low Uncertainty

Low Uncertainty

Low Uncertainty

Medium Uncertainty

Low Uncertainty

Low Uncertainty

High Uncertainty

Low Uncertainty

Certain

Medium Uncertainty

High Uncertainty

Certain

High Uncertainty

Certain

Certain

Certain

Medium Uncertainty

Low Uncertainty

Medium Uncertainty

Low Uncertainty

Combine Project

AGC Calculator

Default Uncertainty

Delete Project

NEWT

PESSIMISM

Change Default Pessimism

NOSTROMO Input Sheet – Notional Data (U)

Software Item Name

Scale Factors

Code Size

Cost Drivers

Iterations

Putnam Productivity

Function Buttons

Conditional Formatting

Pessimism Toggle

Pessimism toggle automatically sets the Requirements Evolution and Volatility to 0% to avoid double counting

UNCLASSIFIED

16UNCLASSIFIED

NOSTROMO Pessimism Worksheet (U)

PessimismPhase Relative SizeFeasibility 0.23Plans and Requirements 0.3Product Design 0.4Detailed Design 0.54Development and Test 0.71Final Qualification Test 0.95

Instructions: Change the relative sizes in the format of the Boehm horn curve, where each value is equal to the relative estimate size to the final size.

Inverted, this value will serve as the top range multiplier for new size. Select "Return To Main" when finished.

Return to Main

Allows the user to develop their own relative size numbers for Pessimism and use in NOSTROMO

UNCLASSIFIED

17UNCLASSIFIED

NOSTROMO Outputs – 5000 SLOC (U)

Percentile Effort Schedule Size Productivity EAF99% 35.48 11.44 7394 208.40 1.5795% 31.27 10.98 6955 222.45 1.4190% 29.23 10.75 6732 230.32 1.3375% 26.11 10.36 6374 244.10 1.2170% 25.40 10.27 6289 247.63 1.1860% 24.15 10.11 6138 254.14 1.1450% 23.04 9.95 6000 260.38 1.0940% 21.98 9.80 5865 266.78 1.0525% 20.33 9.56 5647 277.75 0.9820% 19.71 9.47 5563 282.23 0.9610% 18.16 9.22 5347 294.37 0.895% 16.98 9.02 5176 304.79 0.841% 14.96 8.66 4868 325.34 0.76

Default 20.61 9.60 5875 285.09 1.00


Default 25.50 10.28 7130 279.64 1.00

5000 SLOC – Nominal REVL – No Pessimism 5000 SLOC – Pessimism On

Pessimism accounts for an 18% increase in Size in the 50th Percentile NOSTROMO Run

UNCLASSIFIED

18UNCLASSIFIED

NOSTROMO Outputs – 500000 SLOC (U)

500 KSLOC – Nominal REVL – No Pessimism 500 KSLOC – Pessimism On

Pessimism accounts for a 20% increase in Size in the 50th Percentile NOSTROMO Run


Default 3261.56 48.05 587500 180.13 1.00


Default 4035.20 51.42 712963 176.69 1.00

UNCLASSIFIED

19UNCLASSIFIED

Demonstration

UNCLASSIFIED

20UNCLASSIFIED

Future NOSTROMO Additions and Efforts (U)

• Addition of Exception Conditions from Ray Madachy’s Expert COCOMO

• Transition to an application environment (MS-Access or .NET)

• Expansion of the NOSTROMO tool to address other COCOMO Family models with uncertainties COSYSMO – especially relevant in development of complex DoD systems

COSOSIMO – System of Systems models are becoming increasingly prevalent in DoD efforts

• Expansion of the NOSTROMO methodology and tool into Readiness Level models Software Readiness Levels (SWRL)

Technology Program Management Model (TPMM)

UNCLASSIFIED

21UNCLASSIFIED

Conclusion (U)

Analysis of size growth potential gives valuable insight into the true risks of software development

• Software size in many development environments has a tendency to grow, even late into the project development

• Software estimates need to address any pessimistic growth potentials and software estimators have an obligation to report the risks associated with developments that continue to grow in size

• New NOSTROMO methodology incorporates pessimistic sizing data to predict the worst-case scenario

unclassified schopenhauer's proof for software: pessimistic bias in the nostromo tool (u) dan...

Documents

size slide

horn curve slide

expected code size growth

constant code size growth

pessimistic size growth

relative size

smoothed pessimism curve

smoothed curve