galorath.dan

NASA PM Challenge 2011What Program Managers Need to Know About Software Estimating and its Impact On Successful MissionsDan Galorath

[email protected]

Used with permission

mailto:[email protected]

© 2010 Galorath Incorporated

Key Points

Software is critical to systems and is getting more

complex

Software estimation & metrics can

provide program

managers significant information

Software estimating

processes are core to

reducing software risk


Software Is A Critical Component of Military & Aerospace Systems Failures Ariane 5

• Ariane 5 video

• Note: $500 Million payload

• Failure due to reused software from (Ariane 4)

• With incompatible assumptions for exception condition that was not required

• “the culture within the Ariane program…” only addressed “random hardware failures… which can quite rationally be handled by a backup system”

• “the view had been taken that software should be considered correct until it is shown to be at fault”!

ariane_5_explosion_video.mp4


Software Is Getting More Complex Source: (Impact of Weapon System Complexity on Systems Acquisition by Robert A. Dietrick, Major, USAF)

Why should we care: Software is already difficult, costly, and error prone. More complexity means more

problems


Software Is Providing an Increasing Percentage of Functionality

• Software development has been problematic for military aircraft developers– Consuming about 40 percent of the USAF’s RDT&E budget

– Average software schedule overrun was 222 percent of the original estimate

– Overruns often justified by stating weapon systems performance requirements are evermore demanding

• And thus cause greater reliance on software

• Software provided about 10 percent of an F-4’s functionality in the 1960s but provides over 80 percent of an F-22’s

Why should we care: “unrealistic cost estimates lead to unrealistic budgets and un-executable programs”

Meier Study… DAU


Software Failures Are Costly and Dangerous

• C130J: December 23, 2004, internal memo by Deputy Secretary of Defense called for the cancellation of the U.S. Air Force’s C-130J transport aircraft to cut its losses on the overpriced, unneeded, and problem-plagued C-130J. Estimated cancellation cost were $1.78 Billion – in 2004 dollars!

• Satellite: The loss of a classified satellite after only 7 seconds on orbit prompted the review of software and processors that has caused the most recent delay and a potential $1 billion overrun in Lockheed Martin's Space-Based Infrared System (SBIRS). Strategic Command is worried about potential gaps in coverage for some mission areas in part because satellites are being delivered later than planned.

• Patriot: Iraqi Scud missile slipped through Patriot missile defenses a year ago and hit U.S. Army barracks in Dhahran, Saudi Arabia, killing 28 servicemen.

• NASA Mars Climate Orbiter: Mathematical mismatch that was not caught until

after the $125-million spacecraft, a key part of NASA's Mars exploration program,

was sent crashing too low and too fast into the Martian atmosphere.


Software Is A Key Risk Item In Weapons Systems• GAO identified 42 programs at risk for cost & schedule

1. Military requirements changes

2. Software development challenges

3. Workforce issues

• Navy Mobile User Objective Satellite Communication System delays to the Joint Tactical Radio System, a set of software-defined radios causes advanced MUOS capabilities to be drastically underused… GAO

• National Institute of Standards and Technology (NIST)– Software defects cost nearly $60 Billion Annually

– 80% of development costs involve identifying and correcting defects

Why should we care: Software, not Hardware or technology readiness levels were called out


ESTIMATION & PLANNING: An Estimate Defined

• An estimate is the most knowledgeable statement you can make at a particular point in time regarding:– Effort / Cost

– Schedule

– Staffing

– Risk

– Reliability

• Estimates more precise with progress

• A WELL FORMED ESTIMATE IS A DISTRIBUTION

8


Estimation Methods 1 of 2

Model Category

Description Advantages Limitations

Guessing (Widely Used)

Off the cuff estimatesQuick Can obtain any answer desired

No Basis or substantiationNo ProcessUsually Wrong

Analogy (Widely Used)

Compare project with past similar projects.

Estimates are based on actual experience.

Truly similar projects must exist.

Expert Judgment (Widely Used)

Consult with one or more experts.

Little or no historical data is needed; good for new or unique projects.

Experts tend to be biased; knowledge level is sometimes questionable; may not be consistent.

Top Down Estimation (Widely Used)

A hierarchical decomposition of the system into progressively smaller components is used to estimate the size of a software component.

Provides an estimate linked to requirements and allows common libraries to size lower level components.

Need valid requirements. Difficult to track architecture; engineering bias may lead to underestimation.

9


Estimation Methods 2 of 2

Model Category Description Advantages Limitations

Design To Cost

Uses expert judgment to determine how much functionality can be provided for given budget.

Easy to get under stakeholder number

Little or no engineering basis.

Simple CER’s (Widely Used)

Equation with one or more unknowns that provides cost / schedule estimate

Some basis in data

Simple relationships may not tell the whole storyHistorical data may not tell the whole story

Comprehensive Parametric Models (Widely Used)

Perform overall estimate using design parameters and mathematical algorithms.

Models are usually fast and easy to use, and useful early in a program; they are also objective and repeatable.Easy tradeoffs can provide better plans

Models can be inaccurate if not properly calibrated and validated; historical data may not be relevant to new programs; optimism in parameters may lead to underestimation.

10

Why should we care: The methods of estimation provide insights into the likelihood of its viability


Initial Cost Estimation Problems (Software Program Managers Network)

• Many programs that have been evaluated tend to initially estimate using a very optimistic method.

– Low bid to win contract

– Naïve level of effort estimation

– Human optimism (HBR & Rich Hartley USAF Undersec Acquisition)

– Often wrong since they are not based on a thorough analysis of requirements. The formula

Cost = Size x Complexity/Productivity • Has three unknowns upfront: size, complexity, and productivity.

• Methods & estimating tools determine size given system requirements

• Organizational databases of productivity on comparable size and complexity projects can be used

– Or other bottom-up estimation techniques

“In any event, all initial software cost estimates should be considered as potential

high risk and should be reviewed at each program review” SPMN


13

10 Step Software Estimation ProcessConsistent Processes = Reliable Estimates1. Establish

Estimate Scope

2. Establish Technical Baseline, Ground

Rules, Assumptions

3. Collect Data

4. Estimate and Validate Software Size

5. Prepare Baseline

Estimates

7. Quantify Risks and Risk Analysis

6. Review, Verify and Validate Estimate

8. Generate a Project Plan

9. Document Estimate and Lessons

Learned

10. Track Project Throughout

Development


Balancing Resources & Schedule Is A Science

For a given Size, Complexity and Technology

Impo

ssib

le

Minimum TimeTo Complete

(Effort Increases

to Reduce Schedule)

Work ExpandsTo Fill Time

(Effort Increases

due to lack of pressure)

Inefficient

Minimum Time

Optimal Effort(Lower Effort

for Longer Schedule)

Calendar Time

Effo

rt M

onth

s

Effort Increasedue to Longer

Schedule

Why should we care: These impact both staffing (effort) schedule


Avoid “Death Marches” and Failed Projects By Applying “Brooks Law”

CostOverrun

UnaccomplishedWork

0

2

4

6

8

10

12

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46

Elapsed Calendar Time (months)

Sta

ff L

ev

el (

FT

E p

eo

ple

)

Effective Staffing Staffing Beyond Plan Overstaffed Understaffed

OptimalStaffing

LevelStaffing

ActualDelivery

PlannedDelivery

ScheduleSlip


Size is a Very important Software Metric• Software Size is the main driver of software development cost,

effort and schedule -- use the best available estimate of size range!– Knowing size allows analysts to determine effort (cost)

– Better size estimates = better cost estimates

0

5

10

15

20

25

30

35

40

20000 40000 60000 80000 100000

Size (SLOC)

Dev

elo

pm

ent

Sch

edu

le

Mo

nth

s

0100200300400500600700800900

1000

20000 40000 60000 80000 100000

Size (SLOC)

Dev

elo

pm

ent

Eff

ort

Mo

nth

s


Proper Definition is Important

Comments 23%

Blanks 25%

Debug Code 6%

SLOC 46%

Actual Project Example:Size Estimate at Start of Development = 1.6M Lines

Actual Size = 736,000 SLOC

46%

25%

23%

6%

Why should we care: Poor sizing is a common reason for poor software estimates


Sizing Pitfalls

Sizing Mistake Consequence

Wrong sizing metric chosen for level of detail desired

Large variance in estimates

Not enough time/effort spent on software sizing in general

Unbelievable estimates – results don’t match the program and are too optimistic or pessimistic

No clear definition of size Inconsistent estimates – results don’t pass the sanity check, unreliable output, blame the model

Size growth not considered OR size estimates reduced to achieve desired cost

Inaccurate estimates – results are too optimistic, programs will overrun cost / schedule estimates

Why should we care: Bad sizing yields bad estimates


Reuse: Watch Out For Low Cost Assumptions on “Heritage”

• Reuse or Heritage: applying existing software to a new mission (or additional innovation in its current mission)

• Effort to reuse software is routinely under estimated

Design

Test

ImplementationWhy should we care: Heritage is often underestimated

and causes major schedule / cost overruns


Software Design For Reuse (Lower Cost Heritage)

• Designed for reuse– Reusability requirements during original development– Significant extra coding and documentation effort

during original development to insure reusability– Minor to moderate work (mostly retest) required when

reused IF NOT MODIFIED

• Not designed for reuse– Developed without extra effort to ensure reusability– Sources include any legacy code not specifically

designed for reuse• Other projects & programs• Prior builds of the current program

– Moderate to major rework required• Redesign• Reimplementation• Retest

Reuse OFTEN optimistic… look atrisk


Rework Causes Software Project Issues• Rework: Doing the same work over again because it

was incorrect the first time– Prototyping is not rework

– Refactoring (tuning to make software better) is not rework

• Between 40% and 50% of the total effort on software projects is spent on rework.. Barry Boehm

• Initiatives to reduce rework can save significantly

Why should we care: Unplanned rework can drive cost up dramatically AND reducing rework can be a boon to

projects


Customization

COTS or GOTS Hoped For Reuse In Embedded Systems• Developmental

Software:– Functionality

developed specifically for the project at hand

– May include customization of COTS

• “Glue” Code:– Code written to bind

COTS to developmental software

– Development effort must be captured

• COTS Software:– Purchased functionality

• Direct Cost component of COTS integration

• COTS Cognition: – Required functionality within the COTS

software that must be understood• Effort component of COTS integration

DevelopmentalSoftware

COTS Software

“COTS Cognition”

Glue

Why should we care: COTS & GOTS promises are often not accurate meaning additional development cost & Schedule


Software Implemented Security and Safety Requirements Add Significant Cost & Schedule

Why should we care: Software implemented security and safety requirements can drive costs thru the roof


Understand Project Total Ownership Costs Up FrontMost Projects Spend Low During Maintenance

Staff Vs Maintenance Rigor

0500

100015002000250030003500

1 7 13 19 25 31 37 43 49 55 61 67 73 79 85

Time

sta

ff h

ou

rs p

er

mo

nth

develop

Rigor vhi+

Rigor nom

Rigor vlo

Why should we care: Maintenance is costly and usually not well considered in estimation & planning


Example Sophisticated Parametric Model Can Provide Significant Insight


Example Sophisticated Parametric Model Risk Analysis


Example Sophisticated Parametric Model Results 3


Example Benchmark From an Estimation Model.. Why Are We So Expensive?


0 4 8 12 16 20

Schedule ProbabilityExample Application 1

Probability

Time (calendar months)

1%10%20%30%40%50%60%70%80%90%99%

Understand Project Risks Include Them In Planning Decisions (Example SEER-SEM Outputs)

0 1800 3600 5400 7200 9000

Effort (person-hours)

1%10%20%30%40%50%60%70%80%90%99%

Effort ProbabilityExample Application 1

Probability

0 12 24 36 48 60

Defects ProbabilityExample Application 1Probability

Defects (count)

1%10%20%30%40%50%60%70%80%90%99%

30

Why should we care: Many programs plan at a 70 or 80% probability to reduce risk


Total Cost Growth for Two Space ProgramsDavid Graham, NASA

5 “The Success Triangle of Cost, Schedule, and Performance: A Blueprint for Development of Large-Scale Systems in an Increasingly Complex Environment” - (Booz|Allen|Hamilton, 2003)

Development Growth Causes

25%

11%

25%

9%

11%

11%

8%

RequirementsGeneration & Translation

Budget/Funding

Cost Estimation

Underestimation of Risk

Schedule Slips (Govt &Contractor)

Price Increases

Other

Quantitative Framework


GAO-093SP Cost Estimating and Assessment Guide of March 2009

32

Why should we care: Estimates have uncertainty. We need to know probability of estimate or our

project could fail


33

Software Has Additional Characteristics That Should Be Considered

Track defect discovery and removal rates

against expected rates

Increased defect

reporting rate shows a

worsening trend

Heath and Status Indicator shows status

and trends from the previous snapshot

Thresholds are user definable

Why should we care: EVM can be misleading in software… performance should be considered


• Maintenance typically 50% to 75% of the total software workload:

Dependent on maintenance rigor & operational “life expectancy”

• IEEE and others generally identify four categories of Software Maintenance efforts– Correct - maintenance is a change made in order to remove

a fault – 20%

– Adapt - maintenance is a change made in order to become suited to a different condition – 25%

– Perfect - maintenance is a change made in order to improve – 5%

– Enhance - maintenance is a change made to forestall or reverse a deterioration – 50%

– Plus Innovation which is generally block changes

Software Maintenance -Background

© 2010 Galorath Incorporated A1-35

Software Project Management Challenges• “No good deed will go unpunished.” Unreasonable

expectations on the next project are supported by heroic behavior on the previous project

• The most important business decisions about a software project are made at the time of minimum knowledge and maximum uncertainty.

• Adding and/or changing software means more time and/or more effort

• When a project is in trouble ask for more time rather than more people. Deferring functionality common approach to asking for more time

• Increasing concurrency is usually not a solution (e.g. designing several releases concurrently)


36

Summary

• Software projects are manageable

• Basis of management is a viable estimate

• The 10 step process provides a repeatable approach to estimation process

• You can help ensure useful results by adopting a process that is standardized and repeatable

• Measurement is a key part of the estimation process

• Beware of using simple measurements to drive estimates

galorath.dan

Technology

software failures

software component

set of software

reused software

review of software

galorath incorporated

ariane program

unrealistic cost estimates