successful projects on schedule and on cost: lessons - scaf

More Successful Projects On Schedule and On Cost:Lessons Learned from Three Decades and Best Practices

For The Next DecadeSCAF Conference September 2011

Dan Galorath

[email protected]

Copyright 2011 Galorath Incorporated

mailto:[email protected]�

Human Nature Optimism in Estimates

HBR Article explains this Phenomenon:

• Problem: Humans seem hardwired to be optimists

• We routinely exaggerate benefits and discount costs

Delusions of Success: How Optimism Undermines Executives' Decisions (Source: HBR Articles | Dan Lovallo, Daniel Kahneman | Jul 01, 2003)

2

Solution: Temper with “outside view”Viable estimation is tempering

Past Measurement Results traditional forecasting w/ statistical Parametrics Can help

Don’t remove optimism, but balance optimism & realism

http://hbr.org/search/Dan Lovallo/�

http://hbr.org/search/Dan Lovallo/�

http://hbr.org/search/Daniel Kahneman/�

Example of Tempering: (Source How To Measure Anything)

• German Mark V Tanks

• Allies estimated production by analyzing serial numbers

• Used the captured tanks as a random sample and predicted probability of various production levels

An Estimate Defined

• An estimate is the most knowledgeable statement you can make at a particular point in time regarding:

• Effort / Cost

• Schedule

• Staffing

• Risk

• Reliability

• Estimates more precise with progress

• A WELL FORMED ESTIMATE IS A DISTRIBUTION

© 2011 Copyright Galorath Incorporated 4

Viable Estimation Is Critical

• Estimating is critical for all kinds of systems

• Yet many treat is as a second rate process

• Everyone estimates…. Just most get it wrong and don’t have a process

• Having a repeatable estimation process is critical to both estimating AND to successful projects

• Estimation and measurement go hand in hand

Poor Estimates Effect on Projects• Inaccurate estimates can reduce project success:

• Poor implementations

• Critical processes don’t scale

• Emergency staffing

• Cost overruns caused by underestimating project needs

• Scope creep• Forever changing project goals

• Frustration

• Customer dissatisfaction

• Cost overruns and missed schedules

• Project Failures

• Important project business decisions made early with minimum knowledge & maximum uncertainty

6

Why should we care: Poor estimates and plans are a root cause of program failure

Reducing Costs & Spending

© 2011 Copyright Galorath Incorporated

USD (AT&L) Efficiency 5 Initiatives “Carter Memo” https://acc.dau.mil/bbp


1. Target Affordability

2. Incentivize Productivity & Innovation in Industry

3. Promote Real Competition

4. Improve Tradecraft in Services Acquisition

5. Reduce Non-Productive Processes & Bureaucracy

Director, Acquisition Resources & Analysis will establish tracking system to monitor progress

and compliance

Proposed Support on Carter Initiatives: Better Buying (Examples)• “Should cost” vs “will cost”

• Manage affordability as a requirement

• Milestone a affordability target

• Cost KPP Key Performance Parameter: Failure to meet it causes reevaluation or assessment of program

• Incentivize productivity & innovation in industry:

• Drive productivity growth through Will Cost/Should Cost management

• Increase the use of FPIF contract type where appropriate using a 50/50 share line and 120 percent ceiling as a point of departure

• Eliminate redundancy in warfighter portfolios

• Require non-certified cost and pricing data on single offers

Impact On The Cost CommunityNot just the Carter Memo

• Weapon Systems Acquisition Reform

• DoD Budget Reductions

• Economic Changes/Uncertainty

Change is being driven by continued and“sometimes surprising growth” in cost and schedule.

We know we must:

• Rise above “business as usual”

• Ensure estimates and estimating process are realistic, reliable, and meaningful for today's decision makers

Dealing With Data


© 2011 Galorath Incorporated

Data Doesn’t Have To Be Perfect To Be Useful: But Is Has To Be Viable

• 80 Calories per serving

• 2.5 Servings per can

• 4 Ounces, Condensed, 8 Ounces With Water


You have an estimate …Now what?

13


Use Historical Measurement to evaluate your estimate!

It’s easy to dig deeper and deeper to justify an estimate! 14


• Correlation does not imply causation– Just because two data points may sit side by side

doesn’t mean they are the same or will have the same outcome

• Casual analysis is a recognized error in medicine

Perhaps ???

The Error of Causal AnalysisCreating a False Association

Tumor Can Cause Headache

Headache doesn’t mean a tumor

15


16

Normalization: Adjustments for homogeneity

• Data comes in all “sizes and shapes”

• Development of models and calibration of models require that format and scale of data used be internally consistent – Inches versus centimeters – Full dollars versus “K” dollars versus Euros– Labor phases and activities– Handling of overhead– … and the list goes on

Estimation Process

17


18

Basic Cost Estimating Process (Source CEBOK)

• Work Breakdown Structure (WBS) Development

• Program/System BaselineDevelopment

WBS

Baseline

Data Collection

Data Analysis

Methodology

Validation

Reports

© 2011 Galorath Incorporated19

US GAO process for Credible Estimates


20

10 Step System Estimation Process 2011

1. Establish Estimate Scope

2. Establish Technical Baseline, Ground

Rules, Assumptions

4. Refine Technical Baseline Into

Estimable Components

4. Collect data / estimation inputs

5. Estimate Baseline Cost, Schedule, Affordability Value

6. Validate Business Case Costs &

Benefits (go / no go)

6. Quantify Risks and Risk Analysis

8. Generate a Project Plan

9. Document Estimates and Lessons

Learned

10. Track Project Throughout

Development


Estimation Organizational Maturity V1.7

Level 0

Informal or no estimating

Manual effort estimating without a process

Level 1

Direct Task Estimation Spreadsheets Ad Hoc

Process

Level 2

Formal Sizing (e.g.

function points)

Direct Task

Estimation

Simple model (Size *

Productivity) or informal SEER Use

Some measureme

nt & analysis

Informal Process

Level 3

FormalSizing

Robust Parametric estimation

(SEER)

Estimate vs. actual capture

Formalized Multiple Estimate Process

Rigorous measurement

& analysis

Parametric planning &

Control

Risk Management

Repeatable process

Level 4

Formal sizing Repeatable process

Robust parametric estimating

(SEER)


& analysis

Parametric estimation

with tracking & control

Risk Management

Process improvement via lessons

learned

Level 5

Formal sizing Repeatable process

Robust parametric estimating

(SEER)


& analysis

Parametric estimation

with tracking & control

Risk Management

Continuous process

improvement

Why should we care? Maturity is related to estimate viability… With better estimation process, projects more likely to be successful in execution


Estimating Core Governance Component - A standard Corporate Governance Model (Source: K. Aguanno)

© 2011 Copyright Galorath Incorporated22

GreatIdea

Opportunity Analysis

Preliminary Business

Case

Committed Business

Case

AchieveBusiness

Case

Concept

- DescribeIdea &Possiblebenefits

MarketingAnalysis

FeasibilityStudy

- Pilot orproof ofconcept

-Determinecustomer acceptance

-Interviewsfocus groups,etc.

- Design solution- Estimate cost / schedule- Analyze risk- Determine feasibility / ROI

- Validate & commit to design & approach

-Revised estimates & schedule

-Risk reduction

-Baselined plan

-Build solution

- Deploy

-Achieve -business case-Capture lessonslearned -incl: estimating

Full Executionor Deployment

Gate 1 Gate 2 Gate 3 Gate 4


Estimation Should Be Key Part of Process (Source Q Redman, APMP Just Say No)

23

Sco

pe &

Acc

ura

cyU

s e1 2 3 4Phase 0-1

ROM

ROM

ROM

ROM

AcquisitionPlanning/ POMand Plus Ups

Market Assessment/

“What If’s”

OpportunityCreation/ Customer

Decision PlansProcurement

Initiation Draft RFP RFP

DDE

Modified Budgetary Estimate

Draft RFP/Gate 36-8 people, 3

weeks(Bid Stds + History

)

Formal BidGate 4

15-20 people4 weeks

(Bid Stds+ History)

EARLY ESTIMATING3-5 people, 3 - 5 days

Top Down, parametric model based price estimating

Vs.Current state: 90 people, 6wks


Ask These Questions When Identifying Estimate Scope• Challenged projects

– Would you still go forward if you knew• Schedule would be significantly longer?

• Cost would be dramatically higher?

– Probably: but perhaps more insight could identify mitigation

• Plan functionality differently

• Certainly you could notify stakeholders of real costs

• Ensure staffing is appropriate for the constraints

• Failed Projects– Would you start a project you knew was unaffordable?

Or if schedule was completely unrealistic?

– If knowing up-front could you do something about it?

– Often better to kill project before it begins than waste resources & let the organization down

24


Lesson Learned: Estimate Must Quantify Risk & Uncertainty

2525

What is likely to happen

Feel lucky?

Firm Fixed Price?

Understand the risk before you commit!

Lesson Learned Estimate Total Ownership Costs – IT Example

• Software Development

• Software Maintenance

• IT Infrastructure

• IT Services


Estimation Doesn’t Always Meet Management Needs -The Gap Between IT & Management Needs (Source: Fraunhofer)


Financial

Customer Business Process

Innovation / Growth

Vision & Strategy

Goals

Data Collection

MeasurementMetricsAnswersQuestions

Goal Attainment

Software Development & Maintenance

6σ

GQM

PSM

CoBIT…

Business/OrganizationLevel

ITGovernance

ITGovernance

CustomerFocus

An ROI Analysis of A New System: ESTIMATION SHOULD BE INVOLVED


• Can we do better?• Will stakeholders tolerate a

loss for 3 years?• What is the risk?

Estimation Best Practices


Unit I - Module 1

30

Cost Estimate Qualities (Adapted from CEBOK)

• The characteristics of high quality cost estimates are:

• Accurate (Viable Within Range)

• Comprehensive

• Replicable and Auditable

• Traceable

• Credible

• Timely

31

System Description (Parametrics Can Estimate More, Earlier) Adapted from CEBOK

“If you can’t tell me what it is, I can’t tell you what it costs.”

-Mike Jeffers

“If you can tell me the range of what it might be I can tell you the

range of cost, schedule & probability”

-Dan Galorath

Unit I - Module 1

32

Types of Cost Estimates (Adapted From CEBOK)

• Life Cycle Cost Estimate (LCCE)

• Independent Cost Estimate (ICE)

• Budget Estimate

• Rough Order of Magnitude (ROM)

• Estimate At Completion (EAC)

• Independent Cost Assessment (ICA)

• Analysis of Alternatives (AoA)

• Economic Analysis (EA)

1

33

Aircraft Example: Estimating Techniques Adapted from CEBOK)

• Where applicable, use primary methods

• Analogy• Model X100 series jet engines have only been used on one other

plane, but weight is 100% higher on this model; estimate 2x other model

• Simple Parametric• As shown, need to estimate 2 lbbrake rotors, should be roughly $4Mfrom regression curve

• Commercial Parametric

• Key characteristics range are

• 30-50k lines of code and 600-800

• kilos engine & 7 PC boards

• Engineering Build-Up• Know Air Conditioning (AC) system costs on plane because received

quotes from HVAC vendor for all duct work and AC blower off the shelf

Parametric Estimate

0

2

4

6

8

10

12

0 2 4 6 8 10

Parameter (ie. weight)

Cos

t

Manual Estimates: Human Reasons For Error (Metrics Can Help)

• Manual Task estimates yield SIGNIFICANT error

• Desire for “credibility” motivates overestimate behavior (80% probability?)• So must spend all the time to be “reliable”

• Better approach: force 50% probability & have “buffer” for overruns

• Technical pride sometimes causes underestimates

34

35

Lesson Learned: The Risk In Risk Analysis

"It's tough to make predictions, especially about the future." -- Yogi Berra.


• Decide Why You Want An Estimate

• Map Estimation Goals To Estimate Process Maturity & Develop Plan To Achieve The Maturity

• Have A Documented, Repeatable Estimation Process

• Make The Estimating Process As Simple As Possible; But No Simpler

• Be Proactive: The Process Is Important, The Tools Go Along With The Process

• Get Buy-in From Program Managers

• Hold People Accountable: Center Of Excellence Can Prepare Estimate But Program Managers Must Own Them

• Tie The Estimate To The Plan37

Estimation Best Practices 2

• Evaluate Total Ownership Cost; Not Just Development

• Estimate A Range And Pick A Point For The Plan

• Re-estimate The Program When It Changes

• Avoid Death Marches: Programs With Unachievable Schedules Are Likely To Fail And Drain Morale

• Keep A History: Start An Enterprise Database NOW…

• Business Case: Evaluate ROI In Addition To Costs

• Convert Expert Spreadsheets Into A Common Language

38


• Track Progress Vs. Estimate Throughout The Life Cycle

• Estimate Schedule As Well As Effort (Cost) For Complete Picture

• Tie The Business Case Into The Estimating Process

• Attack Non-productive Rework As Part Of The Process

39


• Have clear definitions:

• What does “complete” mean

• What activities are included and excluded (E.g. development only or total ownership; help desk included or excluded, etc.)

• Which labor categories are included and excluded in the estimate (e.g. are managers included? Help desk? Etc.)

• Don’t ignore IT infrastructure and IT services costs

• Tracking defect sources can go along with the process

40

The Vision of Parametrics Over the Next 20 Years (Originally Published in 2004)• Parametrics integrated into engineering processes and

engineering decision making• E.G. Cost derived from simulation models of that system

• Parametrics used throughout Industry & Government

• Parametrics will lose its “magic” reputation• Improved processes will yield better data

• Augmentation of parametrics with more viewable data will increase believability among engineers and management

• The nay-sayers who say that can make parametric models say anything they want will be replaced with belief

• More dynamic parametrics based on both historical and real time data

• Parametric models will be available to use as “objects” in financial and engineering analysis

41

Actions

• Have a Repeatable estimation process

• Ensure data are understood and normalized

• Bring ROI under the estimation umbrella

• Quantify affordability requirements and use estimating to plan and execute them


successful projects on schedule and on cost: lessons - scaf

Documents