presentation on 'why cant people estimate' event, 23rd june 2016

Why Can’t People Estimate: Estimation Bias and Mitigation With SEER Introduction

David Simms

Galorath International

© 2016 Copyright Galorath Incorporated

Key Points for today . . .

Estimates can be better, reducing bias & strategic mis-estimation… Parametrics can

help.

© 2016 Copyright Galorath Incorporated 2

Tempering with an

“outside view” can mitigate some bias

Without care estimates are usually biased (even with experts)

Some of the most important

business decisions about a

project are made at the time of

minimum knowledge and

maximum uncertainty.

ESTIMATION & PLANNING: An Complete Estimate Defined

• An estimate is the most knowledgeable statement you can make at a particular point in time regarding:

• Effort / Cost

• Schedule

• Staffing

• Risk

• Reliability

• Estimates more precise with progress

• A WELL FORMED ESTIMATE IS A

DISTRIBUTION

4

Human Nature: Humans Are Optimists

Harvard Business Review explains this Nobel Prize Winning Phenomenon:

• Humans seem hardwired to be optimists

• Routinely exaggerate benefits and discount costs

Delusions of Success: How Optimism Undermines Executives' Decisions (Source: HBR Articles | Dan Lovallo, Daniel Kahneman | Jul 01, 2003)

5

Solution - Temper with “outside view”:

Past Measurement Results, traditional forecasting, risk

analysis and statistical parametrics can help

Don’t remove optimism, but balance optimism and

realism

http://hbr.org/search/Dan Lovallo/

http://hbr.org/search/Dan Lovallo/

http://hbr.org/search/Daniel Kahneman/

Why Estimate?

Estimation Methods 1 of 2

Model

Category Description Advantages Limitations

Guessing Off the cuff estimates

Quick

Can obtain any answer

desired

No Basis or substantiation

No Process

Usually Wrong

Analogy Compare project with past

similar projects.

Estimates are based on

actual experience. Truly similar projects must exist

Expert

Judgment

Consult with one or more

experts.

Little or no historical data

is needed; good for new or

unique projects.

Experts tend to be biased;

knowledge level is sometimes

questionable; may not be

consistent.

Top Down

Estimation

A hierarchical decomposition

of the system into

progressively smaller

components is used to

estimate the size of a

software component.

Provides an estimate

linked to requirements and

allows common libraries to

size lower level

components.

Need valid requirements.

Difficult to track architecture;

engineering bias may lead to

underestimation.

Estimation Methods 2 of 2

Model Category Description Advantages Limitations

Bottoms Up

Estimation

Divide the problem into

the lowest items.

Estimate each item…

sum the parts.

Complete WBS

can be verified.

The whole is generally bigger than the

sum of the parts.

Costs occur in items that are not

considered in the WBS.

Design To Cost

Uses expert judgment to

determine how much

functionality can be

provided for given

budget.

Easy to get under

stakeholder

number.

Little or no engineering basis.

Simple CER’s

Equation with one or

more unknowns that

provides cost / schedule

estimate.

Some basis in

data.

Simple relationships may not tell the

whole story.

Historical data may not tell the whole

story.

Comprehensive

Parametric Models

Perform overall estimate

using design

parameters and

mathematical

algorithms.

Models are usually

fast and easy to

use, and useful

early in a program;

they are also

objective and

repeatable.

Models can be inaccurate if not

properly calibrated and validated;

historical data may not be relevant to

new programs; optimism in parameters

may lead to underestimation.

8

Improving Estimation Is Important

• Estimation is critical for defining success for all types of projects

• Many organizations treat estimation as a second rate activity and do not invest in the process

• Everyone estimates….but the process and the quality or success rate varies greatly

• Estimation and measurement go hand in hand – BOTH contribute to improvement

• Having a repeatable estimation process is critical to both estimating AND to successful projects

Cognitive Bias: How Fair Are We (Source BeingHuman.org)

•Human beings exhibit inherent errors in thinking

•Perception has more to do with our desires—with how we want to view ourselves—than with reality." Behavioral economist Dan Ariely

Cognitive bias: Tendency to make

systematic decisions based on PERCEPTIONS

rather than evidence

•Our brains using shortcuts (heuristics) that sometimes provide irrational conclusions Researchers theorize in

the past, biases helped survival

•from deciding how to handle our money

•to relating to other people

•to how we form memories “Bias affects everything:


The Planning Fallacy (Kahneman & Tversky, 1979)

•Manifesting bias rather than confusion

•Judgment errors made by experts and laypeople alike

•Errors continue when estimators aware of their nature

Judgment errors are systematic & predictable,

not random

•Underestimate costs, schedule, risks

•Overestimate benefits of the same actions Optimistic due to overconfidence

ignoring uncertainty

•“inside view” focusing on components rather than outcomes of similar completed actions

•FACT: Typically past more similar assumed

•even ventures may appear entirely different

Root cause: Each new venture

viewed as unique


Reference Class Forecasting

Attempt to force the outside view and

eliminate optimism and

misrepresentation

Choose relevant “reference class”

completed analogous projects

Compute probability distribution

Compare range of new projects to

completed projects

Provide an “outside view” focus on

outcomes of analogous projects


Best predictor of performance is actual performance of implemented comparable projects (Nobel Prize

Economics 2002

Predicts outcome of planned action based on actual outcomes in a reference class: similar actions to that being forecast.

Dealing With the “Problem of Assumptions” • Assumptions are essential Write them down but…

Wrong assumptions can drive an estimate to uselessness

• Use an assumption verification process


1. Identify assumptions

2. Rank order assumptions based on

estimate impact

3. Identify high ranking assumptions

that are risky

4. Clarify high ranking, high risk assumptions

& quantify what happens if those

assumptions change

5. Adjust range of SEER inputs to describe the

uncertainty in assumption

We should always listen to the experts . . . . Shouldn’t we?


Trouble Starts By Bias or Strategic Mis- Estimation Ignoring Iron Triangle

• Typical Trouble: Mandated features needed within specific time by given resources

• At least one must vary otherwise quality suffers and system may enter impossible zone!

Quality Resources Schedule

Scope (features, functionality)

Sometimes strategic mis-estimation is used to get projects started or to win

Some customers think price to win is strategic mis-estimation (it is not)

Plans Based on Assumptions About Average Conditions Usually Go Wrong

• Executive desire to “Show me the number” forces averages

• When average is used to represent an uncertain quantity….

• it ends up distorting the results

• Because it ignores the impact of the inevitable variations

• Flaw may be fixed by tools such as Monte Carlo Analysis

• Adds a distribution & probability to “the number”

16

Adding Reality to Estimates – Example – 2 (Source SEI) Step Best Expected Worst

1 27 30 75

2 45 50 125

3 72 80 200

4 45 50 125

5 81 90 225

6 23 25 63

7 32 35 88

8 41 45 113

9 63 70 175

10 23 25 63

500

What would you forecast the schedule duration to be

now?

Monte Carlo Analysis To Quantify and Mitigate Bias

90% confidence, project under 817

days

Almost guaranteed to miss the 500

days

50% confidence,

project will be under 731 days

Example: Project Cost Alone Is not The Cost of IT Failure (Source: HBR)

• Case Study: Levi Strauss

• $5M ERP deployment contracted

• Risks seemed small

• Difficulty interfacing with customer’s systems

• Had to shut down production

• Unable to fill orders for 3 weeks

• $192.5M charge against earnings on a $5M IT project failure

“IT projects touch so many aspects of organization they pose a new singular risk”

http://hbr.org/2016/09/why-your-it-project-may-be-riskier-than-you-think/ar/1

Causes of Project Failure Source: POST Report on UK Government IT Projects

Lack of a clear link between the project and the organisation’s key

strategic priorities, including agreed measures of success.

1.

Lack of clear senior management and ministerial ownership and leadership 2.

Lack of effective engagement with Stakeholders 3.

Lack of skills and proven approach to project management and risk

management

4.

Lack of understanding of and contact with the supply industry at senior levels

within the organisation

5.

Evaluation of proposals driven by initial price rather than long-term

value for money (especially securing the delivery of business benefits)

6.

Too little attention to breaking development and implementation into

manageable steps

7.

Inadequate resources and skill to deliver the total delivery portfolio 8.

Source: POST Report on UK

Government IT Projects

Fallacy of Silent Evidence What about what we don’t know?

How confident would you feel if the Silent Evidence was visible?

What is Parametric Estimating?

• Parametric Approach:

• Employs cost estimating relationships

• Uses mathematical equations

• Allows estimation based on measurement of key parameters that influence the relationship

• Based on technical or physical characteristics of the product and production environment

• SEER’s Parameters Cover 5 Key Areas

• People

• Process

• Technology

• Platform (Environment)

• Size

22

Draw Out Range By Obtaining 3 Estimates

• Optimistic value (sopt)

• Most likely value (sm)

• Pessimistic value (spess)

• Expected value (EV)


6

)4( pessmopt sssEV

Sophisticated Schedule Modeling: Essential For Viable Project Plans

For a given Size, Complexity and Technology Work Expands To Fill Time

(Work expands due to lack of pressure)

Minimum Time To Complete

(Effort reduces to reduce schedule)

Optimal Effort for staff

(Lower Effort for Longer Schedule)

Effort Increase

due to Longer Schedule

Calendar Time

Effort

Month

s

Reasonable Solution Range

Software Solution

Range vs. Point Estimates (Source US Army)

Range o

f Ris

k &

Uncert

ain

ty

Estimating Accuracy Trumpet

Technical and Program Maturity

RO

M -

30%

to

+7

5%

Para

me

tric

-

10%

to

+2

0%

An

alo

gy

-15%

to

+3

0%

En

gin

ee

rin

g

-5%

to

+15%

Actu

al

-3%

to

+1

0%

Target Cost

Point estimate is most likely within range estimate

with higher potential for cost increase

Range estimate provides a degree of risk and uncertainty

+75%

-30%

B A Materiel Solution Analysis

Systems Acquisition Sustainment

Technology

Development

Production &

Deployment

Operations &

Support

Engineering and

Manufacturing Development

C

Pre-Systems Acquisition

What can a parametric model tell you?

26

What is likely to happen

Feel lucky?

Firm Fixed Price?

Understand the risk before you commit!

Difference Between Bottoms Up and Parametric

• “Bottom-up” or Manual Approach (Detailed Work Measurement)

• Measures everything directly and exactly

• Little or no analysis of alternatives

• Predictive value is limited at best

• Parametrics

• Measures or estimates key parameter values and ranges

• Understand (mathematically) their effect on time, effort

• Once relationships are known, parameter values can be changed and the effects evaluated


Example: Parametric Estimate Compared With History

SEER MAJOR Components

• SEER-SEM – software/application development, maintenance, integration and testing

• SEER-H – system, hardware and electronics development, production and support

• SEER-IT – IT infrastructure, services and operations and systems engineering

• SEER-MFG – hardware manufacturing and assembly

• SEER-SYS – Systems Engineering

• SEER-Cost IQ Case based reasoning estimation


Multiple Estimation Methods Used

• Knowledge based parametrics

• Built-in estimating relationships, ready to use

• Can be tuned and calibrated to improve accuracy

• Expert judgment – utilize SME inputs for a task

• Analogy – estimate based on past examples

• Top down – estimate total cost and allocate

• Bottom up – estimate each detailed task or process step


SEER Enterprise Costing Solution: Parts thru Total Ownership Cost

SEER-MFG

SEER-SEM SEER-IT

Detailed SEER-H Estimate:

• Development • Production

• Operation & Support

• System Level Costing

Detailed Estimate

Parametric Estimate

Software/IT Estimate Actual Procurement

Cost Data

SEER-H


SEER – Risk Driven Estimates The Engine For Project Evaluation

Least, likely, and most inputs provide a range of

cost and schedule outcomes

Confidence (probability) can be set and displayed for any estimated item

• SEER predicts outcomes

• SEER uses inputs to develop probability distributions

• The result is a probabilistic estimate

• SEER will predict a likely range of outcomes

• Monte Carlo provides project-level assessments of risk


Bia

s M

itig

ation:

Ris

k

Knowledge Bases

• SEER-H uses knowledge bases to derive and preset initial parameter settings based on industry experience

• Choose knowledge bases for:

• Application (Aircraft Stabilizer, Analog Display, Electro-Mechanical Control, etc.)

• Platform (Air-Manned, Air-Unmanned, Groung Mobile, etc.)

• Acquisition Category (Build to Print, Buy & Integrate, Make, Modification, etc.)

• Standard (Commercial, Military, Space, etc.)

• Operations & Support (Remove & Replace, Repair, Scheduled Maintenance, etc.)

• Users may add their own knowledge bases if desired

Bia

s M

itig

ation:

Com

ple

ted A

ctu

als


SEER for Software Example: Top 10 Impacts & Sensitivity Analysis

Estimates Are Validated Against Data (Yours and/or Other)

SEER-SEM

Estimate

Your Data

Regression

Trendline

Galorath

Benchmark

Trendline

Your History

Data

36

Project Monitoring & Control “When Performance is Measured Performance Improves”

• Adds Performance Measurement (Earned Value) methods to SEER-SEM

• Accepts progress & expenditure inputs

• Provides cost, schedule, and time variances

• Provides cost, schedule, and time indices

• Performance-based cost & schedule Estimate at Completion

• Displays health and status indicators

© 2016 Copyright Galorath Incorporated

36

SEER-SEM Total Cost of Ownership

Why Should We Care: Impacts ROI & development decisions can impact maintenance costs.

37

Staff Vs Maintenance Rigor

0

500

1000

1500

2000

2500

3000

3500

1 7 13 19 25 31 37 43 49 55 61 67 73 79 85

Time

sta

ff h

ou

rs p

er

mo

nth

develop

Rigor vhi+

Rigor nom

Rigor vlo

Bia

s M

itig

ation:

Forg

ott

en C

osts

Searchable Cost Catalogs

• Catalogs used for actual cost metrics for common tasks and parts

• Configurable to the customers requirements

• Refresh estimates based on the latest metrics


Bia

s M

itig

ation:

Analo

gie

s

Ply Cost Estimator – CATIA V5


Bia

s M

itig

ation:

via

ble

cost

trades

Cost IQ - Obtaining Cost Directly From Limited Requirements

A Case Based Reasoning system that can transform high level requirements and specifications into a cost modeling workup within a sophisticated cost estimating model.

This approach can provide even concept planners with an advanced understanding of the cost of

potential designs.


Bia

s M

itig

ation:

Refe

rence c

lasses

• Scenarios can be generated proactively or incidentally

• Deliberately defining a standard estimation WBS pattern

• Turning an exemplar estimate into a WBS pattern

Scenarios

• Pre-configured WBS patterns may be stored as a “scenario”

• Scenarios may be configured with options for the user to select which elements to include

• The WBS structure will be loaded with the user’s tailoring


Bia

s M

itig

ation:

Com

ple

ted A

ctu

als

SEER-DB

• Store all estimates in a secure repository

• Secure access to individuals or user groups

• Supports estimate revisions and locking


Bia

s M

itig

ation:

Actu

al H

isto

ry

SEER – In Summary

• SEER Parametric cost estimation software provides estimates of Effort, Cost, Schedule For Information Technology projects & Program’s

• Gives a consistent framework for estimating and planning projects

• Helps you understand why projects cost what they do


Key Points for today . . .

Estimates can be better, reducing bias & strategic mis-estimation… Parametrics can

help.


Tempering with an

“outside view” can mitigate some bias

Without care estimates are usually biased (even with experts)

At Galorath – We Do Estimation

Galorath solutions: Tools and consulting

Galorath works with industry leaders to help adopt and improve organizational estimation best practices

• Over 30 years in business

• Hundreds of customers in government & industry; many Fortune 500

• Headquartered in El Segundo, CA; International office in Andover, U.K. with staff in the U.S. and

Europe

• Professional services organization provides consulting and training

presentation on 'why cant people estimate' event, 23rd june 2016

Education