Todd Little

Uncertainty Surrounding the

Cone of Uncertainty

IEEE Software, May/June 2006

Managing the Coming Storm Inside the Tornado

When will we get the requirements?All in good time, my little pretty, all in good timeBut I guess it doesn't matter anyway

Doesn't anybody believe me?

You're a very bad man!

Just give me your estimates by this afternoon

No, we need something today!

I already promised the customer it will be out in 6 months

No, we need it sooner.

Not so fast! Not so fast! ... I'll have to give the matter a little thought. Go away and come back tomorrow

Ok then, it will take 2 years.

Team Unity

Project Kickoff

We’re not in Kansas Anymore

My! People come and go so quickly here!

I may not come out alive, but I'm goin' in there!

The Great and Powerful Oz has got matters well in hand.

"Hee hee hee ha ha! Going so soon? I wouldn't hear of it! Why, my little party's just beginning!

Developer HeroReorg

Testing

The Cone of Uncertainty (Boehm)

0.1

1

10

Feasibility Concept ofOperation

RequirementsSpec

ProductDesign Spec

Detail DesignSpec

AcceptedSoftware

Minimum

Maximum

0.5

2

Cone of Uncertainty: The amount of possible error in a software project estimate, which is very large in the early stages of a project and shrinks dramatically as the project nears completion – Steve McConnell

Hurricane Rita

Landmark Product Suite

Common Model Representation

Well data

Production data

Seismic data

Velocity data

Reservoir /Fluid data

Structural /Stratigraphic data

Common Model Representation

Data from LGC

Initial Estimate vs. Actual Project Duration (from LGC Portfolio Database)

y = 1.6886x

0

200

400

600

800

1000

1200

0 100 200 300 400 500 600 700 800 900 1000

Initial Estimate

Ac

tua

l

LGC Data

Ideal

Linear (LGC Data)

Data from Tom DeMarco

It’s déjà vu all over againIndustry data from Tom DeMarco

0

200

400

600

800

1000

1200

0 100 200 300 400 500 600 700 800 900 1000

Estimated Effort

Ac

tua

l E

ffo

rt

Actual

2X

Ideal

Cumulative Distribution Curve for Actual/Estimate (DeMarco)

DeMarco Cumulative Distribution Function

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0.1 1 10

Ratio

CD

F P

rob

ab

ilit

y

Data

Log Normal Curve

p(10) 0.79p(50) 1.74p(90) 3.81

CDF Distribution Curve (LGC)

Landmark Graphics Cumulative Distribution of Portfolio Projects

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0.1 1 10

Ratio of Actual to Estimate

Cu

mm

ula

tive

Pro

bab

ility

p(10) 0.96p(50) 1.76p(90) 3.23

Probability Distribution Curve

Distribution Curve of Actual/Estimated (DeMarco data vs. LGC)(Demarco data is Effort/Effort; LGC data is Duration/Duration)

0

0.5

1

1.5

2

2.5

0 1 2 3 4 5 6 7 8

(Actual/Estimated)

Fre

qu

en

cy

DeMarco

LGC

Log Normal Distribution

Estimation Accuracy follows a Log Normal distribution

0.1 1 10 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

Median

Mean

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50.1 1 10 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50.1 1 10 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

“It’s tough to make predictions, especially about the future.”Yogi Berra, Niels Bohr

That idea is so damned nonsensical and impossible that I'm willing to stand on the bridge of a battleship while that nitwit tries to hit it from the air.

Newton Baker, U.S. secretary of war in 1921, reacting to the claim of Billy Mitchell (later Brigadier General Mitchell) that airplanes could sink battleships by dropping bombs on them.

“It’s tough to make predictions, especially about the future.”Yogi Berra, Niels Bohr

Heavier-than-air flying machines are impossible.Lord Kelvin, British mathematician, physicist, and president of the British Royal Society, spoken in 1895.

How does Estimation Accuracy Improve Over Time? (Boehm)

0.1

1

10

Feasibility Concept ofOperation

RequirementsSpec

ProductDesign Spec

Detail DesignSpec

AcceptedSoftware

Minimum

Maximum

0.5

2

0

200

400

600

800

1000

1200

0 100 200 300 400 500 600

Ideal

From Start

Post Env

Post Plan

Post Dev

So what does LGC data look like?

Landmark Cone of Uncertainty

But is Uncertainty Really Reduced?

“Take away an ordinary person’s illusions and you take away happiness at the same time.”

Henrik Ibsen--Villanden

Cumulative Distribution (CDF) Curve

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0.1 1 10 100

Initial

Post Env

Post Plan

Post Dev

Log Normal

Remaining Uncertainty

The Pipe of Uncertainty

0.1

1

10

Envisioning Planning Developing Stabilizing

Minimum

Maximum

0.5

2

Does Landmark Suck at Estimation?

A severe depression like that of 1920-21 is outside the range of probability.Harvard Economic Society, Weekly Letter, November 16, 1929.

Does Landmark Suck at Estimation?

I think there is a world market for about five computers.Thomas J. Watson, chairman of IBM, 1943.

Does Landmark Suck at Estimation?

They couldn't hit an elephant at this dist…General John B. Sedgwick, Union Army Civil War officer's last words, uttered during the Battle of Spotsylvania, 1864

Estimation Quality Factor (EQF)

Elapsed Time

Val

ue to

be

Est

imat

ed

Actual Value

Initial Estimate

Actual End Date

Link to article by Tim Lister

Blue Area

Red AreaEQF =

EQF from Lister/DeMarco

EQF Comment

1.8 Typical disaster project

3.8 Median of Medians

5.0 Pretty Good

8.0Excellent sustained Median EQF

10.0 Not Sustained

EQF Distribution Curve (LGC)

Landmark Graphics Cumulative Distribution of EQF

-0.2

0

0.2

0.4

0.6

0.8

1

0.1 1 10 100

EQF

CD

F/P

rob

Raw EQF

LogNormal

p(10) 2.80p(50) 4.78p(90) 11.68

EQF for duration has a theoretical minimum of 2.0

We slip one day at a time, EQF=2

Elapsed Time

Val

ue to

be

Est

imat

ed

Actual Value

Initial Estimate Actual End Date

Blue Area

Red AreaEQF =

(EQF-2) Distribution Curve (LGC data)

Landmark Graphics Cumulative Distribution of EQF

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0.1 1 10 100

EQF

CD

F/P

rob

EQF Data - 2

Log Normal Curve

p(10) 2.80p(50) 4.78p(90) 11.68

From the home office in Duncan, Oklahoma

Top Ten reasons why we are late in 2008

Dubai, UAEDubai, UAE

10: Requirements, what Requirements?

What you want, baby I got it

R-E-Q-U-I-R-E Find out what it means to me

Top Ten reasons why we are late in 2008

9: Dependencies on other groups that were late

Top Ten reasons why we are late in 2008

8: Over-optimistic Schedule Estimation

Always look on the bright side of code. . . . . . .Always look on the bright side of code. . . . . . .The code’s a piece of $#!^, when we look at itWe can always overlook a minor kink . . . .It probably compiles, it might even link . . .Surely that must mean it doesn’t stink

Top Ten reasons why we are late in 2008

7: Those weren’t MY estimates

How low can you go!

Scheduling Ritual

Top Ten reasons why we are late in 2008

6: Not enough testers or documentation resources.

Who needs them anyway? We put those bugs--I mean features--in there on purpose. Besides, it was difficult to program, it should be difficult to use.

Top Ten reasons why we are late in 2008

5: Offshore and Outsourcing issues

My source code lies over the ocean, My source code lies over the ocean, My source code lies over the sea .My source code lies over the sea .

My source code lies over the ocean, My source code lies over the ocean, Oh bring back my source code to meOh bring back my source code to me

. . . . .. . . . .Bring Back, Bring Back, Bring Back, Bring Back,

oh bring back my source code to me, to meoh bring back my source code to me, to meBring Back, Bring Back, Bring Back, Bring Back,

oh bring back my source code to meoh bring back my source code to me

Top Ten reasons why we are late in 2008

4: One word, Ch-ch-ch-changes

Top Ten reasons why we are late in 2008

3: I can’t get no, System Admin I can’t get no, CM action ‘cause I try, ..and I try, ….and I try, ……and I try….

Top Ten reasons why we are late in 2008

2: You didn’t give me the headcount that you promised

Top Ten reasons why we are late in 2008

1: Weren’t you doing the backups!?1: Weren’t you doing the backups!?

Top Ten reasons why we are late in 2008

You can get better

How to get better

Start Measuring Understand bias Collaborate Manage Uncertainty

Start Measuring

Understand Bias

"What gets us into trouble is not what we don't know. It's what we know for sure that just ain't so.“

» Mark Twain

Don’t know thatwe don’t know

Knowable

Unknowable

Uncertainty

Know that we know

Know that we don’t know

Don’t know that we know

Don’t know thatwe don’t know

Knowable

Unknowable

Uncertainty

Know that we know

Know that we don’t know

Don’t know that we know

UncertaintyManagement

Planning Risk Management

p10 p50

p90

Just Double the Estimate

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0.1 1 10

Actual/Estimate Ratio

CD

F

Defending an Unpopular Schedule

http://www.stevemcconnell.com/ieeesoftware/bp03.htm

Developers tend to be temperamentally opposed to the use of negotiating tricks. Such tricks offend their sense of technical accuracy and fair play. Developers don't want to offer lopsidedly high initial estimates even when they know that customers, marketers, or bosses will start with lopsidedly low bargaining positions.

Steve McConnell

We want this

Negotiation Bias

"It is difficult to get a man to understand something when his salary depends upon his not understanding it.“

» Upton Sinclair:

Trust: The Cooperative or Non-Cooperative Game?

Customer perception of Team's ability to Deliver

Desire to set aggressive targets

Team’s ability to Deliver against targets

o

o

s

R

 

Customer perception of Team's ability to Deliver

Team’s ability to Deliver against targets

Test 1 (Jørgensen)

Group Guidance Result

A 800

B 40

C 4

D None 160

Test 1

Group Guidance Result

A 800 300

B 40 100

C 4 60

D None 160

Test 2

Group Guidance Result

A Minor Extension

B New Functionality

C Extension 50

Test 2

Group Guidance Result

A Minor Extension

40

B New Functionality

80

C Extension 50

Test 3

Group Guidance Result

A Future work at stake, efficiency will be measured

B Control 100

Test 3

Group Guidance Result

A Future work at stake, efficiency will be measured

40

B Control 100

Collaborate: Planning Poker

Collaborative Estimation Activity

Modified Fibonacci Sequence 0,½,1,2,3,5,8,13,20,40,100

,∞,?

Interactive version at www.planningpoker.com

Exercise

Low Med High

Distance from Las Vegas to Houston, Texas

Height of the Empire State Building

Population of Sweden

U.S. Oil Consumption/day

Water in a 100 gallon vat filled with sand

Exercise

Low Med High

Distance from Las Vegas to Houston, Texas

1222

Height of the Empire State Building

1453

Population of Sweden 9MM

U.S. Oil Consumption/day 20MM

Water in a 100 gallon vat filled with sand

35

Managing for Uncertainty

We expect uncertainty and manage for it through iterations, anticipation, and adaptation.

The Cone of Uncertainty

Uncertainty

Time

Val

ue

ContractMinimum scope for the release at the latest date that it can be released.

PlanPlanned scope for the

release at the optimal time that it can be released.

TargetBest possible scenario if

everything went perfectly.

The A/B/C List sets proper expectations

A MUST be completed in order to ship the product.

B SHOULD be completed in order to ship the product.

C MAY be completed prior to shipping the product if time allows.

Only “A” features may be committed to customers.

“A” features must fit in a p90 confidence schedule. No more than 50% of the planned effort can be allocated to “A” items

A

A/B/C List

50% 100%

Backlog Plan

Typical Delivery

25%

A B C

B C D

50% 25%

Target Delivery Date

A/B/C List

50% 100%

Backlog Plan

Uncertainty Risk

25%

A B C

B C D

50% 25%

Target Delivery Date

A

Uncertainty

Time

Val

ue

C

B

A

Uncertainty

Time

Val

ue

Failure

MaximalSuccess

ScheduleFlexible

ScopeFlexible

MinimalSuccess

Estimation accuracy improves(Eveleens and Verhoef)

Estimation accuracy constant(Eveleens and Verhoef)

Estimation accuracy decreases(Eveleens and Verhoef)

Take Aways

Estimation follows log-normal patterns There is a cone, sorta Relative uncertainty can reduce, but

often may stay relatively constant Improvement is possible Watch out though

Successful Projects?

Customer Satisfaction

Contact

Todd Little Senior Development Manager, Landmark

Graphics Corporation, www.lgc.com tlittle@lgc.com www.toddlittleweb.com

Coming Soon from Addison-Wesley!

Stand Back and Deliver: A Leaders Guide to the Agile Enterprise

Your Questions?

The Cone of Uncertainty