Learn. Perform. Succeed

Cost and Schedule

Confidence Levels

Roberta Tomasini

Professor of Financial Management

Defense Acquisition University

roberta.tomasini@dau.mil

(703) 805-3764

1

2

Outline

• Sample Program – Are we OK?

• What About Uncertainty?

• Create a Program Model

• Simulate the Range of Outcomes

• Sensitivity/Risk Drivers “Tornado Chart”

• Show Estimates as Ranges

• Risk and Opportunity

• Releasable Management Reserve

• Schedule Analysis

• Final Thoughts

3

OK?

Sample Program – Are we O.K.?

APB Objective/Threshold = $130M/$143M Budget = $141.20M

We can’t tell

Cost Estimate Example

Integration

Design $6.00

Fabricate subsystem prototypes $1.00

Subsystem Prototype testing $0.70

Demonstration

System Prototype Build $7.00

Reliability Growth Test/Redesign $0.80

System Developmental Testing $2.00

Operational Test $1.20

Producton/Fielding

Produce Systems $111.00

Initial Spares $10.00

Fielding/NETT $1.50

Total Program Acquisition Cost $141.20

4

What About Uncertainty?

• To get a better handle on range of program outcomes…

The team sits down and looks at the uncertain elements to build in risk.

What’s the least this could cost (Best Case)?

What’s the most probable cost (Most Likely)?

What’s the most it could end up costing (Worst Case)?

$140M (Worst case) $95M(Best Case)

$111M (Most Likely)

Produce Systems

Cost Estimate Example

Integration Design $6.00 Fabricate subsystem prototypes $1.00

Subsystem Prototype testing $0.70

Demonstration System Prototype Build $7.00

Reliability Growth Test/Redesign $0.80 System Developmental Testing $2.00 Operational Test $1.20

Producton/Fielding Produce Systems $111.00

Initial Spares $10.00 Fielding/NETT $1.50

Total Program Acquisition Cost $141.20

5

Create a Program Model Risk-Adjusted

• Then plug your distributions into a risk program

@RISK EXAMPLE

Point Estimate* Best Case Most Likely Worst Case Mean Value

Case

Integration

Design $6.00 $6.00 $6.00 $6.00 $6.00

Fabricate subsystem prototypes $1.00 $0.80 $1.00 $1.20 $1.00

Subsystem Prototype testing $0.70 $0.50 $0.70 $0.90 $0.70

Demonstration

System Prototype Build $7.00 $5.60 $7.00 $10.50 $7.70

Reliability Growth Test/Redesign $0.80 $0.60 $0.80 $1.20 $0.87

System Developmental Testing $2.00 $1.60 $2.00 $2.50 $2.03

Operational Test $1.20 $0.90 $1.20 $1.60 $1.23

Producton/Fielding

Produce Systems $111.00 $95.00 $111.00 $140.00 $115.33

Initial Spares $10.00 $8.00 $10.00 $12.00 $10.00

Fielding/NETT $1.50 $1.50 $1.50 $1.50 $1.50

Total Program Acquisition Cost $141.20 $120.50 $141.20 $177.40 $146.37

*Point Estimate should be the Program Office Estimate, which should be the Most Likely Case

6

Simulate the Range of Outcomes

•In Breach: <40% chance of coming in under threshold

•Under-funded: < 35% chance of coming in under budget

• Now we have a picture of our cost uncertainty

• Then run a Monte Carlo simulation

APB Threshold $143M

APB Objective $130M

Budget $141.2M

7

Sensitivity/Risk Drivers “Tornado Chart”

Correlations for Total Program Cost /Expected Value/F17

Correlation Coefficients

Operational Test / Expecte.../F10-.007

Fabricate subsyetem Protot.../F3-.019

System Developmental Testi.../F9-.024

Reliability Growth Testing.../F8 .042

System Prototype Build / E.../F7 .072

Initial Spares / Expected .../F14 .082

Subsystem Prototype testin.../F4 .089

Produce Systems / Expected.../F13 .989

-1 -0.75 -0.5 -0.25 0 0.25 0.5 0.75 1

• We can also see what drives our cost uncertainty Big numbers with wide ranges – no surprise!

Does this coincide with your risk cube?

8

Show Estimates as Ranges

Dec-0

5

Jan-0

6

Feb-0

6

Mar-

06

Apr-

06

Do

lla

rs (

M)

Time

Threshold

Objective

FY

11

FY

12

FY

10

FY

13

Cur

rent

9

Risk and Opportunity

Budget

Opportunity: Uncertainty covered by budget Risk: Uncertainty not covered by budget

10

Releasable Management Reserve

(No Right Answer)

• In the above case, what’s your releasable MR if your budget is $175M?

$153M?

$146.3667M?

• Concept: Releasable MR is budget beyond your desired confidence level

11

Schedule Analysis

• Schedule Analysis must be done in a tool

• MS Project, Primavera, …. with Monte Carlo tool

• Basic schedule…start 1 Jun with finish on Sept 3rd

• 3 events with duration

• Success oriented…no failure planned

ID Task Name Duration Start

1 Project 95 d 6/1/14

2 Start 0 d 6/1/14

3 Design Unit 30 d 6/1/14

4 Build Unit 40 d 7/1/14

5 Test Unit 25 d 8/10/14

6 Finish 0 d 9/3/14

6/1

6/1 6/30

7/1 8/9

8/10 9/3

9/3

May June July August September October

12

Schedule Triangular Probability Distribution

Relative

Likelihood

of

Occurring

Low Most Likely High

Threats

Opportunities

ID Task Nam e Rept ID Min Rdur ML Rdur Max Rdur Curve

1 Project 2 0 d 0 d 0 d 0

2 Start 0 0 d 0 d 0 d 0

3 Design Unit 0 20 d 30 d 45 d 2

4 Build Unit 0 35 d 40 d 50 d 2

5 Test Unit 0 20 d 25 d 50 d 2

6 Finish 0 0 d 0 d 0 d 0

13

Schedule Monte Carlo provides range

• 3 Sept less than 15% likely …

Date: 8/12/2013 3:42:21 PM

Samples: 5000

Unique ID: 2

Name: Project

Completion Std Deviation: 8.98 d

Confidence Interval: 0.25 d

Each bar represents 3 d

Completion Date

Fre

qu

en

cy

Cum

ula

tive

Pro

ba

bili

ty

9/13/14 8/20/14 10/15/14

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0.02

0.04

0.06

0.08

0.10

0.12

Completion Probability Table

Prob Prob Date Date

0.05 8/31/14

0.10 9/2/14

0.15 9/4/14

0.20 9/5/14

0.25 9/7/14

0.30 9/8/14

0.35 9/10/14

0.40 9/11/14

0.45 9/12/14

0.50 9/13/14

0.55 9/14/14

0.60 9/15/14

0.65 9/17/14

0.70 9/18/14

0.75 9/20/14

0.80 9/21/14

0.85 9/23/14

0.90 9/26/14

0.95 9/29/14

1.00 10/15/14

Nominal 9/3 Finish Date

14

Final Thoughts

• QRA can lead to a better understanding of your program

• It won’t tell you for sure… what your program will cost

or how long it will take

• It will tell you… what your team really thinks your program will cost

how long your team really thinks it will take

what your team really thinks your probability of success is

potential risk/cost drivers

confidence level of your program now

confidence level after budget cuts

confidence level adjusted for risks and opportunities

• The power is in the team – not the tool

Additional DAU on-line resources:

CLB024 Cost Risk Analysis Introduction

CLM017 Risk Management

https://learn.dau.mil/html/clc/Clc1.jsp?cl=

15

Any Questions?

16

Standard Deviation Normal Curve

95.4%

99.7%

68.2%