coonce.tom
DESCRIPTION
TRANSCRIPT
Confidence Level Estimating and Budgeting
February 2011
Tom Coonce
Topics
• A brief review of confidence level estimating, budget-setting policy and rationale
• Status of program and project Joint Cost and Schedule Confidence Level (JCL) calculations
• JCL issues and lessons learned• Actions completed or planned to
address lessons learned
2
3
Recap of JCL Policy and Rationale
• NPD 1000.5– Programs are to be baselined at a 70% joint confidence
level– Projects are to be baselined/budgeted at JCL that supports
the program approved JCL– Projects are to be funded at no less than a 50% JCL – JCLs are to be developed and maintained through launch– Program and project proposed cost and schedule baselines
will be assessed by an independent review team
• Policy Rationale– Improve cost and schedule performance– Improve expectations with stakeholders– Improve project planning– Improve understanding of project plans
JCL – Defined by a Product, Refers to a Process
• JCL = Joint Confidence Level – Identifies probability that a given project or
program’s cost will be equal or less then the targeted cost AND the schedule will be equal or less then the targeted schedule date
– Per NPD 1000.5 used at KDP-C to set Program/Project Budget Baseline
HeadquartersOffice of Program Analysis and Evaluation
5/18/09 2
MMS SRB ICAJCL Scatter Plot, Corr=0.646 (ICE 70%=1079)
• JCL is More than the Scatter Plot– It’s a systematic process of integrating cost, schedule, and risk – Provides a cohesive and holistic picture of the project or program’s ability to
achieve cost and schedule goals and to help the determination of reserves (schedule and cost)
– Provides key Decision Support information– Does the project have enough funds?– Can the project meet the schedule?– What are areas of risk toward successful execution of the project? – What risk mitigation strategies provide the best project benefit?– What are project phasing (fund) needs? 4
5
Acceptable JCL Methods
• Resource-Loaded Schedule with Monte-Carlo simulation, with three different techniques– Bottoms-up Resource Loaded Schedule or BURLS technique is
based on resource or cost-loaded schedule that focuses on the project risk lists (known-known risks) and uncertainties (known and unknown unknowns) to set the cost and schedule possibilities
– Cost Estimate/Program Assessment (CEPA) technique is based on a top-level resource or cost loaded schedule that focuses on project performance as well as project risks to set cost and schedule possibilities
– Project Risk Evaluation Process or PREP technique is based on the projects’ top-level resource or cost loaded schedule that focuses on the project’s risk list as well as the views of an Independent panel of experts to set cost and schedule possibilities
• Parametric – Uses cost and schedule estimating relationships, data dispersion,
and relationships between cost and schedule to determine the overall cost and schedule target and associated JCL
An Example of JCL Output
6
Points representcost/Schedule resultpair from each iterationof the simulation
Lower quadrantidentifies iterationssatisfying cost andschedule targets ~ the JCL
Horizontal barindicates Y-axis (Cost) confidencelevel
Vertical bar indicates x-axis (Schedule) Confidence level
Frontier line identifiescost/schedule surface that meets Target JCL
7
Management Councils Revisions the Policies and Expectations
• Eliminated the requirement at KDP-B and D, but stipulated that KDP-Cs JCL must be based on probabilistic resource loaded schedules (May 2009 SMC)– Directed pilot projects be completed for projects at KDP-B
• Eliminated the requirement to maintain the JCL during implementation (November 2009 PMC)– Project are not required to develop a JCL at any other time during
implementation if it is “on plan”– Projects are to be monitored as part of the Baseline Performance Review
(BPR) process– If a project is re-baselined, scored red in the BPR, or if BPR leadership
directs, it must re-compute a JCL– The agency is to use existing metrics and/or establish new factors for
determining when a project turns Red • Eliminated JCL requirement for loosely coupled programs, retained the
requirement for tightly coupled programs, and required MDs to submit program impact assessment whenever a new project enters the program (June 2010 PMC)– Eliminate JCL language from NPD 1000.5; reconstitute and update in NPD 7120.5E
8
Management Councils Revisions the Policies and Expectations (Continued)
• Include cost and schedule range estimates and associated probabilities for all projects at KDP-B with LCC >$250M (June and Aug 2010 PMCs)– Coordinate policy language and place into
7120.5E• Projects must develop, explain and
defend plans and associated probability calculations at KDP-B and C (June 2010 PMC)
9
Final Proposed 7120.5E Language(After the 4 August 2010 PMC)
Under 2.5.4.2 Confidence Level Estimating and Budgeting
2.5.4.2.4 Projects with an expected life cycle cost in excess of $250 million are required to develop cost and schedule estimates and associated confidence levels for meeting those estimates.
(1) At KDP B, a range of cost and schedule estimates and associated confidence levels of meeting those estimates will be provided at KDP-B for projects and single-project programs (see the footnote for paragraph 2.5.4.2.2). This policy applies to both competed (AO) and directed (non-AO) missions. The decision authority will use this information to guide the formulation activities and to establish the target cost and schedule ranges and resources phasing until KDP-C estimates are developed..
(2) At KDP C, projects are budgeted to a 70 percent joint cost and schedule confidence level or as approved by the decision authority.
10
So, What is Required to Meet the KDP-B Programmatic Requirements?
• For non-AO missions, projects must provide:– A “high” development (thru launch) cost estimate and
associated probability to achieve that cost– A “low” development cost estimate and associate probability
to meeting that cost– Cost estimates must include all cost elements, including
those for which the project managers is not directly responsible, e.g. the launch vehicle and external partners contributions
– High and low schedule durations (thru launch) and associated probabilities of meeting those durations
– A high level schedule of activities associated with the low and high life cycle cost estimates
– High and low life cycle cost estimates– Year-by-year required resources to meet both the low and
high estimates
11
Programmatic Expectations at KDP-B
Non-AO Missions (Concluded)– Basis of Estimates. Examples include:• How the cost and schedules were derived• How risks (known and unknown were addressed in the
estimates)• How the year-by-year resources were determined
• For AO missions, the mission directorate, or designated entity, will provide the same information and calculations specified for the Non-AO missions to the decision authority
• Expect to set budget at higher cost range number, but set development cost and schedule estimates at lower numbers– UFE held by program or mission directorate– AO mission development and life cycle costs based on
winning bid
12
Status of Program and Project JCLs
• During past two years, the following JCLs were completed and presented at PMCs– LDCM, MMS, MSL, SOFIA, GPM*, MAVEN,
LADEE, NuStar, and IRIS,
• Following JCLs are still in process– JWST
• Pilot projects at KDP-B– CxP (cost loaded schedule)– SMAP (parametric)– MAVEN (parametric)
* GPM used the CEPA technique; remainder used BURLS
13
JCL Implementation Issues
• Most projects had not assembled multiple schedules into a top-level electronically integrated schedule. Required extensive time and effort
• Most projects had not quantified their discrete risks nor were they linked to their cost or schedule estimates. Required extensive time and effort
• Few projects had a basis for their cost estimates. Most were based on assigned budgets. Costs were usually not separated into fixed and variable
• Most projects had lower level schedule of activities detail than costs• Project personnel were not staffed or trained to perform the analysis or use the
tools• Use of low cost COTS scheduling and simulation software were inadequate to
perform the task– Microsoft Project with @Risk took too long to perform computations
• Projects and Standing Review Boards had relatively narrow opinions about discrete risks (Known-knowns)
• Consultants were less than successful in convincing projects or SRB members to add other uncertainty (due to known and unknown-unknown risks)
In spite of these issues, most projects developed improved plans and explained them better than without the JCL requirement
14
JCL Lessons Learned
• Improve project control functions– Ensure projects develop credible cost and schedule basis of
estimates– Ensure projects maintain an integrated master schedule that
reflects costs and schedule risks, i.e., maintain resource-loaded schedules
– Ensure projects continuously quantify and monitor risks and their impact on their plans
• Provide projects with historical cost and schedule uncertainly data – To assembly level if possible, but to the subsystem level at a
minimum
• Provide projects with launch vehicle price and schedule uncertainty data
• Provide projects with tools that will permit required turnaround times
15
Action Completed and Planned
• Completed– Created near term optimization techniques for MS Project/@Risk
• Schedule Shorting Tool• Risk Scenario & Automation Tool (R-SAT)
• In-Process– Developing rapid stand alone simulation tools to be used in
conjunction with MS Project (replaces @Risk)• DICE• JACS
– Collecting cost and schedule uncertainties data – Collecting LV price and schedule uncertainty data– Brainstorming program control revitalization plan
• To Be Done:– Deploy new simulations tools– Disseminate LV uncertainty data– Disseminate cost and schedule uncertainty data and educate
project staffs on their use– Decide on program control improvements and implement
Backup
17
Detailed Program and Project Status on JCLs
18
Detailed Program and Project Status on JCLs (Concluded)