ASMC PDI2010

Nashville, TNWorkshop #57

X (you are here)

Risk, Uncertainty, and Trouble: Escaping the

RUT of Program Instability

Col Brian Shimel, CDFMChief, Financial Analysis

Air Force Space Command2010

Risk, Uncertainty, and Trouble: Escaping the

RUT of Program Instability

Col Brian Shimel, CDFMChief, Financial Analysis

Air Force Space Command2010

What is Behind the Curtain for this Lucky Audience?

•Appropriations—The Hidden Assumptions• Purpose, Amount, and Time

•Risk, Uncertainty & a bit of statistics•Examples in Operations, Acquisition•Risk, Uncertainty, and the Point Estimate•Defending the Budget•Acquisition: Why do we have to keep reforming it?•Ethics•8 Step Problem Solving Process•Decision Support•Backups -- Sources & GAO Excerpts

Appropriations• O&M--3400/3500 (prd of performance-1yr)• R&D – 3600 (2 yr)• Procurement -- 3010/3020/3080 (3 yr)• MILCON & MFH (5 yr)• Accept the $$, accept a “project” schedule

– Base Year (BY) -- $B, $M, $K– Then Year (TY) composite rate (outlay profile)

• 11% 42% 34% 9% 4%

Risk

• The Chance of an Unfavorable Event– Cost Risk – How Much?– Technical Risk – Will it Work?– Schedule Risk -- When? How long?

UNCERTAINTY

What should I say about it?

UNCERTAINTY• Unknown, unpredicted variation around an

expected value/outcome (a prediction of a future state)

• Risk and uncertainty are situational and can be inverse, they can be related– Affected by expectations

UNCERTAINTY

U

UNCERTAINTY

Y

UNCERTAINTY

YU

A Distribution

Independent

Dep

ende

nt

A Distribution

Independent

Dep

ende

nt

A Distribution

Independent

Dep

ende

nt

A Distribution

Independent

Dep

ende

nt

A Distribution

Independent

Dep

ende

nt

Normal Mode = Median = Mean

Small Measure of Dispersion

Narrow, Normal, Certain

Age of Children in Kindergarten

Making Light Bulbs

A-12

Widely distributed, Normal, Varied

Age or Height of People in Audience

Right Skewed

Mean > Median

Right Skeweddispersion strongly favors one side

Right

Right Skewed

Air Force Right

BASELINE

AAFES in Cheyenne Mt

• August 2004; Mess Attendant contract• New proposal --$500K over budget• Dining Facilities in 2 locations• 3 people without separate rations• A captive clientele

Narrow, Normal, Certain

Narrow, Normal, CertainThe Point Estimate

Normal but Not Certain

Normal but Not CertainThe Point Estimate?

2 Different Processes

2 Different Owners

3 Different Processes

What Do We Communicate?

Program Bid, Baseline, EstimateActual Space Program Cost Distribution

-

50

100

150

200

250

5,500 6,000 6,500 7,000 7,500 8,000 8,500 9,000 9,500 10,000

BY02$M

Fre

qu

en

cy

Should a leader make his/her decision based on an 80% CL value for this estimate of total

program costs …

… or an 80% CL value for this estimate?

The elephant in the room

Making the Elephant Disappear

This is our Story & we’re . .

HQ Budget Drill

Not Gone at the Working Level

Accepted Risk

Unexpected Risk

Little Big Problem

Taking on the Management Challenge

Estimates –v- Reality

Right Skewed—Mean is greater than the median

Classic DoD Project

Classic DoD Project

Classic DoD ProjectWeapons Development

Complex Development

Sub-Components

Improved Cost & Schedule Control

Stable and Predictable

GAO-08-619, July 2008

Knowledge-based Acquisition

I want it now I am willing to think it through

Unc

erta

inty

Time

GAO claims DoD overruns its acquisition baseline by 30% (so does the RAND Corp, and so does AFCAA)

GAO-08-619, July 2008• Cost estimates that lack knowledge and rely

heavily on assumptions have inherently high levels of risk and uncertainty

• DOD’s flawed funding process is largely driven by decision makers’ willingness to accept unrealistic cost estimates and DOD’s commitment to more programs than it can support.

Risk Uncertainty and Trouble

• It is a disservice to present a decision-maker with an estimate for a new groundbreaking weapon system (at a high confidence with only a small increase in funding) . . . And it is foolhardy for a decision-maker to accept that estimate.

UNITED STATES DEPARTMENT OF DEFENSE Fiscal Year 2011 Budget Request

Chapter 5REFORMING HOW DoD DOES BUSINESS

Reform How DoD BuysAcquisition reform is the process by which DoD is changing how it buys its weapons and other important systems and investments. DoD is focused on acquisition reform at the strategic and tactical level.• At the strategic level, we are ensuring investments are affordable and consistent with warfighting priorities.• At the tactical level, we are focused on acquiring warfighting capability via a disciplined, effective and knowledge- based acquisition process.

Acquisition Reform

• WSARA (2009)• OSD

– CAPE– BCEFM– PARCA (Nunn-McCurdy Root Cause?)

• AF – AIP– AFCAA evaluation of POM inputs– Section 852 funding

Implementation of WSARAPerformance Assessment & Root Cause Analysis (PARCA)

Role of the senior official for PARCA:• Conduct performance assessments for MDAPs periodically or

when requested by SECDEF, USD(AT&L), Secretary of Military Dept, or head of Defense Agency

• Conduct root cause analysis for MDAPs as required by 10 USC 2433a, or when requested by SECDEF, USD(AT&L), Secretary of Military Dept, or head of Defense Agency

• Advise acquisition officials on MDAP performance issues:ـ Prior to certification under 10 USC 2433a (critical cost growth in

MDAPs);ـ Prior to entry into full-rate production; andـ Upon consideration of decision to request authorization for multi-year

procurement contract

Policy Impact: Newly established position to perform required functions

TITLE 10—ARMED FORCES

• Subtitle A—General Military Law • Subtitle B—Army • Subtitle C—Navy and Marine Corps • Subtitle D—Air Force • Subtitle E—Reserve Components

TITLE 10 > Subtitle A > PART IV

• PART IV—SERVICE, SUPPLY, AND PROCUREMENT

• TITLE 10 > Subtitle A > PART IV > CHAPTER 144• CHAPTER 144—MAJOR DEFENSE

ACQUISITION PROGRAMS

Unit Cost Reports• § 2430. Major defense acquisition program defined • § 2430a. Major subprograms • § 2431. Weapons development and procurement schedules § 2432.

Selected Acquisition Reports • § 2433. Unit cost reports • § 2434. Independent cost estimates; operational manpower requirements • § 2435. Baseline description • § 2436. Major defense acquisition programs: incentive program for

contractors to purchase capital assets manufactured in United States • § 2437. Development of major defense acquisition programs: sustainment

of system to be replaced • [§§ 2438, 2439. Repealed.] • § 2440. Technology and industrial base plans

Nunn-McCurdy

• § 2433(e)(2). If the program acquisition unit cost or procurement unit cost of a major defense acquisition program or designated major subprogram (as determined by the Secretary under subsection (d)) increases by a percentage equal to or greater than the critical cost growth threshold for the program or subprogram, the Secretary of Defense, after consultation with the Joint Requirements Oversight Council regarding program requirements, shall carry out an assessment— (A)

• and state that, (B)– (i) such acquisition program is essential to the national security; – (ii) there are no alternatives to such acquisition program which will provide equal or

greater military capability at less cost; – (iii) the new estimates of the program acquisition unit cost or procurement unit cost are

reasonable; and – (iv) the management structure for the acquisition program is adequate to manage and

control program acquisition unit cost or procurement unit cost;

The Panel’s approach is to expand the mandate of the Office of Performance Assessment and Root Cause Analysis (PARCA) to serve as a performance assessment function for the acquisition system. PARCA would track organizations throughout the defense acquisition system in meeting pre-negotiated goals for acquisition performance. Organizations would see real consequences for both success and failure in meeting their goals. The intent of the performance metrics and performance management recommendations in this report is not to create new layers of bureaucracy. If the performance management measures envisioned in the Panel’s recommendations are implemented properly, program managers could be freed from certain reporting responsibilities specific to their individual program.

HOUSE ARMED SERVICES COMMITTEE PANEL ON DEFENSE ACQUISTION REFORM FINDINGS AND

RECOMMENDATIONS March 23, 2010

The IMPROVE Acquisition Act, 14 Apr 10The IMPROVE Acquisition Act, H.R. 5013, would implement recommendations of the House Armed Services Committee Panel on Defense Acquisition Reform. The panel issued its final report in March after a year-long comprehensive review of how Defense buys goods and services. “When you consider that 60 percent of the Pentagon's procurement dollars are for services contracts alone, the legislation we are introducing today has the potential to affect major changes at the Department of Defense and save billions of taxpayer dollars," said Rep Howard McKeon, R-Calif.

Among other things, the bill would require the department to revamp the requirements process, apply performance management to the defense acquisition system and create metrics supported by goals and standards. The newly created Office of Performance Assessment and Root Cause Analysis would be responsible for assessing compliance with the metrics.

Estimates -v- Reality

?

8 Step Problem Solving Process1. Clarify and Validate the Problem2. Break Down the Problem and Identify Performance Gaps

(Observe)3. Set Improvement Target4. Determine Root Cause (Orient)5. Develop Countermeasures (Decide) 6. See Countermeasures Through 7. Confirm Results and Process8. Standardize Successful Processes (Act).

Establishing a Culture of AccountabilityCNA Independent Assessment:

Air Force Acquisition Return to Excellence, Feb 2009

Weapons DevelopmentWhere do we draw the bullseye?

?

Winnie The Pooh on Process Improvement

Here is Edward Bear, coming downstairs now, bump, bump, bump, on the back of his head, behind Christopher Robin. It is,

as far as he knows, the only way of coming downstairs, but sometimes he feels that

there really is another way, if only he could stop bumping for a moment and think of it.

-- Winnie The Pooh by A A Milne

Why?• Personal -- Money, sex, power, envy, greed &

sloth, a sense of entitlement?• Professional

– Misplaced loyalty to the mission? Group think?

Defending the Chain of Command

• “Our role is to provide Ground Truth, so leaders can make an informed decision”

• Before we “ . . . convince ourselves we can do it”

• Lt Gen Ted Bowlds

The Danger of Compromise

Ethical Bullseye

The Danger of Compromise

Ethical Bullseye

Initial Decision

The Danger of Compromise

Ethical Bullseye

Initial Decision

NEW AIMING RING!

The Danger of Compromise

2nd generation decision:Subordinate’s action

Ethical Bullseye

Initial Decision

The Danger of Compromise

2nd generation decision:Subordinate’s action

Ethical Bullseye

Initial DecisionACTUAL AIMING RING

How Do We Mentor Ethics?

1. Discuss Ethical Concepts2. Be Vigilant for Warning Signs3. Correct Misconduct

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

85

Top 11 List - Rationalization1. What’s the worst thing they can do to us2. That’s the way we’ve been doing it for years3. Don’t ask a question if you can’t stand the answer4. Not enough time for a legal review5. It’s a stupid rule6. Boss said to make it happen7. Warfighter needs it now8. I saw someone else do it that way9. No harm, no foul10.We’re thinking outside the box11.It’s for a good cause

Rationalized Standards Jeopardize Core Values & Fiscal Integrity

Adapt to Decision Support

• Get involved earlier in the decision process– Give advice BEFORE the decision is made, and --“It must be brilliant because the Boss likes it”

• Use more math, law, policy, objective analysis• Accept more Risk and Uncertainty• Do more public speaking

– A little bit scarier than death to many

Estimates -v- Reality

Questions?

Sources• Risk, Uncertainty and Trouble

• Nov/Dec 08 Defense AT&L Magazine– http://www.dau.mil/pubscats/PubsCats/atl/new/nov-dec08.

pdf

• The National Estimator, Fall 2008• Air Force Comptroller, Summer 2008

• Looking at the Root Cause of Problems• May/Jun 08 Defense AT&L Magazine

– http://www.dau.mil/pubscats/PubsCats/atl/new/may-june08.pdf

• Army History of Acquisition

• Adapting to Decision Support• Armed Forces Comptroller, Spring 2009

Cheyenne Mountain revisited

• NORAD Mission Uncertainty• AAFES in the Mountain• Long Memories

Range of RDT&E EstimatesBY$M

ACAT I Threshold ($BY) = $428M

NewESTIMATE

Original Estimate

BaselineDEVELOPMENT PHASE

StatisticsCalculated with 500 iterations

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0 200 400 600 800 1000

BY2008 $M

Co

nfi

den

ce L

evel

(C

DF

)

Baseline (cdf) Point Estimate Mean

50% Confidence Level 80% Confidence Level Budget

ACAT I

Original EstimateContractor RFI ROMs

CONTRACTORRFI ROMS

91

Narrow, Normal, Certain

pdf

Cumulative Density Function

Narrow, Normal, Certain

Not so Much

GAO-08-619, July 2008

• A Knowledge-Based Funding Approach Could Improve Major Weapon System Program Outcomes

• DOD often does not commit full funding to develop its major weapon systems when they are initiated, despite the department’s policy to do so. For a majority of the weapon system programs GAO reviewed, costs have exceeded the funding levels initially planned for and reflected in the Future Years Defense Program (FYDP)—DOD’s investment strategy. To compensate for these shortfalls, DOD makes unplanned and inefficient funding adjustments, like moving money from one program to another, deferring costs into the future, or reducing procurement quantities.

GAO-08-619, July 2008

• DOD’s flawed funding process is largely driven by decision makers’ willingness to accept unrealistic cost estimates and DOD’s commitment to more programs than it can support. DOD often underestimates development costs—due in part to a lack of knowledge and optimistic assumptions about requirements and critical technologies. At the same time, DOD’s continued failure to balance its needs with available resources promotes unhealthy competition among programs for funding. This creates incentives for service and program officials to establish requirements that make their particular weapon systems stand out, with less consideration of the resources needed to develop them. Ultimately, DOD tends to push the need for funding to the future rather than limit program length or adjust requirements.

• Inaccurate cost estimates are often the result of limited knowledge about requirements and technologies. Our best practices work has shown that conducting early disciplined analysis, such as systems engineering, builds knowledge that enables a developer to identify and resolve gaps between requirements and available resources before beginning product development.

• The successful commercial companies that GAO has previously reviewed achieve adequate and stable funding for product development programs by following a disciplined, knowledge-based approach to estimating program costs; using manageable development cycles to increase the predictability of funding needs and the likelihood of program success; and using portfolio management practices to make decisions about which programs to pursue. Once programs are approved, these companies firmly commit to fully fund them.

• Cost estimates that lack knowledge and rely heavily on assumptions have inherently high levels of risk and uncertainty. Conducting quantitative risk and uncertainty analysis provides a way to assess the variability in an estimate. Using this type of analysis, cost estimators can model such effects as a schedule slipping or a key technology failing to materialize, thereby identifying a range of likely costs around an estimate. Presenting decision makers with the range of likely costs around an estimate provides insight into the amount of cost, schedule, and technical risks they are being asked to accept and conveys a level of confidence associated with achieving the proposed estimate. A range of costs also provides a basis for deciding how much funding a program needs to be successful.

• In developing cost estimates, the services and the CAIG often do not present decision makers with the range of costs around an estimate. Instead, they present a single, or point, estimate as the most probable cost, which OSD expects to be a 50 percent chance that actual program costs will be at or below the estimated value. Several recent studies have questioned DOD’s approach and recommended establishing estimates with higher levels of confidence under the assumption that the result will be more realistic estimates.

• For example, the Defense Acquisition Performance Assessment Panel recommended adjusting program cost estimates to reflect “high confidence”—defined as an 80 percent chance of completing development at or below the estimated amount. Requiring a higher confidence level could provide a better basis for determining program funding, but only if the quality of the cost estimate is sound and the underlying risk and uncertainty associated with the estimate are accurately captured. If the quality of the estimate is poor to begin with, simply applying a higher confidence level to the estimate will not make it any more realistic

Findings #1 & #2

• Develop and implement a strategy to bring the department’s current portfolio into balance by aligning the number of programs and the cost and schedule of those programs with available resources.

• Require that all new programs have manageable development cycles, realistic cost estimates, and have planned and programmed full funding for the entire development cycle.

Findings #3

• Require all cost estimates submitted for funding a program at milestone decisions to be reported as a range of likely costs and reflect the associated levels of risk and uncertainty. At Milestone A, require estimates that allow for a wide range of likely costs. At Milestone B, require estimates that, based on knowledge gained, are more precise—in line with best practice standards.

GAO response to DoD Reply• In written comments on a draft of this report, DOD partially

concurred with our first and second recommendations and non-concurred with the third. DOD’s partial concurrences are rooted in the belief that its current policies and initiatives address our recommendations. We agree that aspects of DOD’s current policies appear consistent with our recommendations, and the initiatives could contribute to better program outcomes if implemented appropriately. However, we have found no evidence of widespread adoption of these policies or any other process that would better ensure adequate funding for DOD’s major weapon system acquisition programs and increase the likelihood of achieving successful outcomes.

More Chapter 5Reform How DoD Buys

• Our objective is to achieve predictable cost, schedule and performance outcomes based on mature, demonstrated technologies and realistic cost estimates. We are also implementing initiatives that will increase the numbers and capabilities of the acquisition workforce, improve funding stability, enhance the source selection process, and improve contract execution. Our intent is to provide the warfighter with world class capability while being good stewards of the taxpayer dollar.

8 Step Problem Solving Process1. The OODA Loop (Observe, Orient, Decide, Act) was originated in the 1950's

by Col. John Boyd. It is an objective description of the decision making process. Due to an emphasis on the infinitely repeating nature of decision making, the OODA Loop is an excellent match for the AFSO21 principal of continuous improvement. The eight steps of the Problem Solving Model fit within the frame of the OODA Loop. The eight steps are (1) Clarify and Validate the Problem, (2) Break Down the Problem and Identify Performance Gaps (Observe), (3) Set Improvement Target, (4)Determine Root Cause (Orient), (5) Develop Countermeasures (Decide), (6) See Countermeasures Through, (7) Confirm Results and Process, and (8) Standardize Successful Processes (Act). Following the eight steps will ensure that actions lead to the desired results with an absolute minimum of wasted effort. Following the steps will also ensure the results are aligned with the needs of the organization.

Considers the underlying cause or causes for shortcomings in cost, schedule, and performance including the role, if any, of:

ـ Unrealistic performance expectations;

ـ Unrealistic baseline estimates for cost and schedule;

ـ Immature technologies or excessive manufacturing or integration risk;

ـ Unanticipated design, engineering, manufacturing, or integration issues arising during program performance;

ـ Changes in procurement quantities;

ـ Inadequate program funding or funding instability;

ـ Poor performance by government or contractor personnel responsible for program management;

ـ or any other matters.

Implementation of WSARARoot Cause Analysis

Weapons Systems Acquisition Reform Act

“The key to successful acquisition programs isgetting things right from the start with sound

systems engineering, cost estimating, anddevelopmental testing early in the program cycle.

The bill that we are introducing today will require theDepartment of Defense to take the steps needed toput major defense acquisition programs on a sound

footing from the outset. If these changes aresuccessfully implemented, they should help ouracquisition programs avoid future cost overruns,

schedule delays, and performance problems.”

–Senator Carl Levin, Chairman, Senate Armed Services Committee

Reducing Cycle Time

“The only way to reduce cycle time is to know what you are going to do before you start”

-- Any parent who ever put together anything on Christmas morning while their child waited for them to finish before they could start playing.

Implementation of WSARACritical Cost Growth (1)

DTM contains policy implementing new 10 USC 2433a, Critical Cost Growth of MDAPs, that amends 10 USC 2433, Unit Cost Reports, and supersedes all previous USD(AT&L) policies addressing actions that must be taken following critical cost growth of a MDAP or designated subprogram

•PM shall notify the CAE immediately, whenever there is a reasonable cause to believe that the current estimate of either the program acquisition unit cost (PAUC) or average procurement unit cost (APUC) of a MDAP or designated subprogram (in base-year dollars) has increased by 25 percent (or more) over the PAUC or APUC objective of the currently approved APB estimate, or 50 percent (or more) over the PAUC or APUC of the original APB estimate.

•If the CAE determines that there is an increase in the current estimate of the PAUC or APUC of at least 25 percent over the PAUC or APUC objective of the currently approved APB, or 50 percent over the PAUC or APUC of the original APB, the CAE shall inform the USD(AT&L) and the Head of the DoD Component.

Implementation of WSARACritical Cost Growth (2)

• If the Component Head subsequently determines that there is, in fact, an increase in the current estimate of the PAUC or APUC of at least 25 percent over the currently approved APB, or 50 percent over the PAUC or APUC of the original APB, the Head of the DoD Component shall notify Congress, in writing, of the determination of critical cost growth and the increase with respect to the program or subprogram concerned.

• The notification shall be not later than 45 days after the end of the quarter, in the case of a quarterly report; or not later than 45 days after the date of the report, in the case of an out-of-cycle report based on critical change occurring between quarters. In either case, notification shall include the date that the Head of the DoD Component made the determination.

• In addition, the Component Head shall submit an SAR for either the fiscal year quarter ending on or after the determination date, or for the fiscal year quarter that immediately precedes the fiscal year quarter ending on or after the determination date. This SAR shall contain the additional critical cost growth-related information.

Implementation of WSARACritical Cost Growth (3)

• The USD(AT&L), after consultation with the JROC, shall determine the root cause or causes of the critical cost growth in accordance with applicable statutory requirements and DoD policies, procedures, and guidance based upon the root cause analysis conducted by the senior official for PARCA; and in consultation with the DCAPE, shall carry out an assessment of:

a. The projected cost of completing the program if current requirements are not modified.

b. The projected cost of completing the program based on reasonable

modification of such requirements.

c. The rough order of magnitude of the costs of any reasonable alternative system or capability.

d. The need to reduce funding for other programs due to the growth in cost of the program.

Implementation of WSARACritical Cost Growth (4)

• After conducting the reassessment, the USD(AT&L) shall terminate the program unless the USD(AT&L) submits a written certification to Congress before the end of the 60-day period beginning on the day the SAR containing the unit cost information is required to be submitted to Congress. The certification must state:

a. The continuation of the program is essential to the national security.

b. There are no alternatives to the program that will provide acceptable capability to meet the joint military requirement (as defined in section l8l(g)((1) of Title 10, U.S.C) at less cost.

c. The new estimates of the PAVC or APUC have been determined by the DCAPE, to be reasonable.

d. The program is a higher priority than programs whose funding must be reduced to accommodate the growth in cost of the program.

e. The management structure for the program is adequate to manage and control PAUC or APUC.

Implementation of WSARACritical Cost Growth (5)

• The written certification shall be accompanied by a report presenting the root cause analysis and assessment and basis for each determination made in accordance with the five certification criteria listed previously

• If the USD(AT&L) elects NOT to terminate a MDAP that has experienced critical cost growth, the Secretary of Defense shall:

a. Restructure the program in a manner that addresses the root cause or causes of the critical cost growth, and ensures that the program has an appropriate management structure as set forth in the written certification;

b. Rescind the most recent milestone approval for the program or designated subprograms and withdraw any associated certification(s) pursuant to section 2366a or 2366b.

c. Require a new milestone approval for the program or designated subprograms before taking any contract action to enter a new contract, exercise an option under an existing contract, or otherwise extend the scope of an existing contract under the program, except to the extent determined necessary by the MDA, on a non-delegable basis, to ensure that the program can be restructured as intended by the Secretary of Defense without unnecessarily wasting resources.

d. Include in the report a description of all funding changes made as a result of the growth in cost of the program, including reductions made in funding for other programs to accommodate such cost growth. (The report specified here is the first SAR for the program submitted after the President submits a budget in the calendar year following the year in which the program was restructured.)

Implementation of WSARACritical Cost Growth (6)

• Additionally, for each MDAP that has exceeded the critical cost thresholds, but has not been terminated, the senior official for PARCA shall conduct semi-annual reviews until 1 year after the date a new milestone approval is received. The senior official for PARCA, shall report the results of the semi-annual reviews to the USD(AT&L) and summarize the results in the Director's next annual report.

• If a MDAP is terminated after experiencing a critical cost breach, the USD(AT&L) shall submit to Congress a written report with the following information:

a. An explanation of the reasons for terminating the program.

b. The alternatives considered to address any problems in the program.

c. The course the Department of Defense plans to pursue to meet any continuing joint military requirements otherwise intended to be met by the program.