practical guidance to strengthen facility estimates - nsf · practical guidance to strengthen...
TRANSCRIPT
Practical Guidance to Strengthen Facility
EstimatesNSF Large Facilities Workshop Jason Lee, GAO
May 1-3, 2017 Kevin Porter, NSF
Erik Nylund, CH/KGS
1
Purpose
Strengthen Estimates – Construction, Operations & Maintenance
Highlight Requirements and Best Practices
Emphasize Importance of Cost Estimating Plans, Basis of Estimate
Provide Practical Examples
Answer Questions
2
Overview
Background – AICA, LFM, Definitions
GAO
12 Steps & 4 Characteristics
Cost Estimating Plans (CEP) Examples
Basis of Estimate (BOE) Examples
Independent Contractor – Kforce Government Solutions & Crowe Horwath
Role doing independent cost assessments for NSF per GAO
NSF CEP & BOE Examples
Misc Items
3
American Innovation & Competitiveness Act- Oversight of NSF Large Facilities
“shall strengthen oversight and accountability over the full life-cycle… in order to maximize research investment”
“ensure that policies for estimating and managing costs and schedules are consistent with the best practices described in the Government Accountability Office Cost Estimating and Assessment Guide”
“require that any pre-award analysis of a major multi-user research facility project includes the development and consideration of the full life-cyclecost”
4
AICA - NSF - Construction
“may not approve or execute any agreement to start construction on any proposed major multi-user research facility project unless…
“external analysis of the proposed budget has been conducted”
“independent cost estimate (ICE) of the construction of the project has been conducted using the same detailed technical information as the project”
“considered the analyses… and the independent cost estimate… and resolved any major issues”
5
AICA - NSF - Operations
“shall require an independent cost analysis (ICA) of the operational proposal for each major multi-user research facility”
6
Large Facilities Manual – Section 4.2
Strengthen Estimates & Clarify NSF expectations
Implement AICA Requirements
Supplement GAO Cost Guide w/ NSF specific info
Clarify NSF Cost Analysis process
Duration & NSF participants
Conducted at each Design Stage-Gate Review for Construction
Conducted at proposal submission for Operations
ICE and ICA inform NSF Cost Analysis > They DO NOT replace it!
Correct information/detail into estimate at proposal submission
Reduce burden & frustration for Recipient & NSF
Reduce time to award
7
Cost Estimating Plan (CEP)
AACE International – Recommended Practice No. 36R-08
“establish and communicate how the preparation, development, review and approval of the estimate will be completed”
LFM – includes NSF specific needs
“A plan describing how the cost estimating guidance in this manual will be implemented, how the cost estimate will evolve over time, and how the “Cost Model Data Set” will meet the various needs of the project. The CEP should typically include a narrative and sufficient detail explaining the ground rules and assumptions, roles and responsibilities, practices, systems, and calculations used to develop the cost estimate. “
8
Basis of Estimate (BOE)
AACE International – Recommended Practice No. 34R-05
“Written documentation that describes how an estimate, schedule, or other plan component was developed and defines the information used in support of development. A basis document commonly includes, but is not limited to, a description of the scope included, methodologies used, references and defining deliverables used, assumptions and exclusions made, clarifications, adjustments, and some indication of the level of uncertainty”
Project Management Institute
“Supporting documentation outlining the details used in establishing project estimates such as assumptions, constraints, level of detail, ranges, and confidence levels.”
LFM
PMI definition
Additional guidance on our expectations, level of detail, acceptable justifications
9
Known as the investigative arm of Congress, GAO exists to support Congress in meeting its constitutional responsibilities. To that end, GAO works to
• Help improve the performance of federal government
• Ensure government agencies and programs are accountable to the American people
• Examine the use of public funds, and
• Evaluate federal programs by providing analyses and recommendations to help Congress make informed oversight and funding decisions
Page 10
The Role of GAO in Government
• Drafted 2005-2007, published in 2009
• Outlines GAO’s criteria for assessing cost estimates during audits
• Contains 20 chapters with supporting appendixes
• Chapters 1-17: developing credible cost estimates and the 12-step cost estimating process for developing high quality cost estimates
• Chapters 18-20 address managing program costs once a contract has been awarded and discuss Earned Value and risk management
• Also provides case studies of prior GAO audits to show typical findings related to the cost estimating process
Page 11
Cost Estimating and Assessment Guide
12
A Reliable Process for Credible Cost Estimates
Page 13
Characteristics of Reliable Estimates
13
Comprehensive
• Develop the estimating plan
• Determine the estimating approach
Accurate
• Develop the point estimate
• Compare the point estimate to an independent estimate
• Update the estimate with actual costs
Credible
• Create an independent cost estimate
• Conduct sensitivity analysis
• Conduct risk and uncertainty analysis
Well Documented
• Define the program
• Identify ground rules and assumptions
• Obtain data• Document the
estimate• Present estimate
to management
Are all costs included?
Is the estimate unbiased?
What is the uncertainty?
Can the estimate be recreated?
A comprehensive estimate• Includes both government and contractor costs of
the program
• Covers the full life cycle of the program, from inception of the program through retirement (not just the Construction Stage)
• Applies to both FAR-based contracts and Cooperative Agreements
Comprehensive Cost Estimates
California High-Speed Passenger Rail: Project Estimates Could Be Improved to Better Inform Future Decisions – GAO-13-304
“While the O&M estimate includes common elements for administration and support costs, the O&M WBS is greatly simplified. As a consequence, up to two-thirds of O&M costs are collected in a single cost element.”
DHS and GSA Need to Strengthen the Management of DHS Headquarters Consolidation – GAO-14-648
“GAO found that the 2013 cost estimate…does not include a life-cycle cost analysis of the project, including the cost of operations and maintenance.”
VA Construction: Improved Processes Needed to Monitor Contract Modifications, Develop Schedules, and Estimate Costs – GAO-17-70
“All applicable costs for the construction contract appear to be included in the cost estimate…. However, VA’s $341-million cost estimate for activating the Denver facility is not well supported.”
Comprehensive Cost Estimates - Examples
16
A Focus on Step 2 and Step 10
1. A written study plan• Determines the estimating team’s composition
• Identifies subject matter experts
• Includes a schedule for the cost estimating effort
2. Team includes experienced and trained cost estimators
3. Estimating team is from a centralized office
Not meeting Step 2 criteria is typically a cause for why other best practices are not met
Step 2: Develop the Estimating Plan
2020 Census: Census Bureau Needs to Improve Its Life-Cycle Cost Estimating Process – GAO-16-628
“We found the Bureau had little planning information among its documents supporting its cost estimate. Early fundamental planning and guidance documents such as general policies and procedures for cost estimation…can contribute to consistent control over the process used to develop a cost estimate and help ensure that desired standards and practices are implemented.”
“Eight years later, the absence of guidance to control the cost estimation process persists. Investment in the planning documents to help control and support cost estimation early in the estimation cycle, such as…guidance on key steps and process flows [and] assignment of responsibilities…can help institutionalize practices and ensure that otherwise disparate parties in the process operate consistently.”
Step 2: Census Bureau Estimate Plans
Step 10: Document the Estimate
Cover Page and Introductio
n
Executive Summary
Introduction
System Description
Program Inputs
Methodology by Cost Element
Data by Cost
ElementNarrative
text
Cost tables Sensitivity analysis
Uncertainty analysis
Management approval
Tracked updates
Contingency
derivation
Good documentation should describe the cost estimating process, data sources, and methodologies.
Checked Baggage Screening: TSA Has Deployed Optimal Systems at the Majority of TSA-Regulated Airports, but Could Strengthen Cost Estimates – GAO-12-266
“TSA did not adequately document many assumptions or methodologies underlying its cost model to the extent that would allow someone unfamiliar with the cost estimate, using only the available documentation, to easily re-create the estimate.”
Step 10: Documentation for a “Never-Ending” Program
NSF – High Quality CEP & BOE Example –Construction
Antarctic Infrastructure Modernization for Science (AIMS) Successfully completed Preliminary Design Review (PDR)
21
AIMS – CEP
22
AIMS - CEP
23
AIMS – BOE* - WBS 5.4.1
*BOE organized per AACE International Recommended Practice No. 34R-05. (2010)
AIMS – BOE – WBS 5.4.1
AIMS – BOE – WBS 5.4.1
NSF – High Quality CEP & BOE Example –O&M
National Ecological Observatory Network (NEON) Some elements under construction
Some elements in operation
27
NEON CEP & BOE Example – O&M
NEON
28
NEON CEP & BOE Example – O&M (continued)
29
NEON CEP & BOE Example – O&M (continued)
30
NSF - CEP & BOE Examples – Summary Leading Practices/Observation
Complete BOE document with good project and scope description
Clear linkages from BOE to Cost Book (via WBS)
Thorough and well-documented assumptions for cost drivers, cost estimating methods, and data sources
Integrated cost model which includes build up from lowest to highest levels of the WBS
Use of a “quality standard” as though cost estimate will be subject to third party review and such that third party can replicate
Supportable cost escalations; and indirect/fringe/overhead rates
Sufficient documentation of risk and sensitivity analyses; discussion of cost estimate limitations
31
NSF - CEP & BOE Example – O&M
NSF is not prescriptive – Tailor to your facility!!
Academic Research Fleet
Ship Operations format established by OCE is both the CEP and the BOE
“WBS” elements are the various Level of Effort cost categories defined by OCE
Suitable for NSF to conduct a cost analysis (mainly Program)
32
Can Contingency be used during Operations Stage?
Yes (per LFM 4.2.6)
Must comply with paragraph 200.433 of the Uniform Guidance (Held separately to manage project risk)
Must comply with LFM - formal risk assessment, Risk Management Plan, Risk Register, contingency calculation
Approved by NSF
Generally for major up-grade sub-elements of Ops proposal
Example?
33
§ 200.433 Contingency provisions
(a) Contingency is that part of a budget estimate of future costs (typically of large construction projects, IT systems, or other items as approved by the Federal awarding agency) which is associated with possible events or conditions arising from causes the precise outcome of which is indeterminable at the time of estimate, and that experience shows will likely result, in aggregate, in additionalcosts for the approved activity or project. Amounts for major project scope changes, unforeseen risks, or extraordinary events may not be included.
(c) Payments made by the Federal awarding agency to the non-Federal entity's “contingency reserve” or any similar payment made for events the occurrence of which cannot be foretold with certainty as to the time or intensity, or with an assurance of their happening, are unallowable, except as noted in § 200.431 Compensation—fringe benefits regarding self-insurance, pensions, severance and post-retirement health costs and § 200.447 Insurance and indemnification.
34
Do you really mean Allowance not Contingency? Allowance*: Resources included in the basis of estimate for baseline cost estimates to
cover the cost of known but as-of-yet undefined details or requirements for an individual WBS element. May be used when the level of project definition may not enable certain costs to be estimated definitively or times when it is simply not cost effective to quantify and cost every small item included with the WBS element, but reliable correlations are available.
Past project experiences, demonstrated statistical correlation Most likely costs
Examples:
Preliminary Design – design or material take-off allowance
Small Items - Bolts, structural steel connections
Hand excavation/backfill (vs. machine excavation/backfill)
35
*Adapted from AACE International
