Slide 1

Example 2012 Lockheed Martin Corporation. All Rights Reserved. October 2012 Proxy Estimation Costing for Systems (PECS) Reggie Cole Lockheed Martin Senior Fellow

Slide 2

2 Example 2012 Lockheed Martin Corporation. All Rights Reserved. Discussion Topics Why Do We Need Yet Another Cost Model? The gap in early-stage system cost modeling Systems Engineering Effort as a Proxy Estimator for System Cost And the role of COSYSMO is arriving at this proxy estimate Proxy Estimation Costing for Systems (PECS) Derivation of the PECS Model The PECS modeling approach Case Study for Affordability Analysis Using the PECS Model The real power of the PECS model

Slide 3

3 Example 2012 Lockheed Martin Corporation. All Rights Reserved. Cost Modeling Needs Change Over Time in Terms of Speed and Accuracy So Does Solution Information Problem-Space Description Cost Estimate 25% High-Level Solution Description Cost Estimate 10% Detailed Solution Description Cost Estimate 5% High-Level Solution Assumptions Cost Estimate 20% Increasing Effort and Cost-Modeling Expertise Increasingly Refined Information About the Solution Increasingly Refined Cost Estimate Increasingly Refined Solution We Have a Good Selection of Tools for Late-Stage Cost Modeling We Have Gaps in Early-Stage Cost Modeling

Slide 4

4 Example 2012 Lockheed Martin Corporation. All Rights Reserved. Systems Engineering Effort as a Proxy Measure of Overall System Size and Complexity Proxy Measures Proxy measures are used when you cannot directly measure what you want to measure and when an indirect measure provides sufficient insight Proxy measures are often used in clinical studies since direct measurement is often infeasible or can even alter the outcome It is not always possible to directly measure what you want to measure or directly estimate what you want to estimate System Engineering Effort is a Proxy Measure for System Cost There is strong evidence for the link between systems engineering effort and program cost dating back to a NASA study in the 1980s The optimal relationship between systems engineering effort and overall program cost is 10% - 15% Industry has long used a parametric relationship between software cost and systems engineering cost for software-intensive systems Systems engineering effort can be an effective proxy measure for overall system cost H. Dickinson, S. Hrisos, M. Eccles, J. Francis, M. Johnston, Statistical Considerations in a Systematic Review of Proxy Measures of Clinical Behaviour, Implementation Science, 2010 E. Honour, Understanding the Value of Systems Engineering, INCOSE, 2004

Slide 5

5 Example 2012 Lockheed Martin Corporation. All Rights Reserved. COSYSMO 2.0 Model Parameters Provide a Rich Assessment of System Size, Complexity and Reuse Number of System Requirements Number of Major System Interfaces Number of Critical Algorithms Number of Operational Scenarios Size Drivers Requirements Understanding Architecture Understanding Level of Service Requirements Migration Complexity Technology Risk Level of Documentation Required Diversity of Installed Platforms Level of Design Recursion Stakeholder Team Cohesion Personnel / Team Capability Personnel Experience / Continuity Process Capability Multisite Coordination Level of Tool Support Cost Drivers Managed Elements Adopted Elements Deleted Elements Modified Elements New Elements Reuse Factors Initial Estimate of System Size Scaled Estimate of System Size Consolidated Cost Driver Factor Estimate of Systems Engineering EffortAlso a Biased Proxy Estimator for System ScopeAnd System Cost

Slide 6

6 Example 2012 Lockheed Martin Corporation. All Rights Reserved. An Approach for De-Biasing the Proxy Estimator Relationship Between SE Effort and Total Effort NASA data supports a 10%-15% optimal allocation of systems engineering effort as a portion of overall program effort W. Gruhl, Lessons Learned, Cost/Schedule Assessment Guide, Internal Presentation, NASA Comptrollers Office, 1992 E. Honour, Understanding the Value of Systems Engineering, INCOSE, 2004 INCOSE study on the value of systems engineering also supports a 10%-15% optimal allocation of systems engineering as a portion of overall program effort

Slide 7

7 Example 2012 Lockheed Martin Corporation. All Rights Reserved. The PECS Cost Function VariableTypeDescription COSYSMO Calibration FactorDeterministic Scalar ValueOrganization-specific calibration factor Effort Conversion FactorTriangular Distributed Random VariableThree-point estimate of factor to convert SE effort to total program effort (nominally 0.08, 0.12 and 0.16) SE EffortTriangular Distributed Random VariableThree-point estimate for SE effort, generated using COSYSMO Labor RateTriangular Distributed Random VariableThree-point estimate for composite labor rate Material CostsTriangular Distributed Random VariableThree-point estimate for material costs Travel CostsTriangular Distributed Random VariableThree-point estimate for travel costs This Model is Well Positioned for Monte Carlo Analysis

Slide 8

8 Example 2012 Lockheed Martin Corporation. All Rights Reserved. The PECS Model Putting It All Together Size Drivers (Problem Space) Customer Requirements System Interfaces Major Algorithms Operational Scenarios Complexity Drivers (Problem/Solution) Requirements Understanding Architecture Understanding Level of Service Requirements Migration Complexity Technology Risk Documentation Needs Installations/Platform Diversity Levels of Recursion in the Design Stakeholder Team Cohesion Personnel/Team Capability Personnel Experience/Continuity Process Capability Multisite Coordination Tool Support Reuse Factors (Solution Space) New Modified Deleted Adopted Managed SE Effort is an estimator for total system costbut it is a biased estimator Estimator Bias Function is Based on the Well-Established Relationship Between SE Effort and Overall Program Effort Proxy Estimation Costing for Systems (PECS) Estimator De-Biasing Monte Carlo Analysis of System Cost Three different COSYMO scenarios optimistic, expected & pessimistic provide the basis for the Monte Carlo analysis of system cost

Slide 9

9 Example 2012 Lockheed Martin Corporation. All Rights Reserved. Case Study The COSYSMO Scenarios The case study is based on a large C2 system. Initially developed 20 years ago, the system was unprecedented. Twenty years later, a replacement system is needed. While the initial development was unprecedented, the replacement system is not, which drives down the size drivers (through reuse) and cost drivers. The case study looks at three cost scenarios: Case 1 The original unprecedented system (for calibration purposes) Case 2 Replacement system (as a new development) Case 3 Replacement system (as a largely COTS/GOTS approach) COSYSMO Scenarios for PECS Three Scenarios for Each Case

Slide 10

10 Example 2012 Lockheed Martin Corporation. All Rights Reserved. Case Study The Monte Carlo Analysis Case 1 Average 80/20 Cost = $1.9B Used as a calibration point for the model Case 2 Average 80/20 Cost = $77M Initial Solution for Replacement System Case 3 Average 80/20 Cost = $30M More Affordable Solution, Based on COTS/GOTS Solution The PECS Model Enables Rapid Turn-Around Analysis of Alternatives and Should Cost Analysis

Slide 11

11 Example 2012 Lockheed Martin Corporation. All Rights Reserved. Conclusion The PECS Model is Based on Well-Established Approaches COSYSMO provides the basis for estimation of systems engineering effort and a biased proxy estimator for overall system cost There is a well-established relationship between systems engineering effort and overall effort used to de-bias the COSYSMO-modeled effort Monte Carlo analysis is a well-established technique for cost modeling The PECS Model Can Improve System Cost Modeling The PECS Model occupies an important niche fully parametric system cost modeling in the early stages of system definition The PECS Model can serve as a powerful affordability analysis tool supporting rapid-turnaround analysis of alternatives Butthe PECS Model is not a replacement for existing models Next Steps Broader validation of the model Cross-industry review of the model

Slide 12

12 Example 2012 Lockheed Martin Corporation. All Rights Reserved.