oscam: 20 years of making better operating and...

ENERGY | GOVERNMENT | DEFENCE ©2016 of Decision Analysis Services Ltd

[email protected] ENERGY | GOVERNMENT | DEFENCE

This document is Copyright ©2016 Decision Analysis Services Ltd. Its contents wholly or in part shall not be communicated or copied by any means whatsoever to any third party, individual or organisation or government without consent of Decision Analysis Services Ltd.

OSCAM:20 years of making better operating and support estimates of fleets of ships and aircraft for the US Department of Defense

Stephen Curram, Decision Analysis Services Ltd.

Brittany Basilone, Naval Surface Warfare Center

UK Chapter of the System Dynamics Society2016 Annual Conference

Shard, London14 April 2016


www.das-ltd.co.uk

Contents

2

•What is OSCAM?

•The use of OSCAM in US DoD

•A brief history of OSCAM

•The user experience of OSCAM

• Lessons learned over the 20 years of OSCAM lifetime

•Discussion


www.das-ltd.co.uk

What is OSCAM?

3

• Operating & Support Cost Analysis Model

• There are several OSCAMs in use by US DoD*:- The US Navy uses OSCAM Ship, OSCAM Air and OSCAM Sys

- OSCAM JSF is used by the F-35 Lightning II Program Office

• It is a software tool

• It is used to generate operating and support costestimates for equipment programmes

• It captures the business processes involved in operating and supporting a class of military platforms or systems

• At its core is a System Dynamics simulation model that steps through time (in months) and determines the costs generated each month based on the status and activities of the equipment

• It has a powerful front-end for data entry, analysis of results, comparison of different scenarios and analysis of uncertainty

* US Navy versions of OSCAM are sponsored and owned by US DoD Naval Center for Cost Analysis (NCCA)


www.das-ltd.co.uk4


www.das-ltd.co.uk5


www.das-ltd.co.uk6


www.das-ltd.co.uk7


www.das-ltd.co.uk

Use of OSCAM in US DoD – example ship programmes

8

OSCAM has been used to generate milestone cost estimates for a range of US ship programmes including:

• T-AKE Solid Support Ship

• Virginia Class Block III Upgrade

• LPD

• LCS

• DDG1000

• Ohio replacement, Joint High Speed Vessel, Unmanned Naval Surface Combatant, Coast Guard National Security Cutter & Off-shore Patrol Cutter


www.das-ltd.co.uk

Use of OSCAM in US DoD – example aircraft programmes

9

… and a range of aircraft programmes, including:

• F-35 Lightning II

• V-22 Osprey

• Fire Scout MQ-8B

• P-8A Poseidon

• E-2D Advanced Hawkeye

• E/A-18G Growler


www.das-ltd.co.uk

Use of OSCAM in US DoD – types of studies

10

Milestone submissions:

• Formal stages in the procurement process that are needed to advance to the next stage of procurement

• Milestones A, B, C – each requires increasing levels of precision

• Each includes the requirement for an operating and support cost estimate over the life of the programme, broken down into cost categories

In-service annual cost submissions:

• In-service programmes are expected to provide annual operating and support cost actuals, and estimates to the end of life

Analysis of Alternatives:

• Development of operating & support cost estimates at concept phase to analyse alternative concepts (e.g. nuclear vs conventional propulsion concepts)

Bespoke Studies:

• Use in bespoke studies, e.g. analyse impact of an equipment modernisation programme on operating & support costs for the fleet


www.das-ltd.co.uk

A brief history of OSCAM

11

• First OSCAM Ship and System models developed in 1996, jointly for US DoD and UK MoD

• OSCAM Land variant developed in 1998, with some spin off models for specific vehicles

• OSCAM Air demonstrator developed in 1999. First full-scale tool developed for USAF in 2002. Adapted for US Navy aircraft in 2003. Joint Strike Fighter (F-35) version developed in 2010

• Over 550 people trained for OSCAM Ship/System

• Close to 400 people trained for OSCAM Air variants

Ship & System NCCA MOD

ADGE MOD

Land MOD

Air NCCA / NAVAIR

FASH MOD

JSF JPO JSF v2

Ship v8 NCCA

EFV DRPM EFV

USAF C-17 SPO

• Current versions: OSCAM Ship v8.2, OSCAM Air v4, OSCAM Sys v7, OSCAM JSF v2

• OSCAM Ship v8.3 and Air v4.1 are currently in development

• Plans for updates to OSCAM System and to develop a Land vehicles model for the US Marines


www.das-ltd.co.uk

The user experience (1)

12

Core requirements by cost analysts

• Produce operating and support cost estimates to agreed DoD standard reporting structure- OSCAM produces outputs in agreed reporting structure, flexibility roll-up or expand to different levels

of details in cost breakdown structure

• Must be able to withstand detailed scrutiny by US DoD authorities- US cost analysts and SMEs involved in OSCAM development process throughout, from workshops to

develop influence diagrams through to detailed testing – assures appropriate breadth, depth and flexibility included in the representation of the business processes

- Confidence in verification and validation of the model signed off by US cost analysts

- Ability of OSCAM to include recognised US Navy data sources (VAMOSC database) for OSCAM inputs

- Clear understanding of business process used by OSCAM – ability to view appropriate interactive influence diagrams by selecting any input and output in the model and pressing F1 key

- Non-cost outputs generated that show the levels of activity that drive the costs – allows understanding of why the cost outputs are being generated

• Include uncertainty analysis on inputs using probability distributions- OSCAM models have an Uncertainty Tool embedded that allows probability distributions to be used

for any input values. Repeated model runs are undertaken, sampling from the probability distributions for each run to generate S-Curves for costs


www.das-ltd.co.uk

The user experience (2)

13

Core requirements by cost analysts

• Ability of Programme Offices to undertake numerous what-if studies prior to submission- OSCAM makes it easy to edit baseline estimates to examine alternative scenarios for use and design

- OSCAM handles knock-on impacts of changes – e.g. impact of change in personnel numbers on training and facilities costs

- Turnaround time for answers to what-if questions is a couple of hours using OSCAM versus a couple of days for spreadsheet models

- Comparative graphs (for up to 20 scenarios) and Delta results (difference between two results sets) can be generated

• Ability to add detail for later milestones- OSCAM contains options for “Simplified” and “Detailed” representations. Early stage cost estimates

can use the simplified options and then transition to more detailed representations as knowledge of the equipment increases

- Can use a mixture of simplified and detailed representations within the same model

- Allows OSCAM to be adopted early and for detail to grow over time

• Sharing and handover between cost analysts- Standardised but flexible structure – allows analysts to concentrate on validity of input data rather

than trying to understand someone else’s spreadsheet

- OSCAM contains a number of audit trail features to support documentation


www.das-ltd.co.uk

The user experience (3) – “F1 Help” for interactive influence diagrams

14

Select input or output and press F1 key

Hover mouse over text on left to highlight input or output (and

associated inputs) on the influence diagram, along with appropriate

documentation


www.das-ltd.co.uk

Key lessons learned over 20 years of OSCAM development

15

OSCAM is not mandated in the US Navy but is used for the majority of ship/submarine procurement programmes, and an increasing proportion of aircraft procurements

• OSCAM has been successfully adopted in the US Navy. It is going strong after 20 years and use of OSCAM is on the increase for both ships/submarines and for aircraft

• It has taken longer for use in aircraft costing to reach critical mass, but it is reaching that point now

• OSCAM has not been widely adopted by the UK MoD, despite their involvement in the early OSCAM models

• The following slides provide a synthesis of key factors that have led to the successful adoption and longevity of OSCAM in the US. Equally some of these factors have been absent in the UK:


www.das-ltd.co.uk

1) Winning hearts and minds

16

• Ease of use is a key requirement for widespread adoption of a tool- Users must be able to do what they did before, with greater ease than before. If they can do the

task better than before then that is a benefit but is not usually sufficient in its own right

- New features must never make existing features more complicated to use

• A core group of end-users must be involved in the development of the tool- There must be a bought-in group of users that will spread the word and provide expert support to

new adopters

- Success breeds success – model users become senior staff and team managers and introduce the tool as standard practice. New analysts learn the job using the tool

• Evolution not revolution - The process of using the tool must be logical and sufficiently similar to what analysts were doing

before. Analysts can adapt their existing practices while maintaining confidence in their outputs

- Use of the greater analytical capabilities of the tool comes over time as confidence in use of the tool increases

• Transparency is important- Analysts have to be able to defend their estimates and justify them to peers and authorities.

There must be sufficient information and transparency to understand and logically justify the outcomes


www.das-ltd.co.uk

2) Data, Data, Data

17

• A recognised and reputable source of data must be available- The US Navy VAMOSC database is used to collect and disseminate operating and support costs for

in-service ships, submarines, aircraft and major sub-systems (e.g. weapons systems)

- The Naval Center for Costs Analysis (NCCA) is the sponsor for both OSCAM and VAMOSC

- Historical datasets are produced for OSCAM for in-service (and a number of retired) ships, submarines, aircraft and major systems which can be loaded directly into OSCAM. These can be used as OSCAM inputs to act as benchmarks and starting points for cost estimates for new equipment

• The data source must be able to provide data in a format that is usable for System Dynamics

- It must be possible to understand existing behaviour in terms of rates rather than just absolutes ($ per person per year, $ per steaming hour, $ per flying hour, $ per ship per year, $ per overhaul, etc.)

- Collecting historical spend in terms of accounting budgets will tell you how much has been spent, but not why

- Data collection must include both capture of costs and the non-cost activities that drive those costs

- The available VAMOSC data has influenced what is in OSCAM, however, the development of OSCAM has had a big impact on what is collected in VAMOSC


www.das-ltd.co.uk

3) Getting model granularity right, and correct behaviour for the correct reasons, is key to model longevity

18

• The underlying System Dynamics models within OSCAM are not changed very often- The OSCAM Ship and System models went through incremental changes in the early years, partly

due to constrained budgets and partly though learning, apart from that:- The OSCAM Ship SD model has had one major revamp (OSCAM Ship v8, released in 2010)

- The OSCAM Air SD model has had one major revamp (OSCAM Air v4, released in 2013)

• The level of granularity in the models is such that they capture the key relationships between inputs and outputs while avoiding excessive detail that could cause models to lose relevancy quickly

- Helped by the need to be able to represent a range of types of ships/submarines classes and aircraft types with one Ships model and one aircraft model

- e.g. OSCAM USAF was developed for the C17 programme office (a large cargo aircraft) but personnel from the F22 programme (the world’s most advanced fighter aircraft) were involved in the design process to ensure the model was sufficiently generic

• Modelling correct behaviour for the correct reasons (rather than good enough behaviour for an acceptable range of input values) contributes to model longevity

• Even where models are re-vamped, careful modular design means high re-use of parts of models and ease of fitting them together


www.das-ltd.co.uk

4) OSCAM development is treated as a software development project

19

• Model stability is vitally important when there is a large user base. Errors or major gaps in model logic will quickly lead to loss of confidence in the model, and loss of users

• OSCAM models have requirements, design and testing documents in common with software development practices

- These practices extend to the System Dynamics model where influences diagrams, within an explanatory structure document, are part of the formal design process

- Front-end designer/developers and SD model designer/developers must work closely together, and with the client, at all stages of development to ensure a well focused product (this is easier when the lead designer for both is the same person)

• Balance of effort (including client effort) for a major OSCAM development has a large component for design and testing

For OSCAM Ship v8, it was approximately:

- Design- SD model 10%, user interface 15%

- Development- SD model 10%, user interface 25%

- Formal Testing- SD model 20%, user interface 15%

- End user documentation (excluding training materials)- SD model 4%, user interface 1%


www.das-ltd.co.uk

Authors

20

Stephen Curram is a Managing Consultant at Decision Analysis Services Ltd. He is an Operational Research practitioner, with more than 25 years' experience in the field, specializing in cost modelling, System Dynamics, and applications software development. He was previously a lecturer in Operations Research at Warwick Business School where he also gained a PhD in simulation and artificial intelligence.

Brittany Basilone is the submarine cost team lead for the Naval Surface Warfare Center, Carderock Division in the Cost Effectiveness Branch. With 8 years of cost estimating experience, Brittany has provided cost support to several ship platforms for NAVSEA. She also acts as OSCAM Program Manager. Brittany received her Bachelor's degree from Washington & Jefferson College in Mathematics and Education, and a Master of Science degree from George Mason University in Operations Research.


www.das-ltd.co.uk

Thank You

21

oscam: 20 years of making better operating and...

Documents