1 cosysmo 2.0: a cost model and framework for systems engineering reuse jared fortune university of...

University of Southern California Center for Systems and Software Engineering

1

COSYSMO 2.0: A Cost Model and Framework for

Systems Engineering Reuse

Jared FortuneUniversity of Southern California

Ricardo ValerdiMassachusetts Institute of Technology

2009 COCOMO ForumMassachusetts Institute of Technology


Outline• Introduction

– Research Question and Hypotheses

• Methodology– Research Approach

• Research Products– COSYSMO 2.0

– Reuse Framework

• Results– Model Validation

– Framework Validation

• Conclusion– Future Research Areas

2


Need to Estimate Effect of Systems Engineering Reuse

• Practitioners noted large errors between model estimates and actuals– Variations discovered to be attributable to programs with reuse

• Systems engineering reuse – The utilization of systems engineering products from previous efforts– Ex: architectures, requirements, test plans, interfaces– Means of reducing cost, schedule, and/or risk

• Current version COSYSMO assumes all systems engineering products will be “built from scratch”– Unable to account for the impact of systems engineering reuse

…COSYSMO 2.0

3


Reuse is a Universal Concept

4


Academic Perspective

Literature Review Results

1. Reuse is done for the purpose of economic benefit, intending to shorten schedule, reduce cost, and/or increase performance.

2. Reuse is not free, upfront investment is required.

3. Hardware, software, processes, and knowledge are all reusable products.

4. Reuse needs to be planned from the conceptualization phase of system development.

5. Reuse is as much of an organizational issue as it is a technical one.

6. The benefits of reuse are limited to related domains and do not scale linearly.

5


Practitioner Perspective

6

Linear25%

Other13%

Non-linear62%

Schedule21%

Cost29%

Risk19%

Quality15%

Performance16%

Requirements

45%

Test Data11%

Documentation22%

Other22%

Transition to Operations

Test and Evaluation

Develop

Conceptualize

Unaware Ad Hoc Planned

Requirements

Other

Architecture/Design Models

Test Data/Procedures

COTS Products

Documentation/Templates

Never Occasionally Always

Industry Survey Results

1. Requirements reuse is only performed occasionally, but has the largest benefit associated with it.

2. Reuse occurs more frequently early in the life cycle than later.

3. Cost savings is the most promoted benefit for reuse, but benefits also exist in risk, schedule, and performance.

4. Experienced personnel is a key factor for successful reuse.


Research Evolution

Previous Hypotheses

1. Each COSYSMO size driver can be further decomposed into New, Modified, Adopted, Managed, and Deleted categories of reuse each with corresponding rating scales and weights, and function as statistically significant predictors of equivalent size from the systems engineering standpoint.

2. Two additional COSYSMO cost drivers, Reuse Understanding and Artifact Unfamiliarity, capture the important organizational, personnel, process, and project factors that lead to the accurate estimation of systems engineering effort for systems that involve reuse.

7

Updated to six Categories

Focused on capturing reuse in size drivers


Research Question

What technical and organizational characteristics drive the amount of reduced or added systems engineering effort due to systems engineering product reuse?

8


Hypothesis #1

The requirements size driver can be further decomposed into New, Design for Reuse, Modified, Adopted, Managed, and Deleted categories of reuse, each with corresponding definitions and weights, and function as a good predictor of equivalent size from a systems engineering standpoint.

9


Hypothesis #2

By comparing the normative and descriptive models for managing systems engineering reuse, the systems engineering reuse framework provides a prescriptive model for enabling organizations to take advantage of reuse in systems engineering.

10


Hypothesis #3

The COSYSMO 2.0 model and the associated systems engineering reuse framework help organizations think differently about systems engineering reuse, and its associated costs and benefits.

11


Mixed-Methods Research Approach

12


COSYSMO 2.0 Operational Concept

13


Model Form

14


Reuse CategoriesCategory Sub-Category Definition

I) New Products that are completely new

i) Designed for Reuse

Products that require an additional upfront investment to improve the potential reusability

II) Modified Products that are inherited, but are tailored

ii) Deleted Products that are removed from the system

III) Adopted Products that are incorporated unmodified (a.k.a. “black box” reuse)

iii) Managed Products that are incorporated unmodified and with minimal testing

15


Reuse Category Weights

16


Example COSYSMO 2.0 Estimate

Estimated as 129.1 Person-Months by COSYSMO (without reuse)

…a 30.4% difference


COSYSMO Calibration Data Set Analysis (1)

• Data set used to calibrate and validate the COSYSMO tool when it was published in 2005

• Immature form of reuse data was collected– Reported amount of reuse in the system– Ex: If a responder reported the project had 100 nominal requirements

and that 20 of those 100 requirements where reused, they would report 20% of the nominal requirements were reused

– No consistent definition of term

• Results indicated potential for overestimation of effort– 42 projects in the data set– 54% reported some amount of reuse in one of the four size drivers – 13% reported some amount of reuse in all of the size drivers.


• In most cases, COSYSMO is capable of estimating the effort of a systems engineering project within 30% of the actual, 50% of the time– PRED(.30)=50%

• Experimental model created to test hypothesis that a single reuse category is inadequate– Experimental model utilized one reuse category instead of six – Category weights varied parametrically from 0.0 to 1.5 – Estimation power of one versus no reuse categories were compared

• Experimental model (one category) consistently performed worse than standard COSYSMO (no reuse categories) – Supports the conclusion that a single reuse category is inadequate,

multiple reuse categories are required

COSYSMO Calibration Data Set Analysis (2)


COSYSMO 2.0 Implementation Results• Across 44 projects at 1

diversified organization

• Using COSYSMO:– PRED(.30) = 14%

– PRED(.40) = 20%

– PRED(.50) = 20%

– R2 = 0.50

• Using COSYSMO 2.0:– PRED(.30) = 34%

– PRED(.40) = 50%

– PRED(.50) = 57%

– R2 = 0.72

• Result: 36 of 44 (82%) estimates improved

20


Reuse Framework

21


Reuse Framework Interview Results• Conducted interviews and mini-case studies with five systems

engineering organizations– BAE Systems U.S., BAE Systems U.K., Boeing, British Petroleum (BP),

and Rolls-Royce

“Applicable and valuable to organizations at multiple levels of reuse maturity.” Organization A

“A valid and reasonable illustration of the reuse process.”Organization B

22


Research makes SE’s think differently about reuse (1)

“Systems Engineers are frequently tasked to leverage previously developed systems engineering products for the benefit of future projects. The inclusion of reuse in COSYSMO has an inherent value by adding clarity to project characterization and measurement refinement to the estimation of systems engineering cost.”

Marilee Wheaton, Systems Engineering Division, The Aerospace Corporation

"Metrics-based estimates do account for the cost of systems engineering or systems engineering reuse very well. The parametric nature of COSYSMO 2.0, as COCOMO did for software reuse, improves fidelity and confidence in estimates of systems engineering reuse and provides systems engineers with valuable feedback on reuse decisions.”

Allan McQuarrie, Leveraged Technology & Products, BAE Systems

23


“The benefits of the COSYSMO reuse model are significant, including improving the accuracy of COSYSMO and demonstrating its validity across a wider set of systems engineering projects. The reuse model fundamentally enables the development of the Total Engineering Estimation Model and Process (TEEMaP) estimating model, an implementation of COSYSMO, to achieve high degrees of estimation accuracy and robustness. TEEMaP is now widely used in different Lines of Businesses.”

Gan Wang, Electronics and Integrated Systems, BAE Systems

24

Research makes SE’s think differently about reuse (2)


Threats to ValidityMostly Controllable

• Construct Validity– Minimized COSYSMO overlap– Identified products for reuse

• Result Convergence– Utilized mixed-methods approach

• Internal Validity– Iterated definitions at multiple

COSYSMO workshops– Collected multiple Delphi results

• Generalizability– Validated model with multiple projects– Validated framework with interviews

at multiple organizations

25

Mostly Uncontrollable

• Limited Visibility in Data Analysis– Performed using inputs and guidance

from this research

• Relied on Subjective Inputs– Utilized experts familiar with

COSYSMO research

• Increased Number of User Inputs– Created 3 prime & 3 sub-categories to

address needs raised in discussions


Contributions• Identification of needs and gaps in the research of systems

engineering reuse– Literature review

• Definition and characterization of systems engineering reuse– Reuse categories

• Improvement of the COSYSMO tool to account for systems engineering reuse– COSYSMO 2.0

• Documentation of heuristics for systems engineering reuse – Reuse framework

• Systems engineers will think about their responsibilities differently– COSYSMO 2.0 and reuse framework

26


27

Academic COSYSMO 2.0(without reuse)


28

Academic COSYSMO 2.0(with reuse)


Future Areas of Research• Tailoring COSYSMO 2.0 to individual organizations

– Developing industry and local calibrations

• Exploring a range of weights for the reuse categories– Evaluating effect of variation within categories

• Identifying reuse opportunities and mismatches– Extending development of Expert COSYSMO

• Quantifying the effect of reuse on risk and schedule– Supporting development of new COSYSMO extensions

• Harmonizing reuse across hardware, software, and systems engineering– Continuing efforts to integrate COCOMO and COSYSMO

29

1 cosysmo 2.0: a cost model and framework for systems engineering reuse jared fortune university of...

Documents

reuse systems engineering

requirements reuse

benefits of reuse

successful reuse

deleted categories of

systems engineering

cosysmo size driver

cost model