characterizing the impact of requirements volatility on systems engineering effort

1

University of Southern CaliforniaCenter for Systems and Software Engineering

Mauricio E. PeñaDr. Ricardo Valerdi

March 8, 2011

Characterizing the Impact of Requirements Volatility on

Systems Engineering Effort

2

University of Southern California Center for Systems and Software Engineering

Agenda1:30 – 1:45 pm Introductions & objectives

1:45 – 2:00 pm COSYSMO overview and updates

2:00 - 2:30 pm COSYSMO requirements volatility extension: introduction and research results

2:30 – 3:30 pm Requirements volatility measures: definitions & counting rules

3:30 – 3:45 pm Break

3:45 – 4:45 pm Survey Exercise: expected level of volatility and impact on systems engineering effort

4:45 – 5:00 pm Discussion

3


Objectives of the Workshop• Learn about COSYSMO and the latest research

updates• Explore the causes of requirements volatility and its

impact on systems engineering effort• Obtain feedback on a proposed requirements

volatility extension to COSYSMO• Reach agreement on definitions and rating scales for

requirements volatility • Provide an opportunity for participants to exchange

lessons learned on requirements volatility and influence the direction of future research

4


COSYSMO Overview & Updates

5


CSSE Parametric Cost Models• The Constructive Systems Engineering Cost Model

(COSYSMO) was developed by the USC Center for Software and Systems Engineering (CSSE) in collaboration with INCOSE and Industry affiliates

• COSYSMO is the first generally-available parametric cost model designed to estimate Systems Engineering effort

• Built on experience from COCOMO 1981, COCOMO II– Most widely used software cost models worldwide– Developed with Affiliate funding, expertise, data support

6


COSYSMO

Size*Drivers

EffortMultipliers

Effort

Calibration

# Requirements# Interfaces# Scenarios# Algorithms

- Application factors-8 factors

- Team factors-6 factors

WBS guided by EIA/ANSI 632

COSYSMO Operational Concept

Source: Valerdi, R. (2005)

*Each weighted by complexity: easy, nominal, and difficult

7


Scope of the Model

ISO/IEC 15288 – Common framework for describing the life cycle of systemsEIA/ANSI 632 – Integrated set of fundamental systems engineering processes

Source : Draft Report ISO Study Group May 2, 2000

ISO/IEC 15288

Conceptualize DevelopTransition to

Operation

Acquisition & Supply

Technical Management

System Design

Product Realization

Technical Evaluation

Operational Test &

Evaluation

AN

SI/E

IA 6

32ISO/IEC 15288

Conceptualize DevelopTransition to

Operation

Acquisition & Supply

Technical Management

System Design

Product Realization

Technical Evaluation

Operational Test &

Evaluation

AN

SI/E

IA 6

32

8


Life Cycle Phase Definition• Conceptualize phase focuses on identifying stakeholder needs,

exploring different solution concepts, and proposing candidate solutions.

• The Development phase involves refining the system requirements, creating a solution description, and building a system.

• The Operational Test & Evaluation Phase involves verifying/validating the system and performing the appropriate inspections before it is delivered to the user

• The Transition to Operation Phase involves the transition to utilization of the system to satisfy the users’ needs.


9


Parametric Cost Estimating Relationship

• Where:PMNS = effort in Person Months (Nominal Schedule)A = constant derived from historical project dataSize = determined by computing the weighted average of the (4) size driversE = represents diseconomy of scalen = number of cost drivers (14)EM = effort multiplier for the ith cost driver. The geometric product results in an overall effort adjustment factor to the nominal effort


10


COSYSMO Cost Drivers

• Application Factors– Requirements understanding – Architecture understanding – Level of service

requirements– Migration complexity – Technology Maturity – Documentation Match to Life

Cycle Needs– # and Diversity of

Installations/Platforms– # of Recursive Levels in the

Design

• Team Factors– Stakeholder team cohesion – Personnel/team capability – Personnel

experience/continuity – Process maturity – Multisite coordination – Tool support


11


Cost Driver Rating Scales

Very Low Low Nominal High Very High Extra High EMR

Requirements Understanding 1.87 1.37 1.00 0.77 0.60 3.12

Architecture Understanding 1.64 1.28 1.00 0.81 0.65 2.52

Level of Service Requirements 0.62 0.79 1.00 1.36 1.85 2.98

Migration Complexity 1.00 1.25 1.55 1.93 1.93

Technology Risk 0.67 0.82 1.00 1.32 1.75 2.61

Documentation 0.78 0.88 1.00 1.13 1.28 1.64

# and diversity of installations/platforms 1.00 1.23 1.52 1.87 1.87

# of recursive levels in the design 0.76 0.87 1.00 1.21 1.47 1.93

Stakeholder team cohesion 1.50 1.22 1.00 0.81 0.65 2.31

Personnel/team capability 1.50 1.22 1.00 0.81 0.65 2.31

Personnel experience/continuity 1.48 1.22 1.00 0.82 0.67 2.21

Process capability 1.47 1.21 1.00 0.88 0.77 0.68 2.16

Multisite coordination 1.39 1.18 1.00 0.90 0.80 0.72 1.93

Tool support 1.39 1.18 1.00 0.85 0.72 1.93


12


13


COSYSMO 2.0 Operational Concept

Source: Fortune (2009)

14


COSYSMO 2.0 Model Form


15


Reuse Category Weights


16


COSYSMO 2.0 Implementation Results

• Across 44 projects at 1 diversified organization

• Using COSYSMO:– PRED(.30) = 14%– PRED(.40) = 20%– PRED(.50) = 20%– R2 = 0.50

• Using COSYSMO 2.0:– PRED(.30) = 34%– PRED(.40) = 50%– PRED(.50) = 57%– R2 = 0.72• Result: 36 of 44 (82%) estimates

improvedSource: Fortune (2009)

17


COSYSMO Extension:Requirements Volatility

18


Importance of Understanding Requirements Volatility

• Requirements volatility has been identified by numerous research studies as a risk factor and cost-driver of systems engineering projects1

• Requirements changes are costly, particularly in the later stages of the lifecycle process because the change may require rework of the design, verification and deployment plans2

• The Government Accountability Office (GAO) concluded in a 2004 report on the DoD’s acquisition of software-intensive weapons systems that missing, vague, or changing requirements are a major cause of project failure3

System developers often lack effective methods and tools to account for and manage requirements volatility

Source: 1- Boehm (1991), 2- Kotonya and Sommerville (1995), 3- GAO-04-393

19


Requirements Volatility is Expected• Changes to requirements are a part of our

increasingly complex systems & dynamic business environment

– Stakeholders needs evolve rapidly– The customer may not be able to fully specify the system

requirements up front– New requirements may emerge as knowledge of the system

evolves– Requirements often change during the early phases of the

project as a result of trades and negotiations

Sources: Kotonya and Sommerville (1995); Reifer (2000)

Requirements volatility must be anticipated and managed

20


Questions that will be covered• How do we account for the impact of requirements

volatility on functional size and effort estimates?• What counting rules should be used for requirements

volatility?– How do we distinguish between typical changes/iterations

and unplanned changes that require more effort than originally projected?

– How do we prevent from counting the elaboration of requirements as volatility (added requirements)?

• Is it feasible to track the volatility of other size drivers (interfaces, operational scenarios, algorithms)?

21


Requirements Volatility Definitions• Requirements volatility is typically defined as the

change in requirements (added, deleted, and modified) over a given time interval

• Also known as:• Requirements creep: An increase in scope and

number of system requirements• Requirements churn: Instability in the requirements

set – requirements are modified or re-worked without necessarily resulting in an increase in the total number of requirements

Source: MIL-STD-498

22


Requirements Volatility Metrics (1 of 3)

Systems Engineering Leading Indicators Guide, Version 2.0, 2010

100%90%80%70%60%50%40%30%20%10%0%

Planned SRR

Metric-Driven

SRR

Total Changes AddedDeletedModified

Projected Trend

% o

f tot

al R

equi

rem

ents

Cha

ngin

g

Time

Program “A” Requirements Volatility100%90%80%70%60%50%40%30%20%10%0%

Planned SRR

Metric-Driven

SRR

Total Changes AddedDeletedModified

Projected Trend

% o

f tot

al R

equi

rem

ents

Cha

ngin

g

Time

Program “A” Requirements Volatility

Volatility expressed as a % of the total number of requirements

23



Source: Hammer, T., Huffman, L., and Rosenberg, L. (1998).

Program “B” Requirements Volatility

Time (months)

50

40

30

20

10

0

Num

ber o

f Req

uire

men

ts DeletedAdded Modified

Program “B” Requirements Volatility

Time (months)

50

40

30

20

10

0

Num

ber o

f Req

uire

men

ts DeletedAdded Modified

Volatility metrics track the number and type of changes over time

24



0

20

40

60

80

100

120

140

160

180

J F M A M J J A S O N D

Req. Added

Req. Deleted

Req. Modified

Cum. Deleted

Cum. Modified

Cum. Added

Cum. Changes

# of

Req

uire

men

ts C

hang

es

Conceptualize Development Operational Test & Evaluation

Transition to Operation

0

20

40

60

80

100

120

140

160

180

J F M A M J J A S O N D

Req. Added

Req. Deleted

Req. Modified

Cum. Deleted

Cum. Modified

Cum. Added

Cum. Changes

# of

Req

uire

men

ts C

hang

es



25


Systems Engineering Reviews and Acquisition Lifecycle Phases

Materiel Solution Analysis

Technology Development

Engineering and Manufacturing Development

Production & Deployment

Operations & Support

Pre-Systems Acquisition Systems Acquisition Sustainment

A B C(Program Initiation) IOC FOC

Acquisition Milestones Evolutionary Acquisition or Single Step toFull Capability

ASR SRR SFR PDR CDR FCA PCATech Reviews

Documents

Sys Perf. Spec.

Item Perf. Specs.

Item Detail

Requirements Review

System Definition SY

S

CI

Sys Tech Requirements

Item Design Rqmts.

Detailed Design Rqmts.

Draft

User Needs

Materiel Solution Analysis

Technology Development

Engineering and Manufacturing Development

Production & Deployment

Operations & Support

Pre-Systems Acquisition Systems Acquisition Sustainment

A B C(Program Initiation) IOC FOC

Acquisition Milestones Evolutionary Acquisition or Single Step toFull Capability

ASR SRR SFR PDR CDR FCA PCATech Reviews

Documents

Sys Perf. Spec.

Item Perf. Specs.

Item Detail

Requirements Review

System Definition SY

S

CI

Sys Tech Requirements

Item Design Rqmts.

Detailed Design Rqmts.

Draft

User Needs

Source: DoD Instruction 5000.02 (2008)

A Requirements baseline is established when the functional requirements and characteristics of the system are formally documented and placed under configuration control

26


Requirements and Evolutionary Acquisition Process

Source: DoD Instruction 5000.02 (2008)

27


Requirements Trends as a Leading Indicator of Project Performance

• Evaluates trends in the growth, change, completeness and correctness of the system requirements.

• It helps to determine the stability and completeness of the system requirements which could potentially impact project performance

Source: Systems Engineering Leading Indicators Guide, Version 2.0, 2010

Planned # of RequirementsActual # of RequirementsProjected # of Requirements

Num

ber o

f Req

uire

men

ts

Requirements Growth Trends

TimeSRR PDR CDR

Corrective Action Taken

28


Implications to COSYSMO• During the development of COSYSMO, volatility was

identified as a relevant adjustment factor to the model’s size drivers

• However, there was insufficient data to incorporate volatility effects into the initial version of the model

• The primary objective of the research is to complete the requirements volatility extension to COSYSMO within the existing structure and scope of the model

Source: Valerdi (2005)

29


COSYSMO Volatility Factor

COSYSMO

SizeDrivers

EffortMultipliers

Effort

Calibration




ReuseCategories

Volatility Factor

COSYSMO

SizeDrivers

EffortMultipliers

Effort

Calibration




ReuseCategories

Volatility Factor

COSYSMO

SizeDrivers

EffortMultipliers

Effort

Calibration




ReuseCategories

Volatility Factor


30


Overview of Research to Date• Data were collected through surveys and interviews

of subject-matter experts in four different workshops• A variety of industries were represented with an

emphasis on aerospace & defense • Exploratory survey administered at:

– Workshop # 1: 2010 USC-CSSE Annual Research Review– Workshop # 2: 2010 LAI Knowledge Exchange Event

• Survey exercises were conducted at:– Workshop # 3: 2010 Practical Software and Systems

Measurement (PSM) Users Group Conference – Workshop # 4: University of Southern California 25th Annual

COCOMO Forum

31


Observations from the Literature and Exploratory Research

1. Requirements volatility is correlated with an increase in project size and cost

2. The level of requirements volatility is a function of the system life cycle phase and does not necessarily behave linearly

3. The cost / effort impact of a requirements change increases the later the change occurs in the system life cycle

4. The impact of requirements volatility varies depending on the type of change: added, deleted, or modified

5. Removing a requirement may not necessarily result in a net decrease in engineering effort and cost

A causal model diagram was developed that relates technical, organizational and contextual project factors to requirements volatility

32


Requirements Volatility Causal Model Diagram

Contextual / Environmental

ChangesExperienced

StaffChanges in

COTS Products

Changes in Org / structure/

Process

Poor Understanding of

the System & Customer Needs

Customer-driven Scope Change

Changes in Co-dependent

Systems

Requirements Process Maturity

Technology Maturity

Requirements Volatility

Sources: Kotonya and Sommerville (1995); Hammer et al. (1998); Malaiya and Denton (1999); Stark et al. (1999); Houston (2000); Zowghi and Nurmuliani (2002); Kulk and Verhoef 2008; Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009)

33


Impact of Requirements Volatility

Project Effort

Rework / Defects

Number of System

Requirements

Customer Satisfaction

Quality

Project Cost

Project Schedule

Requirements Volatility +/-

+/- +/-+/- +/-

+/-


34


Exploratory Survey• Developed to gather the perspectives of subject-

matter experts on the causes, impacts, and expected level of requirements volatility for a given system of interest

• Organizations represented: – The Aerospace Corporation, Northrop Grumman

Corporation– The Boeing Company, Softstar Systems, Raytheon– United Launch Alliance, Massachusetts Institute of

Technology, University of Southern California, and– United States Army– Unites States Navy

35


Exploratory Survey Participant Background

System Application Domain

System H/W –S/W breakdown

100% Software, 125% Hardware, 75% Software, 5

50% Hardware, 50% Software, 11

75% Hardware, 25% Software, 3

100% Hardware, 0

No response, 2

Telecomm.2%

Space Systems22%

Transportation Systems

2%

Automotive0%

Scientific / Research

9%

Military / Defense41%

Other7%

Infrastructure 2%

Aircraft/Avionics11%

Data Systems / IT4%

• 22 respondents

• 23 years average industry experience

• Primarily from a Military/Defense and Space Systems Background

• Experienced on Systems with a fairly balanced H/W and S/W work content

36


>20%

>20%

>20% >20%

10-20%

10-20%

10-20%

10-20%

5-10%

5-10%

5-10%

5-10%

<5% <5%

<5%

<5%

don't know don't know don't know don't know

0

5

10

15

20

Conceptualize Development Operational Test &Evaluation

Transition toOperation

Expected Requirements Volatility (%)

Num

ber o

f Res

pond

ents

Expected Level of Volatility Across Lifecycle Phases

Most respondents expect >20% volatility during the conceptualize phase of the project, decreasing to <5% in the transition to operation phase

Source: LAI Knowledge Exchange / CSSE ARR Survey (2010)

37


Impact of Hardware/Software Project Breakdown on Expected Volatility

Operational Test & Evaluation Lifecycle Phase Transition to Operation Lifecycle Phase

0%

20%

40%

60%

80%

100%

<5% 5-10% 10-20% >20%Expected Requirements Volatility (%)

75% Hardware, 25% Software 50% Hardware, 50% Software25% Hardware, 75% Software

% o

f Res

pond

ents

per C

ateg

ory

0%

20%

40%

60%

80%

100%

<5% 5-10% 10-20% >20%

Expected Requirements Volatility (%)75% Hardware, 25% Software 50% Hardware, 50% Software

25% Hardware, 75% Software

% o

f Res

pond

ents

per C

ateg

ory

Respondents from S/W intensive projects tend to expect more volatility later in the lifecycle

Source: LAI Knowledge Exchange / CSSE ARR Survey (2010)

38


0%

20%

40%

60%

80%

100%

Re-work of work products Project size (Total # ofrequirements)

% o

f res

pond

ents

Impacts of Volatility• In general, results of the

survey support observations from the literature and causal model

• Most respondents stated that requirements volatility will cause a moderate to large increase in the number of system requirements and rework

Moderate decrease Moderate Increase

No impact Large Increase

39


Ranked Causes of Requirements Volatility (Exploratory Survey)

Contextual / Environmental

ChangesExperienced

StaffChanges in

COTS Products

Changes in Org / structure/

Process

Poor Understanding of

the System & Customer Needs

Customer-driven Scope Change

Changes in Co-dependent

Systems

Requirements Process Maturity

Technology Maturity

Requirements Volatility

1 2 3 4 5

6 7 8 9


40


Workshop # 3: PSM Conference• Survey Exercise administered during the 2010

Practical Software and Systems Measurement Conference

• Attended by 9 participants from diverse engineering organizations and academic institutions

• Organizations Represented:– Distributed Management, Northrop Grumman– Lockheed Martin, Ericsson España– Samsung SDS, U.S. Navy– Massachusetts Institute of Technology (MIT)

41


Workshop # 3: Survey ExerciseParticipants were asked to:1.Draw a requirements volatility profile across the

lifecycle phases covered by COSYSMO2.Draw an “ease of change” profile across the same

life cycle phases to determine the volatility weighting factor

3.Discuss variation in 1 and 2 above for:1. Large and small projects2. Hardware and software projects3. Development and recurring projects

42


Sample Volatility Profiles



% o

f Req

uire

men

ts, A

dded

, Del

eted

or M

odifi

ed

50%

40%

30%

20%

10%

5%

15%

25%

35%

45%

0%

Localized peaks in volatility

Participant inputs#1 # 5#2 #6#3 #7#4



% o

f Req

uire

men

ts, A

dded

, Del

eted

or M

odifi

ed

50%

40%

30%

20%

10%

5%

15%

25%

35%

45%

0%

50%

40%

30%

20%

10%

5%

15%

25%

35%

45%

0%

Localized peaks in volatility

Participant inputs#1 # 5#2 #6#3 #7#4

Source: Practical Software & Systems Measurement Conference Survey (2010)

43


Ease of Change Profile



Ease

of C

hang

e 1

0.5

0.1

0

0.2

0.3

0.4

0.9

0.6

0.7

0.8

Representative of Participant Feedback



Ease

of C

hang

e 1

0.5

0.1

0

0.2

0.3

0.4

0.9

0.6

0.7

0.8

Representative of Participant Feedback

Cost Penalty defined as 1 / ease of change

Average Ease of Change Factor (Estimated)

Average Cost Penalty (Estimated)

0.8 0.4 0.1 0.05

1.25 2.5 10 20

44


Workshop #4: COCOMO Forum• Attended by 15 participants from 10 different engineering

organizations and academic institutions• The participants were asked to estimate:• Expected level of volatility

– The cumulative volatility (%) over the four life cycle phases covered by COSYSMO

– How those changes in requirements would be partitioned per life cycle phase

– The breakdown of volatility per type of change (added, deleted, modified)

• Cost Penalty– Provided a change cost penalty as a function of lifecycle phase– Estimated the cost penalty per type of change

45


Expected Volatility per Lifecycle Phase

-5%

0%

5%

10%

15%

20%

25%



Req

uire

men

ts V

olat

ility

(% c

hang

es)

Average 10% 11% 5% 2%

Standard Dev 7% 8% 5% 3%

Expected Cumulative Volatility = 28%, STD = 14%

46


Weighting Factor per Lifecycle Phase

-20

-10

0

10

20

30

40

50



1 = no penalty

Conceptualize DevelopmentOpr. Test & Evaluation


Average 1 7 13 16Standard Dev 0.3 15 26 30

47


Breakdown per type of Change

-2

0

2

4

6

8

10

12

Conceptualize Development Operational Test &Eval

Transition to Ops

added

deleted

modified

Cos

t Pen

alty

Fac

tor

Source: COCOMO Forum Survey Exercise (2010)

Expected Breakdown of Volatility per type of change: 34% added, 51% modified, 15% deleted

48


Proposed Requirements Volatility Extension

49


Requirements Volatility Size Driver Adjustment Factor

REVL is defined as the % of the baseline set of requirements that changes throughout the lifecycleA weighting factor is added to account for the effort penalty due to rework

This relationship is expressed through the following equation:

Where,Rx,r = Baseline number of requirements (easy, nominal, difficult)Reff = Effective number of requirements at the end of the projectwv = Requirements volatility weighting factor

Source: Boehm (2000)

rdrnreveff RRRwREVLR ,,," 1001

50


Aggregate Weighting Factor

As previously discussed, the effort penalty due to requirements volatility is likely to increase across lifecycle phases

In order to capture this potential lifecycle phase dependency in the model, the following relationship is proposed:

Wherewv = Requirements volatility weighting factorwx,l = Weighting factor for added, deleted, or modifiedΘx,l = % of total changes that were added, deleted or modifiedl = lifecycle phases

l

lmlmldldlalav wwww ,,,,,,

51


Proposed Model (2)As previously discussed, requirements volatility may be decomposed into added, modified or deleted requirement; each with a different weighting factor

Rv = wa · Radded + wm · Rmodified + wd · Rdeleted

Where,Rv = Effective number of requirements changeswx = Weighting factor for added, deleted, or modifiedRx = Number of requirements added, modified, deleted

52


Requirements Volatility Measures

53


Requirements Size Driver Definition

This driver represents the number of requirements for the system-of-interest at a specific level of design. The quantity of requirements includes those related to the effort involved in system engineering the system interfaces, system specific algorithms, and operational scenarios.

Requirements may be functional, performance, feature, or service-oriented in nature depending on the methodology used for specification. They may also be defined by the customer or contractor.

Each requirement may have effort associated with is such as V&V, functional decomposition, functional allocation, etc.

System requirements can typically be quantified by counting the number of applicable shalls/wills/shoulds/mays in the system or marketing specification

Source: Valerdi (2005)

54


Requirements Counting Rules1. Determine the system of interest2. Decompose system objectives, capabilities, or measures of

effectiveness into requirements that can be tested, verified, or designed• Different systems will exhibit different levels of requirements

decomposition depending on the application domain, customer’s ability to write good system requirements, and the functional size of the system

3. Provide a graphical or narrative representation of the system of interest and how it relates to the rest of the system

4. Count the number of requirements in the system/marketing specification or the verification test matrix for the level of design in which systems engineering is taking place in the desired system of interest

5. Determine the complexity, volatility, and reuse of requirements Source: Valerdi (2005)

55


Requirements Volatility Measures• Base Measures

– Number of requirements changes– Number of added, deleted or modified requirements

• Measurement methods1. Count the number of requirements changes from

Engineering Change Proposals (ECPs)2. Count the number and type of changes using a

requirements management tool such as the Dynamic Object Oriented Requirements System (DOORS)

3. Use a text differencing tool to determine the number and type of requirements changes between two versions of a specification

56


Requirements Volatility Counting Rules

• We need to formulate specific counting rules for the volatility of the requirements size driver

• Questions for discussion:– How do we distinguish between typical changes/iterations

and unplanned changes that require more effort than originally projected?

– How do we prevent from counting the elaboration of requirements as volatility (added requirements)?

– How do we avoid counting editorial changes (typos, etc.)?– How do we identify major changes in scope that could

disrupt the results?

57


Cone of Uncertainty

Source: Boehm et al., 2004

58


Requirements Decomposition Framework• Sky level

– Top-level objective– Summary use case

• Kite level– Additional information as to how the

sky-level objective is to be will be satisfied

• Seal level– User level task– Environment in which the developer

interacts with the stakeholder• Underwater level

– Detailed design and implementation

Sources: Cockburn (2001); Valerdi (2005)

59


Requirements Elaboration (S/W Example)

Capability Requirements

Use Cases

Use Case Steps

expansion

expansion

Contractor

Customer

expansion

SLOC

expansion

Capability Goal

Sources: Cockburn (2001); Malik (2010)

60


Elaboration Factors

Elaboration FactorsAverage Std. Dev.

Study 1 Study 2 Study 1 Study 2

Capability Goal to Capability Requirement

1.95 2.46 0.67 0.94

Capability Requirement to Use Case

4.89 0.89 1.49 0.55

Use Case to Use Case Steps

5.67 7.06 1.60 2.97

Use Case Steps to SLOC 266.93 66.91 72.77 64.46

Source: Malik (2010)

61


Requirements Decomposition Example• Objective: Broadcast television signals over the Continental

United States (CONUS) compatible with 18 inch receive antennas1. The system shall be able to receive a Ku-band signal from the

customer ground station and downlink the signal to CONUS coverage with a minimum Equivalent isotropically radiated power (EIRP) of 30 dBm

1.1 The system shall have a pointing accuracy of 0.5 degrees1.2 The system shall radiate a minimum of 3000 W of RF power

1.2.1 The system shall be able to produce 4000 W of DC power1.2.2 The system shall be able to store 500 W-Hr of energy1.2.3 The system shall be able to dissipate 2000 W of heat

1.2.3.1 The radiator surface area shall be greater than…

1.2.3.2 The radiator surface properties shall be…….

1.2.3.3 Metal surfaces shall be blanketed…….

Too High?

Too Low?

Just Right

62


Proposed Guidelines• Count only requirements changes after the

requirement set is baselined and under configuration control (SRR, PDR)

• Count only changes to requirements for the system-of-interest at a specific level of design (use COSYSMO requirements counting rules)

• Identify disruptive changes in scope / re-baselining in the data collection instrument

• Use counting logic rules to exclude multiple changes to a requirement in one day (avoid counting editorial changes)

63


Feedback on Data Collection Instrument

Data Collection Instrument Link

64


Questions on the Approach• Should we attempt to count changes per type of

requirement (easy, nominal, difficult)?– Is it feasible?– Does it add value to the analysis?

• Do you keep metrics on the volatility of the other size drivers?

– Operational scenarios– System interfaces– Algorithms

• Would measuring the impact of requirements volatility envelope the impact of changes to the other size drives?

65


Survey Exercise

66


Thank you!

67


References• Boehm, B. (1991). “Software Risk Management: Principles and Practices.” IEEE Software. Vol 8 (No.1).pp 32-41

• Boehm, B., Abts, C., Brown, A.W., Chulani, S., Clark, B., Horowitz, E., Madachy, R., Reifer, D.J., and Steece, B. (2000). Software Cost Estimation with COCOMO II. Prentice Hall

• B. Boehm et al., Guidelines for Model-Based (System) Architecting and Software Engineering (MBASE) Version 2.4.1, USC-CSE, 2004.

• Cockburn, A. (2001). Writing Effective Use Cases, Addison-Wesley.

• EIA (2002). EIA-731.1 - Systems Engineering Capability Model

• Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009). “Understanding the effects of requirements volatility in software engineering by using analytical modeling and software process simulation.” The Journal of Systems and Software. Vol. 82, pp 1568-1577.

• Fortune, J. (2009) Estimating Systems Engineering Reuse with the Constructive Systems Engineering Cost Model (COSYSMO 2.0). Doctoral Dissertation. University of Southern California

• General Accounting Office (2004). Stronger Management Practices are Needed to Improve DOD’s Software-intensive Weapon Acquisitions (GAO-04-393). Defense Acquisitions.

• Hammer, T., Huffman, L., and Rosenberg, L. (1998). “Doing requirements right the first time.” Crosstalk, the Journal of Defense Software Engineering. Pp 20-25.

• Houston, Dan X. (2000). A Software Project Simulation Model for Risk Management, Ph.D. Dissertation, Arizona State University

• IEEE (1990). IEEE Standard Glossary of Software Engineering Terminology (ANSI), [1-55937-067-X] [SH13748-NYF], IEEE

• INCOSE (2010). Systems Engineering Handbook. INCOSE Version 3.2

• Kotonya, G., Sommerville, I., (1998). Requirements Engineering: Processes and Techniques. John Wiley and Sons, Ltd.

• Malik, A. (2010). Quantitative and Qualitative Analyses of Requirements Elaboration for Early Software Size Estimation. Doctoral Dissertation. University of Southern California.

• MIL-STD-498. 1994. Software Development and Documentation. U.S. Department of Defense.

• Reifer, Donald J. 2000. “Requirements Management: The Search for Nirvana.” IEEE Software. Vol 17 (No. 3), pp 45-47

• Roedler, G. and Rhodes, D. (2007). Systems engineering leading indicators guide. Version 1. Massachusetts Institute of Technology, INCOSE, and PSM

• Valerdi, R. (2005). The constructive systems engineering cost model (COSYSMO). Doctoral Dissertation. University of Southern California

• Weinberg, G. (1993). Quality Software Management: Volume 2 First-Order Measurement. Dorset House.

• Zowghi, D. and Nurmuliani, N. (2002). A Study of the Impact of Requirements Volatility on Software Project Performance. Proceedings of the Ninth Asia-Pacific Software Engineering Conference

68


Call for Participation• In order to complete the requirements volatility extension of

COSYSMO, we are seeking industry data for engineering projects in terms of:

– Systems engineering effort actuals (labor hours)– Requirements volatility: the number of requirements, added, deleted, and

modified added after the requirements baseline• By providing these data your organization will benefit by:

– Improving its ability to estimate the impact of requirements changes on project cost

– Calibrating and tailoring the updated Model for your application domain• USC-CSSE and LAI at MIT have proven processes in place to ensure

the confidentiality and protection of the data with its Corporate Affiliates and Consortium Members

• Contact:Mauricio E. Pena [[email protected]]Ricardo Valerdi [[email protected]]