welcome [sercuarc.org]...2019/12/11  · ray madachy, naval postgraduate school david jacques, air...

SERC Talks December 11, 2019 1

Today’s session will be recorded.

An archive of today’s talk will be available at: www.sercuarc.org/serc-talks/ as well as on the SERC

YouTube channel.

Use the Q&A box to queue questions, reserving the chat box for comments, and questions will be

answered during the last 5-10 minutes of the session.

If you are connected via the dial-in information only, please email questions or comments to

SERCtalks@stevens.edu.

Any issues? Use the chat feature for any technical difficulties or other comments, or email

SERCtalks@stevens.edu.

WELCOME

“How Can System Architecture and Cost Models be Integrated for UAS TradespaceAnalysis?December 11, 2019 | 1:00 PM ETDr. Raymond J. Madachy, Professor of Systems Engineering, Naval Postgraduate Schoolrjmadach@nps.edu

SERC Talks December 11, 2019 2

The Systems Engineering Research Center (SERC) is a federally funded University Affiliated Research Center managed by Stevens Institute of Technology.

Any views, opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Department of Defense, ASD(R&E), nor the SERC.

No Warranty. This Stevens Institute of Technology Material is furnished on an “as-is” basis. Stevens Institute of Technology makes no warranties of any kind, either expressed or implied, as to any matter including, but not limited to, warranty of fitness for purpose or merchantability, exclusivity, or results obtained from use of the material. Stevens Institute of Technology does not make any warranty of any kind with respect to freedom from patent, trademark, or copyright infringement.

This material has been approved for public release and unlimited distribution.

How Can System Architecture and Cost Models Be Integrated for UAS

Tradespace Analysis?Ray Madachy, Naval Postgraduate School

David Jacques, Air Force Institute of Technology

Kristin Giammarco, Naval Postgraduate School

John (Mike) Green, Naval Postgraduate School

SERC Talk

December 11, 2019

Agenda

4

• Introduction and Background• SysML and Systems Engineering Cost

• System Behavioral Modeling and Software Cost

• Case Study: Remote Targeting System UAS

• System Product Line Architecture Variant Modeling and Product Line Cost with ROI

• Conclusions and Future Work

Introduction

• Research with DoD Systems Engineering Research Center (SERC) System Qualities Ontology, Tradespace and Affordability (SQOTA) Project, and the NPS Acquisition Research Program (ARP).

– Joint research by NPS and AFIT developing affordability tradeoff analysis methods with architecture and cost models applied to Unmanned Aerial Systems (UAS) case studies.

– ARP research developed product line modeling methods applied to Naval combat systems which are transferable to the UAS domain.

• Addresses problem that performance-based and cost tradeoff analyses are generally distinct modeling activities in the DoD.

• A focus is at the interface of cost and architecture models for automatic extraction of system size attributes from the architecture models at the levels of software, system and product line.

– Costs can then be associated with architecture variants to assess tradeoffs between affordability and other ilities involving performance measures.

• Requires translations between MBSE models to map architectural elements with cost model factors.

• Cost estimation is improved by using consistent architecture entities for cost model inputs, without the need for independent cost modeling expertise and effort expenditure.

5

Method Overview

• Use MBSE methods and tools to evaluate behavior, performance and cost in the face of requirements changes and System of System (SoS) architectural variations.

• Develop operational and system architectures in MBSE environments to capture military scenarios for actual and notional case studies

– Modeling with SysML, DoDAF, Orthogonal Variability Models, Monterey Phoenix, and executable activity models using various tools

• Develop cost model interfaces for components of the architectures in order to evaluate cost effectiveness in uncertain future environments.

• Evaluate tradespace including cost in integrated MBSE environment with static and executable models of architectures

6

Cost Model Background

• Cost models provide an easy-to-use framework for performing broader ility and affordability analyses when tied to architecture models.

• Parametric cost models use cost estimating relationships (CERs) as mathematical algorithms relating cost factors. They use numeric inputs for explanatory variables reflecting system characteristics to compute cost.

• This research leverages parametric cost models in the public domain, with the formulas open and available for use. All stakeholders can ascertain why the models produce the results they do and be better informed for making decisions. These models are called “constructive” for these reasons.

• System size is the largest source of cost variance in all parametric cost models. It is the foremost cost factor to capture.

7

Parametric Effort Formula for Constructive Cost Models

Where– Effort is in Person-Months (PM)– A is a constant derived from historical project data– Size is a measure of the work product– B is an exponent for the diseconomy of scale– EM

i is an effort multiplier for the ith cost driver. The geometric product

of N multipliers is an overall Effort Adjustment Factor (EAF) to the nominal effort.

Constructive - A user understands why the model gives the estimate it does, and gains a better understanding of the job being estimated through using the cost model.

8

Cost Models Used

• Constructive Systems Engineering Cost Model (COSYSMO) estimates the labor cost of systems engineering. – A mapping between SysML entities and COSYSMO size attributes was

developed.

• Constructive Cost Model (COCOMO) estimates software development costs.– A mapping between behavioral modeling entities and software size was

developed.

• Constructive Product Line Investment Model (COPLIMO) assesses the labor costs, savings, and return on investment (ROI) for developing and reusing software product lines. It was extended to the system level for this research to include hardware development and non-labor costs.– A mapping between architecture model variations and system component

degree of reuse was developed.

9

Agenda

10

• Introduction and Background• SysML and Systems Engineering Cost

• System Behavioral Modeling and Software Cost

• Case Study: Remote Targeting System UASSystem Product Line Architecture Variant Modeling and Product Line Cost with ROI

• Conclusions and Future Work

COSYSMO

SizeDrivers

EffortMultipliers

Effort by Phase and Activity

# Requirements# Interfaces# Scenarios# Algorithms +Complexity Adjustments

- Application factors-8 factors

- Team factors-6 factors

COSYSMO Black Box Model

11

COSYSMO Size Inputs

Size Type DescriptionRequirements The number of requirements for the system-of-interest at a

specific level of design. Requirements may be functional, performance, feature, or service-oriented.

Interfaces The number of shared physical and logical boundaries between system components or functions (internal interfaces) and those external to the system (external interfaces).

Algorithms The number of newly defined or significantly altered functions that require unique mathematical algorithms to be derived in order to achieve the system performance requirements.

Operational Scenarios (Threads)

Operational scenarios that a system must satisfy, including nominal and off-nominal threads.

12

SysML to COSYSMO Mapping

# Requirements

Parametric Diagram

<<constraint>>

SysML COSYSMO

System Size

# Interfaces

# Algorithms

# Operational Scenarios (Threads)

Internal Block Diagram

port

Block Definition Diagram

port Σ

Use Case Diagram

use case

Package Diagram

<<requirement>>

Requirements Diagram

<<requirement>>

Block Definition Diagram

+

13

Example UAS Missions

14

• Single UAS Search and Target Tracking (Simple Mission)

• UAS Pair Search and Target Tracking• Find, Fix and Finish Terrorist Leadership (1)• Find, Fix and Finish Terrorist Leadership (2)• Mobile Missile Launcher Monitoring (1)• Mobile Missile Launcher Monitoring (2)

Single UAS Simple Mission Threads

• Launch• Navigation and flight• Search and target ID including evaluation• Target tracking• Return/recovery

Enumeration of these use cases in MBSE models constitutes a primary size input for COSYSMO

15

UAS Mission Cost Comparisons

16

Agenda

17

• Introduction and Background• SysML and Systems Engineering Cost

• System Behavioral Modeling and Software Cost

• Case Study: Remote Targeting System UAS

• System Product Line Architecture Variant Modeling and Product Line Cost with ROI

• Conclusions and Future Work

Monterey Phoenix Overview

• Monterey Phoenix (MP) is an approach to formal software and system specification based on behavior models.

• A view of the architecture model as a high level description of possible behaviors of subsystems and interactions between subsystems.

• The emphasis is on specifying the interaction between the system and its environment.

• The behavior composition operations support architecture reuse and refinement toward design and implementation models.

• Executable architecture models provide for system architecture testing and verification with tools.

• See http://wiki.nps.edu/display/MP

• Run models at http://firebird.nps.edu

18

COCOMO Black Box Model

COCOMO II

product size estimate

product, process, platform, and personnel attributes

reuse, maintenance, and increment parameters

development, maintenance cost and schedule estimates

cost, schedule distribution by phase, activity, increment

19

Behavioral Modeling Method Overview

• Monterey Phoenix models of software systems and user behaviors can indicate software size.

– A subset of system size attributes also available• The methodology extracts an unadjusted function point (UFP)

count from Monterey Phoenix’s executable architecture models for use in software cost estimation.

• The COCOMO II model is used to input the UFP count to determine cost estimates.

• Allows the assessment of architecture design decisions and their cost impacts.

• The methodology was applied to three standardized function point counting examples with empirical data for comparison [Farah-Stapleton 2016].

20

Function Point Analysis - Functionality From User’s Perspective

Identify the Following: • External Inputs (EI): Data that is entering the system• External Outputs (EO) and External Inquiries (EQ): Data that is leaving

the system• Internal Logical Files (ILF): Data that is processed and stored within the

system• External Interface Files (EIF): Data that is maintained outside the system

but is necessary to satisfy a requirement

Function Point Analysis Steps1. Count Data and Transactional Function Types2. Determine UFP Count3. Determine the Value Adjustment Factor4. Calculate final Adjusted FP Count

21

Function Points: • Normalized metric used to

evaluate software deliverables• Measure size based on

well-defined functional characteristics of the software system

• Must be defined around components that can be identified in a well-written specification

Source: International Function Point User Group (IFPUG)

Steps 1 and 2 Result in UFP

Application Boundary

Example MP ModelAnnotated for Function Point Counts

01 SCHEMA It_Is_Tee_Time_EQ 02 ROOT User_GCL: Inquire_on_state_data something_else; 03 Inquire_on_state_data: Click_state_arrow_dropdown Receive_state_list_display;04 ROOT Golfcourses_ILF: Get_result anything_else;05 Get_result: Receive_state_arrow_prompt Send_state_list_display;

06 COORDINATE $x: Inquire_on_state_data FROM User_GCL, 07 $y: Get_result FROM Golfcourses_ILF08 DO

22

Lines 02-05 represent the behaviors of ROOTs (Actors)

Lines 06-17 represent the behaviors of

the COORDINATE for EQ: State Drop Down

MP Calculation• 1 FTR and 1 nested COORDINATE (COORDINATE & 2 ADDs) correspond to 1 FTR and 2 DETs

and a functional complexity weighting of 3• EQ State Drop Down = (1 COORDINATE) * 3 UFP/COORDINATE = 3 UFPs

Line 01 Schema Name for EQ

17 OD;

09 COORDINATE $xx: Click_state_arrow_dropdown FROM $x, 10 $yy: Receive_state_arrow_prompt FROM $y,11 $x11: Receive_state_list_display FROM $x, 12 $y11: Send_state_list_display FROM $y13 DO14 ADD $xx PRECEDES $yy;15 ADD $y11 PRECEDES $x11; 16 OD;

Lines 09 -16 represent

behaviors of nested

COORDINATE, DETs and FTRs. The number of

ADDs determines the weight of this

EQ

• SW

23

MP/COCOMO tool Created by Dr. Ray Madachy, NPS, rjmadach@nps.edu/

Example Effort Estimate

• Sizing Method: Function Points

• UFP: 88

• Nominal options

• Software Development– Elaboration and Construction Phases

– Effort: 16.0 Person-months

– Schedule: 9.2 Months

– Cost: $319,750

• Total Equivalent Size: 4664 SLOC– SLOC = (# UFP) x ( language

SLOC/UFP)

– Java language conversion ratio for UFP to SLOC is 53

– 4664 SLOC = (88 UFP) x (53 conversion ratio)

*SLOC Source Lines of Code

Agenda

24

• Introduction and Background• SysML and Systems Engineering Cost

• System Behavioral Modeling and Software Cost

• Case Study: Remote Targeting System UAS

• System Product Line Architecture Variant Modeling and Product Line Cost with ROI

• Conclusions and Future Work

Remote Targeting System (RTS) Background

• Comparing two architecture variants of Remote Targeting System (RTS) performed by semi-autonomous vehicles

• Autonomous Target Recognition (ATR) variant has heterogeneous sensors, and can use multiple vehicles to auto confirm target declarations without requiring a human.

– Vehicle needs an autonomous target recognition algorithm (ATR)– Vehicle requires a data link between vehicles, in addition to the data link back to the human operator

(which must be modified to accommodate the ATR declarations); – The Plan Mission Use Case must be modified to include loading of target templates– Search Use Case must be modified to accommodate the ATR activities– Additional <<include>> Use Cases, “Perform ATR” and “Confirm Target” must be added– New and modified requirements must be accommodated

• Both variants are modeled in SysML from which cost estimates are directly derived

25

RTS Scenarios (Use Cases)

26

Use Case Level for Costing

27

Perform SurveillanceDescription: This Use Case covers surveillance activitiesPreconditions: Target has been identified and Air Vehicle has entered Surveillance modePrimary Flow: 1. Air Vehicle transmits telemetry to Ground Station(s)2. Ground Station(s) receives and displays flight data3. Ground Station(s) stores telemetry data4. Air Vehicle loiters over target5. Air Vehicle continues video transmission to Ground Station and Off-Board C26. Ground Station(s) receives and displays video transmission7. Operator and Off-Board C2 monitor video and flight data8. Ground station(s) calculate target coordinates based on video and telemetry9. Ground station(s) displays target coordinates10. Operator initiates RTL11. Ground Station sends RTL command to Air Vehicle12. Air Vehicle enters RTL modeAlternate Flow: At any time: a. If bad vehicle health, Operator enters RTL command on Ground Station b. Ground Station sends RTL command to Air Vehicle c. Air Vehicle enters RTL modeAt any time: a. Operator initiates <<include>> Plan Mission Use Case b. Vehicle ingresses to new Search Insertion pointAt any time: a. If vehicle compromise is evident, execute <<include>> Self Destruct Use Case Postconditions: Air Vehicle is loitering over the target for > 10 minutes and target coordinates are calculated and displayed on Ground Station(s); Air Vehicle enters RTL mode

Perform Surveillance MP Model - Primary Flow

28

● This new model is being subjected to function point extraction to estimate software development cost of implementing the use case.

...

Perform Surveillance - Initiate Plan Mission Trace

29

Example RTS Requirements

30

RTS Interfaces (Ports)

31

RTS Baseline Systems Engineering Cost Estimate

32

RTS Full Program Cost Extrapolation

33

RTS Architecture Cost Comparisons

34

Agenda

35

• Introduction and Background• SysML and Systems Engineering Cost

• System Behavioral Modeling and Software Cost

• Case Study: Remote Targeting System UAS

• System Product Line Architecture Variant Modeling and Product Line Cost with ROI

• Conclusions and Future Work

System Product Line Architecture Research Overview

• An MBSE approach has been developed that integrates parametric cost and product modeling methods for economic tradeoff analysis of system product lines.

– Product Line: A set of systems that share a common, managed set of features that satisfy the specific needs of a particular market segment or mission developed from a common set of core assets in a prescribed way.

• The modeling framework includes a reference architecture and cost model for a general combat system product line that is extensible to other DoD and government domains.

• It was applied to assess the economics of Navy combat system product line architecture approaches in coordinated case studies performed by student capstone teams and on individual theses.

• 3-Tier Cruise Missile Defense System [Hall 2018]• Aegis ship class software product line economics [Chance 2019]• ASW cross-domain product line for air, surface, and subsurface applications.

[Manfredo, Jackson-Henderson and Fraine 2019]

• An overall business case analysis as a synthesis of case studies for product line practices was performed with recommendations.

• Results promising for application in the UAS domain using AFIT reference architecture. 36

Product Line Modeling Method Overview

• Describe a general domain model of the given system with common elements

– Combat systems architectures including sensors, weapons, and hardware/software are formally modeled to identify common functions and variations for different case studies.

• Develop a reference product architecture with variation points– Variation points are identified for sensors, HSI / consoles, weapons, and data links with

alternative choices for a combat system product line which also serve as cost model inputs.

• Map existing systems to the reference architecture

• Collect empirical costs and map them to system elements from above

– Empirical cost data from DoD programs is allocated to the system functions in the architecture models to calibrate and populate cost model for specific system configurations.

– Note that small UAS costs may be extensively composed of commercial off-the-shelf hardware

37

Product Line Modeling Method Overview (Cont.)

• Tailor the System Constructive Product Line Investment Model (COPLIMO) framework for the reference architecture or develop new cost models for each application, as necessary.

• Enumerate architecture commonalities to determine percentage of unique, reused, and adapted components.

• Use the portions in System COPLIMO to determine return on investment and Total Ownership Cost of product line approach vs. traditional one-off designs

38

System COPLIMO Black Box Model

39

Example Architecture Flow Diagram

4040

Example Orthogonal Variability Model

4141

Example Product Line Components

4242

Example Cost and ROI Results

• Results for Cruise Missile Product Line

4343

Agenda

44

• Introduction and Background• SysML and Systems Engineering Cost

• System Behavioral Modeling and Software Cost

• Case Study: Remote Targeting System UAS

• System Product Line Architecture Variant Modeling and Product Line Cost with ROI

• Conclusions and Future Work

Conclusions

• Cost models provide a simple framework for performing broader ility and affordability analyses when tied to product-oriented architecture models.

• By developing mapping rules between MBSE models, we demonstrated integrated approaches for architectural analysis, behavior/performance analysis, and cost estimation applied to real DoD mission types.

• There is a strong correspondence between SysML constructs and system size measures of requirements, interfaces, algorithms, and operational scenarios.

– Require additional attributes for modeling complexity levels of size drivers

• The MP simulation language encapsulates function point measures for software size, as well as system size attributes.

• DoD product line architecture variabilities for software and hardware subsystems can be quantified with OVM modeling, and used for costing and investment analysis.

45

Future Work

• Continue expanding the COSYSMO framework for architectural tradeoffs with better fidelity for hardware aspects.

– Further collaboration with AFIT applied to Small UAS (SUAS) domain.

• Model complexity levels of size attributes• Use AFIT SUAS Reference Architecture for product line modeling.

– New empirical data available for Wide Area Search and Surveillance (WASS) software agent implementation:

• Enhance product line cost models for individual component complexities and non-homogeneous change percentages.

• Develop guidelines with examples for practitioners on modeling decomposition levels of detail.

• Collect more empirical data for further calibration and validation of cost models• Data mining of MBSE models for new correlative measures to size and/or cost. 4646

References

• Boehm, Barry, A. Winsor Brown, Ray Madachy, and Ye Yang. 2004. “A Software Product Line Life Cycle Cost Estimation Model.” Proceedings of the 2004 International Symposium on Empirical Software Engineering.

• Maj. Zhongwang Chua (Singapore AF), “Application of Executable Architecture in Early Concept Evaluation using DoDAF”, M.S. Thesis, AFIT, September 2016

• K. Chance, “Naval Combat Systems Product Line Economics: Extending the Constructive Product Line Investment Model for the Aegis Combat System”, MS Thesis, Naval Postgraduate School, June 2019

• CPT Dennis Edwards (USArmy), “Exploring the integration of COSYSMO with a model based system engineering methodology in early trade space analytics and decisions”, M.S. thesis, NPS, June 2016

• Monica Farah-Stapleton, “Resource Analysis Based On System Architecture Behavior”, Ph.D. thesis, NPS, June 2016

• Hatley, Derek, Peter Hruschka, and Imtiaz Pirbhai. 2000. "Process for System Architecture and Requirements Engineering." New York, NY: Dorset House Publishing.

• LT Hall, R., “Utilizing a Model Based Systems Engineering Approach to Develop a Combat System Product Line,” MS Thesis, Naval Postgraduate School, June 2018

47

References (Cont.)

• D. Jacques and R. Madachy, “Model-Centric UAV ISR Analysis,” presented at Systems Engineering Research Center, 7th Annual SERC Sponsor Research Review, Washington, DC, December 3, 2015.

• R. Madachy, Systems Engineering Cost Estimation Workbook, Naval Postgraduate School, October 2015

• R. Madachy, J. Green, , "Naval Combat System Product Line Economics," Proceedings of the 16th Annual Acquisition Research Symposium, Monterey, CA, May 2019

• V. Manfredo, T. Jackson-Henderson, N. Fraine, "Naval ASW Combat System Product Line Architecture Economics," MS Capstone, Naval Postgraduate School, June 2019

• Peak, R.S. and Lane, J.A., “SysML Building Blocks for Cost Modeling: Towards Model-Based Affordability Analysis”, INCOSE International Workshop (IW14), Torrance, California, 2014

• Maj. Ryan Pospisal (DTRA/A9, Kirtland AFB), “Application of Executable Architectures in Early Concept Evaluation”, M.S. thesis, AFIT, December 2015

• SERC, "Valuing Flexibility Phase II." A013 Final Technical Report SERC-2012-TR-10-2, 2012

• SERC, “System Qualities Ontology, Tradespace, and Affordability (SQOTA) Phases 1-7”, SERC-2019-TR-012-S, 2019

48

SERC Talks December 11, 2019 4

Thank you for joining us!Please check back on the SERC website for today’s recording

and future SERC Talks information.

We’ll be exploring Autonomy and Trust through Spring 2020

Happy Holidays! We look forward to seeing you in the new year!

Subscribe and follow SERC on our social channels: