measures, metrics, and systems-of-systems

Measures, Metrics, and Systems-of-Systems

Bridging a Gap Between Academic and DoD Systems Engineering Terminology

Dr. Thomas Holland, Ph.D.Naval Surface Warfare CenterDahlgren, VA

NSWCDD-PN-14-00163 Public Release Statement A: Approved for Public Release

O. T. Holland NSWCDD-PN-14-00163 SEDC 2014, April 3-5

Overview

Measure or Metric? Systems Perspective and the DoD MOEs & MOPs Systems vs. Systems-of-Systems Military Definitions of SoS In Conclusion

2Public Release Statement A: Approved for Public Release


Measure or Metric?

“By measuring only the amount of merchant tonnage sunk, they did not measure the progress of their strategy in a holistic manner and lost sight of their ultimate strategic goal. In addition to shipping sunk, the planners should have determined a method to measure not only British public opinion, but also grain supply levels and the availability of bread. Another metric worthy of measure might have been American war intentions. These additional metrics assess other aspects of translating their strategy into the desired end state.”



Measure or Metric?




Measure or Metric?

Measure: a dimension, quantity, or capacity (degree), ascertained by comparison to an accepted standard. • length in inches• weight in pounds• temperature in degrees Celsius

(Oxford Dictionary, 2013)



Measure or Metric?

Metric: a technical system or standard of measurement; a set of figures or statistics that measure results. • Productivity: number of sales calls an employee

makes per day.• Efficiency: probability of correct data entry.

(Oxford Dictionary, 2013)



Systems Perspective & the DoD

In SE, “metric” refers to performance.• “The SoS metrics are collected and analyzed as

part of analyses to assess whether the SoS is making progress towards objectives.” (OAD, 2008)

• Despite these positive indications, the United States has not declared victory against terrorism nor has the U.S. government satisfied world or domestic opinion with explanations of forward progress in the war. Metrics must be developed to give meaning to our military actions.” (Walker, 2005)



Systems Perspective & the DoD We can measure many things. But there are certain measures and collections

of measures that are important within the context of our interest.

The right metric helps us answer our specific questions.



Systems Perspective & the DoD

Metric: a set of measures or methods that ascertain the progress a system is making toward achieving its goal.

Did you know, “Metric” is not defined in the DAU Glossary?



MOEs and MOPs

Measure of Effectiveness (MOE): “The data used to measure the military effect (mission accomplishment) that comes from the use of the system in its expected environment. That environment includes the system under test and all interrelated systems, that is, the planned or expected environment in terms of weapons, sensors, command and control, and platforms, as appropriate, needed to accomplish an end-to-end mission in combat.” (Defense Acquisition University, 2013)



MOEs and MOPs

Measure of Performance (MOP): “System-particular performance parameters such as speed, payload, range, time-on-station, frequency, or other distinctly quantifiable performance features. Several MOPs may be related to the achievement of a particular Measure of Effectiveness (MOE).”

(Defense Acquisition University, 2013)



MOEs and MOPs

Public Release Statement A: Approved for Public Release 15


Systems vs. Systems-of-Systems

System: any interacting or interdependent group of entities that forms a unified and purposeful whole.

System-of-systems: any adaptive, emergent, scalable, and autonomous system where the behavior changes continuously in response to influence of stakeholders and is composed of systems that are themselves autonomous.

(Fisher, 2006)






(Fisher, 2006)





Traditional Monolithic Systems Systems-of-Systems

Centralized Control Independent Decisions

Brittle (linear) Adaptive (complex)

Behaviors Specified Behaviors Emerge

Closed System Constraints Unbounded Constraints

Hierarchical Structures (Components) Autonomous Decisions (Constituents)

Components may have Global Visibility Constituents have Local Visibility

Tight Coupling between Components Dynamic Interactions between Constituents

Components are Integrated Constituents are Interoperable


Military Definitions of SoS

System: A functionally, physically, and/or behaviorally related group of regularly interacting or interdependent elements; that group of elements forming a unified whole.




System-of-systems: a set or arrangement of systems that results when independent and useful systems are integrated into a larger system that delivers unique capabilities. Both individual systems and SoS conform to the accepted definition of a system in that each consists of parts, relationships, and a whole that is greater than the sum of the parts; however, although an SoS is a system, not all systems are SoS.





SoS: Context not entirely in original systems’ contexts

SoS may exclude some of the original context

FOS: Shared Context of systems operating independently

Context for A

Context for B

SoS makes some of the original context internal



AcademicSystem: any interacting or

interdependent group of entities that forms a unified and purposeful whole.

DoDSystem: a functionally,

physically, and/or behaviorally related group of regularly interacting or interdependent elements; that group of elements forming a unified whole.




AcademicSystem-of-systems: any adaptive,

emergent, scalable, and autonomous system where the behavior changes continuously in response to influence of stakeholders and is composed of systems that are themselves autonomous.

DoDSystem-of-systems: a set or arrangement

of systems that results when independent and useful systems are integrated into a larger system that delivers unique capabilities. Both individual systems and SoS conform to the accepted definition of a system in that each consists of parts, relationships, and a whole that is greater than the sum of the parts; however, although an SoS is a system, not all systems are SoS.



In Conclusion Academic definitions of Systems, Systems-of-Systems,

Measures, and Metrics have Military Domain Counterparts.

A working knowledge of the terms are critical to bridge the audiences when addressing different communities or domains.

Do not introduce new terminology, or redefine existing terms, if it is already authoritatively defined.

Introduce new terms in light of new concepts only if necessary.



In Conclusion

Original“By measuring only the amount of merchant tonnage sunk, they did not measure the progress of their strategy in a holistic manner and lost sight of their ultimate strategic goal. In addition to shipping sunk, the planners should have determined a method to measure not only British public opinion, but also grain supply levels and the availability of bread. Another metric worthy of measure might have been American war intentions. These additional metrics assess other aspects of translating their strategy into the desired end state.”

(Walker, 2005)

Revised“By using only the MOP of shipping sunk, they were not able to ascertain the progress of their strategy in a holistic manner and lost sight of their ultimate strategic goal. In addition to shipping sunk, the planners should have developed an MOE representative of the relationship between British public opinion, grain supply levels, and the availability of bread. Another useful MOE might have assessed American war intentions. These MOEs would have assessed the extent to which their capability achieved the desired end state.”



Thank You!

Thomas Holland, Ph.D.Naval Surface Warfare Center

Dahlgren, VA



References ASN(RDA) Chief Systems Engineer, 2006. Naval “Systems of Systems” Systems

Engineering Guidebook, Vol. II. Defense Acquisition University, 2013. Glossary of Defense Acquisition Acronyms

and Terms. [Online] Available at: https://dap.dau.mil/glossary/Pages/GlossarySearch.aspx?rec=MOE

Fisher, D. A., 2006. An Emergent Perspective on Interoperation in Systems of Systems, Pittsburgh: Carnegie-Mellon Software Engineering Institute.

Maier, M. W., 1998. Architecting Principles for Systems-of-Systems. In: Systems Engineering. pp. 267-284.

Meyers, B. C., Monarch, I. A., Levine, L. & Smith, J. D., 2005. Including Interoperability in the Acquisition Process, Pittsburgh: Carnegie Mellon University, Software Engineering Institute.

Moffat, J., 2003. Complexity Theory and Network Centric Warfare, Washington, DC: DoD Command and Control Research Program.



References continued

Office of the Deputy Under Secretary of Defense for Acquisition and Technology, Systems and Software Engineering, 2008. Systems Engineering Guide for Systems of Systems. Version 1.0 ed. Washington, DC: ODUSD(A&T)SSE.

Office of the Deputy Under Secretary of Defense for Acquisition and Technology, 2008. Systems and Software Engineering. Office of the Deputy Under Secretary of Defense for Acquisition and Technology.

Oxford, 2013. Oxford Dictionaries. Available at: http://oxforddictionaries.com/ Shah, N. B., Rhodes, D. H. & Hastings, D. E., 2007. Systems of Systems and

Emergent System Context. Hoboken, NJ Smith, J. D., 2006. Topics in Interoperability: Structural Programmatics in a System

of Systems, CMU/SEI-2006-TN-037, Pittsburgh: Carnegie Mellon Software Engineering Institute .

Walker, D. H., 2005. Developing Metrics in the Global War on Terrorism, Newport, RI: Naval War College.


measures, metrics, and systems-of-systems

Documents

public release statement

metric worthy of measure

british public opinion

holland nswcddpn

additional metrics

lost sight

holistic manner

supply levels