ilities in the acqoutsi i i n of dod systems › library › arj › arj › arj87 › arj87... ·...

This article examines Department of Defense (DoD) systems, both retired and extended, to determine what nonfunctional attributes, specified as iltiies, the two groups of systems exhibit to determine if the decision to retire a system is based on these attributes. The research applies a grounded theory approach to determine what attributes extended DoD systems exhibit and compares these findings to retired systems. Research showed that the DoD systems extended beyond their planned service life demonstrate the ilities of extensibility, flexibility, interoperability, versatility, and robustness. The research also explores the application of social network analysis tools and metrics to a network of U.S. Air Force systems to understand if these tools can quantify the ilities of interoperability and versatility. For this analysis, a data set of 136 systems provided the basis for a network of systems that examines shared resources (financial, physical, and information) as well as shared operational activities.

DOI: https://doi.org/10.22594/dau.18-799.26.01 Keywords: Grounded Theory Approach, Social Network Analysis, Systems Engineering, illities, Weapons Acquisition

IDENTIFYING AND QUANTIFYING Critical ilities in theACQUISITION of DoD Systems

LTC James R. Enos, USA, John V. Farr, and Roshanak R. Nilchiani

Image designed by Michael Krukowski

20 21Defense ARJ, January 2019, Vol. 26 No. 1 : 18-43 Defense ARJ, January 2019, Vol. 26 No. 1 : 18-43

Identifying and Quantifying Critical ilities in the Acquisition of DoD Systems https://www.dau.mil January 2019

The portfolio of systems the Department of Defense (DoD) has developed over the past 30 years has been a challenge for the DoD when making the decision whether to retire or extend a system. In 1996, then-Vice Chairman of the Joint Chiefs of Staff ADM William A. Owens proposed warfighting capability would be more reliant upon system-of-systems and network-cen-tric operations (Owens, 1996). As such, DoD systems are becoming more and more interconnected and reliant upon other systems to provide capability to the user, making the decision to retire a system much more complex. Often, individual systems on a battlefield cross Service boundaries, making collab-oration difficult in traditionally hierarchal military structures (Dahmann & Baldwin, 2008). Because of the emergence of network-centric warfare, the Government Accountability Office (GAO) found the DoD lacked methods and tools for conducting portfolio management at the enterprise level for capabilities. Additionally, they noted gaps in the DoD’s ability to identify, understand, and assess the capability portfolio (GAO, 2015). Specifically, the DoD lacks a method or approach to understand legacy systems and when they should be retired versus upgraded to improve the capabilities of an existing system (GAO, 2015).

Over the past several decades, the DoD demonstrated significant failures when acquiring new weapon systems to replace

its legacy systems. Each of the Services has expe-rienced these failures, resulting in the DoD’s

loss of billions of dollars without a deliv-ered capability. In 2004, the Army

finally cancelled the Comanche helicopter after 22 years

of development and s p en d i n g a r ou n d

$6.9 billion (Ward, 2012). The DoD attri-

butes the Coma nche program’s cancellation to

requirements creep, but they intended the helicopter to be used

on a conventional battlefield for a narrow purpose that was never realized (Magnuson,

2011). One of the most expensive failed systems is the Army’s Future Combat Systems (FCS) program on

which the Army spent just under $20 billion without fielding any systems (Freedberg, 2012).

The Army is not alone in these failures. In 1991, the Navy cancelled the A-12 stealth bomber program, which cost them about $1.3 billion and 23 years of litigation (Thompson, 2014). The Marines cancelled their Expeditionary Fighting Vehicle (EFV) in 2011 after spending $3.3 billion in development costs due to delays and cost overruns (Rodriquez, 2014). Additionally, the Air Force cancelled the Combat Search and Rescue program (CSAR-X) and opted to modify the Army’s proven UH-60M helicopter into a CSAR version designated the HH-60M (Warwick, 2013). The DoD at large cancelled its Transformation Satellite Communications System (TSAT) in 2009 as it had redundant capability with the Advanced Extremely High Frequency (AEHF) satellites at a price tag of $3.2 billion (Rodriquez, 2014).

The DoD intended to replace legacy systems with these failed systems as they neared the end of their service life. However, these legacy systems continued to provide value to DoD warfighters on the battlefield in the 2000s. The Army planned to replace the OH-58D, a Vietnam-era helicopter, with the Comanche program; and the Abrams, Bradley, and M-113 tracked vehicles with the FCS program (Freedberg, 2012; Magnuson, 2011). When the DoD terminated these programs, they effectively forced the Services to extend the legacy systems; however, upon further investigation, these legacy systems possess attributes that may have led to a successful service life extension. By understanding these attributes, it may be possible to inform the retirement decision and development of a new system prior to expending billions of dollars. With unpredictable and constrained budgets, understanding how these ilities positively affect the life cycle of a system may also lead to better design and procurement decisions.

Figure 1 presents a systemigram of the problem domain the DoD currently faces and provides an overview of the methods and tools used to explore the problem. Systemigrams generate a rich picture of the system and the prob-lem as a means to convey the complexities of systems and their environment (Boardman & Sauser, 2008). In this case, the DoD manages systems that age to a certain point, triggering a retirement decision where the DoD might not base its decision solely on the system’s age. This article explores the first hypothesis—that retired and extended systems have different system

Over the past several decades, the DoD demonstrated significant failures when acquiring new weapon systems to

replace its legacy systems.



attributes as defined by systems engineering ilities. The second hypothesis describes how social network analysis (SNA) quantifies two of these crit-ical ilities to assist the DoD with this decision. Future work will continue to explore and validate these two hypotheses and determine how they can contribute to a decision model for retiring a system.

FIGURE 1. SYSTEMIGRAM OF THE PROBLEM

Departmentof Defense

(DoD)

DoD SystemRetirementDecisions

DoD Systems

PlannedRetirement

Date

Retired Systems

Extended Systems

WarfighterNeeds

Critical ilities

SystemsEngineering

(SE)

SE Literature

System ofSystems

EngineeringSystemArchitecture

MathematicsLiterature

NetworkAnalysis

Social Network Analysis

Betweenness

Degree Centrality

EigenvectorCentrality

LiteratureGap

TheoreticalReal System

Age

interoperability

versatilityrobustness

LegendStart/End Node

Mainstay

Hypothesis 1

Hypothesis 2

Hypothesis 3

F-117 F-14 E/A-6B

CH-46 OH-58D

B-52 A-10 M-1A2

UH-60 E-2E

retir

ed

include

manages

deve

lop creates includes

includes

includes

include

meet

age

exhibit

exhi

bit

filled by

provides basis for

triggers

identify

build

lack

quantify

influence

exhibit

cont

inue

to m

eet

usesapplies

Literature ReviewA review of the relevant literature, including a discussion of systems

engineering with an emphasis on the nonfunctional attributes of systems, found these attributes of systems have become incredibly important to the design of systems. Systems engineers have defined ilities to represent these attributes. These attributes become the foundation of the grounded theory approach to understand how extended DoD systems share common attributes. Additionally, the literature on SNA identifies several metrics to quantify various attributes of nodes within a social network. Likewise, the SNA methodology provides a means to quantify a subset of these critical ilities to assess legacy DoD systems.

Systems EngineeringAs a discipline, systems engineering faces increasing complexity of sys-

tems as technology progresses and systems become more interconnected. The International Council on Systems Engineering (INCOSE) defines systems engineering as—

An interdisciplinary approach and means to enable the realization of successful systems. Systems engineering focuses on def ining customer needs a nd required functionality early in the development cycle, documenting requirements, and then proceeding with design synthesis and system validation while considering the complete problem (INCOSE, 2007, Appendix C-5).

Systems engineers differ from traditional engineers because they consider the system in its entirety, lead the conceptual design of systems, and bridge the gaps between traditional engineering (Kossiakoff, Sweet, Seymour, & Biemer, 2011). Buede (2000) writes that systems engineers translate customer and stakeholder needs into system requirements and design the system by allocating these requirements to functions and, eventually, the physical aspects of the system. Rouse (2005) describes how systems engi-neers decompose a system of interest, both functionally and physically, design these decomposed subsystems and components, then combine them to achieve the overall function of the system. Although systems engineer-ing has progressed significantly over the last century, it remains focused internally to a system.

Systems engineers have adopted the ilities as a construct for understanding nonfunctional attributes of systems and the complex interactions between systems. As a result, systems engineers have begun to recognize the critical-ity of these nontraditional design criteria and have begun to include them in the design of systems (McManus, Richards, Ross, & Hastings, 2009).



However, these properties and attributes of a system often emerge after engineers have designed and put the system into operation (de Weck, Ross, & Rhodes, 2012). The literature identifies dozens of ilities; Table 1 presents definitions from the literature for a subset of the ilities.

TABLE 1. ILITY DEFINITIONS

ility Definition Sources

Affordability“the ability to allocate resources out of a future total budget projection to individual activities”

(DoD, 2013, p. 96)

Extensibility “the ability to accommodate new features after design”

(de Weck, Ross, & Rhodes, 2012, p. 6)

Flexibility“the ability of a system to be changed by a system-external change agent”

(McManus, Richards, Ross, & Hastings, 2009, p. 3)

Interoperability “the ability to effectively interact with other systems”

(de Weck et al., 2012, p. 6)

Quality“ability to deliver requirements at a ‘high’ level, as perceived by people relative to other alternatives that deliver the same requirements”

(Engineering Systems Division Symposium Committee, 2002, p. 5)

Robustness

“the ability of a system to maintain its level and set of specification parameters in the context of changing system external and internal forces”

(McManus et al., 2009, p. 3)

Versatility“the ability of a system to satisfy diverse expectations on the system without the need for changing form”

(McManus et al., 2009, p. 3)

Social Network AnalysisOver the past decade, the DoD successfully applied SNA to analyze,

visualize, and understand complex networks of terrorists and individuals of interest in both Iraq and Afghanistan. Most notably, the DoD and the Central Intelligence Agency (CIA) used SNA to identify the location of Osama bin Laden (Schmidle, 2011). For years, special operations forces searched the mountains of Afghanistan and Pakistan trying to locate the most wanted man in the world. Finally, a CIA analyst identified a courier within the bin Laden network and the DoD was able to pinpoint his location and execute a daring night raid against the leader of al-Qaeda (Schmidle, 2011). However, the DoD has not used these methods and techniques to look inward at their complex network of systems to better understand how their network of sys-tems has emerged over the past several decades (Enos & Nilchiani, 2017b).

SNA is a specific application of network analysis sociologists use to ana-lyze networks in which the nodes are individuals, groups, or organizations. These actors share interests, social contacts, membership in organiza-tions, participation in events, family ties, or financial ties (Serrat, 2010). SNA models represent a network’s nodes (people) and edges (relationship), and allow for the calculation of metrics such as degree centrality, closeness, and betweenness that provide valuable information about an individual node (Freeman, 2004). One distinction of SNA is the emphasis on rela-tionships between the nodes as opposed to an individual node’s attributes (Hanneman & Riddle, 2005).

An important aspect of SNA is that researchers can merge multiple rela-tionship types into one network to identify hidden relationships and provide insight into the underlying social structure. White, Boorman, & Breiger (1976) discuss a method to determine social structure from multiple net-works that independently do not reveal the true social structure, but when aggregated begin to display the underlying network. Similar to network analysis, SNA faces several challenges including dynamic networks with fuzzy boundaries and an incomplete data set for the network (Everton, 2012). However, SNA overcame some of these challenges through network analysis algorithms and advanced computer process and visualization to incorporate thousands of nodes and relationships.

MethodologyThe proposed methodology applies grounded theory to identify critical

ilities and SNA to quantify two of these ilities. First, grounded theory pro-vides a method to analyze multiple DoD systems that the DoD has extended well beyond their initial retirement date along with several retired DoD systems. This approach uses systems engineering ilities to code literature describing DoD systems of interest to determine if retired and extended systems possess different attributes. Second, the SNA methods and metrics can generate the emergent DoD network of systems and provide quantifiable metrics for these attributes. Specifically, the centrality metrics from SNA quantify the interoperability and versatility of an individual system within the network of DoD systems.

As a discipline, systems engineering faces increasing complexity of systems as technology progresses and systems

become more interconnected.



Grounded TheoryGlaser and Strauss (1967) introduced grounded theory as a research

method for developing a hypothesis from systematically obtained and ana-lyzed data. One of the unique characteristics of grounded theory is the iterative process of simultaneously collecting and analyzing data so the researcher does not wait to begin analysis until the entire data set is col-lected (Cho & Lee, 2014). Grounded theory uses a combination of induction and abduction to develop a hypothesis. Induction begins with a study of several cases and the development of conceptual categories-based pat-terns contained within the cases (Thornburg & Charmaz, 2011). Whereas in abduction, researchers start with an initial hypothesis based on a few observations and then seek to strengthen their hypothesis with empirical evidence similar to inductive reasoning (Timmermans & Tavory, 2012). Although rooted in the social sciences, researchers in healthcare, informa-tion systems, and even engineering have begun to use aspects of grounded theory to support their research.

One of the fundamental aspects of grounded theory is coding the underlying data in a meaningful manner to generate a hypothesis. Grounded theory necessitates coding data because the research often focuses on qualitative, textual data a researcher must interpret prior to analyzing (Birks & Mills, 2015). Coding is the process for naming segments of data to categorize, sum-marize, and account for data identified during the research (Thornburg & Charmaz, 2011). A researcher codes segments of text, then combines these codes into clusters and groups; and in pure grounded theory, the hypothe-sis emerges from these clusters of data (Fendt & Sachs, 2008). Grounded theory consists of three main phases of coding: open coding—data concep-tualization and categorization; axial coding—grouping the data based on observed patterns; and selective coding—identifying the core phenomenon (Starks & Brown-Trinidad, 2007). As researchers analyze new data, they organize it against the developed categories through constant comparison, a fundamental process in grounded theory (Urquhart & Fernandez, 2013).

The researcher continues this until they reach the point of theoretical sat-uration when the grounded theory emerges from the coded and categorized data (Glaser & Strauss, 1967).

Social Network Analysis MetricsSNA metrics range from individual nodal metrics to metrics for the

entire network of actors, which assists in the understanding of a social net-work. Individual centrality metrics include the degree centrality, closeness, betweenness, and eigenvector centrality. These metrics determine how well a node connects to the network through the relationships and assist in understanding how important a node is within the network. Based on the findings of the grounded theory portion of this article, SNA metrics quantify the ilities of interoperability and versatility.

Sociologists base degree centrality on the notion that actors with the most ties to other actors must be important within the network (Wasserman & Faust, 1994). Graphically, these would be nodes with the most edges extending from them to other nodes within the network. Mathematically, in a network the degree centrality is equal to the number of connections, d:

Cd (ni ) = d(ni ) (1)

Researchers use this formula to identify active actors in the network who connect to several other actors (Faust, 1997). In a network of systems, the degree centrality could provide an assessment of the interoperability or ver-satility of a system based on the relationships to other systems. Generally, a DoD system is considered interoperable if it can enter into the DoD’s network, but this fails to consider or account for physical interoperability or resource flows. However, in a network of systems, the degree centrality metric could determine how connected the individual system is to the over-all network, and thus determine the true interoperability of a system (Enos & Nilchiani, 2017c). When shared operational activities define the edges of a network, the versatility of a system becomes evident.

Another measure of centrality is the closeness metric, which views an actor’s centrality based on the distance to all other nodes in the network (Wasserman & Faust, 1994). For this metric to be meaningful, the nodes in the network must connect to the node of interest. Mathematically, closeness centrality in the network described above is calculated by:

C’ c(ni ) = g–1

(2)∑ d(ni ,nj )

gj=1



This provides insight into the speed at which information or resources travel along a network between the different nodes. This can be used to identify actors who may receive information in a gossip network first or identify companies that may be first to develop and bring a product to mar-ket (Borgatti, 2005). For systems engineers, closeness centrality could help determine information flows across the network or the vulnerabilities in the network. In an infrastructure network, it could identify and prioritize nodes to repair, enabling quick restoration of power to the entire grid (Enos, Mansouri, & Nilchiani, 2017).

The third SNA metric that may provide value to systems engineers in their analysis of networks of systems is the betweenness centrality. The betweenness metric identifies situations in which interactions between two nonadjacent nodes in the network depend on another actor—referred to as a bridge—in the network (Wasserman & Faust, 1994). Going back to the bin Laden example, the identification of the courier who relayed messages between bin Laden and other high members of al-Qaeda was the key to find-ing bin Laden (Schmidle, 2011). Several assumptions go into the calculation of betweenness, but the metrics capture how communication in the network will follow any path and uses the probability of a communication between two nodes. Mathematically, in a network, gjk is the number of edges linking j and k; the betweenness is:

CB (ni ) = ∑ j<k gjk (ni )/gjk (3)

SNA uses the betweenness metric to identify influential actors that affect the flow of information between groups of actors in a network (Faust, 1997). In a network of systems, identifying bridges with a high betweenness score could identify systems critical to the operation of the overall network. These bridge systems may represent systems engineers must harden to develop a robust network of systems (Enos & Nilchiani, 2017b). Additionally, these nodes in the network may require additional attention if they fail, as in an infrastructure network, as they will return service to the most number of customers (Enos, Mansouri, & Nilchiani, 2017).

Finally, eigenvector centrality is similar to degree centrality as it uses the number of connections from a node, but it weights a node’s connections based on the connected node’s importance in the network (Everton, 2009). If a node only connects to a few, but highly connected nodes, its eigenvector centrality will be higher than its degree centrality. Mathematically, the eigenvector centrality metric is the eigenvector of the network’s adjacency matrix (Everton, 2012):

λv = Av (4)

The concept behind this metric is nodes that are only adjacent to one or two other nodes may be important to the overall network if the adjacent nodes are highly connected (Everton, 2012). In a network of systems, a system may only provide information to one other highly connected node, so it will have a low degree centrality, but may be important to the network.

Application to DoD SystemsThis section applies the two portions of the methodology to identify

critical ilities for DoD systems and then leverages SNA metrics to quantify two of these metrics. The grounded theory approach to identifying critical ilities uncovered how systems the DoD has extended beyond their planned life cycle demonstrate several common ilities not present in systems they have retired. This provided the basis for selecting ilities to quantify with SNA metrics, specifically interoperability and versatility. The portion of this work focused on quantifying these ilities is not yet validated, but the results appear consistent with common perceptions of the DoD systems as a whole.

Identifying Critical IlitiesThe authors used grounded theory application to examine a set of articles

on 10 DoD systems to identify common attributes of these systems and build a theory on why the DoD retires certain systems and extends others. This work builds upon previous work that examined only the B-52 and the F-117, but the findings are consistent with work and identify several critical ilities for DoD systems (Enos & Nilchiani, 2017a). Based on an examination of the 10 DoD systems, the decision to retire a system is statistically dependent on the nonfunctional attributes of the system. Table 2 presents a summary of the nonfunctional attributes the literature uses to describe operational systems including the B-52, A-10, M-1, UH-60, and E-2 as well as the retired systems including the F-117, F-14, OH-58D, CH-46, and E/A-6B. For each system, a sample of 20 articles provided the data and the ilities served as the coding for the grounded theory approach. The literature describing active, extended systems (B-52, A-10, M-1, UH-60, and E-2) mentions extensibility, flexibility, interoperability, robustness, and versatility much more frequently than the literature describing the retired systems. Literature often describes the quality of the F-117, F-14, OH-58D, CH-46, and E/A-6B; however, it rarely uses other ilities to describe these systems. It is likely these systems lacked critical attributes, which influenced the DoD’s decision to retire the F-117, F-14, OH-58D, CH-46, and E/A-6B.



To establish a statistically significant difference in the two groups, the chi-squared test evaluates categorical data to determine if the decision to retire a DoD system is dependent on the ilities of the system. The chi-squared test determines if two categories of data are independent of each other or, alter-natively, if the two groups are dependent upon each other (Kiemele, Schmidt, & Berdine, 2000). In this case, the p-value of the test is extremely low, so we reject the null hypothesis and can state that retired systems and extended systems have a statistically significant difference in their ilities. Further work will be required to extend this analysis beyond just these DoD systems to capture additional data points for other retired and extended DoD systems to build additional confidence in the analysis.

Quantifying Interoperability The first network follows traditional SNA analysis and examines direct

links between systems to determine the interoperability of systems within the network. In traditional systems engineering, engineers would identify this through a design structure matrix, similar to an adjacency matrix (Enos & Nilchiani, 2017b). The U.S. Air Force’s (USAF) fact pages and Gallery of Weapons provided the source for the data in this network to include the connections between systems (Church, 2015; U.S. Air Force, 2018). Figure 2 presents the interoperability network for the 136 USAF systems. The dia-gram uses the Fruchterman-Reingold algorithm for organizing nodes with a force-directed algorithm to arrange the nodes based on their connections, generating a force between the two nodes (Fruchterman & Reingold, 1991). Upon initial visual inspection, the Global Positioning System III (GPS III),

which provides positioning, timing, and navigation data, stands out as highly connected to other systems. Along the edge of the network, systems are less connected and only have a degree centrality of one or two.

FIGURE 2. USAF INTEROPERABILITY NETWORK

TABLE 2. CRITICAL ILITIES FOR DOD SYSTEMS

Status Service Sources Affordability Extensibility Flexibility Interoperability Robustness Versatility Quality Other

B-52 Stratofortress Active USAF 20 1 9 5 7 4 10 11 4

A-10 Thunderbolt II Active USAF 20 7 7 3 7 13 12 15 3

M-1 Abrams Active USA 20 1 10 10 7 12 7 10 8

UH-60 Black Hawk Active USA 20 2 11 9 7 3 14 10 4

E-2 Hawkeye Active USN 20 0 7 10 13 3 11 13 2

Total Active Systems 11 44 37 41 35 54 59 21

F-117 Stealth Fighter Retired USAF 20 2 1 1 2 3 0 16 10

OH-58D Kiowa Retired USA 20 2 6 10 2 1 6 3 9

F-14 Tomcat Retired USN 20 1 3 3 4 2 9 8 5

E/A-6B Prowler Retired USN 20 0 3 4 14 1 6 11 2

CH-46 Sea Knight Retired USMC 20 1 3 11 0 0 7 16 5

Total Retired Systems 6 16 29 22 7 28 54 31



Interoperability focuses on the ability of a system to interact with other sys-tems and provide to, or receive resources from, those other systems; degree centrality aligns with this description of interoperability. Table 3 presents the top 10 systems by their degree centrality, or their direct links to another system in the network. The GPS III system, not surprisingly, is the highest ranking system with a degree centrality of 44—consistent with the visual depiction of the network. The next three systems in the list are the tanker aircraft, which rank very high as nearly every other aircraft in the USAF’s inventory relies on them for air-to-air refueling. The B-52, B-1, F-15E, and A-10C all attribute their high-degree centrality scores to the wide variety of weapons each of these aircraft has the ability to employ. Finally, the Link-16 system is a command and control system used to distribute tactical infor-mation between various systems in the network.

TABLE 3. TOP 10 USAF SYSTEMS RANKED BY DEGREE CENTRALITY

System Degree Closeness Betweeness Eigenvector

GPS III Satellite 44 0.5081 3251.6 1

KC-135 Stratotanker 30 0.4701 1134.1 0.9966

KC-10 Extender 28 0.4549 721.3 0.976

KC-46 Pegasus 28 0.4549 721.3 0.976

B-52H Stratofortress 23 0.4615 1040.6 0.6856

B-1B Lancer 22 0.4468 893.5 0.6531

C-130J Hercules 17 0.3663 264.4 0.5896

Link-16 17 0.4118 804.8 0.5326

F-15E Strike Eagle 16 0.4286 548.2 0.533

A-10C Thunderbolt II 15 0.4271 403.3 0.5532

Previous work evaluated two aspects of eigenvector centrality as a measure of interoperability: first, the direct measurement of a system’s eigenvector centrality; and second, examining systems with a low-degree centrality, but a relatively high eigenvector centrality (Enos & Nilchiani, 2017c). Table 4 presents the 10 highest ranking systems according to the eigenvector centrality, which takes into account the importance of the nodes to which an individual node is connected in addition to the quantity of links in the

network (Golbeck, 2013). This list is almost the same as the degree central-ity list so it initially appears eigenvector centrality is not as important to determining the interoperability of a system.

TABLE 4. TOP 10 USAF SYSTEMS RANKED BY EIGENVECTOR CENTRALITY


GPS III Satellite 44 0.5081 3251.6 1

KC-135 Stratotanker 30 0.4701 1134.1 0.9966

KC-10 Extender 28 0.4549 721.3 0.976

KC-46 Pegasus 28 0.4549 721.3 0.976

B-52H Stratofortress 23 0.4615 1040.6 0.6856

B-1B Lancer 22 0.4468 893.5 0.6531

C-130J Hercules 17 0.3663 264.4 0.5896

A-10C Thunderbolt II 15 0.4271 403.3 0.5532

F-15E Strike Eagle 16 0.4286 548.2 0.533

Link-16 17 0.4118 804.8 0.5326



To expand the potential for eigenvector centrality to measure the interop-erability of a system, Table 5 examines the top 10 systems, with a degree of one, by their eigenvector centrality. In this case, the evaluation provides some interesting insights into a system’s interoperability based on the eigenvector centrality. First, the GPS OCX, which monitors and controls the GPS III satellites, has a relatively high eigenvector centrality despite the degree centrality of one. Additionally, the AN/ARC-210 BLOS Satcom enables tactical aircraft to connect to satellite communications networks. So these types of systems, which may seem to lack interoperability via the degree metric, benefit from an analysis of their eigenvector centrality.

TABLE 5. TOP 10 USAF SYSTEMS WITH A DEGREE CENTRALITY OF 1 RANKED BY EIGENVECTOR CENTRALITY


C-145 Skytruck 1 0.3378 0 0.0887

GPS OCX 1 0.3378 0 0.0887

B61-11 Nuclear Bomb 1 0.3096 0 0.057

B61-7 Nuclear Bomb 1 0.3096 0 0.057

Mk 65 Sea Mine 1 0.3096 0 0.057

AN/ARC-210 BLOS Satcom 1 0.3007 0 0.0463

AGM-88 HARM 1 0.2979 0 0.0449

BLU-111 Bomb 1 0.2883 0 0.0299

BLU-113 Bomb 1 0.2806 0 0.0187

BLU-120B 1 0.2727 0 0.0129

Quantifying VersatilityThe second network applies techniques developed for dealing with dark

networks and applies SNA methods to examine other relationships between actors in a network (Everton, 2012). For this network, links represent con-nections between systems that can perform the same function and the metrics represent the versatility of a system. As in the dark network analysis, when these relationships combine with the direct links, additional insights may emerge. Figure 3 displays the network consisting of the 136 systems and 55 operational activities. Versatility captures the ability of a system to meet a range of stakeholder expectations without changing physical form

(McManus et al., 2009). In the network, versatility represents the USAF’s ability to use an individual system in various roles—possibly as a replacement for other systems. A binary link in the network connects systems that per-form the same function; however, it does not capture the level of performance.

FIGURE 3. USAF VERSATILITY NETWORK

Table 6 presents the highest ranking systems by degree centrality to rep-resent the traditional definition of versatility. As such, it is not surprising that multirole aircraft like the F-22A, F-16C, F-15E, and the F-35A rank high according to the degree metric. Additionally, literature often refers to the B-52H and the A-10C as highly versatile aircraft (Church, 2015; USAF, 2018). The degree metric would still represent a system’s ability to replace other systems within the network, thus meeting the expectations of various stakeholders without changing form. Future work could expand on this to capture subject matter experts’ opinions on these systems to determine how well the degree metric represents true versatility or determine if closeness or betweenness are better metrics to quantify a system’s versatility.



TABLE 6. TOP 10 USAF SYSTEMS IN VERSATILITY NETWORK BY DEGREE CENTRALITY

System Type Degree Closeness Betweenness

F-22A Raptor Aircraft 118 0.6368 260.1

F-16C Fighting Falcon Aircraft 118 0.6368 260.1

F-35A Lightning II Aircraft 118 0.6368 260.1

A-10C Thunderbolt II Aircraft 112 0.6853 1295.1

F-15E Strike Eagle Aircraft 112 0.6398 269.6

B-52H Stratofortress Aircraft 111 0.6553 943.1

F-15C/D Eagle Aircraft 104 0.6368 242.1

AC-130J Ghostrider Aircraft 98 0.6279 152.1

AC-130U Specter Aircraft 98 0.6279 152.1

AC-130W Stinger II Aircraft 98 0.6279 152.1

Summary, Conclusions, and Future WorkThis article explores DoD systems to determine if systems the DoD

extends past their planned life cycle exhibit different attributes, as defined by the ilities, than those the DoD retires. Additionally, it begins to extend beyond identifying these attributes to determine methods to quantify these attributes through the application of SNA. The various centrality metrics from SNA appear to capture the interoperability and versatility of DoD sys-tems within the network of systems the DoD developed over the past several decades. However, additional work will be required to validate these findings and apply them to a broader range of DoD systems.

The initial findings from this research identify several critical ilities or nonfunctional attributes of systems as predictors for service life extensions beyond their planned life cycle. Grounded theory provided a methodology to examine four DoD systems: the B-52 bomber, A-10 aircraft, F-117 stealth fighter, and the OH-58D helicopter. It appears the literature describing extended DoD systems mentions extensibility, flexibility, interoperability, versatility, and robustness significantly more often than for the retired systems. This may assist DoD decision makers with the decision to pursue a new system design or extend the life of the legacy system based on these attributes. Additionally, understanding these attributes may inform sys-tems engineers as they design new systems and incorporate these types of attributes into their design.

In the SNA portion of their research, the authors explored the application of centrality metrics to a network of DoD systems—in this case, 136 USAF systems—to determine if these metrics could quantify interoperability and versatility. It appears degree or eigenvector centrality provides potential means to quantify interoperability in a network where edges represent the exchange of resources, information, or physical connections. Additionally, degree centrality has the potential to quantify versatility in a network where the links represent common operational activities between systems.

Future work is required to expand this research to ensure the results are valid for an expanded data set of DoD systems. First, the grounded theory research must extend beyond these four systems to capture a broader range of DoD systems across each of the Services and domains. Once a broader data set is examined, the hypothesis will become more clear and applicable to a wider range of DoD systems. Additionally, validation through other means—potentially stakeholder interviews—will be necessary to ensure the research accurately identifies critical nonfunctional attributes of DoD systems. Research should also extend to newer systems, generally more complex and software-dependent, to determine how the DoD is upgrading new systems to achieve these attributes. The use of ilities may also have application in predicting when a system is ripe for a disruptive technology.

The authors used grounded theory application to examine a set of articles on 10 DoD systems to identify common

attributes of these systems and build a theory on why the DoD retires certain systems and extends others.



From an acquisition perspective, system ilities could become key perfor-mance parameters in the design of new systems. If we understand how the ilities affect a service life, designs might produce a system that can remain in the inventory longer with reduced total ownership costs.

For the SNA portion of the research, an extension to more DoD systems will assist in validating the findings. One of the benefits of SNA is the scalability of the metrics, so as the network increases beyond the Air Force systems, they will remain valid. Incorporating Army, Navy, Marine, and DoD systems into the network will increase the accuracy of the network and better reflect how these systems interoperate on the battlefield. Once a completed network is developed, the metrics quantifying interoperability and versatility can be validated against stakeholder interviews or surveys to determine if the calculated metric aligns with the perception of these systems.

Finally, the research should be extended beyond just the DoD to determine if it applies to broader types of systems.

First, it could explore cyber-networks to determine if the SNA metrics inform decision makers about

the interoperability of these systems or the vul-nerabilities of these types of network systems.

Additionally, it could extend to infrastructure systems to understand how power grids, water

treatment, and other utilities depend upon one another to operate.

Likewise, it could relate back to DoD systems once com-

mon attributes of systems are identified that make decision ma kers more likely to extend a system to determine how this affects the design of new DoD systems.

References

Birks, M., & Mills, J. (2015). Grounded theory: A practical guide (2nd ed.). Los Angeles, CA: Sage.

Boardman, J., & Sauser, B. (2008). Systemigrams. In Systems thinking: Coping with 21st century problems (chap. 6, pp. 95–115). New York: Taylor & Francis Group.

Borgatti, S. P. (2005). Centrality and network flow. Social Networks, 27(1), 55–71.Buede, D. (2000). The engineering design of systems: Models and methods. New

York: Wiley.Cho, J. Y., & Lee, E.-H. (2014). Reducing confusion about grounded theory and

qualitative content analysis: Similarities and differences. The Qualitative Report, 19(32), 1–20.

Church, A. M. (2015, May). Gallery of USAF weapons. Air Force Magazine, 98(5), 80–103.

Dahmann, J., & Baldwin, K. (2008, April). Understanding the current state of US defense systems of systems and the implications for systems engineering. In Proceedings of the 2nd Annual Institute for Electrical and Electronics Engineers (IEEE) Systems Conference (pp. 1–7). Montreal, Canada: Piscataway, NJ: IEEE.

Department of Defense. (2013). Defense acquisition guidebook. Washington, DC: Author.

de Weck, O., Ross, A., & Rhodes, D. (2012, June). Investigating relationships and semantic sets amongst system lifecycle properties (ilities). Paper presented at the Third International Engineering Systems Symposium, Delft University of Technology, Delft, The Netherlands.

Enos, J. R., Mansouri, M., & Nilchiani, R. (2017). Applying social network analysis to systems of systems: The case of the 2016 Puerto Rico power outage. In Proceedings of the 12th Annual Institute of Electrical and Electronics Engineers (IEEE) International System of Systems Engineering Conference. Waikoloa, HI: IEEE.

Enos, J. R., & Nilchiani, R. R. (2017a). A tale of two aircraft: How non-functional attributes impact a system’s lifecycle. In Proceedings of the American Society for Engineering Management (ASEM) 2017 Annual International Conference. Huntsville, AL: ASEM.

Enos, J., & Nilchiani, R. (2017b). Understanding how social network analysis can provide insights into emergent networks of systems. Paper presented at the 15th Annual Conference on Systems Engineering Research (CSER) 2017, Redondo Beach, CA.

Enos, J. R., & Nilchiani, R. R. (2017c). Using social network analysis to quantify interoperability in a large system of systems. In Proceedings of the 12th Annual Institute of Electrical and Electronics Engineers (IEEE) International System of Systems Engineering Conference (pp. 1–5). Waikoloa, HI: IEEE.

Engineering Systems Division Symposium Committee. (2002). ESD terms and definitions (Version 12). Cambridge, MA: Massachusetts Institute of Technology.

Everton, S. (2009, April). Terrorist networks. Paper presented at the Annual Conference of the Association for the Study of Economics, Religion and Culture, Washington, DC.

Everton, S. (2012). Disrupting dark networks. New York: Cambridge University Press.Faust, K. (1997). Centrality in affiliation networks. Social Networks, 19(2), 157–191.



Fendt, J., & Sachs, W. (2008). Grounded theory method in management research: Users’ perspectives. Organizational Research Methods, 11(3), 430–455.

Freedberg, S. J. (2012, June 19). Total cost to close out cancelled Army FCB could top $1 billion. Breaking Defense. Retrieved from https://breakingdefense.com/2012/06/total-cost-to-close-out-cancelled-army-fcs-could-top-1-billion/

Freeman, L. (2004). The development of social network analysis. Vancouver, BC: Empirical Press.

Fruchterman, T. M., & Reingold, E. M. (1991). Graph drawing by force-directed placement. Software: Practice and Experience, 21(11), 1129–1164.

Glaser, B., & Strauss, A. (1967). The discovery of grounded theory: Strategies for qualitative research. London: Aldine Transaction.

Golbeck, J. (2013). Analyzing the social web. Boston, MA: Morgan Kaufmann.Government Accountability Office. (2015). Opportunities exist to improve the

Department of Defense’s portfolio management (Report No. GAO-15-466). Washington, DC: U.S. Government Printing Office.

Hanneman, R., & Riddle, M. (2005). Introduction to social network methods. Riverside, CA: University of California.

International Council on Systems Engineering. (2007). Systems engineering handbook v3.1: A guide for system life cycle processes and activities. Retrieved from http://smslab.kaist.ac.kr/Course/CC532/2012/LectureNote/2012/INCOSE%20Systems%20Engineering%20Handbook%20v3.1%205-Sep-2007.pdf

Kiemele, M. J., Schmidt, S. R., & Berdine, R. J. (2000). Basic statistics: Tools for continuous improvement. Colorado Springs, CO: Air Academy Press.

Kossiakoff, A., Sweet, W., Seymour, S., & Biemer, S. (2011). Systems engineering principles and practice (2nd ed.). Hoboken, NJ: Wiley.

Magnuson, S. (2011, April 1). Ghost of Comanche haunts Army helicopter leaders as they push for new models. National Defense. Retrieved from http://www.nationaldefensemagazine.org/articles/2011/4/1/2011april-ghost-of-comanche-haunts-army-helicopter-leaders-as-they-push-for-new-models

McManus, H., Richards, M., Ross, A., & Hastings, D. (2009, September). A framework for incorporating “ilities” in tradespace studies (AIAA Paper 2007-6100). In Proceedings of the American Institute of Aeronautics and Astronautics (AIAA) Space 2007 Conference & Exposition. Long Beach, CA: AIAA.

Owens, W. A. (1996). The emerging U.S. system-of-systems (National Defense University Strategic Forum No. 63). Washington, DC: National Defense University Press.

Rodriquez, S. (2014, December 2). Top 10 failed defense programs of the RMA era. War on the Rocks. Retrieved from https://warontherocks.com/2014/12/top-10-failed-defense-programs-of-the-rma-era/

Rouse, W. (2005). Enterprises as systems: Essential challenges and approaches to transformation. Systems Engineering, 8, 138-150. https://doi.org/10.1002/sys.20029

Schmidle, N. (2011, August 8). Getting Bin Laden. The New Yorker. Retrieved from https://www.newyorker.com/magazine/2011/08/08/getting-bin-laden

Serrat, O. (2010). Social network analysis. Washington, DC: Asian Development Bank.Starks, H., & Brown-Trinidad, S. (2007). Choose your method: A comparison of

phenomenology, discourse analysis, and grounded theory. Qualitative Health Research, 17(10), 1372–1380.

Thompson, M. (2014, January 27). The A-12: Admiral Peabody’s ultimate wayback machine. Time. Retrieved from http://swampland.time.com/2014/01/27/the-a-12-admiral-peabodys-ultimate-wayback-machine/

Thornburg, R., & Charmaz, K. (2011). Grounded theory and theoretical coding. In The SAGE handbook of qualitative data analysis (pp. 153–160). Los Angeles, CA: Sage.

Timmermans, S., & Tavory, I. (2012). Theory construction in qualitative research: From grounded theory to abductive analysis. Sociological Theory, 30(3), 167–186.

Urquhart, C., & Fernandez, W. (2013). Using grounded theory method in information systems: The researcher as blank slate and other myths. Journal of Information Technology, 28(3), 224–236. https://doi.org/10.1057/jit.2012.34

U.S. Air Force. (2018). Air Force fact sheets. Retrieved from http://www.af.mil/AboutUs/FactSheets

Ward, D. (2012, May 25). Real lessons from an unreal helicopter. Time. Retrieved from http://nation.time.com/2012/05/25/real-lessons-from-an-unreal-helicopter/

Warwick, G. (2013, January 11). CSAR-X to CRH – Then there was one. Aviation Week. Retrieved from http://aviationweek.com/blog/csar-x-crh-then-there-was-one

Wasserman, S., & Faust, K. (1994). Social network analysis: Methods and applications. New York: Cambridge University Press.

White, H., Boorman, S., & Breiger, R. (1976). Social structure from multiple networks. American Journal of Sociology, 81(4), 730–780.



Author Biographies

LTC James R. Enos, USA,is a PhD candidate at Stevens Institute of Technology. During his Army career, he served as systems engineer branch chief on the Joint Staff and an assistant professor in the Department of Systems Engineering at the United States Military Academy. He also held numerous leadership positions as an infantry officer. LTC Enos holds a master’s degree in Engineering and Management from MIT. Upon completion of his PhD, he will

return to West Point as a program director.

(E-mail address: [email protected])

Dr. John V. Farris currently a program director at the University of Central Florida and Professor Emeritus of Engineering Management at West Point. He has authored or edited over 200 technical publications on engineering management and systems engineering. Dr. Farr has served on numerous defense national and academic advisory boards to include membership on the Army Science Board and the Air Force Studies Board of the National Academies. He holds

a PhD in Civil Engineering from the University of Michigan.


Dr. Roshanak R. Nilchianiis an associate professor in the School of Systems and Enterprises at Stevens Institute of Technology. At Stevens, Dr. Nilchiani resea rches risk-ba sed complex systems desig n, critica l infrastructure resilience, dynamics of disruptive technologies, and systems and enterprise architecture. She received her PhD in Aerospace Engineering at MIT where she worked on flexible space systems design for the Defense Advanced Research Projects

Agency’s Orbital Express program.


ilities in the acqoutsi i i n of dod systems › library › arj › arj › arj87 › arj87... ·...

Documents