AFRL S5 Symposium 2014

Yuri Gawdiak Manager Strategic Analysis

Strategy, Architecture and Analysis Office (SAA)

Aeronautics Research Mission Directorate (ARMD)

National Aeronautics & Space Administration (NASA)

yuri.o.gawdiak@nasa.gov June 12, 2014

Strategic Portfolio Analysis:

Complex Systems

Development Risks

NextGen Enterprise Architecture 2009

$(150.00)

$(100.00)

$(50.00)

$-

$50.00

$100.00

$150.00

$200.00

$250.00

FY 2

010

FY 2

011

FY 2

012

FY 2

013

FY 2

014

FY 2

015

FY 2

016

FY 2

017

FY 2

018

FY 2

019

FY 2

020

FY 2

021

FY 2

022

FY 2

023

FY 2

024

FY 2

025

FY 2

026

FY 2

027

FY 2

028

FY 2

029

FY 2

030

FY 2

031

FY 2

032

FY 2

033

FY 2

034

FY 2

035

FY 2

036

FY 2

037

FY 2

038

FY 2

039

FY 2

040

FY 2

041

FY 2

042

FY 2

043

FY 2

044

FY 2

045

FY 2

046

FY 2

047

FY 2

048

FY 2

049

FY 2

050

$(150.00)

$(100.00)

$(50.00)

$-

$50.00

$100.00

$150.00

$200.00

$250.00

Cost

Benefit

NPV

Note: Costs represented in the table and charts are based on point estimate results; these figures do not reflect all

estimating uncertainty, the influence of any programmatic risk –adjustment, or the impact due to environmental

factors

Incremental Cumulative NextGen Net Cash Flows (PV $M)

$104.2 B

In B

illio

ns o

f D

iscounte

d D

olla

rs

NPV: $ 104.2 B Discounted Payback: 2027 Benefit/Cost Ratio: 2.00 IRR: 10%

The Society/Passengers NPV for the Initial Alternative results in a positive $104.2B, without accounting for environmental impact or risk

2027 2050

SOCIETAL NET PRESENT VALUE JPDO NGOps-5 Architecture by 2025

IPSA Results: Monetized impacts of policy relative to baseline (2009 billions of US dollars) for initial alternative

ENVIRONMENTAL ANALYSIS

• The table shows monetized environmental impacts of the NextGen N+1 policy relative to the baseline in billions of 2009 U.S. dollars for the lower, mid, and upper bound assumptions (including discount rates within the recommended OMB and EPA ranges).

• Lower discount rates place greater value on future years, which contributes to the higher cost of impacts, especially for CO2 effects on climate.

• Negative values represent projected impact reductions (improvements) for policy minus baseline; positive values are projected costs, which stem primarily from the increased number of flights.

• Climate results reflect global impacts and are integrated out to 800 years, which is consistent with EPA guidance. Noise and Air Quality results reflect regional impacts and are assumed to be realized during the period of operation (2006-2050).

• The monetized environmental impacts are sensitive to many assumptions, as reflected in the low, mid, and upper bound ranges. For instance, climate estimates are sensitive to the following assumptions: discount rates, climate sensitivity, damage coefficients, short-lived radiative forcing, background CO2 scenario, NOx effects.

• Important to Note: Results are highly sensitive to emissions, noise, and fuel burn inventories provided by IPSA

Environmental Impacts 2006-2050 Mid-range Upper Bound Lower Bound

Discount Rate 3.5% 2.0% 5.0%

Climate $63bn $808bn $2bn

Air Quality -$4bn -$36bn -$0.4bn

Noise (for ≥ 55 dB DNL) -$29bn -$229bn -$3bn

Total (NextGen Policy minus Baseline) $30bn $543bn -$1.4bn

When comparing programs from various studies and sources, the

average cost overrun percentage ranged from 28% to 79%

Program Type Number of

Programs

Average

Program Cost

Growth

Dollar

Weighted

Average

Growth*

Cost

Growth

Standard

Deviation

FAA 11 79% 112% 86%

NASA 83 45% 59% 67%

DoD Weapons

Systems 76 45% ~40% 80%

Public Works

Transportation 258 28% Not Available 39%

* Percent overrun weighted by the program dollar value relative to total dollar value of all programs

Cost Overruns of Federal Programs

When considering environment and programmatic risks, the

NPV for Society/Passengers drops substantially

Point

Estimate

Environment

Impact

Unmitigated Historical

Risk-Adjusted

Discounted Dollars NPV NPV NPV

Environmental Lens 1

$ 104.2

B $ 75.4 B $ (131.9) to $96.1 B

Environmental Lens 2

$ 104.2

B $ (439.9) B $ (647.2) to $ (419.1) B

Environmental Lens 3

$ 104.2

B $ 105.7 B $ (101.6) to $126.4 B

Based on an assessment of the Initial Alternative, environment and

risks severely impact the NPV for NextGen

• Environment can negatively impact the NPV by a factor of 5

• Risks can negatively impact the NPV by a factor of 3

Including Environment Including Historic Risk

INTEGRATED RESULTS, SUMMARY, and RECOMMENDATIONS

Question: How do we stand now? Five Years Later?

Complex Hardware & Software Systems key examples

The new program baseline projects total acquisition costs of $395.7 billion, an increase of $117.2 billion (42 percent) from the prior 2007 baseline. Full rate production is now planned for 2019, a delay of 6 years from the 2007 baseline. Unit costs per aircraft have doubled since start of development in 2001. Since 2002, the total quantity through 2017 has been reduced by three-fourths, from 1,591 to 365. The original plan was that about 70 percent of all the parts on the airplanes would be common; the actual figure today is about 25 percent.

FAA now projects that ERAM will be almost 4 years behind schedule, with an uncertain final completion date. If problems persist, cost increases could reach in excess of $500 million and interfere with program execution. FAA’s problems in advancing ERAM are attributable to a number of fundamental program management weaknesses that have impeded the Agency’s ability to effectively implement ERAM and effectively manage other major acquisitions.

JWST was projected to cost just $1 billion to build and launch. By 2011, however, the program had seen almost a decade of cost overruns and schedule delays. Under pressure from lawmakers, NASA rebaselined the program with a revised cost estimate of $8.8 billion, and a new launch date of October 2018 - 10 years later than the original date of 2008.

NASA JWST FAA ERAM

DoD JSF

Simplified FAA ATM System Diagram Existing Environments are Complex

And we want to add new Functions & More Complexity

Airport Robotic Systems

Control Towers

Safety & Security Systems

AutoMax Single Pilot Ops

Airline & Airport Ops Centers Passengers/Public Ticketing & Scheduling Systems

Manufacturers Operations Center (Boeing)

Weather & External Services

FAA Command Center

Pilots/Intelligent Flight Decks

Intelligent Autonomous Systems

Gates & Ramps

Gate to Gate TBO/4DT/Autonomous

Intelligent Swarming Systems

Complete Success Requires Addressing Complex Organizational

Environments

UAS Stakeholders – Joint Planning & Development Office Coordination

― Policy/Regulatory

― Research and Development

― Operators

― Operations

― Strategic

― NAS Community & Public Advocacy

― Manufacturers

This is a high-level view

of the UAS Stakeholders

and their associated

roles/functions, which

are categorized by the 7

bins represented in the

Key below

This is a one time slice. Hand generated, expensive and already out of date. V&V of complex systems/ops require this to be continuously updated.

Key Research & Development Related

Strategic Decisions*

* Results based on a 20% sampling of the JPDO Enterprise Architecture

RISK ANALYSIS & RESULTS 2009 JPDO IWP

Note: Intensity is a function of the frequency and complexity of decision events

Question: Are we missing something obvious in our complex

problem spaces?

Humans/Organizations V&V?: Configuration Management, Baselines, Test, Demonstrations & Formal Methods

V&V?:

Software Hardware

Total System of Systems

V&V?: Configuration Management, Baselines, Test, Demonstrations & Formal Methods

V&V?:

Software Hardware

Total System of Systems

Humans/Organizations

1. Most Dynamic 2. Greatest Lifecycle

Impact 3. Least Predictable 4. Least Calibrated 5. Least Tracked 6. Least Supported

Columbia Accident Investigation

Board Organizational Issues After the accident, Program managers stated privately that if engineers had a safety concern, they were obligated to communicate their concerns to

management. Managers did not seem to understand that as leaders they had a corresponding and perhaps greater obligation to create viable routes for the

engineering community to express their views and receive information. This barrier to communications not only blocked the flow of

information to managers, but it also prevented the downstream flow of information from managers to engineers, leaving

Debris Assessment Team members no basis for understanding the reasoning behind Mission Management Team decisions.[1]

Encouraging Minority Opinions: The Naval Reactor Program encourages minority opinions and “bad news.” Leaders continually emphasize that when

no minority opinions are present, the responsibility for a thorough and critical examination falls to management. Alternate perspectives and critical

questions are always encouraged. In practice, NASA does not appear to embrace these attitudes. Board interviews revealed that it is difficult for

minority and dissenting opinions to percolate up through the agency’s hierarchy, despite processes like the anonymous NASA

Safety Reporting System that supposedly encourages the airing of opinions.[2]

NASA had conflicting goals of cost, schedule and safety.[3]

Allegiance to hierarchy and procedure had replaced deference to NASA engineer’s technical expertise.[4]

The organizational structure and hierarchy blocked effective communication of technical problems. Signals were overlooked, people were

silenced, and useful information and dissenting views on technical issues did not surface at higher levels.[5]

Engineers at Thiokol who still objected to the decision later testified that they were intimidated by management authority, were accustomed to

turning their analysis over to managers and letting them decide, and did not have the quantitative data that would empower them to object further.[6]

In the more decentralized decision process prior to Columbia’s re-entry, structure and hierarchy again were responsible for an

absence of signals.[7]

As what the Board calls an “informal chain of command” began to shape STS-107’s outcome, location in the structure empowered some to

speak and silenced others.[8]

Strategies must increase the clarity, strength, and presence of signals that challenge assumptions about risk. NASA’s

challenge is to design systems that maximize the clarity of signals, amplify weak signals so they can be tracked, and account for missing signals.[9]

1

2

3

4

5

6

7

8

9

[Notes] Columbia Accident Investigation Board, Report Volume 1, pages 169, 200-201, 203

General Motors Switch Failure

GM Internal Assessment: 1. Some were dismissed due to misconduct or incompetence, while others simply did not do enough to fix the problem. 2. It represents a fundamental failure to meet the basic needs of these customers. 3. We simply didn't do our jobs. We failed these customers. 4. GM's personnel's inability to address the ignition switch problem, which persisted for more than 11 years, represents a history of failures. 5. While everybody who was engaged on the ignition switch issue had the responsibility to fix it, nobody took responsibility. 6. Throughout the entire 11-year history, there was no demonstrated sense of urgency, right to the very end. 7. A pattern of management deficiencies and misjudgments -- often based on incomplete data -- that were passed off at the time as business as usual. 8. But the lack of information at the highest executive levels, the report concluded, was at the heart of G.M.’s failings on this issue.

http://money.cnn.com/2014/06/05/news/companies/gm-recall-probe/index.html?iid=TL_Popular

http://www.nytimes.com/2014/06/06/business/gm-response-to-a-fatal-flaw-was-to-shrug.html?hp&_r=0

Wetware (highest risks)

Software (high risk)

Hardware (lowest risk)

Mature; rich set of standards;

explicit; significant laws of

physics constraints; more

limited interfaces and

behaviors; & extensive full-

lifecycle tools.

Human &

Organizational

Components

Intelligent,

Autonomous

Systems

Avionics, Operating Systems,

Embedded Systems, Apps, Tools,

etc. Physical,

Mechanical,

Components

& Systems

Evolving; growing set of

standards; highly dynamic;

limited laws of physics

constraints; highly susceptible

to unintended interfaces and

behaviors; & growing &

evolving full-lifecycle tools.

Significant gaps; very limited set

of standards & certifications;

highly dynamic; moderate to

limited laws of physics

constraints; very susceptible to

unintended interfaces and

behaviors; & a dearth of full-

lifecycle tools.

Complex Systems Matrix

Key First Principles for Human/Organizational Complex Systems

Risks

KEY PREMISE ON

HUMANITY:

MOST OF US HAVE THE

SAME STRATEGIC

GOALS

HUMAN/ORGANIZATION

GOALS

1. PEACE

2. DIGNITY

3. PROSPERITY

4. GROWTH

5. SUSTAINABILITY

HUMAN/ORGANIZATION

GOALS - Pedigree

1. PEACE

2. DIGNITY

3. PROSPERITY

4. GROWTH

5. SUSTAINABILITY

1. Life

2. Liberty

3. Pursuit of

Happiness

SYSTEMS OWNERS/DESIGNERS

HAVE SIMILAR GOALS

1. PEACE

2. DIGNITY

3. PROSPERITY

4. GROWTH

5. SUSTAINABILITY

HUMANS/ORGANIZATION

S

1. FUNCTIONALITY

2. INTEGRITY

3. STABILITY

4. MARKET SHARE

5. SUSTAINABILITY

SYSTEMS and/or SYSTEMS

OF SYSTEMS

1. PEACE

2. DIGNITY

3. PROSPERITY

4. GROWTH

5. SUSTAINABILITY

HUMANS/ORGANIZATION

S

1. FUNCTIONALITY

2. INTEGRITY

3. STABILITY

4. MARKET SHARE

5. SUSTAINABILITY

=

SYSTEMS OWNERS/DESIGNERS

HAVE SIMILAR GOALS

SYSTEMS and/or SYSTEMS

OF SYSTEMS

1. PEACE

2. DIGNITY

3. PROSPERITY

4. GROWTH

5. SUSTAINABILITY

HUMANS/ORGANIZATION

S

1. FUNCTIONALITY

2. INTEGRITY

3. STABILITY

4. MARKET SHARE

5. SUSTAINABILITY

SYSTEMS OWNERS/DESIGNERS

HAVE SIMILAR GOALS

SYSTEMS and/or SYSTEMS

OF SYSTEMS

HOWEVER INDIVIDUAL HUMANS,

ORGANIZATIONS, & SYSTEMS

DON’T HAVE COMMON ANCHORS:

1. STARTING POINTS

2. KNOWLEDGE/EXPERIENCE

3. BELIEFS

4. FILTERS

5. COMFORT LEVELS

6. RESOURCES

7. TIMEFRAMES

8. LOCAL ENVIRONMENTS

9. CONSTRAINTS

10. DRIVERS

THESE DIFFERENCES LEAD TO:

1. STARTING POINTS

2. KNOWLEDGE/EXPERIENCE

3. BELIEFS

4. FILTERS

5. COMFORT LEVELS

6. RESOURCES

7. TIMEFRAMES

8. LOCAL ENVIRONMENTS

9. CONSTRAINTS

10. DRIVERS

1. PERSPECTIVES

2. PRIORITIES

3. APPROACHES

4. RISK MITIGATION

STRATEGIES

5. IMPLICIT VS. EXPLICIT

POSITIONS/BEHAVIORS

6. EXTRAPOLATIONS INTO

THE FUTURE

7. EXPECTATIONS

ANCHORS POINTS INCONGRUENT

INCENTIVES/BEHAVIORS

THESE BEHAVIORS DRIVE:

1. STARTING POINTS

2. KNOWLEDGE/EXPERIENCE

3. BELIEFS

4. FILTERS

5. COMFORT LEVELS

6. RESOURCES

7. TIMEFRAMES

8. LOCAL ENVIRONMENTS

9. CONSTRAINTS

10. DRIVERS

1. PERSPECTIVES

2. PRIORITIES

3. APPROACHES

4. RISK MITIGATION

STRATEGIES

5. IMPLICIT VS. EXPLICIT

POSITIONS/BEHAVIORS

6. EXTRAPOLATIONS INTO THE

FUTURE

7. EXPECTATIONS

ROOT CAUSE OF

MOST GAPS,

SOURCES OF

FRICTION &

CONFLICTS

ANCHORS POINTS INCONGRUENT

INCENTIVES/BEHAVIORS

I believe a critical priority is to illuminate this area. To find sufficient situational

awareness for the entire systems of systems lifecycle

1. STARTING POINTS

2. KNOWLEDGE/EXPERIEN

CE

3. BELIEFS

4. FILTERS

5. COMFORT LEVELS

6. RESOURCES

7. TIMEFRAMES

8. LOCAL ENVIRONMENTS

9. CONSTRAINTS

10. DRIVERS

1. PERSPECTIVES

2. PRIORITIES

3. APPROACHES

4. RISK MITIGATION

STRATEGIES

5. IMPLICIT VS. EXPLICIT

POSITIONS/BEHAVIOR

S

6. EXTRAPOLATIONS

INTO THE FUTURE

7. EXPECTATIONS

ROOT CAUSE OF

MOST GAPS AND

SOURCES OF

FRICTION &

CONFLICTS

DIFFERENT ANCHOR

POINTS

1. PEACE

2. DIGNITY

3. PROSPERITY

4. GROWTH

5. SUSTAINABILITY

1. FUNCTIONALITY

2. INTEGRITY

3. STABILITY

4. MARKET SHARE

5. SUSTAINABILITY

COMMON

STRATEGIC GOALS INCONGRUENT

INCENTIVES/BEHAVIO

RS

Thank You

Questions?