application of fast-time computer modeling for atm...

Application of Fast-Time Computer Modeling for ATM Systems Presented by the FAA’s Modeling and Simulation Branch (ANG-C55) for the 7th International Conference on Research in Air Transportation June 22, 2016

Outline

• Background: NAS & ATM • Concept Validation • Fast-Time Simulation • Process Steps • Examples • Demo Videos

2

Introduction to NAS/ATM

3

“Simplified” NAS/ATM

4

NAS: A System of Systems

• Airspace • Airports • Facilities • Airlines

5

• Regulations • Services • Workforce • Military

FAA Areas of Responsibility

6

Oceanic

Enroute

Airspace • Center • Sector • Terminal • Oceanic

National Airspace System

Sector

Airport

Terminal Airspace

Center Airspace

7

8

Terminal Sectors

En Route Low Altitude Sectors

En Route High Altitude Sectors

En Route Low Altitude Sectors

Terminal Sectors

Command Center

Navigation and Surveillance

9

Next Generation Air Transportation System (NextGen)

• Modernization of NAS On-going implementation Realization by 2025

• Transform air traffic systems GPS technology will be used to save time and fuel,

reduce delays, increase capacity, and increase safety DataComm (automated data exchange) will reduce

the workload associated with manually processing information

• New entrants into the NAS

10

Concept Development & Validation

12

Advancing NextGen

13

NextGen Implementation

Plan and Operational

Improvements

Congressional Mandates

Aviation Research in Industry and

Academia

National Airspace System (NAS)

Enterprise Architecture and

Infrastructure Roadmaps

Concept Proposal

And Approval Process

Stakeholder Involvement

Alignment with

NextGen Developments

Concept Requirements

Benefits Analysis

INPUTS CONCEPT DEVELOPMENT OUTPUTS

CD&V Process

Concept Development

• Definition and analysis of alternative concepts to meet the ATM need

• Systematic investigation of feasibility • Selection of one or more concepts to be

pursued • Development of an integrated concept

evaluation environment

14

Concept Validation

• Process that Underlies concept development Ensures that the correct system is being built to

meet the defined service needs

• Utilize variety of methods and disciplines with a broad foundation of capabilities and tools

15

Introduction to Fast-Time Simulation

16

Why Simulation?

• Provides a useful representation of reality

• Captures the variability that exists in reality

17

Definitions

• Model refers to the algorithms and equations used to capture the behavior of a system

• Simulation refers to the execution of a program that contains a model of a system

• Fast-time simulation refers to a simulation that runs faster than real-time

• Multi-agent system refers to a computerized system composed of interacting intelligent agents within an environment

18

Misconceptions

• Fast-time simulation studies are quick turnaround studies Sample from AENA 35 Scenarios – 10 months to complete Airport Design Teams – 1-2 years

• Any model will do Scope Number of scenario runs expected

• Data is readily available Traffic data, weather, procedures, airports, waypoints,

aircraft types • Models are easy to learn

19

Categories of Models: Variability

• Stochastic (random) Will accept probabilistic input parameters and capture

the impacts of uncertainty Monte Carlo simulations use stochastic models

• Deterministic Does not model variation Will produce the same results given the same input

parameters

20

Categories of Models: Time Progression

• Discrete Models the operation of a system

as a sequence of events in time • Continuous

Models a system by continuously tracking system response over time

21

Categories of Models: Configuration

Local • Simulation

model located on one system

Distributed • Simulation

model runs on a network of interconnected computers

22

Fidelity and Scope of Models

Fidelity • Macroscopic

(Low) • Mesoscopic (Mid) • Microscopic

(High) Scope • Airport • Terminal • En Route • NAS-wide

23

Examples of FAA Used Simulation Tools

24

Variability Time Progression

Configuration Fidelity Scope

RAMS Plus

Stochastic Continuous Local High NAS-Wide

SIMMOD Stochastic Discrete Local Mid Airport/ Terminal

ADSIM Stochastic Discrete Local Low Airport

RDSIM Stochastic Discrete Local Low Runway

AirTOp Stochastic Continuous Distributed High NAS-Wide

Agentfly Stochastic Continuous Distributed High NAS-Wide ATC Simulation

- George E. Box

26

Essentially, all models are wrong, but some are useful.

27

Conducting a Simulation Study

Select Research Team

• Project Lead Responsible for overall planning (Project

Management) Interfaces with stakeholders and management

• Principal Investigator/Task Lead Coordinates simulation test plan and final report

• Support staff • Subject Matter Experts (SMEs)

Provides operational insight Ensures simulation “realism”

28

Simulation Process Steps

1. Project Plan/Problem Definition 2. Metrics Selection 3. Analysis Design 4. Simulation Plan 5. Execution 6. Reporting

29

Process Steps

31

QUESTIONS TO BE ANSWERED THROUGH M&S

METRICS

STAKEHOLDERS ANALYSTS

Step 2. Metric Selection

DATA • System

• CONOPS

• Environment

• Scenarios

•Data Availability

• Data Assumption

• Data Validation

Experiment Design

Model Selection

EXISTING NEW

Modify As-Is

Step 3. Analysis Design

Run Matrix


Step 4. Simulation Plan

FINAL REPORT

STAKEHOLDERS ACCEPTANCE

Step 6. Reporting

PROBLEM


STUDY OBJECTIVES

Step 1. Project Plan/Problem Definition

Scenario/Data Validation

Conduct Test/ Verification Runs

Conduct Analysis

Conduct Runs

Credibility Assessment

Step 5. Execution

Process Steps QUESTIONS TO BE ANSWERED

THROUGH M&S

METRICS



DATA • System

• CONOPS

• Environment

• Scenarios


• Data Assumption

• Data Validation

Experiment Design

Model Selection

EXISTING NEW

Modify As-Is


Run Matrix



FINAL REPORT


Step 6. Reporting

PROBLEM


STUDY OBJECTIVES




Conduct Analysis

Conduct Runs


Step 5. Execution

32

Project Plan • Define scope of project

Clear definition of problem being studied • Concept of Operations document (if available) • Stakeholder input

Objectives Research questions

• Identify necessary resources Personnel, hardware/software, data

• Describe key tasks • Schedule

33


THROUGH M&S

METRICS



DATA • System

• CONOPS

• Environment

• Scenarios


• Data Assumption

• Data Validation

Experiment Design

Model Selection

EXISTING NEW

Modify As-Is


Run Matrix



FINAL REPORT


Step 6. Reporting

PROBLEM


STUDY OBJECTIVES




Conduct Analysis

Conduct Runs


Step 5. Execution

34

Metric Selection

• Driven by research questions • Air Traffic Modeling of Operational Concepts:

Performance Measures and Metrics • Key Performance Areas

Capacity Efficiency Environment Predictability Safety

35

Capacity Metrics

• Throughput Number of operations per unit time such as arrivals, departures,

sector entries, etc. • Operations

Count of the operations using a resource such as sector, center, airway, etc.

• Controller workload & airspace complexity Number/type of controller tasks Time spent performing

controller tasks Dynamic Density

36

• Flight time Gate to gate Taxi time Runway occupancy time Time spent in terminal area

• Fuel burn • Flight distance • Delay

Efficiency Metrics

37

Environment and Predictability Metrics

• Environment Emissions Noise and sound exposure

• Predictability Ability of the airspace users and ATM service

providers to deliver consistent and dependable levels of performance (such as arrival times)

Typically considers the variance of other metrics

38

Safety Metrics

• Separation assurance Conflict events Loss of separation events Spacing

• Risk of a safety significant event Propensity

• Operational errors Deviation and spacing errors

39

Project Example: UAS Airspace Integration

40

UAS Airspace Integration: Project Plan

• Collaborated with DOD stakeholders • Project Scope Statement

Problem: no standard procedures for UAS Objective: validate UAS procedures at one Air Force Base

for use in a real-time simulation Research Questions:

• Can UAS perform the procedures? • Do the procedures conflict with typical traffic flows?

Resources: analysts, DOD and ATC SMEs, AirTOp simulation model

Key tasks: perform iterative fast-time simulation study Schedule: 7 months

41

UAS Airspace Integration: Metrics Selection

• Safety was a key performance area

• Separation assurance Number of conflicts Types of conflicts Separation statistics

42

UAS Airspace Integration: Metrics Selection (cont)

• Risk Metric Propensity Identified

segments of the UAS procedures that presented the most collision risk for the UA

43


THROUGH M&S

METRICS



DATA • System

• CONOPS

• Environment

• Scenarios


• Data Assumption

• Data Validation

Experiment Design

Model Selection

EXISTING NEW

Modify As-Is


Run Matrix



FINAL REPORT


Step 6. Reporting

PROBLEM


STUDY OBJECTIVES




Conduct Analysis

Conduct Runs


Step 5. Execution

44

Analysis Design

• What is the mechanism we are demonstrating? • How will the effect (if any) be realized? • What are the available inputs and expected

outputs?

• A ‘use case’ outlines the setting(s) “Picture this…”

45

Model Selection

• Which fast-time model is most appropriate? • Considerations

Fidelity Scope Availability Setup

46

Factor Selection

• Factor = variable that will change • Level = setting or value of the factor

47

Traffic Schedule

Forecast Year 2020

Forecast Year 2030

Automation Parameters

Setting 1

Setting 2

New Prototype

On

Off

Simulation Scenarios

• Baseline: reference point, aka “control” Current operations Future conditions

• Treatment: modification(s) Flights (count, attributes) Procedures Automation

48

Design of Experiments (DOE)

• Systematic approach to study the relationship between factors and responses

• “One factor at a time” vs factorial experiment Capture interactions Run efficiently

• Custom DOE built using JMP®

50

Theoretical Model

51

Rjk = response Yj = forecast years, j = 1, 2 Ak = automation parameter, k = 1, 2, 3 εn(jk) = random error, n = 1, 2, … for all j, k

Rjk = µ + Yj + Ak + YjAk + AkAk + εn(jk)

Project Example: CRA Benefit Study

• Conflict Resolution Advisories Decision support tool Provides efficient 2-

part resolution maneuvers

Underlying resolution intent entry creates a closed-loop system

Inter-sector coordination of trial plans

52

53

A

C

Controller intent

Ground automation

FIX

CRA Benefit Study: Overview

• Investigate benefit from improved entry of controller intent Knowledge of resolution intent improves trajectory

modeling and conflict prediction accuracy • Model varying levels of intent by removing

complete 2-part amendment clearances and replacing with open clearances

54

CRA Benefit Study: Analysis Design

• 3 controllable factors Center (ARTCC) – 5 levels Traffic level (year) – 2 levels Intent entry (% of clearances

entered) – 5 levels • Experiment design: full factorial • Analysis

Internally developed suite of software tools

55

Process Steps

56


METRICS



DATA • System

• CONOPS

• Environment

• Scenarios


• Data Assumption

• Data Validation

Experiment Design

Model Selection

EXISTING NEW

Modify As-Is


Run Matrix



FINAL REPORT


Step 6. Reporting

PROBLEM


STUDY OBJECTIVES




Conduct Analysis

Conduct Runs


Step 5. Execution

57

“I’ve forgotten what this experiment is all about”

Simulation Test Plan

• Document approach, for sanity and transparency

• Test plan components Problem definition Assumptions Data sources Protocol for data analysis Run matrix

58

Run Matrix

• Write it out • Specify

Factor levels Scenario definitions Total number of runs

• Get agreement from stakeholders • Use for tracking progress

59

Project Example: Space Vehicle Operations

60

SVO Run Matrix

62

Scenario ID

Airspace Closure Strategy

Forecast Year

Number of Flights

SV Operations

Level

Number of SV

Operations

ARTCC Location of SV Operations

BL2018 None 2018 51,749 None 0 N/A

BL2025 None 2025 56,478 None 0 N/A

Current Day 1 Current

2018 51,749 Low 3 ZAB, ZDC, ZFW, ZHU 4DE Day 1 4D Compact

Envelopes Current Day 2 Current

2018 51,749 Medium 4 ZAB, ZDC, ZLA, ZOA 4DE Day 2 4D Compact


2018 51,749 High 7 ZAB, ZDV, ZFW, ZHU, ZJX, ZMA 4DE Day 3 4D Compact


2025 56,478 Low 6 ZAB, ZAN, ZFW, ZHU, ZJX, ZLA,

ZMA 4DE Day 4 4D Compact Envelopes

Current Day 5 Current

2025 56,478 Medium 8 ZAB, ZDV, ZFW, ZHU, ZLA 4DE Day 5 4D Compact


2025 56,478 High 15

ZAN, ZDC, ZDV, ZFW, ZHN, ZHU,

ZJX, ZKC, ZLA, ZMA, ZOA 4DE Day 6 4D Compact

Envelopes


THROUGH M&S

METRICS



DATA • System

• CONOPS

• Environment

• Scenarios


• Data Assumption

• Data Validation

Experiment Design

Model Selection

EXISTING NEW

Modify As-Is


Run Matrix



FINAL REPORT


Step 6. Reporting

PROBLEM


STUDY OBJECTIVES




Conduct Analysis

Conduct Runs


Step 5. Execution

63

Execution Phase

• Construction of model • Model validation &

verification • Model execution • Data analytics • Credibility assessment

64

Model Construction

• Input data collection Air traffic

• Recorded track data, flight plans, forecasted traffic schedules Aircraft Performance Model (APM)

• EUROCONTROL’s Base of Aircraft Data (BADA) Airspace adaptation

• Navigational aids, airways, sector boundaries, airport locations Operational knowledge

• Capturing key decision points & logic Other infrastructure

• Airport runways, taxiway usage, Special Activity Airspace (SAA) • Data transformation

Information gained Information the model can use

65

SVO Project: Input Data Collection

• Blue Origin Planned Launch – Van Horn, Texas on August 24, 2011 NOTAM Facility: ZAB Albuquerque NOTAM number: FDC 1/3552

• Atlas Launch – Vandenberg Air Force Base, California on February 11, 2013 NOTAM Facility: ZLA Los Angeles NOTAM Number: 02/095, 02/192, 02/193, 02/194, 02/195, 02/197 NOTAM Facility: ZAK Oakland NOTAM Number: 02/096

• SpaceX Falcon 9 Launch – Cape Canaveral, Florida on March 1, 2013 NOTAM Facility: ZMA Miami NOTAM Number: A0177/13, FDC 3/1587

• Orbital Sciences Pegasus Launch – Vandenberg Air Force Base, California on June 27, 2013 NOTAM Facility: ZAK Oakland NOTAM Number: 06/134

66

SVO Project: Model Construction

67

2018 Scenarios Low

Time (Local) Time (UTC) Locations State Space Vehicle Timing of Airspace Closure (UTC) Azimuth Notes

9:00 16:00 Spaceport America NM Virgin Galactic SpaceShip 2 15:50-16:20 0 SME input to assume 10min before, 20min after launch

11:20 16:20 Wallops VA Orbital Sciences Pegasus W-386: 16:05-16:47; others: 16:05-16:47 150 based on Pegasus launch from VAFB on 06/27/2013

14:15 20:15 Midland TX XCOR Lynx 20:05-20:35 120 SME input to assume 10min before, 20min after launch

Medium


5:30 13:30 Vandenburg Air Force Base CA United Launch Alliance Atlas

V W-289S,W-537, W-532S/E/N: 13:13-17:28; others: 13:15-

17:57 191 based on launch on 02/11/2013

10:00 15:00 Wallops VA Orbital Sciences Antares W-386, W-72A/B: 14:53-16:23; others: 14:53-16:23 110 based on launch on 07/13/2014


15:23 23:23 Pacific Ocean N/A SpaceX Dragon Reentry 23:03-23:30 135 based on Dragon reentry on March 26, 2013

High


6:30 11:30 Titusville FL Virgin Galactic SpaceShip 2 11:20-11:50 50 SME input to assume 10min before, 20min after launch

8:20 14:20 Van Horn TX Blue Origin PM2 14:10-14:40 0 SME input to assume 10min before, 20min after launch

11:10 16:10 Cecil Field FL XCOR Lynx 16:00-16:30 180 SME input to assume 10min before, 20min after launch


12:00 19:00 White Sands Missile Range NM Sounding Rocket R-5107H & R-5107E: 18:00-21:00 0 airspace based on 07/22/2014 launch at WSMR; timing based on single launch from Wallops on 07/20/2012 and confirmed by SME

13:50 19:50 Midland TX XCOR Lynx 19:40-20:10 0 SME input to assume 10min before, 20min after launch

16:45 23:45 Front Range CO XCOR Lynx 23:35-00:05 150 SME input to assume 10min before, 20min after launch

SVO Project: Data Transformation

68

Model Validation & Verification

• Is model compliant with the simulation plan? Data sources Scope of model

• Are the various components functioning correctly within the model? Implementation of operational knowledge Feasibility of APM Reroute logic

• Is the model a feasible representation of reality? Simulation settings Reasonable flight operations

69

Did I build the right model?

Did I build the model right?

http://images.clipartpanda.com

/question-Kin57o5iq.gif

Model Execution

• Always do several test runs with all integrated components Unexpected simulation

behaviors • Develop a hardware

schedule Formal or informal checklist Useful in managing

computing resources • Have contingency plans

Anticipate hardware malfunctions

Backup simulation data • Follow run matrix

70

Conduct Analysis

• Capture Simulation log Output format

• Collect Database Data Files

• Process and analyze Format conversion Statistical software (COTS) Specialized analysis tools

• Summarize results Interpret findings

71


• “Sanity checking” data analytics results Checking against

“truth” data available • Further examination of

data outliers • Document lessons

learned The good, bad, and

future areas of research

72

SVO Project: Conduct Analysis

73

SVO Project: Analysis Results

74

Number of rerouted flights by scenario

Difference in average flight distance from baseline scenarios

Process Steps

75


METRICS



DATA • System

• CONOPS

• Environment

• Scenarios


• Data Assumption

• Data Validation

Experiment Design

Model Selection

EXISTING NEW

Modify As-Is


Run Matrix



FINAL REPORT


Step 6. Reporting

PROBLEM


STUDY OBJECTIVES




Conduct Analysis

Conduct Runs


Step 5. Execution

Reporting

• Target audience Stakeholders Research community at

large Management

• Methods Government technical note White papers Memorandums Presentation briefings Journal publications Conference papers Conference presentations

76

Process Steps

77

Project Plan

Metric Selection

Analysis Design

Simulation Plan Execution Reporting

QUESTIONS?

78

application of fast-time computer modeling for atm...

Documents