Dr. Tulinda LarsenVice Presidenttulinda@masflight.comMobile. +1 (443) 510-3566

4833 Rugby Avenue, Suite 301Bethesda, Maryland 20814www.masflight.com

Cross-Platform Aviation Analytics Using Big-Data Integration Methods

2013 Integrated Communications Navigation and Surveillance (ICNS) Conference

April 25, 2013

The Analysis ChallengeT H E A N A L Y S I S C H A L L E N G E

Scale and complexity of aviation data limits research applications

Obtaining radar and airport data, schedules, weather maps and forecasts, fleet information

• Real-time transmission of very large data

• Proprietary and inconsistent formats

• No conditioning or validation

Problems Acquiring Data

Problems Analyzing InformationUsing data for strategic planning and recovery,cost improvement and new market opportunities

• Goes beyond desktop capability

• Time-consuming manual slicing of data

• Need weather and competitor information to answer key operational questions

Fuel and OilConservation

Gate and Terminal Use

Weather Plan & Ops Recovery

Pilot and Crew Staffing

OperationalOptimization

Big-data analytical methods can address these challenges

• Figurative “Cloud” The term comes from the common use of a cloud-shaped symbol as an abstraction for the Internet but application to virtual servers is as recent as 2006

• Cloud computing is the use of resources (hardware and software) that are delivered as a service over the Internet or other network

What is Cloud Computing?

C L O U D C O M P U T I N G

The cloud consists of terrestrial servers across the Internet that collectively store, manage and process data

ApplicationPlatform

Infrastructure

Monitoring

Content

Object Storage

Computation

StorageCollaboration

Communication

Financial Metrics

Content

Identity

Databases

Network

Cloud Computing Architecture

Public Cloud Providers

Community Cloud

Providers

Managed Services Private

Providers

Company Private Clouds

Shared between users

Private Infrastructure

Low Industry Expertise

High Industry Focus

• Cloud computing services can be delivered by an internal IT organization (company-owned private cloud) or

• By an external service provider (managed services private cloud or public cloud provider) or

What are Cloud Architectures?

C L O U D C O M P U T I N G

In aviation cloud resources can be customized and shared among consortiums of customers (community cloud) or

shared with customers in other industries (public cloud)

What is Big-Data Analytics?• The process of examining diverse, large-scale data sets to uncover

patterns, unknown correlations and other useful information

• Organizations have different levels of (1) database management expertise and (2) knowledge to process and analyze big data sets– “Big data” is a relative term based on the user

– Data tables in excess of ten terabytes (10TB) are difficult to work with using most relational database management systems, and particularly using desktop statistics and visualization packages, including Microsoft Excel and Access

• Unstructured data sources in the operational world simply do not fit into desktop or small-scale database structures– They can be hosted using cloud computing at lower cost, and mined

more efficiently, than with on-premises database architectures

B I G - D A T A A N A L Y T I C S

What are Big-Data Analytics Tools?• Big-data analytics employ software tools from advanced analytics

disciplines such as data mining and predictive analytics.

– Mining data, trends or analysis of these multi-terabyte data sets requires parallel software running on tens, hundreds, or even thousands of servers to keep pace with user demands and processing expectations.

• A new class of big-data methods have emerged to address user demands for horizontal scaling and availability of underlying data

– Hadoop and MapReduce, among others, offer fast processing speed.

– Great for large-scale static data sets, but not so great for real-time data

– Most organizations employ a hybrid method combining technologies

• A robust open source framework supports processing in clustered systems.

• Platform-as-a-service vendors (Microsoft, Amazon, Google) offer turn-key solutions for analysts to simply upload, link and compute basic data sets

– Great for simple historical analysis; bad for real-time or diverse data sets

B I G - D A T A A N A L Y T I C S

masFlight: A Global Aviation Data Warehouse and Big-Data Analytics Platform

M A S F L I G H T

Hybrid Architecture

• Physical architecture for secure data feeds

• Cloud-based instances for linking

• Managed cloud data tables

• Integrates with local BI and warehouses

Redundancy

• Multi-source data acquisition

• Real-time validation and processing

• Replication across cloud infrastructure

• Load balancing and parallel processing

Backup

• Cluster processing to reduce dependencies

• Monitored data integrity and performance

• Multiple geographic zones and clusters

• Imaging of tables for replication

Customization

• Customizable for specific user requirements

• Dashboards and web templates

• Integrated internal data in warehouse

• Connect to local BI systems

D A T A A N D A P P L I C A T I O N S

masFlight’s Data and Applications Platform

Data Input Feeds

Airport & Gate Status Multisource, real-time feeds

Current WeatherGlobal hourly conditions

Forecast WeatherStandard and severe forecasts

Flight SchedulesWhat’s planned to operate

Reference and Static DataGeospatial, airline, airport info

Government Economic Data Revenue and audited data

Secure U.S./Canada Radar

Authorized direct accessOther Airspace DataSatellite and transponder info

In-House ServersFor private gov’t feeds

Cloud WarehouseLinked Information

60TB structured data

Robots and Java Applications

Secure External Network

Automated collection

OUR CLOUD-BASED DATA WAREHOUSE OUR CUSTOMER APPLICATIONS

Web Application(masflight.com)

Cloud Managed Database Hosting

Dashboards & Web Services

HTML 5 / RubyAnalyst focusedCustomizableFast deploymentSaaS revenue model

REST web servicesFeed internal systemsCustom dashboardsFlexible interfaces

Virtual tablesUpdated in real timeBypass constraintsUltimate customization

masFlight Platform

M A S F L I G H T P L A T F O R M

Multisource, integrated airline operations data

Our platform shows where, when and why problems occur

• Examine diversions, cancellations, delays and determine root causes

• Deep-dive into airport gates, taxi times, and runway patterns

• Analyze air space usage and air traffic management

Planned Flight Schedules

Multisource Flight Status

Global WeatherData and Maps

AirlineOps Data

U.S. Radar Data

Airline FleetInformation

Airport Gate & Terminal Data

AirportRunway Data

Key Partners and Suppliers:

Origin weatherOrigin informationOperating airlineScheduled timesDeparture gate/timeTaxi-out/takeoff times

Arrival weatherDestination information

Landing/taxi timesArrival gate/time

Diversion dataAircraft information

Flight plan filedActual path flown

CongestionWeather diversions

En-route times and fixes

KIAD V268 SWANN

1502Z 1550Z 1620Z

E N D T O E N D C A P A B I L I T Y

Big-Data Analytics Facilitates End-to-End AnalysisA full picture of each flight is critical for analyzing operations

Query flights from planned schedule through post-operation recoveryUp to 500 data points per flight

Other sources only offer limited, disaggregated and unformatted regional data

A Global Solution

C O V E R A G E

North and South America EMEA and Asia

White lines are flights in the masFlight platform from February 8, 2013.Yellow pins are weather stations feeding hourly data to our platform.

Maps from Google Earth / masFlight

masFlight tracks flights, airports and weather around the world

• Global daily flight information capture ― 82,000 flights― 350 airlines ― 1700 airports

• Integrated weather data for 6,000 stations― Match weather to delays ― Validate block forecasts

at granular level― Add weather analytics to

IRROPS review and scenario planning

Example 1: Proposed FAA Tower ClosuresmasFlight used big-data to link airport operations across three large data sets:

– Current and historical airline schedules

– Raw Aircraft Situation Display to Industry (ASDI) radar data from the FAA

– Enhanced Traffic Management System Counts (ETMS), including Airport operations counts by type (commercial, freight, etc.), departure & arrival

T O W E R C L O S I N G S

Dots indicate closures; Red dots have scheduled service

Based on scheduled service March 1 – 7, 2013; scheduled service includes scheduled charter flights, cargo flights, and passenger flights

Findings: Proposed Tower Closings• From schedules database: 55 airports with

scheduled passenger airline service– 14 EAS Airports

• From ASDI & ETMS: 10,600 weekly flights on a flight plan (ex. VFR and local traffic)

– 6,500 Part 91/125 weekly flights

– 4,100 Part 135/121 weekly flights

Example 1: Big-Data Analytics Applied to ASDI and ETMS To Analyze Operations

T O W E R C L O S I N G S

Up to 5 5-10 10-15 15-20 20-25 25-30 30-35 35-40 40-45 45+

26

44

24 23

11 106

2 1 2

Distribution of Airports By Average Number of “Daily” Impacted Flights

Airports Affected by Tower Closures

Average Number of Daily Operations with a Flight Plan Filed

Cou

nt o

f Airp

orts

Source: ASDI radar data – Part 91/151 flying and Part 135/121 flying – March 1-7, 2013; masFlight analysisNote: Average “daily“ operations based on 5-day week

Example 2: Aviation Safety Causal Factor

For example, consider the following ASRS report (ACN 1031837):

“Departing IAH in a 737-800 at about 17,000 FT, 11 miles behind a 737-900 on the Junction departure over CUZZZ Intersection. Smooth air with wind on the nose bearing 275 degrees at 18 KTS.

We were suddenly in moderate chop which lasted 4 or 5 seconds then stopped and then resumed for another 4 or 5 seconds with a significant amount of right rolling… I selected a max rate climb mode in the

FMC in order to climb above the wake and flight path of the leading -900. We asked ATC for the type ahead of us and reported the wake encounter. The -900 was about 3,300 FT higher than we were.”

• Synopsis– B737-800 First Officer reported wake encounter from preceding B737-900

with resultant roll and moderate chop.

What causal factors can be identified from this narrative thatcould be applied to future predictive applications?

C A U S A L F A C T O R S

Data-mining algorithms can mine the text of safety reports to obtain specific data that can be used to analyze causal factors.  

Example 2: Identifying Causal Factors

C A U S A L F A C T O R S

Indicators – Data Element Methods – Identifying Context and Causes

• Time of day

• Date range (month, day)

• Aircraft type

• Fix or coordinates

• Originating airport

• Destination airport

• Weather notes

We pinpoint the sequencing of flights on the IAH Junction Seven departure (at CUZZZ) during the specified wind conditions to find cases where a B737-900 at 20,000 feet precedes by 11 miles a B737-800 at 17,000 feet

• Search related data sets including ASDI(flight tracks, local traffic and congestion)

• Weather conditions for alternative causes (winds aloft, shear and convective activity)

• Airline specific information (repeated occurrence of event in aircraft type)

Big data gives us visibility into contextual factors even if specific data points are missing such as a specific date or route.

Big-data analytics gives us insight into unreported factors as well.

Example 3: Correlating Utilization and Delays

7 8 9 10 11 12 1360%

65%

70%

75%

80%

85%

90%

95%

100%

HOURS OF DAILY UTILIZATION

ON

TIM

E D

EPA

RTU

RE

PER

FOR

MA

NC

E

7.0 8.0 9.0 10.0 11.0 12.0 13.060.0%

70.0%

80.0%

90.0%

100.0%

NarrowbodiesBy Day of Week

7.0 8.0 9.0 10.0 11.0 12.0 13.0 14.060.0%

70.0%

80.0%

90.0%

100.0%

Widebodiesby Day of Week

Daily Utilization vs. On-time DeparturesJanuary 2013 System Operations

Correlation Coefficient -0.53

Includes AA, AC, AS, B6,

F9, FL, NK, UA, US, VX and WN

SOURCE: masFlight (masflight.com)

C O M P A R I N G O T P A N D U T I L I Z A T I O N

LGB

JFK

BOS

MCO

DCA

FLL

6.2

6.0

5.8

5.8

5.2

4.9

JetBlue FocusAverage Daily Deps

per Gate Used

U T I L I Z A T I O N B Y H U B

Example 4: Daily Utilization of Gates, by HubBig-data analysis of different carriers – daily departures per gate used

SOURCE: masFlight (masflight.com)

June 1 through August 31, 2012. Gates with minimum 1x daily use

ORDLAXSFOEWRDENIAHIAD

CLE

7.7 7.4

7.2 6.2 6.1

5.8 3.8

3.6

United Airlines HubsAverage Daily Deps

per Gate Used

SEASANPDXANCSFOGEGLAXSJC

7.8 6.4

5.5 5.4

5.3 4.4 4.3

4.0

Alaska Airlines HubsAverage Daily Deps

per Gate Used

ORD

DFW

LAX

LGA

MIA

JFK

7.2

6.9

6.8

6.4

5.0

2.7

American HubsAverage Daily Deps

per Gate Used

CLT

DCA

PHL

PHX

BOS

7.2

6.9

6.6

4.9

4.2

US Airways HubsAverage Daily Deps

per Gate Used

MCO

BWI

ATL

MKE

6.6

5.9

5.5

4.7

AirTran HubsAverage Daily Deps

per Gate Used

Conclusions for Big Data in Aviation• Big-data transforms operational and commercial problems that were

practically unsolvable using discrete data and on-premises hardware

• Big data offers new insight into existing data by centralizing data acquisition and consolidation in the cloud and mining data sets efficiently

• There is a rich portfolio of information that can feed aviation data analytics

– Flight position, schedules, airport/gate, weather and government data sets offer incredible insight into the underlying causes of aviation inefficiency.

– Excessive size of each set forces analysts to consider cloud based architectures to store, link and mine the underlying information

– When structured, validated and linked, these data sources become significantly more compelling for applied research than they are individually

• Today’s cloud based technologies offer a solution

C O N C L U S I O N S

Conclusions: Our Approach• masFlight’s data warehouse and analysis methods provide a

valuable example for others attempting to solve cloud based analytics of aviation data sets

• masFlight’s hybrid architecture, consolidating secure data feeds in on-premises server installations and feeding structured data into the cloud for distribution, addresses the unique format, security and scale requirements of the industry

• masFlight’s method is well suited for airline performance review, competitive benchmarking, airport operations and schedule design, and has demonstrated value in addressing real-world problems in airline and airport operations as well as government applications

C O N C L U S I O N S