Slide 1

Free Flight Phase 1 & 2 Performance Measurement Dave Knorr May 16, 2002 Slide 2 2 Outline Free Flight Phase 1 and 2 Measurement Process How and What do we measure? Mechanisms versus Operational Benefits Free Flight Phase 1 Case Studies Other Exploratory Metrics Slide 3 3 Free Flight Phase 1 & 2 Established to deliver new automation tools to controllers at limited locations User Request Evaluation Tool (URET) Traffic Management Advisor (TMA) Passive Fianl Approach Spacing Tool (pFAST) Collaborative Decision Making tools (CDM) Metrics program developed in collaboration with stakeholders Expanded deployment (FFP2) based on FFP1 measured benefits Slide 4 4 User Request Evaluation Tool Capabilities R-side Controller D-side Controller Assistant Controller Primary User En Route Facility For controllers providing en route services, URET offers the following: Automatic conflict detection Trial planning to assist with conflict resolution or user requests Conformance monitoring of current trajectory Electronic flight data capability Slide 5 5 Traffic Management Advisor TMA is a time-based system that: Provides a schedule for all arrivals Computes arrival times to the runway and other designated fixes Handles dynamic traffic situations Assists the controllers in meeting schedules by providing advisories Enhances the sequencing process by refining super stream class protocols based on prop and jet engine characteristics Slide 6 6 Measuring Impacts of New Tools Why is measurement of performance in an operational environment important? Can we measure actual performance of specific tools? Slide 7 7 Why Measure? Verify projected benefits from models/simulations Feedback to program management Accountability for capital investments Improve future goal setting and models Model the future Measure Establish goals Slide 8 8 How Do We Measure? In Actual Operational Environment With: Real people: controllers, pilots, dispatchers, etc. (not simulated) Varying weather conditions and seasons Varying aircraft mix Varying airport conditions/configurations Partnership With Industry (airlines/airports/GA/others) - share data, share findings and results validation Over significant time periods - before and after implementation Slide 9 9 What Do We Measure? Quantitative Outcomes Actual Operational Change Mechanisms for Improvement Example: Increasing AmendmentsExample: Distance Savings Slide 10 10 What Do We Measure? Quantitative Outcomes Mechanisms for Operational Change: Are tools actually being used during study periods? Are tool recommendations being followed? Is there smoother flow into TRACON airspace? Are trial plans being used? Is controller workload reduced? These metrics should correlate with operational benefits measured. Slide 11 11 What Do We Measure? Quantitative Outcomes Actual Operational Change Throughput During Peak Activity Periods An indicator of capacity changes Efficiency Reduction of flight times and/or distances; fuel usage Predictability Reduction in variation of flight times Delay A good indicator of system problems Not a good indicator of capacity increases FFP Focus Slide 12 12 These Measures Interact Demand Delay Capacity Throughput Capacity Reduced Flight Time Increases Variation in Flight Time Increases Throughput Decreases IFR Weather Slide 13 13 These Measures Interact Demand Delay Capacity Throughput FFP TOOLS Throughput Increases Capacity Increases Demand More Demand Can Be Accommodated and/or Delay Decreases Slide 14 14 Runway Counts and Flying Times Runway Counts (EA/OA) (OA/IA) (IA/MA) (MA/RW) Slide 15 15 Key Findings from Free Flight Phase 1 Must be able to attribute operational change to controller actions (mechanisms) Focus on peak periods Include downstream affects of operational improvements Slide 16 16 Metrics on Operational Performance Changes TMA and URET Case Studies Slide 17 17 Metered Flow of Arrivals Departure Routes Slide 18 18 Unmetered Flow of Arrivals Departure Routes Increased Vectoring Increased Conflicts Missed Slots Long Final Slide 19 19 TMA Supports Efficient Transition Without metering TRACONs can become underutilized or overloaded Metering increases throughput Slide 20 20 CM Sim Data TMA Details CM_Sim : /export/pb Start Time : Tue May 22 14:49:33 2001 Stop Time : Tue May 22 23:15:22 2001 Total Aircraft in cm_sim (including those without crossing or frozen message) : 567 Meter Fix STA to MF STA-Crossing Crossing Host CTAS MF at Crossing Difference Time ACID MF MF TYPE Freeze Time (seconds) ----------- -------------- ------ ----- ----- ---------- --------- --------- 14:50:08Z AAL1001/TUL.B0573 CIVET CIVET MD8 14:49:53Z 14:50:08Z 15 14:50:11Z KAL062/SBGR.B0328 VISTA VISTA B747 14:50:08Z 14:50:11Z 3 14:50:16Z UAL1769/DEN.B0828 CIVET CIVET B757 14:50:08Z 14:50:16Z 8 14:50:38Z EGF214/SAN.B0074 PEBLE DH8 14:50:25Z 14:50:38Z 13 14:51:11Z AAL2821/DEN.B0519 CIVET CIVET B737 14:51:27Z 14:51:11Z 16 14:52:51Z EGF200/SBP.B0658 VTU VTU DH8 14:49:33Z 14:52:51Z 198 14:53:35Z AAL691/LAS.B0295 CIVET CIVET MD8 14:53:27Z 14:53:35Z 8 14:53:53Z SIA7962/PANC.B0567 FIM FIM B74F 14:53:53Z 14:53:53Z 0 14:56:03Z EGF732/SAN.B0621 PEBLE DH8 14:55:52Z 14:56:03Z 11 14:56:18Z MXA912/GDL.B0161 VISTA VISTA B727 14:56:33Z 14:56:18Z 15 14:59:20Z SWA753/OAK.B0148 FIM FIM B73V 14:59:15Z 14:59:20Z 5 15:00:10Z AMX6490/MEX.B0920 VISTA VISTA B757 15:00:12Z 15:00:10Z 2 15:00:13Z N667CX/DAL.B0897 KONZL KONZL G2 15:00:06Z 15:00:13Z 7 15:00:43Z SKW5112/ONT.B0723 LAHAB IARN E120 15:00:46Z 15:00:43Z 3 . .. Slide 21 21 CM Sim Data TMA Summary Arrival Rate Accuracy Summary CM_Sim : /export/pb Start Time : Tue May 22 14:49:33 2001 Stop Time : Tue May 22 23:15:22 2001 Total Aircraft : 298 Total Total Aircraft Difference Accuracy Crossed STA/Crossing Percentage ---------- ----------- ------------ 298 8 97.32 Slide 22 22 TMA Operational Benefits Improved Center to TRACON feed supports higher airport throughput MSP up 4%+ Denver up 2%+ Metering supports delay distribution to more fuel efficient altitudes Miami delay distribution improved significantly Atlanta delay distribution improved at front end of rush Internal Departures receive improved ATC services Slide 23 23 MSP Actual Arrival Peak-Times Slide 24 24 MSP Daily Cumulative Arrivals 12-18 March 2000 Slide 25 25 MSP Peak Period Cumulative Arrivals 16 March 2000 Slide 26 26 Before TMA vs After TMABefore TMA vs Oct 01 MSP Throughput Post 9/11 Throughput Higher Than Baseline Slide 27 27 Impact of TMA at DEN Arrival Rate up 1-2 an hour when stressed DEN only stressed ~2% of the time Slide 28 28 Arrival Demand & AAR at DEN, 23 OCT 00 Slide 29 29 MIA Arrival Streams Outer Arc 180 nmi. Meter Arc 40 nmi. Slide 30 30 Percentage of Flight Time Spent in TRACON Time flown from meter arc to runway divided by time flown from outer arc to runway Only aircraft crossing meter arc during peaks included Error bars represent 95% confidence intervals Difference is statistically significant Slide 31 31 Slide 32 32 Impact of TMA on ZLA internal departures to LAX Gate+Air delay down 23% at smaller airports, 10% at LAS For ZLA-released airports with at least 1 flt/day into LAX Based on ASPM data Slide 33 33 ATL Internal Departures Slide 34 34 Measuring Operational Benefits of URET Clear evidence of URET usage and increased controller productivity Measurable improvement in distance reduction through route amendments Insignificant improvement in flight times Measurable improvement in efficient altitudes for descending a/c Slide 35 35 ZME: January 02 Average TPs, Amendments, and Tracked Aircraft Count by ATC Area Avg TP/dayAvg Accepted AM/day Avg Tracked AC/day Data through 26 January 02 Slide 36 36 ZME: Direct Routing Amendments Notes: - Data Sampling: 2 days/week; between 14Z and 22Z - URET 2-way processing began in July 99 - Includes any Lateral Amendment processed by Host Data through 22 March 02 Slide 37 37 ZID data through 27 Mar; ZME through 22 Mar. Distance Savings for Lateral Amendments Notes: - Data Sampling: 2 days/week - ZID between 13Z and 23Z; ZME between 14Z and 22Z - URET 2-way processing began in July 99 - Includes any Lateral Amendment processed by Host Slide 38 38 EnRoute Distance; shows consistency Slide 39 39 EnRoute Time w/out Seasonal Correction Slide 40 40 EnRoute Time with Seasonal Correction Slide 41 41 Commercial/NRP metrics at ZME. Slide 42 42 URET Direct Savings Direct Route Route Structure Slide 43 43 Average Excess Distances Flown Slide 44 44 Example: Lifting of Restriction 390 370 350 330 310 290 230 250 230 Flight Level 200 Dayton HI (88)Appleton HI (87) TRACON IND 0 98 NMI Trajectory without restriction Trajectory with restriction LO SECTORS 2/24/00: IND arrivals 87/88 at FL310 ZID Airspace Ultra-HI SECTORS Slide 45 45 History of Restriction Relaxation Status ZID and ZME Slide 46 46 Exploratory Metrics Analyzing tool benefits during bad weather Slide 47 47 Traffic Density and Weather Slide 48 48 Back-ups Slide 49 49 Raw Database Logs Data Ingestion NCDC Weather Data Ingestion Database Overview Analysis Database Airport Flight Aircraft Metrics Analysis Host/ARTS Data Ingestion Track Weather Contains information for all flights recorded by the respective computer (ARTS or Host) Contains airport AAR, runway configuration, and restrictions Contains winds, visibility, precipitation, etc. at the airport Data Wrapper Slide 50 50 FFP Metrics Lessons Learned/What works Terminal Area Performance Changes are measurable: Increased throughput during peaks indicates increased capacity Clear Objective Functions: Increased throughput, decreased flight times Normalization achievable (demand, conditions, etc.) Automated analyses possible Slide 51 51 Analyzing Peak terminal Throughput Focus on peaks where throughput is constrained by capacity During slow traffic periods, there is little or no benefit with new tools Determine when system is stressed Demand exceeds capacity Desire to measure throughput not constrained by demand Determine criteria for minimum peak period May depend on site Slide 52 52 MSP Actual Arrival Peak-Times (July 2000) Slide 53 53 Excess Distance Based on ETMS data Draw 100-nmi ring Calculate excess distance compared to great circle Slide 54 54 Daily Total Excess Distance between city pairs (Top 31 airports) Look closer at selected pairs Slide 55 Representative Delay Base CaseFFP1 Case Slide 56 Representative Cumulative Delay Slide 57 57 Meter Fix Arrival Peaks Example Day MIA Jan 14, 2001 Slide 58 58 Arrivals into MSP Average Weekday Runway Count During 5-minute Bins (Jan-Mar 2001) Arrival Banks Peaks indicate capacity of ~5 arrivals during a 5-min period (MSP mixes arrivals and departures on 2 primary runways) Slide 59 59 Average Excess Distance between city pairs (Top 31 airports) Slide 60 60 Putting It All Together Drop EA/OA Time Since No Delay Apparent Show When Metering Generally Occurs Runway Counts (OA/IA) (IA/MA) (MA/RW) Slide 61 61 Runway Counts and Flying Times Full Day Runway Counts (EA/OA) (OA/IA) (IA/MA) (MA/RW) Slide 62 62 Flights On-Time Avg Delay Per Flight Ground Stop Delay Ground Hold Delay % Flown as Filed Avg Daily Capacity OEP/FFP Measures Ground, Terminal and Enroute for Each Flight Time/Distance Flight Time Variance (Predictability) Capacity/ Throughput PBO Measures Cost Per Flight Safety OE/OD/RIs Slide 63 63 Delay Distribution Runway Counts (OA/IA) (IA/MA) (MA/RW) Slide 64 64 Measurement & Reporting Objectives Program Management Focuses Efforts Establishes Expected Outcomes Exercise Management Oversight Future Decision Making Communicate with stakeholders Feedback to service providers, users Customer Accountability Shareholder Accountability Slide 65 65 What Do We Measure? There are four measures that cover most of what people want to know about NAS performance: 1. Capacity How many aircraft are possible? 2. Peak Throughput How many can get there? 3. Flight Time How long does it take to get there? 4. Variation of Flight Time Can we plan on it? These measures can be applied to overall NAS performance as well as Terminal or En Route Slide 66 66 Runway Counts and Flying Times Full Day Runway Counts (EA/OA) (OA/IA) (IA/MA) (MA/RW) Slide 67 67 URET at ZKC IDU 12/3/01 As of 3/28/02, Over 55% of Workforce has Completed URET Training (Over 8,000 Training Hours) Benefit Anecdotes: 20 minute conflict probe works good Increased time for strategic planning More direct routings are being given Slide 68 68 URET Daily Use Metrics at ZID and ZME Count of Directs Define Direct to be all route amendments which reduced the remaining flying distance For busy hours (ZID: 13Z - 23Z; ZME 14Z-22Z) Since May 99, processing 2 days/week Lateral Distance Saved For all lateral amendments (includes directs as well as penalties), the average of the daily sum of nautical miles changed. Slide 69 69 ZID: Direct Routing Amendments Notes: - Direct Data Sampling: 2 days/week; between 13Z and 23Z - URET 2-way processing began in July 99 - Includes any Lateral Amendment processed by Host Data through 27 March 02 Slide 70 70 Related Activities ZID A URET benefits team has been created, includes airline participation. Any proposed new airspace restrictions will be evaluated by the URET benefits team for validity, prior to implementation. ZME ZME controllers have started dynamically lifting crossing restrictions for SDF arrivals before the formal evaluation is in place. Slide 71 71 Arrivals into MSP Average Weekday Runway Count During 5-minute Bins (Jan-Mar 2001) Arrival Banks Peaks indicate capacity of ~5 arrivals during a 5-min period (MSP mixes arrivals and departures on 2 primary runways) Slide 72 72 Slide 73 73 EnRoute Time -- East/West city-pairs Slide 74 74 Extrapolated Savings ZID Feb. 24-25 Restriction Relaxation Evaluation Per Restriction: fuel saved ~113 gallons Evaluation period represents approximately 40% of total traffic for day Per Day: potential fuel saved: ~282 gallons Per Year: potential fuel saved: ~102,930 gallons ZID identified restrictions for possible removal testing (~70-368) Per Year: potential fuel savings: ~7,205,100 gallons Slide 75 75 Methodology Delays - Where, When, and How Much Airspace around MSP divided into imaginary rings Data collected on flights entering these rings Times at which aircraft cross each arc Flying time through each ring