elizabeth “scottie-beth” fleming committee members: dr. amy pritchett, chair dr. karen feigh
DESCRIPTION
DEVELOPING A TRAINING PROGRAM FOR THE TRAFFIC ALERT AND COLLISION AVOIDANCE SYSTEM IN CONTEXT. Elizabeth “Scottie-Beth” Fleming Committee Members: Dr. Amy Pritchett, Chair Dr. Karen Feigh Dr. Ute Fischer Sponsored by the FAA, Tom McCloy as Technical Monitor. Overview. - PowerPoint PPT PresentationTRANSCRIPT
Elizabeth “Scottie-Beth” Fleming
Committee Members:Dr. Amy Pritchett, Chair
Dr. Karen FeighDr. Ute Fischer
Sponsored by the FAA, Tom McCloy as Technical Monitor
DEVELOPING A TRAINING PROGRAM FOR THE TRAFFIC ALERT AND COLLISION
AVOIDANCE SYSTEM IN CONTEXT
2
Overview
+ Introduction to TCAS and Background Information+ Design of Training Program+ Evaluation of the Training Program+ Results+ Conclusions
Develop a training program intended to improve pilots’ understanding of TCAS use for collision avoidance in a
range of traffic situations
3
Traffic alert and Collision Avoidance System (TCAS)
+ Pilot always shown the Traffic Situation Display (TSD)+ TCAS delivers a two stage advisory and vertical avoidance maneuver
Traffic Advisory (TA) - ‘Traffic Traffic’ Resolution Advisory (RA) - ‘Climb Climb’
+ Pilot is to follow an RA, even if it conflicts with ATC instructions, unless the pilot believes that safe flight would be jeopardized
Federal Aviation Administration (2000). Introduction to TCAS II Version 7. Washington, D.C.
4
Collision Avoidance in a Broader Context
Pilots don’t only interact with TCAS, they also receive information from ATC and the environment
Pilot’s ResponseAdvised Maneuvers for Collision Avoidance
Notification and Awareness
Awareness of other aircraft via TSD
TCAS Traffic Advisory
TCAS Resolution Advisory
ATC Traffic Callout
Visual acquisition of other aircraft
ATC Maneuver
Compliance to TCAS
Compliance to ATC
Personal Assessment and Maneuver
DecisionParty-line Information
5
Current TCAS Training Requirements: Ground Training
+ Classroom or computer based + >60 training requirements+ Measures performance through
quizzes and/or activities
To comply to the RA, you should
Pull the stick back
Push the stick forwardFederal Aviation Administration (2001). Advisory Circular No. 120-55B: Air Carrier Operational Approval and Use of TCAS II. Washington, D.C.
6
Current TCAS Training Requirements: Flight Training
7
Observed TCAS Use and Operation
o The TCAS Operational Performance Assessment (TOPA) monitored the terminal area of 8 major airports and examined pilot compliance to climb and descend advisories
41% to Climb RA’s
59% to Descend RA’s
TOPA observed compliance rates of
Misunderstanding/ Confusion
Aggressiveness
Non-Compliance
Olson, W. and J. Olszta (2010). TCAS Operational Performance Assessment in the U.S. National Airspace. IEEE/AIAA Digital Avionics Systems Conference.
Olszta, J., & Olson, W. (2011). Characterization and Analysis of Traffic Alert and Collision Avoidance Resolution Advisories Resulting for 500' and 1,000' Vertical Separation. Paper presented at the Ninth USA/Europe Air Traffic Management Research and Development Seminar (ATM 2011), Berlin, Germany.
8
Observed TCAS Use and Operation
“[My FO and I] incorrectly interpreted the red 'above' target on the VSI and responded improperly. We further reviewed the procedures, agreeing that a
person should fly 'away' from the red VSI indication, if instructed via RA.”ACN:785761, 2008
Pilot reports flying into the red region on the VSI
Non-Compliance
Aggressiveness
Misunderstanding/ Confusion
NASA. (2009). Aviation Safety Reporting System. Retrieved August 1, 2010: http://asrs.arc.nasa.gov/
“Descending into an airplane that is clearly descending? TCAS software clearly did not give appropriate guidance, nor did it self-correct when the
initial guidance was so clearly wrong” ACN: 854982, 2009
Pilot disagrees with advised descend RA
9
Observed TCAS Use and Operation
Non-Compliance
Misunderstanding/ Confusion
Aggressiveness
Far Eastern Air B757 Response to Descend RA (TCAS advised a descent rate of 1500 FPM)
Image copied from Lacagnina (2008). Easy Does It. Aero Safety World : http://flightsafety.org/asw/oct08/asw_oct08_p44-47.pdf?dl=1
10
Research Statement and Objectives
Develop a training program intended to improve pilots’ understanding of TCAS use for collision
avoidance in a range of traffic situations
(1) Train pilots to understand TCAS use for collision avoidance in the actual traffic and operational traffic environment
(2) Provide pilots with a well-rounded knowledge of different traffic situations that may result in TCAS advisories
(1) Train pilots to understand TCAS use for collision avoidance in the actual traffic and operational traffic environment
(2) Provide pilots with a well-rounded knowledge of different traffic situations that may result in TCAS advisories
11
Demonstration Based Training
Event Based Training
Approach to Training DesignComplement of Two Methods
12
Demonstration Based Training (DBT)
+ Computer-Based Training ~25 minutes Provides conceptual understanding of TCAS Outlines rules for compliance
+ 6 segments Introduction to TCAS Traffic Situation Display TCAS Advising Logic Traffic Advisories Resolution Advisories Example Timeline of RA Evolution
+ Mid-Training Quizzes
13
DBT: Demonstrations of RA’s
Event Based Training (EBT)
+ Presents traffic events that create the requirement to act+ Builds context and complexity into each scenario as the flight
progresses+ Uses a more realistic training environment
Fowlkes, J., Dwyer, D., Oser, R., & Salas, E. (1998). Event-Based Approach to Training (EBAT). The International Jounal of Aviation Psychology, 8(3), 209-221.
14
15
Air Traffic Transcripts
Charts & Checklists
Audio Communications (Aviation Intercom)
TCAS Alerts
Simulation Architecture
Eyetracker
PartyLine First Officer
Captain
Experimenter/Instructor
Coded Log of Flights
B747-400 Simulator (RFS)
TSDPFD
ND
PFDN
D
Touchscree
n
SideStick
ATC->TSDVGA
ATC Air Traffic Sim
ulator (TGF)
TCAS Logic
Simulator Study in Integrated Flightdeck – ATC Environment
Pritchett, A., Fleming, E., Cleveland, W., Zoetrum, J., Popescu, V., & Thakkar, D. (2012). Pilot Interaction with TCAS and Air Traffic Control. Paper presented at the 2nd International Conference on Application and Theory of Automation in Command and Control Systems (ATACCS), London.
16
Structure of Events: Descend RA Example
Training Objective Context of Event Performance Measures Feedback
Session
Accurate interpretation
of and response to TCAS
Descend RA.
Instrument Meteorological
Conditions (daytime, clouds, no winds)
ATC provides no traffic information
Conflict caused by IFR traffic enroute (most likely on departure)
RA maneuvering should not violate ATC
instructions
No conflicting ATC or party-line information
Pilot disengages autopilot and flight directors
If the pilot did not meet any
particular performance
measure, review the
correct response in
regards to that measure
Pilot responds to advisory with appropriate vertical speed
Pilot ensures vertical speed is not excessive
o Pilot notifies ATC of response to TCAS advisory as the maneuver is performed
Pilot reengages autopilot and flight directors
o Pilot notifies ATC of TCAS advisory and response after clear of conflict
Pilot returns to original clearance (if needed)
Evaluating the training program’s impact
18
+ Baseline Study+ Training Study+ 6 scenarios with 2 traffic events per scenario+ Traffic events defined by
RA type ATC information Traffic density
Overview of Study
Comparison to identify the impact of training
Overview of Study
Pre-Training Data Collection
(50 minutes)
Pre-Experiment Questionnaire
Pre-Experiment Quiz
TCAS Training Program
(50 minutes)
Introduction to TCASDemonstration Based TrainingShort Section QuizzesEvent Based Training
Evaluating TCAS Training Program
(120 minutes)
Flight Scenarios
Post Scenario Questionnaires
Debrief(15 minutes) Post- Experiment Questionnaire
19
20
Skill-Based Behavior
Rule-Based Behavior
Knowledge-Based Behavior
Skill-Based Behavior
Rule-Based Behavior
Knowledge-Based Behavior
Does the training program improve pilot performance in response to TCAS advisories and increase pilot understanding of TCAS?
21
Does the training program improve pilot performance in response to TCAS advisories and increase pilot understanding of TCAS?
Decrease time to achieve compliance
Decrease time to disconnect autopilot
Reduce aggressive response features
Increase percentage of RA duration in compliance
Increase appropriate response post-Clear of Conflict
Increase understanding of TCAS
Increase trust in TCAS
Skill-Based Behavior
Rule-Based Behavior
Knowledge-Based Behavior
22
Assessing the Pilot’s Response: Skill-Based BehaviorsVe
rtica
l Spe
ed
time5 seconds
RA Climb
2 ½ sec
TCAS weakens required
vertical rate
TCAS assumed
¼ g pull-up
Clear of Conflict
TCAS assumed constant vertical
RA rate
Autopilot Disconnect
Time
TCAS RA ManeuverPilot’s Response
Time Pilot First Achieves Compliance
Skill-Based Behavior
Rule-Based Behavior
Knowledge-Based Behavior
X
23
Mean Time to Comply: During Training
Skill-Based Behavior
pMM = Significance for the mixed modelps2 = Significance of the variance
pm = Significance of the means
Rule-Based Behavior
Knowledge-Based Behavior
24
+ During training, no significant differences+ Post training
Autopilot disconnect time decreased Pilot response was more consistent for one event
Autopilot Disconnect: Post Training
Skill-Based Behavior
Rule-Based Behavior
Knowledge-Based Behavior
25
Impact of Training Program
Decrease time to achieve compliance
Decrease time to disconnect autopilot
Reduce aggressive response features
Increase percentage of RA duration in compliance
Increase appropriate response post-Clear of Conflict
Increase understanding of TCAS
Increase trust in TCAS
Skill-Based Behavior
Rule-Based Behavior
Knowledge-Based Behavior
Skill-Based Behavior
Rule-Based Behavior
26
Assessing the Pilot’s Response: AggressivenessVe
rtica
l Spe
ed
time5 seconds
RA Climb
2 ½ sec
TCAS weakens required
vertical rate
TCAS assumed
¼ g pull-up
Clear of Conflict
TCAS assumed constant vertical
RA rate
Maximum vertical rate
Maximum vertical rate difference Vertical rate
difference
TCAS RA ManeuverPilot’s Response
Skill-Based Behavior
Rule-Based Behavior
Knowledge-Based Behavior
*Altitude Deviation
27
Assessing the Pilot’s Response: ComplianceVe
rtica
l Spe
ed
Not in Compliance In Compliance
TCAS RA ManeuverPilot’s Response
time5 seconds
RA Climb
2 ½ sec
TCAS weakens required
vertical rate
TCAS assumed
¼ g pull-up
Clear of Conflict
TCAS assumed constant vertical
RA rateSkill-Based Behavior
Rule-Based Behavior
Knowledge-Based Behavior
28
Assessing the Pilot’s Response: Return to Clearance
TCAS RA ManeuverPilot’s Response
TCAS assumed constant vertical
RA rate
TCAS weakens required vertical rate
Clear of Conflict
Alti
tude
time5 seconds
RA Climb
TCAS assumed
¼ g pull-up
Cleared altitude at time of RA
Pilot holds new altitudeContacts ATC and request new clearance
or ask for further instructions
Pilot begins descent back to originally cleared altitude
May inform ATC of response to RA
Skill-Based Behavior
Rule-Based Behavior
Knowledge-Based Behavior
29
Before Training
+ 89% of the pilots did not know that an RA should cause less than 500 feet of altitude deviation
+ 56% of the pilots responded that they would hold current altitude achieved after responding to an RA, as opposed to returning to their clearance
+ When asked about airline procedures for following TCAS, all 18 pilots noted the need for complying with an RA BUT 28% of the pilots commented compliance wasn’t necessary is there was a
TCAS “malfunction” or if the RA would cause an “unsafe situation”
“[Pilots] must always comply with a TCAS RA unless [aircraft] performance is hindered (i.e. operating single
engine) or [there is] an obvious TCAS malfunction (ie you can see traffic and it is not a threat)”
Skill-Based Behavior
Rule-Based Behavior
Knowledge-Based Behavior
30
+ During training, aggressive response features decreased + Post training, same decreasing trend observed
Aggressiveness
Altitude DeviationAverage Vertical Rate Difference
Maximum Vertical Rate DifferenceMaximum Vertical Rate Difference
All decreased
Skill-Based Behavior
Rule-Based Behavior
Knowledge-Based Behavior
31
+ During training, percentage compliance decreased in training event with “Climb RA” (93.1% compared to 99.6%)
+ Post training No significant differences in means observed But, trained pilots had a more consistent response
Percentage Compliance
Skill-Based Behavior
Rule-Based Behavior
Knowledge-Based Behavior
32
Return to Clearance
Skill-Based Behavior
Rule-Based Behavior
Knowledge-Based Behavior
Pilot returns to original clearance
NoYes
33
Impact of Training Program
Decrease time to achieve compliance
Decrease time to disconnect autopilot
Reduce aggressive response features
~ Increase percentage of RA duration in compliance
Increase appropriate response post-Clear of Conflict
Increase understanding of TCAS
Increase trust in TCAS
Skill-Based Behavior
Rule-Based Behavior
Knowledge-Based Behavior
Rule-Based Behavior
Knowledge-Based Behavior
34
Before Training
+ 94% pilots agreed with statement “I understand TCAS maneuvers when they are issued”, but… Only 50% of the pilots correctly identified the assumptions made by TCAS
advisory logic When asked to interpret TSD symbols, only 50% of the pilots got all parts of
the associated questions correct
Skill-Based Behavior
Rule-Based Behavior
Knowledge-Based Behavior
35
“My understanding of TCAS has increased”
+ 50% pilots reported an increase in understanding TCAS logic+ 27% pilots claimed to have learned about different types of TCAS RA’s
(notably, “Crossing RA’s”)
Post Training Understanding of TCAS
“Types of RA's were not previously taught. We were taught simply to comply”
Strongly Disagreed
0
Disagreed
0
Neutral
6%
Agreed
61%
Strongly Agreed
33%
Skill-Based Behavior
Rule-Based Behavior
Knowledge-Based Behavior
36
Post Training Trust in TCAS
“I am more likely to trust TCAS after completing today’s training”
“My trust in TCAS was already at a maximum so I wouldn’t be ‘more’ likely to trust it”
Strongly Disagreed
11%
Disagreed
0
Neutral
39%
Agreed
39%
Strongly Agreed
11%
Skill-Based Behavior
Rule-Based Behavior
Knowledge-Based Behavior
37
Post-Training
Decrease time to achieve compliance
Decrease time to disconnect autopilot
Reduce aggressive response features
~ Increase percentage of RA duration in compliance
Increase appropriate response post-Clear of Conflict
Increase understanding of TCAS
Increase trust in TCAS
38
Conclusions and Future Work
Conclusions+ Pilots may not need more training, but instead need better training
Integrating DBT and EBT methods permits ground-based and flight training material to be more cohesive
EBT structure allows for the design of purposeful training events+ Current FAA mandated TCAS training objectives may not fully reflect all
training areas Language needed to address the reduction of excessive responses to RA’s
Future Work+ What facilities and technologies would be required to implement this type of
training program?+ What implications arise when considering training design versus system
design? Would incorporating human factors considerations in initial design stages
decrease the amount of required training?
39
40
Conclusions and Future Work
Conclusions+ Pilots may not need more training, but instead need better training
Integrating DBT and EBT methods permits ground-based and flight training material to be more cohesive
EBT structure allows for the design of purposeful training events+ Current FAA mandated TCAS training objectives may not fully reflect all
training areas Language needed to address the reduction of excessive responses to RA’s
Future Work+ What facilities and technologies would be required to implement this type of
training program?+ What implications arise when considering training design versus system
design? Would incorporating human factors considerations in initial design stages
decrease the amount of required training?
41
Acknowledgements
+ Work sponsored by the FAA, Tom McCloy as Technical Monitor + 34 pilot participants+ Dr. Amy Pritchett+ Dr. Karen Feigh+ Dr. Ute Fischer+ Dr. Wesley Olson, MIT Lincoln Labs+ Wayne Gallo, FAA+ Roger Sultan, FAA+ Kylie Garey+ TCAS Team: William Cleveland, Vlad Popescu, Justin Mullins, Anil Bozan,
Henry Tran, Jack Ridderhof, Alyssa Whitlock, Colin Ludwig, Dhruv Thaakar, Jonathan Zoetrum, Jelle Wissink
+ CEC Lab Members
42