1

DOT/FAA/COE

Air Traffic Organization Washington, DC 20591

Enhanced AT-CPC Training

Date June 2018

Final Report

U.S. Department of Transportation Federal Aviation Administration

2

ACKNOWLEDGEMENTS

The authors would like to acknowledge the FAA personnel who took the time out of their busy schedules to complete the online surveys and participate in the face-to-face interviews. The team would also like to acknowledge Henry Fair for arranging the visits to the air traffic facilities and Miguel Montgomery, our technical advisor from the FAA for his support and guidance. The authors would also like to thank Industry Partners: Carol Sherman and Doug Siefken from TransLumen Technologies, Chris Malvey, Sean Connolly, and Alex Bowen from UFA. Special thanks to Embry-Riddle Aeronautical University graduate student Mattie Milner for her literature research, write-up, and review of the document. Sincere gratitude to university team members Stephen Rice a Human Factors expert from Embry-Riddle Aeronautical University and retired air traffic controllers Paul Drechsel and Terra Jorgenson from University of North Dakota, and William Coyne, Clyde Rinkinen, and Martin Lauth from Embry-Riddle Aeronautical University.

2

TABLE OF CONTENTS

Page

ACKNOWLEDGEMENTS .......................................................................................................................... 2

TABLE OF CONTENTS .............................................................................................................................. 2

LIST OF FIGURES ...................................................................................................................................... 5

LIST OF TABLES ........................................................................................................................................ 6

LIST OF ACRONYMS ................................................................................................................................ 7

EXECUTIVE SUMMARY ........................................................................................................................ 10

INTRODUCTION ...................................................................................................................................... 11

BACKGROUND ........................................................................................................................................ 12

Current Post-Certification Training and Recommendations ............................................................... 12

Refresher Training. ............................................................................................................................. 13

Skill Enhancement Training (SET). .................................................................................................... 13

Recurrent Training. ............................................................................................................................. 13

Supplemental Training ........................................................................................................................ 14

Skill Degradation .................................................................................................................................... 15

Overlearning ........................................................................................................................................... 17

Retention Interval Length ....................................................................................................................... 17

Task Characteristics ................................................................................................................................ 18

Motivation ............................................................................................................................................... 20

Testing Methods...................................................................................................................................... 21

Retrieval Conditions ............................................................................................................................... 22

Instructional Strategies/ Training Methods ............................................................................................. 23

Refresher Training .................................................................................................................................. 23

Practice .................................................................................................................................................... 25

Testing .................................................................................................................................................... 26

Symbolic Rehearsal ................................................................................................................................ 27

Skill Enhancement Training ....................................................................................................................... 27

Recurrent Training ...................................................................................................................................... 29

METHODOLOGY ..................................................................................................................................... 32

Research Design...................................................................................................................................... 32

Participants .............................................................................................................................................. 32

3

Materials and Procedures ........................................................................................................................ 32

RESULTS – QUANTITATIVE FINDINGS .............................................................................................. 34

Level 5 Facilities. ................................................................................................................................ 36

Level 6 Facilities. ................................................................................................................................ 37

Level 7 Facilities. ................................................................................................................................ 37

Level 8 Facilities. ................................................................................................................................ 38

Level 9 Facilities. ................................................................................................................................ 38

Level 10 Facilities ............................................................................................................................... 38

Level 11 Facilities. .............................................................................................................................. 39

Level 12 Facilities. .............................................................................................................................. 40

RESULTS - QUALITATIVE FINDINGS ................................................................................................. 48

Topics from mid to low level towers and TRACONS. ....................................................................... 50

Topics from high level towers and TRACONS. ................................................................................. 51

Topics from Enroute Centers. ............................................................................................................. 51

Items were identified by every type and level of facility .................................................................... 51

FAA Academy ............................................................................................................................................ 76

DISCUSSION AND PRACTICAL IMPLICATIONS ................................................................................ 79

Resources ................................................................................................................................................ 80

Web Based Training ............................................................................................................................... 81

Performance Assessment ........................................................................................................................ 81

Adaptive Learning .................................................................................................................................. 82

Train the Trainers .................................................................................................................................... 82

Bandwidth / Internet ............................................................................................................................... 83

Biasness in Training Methods ................................................................................................................. 83

Training Topics ........................................................................................................................................... 83

Situational Awareness / Scanning ........................................................................................................... 84

LIMITATIONS ........................................................................................................................................... 85

RECOMMEDATIONS ............................................................................................................................... 86

Bandwidth / Internet ............................................................................................................................... 86

FAA Academy ........................................................................................................................................ 86

Training Topics ....................................................................................................................................... 86

Adaptive Learning .................................................................................................................................. 87

Performance Assessment ........................................................................................................................ 88

Industry Updates for New Technology Insertions .................................................................................. 88

Accessing Training ................................................................................................................................. 88

4

Biasness in Training Methods ................................................................................................................. 89

NATPRO ................................................................................................................................................ 89

Tower Simulation / UFA ........................................................................................................................ 90

Scanning / TransLumen .......................................................................................................................... 93

STEGC .................................................................................................................................................... 93

ATC Training Commonalities Between Diverse and Remote Facilities ................................................ 94

A New Occupation - Requirements for UAS training within NAS for Commercial Businesses ........... 96

REFERENCES ........................................................................................................................................... 97

APPENDIX A ........................................................................................................................................... 103

Interview sites for SOAR Project 6 – Enhanced AT-CPC Training ......................................................... 103

APPENDIX B ........................................................................................................................................... 104

Overview of Results of the CPC Survey ................................................................................................... 104

5

LIST OF FIGURES Figure Page 1 - Presents a word cloud that reveals the most common words and phrases used by the 34 respondents. 2 - Frequency of word or phrase usage by interview participants for question 5. 59 3 - Frequency of word or phrase usage by interview participants for question 8. 69

6

LIST OF TABLES Table Page

1: All Data - Overview of the results of the CPC Survey 96

2: All Respondents - Presents the data as a function of all respondents 101

3: Facility Levels - Presents the data divided by facility levels in two categories 103

4: Years of Experience - Presents data in two categories by years of experience 105 5: Facility Types - Presents the data divided by facility types 107

7

LIST OF ACRONYMS AAM Office of Aerospace Medicine AAT Air Traffic AJI FAA Safety and Technical Training AMA FAA Academy ARTCC Air Route Traffic Control Centers ASO Southern Region, FAA ASOS Automated Surface Observing Systems AT-CPC Air Traffic Certified Professional Controller ATC Air Traffic Control ATCS Air Traffic Control Specialist ATCT Air Traffic Control Tower ATO Air Traffic Organization ATSAP Air Traffic Safety Action Program ATTE Air Traffic Teamwork Enhancement AWSS Automated Weather Sensor System CAMI Civil Aerospace Medical Institute CBI Computer Based Instruction CEDAR Comprehensive Electronic Data Analysis and

Reporting CIC Controller-in-Charge COA Certificate of Waiver or Authorization COE Center of Excellence CPC Certified Professional Controller CRM Crew Resource Management DOT Department of Transportation DOD Department of Defense DYSIM Dynamic Simulator EDST En Route Decision Support Tool eLMS Electronic Learning Management System ERAM En Route Automation Modernization FAA Federal Aviation Administration FAR Federal Aviation Regulations FAA HQ Federal Aviation Administration Headquarters FLM First Line Manager FUSION Fused Display Mode GPS Global Positioning System HCI Human/computer interaction IFR Instrument Flight Rules ILS Instrument Landing System INET Internet Location of ATC Training Courseware IT Initial Training KSA Knowledge, Skills and Attitudes LADP Local Airport De-icing Plan LAHSO Land-and-Hold Short

8

LAWRS Limited Aviation Weather Reporting Station LCMS Learning Content Management System LOA Letter of Agreement LUAW Line Up and Wait METAR Aviation Routine Weather Report MOA Military Operating Area MSAW Minimum Safe Altitude Warning NAS National Airspace System NASA National Aeronautics and Space Administration NATCA National Air Traffic Controllers Association NATPRO National Air Traffic Professionalism NLT Not Later Than NVivo Qualitative Data Analysis Computer Software OJT On-the-Job Training OJTI On-the-Job Training Instructor OKC Oklahoma City PBN Performance-based Navigation PI Proactive Interference PIREP Pilot Report QA Quality Assurance RECAT Recategorization RI Retroactive Interference RNAV Area Navigation RNP Required Navigation Performance SAA Special Activity Airspace SET Skill Enhancement Training SGET Scenario Generation Tool SIDS Standard Instrument Departure Routes SME Subject Matter Expert STAR Standard Terminal Arrival Routes STARS Standard Terminal Automation Replacement

System STARS Elite Standard Terminal Automation Replacement

System - Enhanced Local Integrated Tower Equipment

STEGC Subthreshold Extreme Gradual Changes SUA Special Use Airspace SVFR Special Visual Flight Rules TA Training Administrator TALPA Takeoff and Landing Performance Assessment TMU Traffic Management Unit TRACON Terminal Radar Approach Control TSS Tower Simulator System TTL Transistor - Transistor Logic UAS Unmanned Aircraft System UAV Unmanned Aircraft Vehicle

9

USSS United States Secret Service UTM Unmanned Aircraft System Traffic Management VFR Visual Flight Rules VR Virtual Reality VRR Voice Recognition and Response WCG Washington Consultant Group WIFI Wireless Fidelity

10

EXECUTIVE SUMMARY Comprehensive proficiency training for fully certified controllers is a crucial element of

ensuring the Federal Aviation Administration’s (FAA) mission to provide the world’s safest and

most efficient airspace system. This study examines the FAA’s current proficiency training

program, researching recurrent and skill enhancement training to Certified Professional

Controllers (CPCs).

From controller workforce and the management team participation, using online surveys and

face to face interviews, revealed deficiencies in the FAA’s proficiency training program.

Controller experience levels highlighted differences in learning styles. The effectiveness of

knowledge, skills, and enhanced training was facility dependent, indicating the facility training

should be standardized and have a centralized approach to maintain consistencies in content and

delivery. Facility resources, especially at lower level radar and towers, limit the required

training that can be developed and performed as per FAA Order 3120.4.

Research has demonstrated that training quality, standardization, efficiency, and availability can

be improved using advanced training technology. “Enhanced Training” should incorporate

advanced concepts and competencies spanning an entire spectrum of difficulty, delivered

through classroom instruction, advanced simulation, gamification, web-based instruction,

multiple media formats, and adaptive learning technology. These methods will augment

teaching the teachers and adapting learning to millennials and future controller demographics.

11

INTRODUCTION

The FAA currently provides recurrent and skill enhancement training to CPCs almost

entirely through facility based instruction. This decentralized approach creates systemic

inconsistencies in training content and delivery, adds additional burden on facilities as their focus

is on operations, and inhibits providing training on advanced NextGen concepts and the use of

simulation. In addition, the current controller training practices and methods have struggled to

keep pace with advancements in training technology and design. As a result, training for CPCs is

often ineffective, inefficient, subjective, and inconsistent. The current training system also lacks

the flexibility or adaptability necessary to meet the need for training new technologies and

emerging ATC issues.

Advanced training technologies have the ability to support more dynamic and effective

training materials, and methods. Such technologies can promote accelerated learning and skill

enhancement while tailoring learning to the individual’s strengths and weaknesses. Training can

also be developed specific to the air traffic facility where the particular skills will be applied.

The FAA’s air traffic control (ATC) training program must address the proficiency

training needs of CPCs, with updated technology and methods. The purpose of this research is to

identify changes needed to improve CPC post-certification training.

12

BACKGROUND Current Post-Certification Training and Recommendations

In the past five years the FAA has hired over 4,000 new air traffic controllers and plans

to hire over 6,000 new controllers within the next five years to keep up with expected attrition

and air traffic growth (Schultheis, Winokur, & Hutson, 2014). The FAA’s goal is to provide

sufficient and effective training for air traffic controllers so that the national aerospace system

will remain safe and efficient (Schultheis et al., 2014). Air Traffic Control (ATC) training

programs should address the learning needs of newly hired controllers (typically students),

ongoing refresher and skill enhancement training for Certified Professional Controllers (CPC),

and expose students and CPCs to any new technologies or procedures through training

(Schultheis et al., 2014). Part of successful ATC operations depends on controllers utilizing

individual skills and combining their skills to provide safe and efficient guidance and control

in the aerospace system. A majority of current ATC training methods do not identify specific

skills that air traffic controllers should develop and master to be successful controllers. The

training curriculum typically consists of students learning the necessary information (gaining

knowledge) in a classroom and then they move on to skill acquisition gained through

simulation and on-the-job training (Schultheis et al., 2014). However, acquiring and reaching a

level of proficiency in the necessary ATC skills while also controlling air traffic in a simulated

environment or on-the-job training can be difficult and inefficient (Schultheis et al., 2014).

Air traffic controllers receive initial training and then go through additional

proficiency training to maintain and upgrade their knowledge and the skills necessary to apply

ATC procedures in a safe and efficient manner (Saunders, 2005). This training includes

Refresher, Skill Enhancement, Recurrent, and Supplemental; however, training needs differ

13

according to different facilities and programs should be tailored to meet individual

requirements. The following is a brief description from the Federal Aviation Administration

(FAA) regarding each of the different types of training.

Refresher Training. Each facility must maintain, in writing, an annual (calendar year)

refresher training plan. Annual refresher training contains two elements: nationally and/or

facility-developed curriculum and simulation training. Facilities are encouraged to review

their quality control data (e.g., Quality Control Monitoring, Service Reviews, and

Compliance Verification and data available in the Partnership for Safety Portal) to identify

additional topics for annual refresher training in order to meet each facility’s changing needs

(Saunders, 2005). In addition, there are fifteen topics which must be addressed during refresher

training unless they are deemed unnecessary by the TA (Saunders, 2005).

Skill Enhancement Training (SET). The purpose of SET is to improve an individual’s

knowledge, skills, and abilities. SET may be assigned to individuals certified on a position(s).

SET may not be used to correct deficient performance that resulted in decertification. For an

individual not certified but training on a position(s), SET may be assigned to enhance specific

skills, techniques, or knowledge, to assist in their success with the training program

(Saunders, 2005).

Recurrent Training. Recurrent training is collaboratively-developed national safety

training delivered via electronic means, instructor-led presentations, or any combination

thereof. Recurrent training is intended to increase air traffic controller proficiency, enhance

awareness of human factors affecting aviation, and promote behaviors essential for the

identification, mitigation and/or management of risk. Topics are derived from data collected

through internal and external safety reporting systems and stakeholder input. Recurrent

14

training is conducted via an 8-hour block of training, two rounds delivered yearly. Each round

is comprised of approximately four hours of training selected from the topics listed below,

and four hours of training on relevant and timely safety topics, such as but not limited to:

Human Factors, Safety Culture, Threat and Error Management, Crew Resource Management,

Event Recovery, and learning that promotes the maturity of the Safety Management System.

Recurrent training requirements are identified annually NLT October 1st to be delivered the

following calendar year. Recurrent training on the following items need not be duplicated in

local refresher training (Saunders, 2005):

(1) Safety alerts and traffic advisories, to include Minimum Safe Altitude Warning

(MSAW) procedures and the relationship between charted minimum altitudes and

underlying 09/30/13 JO 3120.4N 4-4 topography.

(2) Weather and other conditions that affect flight (e.g., icing, thunderstorms, wind

shear, and VFR aircraft that encounter instrument flight rules (IFR) conditions).

(3) Bird activity information and dissemination.

(4) Wake turbulence information and application.

(5) Line up and wait (LUAW).

(6) Runway Safety.

(7) Recovery in ATC Operations.

(8) Fatigue awareness

Supplemental Training. All operational personnel must complete supplemental training prior to

the use of new/revised procedures, regulations, or equipment. The TA must review all

supplemental training and update local training materials as appropriate (Saunders, 2005).

15

Skill Degradation

Skill decay refers to the loss or degradation of training or acquired skills (or

knowledge) after periods of nonuse (Kluge & Frank, 2014; Winfred, Bennett, Stanush, &

McNelly, 1998). Skill decay is most common and problematic in situations where individuals

receive initial training on knowledge and skills but then they are not required to use or access

their knowledge and skills on a regular basis (U.S. Department of Transportation, 2016). Skill

degradation commonly happens whenever the task (or knowledge) is replaced by automation

and the human user no longer needs to perform that particular skill anymore, unless there is

an automation failure and an emergency occurs (Kluge & Frank, 2014). As we consider the

different types of proficiency training that CPCs experience (recurrent, refresher, skill

enhancement, SET), it is important to keep in mind any tasks that may have been overtaken

by automation. Air traffic controllers handle situations that can have severe consequences if

anything goes wrong so while we want to reduce workload as much as possible (i.e. through

automation), we also do not want controllers to experience significant skill degradation that

could affect their reaction times in an emergency situation (Casner, Geven, Recker, &

Schooler, 2014; Ebbatson, Harris, Huddlestone, & Sears, 2010; Wickens, Mavor,

Parasuraman, & McGee, 1998). Therefore, it is crucial that the FAA understands the effects

of skill degradation and how these effects can be minimized while also improving air traffic

controller’s proficiency training and making it as efficient as possible.

Several studies have addressed the different factors that affect the decay or retention of

trained/acquired skills over extended periods of nonuse (Annett, 1979; Wickens et al., 1997;

Winfred et al., 1998). These factors include (a) the degree of overlearning (b) the length of the

retention interval, (c) certain task characteristics (closed-loop vs. open-looped tasks; physical

16

vs. cognitive tasks), (d) motivation, (e) testing methods for original learning and retention, (f)

conditions of retrieval, and (g) instructional strategies or training methods (Ryan & Deci,

2000; Sauer, Hockey, & Wastell, 2000; Winfred et al., 1998). It is also important to note that

there is a difference between methodological factors and task-related factors. Methodological

factors are those that can be manipulated during the training process to reduce skill loss. This

include elements such as the degree of overlearning, conditions of retrieval, evaluation

criteria, and the method of testing (Winfred et al., 1998). On the other hand, task-related

factors are integral characteristics of the task and are not easily changed during the training or

learning session.

Task-related factors can include characteristics such as the distinction between closed-loop

and open-looped tasks, physical and cognitive tasks (Winfred et al., 1998).

Cognitive science theories recommend that a Knowledge, Skills, and Attitudes (KSA)

KSA) analysis be performed so that the task may be broken down into its

underlying components and the workers know exactly what is required to successfully

and efficiently complete the job. (Cahillane, MacLean, & Smy, 2015, p. 31).

Depending on the type of skill or knowledge acquired, it may be more or less

susceptible to skill degradation; however, in general, the longer the period of nonuse for the

skill, the greater the decay (Annett, 1979; Kluge & Frank, 2014; Wickens, Mavor, & McGee,

1997; Winfred et al., 1998). The factors listed above are what usually affect skill decay, it’s

important to note that skill decay is usually related to infrequent opportunities to practice or

perform acquired skills or the user receiving absent or inadequate feedback (Kluge & Frank,

2014; Winfred et al., 1998). One of the most important aspects of both skill and knowledge

retention is the amount or degree of overlearning.

17

Overlearning

Overlearning provides additional training beyond what is required for initial

proficiency and as a result, a higher degree of learning or understanding is usually achieved

(Winfred et al., 1998). Overlearning tends to strengthen the bond between stimulus and

response, which decreases the likelihood that the response will be forgotten. Additionally,

increased repetition and practice may provide further feedback to the user, which help the user

understand when they are making correct or incorrect responses and allow them to make

improvements on their performance (Winfred et al., 1998). Overlearning can also help the

user gain more self- confidence and may decrease negative factors such as stress and anxiety,

which could hamper their performance (Winfred et al., 1998). Overlearning has been found to

be negatively correlated with the amount of skill decay on job performance, such that higher

degrees of overlearning result in less skill decay over extended periods of nonuse (Winfred et

al., 1998).

Retention Interval Length

Retention can be defined as the “evidence that the process of memory retrieval has been

effective and the [knowledge and/or skills] from the previous experience has been [reproduced

or recalled]” (Spear, 2014, p. 8). Initial training and refresher training courses are meant to help

the user increase the retention strength of a certain knowledge or skill. However, research

supports the notion that the length of the nonpractice interval is positively associated with the

level of skill decay meaning that the longer the user goes without practicing the skill, the more

likely they are to experience significant degradations in skill (Sauer et al., 2000; Winfred et al.,

1998).

18

Task Characteristics

An important set of factors that can influence a person’s retention and degradation of

skills and knowledge is the characteristics of the task being learned. These tasks may differ in

terms of difficulty, complexity, and level of integration; therefore, researchers tend to classify

them by broad categories, such as closed-loop/open-loop tasks, physical/cognitive tasks,

natural/artificial tasks, and speed/accuracy (Farr, 1987; Winfred et al., 1998).

Closed-loop tasks usually involve discrete responses or other fixed-sequence tasks,

which have a definite beginning and end (i.e. preflight checklist) (Winfred et al., 1998). On

the other hand, open-loop tasks typically involve continuous responses that are repeated and

do not have a definitive beginning or end (i.e. tracking or problem solving) (Winfred et al.,

1998). Research has shown that open-looped tasks have a higher likelihood of being better

retained by an individual, even if the period of non-use is several years long (Childs & Spears,

1986; Farr, 1987; Winfred et al., 1998). It is hypothesized that open-looped tasks are easier to

retain because of their continuous nature, which may allow for repeated practice

(overlearning), and therefore more resistant to decay. In addition, the measurement of a

person’s proficiency level for these two different tasks is probably measured differently. For

example, when a person is performing a closed-loop task, it may be more obvious when there

are slight deviations from the prescribed instructions or directions. However, when a person is

performing an open-loop task, the person may be able to deviate a little more because there are

not precise steps on how exactly the task needs to be accomplished (Winfred et al., 1998).

In addition, the type of effort (physical or cognitive) may moderate the effect of

overlearning and insulate against skill degradation. Physical tasks typically require muscular

strength, exertion of forces, endurance, and coordination; whereas cognitive tasks typically

19

involve perceptual input, mental operations, problem solving, and decision making

(Winfred et al., 1998). Research supports the theory that cognitive skills are more easily

retained (and less susceptible to skill degradation) because they more readily lend

themselves to mental practice (i.e. cognitive repetition of a task without actually carrying

out the task or the accompanying physical movements) (Farr, 1987; Winfred et al., 1998).

Thus, mental rehearsal appears to be an effective strategy at combating skill degradation

and enhancing retention interval. However, it is important to note that this research is based

off the premise that the user is engaging in mental practice. If the user is not mentally

rehearsing the skill then it can be reasonably assumed that the skills required for a physical

task will suffer from less skill degradation than cognitive skills (Winfred et al., 1998).

Therefore, it is important that if the user is engaged in cognitive tasks, that s/he has

appropriate mental rehearsal periods to maintain skill proficiency levels.

Furthermore, the qualities of the task itself, whether it is natural or artificial, will help

determine how well the task will resist skill degradation. Natural tasks are generally more

complex; therefore, they require more effort to learn and process the task (for example, typing,

instrument flying, piano playing, etc.); which has a significant positive influence on how well

they are learned and subsequently, retained. In other words, the more cohesive or ingrained a

task is, the easier it is for the user to take advantage of and less likely to suffer from skill

degradation (Winfred et al., 1998). In addition, when people learn natural tasks they typically

have a genuine interest in acquiring and retaining proficiency; whereas this type of intrinsic

motivation can be difficult to cultivate with artificial tasks (Annett, 1979; Winfred et al.,

1998). A person’s motivation has actually been found to also influence skill retention because

a motivated person is more likely to practice more often, and additional practice can be

20

expected to positively influence skill retention. It has been hypothesized that people engaging

in natural tasks will experience more motivation and therefore the effect of overlearning on

skill retention ought to be greater for natural tasks (Winfred et al., 1998).

Motivation

A person working toward an end goal is typically motivated whether through an

intrinsic (internal) force or extrinsic (external) force. When a person is intrinsically motivated,

they usually do something because they find the task inherently enjoyable or interesting;

whereas if a person is extrinsically motivated, the person usually engages in the task because

they know they will receive some type of reward for completing the task (i.e. money,

promotion, etc.) (Gagne & Deci, 2005; Lee, Cheung, & Chen, 2005; Ryan & Deci, 2000). For

example, a student could be motivated to complete their homework because they are curious

or because they want a good grade in the class.

Interpersonal events and structures that are conducive to feelings of competence during

an action can enhance intrinsic motivation for that action because they help the person feel

satisfied in their own capabilities. When encouraging feelings of intrinsic motivation, the

person should experience optimal challenges, effective feedback, and freedom from

demeaning evaluations are all predicted to facilitate intrinsic motivation (Ryan & Deci, 2000).

Furthermore, feelings of competence will not enhance intrinsic motivation unless they are

accompanied by a sense of autonomy. Thus, the person needs to experience high self-efficacy

and they also need to believe their behavior is self-determined if intrinsic motivation is to be

maintained or enhanced (Ryan & Deci, 2000). A majority of research regarding intrinsic

motivation supports the idea that extrinsic factors (such as tangible rewards, threats, etc.) can

diminish a person’s intrinsic motivation because people may view the extrinsic factors as

21

controlling their behavior.

Unfortunately, many workers do not find intrinsic motivation within their job and

require extrinsic motivation if they are expected to perform at an acceptable standard. When

people are externally motivated they typically act with the intention of obtaining a desired

reward or avoiding an undesired consequence so they may only work or complete the task

when the action is instrumental to accomplishing that goal (Gagne & Deci, 2005). It is

important to note that if a person is extrinsically motivated, they should identify the value of

their behavior for their own self-selected goals. If people make this type of identification, they

typically feel greater freedom and volition because their behavior is more congruent with their

personal goals and identities, even if they are motivated through an external factor (Gagne &

Deci, 2005).

Testing Methods

When determining a person’s level of proficiency on a specific task or knowledge, the

typical method is to deliver the person a recognition or recall test after some period of nonuse

to ensure they have retained the appropriate material. However, a recognition test and a recall

are dissimilar and typically a person’s ability to recognize information or an event is unrelated

to their ability to recall the same applicable information (Winfred et al., 1998). Therefore,

depending on how the person learned the material or skill, they may display different degrees

of retention. Although, as a whole, recall tests usually yield lower scores than recognition tests

because a recognition typically prompts the user with a word or a hint (think of an elementary

school vocabulary test); whereas, a recall test provides the user with no hint so they may have

to work a little harder. If a supervisor is determining his worker’s proficiency level on a

specific task or knowledge in a certain area, in might be more beneficial to have the workers

22

perform a recall test because this will show if the worker truly remembers the applicable

information and can make the appropriate connections to retrieve the critical information.

Retrieval Conditions

The quality of a person’s skill retention is highly dependent on two related factors,

how information was encoded and the types of cues present at retrieval. If a person is

attempting to retrieve the knowledge/skills necessary to complete a task, his/her accuracy may

be affected by “previously acquired memories (proactive interference “PI” and/ or subsequent

learning (retroactive interference “RI”)” (Jonides & Nee, 2006, p. 44; Spear, 2014; Szpunar,

McDermott, & Roediger, 2008). People typically have a more difficult time retrieving the

appropriate information if they have interfering memories which are too similar to or

contradict the critical memory. Furthermore, people may have a more difficult time retrieving

the appropriate memory if the physical environment they are currently in is different from the

one in which they acquired the memory (retention environment) (Spear, 2014; Winfred et al.,

1998). For example, students are often encouraged to study for a test in the room/area where

the test will take place because it will be easier for the brain to recall the information if it is in

the same environment.

The “encoding specificity principle states that information retrieval or retention will be

maximized if the conditions at retentions assessment match as closely as possible to those

present during the original learning” (Winfred et al., 1998, p. 64). A person may fail in

memory retrieval during testing (or an emergency at work) because the stimuli that were

present during training are now absent. On the other hand, the worker may notice additional

stimuli which are not present during training which cause confusion and hinder retrieval

(Spear, 2014). Due to retroactive interference or proactive interference, a conflicting memory

23

may interfere with the person’s ability to appropriately retrieve the critical memory

(particularly if they are in an emergency situation).

Instructional Strategies/ Training Methods

When considering the effect skill degradation can have on a person’s acquired

knowledge and skills, it is important to understand the instructional process in which the

information was first learned because different types may influence a person’s skill retention.

There are several different training methods that an instructor may use, but the most frequently

used training methods include on-the-job training, lecture method, and programmed

instruction (Winfred et al., 1998). Programmed instruction is an instructional technique that

systematically presents information to the learner (typically through a computer or some type

of visual media aid). This particular type of instruction has shown to support higher levels of

knowledge retention in individuals compared to conventional instruction methods (Farr, 1987;

Winfred et al., 1998).

Refresher Training

The main goal of refresher training is to re-establish a specific skill that was acquired at

the end of an initial training (IT), which may need to be relearned after a period of not using

the skill (Kluge & Frank, 2014). In particular, refresher training can help air traffic controllers

practice and maintain the required skills and knowledge that is necessary to successfully

navigate the wide range of challenges they may experience in abnormal or emergency

situations (Mahony, Griffiths, Larsen, & Powell, 2008). An emergency in the national

aerospace system can result in severe consequences; therefore, several precautions are taken to

ensure emergencies and abnormal situations are minimized as much as possible. While this is

24

certainly good, it also means that air traffic controllers do not have ample opportunity to

practice the skills that are necessary to handle these types of situations. Therefore, it is vital

that the refresher training air traffic controllers receive covers all necessary skills and helps

them maintain appropriate levels of proficiency that is required in an emergency or abnormal

situation.

Researchers interested in examining how air traffic controllers reacted during an

emergency situation conducted an observational field study. Researchers used “a prototype set

of individual and team-level cognitive strategies...to investigate dyadic teams of operational

controllers during real and simulated emergencies and abnormal situations in a major European

Area Control Centre” (Malakis & Kontogiannis, 2012, p. 1). The results indicated that air

traffic controllers experienced a significant reduction of performance during the management

of certain “abnormal situations [where] the escalation pattern called for rapid interventions

from the Executive Controller position” (Malakis & Kontogiannis, 2012, p. 1). Fortunately,

situations that required less escalation patterns were handled successfully by the controllers.

After reviewing the type of refresher training that the controllers received, it was

discovered that their training curriculum was technically oriented and focused on situations

where they already displayed high performance scores and the speed of their reaction time was

not a factor (Mahoney et al., 2008). After experiencing the simulated emergency, it is clear

that there was a mismatch between formal training requirements and unattained operational

demands, which could lead to severe safety concerns in the future. It is crucial that air traffic

controllers’ refresher training has an overarching goal of maintaining safety while also

focusing on cultivating coordination between controllers, resilience, and building skills that

can be transferred to different emergency situations (Mahoney et al., 2008).

25

When responding to an emergency, air traffic controllers should demonstrate problem-

detection skills while filling in gaps in information, clarifying assumptions, and evaluating

potential solutions (Mahoney et al., 2008). During an emergency, controllers must synchronize

activities that are typically independent because they are re-allocating duties and

responsibilities. Synchronizing these activities requires successful coordination and appropriate

information exchange between different workers. As new tasks are added or changed, team

workload must be managed, especially in critical situations when errors are unacceptable

(Mahoney et al., 2008).

Refresher training can help air traffic controllers experience these simulated scenarios

where they have the opportunity to build or maintain the required personal skills and team

skills necessary to successfully handle abnormal or emergency situations.

(a) Previous research has indicated that there are three different types of basic

refresher interventions that are assumed to be the most effective (Farr, 1987;

Kluge & Frank, 2014; Wickens et al., 1997) and were selected based on their

effects on skill retention. These three refresher interventions include: “Practice,

based on theoretical assumptions regarding skill proceduralization,

(b) Testing, based on the theoretical assumptions of the testing-effect, and

(c) Symbolic rehearsal, based on the theoretical assumption of mental practice”

(Kluge & Frank, 2014, pp. 176-177)

Practice

Typically, during initial training, the user works on a task repeatedly until the required

proficiency level is reached; whereas refresher interventions can be viewed as a form of

“distributed practice in which the practice is interrupted by longer periods of non-use” (Kluge

26

& Frank, 2014, p. 176). In this context, practice supports the transfer of declarative knowledge

(i.e. facts and figures) into procedural knowledge (i.e. physical and/or cognitive skills). After a

long period of practice, the user begins generalizing specific knowledge sets into mental

schemas (a representation of a plan in the form of a mental model) that are made up

declarative knowledge and rules (Kluge & Frank, 2014). In most cases, skill retention can be

improved by increasing the amount of training through task repetition; and repetitions are

most effective when applied both before and after task proficiency has been achieved (Kluge

& Frank, 2014).

Testing

Previous research has demonstrated that the testing a user’s knowledge and skill is a

legitimate approach for helping the user maintain the appropriate level required (Szpunar et

al., 2008; Wickens et al., 1997). When the user is tested on their knowledge and/or skills

(after an initial training phase), they may experience a “testing-effect,” (Roediger & Karpicke,

2006; Saunders, 2005; Szpunar et al., 2008) which supports knowledge retention more

strongly than additional rehearsal or practice of the learning material (Kluge & Frank, 2014;

Roediger & Karpicke, 2006). Furthermore, testing an individual’s knowledge or skills helps

them learn how to adapt the skill to different situations, which would definitely be a useful

skill during an emergency or abnormal situation. The testing-effect is explained by two

different reasoning’s: (1) the intense retrieval effort (i.e. time and mental effort) that the

learner must go through when in the testing situation to retrieve information from long-term

memory, and (2) this process of retrieving knowledge or a skill is similar to a refresher

intervention and a retention assessment (Kluge & Frank, 2014).

27

Symbolic Rehearsal

Symbolic rehearsal happens when a person visualizes, describes, or draws the steps

necessary to perform a task, without actually physically performing the task (Kluge & Frank,

2014). Interestingly, the more mental steps that the task requires, the more effective that the

symbolic rehearsal is, particularly for novice learners who may not be as familiar with all of the

steps (Farr, 1987). This type of refresher intervention is typically most effective when practicing

cognitive tasks that require a sequence of actions (Ginns, 2005; Kluge & Frank, 2014). For

example, an air traffic controller may not have a physically demanding job but they have

several checklists that they exhaust when guiding pilots and symbolic rehearsal of the

appropriate timing of these checklists may be beneficial. While symbolic rehearsal can hinder

the speed of skill decay, it is not as effective as actual practice (Kluge & Frank, 2014; Kluge,

Burkolter, & Frank, 2012); therefore, this type of refresher intervention should be used in

addition to the previous two interventions and not as a replacement.

Skill Enhancement Training

Skill learning involves any experience that allows a person to improve their

performance on a particular activity; however, it’s important to note that this concept is

separate from knowledge acquisition (Hendrickson, Goldsmith, & Johnson, 2006). For skill

learning, performance requires attaining procedures, whereas for knowledge acquisition,

performance requires attaining facts. Fortunately, these two activities are similar in that they

usually both involve procedural information (skill) and declarative information (facts)

(Hendrickson et al., 2006).

If an individual wants to enhance their skills, there are three general aspects of the

28

enhancement process that should be strengthened: (a) acquisition, (b) retention, and (c)

transfer. To begin with, acquiring the skill (or experiencing training) should be done as

quickly and efficiently as possible to reduce cost and save time and effort. However, it is

important that the training methods, and results, are not compromised as a result from being

taught or completed too quickly. Next, the skills should be made as durable as possible so that

following training the user will still recall the appropriate actions/knowledge even if they

have not used that particular skill for an extended period of time. Depending on the type of

skill, skill degradation can have a range of different consequences and it’s critical for the user

to understand how degradation may affect their particular skill and how to combat it. Third,

training should be conducted in such a way that the skill is learned in a flexible environment.

The user should be able to apply the skill in different situations and not have its use restricted

to the particular situation they encountered during training (Hendrickson et al., 2006).

Within these three guidelines, there are several principles of training that describe in

greater detail how to engage in skill enhancement learning, whether that involves a person’s

performance or skill acquisition. One of the first principles is ‘procedural reinstatement,’

which involves the person engaging in deliberate practice of the task and not waiting until an

emergency or random event occurs before they practice their response. Typically, skill

learning occurs gradually and individuals improve in their response times, the more that they

practice; this is also known as the power law of practice (Hendrickson et al., 2006). This law

basically supports a common school of thought that practice helps most at the beginning of

learning a task or acquiring knowledge, but even after many trials, practice continues to help

individuals perform faster. On the other hand, there is a complementary power law of

forgetting, which accounts for a gradual decline in performance when individuals do not

29

practice their skills for an extended period of time (also known as skill degradation) (Annett,

1979; Hendrickson et al., 2006; Wickens et al., 1997; Winfred et al., 1998). It’s important to

note that deliberate practice requires a high degree of focus and motivation, but this type of

practice can lead to individuals achieving expert levels of performance.

While acquired knowledge may be forgotten very rapidly (especially if it is not

consistently recited by the individual), learned skills are typically easier retained than

across periods of nonuse. For example, if a person is told a seven-digit phone number,

they may not remember that number by the time they go to actually type it into their

phone. However, if a person learns how to ride a bike when they are young, they can still

recall the skill several years later (even if they may be a bit unbalanced, they don’t have to

relearn the entire process of how to ride a bike). However, acquired knowledge is

typically much more flexible and applicable to different situations that a learned skill

(Hendrickson et al., 2006). Continuing with the previous example, the knowledge of

acquiring telephone numbers can be used any time the person needs to learn a sequence of

new informational pieces. However, when riding a bicycle, the individual may be

restricted to the type of bike they originally learned the skill on when they were younger.

Together, these statements comprise the procedural reinstatement principle, which states

that declarative information (acquired knowledge) shows poor retention but robust transfer

and procedural information (learned skill) shows strong retention but limited transfer

(Hendrickson et al., 2006).

Recurrent Training

In work environments that involve critical jobs with human-machine interactions, it is

30

important to consider the most effective training method that will provide operators with the

required skills and knowledge to effectively and efficiently carry out their tasks (Sauer et al.,

2000). Effective training should encourage rapid skill acquisition and support high skill

retention during extended periods of non-use (Sauer et al., 2000). There has also been concern

that as jobs include more automation, traditional training approaches may not be as effective

and operators “need to form a mental model of the overall functional structure of the system to

understand its contingencies and interactions” rather than only learning one part of the job or

task (Sauer et al., 2000; Starter, Woods, & Billings, 1997).

In 1992, the FAA developed its first national human factors course for controllers, Air

Traffic Teamwork Enhancement (ATTE). This course focuses on Crew Resource Management

(CRM) human factors training for air traffic controllers (Grieser, 2001). There are two basic

tenets that underlie this training, (1) air traffic controllers operate both as individuals and as

members of a team; therefore, performance in both areas should be addressed and (2) Error

Management must be a cornerstone of every course because it is so common and necessary

throughout airline CRM. The general outline of the course is provided; however, it is meant to

be facilitated and interactive with participants (Grieser, 2001). There are three different

modules, which include “improving individual performance,” “improving teamwork,” and

“managing errors” (Grieser, 2001; Starter et al., 1997).

The FAA’s CRM Advisory Circular recommends three stages for effective CRM

training:

awareness training communicates basic human factors concepts that are related to

individual and team performance; practice and feedback develops the human factors

attitudes, behaviors and skills identified in the first stage; and continual

31

reinforcement provides repeated exposure to the concepts, as well as on-the-job

feedback and reinforcement from multiple sources. (Grieser, 2001, p. 2)

Previous research has shown that air traffic controllers from different facilities which

held weekly team meetings typically had a better understanding of the shared strengths

and weaknesses throughout the group and they were less likely to seek out, and receive,

outside help. “(Grieser, 2001; Smith-Jentsch, Kraiger, Cannon-Bowers, & Salas, 1998).

32

METHODOLOGY

This study was an assessment of the FAA’s ongoing refresher and skill enhancement

training for CPCs. The assessment was primarily exploratory in nature, to gain a broad

perspective of the skill enhancement training CPC’s receive.

Research Design

The writers used mixed methods procedures. This strategy was used to gain a broader

perspective of skill enhancement training. The quantitative data was taken from CPC survey.

The qualitative data was collected from interviews with non-bargaining employees at field

facilities.

Participants

Sixty-nine CPC controllers volunteered for the survey portion of the data collection.

Sixty-three front line managers, training managers, and air traffic managers, volunteered for the

interview portion of the data collection from 25 air traffic facilities (See Appendix A).

Materials and Procedures

For the online survey, participants received an invitation via email to take part in the

survey portion of the data collection. They first read an electronic consent form and then

proceeded to answer demographics questions, including where their facility service area is

located, their facility type, their ATC level, and years of experience.

Following this, participants responded to a series of questions about their opinions on

proficiently training, refresher training, and recurrent training. Each of these areas was

addressed in a scale measuring how strongly participants agreed or disagreed with statements

33

about each type of training. Lastly, participants rated how useful the overall content of their

training was, and how meaningful the delivery method was. Participants could provide any

additional comments in open-ended questions about additional training issues, etc.

For the interview portion of the study, front-line managers, training managers (larger

facilities), and air traffic managers (smaller facilities) from random air traffic facilities were

recruited. The team attempted to interview individuals from every type and level of facility.

Facilities were also selected to offer a range of geographical variability.

34

RESULTS – QUANTITATIVE FINDINGS There are five tables showing data from the CPC Survey. Questions 1 through 23 address

Proficiency, Recurrent, Refresher and Supplemental Training. The responses available were

strongly disagree (-2), disagree (-1), agree (+1), strongly agree (+2), ranging from -2 to 2.

Questions 24 and 25 ask for input on content and delivery method of proficiency training,

ranging from 1 to 5. The tables can be found in Appendix B.

Table 1 - presents the data as a function of all data, data from all respondents, data

divided by facility levels, data divided by years of experience, and data divided by facility

types. Table 1 gives an overview of the results of the CPC Survey. Included in the result

discussion will be comments from the open-ended questions. The open-ended questions in the

survey were as follows:

1. What improvements would you make to enhance proficiency training?

2. What additional training do you consider necessary, but are not receiving?

3. Do you think training can be delivered by a more meaningful and consistent

method, and if so, please elaborate on your ideas?

4. Please provide any other comments that you wish to share.

Table 2 - in Appendix B presents the data as a function of all respondents. Recurrent Training in

safety alerts, traffic advisories, weather and runway safety imply controllers overall appear to

receive adequate training on these issues. This is not unusual as they are among the top five

safety concerns that the FAA continues to target in training.

Topics, that seem to show a need to be improved upon, are the relationship between

35

charted minimum altitudes and underlying topography; lost aircraft orientation; aviation security

procedures; emergency situations; seldom-used procedures such as different traffic flows; ATO

Operational Contingency Plan; new and revised procedures, regulations and/or equipment; local

airport de-icing plans (LADP); weather observer training on ASOS or AWSS and LAWRS;

special use airspace (SUA) / special Activity airspace (SAA); Primary Backup Mode; and En

Route Decision Support Tool (EDST).

The last two questions of the survey are to receive feedback on the effectiveness of the

overall content and delivery methods of proficiency training. These questions were answered on

a scale of 1 to 5, one being the least effective and five being the most effective. The input implies

a slightly more positive response to effectiveness.

Table 3 - presents the data divided by facility levels in two categories. The first category consists

of levels five through nine and the second category consists of levels 10 through 12 ATC

Facilities.

Both categories seem to require additional training with the relationship between charted

minimum altitudes and underlying topography; seldom-used procedures; Air Traffic

Organization Operational Contingency Plan; and primary backup mode.

Additionally, to the above, the first category, consisting of facility levels 5 through 9, imply

insufficient training in the minimum safe altitude warning (MSAW); recovery in ATC

operations; lost aircraft orientation; aviation security procedures; emergency situations;

Supplemental Training I receive on new / revised procedures, regulations, or equipment; and

special use airspace (SUA) / special Activity airspace (SAA).

The second category, facility level 10 through 12, seem to approve of the training they

36

get in safety alerts and traffic advisories but lower level facilities imply with they require more

training;

Both categories imply there is room to improve the content and delivery method for

proficiency training. With a scale of 1 to 5 the content varies from 2.95 for facility levels 5-9 and

2.86 for facility levels 10-12. Delivery methods seems to show a larger need for improvement

with a 2.45 for facility levels 5-9 and 2.64 for facility levels 10-12.

Comments made on the open-ended questions adds more details to training needs as

follows:

Level 5 Facilities.

Make it applicable to my location and type of facility

Seldom used procedures

UAS operations

Modernization

Completely getting rid of CBI driven courses and take everything via eLMS

Keep these refresher courses updated periodically with new content

You don't want to do the same LAWRS course every six months. Make refresher

versions available as well, in a shorter format

Ask what questions people have. Some items are not covered by the orders, but

are in the FARs. For example, a VFR transitioning a Class D while they are IFR

due to low ceilings. The VFR is high enough and won't encounter the clouds. No

intention to land. Does a SVFR clearance apply?

I do like going to other facilities for recurrent training, as you get to mingle with

other controllers.

37

intricacies of LAWRS are lost in our training program.

If we get a briefing item, sometimes it seems like it's just checking the box, but

comprehension isn't assured.

Level 6 Facilities.

More real-life scenarios from similar type and sized facility.

Less training of systems and programs used at bigger facilities that don't apply

for us.

Importance of ensuring METARs are transmitting the correct information and

the impact of PIREP dissemination to the NAS.

I love the current ELMS system I just wish we could get more courses for

unusual events and emergencies

Scenarios and problems are outdated and people teaching it are outdated on

current procedures as well.

Equipment at some facilities are old and or useless.

We need updated material and updated resources.

Level 7 Facilities.

The courses shall be relating to ATC and only topic regarding air traffic.

Improve eLMS courses.

I am at a small facility that suffers greatly with staffing a mgmt. person and controller

are pulled to teach a class

Via a tablet or faster internet service.

38

Make the training more interactive and scenario based.

Less PowerPoint, more instructor led discussion.

Standardized procedures between adjacent facilities Instructor led discussion.

Level 8 Facilities.

The yearly required refresher training should be combined with recurrent

training and developed at the national level.

Simulators are the best way to prepare tower trainees to become CPCs however

many facilities have no access to these systems even if it shows that they have

access on paper.

The FAA needs to embrace technology that is readily available to provide

meaning full hands on simulator training to all tower controllers

I think we should line training up more like pilots where we come completely off

the boards for a week and do all of our refresher/recurrent at once and have

people who actually know how to do the job instruct us.

Level 9 Facilities.

joined the FAA from the DoD. I state this because I have never received GPS

anomaly or

Certificate of Waiver or Authorization (COA) for UAS operations training with

the FAA.

We very rarely ever have meaningful recurrent or refresher training

Level 10 Facilities.

39

Target issue relevant to local procedures and issues.

Training for situations specifically relevant to local facility deficiencies and

common issues.

It will have to be a cultural shift. From the top down. Time and staffing for

training has to be a priority for managers.

More hands-on training on emergencies and unusual situations so that

controllers learn to think on their feet and make quick and correct decisions.

Recurrent training does a fairly good job of this; however, they really need to

split terminal, tower and enroute. It takes away from the credibility of the

content when it is not applicable to a particular facility.

Refresher training TTL problems should be more flexible to allow facilities to

concentrate on those items that are most relevant to the situations and

deficiencies they are seeing in their own workforce.

Crew breakout and discussion time.

Level 11 Facilities.

Needs to be more interactive and relevant, and include more

discussion/participation. instructors need to be SMEs

create a new simulation environment,

Show value in the training by supporting its use.

Provide an opportunity to the workforce to engage in different types of learning

situations.

More equipment failure scenarios (e.g. radar, comm, etc.) RNAV (SIDS, STARS,

40

and GPS and/or RNP approaches),

Make it applicable to your facility type

Some drone

Yes, make it more interactive, not so boring.

More group sessions

Get rid of CBIs

Have instructors develop the training that are trained as educators.

Level 12 Facilities.

There may only be one or two individual sessions out of the classroom training

that are even applicable or relevant.

Distill the classroom training down to relevant topics on the operation itself.

How to handle emergencies, SVFR, military ops, etc.?

Get rid of all of the videos in the CBIs unless they actually help explain a

situation.

I don't find the classroom training very meaningful, the lab time is good, but

what is being taught during that time needs to be well defined.

Using simulators in an effective manner. Peer-to-peer effective techniques. Use

of replays as a teaching tool.

We do not properly train how to handle emergencies and urgent situations.

More DYSIM, real time problems with recorded live traffic and realistic

examples of traffic flow.

More focus on handling of emergencies and recovery situations.

41

More skill enhancement use of DYSIM,

Our safety trends indicate we're lacking in training.

If we're not taught how and when to issue a safety alert. How will we know

when? And how will it come out right when we try?

More training on emergencies and unusual situations.

More local examples.

Modernize the entire 3120.4. It does not meet the needs of the current ATC

system.

eLMS is awful.

Table 4 - presents data in two categories by years of experience. The first category is less than 10

years’ experience. The second category consists of 10 plus years’ experience.

Both categories imply the need for additional training on seldom used procedures; Air Traffic

Organization Operational Contingency Plan; new / revised procedures, regulations, or

equipment; and Primary Backup Mode. Additionally, the first category, less than 10 years’

experience, implies they are more satisfied with training over all.

One marked difference between the two experience groups are the less experienced

controllers seem to want more training on fatigue whereas the more experienced controllers

imply they had an adequate amount of training on fatigue. A second difference is the less

experienced controllers seem to find the unusual situation training adequate while more

experienced controllers find it lacking. A third difference is less experienced controllers find the

training on local airport de-icing plans (LADP) adequate but the more experienced controllers

find the training inadequate. The fourth difference is less experienced controllers find training as

a weather observer on ASOS or AWSS and LAWRS adequate and the more experienced

42

controllers imply additional training is required. The fifth and last comparison is the less

experienced controllers find the training on En Route Decision Support Tools (EDST) is

adequate while the more experienced controllers imply more training is necessary.

The second category, 10 plus years seems to imply that there should be additional CPC training;

Minimum Safe Altitude Warning (MSAW) procedures; relationship between charted minimum

altitudes and underlying topography; Recovery in ATC Operations;

Comments made on the open-ended questions adds more details to training needs as

follows:

10+ Years’ Experience:

Make it applicable to my location and type of facility

Seldom used procedures,

UAS operations.

Modernization!

Better instructors -maybe a national program that the instructors travel around

instead of using local people

Simulators are the best way to prepare tower trainees to become CPCs however

many facilities have no access to these systems

We are overwhelmed by training. There is way too much inaccurate, poorly

designed, or poorly delivered training given to us--causing an overall dislike for

all things 'training' meant for controller audiences.

Real life scenario with discussion, not an overly simplistic TTL lab

Flight join ups, and break ups, civilian as well as military.

Stop using controller cadres, create a new simulation environment

43

needs to be more interactive and relevant, and include more

discussion/participation.

instructors need to be SMEs.

< 10 Years’ Experience:

Have the training Less often.

Training on topics that apply to me and my facility.

Scenarios and problems are outdated and people teaching it are outdated on

current procedures as well.

Completely getting rid of CBI driven courses and take everything via eLMS.

Via a tablet or faster internet service. Videos are great, but need a faster internet

connection.

IFR/VFR conflict, vectoring VFR aircraft, emergency procedures

Yes. Make the training more interactive and scenario based.

More sim time in meaningful scenarios for CPCs to practice recovery,

emergencies etc.

Crew breakout and discussion time. Increase staffing to accommodate weekly

time off the floor so that controllers are more involved in things like training,

LOAs, airspace, QA, SGET.

headquarters and regional level to replace the monotonous CBIs with

""something”“that ""engages people"" and ""creates dialog""

How to interpret Instrument approach plates and to understand what the pilots

are doing for those approaches.

44

I really believe "Fam Flights" would be incredibly beneficial, but it is virtually

impossible for controllers to get off the schedule due to staffing issues

Table 5 - presents the data divided by facility types. Facility types are divided into three

categories. The categories are Air Route Traffic Control Centers (ARTCC), Air Traffic Control

Towers (ATCT) and Terminal Radar Approach Controls (TRACON).

The overall content and delivery method of proficiency training is more favorably

viewed by the ATCT’s and TRACON’s than ARTCC’s. All three imply there is still

improvement necessary in delivery method.

In ARTCC facilities CPC’s imply that there is a need for wide spread improvement in

training. The areas requiring the most improvement are charted minimum altitudes and

underlying topography; Emergency Situations; Seldom-used procedures; Air Traffic

Organization Operational Contingency Plan; new / revised procedures, regulations, or

equipment; Primary Backup Mode. One last topic was Local airport de-icing plans (LADP)

however ARTCC’s do not use de-icing procedures therefore would not be pertinent.

In ATCT facilities the topics that CPC’s seem to imply a need for additional training on

are Recovery in ATC Operations; unusual situations; lost aircraft orientation; seldom used

procedures; new / revised procedures, regulations, or equipment; and special use airspace (SUA)

/ special Activity airspace (SAA).

In TRACON facilities the topics CPC’s implied needed additional training were seldom

used procedures and Air Traffic Organization Operational Contingency Plan. TRACON’s do not

use de-icing procedures therefore they would tend to answer that they do not receive training on

Local airport de-icing plans (LADP).

Comments made on the open-ended questions adds more details to training needs as

45

follows:

ARTCC:

Show value in the training by supporting its use.

Provide an opportunity to the workforce to engage in different types of learning

situations. Not everyone learns the same way.

Training needs to be quarterly, meaningful and given by real instructors

Running simulator problems that have been updated would be nice

We do not train how to recover adequately when a LoSS has or is about to occur.

Top 5 at a minimum need to be added to recurrent, refresher, and stage 4 radar

training.

Things should be more test based and more accountability to someone’s

knowledge and ability to pass the test.

Swap routes & visual separation in DYSIM. DYSIM needs to be taken more

serious

More hands-on training on emergencies and unusual situations

CBI courses so out of date and technologically archaic

More unusual scenarios

Handling of military flights-formation, refueling

The real-world examples of radio recordings or videos are helpful.

Have the local safety council make it tailored to our facility.

ATCT with Radar:

46

Ensure that it applies to the type of facility that is receiving the information.

eLMS is awful. Half the courses don’t work or show when you are complete

Recurrent training both instructor led and eLMS are light years ahead of where

we were 20 years ago. Keep it moving forward.

Some of my feedback may just be tainted by the lack of enthusiasm at the mid-

level management investment

ATCT/TRACON:

I have never received GPS anomaly or Certificate of Waiver or Authorization

(COA) for UAS operations training with the FAA.

" Better instructors or at least train the instructors to do more than just read the

material "

Have long term national trainers who have been in the job longer.

They can teach from experience instead of just reading me the information.

Target issue relevant to local procedures and issues.

TRACON:

Using simulators in an effective manner.

Get rid of all of the videos in the CBIs unless they actually help explain a

situation.

Training videos that are relevant to each facility type would improve the

consistency and quality of information being shared with the controllers.

More training on emergencies and unusual situations.

47

More local examples.

There would be more time for meaningful training if we weren't saddled with

useless eLMS

48

RESULTS - QUALITATIVE FINDINGS

While the quantitative data in the previous section provides us with a descriptive

snapshot of the data, our secondary goal was to more closely examine the qualitative

comments provided by participants. We asked participants to respond to 13 open-ended

questions. We examine each of these questions in turn.

In doing so, we employed the use of NVivo, which is a software program used to

help organize, analyze and code responses in unstructured qualitative data (e.g. interviews,

surveys, transcripts, etc.). Using NVivo, we were able to generate clouds of data, revealing

the terms and items most commonly used by participants to describe their responses.

49

Question 1 - What, if any facility developed refresher training has been done? 1a) What air traffic operations at your facility do you feel may be considered high-risk but the controller workforce performs infrequently that may require additional training? Example: aircraft with engine out – any other examples?

Figure 1. Most Common Words and Phrases used by Respondents

There were two main themes that emerged from this question. First, most respondents have

received some training, but not in every topic that they were interested in:

“None for refresher training except unusual situations”

“Recurrent training twice a year, however not all topics covered.”

“30% recurrent does not pertain to their local operation (Example Climb via SID). This

is very costly training to send three people to XYZ to spend the night for cadre training”

“Wasted resources training on procedures not used at facility (such as Climb Via).

Training that could be used is how to apply visual separation in the tower. Much of the

workforce is confused by 7110.65 order changes and the mandates from the FAA are not

clear.”

50

Another theme revolved around which other topics should be covered in the refresher training

courses:

“The facility should develop refresher training dealing with a single runway operation

and the use of LAHSO.”

“Controllers need more exposure to VFRs, and possibly include them in tower and radar

simulator. Additional training on wind shear alerts. Running visual approaches to 3

runways and the possibility of a missed approach.”

“Parachute jumping needs to be covered more.”

“No Specific local training for TRACON.”

“Airport hotspots, Line Up and Wait (LUAW) procedures. Both identified through the

Local Training Committee.”

“Airshows – better briefings to all.”

Although there were two main themes, many items were identified as needing additional

training. Most areas identified were specific to a single facility or just a handful of facilities.

Topics from mid to low level towers and TRACONS.

Single runway operations

Land-and-Hold Short (LAHSO)

Construction on Airport

Duel Locals

VFR on Top

Helicopter Operations

Radar Outages

51

Opposite direction operations (multiple facilities)

Topics from high level towers and TRACONS.

VFR mixing with IFR Traffic (multiple facilities)

Wind shear alerts (multiple facilities)

Infrequent airport traffic flows

Triple ILS operations

Wake turbulence and RECAT rules

Airport Hotspots

Line-up-and-wait (LUAW) procedures

Stagger simultaneous approaches

Rarely Done traffic flows (multiple facilities)

Topics from Enroute Centers.

Approach services and operations at uncontrolled airports

Formation breakup and joining (multiple facilities)

Cruise / Through Clearances

Simulation for intercepts

Nonradar after major weather events

Presidential movements

Air refueling

Items were identified by every type and level of facility.

Special VFR (SVFR)

Parachute Jumping

52

Most of the larger facilities have developed some local training to cover the air traffic

issues specific to their situations, however many of the smaller facilities do not have the

resources to do so. It is felt that the FAA should tailor the refresher training to type of facility

(Tower/TRACON/Enroute) and possible level of facility would also help. Overall most

individuals considered the recurrent training good, however it would be better if it was focused

on the type of facility.

“Refresher and recurrent training doesn’t cover items unique to our local area. The training is very generic.” “Facilities need to be able to create training specific to a facility.” “The facility should develop refresher training dealing with single runway operations and the use of LAHSO, however lack of resources, especially personnel, prevent it.” “Recurrent training is done best. IT utilizes videos, and other technology.” “Add local flair to national refresher training. We need to collaborate with other centers to see what they are doing.”

53

Question 2 - Is there any required proficiency or skill enhancement training provided by the FAA that you consider inadequate? If yes, what training? Can you elaborate why? How would you deliver it more effectively? One main theme that revealed itself repeatedly was the issue with eLMS training not producing

the desired results:

“eLMS was supposed to take burden off of facility to conduct training but it didn’t really

accomplish that or meet all the goals.”

“eLMS – waste of time. Most of the time the controllers just click the answers. No

accountability. Many of the eLMS training areas are the same as what they have for

team briefings. Duplication of training material.”

“eLMS – the security training is the same thing every time. They do like that on the

newer eLMS you can test out of certain areas, which is good.”

“eLMS lessons – very repetitive, same info time after time. Keep the lessons shorter.

Would like to take controllers off floor, conduct classroom in a whole day and vary the

delivery. eLMS lessons cover mainly redundant everyday things – should cover things

rarely seen, ex. Holding at XYZ during weather.”

“eELMS courses – ineffective, many don’t apply to CPC such as computer courses,

texting while driving (they don’t operate government vehicles)”

“eLMS – terrible program and platform. Controllers and Management view it as just

checking a box. Pace of the training is bad. Technology is bad.”

Another common issue that was discussed was the TSS simulator:

“Skill enhancement – share TSS simulator with 7 other facilities and only two instructors.

When the simulator is not available then they need to use a table top.”

“Proficiency training is very difficult to schedule TSS simulator time and staffing at the

54

facility is very low so they don’t have enough staffing send controllers to operate

simulator.”

Most of the management personnel interviewed indicated the FAA’s required proficiency

training was “at best” adequate. Many elaborated much of the training was too generic, had no

accountability, and in many cases, such as: training concerning human trafficking, volcanic ash,

texting while driving, had no applicability directly to the controller’s job. Many in management

felt that much of the refresher training was “just checking a box”. Other issues included limited

internet access, especially at smaller facilities and the lack of radar and tower simulation, again

at smaller facilities. With more staff, it was suggested that more discussion / classroom based

training should be done. Another suggestion was to bring in individuals from outside the facility

such as pilots to come and talk to the controller workforce. Facilities would like access to the

facility data from ATSAP reports so training could be developed at the facility level to counter

any operational issues identified.

“All computer courses have no accountability; controllers just click through the course”

“Need more classroom training where questions can be asked and answered.”

“Half the recurrent training does not affect our facility.”

“In many cases it takes a half hour to load an eLMS file.”

55

Question 3 - Is there any required proficiency or skill enhancement training provided by the FAA that you think is done well? If yes, what training? Can you elaborate why? Some of the respondents indicated that none of the training was well done, but that most of it

was adequate.

“Nothing done well but adequate, room for improvement. All radar lab work, simulation,

inadequate staffing and scheduling”

“No nothing is done really well. We are overloaded due to how many courses we are

giving to CPCs. We need to hit a topic hard, from a number of directions, using 3 or 4

different delivery methods”

“No. Not really. The FAA is trying to improve it but most controllers think it is a waste

of time.”

“No. FAA needs to be more proactive in the training. Don’t wait for something to

happen. Much of the material needs to be vetted before being disseminated”

“No FAA training has been done really well”

“Generally, eLMS boring not good. Click through.”

Many of the respondents disagreed with this assessment, and pointed out some parts of the

training that they felt was well done.

“Done well is 7110.65 change 3, 3/19/17 IFR to VFR. This was a very good word

document. Need a little more on human factors issues. The TALPA on braking action

was also good...”

“XYZ ATCT did proficiency training well, because they devoted a whole day for a team

to train together.”

“The eLMS items are pretty good”

“7110.65 Change 3 briefing was good. IFR on visual approaches. Parts of recurrent

56

training done well, but still needs to apply to location and facility type. Good job with

braking action – TELPA.”

“Active shooter was good, however from an operational standpoint not much was good.”

“Recurrent training good especially “Wrong airport/surface” training.”

“Recurrent training at XYZ was good and seemed to motivate everyone.”

“Emergencies was good in the eLMS course. Derelict balloons was good.”

Of the subcategories that make up Proficiency training, most respondents considered Recurrent

done well. Respondents prefer training that is operational in nature, and it is especially effective

when presented in a variety of methods, such as eLMS, videos, radar recreations with voice, and

group briefings. Those cadre instructors trained to perform Recurrent training say it is done

well, however it was noted that since it takes two people, it can be a strain on staffing. A few

respondents believed performing Recurrent training every six months is too much, and it should

be done once a year.

“Recurrent training is done the best. More up-to-date technology in presenting the

material. Variety of delivery methods: videos, briefings etc. Good that NATCA and

Management are involved in this training.”

“Recurrent training engages the workforce. The videos and PowerPoints allow the

controllers to ask questions and have interaction.”

57

Question 4 - Is there any additional training you consider necessary for the CPCs but is NOT currently provided or required? If yes, what training and why? Who should provide this additional training? There were quite a few suggestions for additional training that respondents felt was necessary but

not currently provided or required. All but five respondents offered some suggestions. Some

participants felt that much was missing and listed many examples:

“Better QA/CEDAR training. No real formal course training. Only annual CBI courses

which are outdated and annual CIC refresher training doesn’t really have any substance.

Training doesn’t really teach how to manage the program or write reports. Better

training on CIC duties to include documentation and report events. Better training on

how to handle emergencies – current CBIs very generic and outdated. Once a controller

is signed off as a CIC, there is little or no additional training. It is one of the most

important positions, with little follow up.”

“Emergencies, radio out, how to operate emergency radios, what airspace when taken

from other facilities due to outages. Radio/communication outages and the procedures

for when it happens. Navaid reset if you are responsible for a navaid. What does your

facility do when an adjoining facility goes to ATC 0? What airspace if any does your

facility take over and the what procedures are set in motion.”

“Go around procedures should be trained more especially in the tower simulation lab.”

“We need the big global picture. Something beyond their current job function. Example

Tier 3 ground stop. However, the down side is people go to a lot of meetings and then

nothing develops. Specific procedures to the airport that are not done very often. Set up

specific radar simulations dealing with those type of procedures. Talking about it does

not work. More “big picture” training, and what happens beyond your airspace. The

58

impact of TMU on the workforce.”

Other respondents felt that most things were covered and only listed one or two examples:

“Go- around in the simulators. CPCs should demonstrate and get signed off, with a

written test for accountability.”

“Specific procedures such as Simultaneous to runway XX, XY and visuals to XZ”

“XYZ use of TSS (Tower Simulator).”

“Occasionally train on areas not really used, i. e., non-radar.”

“Human factors training done by outside expertise always very good.”

“IFR to VFR visual separation climb”

“No procedures for e-mail use.”

The need for additional training for CPCs was varied and often depended upon the type of

facility and the level of facility. Respondents from a multiple and varied facilities suggested the

FAA at the national level produce videos for the entire country. Videos would include

interpretations of 7110.65 changes, such as B757 Wake Turbulence. The Academy or FAA HQ

could produce videos geared to a specific option. Misinterpretation of a national order would be

reduced if the same office produced it for all air traffic facilities

“Specifics of a 7110.65 change that has national implications. Give one brief from

Headquarters, but leave room for local implications. Use a video produced for the entire

country. The B757 wake turbulence changes were important, but there was not a

national brief. As a result, implementation was done very poorly.

A number of respondents from high level TRACONS and centers to lower level towers,

indicated additional training needed for OJT instructors.

59

“OJTI training not adequate. More training on how to comprehensibly train and

perform training.”

“Need training on how to train”

“OJTI training needs to include more on how to train. Need more in OJTI training that

deals with working together and communications.”

“Controllers need to be taught how to become an effective communicator / trainer. They

need to be shown how people learn.”.

60

Question 5 - In order to make training more effective, how would you improve the training- / Environment / Delivery / Material:

Figure 2. Frequency of word or phrase usage by interview participants for question 5

There were quite a few suggestions from respondents on how to make training more effective.

Some respondents had several suggestions:

“More customized proficiency/refresher training for the facility. Always a push to have

61

the FLM do the training but needs to be done by the training department. Smaller

facilities don’t have an opportunity for team briefings. The goal should be keeping he

interest in the training. Is the person performing the training creditable? Is the training

relevant to what is happening in the facility? Videos can be more entertaining and hold

your interest, however you cannot ask a video a question. Is the training tailored to the

audience – tower controller vs radar controller.”

“If the FAA does not mandate it, it won’t happen”.”

“Overcome some controller mentality of “you can’t teach me anything”.

“One size does not fit all for CEDAR, eLMS and face to face briefs. Delivery:

Simulation is effective, but we need pilots as resources. Simulation training would be

great if there is accountability by the controller and tested out with an instructor.

Material: DO NOT put surveys on required courses. Need more face to face, rather than

“…go read this”. **Need some type of accountability at the end of training. Part task

trainers may be a problem for CPCs, but good for developmentals (Visuals to parallel

runways and turning on above glide slope). Part task training would need to be

specific.”

“Environment – communication system to mirror what’s on the floor, other areas of

simulation are very realistic and good Delivery - ELMS/CBI; not easiest to use, not

very user-friendly, supervisors can’t easily find out what training has been done by

controllers. Classroom sessions would be best for some subjects, but hard to get people

off. Material – Good; some courses don’t contain many graphics/pictures at all, with

slide after slide of wording. Level of some material somewhat elementary and not quite

as professional, and takes a long time to update material. Simulation – make

62

realistic as possible Computer Games – the present generation could benefit. They

need to be active in the training. Part task training – clearing into MOA, clearing

for ILS, and items like these could provide more efficiency. Make training more

interactive, more engaging.”

Other respondents only had a few, or no, suggestions:

“More classroom for better interaction”

“Tower/TRACON - Incorporate the two facilities in training to encourage more

discussion”

“Delivery – create a new platform to deliver the information quickly, direct and to the

point”

“Material developed for cadre is not at the facility level.”

“Try having briefings at the beginning of shift to bring controllers up-to-date on

operations.”

The recommendations on how to make training more effective were quite varied. Some included

improvements to internet access, expanding radar simulation, and training in a group session.

Most of the suggestions for environmental and delivery issues were from smaller tower facilities.

Lack of resources especially at smaller facilities was a common thread.

“Larger radar lab with more scopes to allow more than one controller at a time to train”. “Bigger radar lab, i.e., more scopes to engage and train more people, how to add culture

change dealing with younger controllers, improved internet speeds.”

“Faster internet speeds to allow eLearning and CBI to move along more efficiently.” “Often takes 10 minutes to get into eLMS.” “Increase internet speed to improve any training requiring computer usage”

63

“Some platforms do not run the required videos”.

“Increase bandwidth – eLMS freeze up, speed of the internet computers is too slow.” Larger TRACON facilities had other suggestions. Several respondents felt the radar simulation

to perform recurrent and refresher training did not have realistic aircraft performance, resulting

in scenarios not matching actual traffic situations and impacting training.

Improved simulation such as better aircraft modeling characteristics to be more

realistic, time allocation.

Rework the simulation to make it more realistic. Don’t have what is trained in the

training department different from what is done in the operation.”

“Simulation is generally good, however some of the simulation is not realistic. How

targets in the simulation act is not real. The turns, climbs, how they slow down are not

what really happens.”

Respondents from all levels and type of facilities prefer face to face briefings as opposed to

“just” reading the briefing material. This allows participants to ask questions and in some

instances, discuss the issue among themselves.

“Face to face briefings would be better than just having someone read the material.” When asked about NATPRO, Respondents who had at least 15 years in the FAA, generally had

good things to say about the program. Most believed that bringing some form of it back for skill

enhancement training would be beneficial.

“Part-task trainers – not at XYZ. NATPRO - need to bring it back, would help with

listening and scanning.”

“Delivery – training not interactive. Should be joint delivery with FAA and NATCA.

Should have Computer games and NATPRO”

64

“NATPRO – some portions were better than others and could be used to enhance skill.

Scanning tool was good.”

“More engaging training is good. How to do that is hard, possible NATPRO

competition.”

“NATPRO – some controllers liked it some did not. A lot of it was how you sell it. If a

controller has a buy in, then they accept the training. If not, then they will not get

anything out of it.”

“Like NATPRO and the competition it created.”

“NATPRO – some felt they received good training from it, while others thought it was a

waste of time. Would it be possible for a controller to earn credit to play NATPRO?”

“We need to bring back NATPRO. It would help with listening and scanning.”

65

Question 6 - Do you feel the trainer’s job/position is a factor in how well the training is received? Please elaborate. For this question, many of the respondents highlighted the need for trainer credibility:

“Credibility a huge factor. The person performing the training needs to be able to

answer questions, or at least know what they are talking about.”

“Credibility is very important. I can’t have a TRACON teach runway separation with

credibility. The trainer for operational items needs to have credibility.”

“Need credibility. Some just read it. The person doing the training needs credibility. A

dedicated team should do the training.”

“A trainer needs credibility. The person has to come in with the mentality of an

instructor, not “why are we doing this”.”

“The person giving the training must be knowledgeable and have credibility”

“Yes, don’t want the least professional person teaching professionalism. Person needs

credibility.”

Related to credibility, many respondents pointed out that trainers need to be knowledgeable and

experienced:

“The person needs to be knowledgeable and experienced. OJT instructors often do not

have the resources to provide additional training.”

“Must be believable and knowledgeable.”

And there were several comments about the trainer being someone who is likeable,

approachable, and well versed on how to teach effectively:

“Likable, good controller, able to work with others, able to provide corrective criticism

in a forceful yet workable method; must be given in a way that is accepted and received.”

“Yes likability and credibility are important. Example is the face to face briefs twice a

66

year.”

“Yes; personality skills, respect; has to be able to know how trainee is learning;

relationship with trainee; been there done that. Instructors need to be able to engage the

group. Trust of the controllers makes a big difference.”

“Yes, not interactive or entertaining, the training session is lost, must keep them

engaged.”

“attitude, professionalism, energy-excited, good”

Respondents from every level and type of facility believed the individual performing the training

or giving the briefing needed to be knowledgeable about the subject. Individuals conducting

briefings, on national order changes, also had to avoid giving their own interpretations or

opinions on the subject material. To minimize individual interpretations, subjects concerning

national issues could be presented by higher-ranking FAA officials. Respondents also suggested

FAA HQ could be involved with briefing some of the “TOP 5” recurrent training items.

Respondents favored supervisors and union reps working together to give briefings.

“Good credibility if someone higher in authority gives a video brief (Top 5), with room

for a local supplement.”

“When top management are involved with briefings it makes a difference for proficiency”

“The higher the position of someone giving the briefing the more the controllers will pay

attention.”

The Top 5 recurrent training could be done by video from FAA HQ, with specifics such

as ‘Last year there was 73 reports of aircraft almost hitting skydivers’.”

67

Question 7 - Do you feel training in a group session is a worthwhile endeavor? From other facilities? Many of the respondents indicated that group training can be worthwhile if done correctly, and

with the right participants:

“Training together on certain topics would be good. Group training within the facility

and multiple facilities.”

“Training together with other controller would be good because it would allow more

thoughts, ideas and discussion concerning a training topic.”

“Some combined training works well. Some training sessions would be good, but others

could create problems. Within the facility training with others would be good. The use

of overtime to get people off the schedule, so groups could participate in training.”

“For the right subjects, group setting is good”

“Groups work well with the right subjects”

“Groups are good for interactions, and sending a cohesive message. Usually groups

create a budget and time issue for the facility.”

Respondents generally agreed that training in group setting was beneficial. Briefing items, order

and procedure changes, and other operational issues were enhanced by the opportunity to discuss

with fellow controllers. Concerns and questions could be resolved as a group. Strong guidance

or facilitating would be required to keep group on task. Respondents were also in favor of

bringing in outside professionals such as pilots, dispatchers, or meteorologist to discuss specific

issues involved with the facility. Visits to other air traffic facilities to perform joint training,

briefings or familiarization could also provide good understanding and cooperation between

facilities.

“Training with other facilities is very productive. Good interaction between different

68

facility controllers.”

“Bring pilots to discuss weather”

“Bring in pilots and representatives from flight schools to talk about their issues.”

“If possible have controllers go to other facilities.”

“Classes with other facilities is good when working on LOA’s.”

69

Question 8 - Do you feel the controller workforce is aware of fatigue? If “yes” is the controller workforce taught ways to mitigate fatigue? Do you feel they need more training?

Figure 3. Frequency of word or phrase usage by interview participants for question 8

Most of the respondents agreed that fatigue is an issue, and that the workforce is mostly aware of

fatigue-related consequences:

“Controllers are aware of it.”

“Most controllers are aware of fatigue”

“Controllers are aware, especially the older more senior controllers.”

“Most controllers through NATCA are aware.”

Many respondents felt that enough was being done about fatigue and that no more training was

necessary:

“The controllers are aware of fatigue through the eLMS courses and the union contract.

No more training needed.”

70

“No more fatigue training”

“No further training needed. The controllers do what is necessary to combat fatigue”

“No further training needed on fatigue”

“Fatigue mitigation briefed but not an issue, so we don't need more training.”

“Yes, CPCs are taught about fatigue and how to manage. No more training necessary,

but they could be reminded annually.”

However, there were some comments questioning whether enough was being done about fatigue

training and remediation:

“Most controllers are aware of fatigue; however, the FAA does not teach enough about

it. Current training is a “band aid”.”

“They need more training on the subject.”

“The current fatigue training needs to be refreshed. “Recovery piece” needs to be more

thorough.”

“Work force knows how to recognize it in themselves. Always can use more training”

An overwhelming number of respondents stated there was sufficient training and briefings on

fatigue and how to recover from it.

71

Question 9 - Is the controller workforce taught ways to optimize scanning of radar displays/out windows? If no, do you feel such training could be useful? If yes, how is the training accomplished and could it be improved? Many of the respondents agreed that the controller workforce is taught ways to optimize

scanning of radar displays/out windows.

“Center controllers are taught how to scan, so yes it [training] would be useful.”

“Trainers focus on trainees for scanning, not much of an issue at facility.”

“Scanning training is done as a part of OJT”

“Learned during OJT”

“Scanning is taught at the Academy with trainer”

But not everyone agreed with this assessment:

“No training now, but it would be good to have some training in this area. Big hole in

this area.”

“There is a lack of awareness”

“There is not specific training for scanning.”

“Controllers could use some form of training to improve scanning. A training module

would be useful”

“Scanning for CPC is not available but very interesting. Would be very beneficial. If

there was a tool – it would be great.”

“No method of scanning provided; if something were available it would be nice, just not

sure how it would be done.”

“As an agency, wish we had something, a game like NATPRO for scanning.”

Others agreed that they are taught ways of scanning, but that it could be improved:

“The controllers are taught ways to scan but it should be tailored to the individual.”

72

“Training on scanning should include ways for the controller to keep a running dialogue

in their heads about looking at the strips and data tags and what the info means. Need a

way to determine on national level on how to scan. Controllers will have “scan drift” if

they don’t practice diligent scanning.”

“Open to something new to teach scan. Sitting down a talking about it won’t work. Need

a video.”

Many respondents indicated that scanning was a personal issue that should be developed when

the individual is in training. However, others acknowledged that some form of scanning training

for CPC’s could be beneficial in both a radar and tower environment.

73

Question 10 - Do you have enough training to address UAVs that may deviate from their flight plan and breach separation from collision avoidance? Do you feel the controllers need training to communicate with UAS operators? Most of the respondents agreed that there was enough training to address UAVs that may deviate

from their flight plan and breach separation.

“Enough training provided.”

“Overall adequate training on UAVs. Most controllers are well aware of the issues.”

“UAS training is adequate”

“We’ve had enough briefing items on UAVs”

“Don’t feeling training is necessary at this time”

However, there were many complaints about the quality of the training:

“The UAS training was useless. We don’t care about size. What happens to the

perpetrator would be interesting? CIC training on how to handle specific requests by

UAS operators. Current training did not apply so was useless.”

“Don't have enough training for UAS. Mandatory training is basically reading. Could

be a requirement for refresher training. So much information that we need a checklist to

ensure check lists are updated.”

“Reference UAV they have enough training, but need continued training as things

change.”

“Not enough training in UAV’s. Supervisors are not given enough training.”

“CPC Contradictory training and constantly retraining.”

There were several suggestions for future training topics in this area:

“Controllers could have more training on how to deal with requests from operators, and

not just say “no” to every request”

74

“Controllers get reports about UAS sightings but are not sure what to do about it. Also,

they do not know how to deal with approval or disapproval process.”

“Should be more site specific to what each facility needs”

“Only training is how to report incidents. Recreational UAV no communications”

The respondents to this question were divided concerning training for UASs. Smaller tower

facilities in general indicated that they received enough training, however those from larger

facilities such as Level 11 and 12 TRACONS acknowledged they could use additional training.

Respondents specified different UAS training may be needed based on whether you deal with

UAS as low altitude / level around an airport compared with UAS circling for extended periods

of time at high altitudes (above FL 600). Many respondents recognized the UAS industry was

changing quickly and it was difficult to keep the current information available. It was suggested

a central depository for UAS material including contact information be made available to the

controller workforce.

75

Question 11 - Any final thoughts on what the FAA could do to make proficiency training more effective and efficient? There were many comments on this question. Please see the Appendix for a complete list. One

theme that arose several times was the need for local training:

“More locally developed, site-specific training.”

“Need LOCAL content and it must be interactive.”

Another theme about accountability was frequently mentioned as well:

“ACCOUNTABITY for the proficiency training. Controllers feel much of the training is

done because someone else messed up. They don’t take personal ownership. Individuals

need to be held accountable for learning the material”

“ACCOUNTABILTY. What does it matter to me?”

In summary, the respondents were mixed on much of the training they received and how it could

be improved. Many indicated training needs to be more engaging and more interactive. Instead

of controllers just “clicking a box” the method of delivery needs to get the individual thinking.

From a front-line manager and training manager perspective, the training material must be

understood by those receiving the training and there must be some accountability. The person

performing the training must know there are consequences to not completing the training

successfully. The workforce prefers training that include “real world” situations and lessons.

Sanitized videos and audio of actual events presenting lessons learned and best practices should

be used to enhance a topic of training. Individuals performing training, especially in group

settings should receive more training on how to conduct briefings sessions. The training should

be broken down to fit specific facility types and be able to address the controllers question of

“how does this affect me?”

76

FAA Academy

The team researched “proficiency training” for Certified Professional Controllers at the

FAA Academy. The FAA Academy has the means to incorporate high levels of proficiency

training, covering advanced concepts and competencies. They have the ability to span various

levels of difficulty and multiple course delivery methods. The Academy uses classroom

instruction, advanced simulation, web-based instruction, and multiple media formats. They use

Blackboard, eLMS, and LCMS (Learning Content Management System), stand-alone trainers,

and NATCA is apparently attempting to obtain IPads for employees. AJI owns all courses and

academy partners assist the FAA to keep the courses current. The academy can help with

changes to resident courses or field course, however field courses are usually not updated by the

FAA Academy.

However, FAA personnel indicated recurrent training is “thrown together to check a box”

and said the “FAA needs to get better”, “Many things are not being done on National basis”,

“We need real world applications”.

The FAA Academy has production capabilities to deliver courses to field facilities. They

appeared to be rarely used. They delivered a Technical Operations course to Denver via the FAA

intranet. They called it a virtual classroom, but from their description it was similar to college

synchronous web based course delivery. Due to band width, the course delivery did not work

properly. At the time of our visit they were considering using a Satellite link. The FAA

individual we spoke to thought approximately 85% of FAA facilities have satellite links. The

FAA official said another method of course delivery may be terrestrial, using T1 lines, but a

business case would be needed to determine the correct method of course delivery. Cost is a

factor (190 down links with $3000 per site and $1500 recurring). On the other hand, it costs

77

approximately $250 per day for employee per diem to bring folks to OKC.

INET is a remote site for downloading field courses. There are many courses. All are

kept up to date by AJI. Tech ops has 512 courses or systems to maintain. The FAA official

indicated it would be nice to have VR (Virtual Reality) for some of the courses and facilities they

are required to maintain. Tech ops personnel may not be comfortable working on a facility due to

lack of proficiency. This is costly for the FAA. For example, Techs may not be proficient on a

facility, so they order a complete set of replacement parts, then return to depot. Sometimes they

say parts don't work, but actually they are good. Additionally, the FAA Academy is

experimenting with Microsoft HoloLens.

The FAA Academy also has STARS Elite that allows field facilities to connect with the

academy or FAA Technical Center. It was tested in the El Paso Texas airspace. Latency was

nonexistent and it appeared to work well. An FAA official said STARS Elite may help to resolve

proficiency issues. This person indicated PBN has degraded ATC vectoring skills, and FUSION

creates some timing issues with proper vectors. "Practice" could be used to solve some of these

issues, and it was developed by AMA500. The “Practice” program works well for ab-initio

training for terminal and enroute. It should be noted that CAMI indicates for every hour of

simulation, OJT is reduced by 3 hours.

The team discussed the practical use is for asynchronous course delivery. This type of course

delivery would help with field facilities during low staffing situations, controlling costs,

consistency of message, etc. The FAA does not have a library of recorded lectures. The industry

partners; Trans-Lumen Technologies, Washington Consultant Group (WCG), UFA, and others

may be able to provide:

1. Preliminary development of recurrent skill enhancement training for CPCs.

78

2. Preliminary development of proficiency training for CPCs

3. Provide a centralized approach that maintains consistencies in content and delivery.

79

DISCUSSION AND PRACTICAL IMPLICATIONS

The FAA’s proficiency training program faces a number of challenges in ensuring CPCs can

meet the demands of the technological and operational changes anticipated in the NAS. The

training needed includes training for changes in airspace and procedures, advances in

automation, skill enhancement, Unmanned Aerial Systems (UAS), and commercial space

operations, and scanning. In addition to meeting these ongoing needs, current CPC training

practices, methods, and lack of resources must be addressed or the post certification training

program could be in jeopardy of not meeting the training demands. Plus, a number of other

issues were identified through the survey and interview process and include:

• Lack of standardization in those topic areas where it would be appropriate and beneficial

• Lack of flexibility in training delivery to meet individual CPC needs

• Inefficient training due to the need to implement training that may not be relevant to a

CPC’s operational position.

For certain types of CPC training, the method of delivery is spelled out in FAA orders. When

training delivery methods are not mandated, the facility usually selects the delivery method. Due

to limited resources, facilities are often forced to select the delivery method that is most

expedient rather than the delivery method that is most effective or best suited for meeting

training objectives.

For training complex skills, simulation training is the most effective delivery method.

However high-fidelity simulation can be expensive and resource-intensive. Due to lack of

resources, facilities sometimes tend to rely on instructor led training or eLMS instead of

simulation training. This results in a mismatch between the training delivery method and the

training content.

80

Resources

In the current air traffic environment, CPC proficiency training is governed by FAA

Order JO 3120.4. While the order provides a high degree of flexibility at the facility level,

allowing sites to tailor some training to meet specific needs, many small facilities do not have the

resources in staffing to accomplish the intent of the order.

The delivery methods for proficiency training vary depending on several factors,

including the immediacy, complexity and amount of training material. Classroom instruction

and CBI are effective for acquiring basic knowledge, but high-fidelity simulation is critical for

refreshing the specialized and complex skills for CPCs, which presents other issues.

Each facility is responsible for developing the simulation training to ensure the

requirement is satisfied. The development of these scenarios in each facility is extremely

resource-intensive and requires specially trained personnel and many days of effort. Facilities

often use previously-developed scenarios rather than develop new simulation scenarios each

year. Most simulator training, requires instructor guidance, remote pilot operators, and the time

and equipment resources of the training lab. Large facilities may have the personnel to conduct

extensive simulator training, again the smaller facilities generally do not.

Large or small, most facilities do not have the type of training technology that will

support targeted, part-task training which allows controllers to focus on an individual skill or

task in isolation. Current ATC training methodologies typically rely on high-fidelity simulation

to convey skill and task development curriculum forcing the controller to practice the skill or

task while at the same time attending to all other controller tasks and actions. Attempting to

enhance a specific skill while trying to successfully control traffic in simulation is considered

difficulty and inefficient.

81

Web Based Training

Web based training or eLMS for the most part has replaced the traditional CBI in the

ATC environment. This type of training provides information generally in a declarative,

academic way with limited interaction by the CPC. The current eLMS and CBI training

platforms make it difficult for the system to quickly and flexibly adapt to deliver new training via

the most effective and efficient means. The eLMS database keeps track of whether CPCs have

completed the training, however there are no performance criteria for eLMS. Other than

completion, there is no way to evaluate any particular controller’s comprehension of the course

materials or provide a degree of assurance that a transfer of knowledge occurred.

Performance Assessment

Performance assessment criteria and measures for proficiency training in some cases is

undefined and generally the only requirement is that the training is completed. Training

personnel stated many training modules do not contain an evaluation component or a way to

objectively assess knowledge gained or level of performance obtained. Training managers and

supervisors expressed that CPCs considered much of the training as just “checking a box” with

little or no ramifications on how an individual scored.

The lack of standardized and objective performance measures contributes to variable

overall quality and creates a challenge for ensuring that all CPCs have achieved the training

objectives. Lack of performance criteria is demotivating. Many CPCs do not engage in active

listening or learn effectively during instructor led training or eLMS courses, which leads to

diminished potential for effective learning.

82

Adaptive Learning

Web based training and simulation training do not adapt to controller performance in

actual time to meet an individual CPC’s training needs. eLMS and CBI do not adapt to

individual performance by providing more practice for certain topics, or by allowing more well-

informed CPCs to skip ahead.

The inability to adjust training as it happens, diminishes learning opportunities for CPCs

who may require additional practice for particular operations. In addition, the current delivery

methods are incompatible with part-task training design that can accommodate varying levels of

skills and abilities and allow for additive skill-building; even though this type of training has

shown to be effective and efficient for training complex tasks.

Train the Trainers

A discussion should ensue as to who should train, on what tasks should they train on, the

diversity of trainers matched to facilities types and the tools that they need to meet instructional

goals for trainee acquisition of knowledge and skills. Factored into the evaluation for the best

path, is how much trainee time can be absorbed by their facility for “not on the job” and trainer

resources either being diverted from their ATC positions or sourced from external areas, such as

retired Air Traffic Controllers from industry, academics and government or individuals

specifically trained for such responsibilities. Mitigating costs would be augmenting tools, such as

gamification and mobile simulators that can be accessed by the students and trainers more

frequently. It is important that trainers who have experience also are up to date on the latest tools

available and can translate them into the life experiences to convey to the current demographic of

students.

83

Bandwidth / Internet

One of the current difficulties encountered in getting the latest in teaching materials rich

with interactivity and digitally intensive data is related to slow or the unavailability for

downloading content to devices due to inadequate bandwidth and WIFI. This issue is particularly

difficult in remote areas where the effort to upload and download has limited connectivity.

Improving network utilization will ultimately produce faster data transfer and less frustration

with using the Internet and therefore, more adoption.

Biasness in Training Methods

Through the interviews there is a conclusion that from one generation to another there are

different methods to teach. While more honed and engaging training modules and topics relevant

to the audience is desired, experience is a necessary component to engage the right lessons.

Therefore, SMEs should come together to identify the required skills and knowledge for part task

training that can be converted to new methods of training such as serious gamification. There is

also a requirement that the trainers need to be trained themselves for how technology innovations

can impact the FAA environment, such as UAS/ USSS centers.

Training Topics

During the interview process a number of topics were identified by the workforce as

needing additional training. Issues include UAS operations around airports and at high altitude.

Scanning both in a radar and tower environment. Commercial space operations and how it will

affect the National Airspace System.

The impact of future commercial use of the NAS, will be composed of a large

number of diverse industries that capitalize on advancing and integrating the UAS into their own

84

business missions. How these businesses will interface with other companies, regional control

centers and comply with manned airspace operations will require the businesses to have a

trained. capable workforce to obtain skills and knowledge for complex flight controls and flight

rules which may not be assessable to all companies. Particularly problematic would be if such

commercial entities do not realize that training is even necessary or budgets do not exist for this

kind of employee role in the organization.

Situational Awareness / Scanning

Situational awareness through scanning by air traffic controllers is essential. Visual

scanning strategies is a challenging issue due to the dynamic movement of multiple aircraft or

targets. Many respondents from the interviews indicated that training to assist in learning to scan

or improve a controller’s scan would be useful. With the requirements to task the ATC with

more visual challenges, additional training to augment scanning techniques lends itself to more

skills acquisition using transference from training methods to the work performed in the field.

There are dozens of scanning methods to provide for visual training.

85

LIMITATIONS The AT-CPC Enhancement Training research project was designed to disseminate surveys

to FAA management, Training Administrators and CPC’s to gather information on the existing

FAA CPC training. After the conclusion of the survey there were interview questions designed

as follow up questions.

The survey was designed by the researchers of AT-CPC Enhancement Training project.

A panel of subject matter experts validated the survey. In addition, FAA management and

NATCA reviewed the survey before being administered to ensure that the survey was acceptable

and the union approved of the questions for the target population. The survey was not approved

by the FAA to be administered to management until it was too late to complete prior to the end

of the project.

The Interview questions were reviewed by FAA management and NATCA however the

project was only able to get approval to interview facility management. We were unable to get

approval by the Union for the interview questions to be administered to CPC’s.

86

RECOMMEDATIONS Many of the aforementioned issues relate to lack of advanced training technology integrated

into the ATC training system. Extensive research by MITRE and other organizations have

demonstrated that the application of different innovative technologies such as: speech

recognition and speech synthesis, game technology, simulation, automated performance

assessment and feedback, real-time coaching and tailored instruction, and adaptive learning,

would benefit training quality.

Bandwidth / Internet

Ensure each facility has sufficient bandwidth and internet access to take advantage of

advanced training technologies.

FAA Academy

A centralized configuration for certain training would enforce standardization of instruction

and use of dedicated educators, and help alleviate bottlenecks at field facilities. Utilize the

capabilities of the academy by having them produce training videos or conduct online courses

dealing with issues at a national level. Briefings concerning updates and changes to the FAA

Order 7110.65 could benefit from a standardize and centralized training approach, and could be

tailored for the particular air traffic option of facility type.

Training Topics

At times, facilities are required to conduct training on topics that are not relevant to their

workforce (e.g., a controller in an enroute area required to receive refresher training on de-icing

procedures). This results in inefficient use of training time and resources. Tailor the refresher

training to local needs as quality control and safety data dictate.

Develop training in UAS management that matches the facility type. Tower management /

87

controllers should learn UAS issues that concern airport and terminal operations. Enroute

controllers should concentrate on low and high altitude UAS issues.

Commercial Space will require guidance through an overarching strategy for assisting

Industry with training for areas where such core competencies do not exist. It is the correct time

to start creating training curriculums with the UTM programs requiring the necessary

competencies before companies launch their systems into a crowded autonomous and manned

NAS.

Scanning techniques are critical to the ATC. A potential solution to accelerating

developmental skill acquisition for lifelong career ability to perform their job, lies in rapid game

development and its distribution for scanning. These can be simple games using transference

theory where visual skills learned through casual-style serious-gaming can augment classroom

and eLMS curriculum by transferring such skills to other visual tasks. These scanning games

then can be designed for more simulation driven games as pre-simulator part task training,

available 24/7 with an easily accessible repository of game modules that can be adapted and

modified for a facility focused experience.

Adaptive Learning

Courses should be designed with modular components that can accommodate varying levels

of skills, knowledge and abilities. CPC post-certification training is currently not designed with

flexible components that enable training to be administered in a linear or non-linear fashion,

thereby supporting training for varying levels of experience and allowing strengths and

weaknesses of controllers to be more readily identified and addressed.

When appropriate use performance assessment measures to develop “pre-tests” for training

in order to measure CPCs existing knowledge and encourage the delivery of more customized

88

training that meets individual CPC needs. Pre-tests can be used in refresher training to assess

knowledge already acquired relieving CPCs of having to spend time learning information that

they already know. Pre-tests will increase training efficiency by enabling CPCs who pass pre-

tests to move ahead and complete required training more quickly.

Performance Assessment

CPC performance should be assessed during and upon completion of training. For

simulation training, instructors should monitor CPCs during scenario runs to ensure completion

and provide an informal, subjective assessment of performance. New simulation upgrades

should include automated evaluation summaries. For web based training such as eLMS,

performance criteria should be developed. Poor CPC training performance should require

additional training.

Industry Updates for New Technology Insertions

To really understand the changes in a particular ATC system, have experts who have

designed the systems and worked with technologies be part of the training curriculum. Have

industry when appropriate, be present in the classroom and in authoring learning tasks. While in

a number of cases this may be done already, the FAA should be requiring the industry who

creates new tools to design the training for integration into existing lessons.

Accessing Training

Develop a central database to maintain locally-developed training content, best practices and

effective techniques. The database could also be used to enhance training distribution. Make the

extensive libraries of training materials accessible from every air traffic facility. Part task and

topic items that are relevant to a given facility or category of facility and job task should be

89

designed to be filtered and searched so there would be less burden on the system while reducing

the time to acquire the correct lessons and follow through to learning. This methodology will

also take off burden off the bandwidth utilization.

Biasness in Training Methods

CPC post-certification training need to reflect the state-of-the art in matching training

delivery methods to specific training objectives. Present-day training delivery methods are

limited to instructor-led classroom training, eLMS, and high fidelity full-task simulation.

Facilities need to have access to the variety of advanced training technologies that facilitate long-

term retention of information and are suited to hand-on learning and practice of practical

information, operational integration, procedures, and benefits. Current delivery methods are

human-resource intensive and need to be updated to incorporate advancements in technology that

enable more stand-alone, always available, standardized, cost-effective training.

NATPRO

The FAA should consider reinstating the National Air Traffic Professionalism

(NATPRO) project at a national level. NATPRO is a program to sharpen and maintain

controllers’ mental skills. It is an example of how information identified by operational analysis

can be turned into strategy and skill enhancement. Rather than relying solely on knowledge-

based training, the approach integrates the concept of “performance coaching,” using an

awareness seminar coupled with a practicum.

Although the skills are generic mental skills, by improving general skills, it was expected

that practice to strengthen individual performance would result in a corresponding improvement

in controllers’ professional performance. The approach was found to be cost-effective, and a

personally rewarding method of training to sustain a highly skilled work force.

90

The first NATPRO program focused on cognitive skills related primarily to visual

attention, such as detection of information, focusing on relevant information and multitasking.

The practicum included activities to exercise the mind and improve concentration through

distributed practice. Interactive web-based computerized skill challenges permitted participants

to gain insight about their own skills. By testing themselves against the computer and

experiencing how performance can vary in relation to factors such as distraction, fatigue,

boredom, and so on, participants gain increased understanding of their own performance and

identified strategies to improve it. The internet-based application gives individuals and teams the

opportunity to compete for high scores, should they desire to do so. Although the competitive

aspect was available to participants, it was not required. It was included to enhance the

experience for those participants who want to engage in interpersonal competition. 1

1 NATPRO Public Affairs Document, November 2003.

Tower Simulation / UFA

Desktop or portable tower air traffic simulators should be made available at every FAA

tower facility. FAA Industry partner UFA is in a position to develop and supply the needed

training equipment.

UFA is one of the world’s leading developers of Air Traffic Control simulation systems,

providing controller training and research tools to air navigation service providers, military

organizations, universities, and airports.

The following recommended products and solutions are based on the COE Project 6

survey, interview, and quantitative findings. These findings show that training needs to be

tailored to the site and to the individuals. Use of smaller part task trainers that can leverage both

91

group or individual training curriculums. Small training devices reduce staffing requirements

and are still capable of achieving similar fidelity to a larger training device but at much lower

cost to the user.

UFA’s ATTower product suite provides a flexible and expandable solution for ATC

Apron, Radar, Ground, and Tower Training. The ATTower software provides a realistic model

of aircraft and vehicle movement and supports training of the Apron, Ground, Tower, Radar, and

Flight Data Positions. The software accomplishes this with commercial off-the-shelf hardware

such as PC, laptop, or tablet devices and uses Windows as the operating system to provide a

user-friendly interface.

ATTower can be used entirely with ATVoice, our voice recognition and response (VRR)

capability. The use of VRR reduces the number of staff needed to provide training and

reinforces proper phraseology and is available with all UFA products.

ATTower has a fully integrated Exercise Development environment that allows a user to create,

modify, and manage airport information, airspace information, maps, flight strips, and all

aspects of training exercises. ATEditor presents information in a modern graphical user

interface that is intuitive to the user. This allows for creation and maintaining of individual site

data and exercises to create a unique experience for controller training.

UFA’s ATCloud infrastructure is a complete part-task training solution optimized for the

Internet or Enterprise Intranet and allows organizations to maximize their training impact by

giving users access to simulation exercises outside the classroom/lab or normal training

environment. ATCloud offers high fidelity simulation capabilities in small-scale, available

devices such as tablets and laptops. This system allows for modernization of normal training

processes and does not require multiple personnel to operate. The system can be hosted on

92

internal networks or over Amazon Web Services. The use of voice recognition software prevents

the need for pseudo pilots. Automatic student assessments and time recording metrics prevent the

need for instructors to monitor live usage of the system. ATCloud consists of two main products

ATSpeak Mobile Edition, which is a Phraseology and Part-Task Trainer, and ATCoach Mobile

Edition, which is a Mobile Radar Simulator.

ATSpeak ME is the ATCloud based voice recognition enabled skills and phraseology

training application. ATSpeak ME can present information, videos, fill-in-the-blank, multiple

choice, and several variations of voice recognition enabled slides to teach and test users on ATC

Operations. An ATSpeak Editor is available that allows for the creation of content and

exercises, such as specific emergency or procedural situations that require unique coordination.

This allows for tailored training for local facilities and phraseology practice outside of normal

operations.

ATCoach ME is the ATCloud based mobile radar simulator based on UFA’s High-

Fidelity Standalone and Embedded Radar simulator product ATCoach. This application has the

ability to create site-specific databases that replicate exact airspace, giving control to smaller

sites to apply more suitable training. The ATPrepare database preparation tool allows users to

maintain and update their own environments at each facility. Users are able to define emergency

and unusual situations to cover all needs of initial or recurring CPC training. Automatic voice

and performance-based statistics are collected and presented in forms of student assessments

that both students and instructors can access through the ATCloud interface.

UFA’s voice recognition enabled products ATTower, ATCloud, ATCoach ME and

ATSpeak ME allow for high fidelity simulation in a cost effective, mobile platform. These

products allow for site specific training to be created and performed by local instructors. These

93

products can be used on mobile devices and laptops to allows for ease of use and cost reduction.

Controller assessment reports integrated into each of the products will hold controllers

responsible for the training and will likely create a more engaged training audience.

Scanning / TransLumen

The follow-on work from this project has many outcomes that would lead to addressing

the challenges for key areas advancing training highlighted in this document. While there are

differences of opinion from the participants for advancing training, dependent upon experience

and personal viewpoints, the feedback does clearly convey many approaches for improvements.

From the TransLumen perspective and the company’s expertise, the following subject areas

would represent follow-on project areas that would add value to future training as the NAS

environment and ATC demographics evolve.

STEGC (Subthreshold Extreme Gradual Changes)

Scanning has been identified as a skill that is critical in Radar and Tower functions.

Training developmentals on scanning methods has benefits throughout their career even as they

adopt their own methods to handle tasks. As an overview of continued work in this area, the

project team would use its visualization, human factors and ATC subject matter expertise to

develop visualization patterns for general scanning that will align with optimized scanning

techniques. The proposed project will develop scanning videos and apps creating interactive

materials to train and test developmentals. Augmenting the expertise and the productions,

relevant research will be reviewed and potentially utilized in the content. Baseline scenarios will

be designed to capture a player’s activities, accuracies and times to formulate scores and if

desired metrics to quantify performance.

94

STEGC on-screen imagery (video or embedded) based upon the initial tasks will be used

to support end user interactive functions over time. The ATC tasks will establish the rules and

decisions to be used for each scenario and develop user testing methodologies. Once established

the optimal requirements for imagery will be implemented. STEGC will be applied to some of

the video content criteria in predefined areas of the environment to better create effects on screen

leading to analysis for the most relevancy in testing. Ultimately the rate of change detection will

be established by applying STEGC algorithms to define ranges for ATC platforms. Simulations

test scenarios will be defined using STEGC technology for image recognition and data transfer

criteria and processed to generate video/animation scene blocks. To achieve this end, the

research team will build upon and extend an existing computational theory and framework for

understanding human visual and spatial working memory and cognitive task load. The research

team will implement a software application suitable for mobile as well as desktop distribution.

ATC Training Commonalities Between Diverse and Remote Facilities

As an approach to addressing training differentials between centralized centers, diverse

and remote facilities, a project study could identify both common core and unique methods for

translation into relevant modules for interactive training. The targeting of common knowledge

and skills with common tasks would mitigate the absence of lessons that get commingled with

subjects that are not as applicable to different facilities.

This analysis could include developing a track for using “transference” through common

imagery and techniques to extend applicability from one dissimilar facility and make it relatable

to others.

95

Laying Groundwork for Training ATC for UAS in UTM areas

While some of the feedback in the report has Controllers satisfied with the training for

handling UAVs in their airspace, the fact is that the criteria for managing UAV intrusions for all

NAS traffic is still being developed by NASA and the FAA. To stay ahead of the training and to

be proactive with common computer user interfaces, symbology and language, remote

UAS/USSS control centers require highly detailed development of communication, conflict, and

cybersecurity protocols. Other groups may be looking at these issues and a combined approach is

needed before procedures and standards are set and industry starts employing remote pilots,

which may or may not be certified. This project could present an opportunity to create a common

operating picture for all operations and become the genesis of ATC training for this emerging

sector.

In using TransLumen’s expertise and the subject matter experts from the team as well as

the FAA Human Factors areas, we can start producing visualization studies that will translate

into UAS command and control HCI design. UAS control centers will certainly need to have

tools to aid in interpreting data intensive, hi-stress operations, by offloading the monitoring of

various levels of criticality for lower level or less pressing tasks while enabling the pilot to focus

on more time critical situations (Richards & Siefken, 2007). Planning and implementing

intelligent visual interfaces is key to facilitating the human in the loop or on the loop in making

decisions and supervising secondary events through augmented visualization and performance

capability. These studies can set the stage for implementations for new interfaces and subsequent

training (Siefken, 2006).

96

A New Occupation - Requirements for UAS training within NAS for Commercial

Businesses

With TransLumen’s experience with serious gamification development for the FAA ATC,

a similar path can be taken for UAS and USS regional centers. Also, new schools are emerging

to teach the UAS operations. The leading schools such as the Embry-Riddle Aeronautical

University will have the students and trainers available to start the process of educating the

mainstream industries that will be using the UAS command and control in the NAS. Such

controls will be manned by a diversity of individuals with varying capabilities to manage UAVs

as it interfaces with the FAA. What is clear is that these individuals will be hired by commercial

companies, in market areas such as distribution, utilities, security and many others. Currently

there are certifications available and schools that train military and FAA personnel, but as UAV

remote pilots interface with regional centers across different geographic areas converge with

manned airspace, we cannot be complacent. Commercial business need to understand how to

assess their technical human talent, understand the current constraints of autonomous systems

and rigorously train personnel as this new occupational job market emerges. Is it too soon? No. It

is never too soon to be proactive even as standards are being developed.

97

REFERENCES Annett, J. (1979). Memory for skill. Applied problems in memory. London, England:

M.M. Gruneberg & P.E. Morris

ATC Refresher Training Manual. Eurocontrol. (2015). Retrieved

from http://skybrary.aero/bookshelf/books/3024.pdf

Casner, S., Geven, R., Recker, M., Schooler, J. (2014). The retention of manual flying skills

in the automated cockpit. Human Factors, 56(8), 1506-1516. doi:

10.1177/0018720814535628

Childs, J., & Spears, W. (1986). Flight-Skill Decay and Recurrent Training. Perceptual

and Motor Skills, 62, 235-242. doi: 10.2466/pms.1986.62.1.235

Cuevas, R. (2009). Do your operators need refresher training? Material Handling

Management, 64(6), 45-46. Retrieved from http://search.proquest.com.ezproxy.

libproxy.db.erau.edu/docview/213365058? accountid=27203

Ebbatson, M., Harris, D., Huddlestone, J., & Sears, R. (2010). The relationship between manual

handling performance and recent flying experience in air transport pilots. Ergonomics,

53(2), 268-277. doi: 1080/00140130903342349

Farr, M. J. (1987). The long-term retention of knowledge and skill: A cognitive and

instructional perspective. New York: Springer-Verlag.

Gagne, M., & Deci, E. (2005). Self-determination theory and work motivation. Journal

of organizational behavior, 26, 331-362. doi: 10.1002/job.322

Gardin, R., & Sittereley, E. (1972). Degradation of learned skills. A review and annotated

bibliography. NASA Scientific and Technical Information Program, 1-123. Retrieved

from https: //ntrs.nasa.gov/search.jsp?R=19730001425

98

Grieser, S. (2001). Recurrent human factors training for air traffic controllers in developing

generative safety culture. Focusing Attention on Aviation Safety. Retrieved from

https://scholar. google.com/scholar?

hl=en&q=NATCA+training&btnG=&as_sdt=1%2C10&as_sdtp=

Healy, A. (2013). Enhancement of skill learning. Encyclopedia of the Mind. Sage

Publication: Thousand Oaks, CA. 691-694.

Hendrickson, S., Goldsmith, T., & Johnson, P. (2006). Retention of airline pilots' knowledge

and skill. Proceedings of the Human Factors and Ergonomics Society.

Jonides, J., & Nee, D. E. (2006). Brain mechanisms of proactive interference in working

memory. Neuroscience, 139(1), 181-193. doi: 10.1016/j.neuroscience.2005.06.042

Kluge, A., & Frank, B. (2014). Counteracting skill decay: Four refresher interventions and

their effect on skill and knowledge retention in a simulated process control task.

Ergonomics, 57(2), 17-190. doi: 10.1080/00140139.2013.869357

Kluge, A., & Frank, B. (2016). Being prepared for the infrequent: A comparative study of two

refresher training approaches and their effects on temporal and adaptive transfer in a

process control task. Proceedings of the Human Factor and Ergonomics Society,

56(1). doi: 10.1177/1071181312561496

Lee, M., Cheung, C., & Chen, Z. (2005). Acceptance of internet-based learning medium: The

role of extrinsic and intrinsic motivation. Information & Management, 42(8), 1095-

1104. doi: 10.1016/j.im.2003.10.007

Mahony, P., Griffiths, R., Larsen, P., Powell, D. (2008). Retention of knowledge and skills in

first aid and resuscitation by airline cabin crew. Resuscitation, 76(3), 413-418. doi:

10.1016/j. resuscitation.2007.08.017

99

Malakis, S., & Kontogiannis, T. (2012). Refresher training for air traffic controllers: Is it

adequate to meet the challenges of emergencies and abnormal situations. The

International Journal of Aviation Psychology, 22(1), 59-77. doi:

10.1080/10508414. 2012.635127

O'Hara, J. (2007). The retention skills acquired through simulator- based training. Ergonomics,

33(9). doi: 10.1080 /00140139008925319

Philip, K., & Kaiser, M. (1994). Perceptual learning modules in flight training. Proceedings of

the Human Factor and Ergonomics Society, 38(18). doi:

10.1177/154193129403801808

Pounds, J. (2003). FAA strategies for reducing operational error causal factors. US

Department of Transportation. Federal Aviation Administration. Report No

DOT/FAA/AM-03/19

Richards, D., & Siefken, D. (2007). Augmented Symbology: A New Approach to Data

Visualization. Foundations of Augmented Cognition 4th Edition (ACI 2007) Retrieved

from http://www.translumen.net/images/papers/services

Roediger, H. L., & Karpicke, J. D. (2006). The power of testing memory. Basic research and

implications for educational practice. Perspective on Psychological Science, 1(3), 181-

210. doi: 10.1111/j.1745-6916.2006.00012.x

Ryan, R. & Deci, E. (2000). Intrinsic and extrinsic motivations: Classic definitions and new

directions. Contemporary Educational Psychology, 25, 54-67. doi:

10.1006/ceps.1999.1020

Saunders, C. (2005). Air Traffic Technical Training. Federal Aviation Administration Report

3120.4L

100

Sauer, J., Hockey, G., & Wastell, G. (2000). Effects of training on short - and long-term skill

retention in a complex multiple task environment. Ergonomics, 43(12), 2043-2064.

doi: 10.1080 /00140130010000893

Schultheis, S. (2014). Integrating advanced technology into air traffic controller training.

MITRE Corporation. 14th Aviation Technology, Integration, and Operations

Conference. Atlanta, GA.

Schultheis, S., Winokur, D., & Hutson, K. (2014). Improving Air Traffic Controller Training

Today and in the Future: Summary of MITRE Research and Recommendations.

MITRE. Technical Report No 140110. 1-26.

Siefken, D. (2006). Subthreshold Data Cognition: Adaptive “Mitigation” Strategy, Foundations

of Augmented Cognition 2nd Edition (ACI 2006). Retrieved from

http://www.translumen.net/images/papers/services

Sitterley, T., & Berge, W. (1972). Degradation of learned skills. Effectiveness of practice

methods on simulated space flight skill retention. Report No 19730001426. Retrieved

from https://ntrs. nasa.gov/search.jsp?R=19730001426

Smith-Jentsch, K., Kraiger, K., Cannon-Bowers, J., & Salas, E. (1998). A data driven model of

precursors to teamwork. Paper presented at the 13th Annual Conference of the Society of

Industrial and Organizational Psychology. Dallas, TX.

Spear, N. (2014). The processing of memories. New York, New York: Psychology

Press. Starter, N. B., Woods, D. D., & Billings, C. E. (1997). Automation surprises.

Handbook of

Human Factors. New York, New York: Wiley. Szpunar, K. K., McDermott, K. B., & Roediger, H. L. (2008). Testing during study insulates

101

against the buildup of proactive interference. Journal of Experimental Psychology,

34(6), 1392-1399. doi: 10.1037/a0013082

U.S. Department of Transportation. (2016). A Plan for the Future: 10 Year Strategy for the Air

Traffic Control Workforce. Federal Aviation Administration. 1-60.

Whelan, T., Shi, X., Yorke, S., Andony, K., & Mckenzie, M. L. (2016). Knowledge and skills

enhancement through perioperative nursing simulation lab training. Ornac Journal,

34(2), 13-19,21-25,28-30. Retrieved from http://search.proquest.

com.ezproxy.libproxy.db.erau.edu/docview/1802562372? accountid=27203

Wickens, C., Mavor, A., & McGee, J. (1997). Flight to the Future: Human Factors in Air

Traffic Control. National Academy Press: Washington, D.C. 1997. Retrieved from

https://www.nap.edu/read/5493/chapter/1

Wickens, C., Mavor, A., Parasuraman, R., & McGee, J. (1998). The future of air traffic

control: Human operators and automation. Washington, D.C.: National Academy

Press.

Winfred, A., Bennett, W., Stanush, P., & McNelly, T. (1998). Factors that influence skill decay

and retention: A quantitative review and analysis. Human Performance, 11(1), 57-101.

Retrieved from

https://www.researchgate.net/profile/Winfred_Arthur_Jr/publication/232904872_Factor

s_That_Influence_Skill_Decay_and_Retention_A_Quantitative_Review_and_Analysis/l

inks/00463524 b1893591d0000000.pdf

Winokur, D., & Sacher, B. (2015). Strategy for Certified Professional Controller (CPC)

Post- Certification Training. MITRE Technical Report 150304

Worden, A., Sheridan, T., Winokur, D., Sacher, B., & Hutson, K. (2016). Certified Professional

102

Controller (CPC) Post-Certification Training: Workflow Analysis and Targeted

Recommendations. MITRE Corporation. Technical Report N

103

APPENDIX A

Interview sites for SOAR Project 6 – Enhanced AT-CPC Training Texas Dallas Ft. Worth Tower (DFW) October 28 Dallas Ft. Worth TRACON (D10) October 28 Florida Tampa Tower/TRACON (TPA) December 1 Orlando Sanford Tower (SFB) December 1 Orlando International Tower (MCO) November 8 Central Florida TRACON (F11) November 8 Miami Tower/TRACON (MIA) November 17 Miami Center (ZMA) November 17 Jacksonville Center (ZJX) November 1 North Dakota Fargo Tower/TRACON (FAR) October 5 Grand Forks Tower (GFK) October 12 Bismarck Tower/TRACON (BIS) October 19 Minnesota Crystal Tower (MIC) September 12 to 14 Flying Cloud Tower (FCM) September 12 to 14 St. Paul Tower (STP) September 12 to 14 Minneapolis Tower (MSP) September 12 to 14 Minneapolis TRACON (M98) September 12 to 14 Minneapolis Center (ZMP) September 12 to 14 Michigan Detroit Tower (DTW) August 16 Detroit TRACON (D21) August 16 Flint Tower/TRACON (FNT) August 15 Lansing Tower/TRACON (LAN) August 15 Pontiac Tower (PTK) August 15 West Ontario Tower (ONT) November 15 Southern CA TRACON (SCT) November 15 FAA Academy – Oklahoma City August 23/34

104

APPENDIX B

Overview of Results of the CPC Survey

Table 1. All Data - Overview of the results of the CPC Survey

All Respon- dents

Levels 5-9

Levels 10-12

< 10 Years’ Experience

10+ Years’ Experience

ARTCC ATCT TRACON

1. The Proficiency Training, I receive as a CPC is overall complete and applicable.

0.05 0.12 -0.05 0.54 -0.22 -0.26 0.33 0.38

2. The Recurrent Training, I receive on relevant and timely safety topics is complete and applicable.

0.14 0.28 0.07 0.40 0.03 -0.17 0.25 0.62

3. The Recurrent Training, I receive on safety alerts and traffic advisories is complete and applicable.

0.40 0.07 0.49 0.60 0.33 0.36 0.50 0.52

4. The Recurrent Training, I receive on Minimum Safe Altitude Warning (MSAW) procedures is complete and applicable.

-0.08 -0.38 0.02 0.12 -0.16 -0.21 -0.17 0.24

5. The Recurrent Training, I receive on the relationship between charted minimum altitudes and underlying topography is complete and applicable.

-0.43 -0.49 -0.45 0.00 -0.68 -0.76 0.00 -0.14

6. The Recurrent Training, I receive on Weather and other conditions that affect flight including: icing, thunderstorms, wind shear and VFR

0.41 0.12 0.48 0.44 0.43 0.38 0.50 0.48

105

aircraft that encounter IFR conditions is complete and applicable.

7. The Recurrent Training, I receive on Bird activity information and dissemination is complete and applicable.

0.24 0.17 0.27 0.21 0.30 0.10 0.67 0.25

8. The Recurrent Training, I receive on Recovery in ATC Operations is complete and applicable.

-0.25 -0.60 -0.14 -0.04 -0.35 -0.28 -0.67 0.10

9. The Recurrent Training, I receive on Fatigue awareness is complete and applicable.

0.22 0.01 0.24 0.04 0.41 0.03 0.58 0.38

10. The Refresher Training, I receive on Unusual situation is complete and applicable.

-0.08 -0.06 -0.07 0.36 -0.32 -0.21 -0.50 0.43

11. The Refresher Training, I receive on Lost aircraft orientation is complete and applicable.

-0.17 -0.76 0.00 -0.04 -0.22 -0.14 -0.75 0.19

12. The Refresher Training, I receive on Aviation Security Procedures including: interceptor procedures and communications, Domestic Events Network (DEN) notification requirements, hijacking, and airspace intruders is complete and applicable.

-0.10 -0.76 0.10 -0.08 -0.05 -0.24 -0.25 0.29

106

13. The Refresher Training, I receive on Emergency Situations is based on real-life incidents and aircraft accidents and stresses a lessons-learned approach.

-0.16 -0.27 -0.17 0.36 -0.46 -0.48 0.25 0.14

14. The Refresher Training, I receive on Seldom-used procedures, such as transitioning to and applying non-radar separation and procedures for special flight handling is complete and applicable.

-0.67 -0.77 -0.64 -0.44 -0.78 -0.83 -0.67 -0.38

15. The Refresher Training, I receive on Air Traffic Organization Operational Contingency Plan is complete and applicable.

-0.67 -0.60 -0.79 -0.56 -0.70 -0.97 -0.17 -0.48

16. The Supplemental Training, I receive on new / revised procedures, regulations, or equipment is complete and applicable.

-0.34 -0.60 -0.27 -0.21 -0.38 -0.54 -0.50 0.10

17. The Recurrent Training, I receive on Runway Safety is complete and applicable.

0.64 0.69 0.43 1.00 0.25 0.22 1.00 0.62

18. The Refresher Training, I receive on Facility procedures for Go Arounds and Missed Approaches is complete and applicable.

0.05 0.00 0.04 0.22 -0.14 -0.19 0.27 0.12

107

19. The Refresher Training, I receive on Local airport de-icing plans (LADP) is complete and applicable.

-0.22 -0.08 -0.54 0.46 -0.86 -0.56 0.38 -0.40

20. The Refresher Training, I receive as a weather observer on ASOS or AWSS and LAWRS is complete and applicable.

-0.10 -0.24 -0.07 0.21 -0.40 -0.20 0.00 -0.09

21. The Refresher Training, I receive on Procedures and responsibilities for special use airspace (SUA) / special Activity airspace (SAA) is complete and applicable.

-0.10 -0.43 0.06 0.11 -0.16 -0.04 -0.71 0.20

22. The Refresher Training, I receive on Primary Backup Mode is complete and applicable.

-0.46 -0.31 -0.58 -0.20 -0.59 -0.78 -0.30 0.00

23. The Refresher Training, I receive on En Route Decision Support Tool (EDST) is complete and applicable.

-0.15 -0.12 -0.17 0.20 -0.38 -0.19 -0.20 0.00

24. Rate the overall CONTENT of the proficiency training you receive from 1 to 5.

2.92 2.95 2.86 3.16 2.81 2.79 3.00 3.14

25. Rate the DELIVERY METHOD used for proficiency training from 1 to 5

2.60 2.45 2.64 2.88 2.46 2.55 2.33 2.90

0.15 0.01 0.15 0.39 0.01 -0.02 0.21 0.38

108

Table 2. All Respondents - presents the data as a function of all respondents

All Respondents

1. The Proficiency Training, I receive as a CPC is overall complete and applicable.

0.05

2. The Recurrent Training, I receive on relevant and timely safety topics is complete and applicable.

0.14

3. The Recurrent Training, I receive on safety alerts and traffic advisories is complete and applicable.

0.40

4. The Recurrent Training, I receive on Minimum Safe Altitude Warning (MSAW) procedures is complete and applicable.

-0.08

5. The Recurrent Training, I receive on the relationship between charted minimum altitudes and underlying topography is complete and applicable.

-0.43

6. The Recurrent Training, I receive on Weather and other conditions that affect flight including: icing, thunderstorms, wind shear and VFR aircraft that encounter IFR conditions is complete and applicable.

0.41

7. The Recurrent Training, I receive on Bird activity information and dissemination is complete and applicable.

0.24

8. The Recurrent Training, I receive on Recovery in ATC Operations is complete and applicable.

-0.25

9. The Recurrent Training, I receive on Fatigue awareness is complete and applicable.

0.22

10. The Refresher Training, I receive on Unusual situation is complete and applicable.

-0.08

11. The Refresher Training, I receive on Lost aircraft orientation is complete and applicable.

-0.17

12. The Refresher Training, I receive on Aviation Security Procedures including: interceptor procedures and communications, Domestic Events Network (DEN) notification requirements, hijacking, and airspace intruders is complete and applicable.

-0.10

13. The Refresher Training, I receive on Emergency Situations is based on real-life incidents and aircraft accidents and stresses a lessons-learned approach.

-0.16

109

14. The Refresher Training, I receive on Seldom-used procedures, such as transitioning to and applying non-radar separation and procedures for special flight handling is complete and applicable.

-0.67

15. The Refresher Training, I receive on Air Traffic Organization Operational Contingency Plan is complete and applicable.

-0.67

16. The Supplemental Training, I receive on new / revised procedures, regulations, or equipment is complete and applicable.

-0.34

17. The Recurrent Training, I receive on Runway Safety is complete and applicable.

0.64

18. The Refresher Training, I receive on Facility procedures for Go Arounds and Missed Approaches is complete and applicable.

0.05

19. The Refresher Training, I receive on Local airport de-icing plans (LADP) is complete and applicable.

-0.22

20. The Refresher Training, I receive as a weather observer on ASOS or AWSS and LAWRS is complete and applicable.

-0.10

21. The Refresher Training, I receive on Procedures and responsibilities for special use airspace (SUA) / special Activity airspace (SAA) is complete and applicable.

-0.10

22. The Refresher Training, I receive on Primary Backup Mode is complete and applicable.

-0.46

23. The Refresher Training, I receive on En Route Decision Support Tool (EDST) is complete and applicable.

-0.15

24. Rate the overall CONTENT of the proficiency training you receive from 1 to 5.

2.92

25. Rate the DELIVERY METHOD used for proficiency training from 1 to 5

2.60

0.15

110

Table 3. Facility Levels - Presents the data divided by facility levels in two categories

Levels 5-9 Levels 10-12

1. The Proficiency Training, I receive as a CPC is overall complete and applicable.

0.12 -0.05

2. The Recurrent Training, I receive on relevant and timely safety topics is complete and applicable.

0.28 0.07

3. The Recurrent Training, I receive on safety alerts and traffic advisories is complete and applicable.

0.07 0.49

4. The Recurrent Training, I receive on Minimum Safe Altitude Warning (MSAW) procedures is complete and applicable.

-0.38 0.02

5. The Recurrent Training, I receive on the relationship between charted minimum altitudes and underlying topography is complete and applicable.

-0.49 -0.45

6. The Recurrent Training, I receive on Weather and other conditions that affect flight including: icing, thunderstorms, wind shear and VFR aircraft that encounter IFR conditions is complete and applicable.

0.12 0.48

7. The Recurrent Training, I receive on Bird activity information and dissemination is complete and applicable.

0.17 0.27

8. The Recurrent Training, I receive on Recovery in ATC Operations is complete and applicable.

-0.60 -0.14

9. The Recurrent Training, I receive on Fatigue awareness is complete and applicable.

0.01 0.24

10. The Refresher Training, I receive on Unusual situation is complete and applicable.

-0.06 -0.07

11. The Refresher Training, I receive on Lost aircraft orientation is complete and applicable.

-0.76 0.00

12. The Refresher Training, I receive on Aviation Security Procedures including: interceptor procedures and communications, Domestic Events Network (DEN) notification requirements, hijacking, and airspace intruders is complete and applicable.

-0.76 0.10

13. The Refresher Training, I receive on Emergency Situations is based on real-life incidents and aircraft accidents and stresses a lessons-learned approach.

-0.27 -0.17

14. The Refresher Training, I receive on Seldom-used procedures, such as transitioning to and applying non-radar separation and procedures for special flight handling is complete and applicable.

-0.77 -0.64

111

15. The Refresher Training, I receive on Air Traffic Organization Operational Contingency Plan is complete and applicable.

-0.60 -0.79

16. The Supplemental Training, I receive on new / revised procedures, regulations, or equipment is complete and applicable.

-0.60 -0.27

17. The Recurrent Training, I receive on Runway Safety is complete and applicable.

0.69 0.43

18. The Refresher Training, I receive on Facility procedures for Go Arounds and Missed Approaches is complete and applicable.

0.00 0.04

19. The Refresher Training, I receive on Local airport de-icing plans (LADP) is complete and applicable.

-0.08 -0.54

20. The Refresher Training, I receive as a weather observer on ASOS or AWSS and LAWRS is complete and applicable.

-0.24 -0.07

21. The Refresher Training, I receive on Procedures and responsibilities for special use airspace (SUA) / special Activity airspace (SAA) is complete and applicable.

-0.43 0.06

22. The Refresher Training, I receive on Primary Backup Mode is complete and applicable.

-0.31 -0.58

23. The Refresher Training, I receive on En Route Decision Support Tool (EDST) is complete and applicable.

-0.12 -0.17

24. Rate the overall CONTENT of the proficiency training you receive from 1 to 5.

2.95 2.86

25. Rate the DELIVERY METHOD used for proficiency training from 1 to 5

2.45 2.64

0.01 0.15

112

Table 4. Years of Experience - Presents data in two categories by years of experience

< 10 Years’ Experience

10+ Years’ Experience

1. The Proficiency Training, I receive as a CPC is overall complete and applicable.

0.54 -0.22

2. The Recurrent Training, I receive on relevant and timely safety topics is complete and applicable.

0.40 0.03

3. The Recurrent Training, I receive on safety alerts and traffic advisories is complete and applicable.

0.60 0.33

4. The Recurrent Training, I receive on Minimum Safe Altitude Warning (MSAW) procedures is complete and applicable.

0.12 -0.16

5. The Recurrent Training, I receive on the relationship between charted minimum altitudes and underlying topography is complete and applicable.

0.00 -0.68

6. The Recurrent Training, I receive on Weather and other conditions that affect flight including: icing, thunderstorms, wind shear and VFR aircraft that encounter IFR conditions is complete and applicable.

0.44 0.43

7. The Recurrent Training, I receive on Bird activity information and dissemination is complete and applicable.

0.21 0.30

8. The Recurrent Training, I receive on Recovery in ATC Operations is complete and applicable.

-0.04 -0.35

9. The Recurrent Training, I receive on Fatigue awareness is complete and applicable.

0.04 0.41

10. The Refresher Training, I receive on Unusual situation is complete and applicable.

0.36 -0.32

11. The Refresher Training, I receive on Lost aircraft orientation is complete and applicable.

-0.04 -0.22

12. The Refresher Training, I receive on Aviation Security Procedures including: interceptor procedures and communications, Domestic Events Network (DEN) notification requirements, hijacking, and airspace intruders is complete and applicable.

-0.08 -0.05

13. The Refresher Training, I receive on Emergency Situations is based on real-life incidents and aircraft accidents and stresses a lessons-learned approach.

0.36 -0.46

113

14. The Refresher Training, I receive on Seldom-used procedures, such as transitioning to and applying non-radar separation and procedures for special flight handling is complete and applicable.

-0.44 -0.78

15. The Refresher Training, I receive on Air Traffic Organization Operational Contingency Plan is complete and applicable.

-0.56 -0.70

16. The Supplemental Training, I receive on new / revised procedures, regulations, or equipment is complete and applicable.

-0.21 -0.38

17. The Recurrent Training, I receive on Runway Safety is complete and applicable.

1.00 0.25

18. The Refresher Training, I receive on Facility procedures for Go Arounds and Missed Approaches is complete and applicable.

0.22 -0.14

19. The Refresher Training, I receive on Local airport de-icing plans (LADP) is complete and applicable.

0.46 -0.86

20. The Refresher Training, I receive as a weather observer on ASOS or AWSS and LAWRS is complete and applicable.

0.21 -0.40

21. The Refresher Training, I receive on Procedures and responsibilities for special use airspace (SUA) / special Activity airspace (SAA) is complete and applicable.

0.11 -0.16

22. The Refresher Training, I receive on Primary Backup Mode is complete and applicable.

-0.20 -0.59

23. The Refresher Training, I receive on En Route Decision Support Tool (EDST) is complete and applicable.

0.20 -0.38

24. Rate the overall CONTENT of the proficiency training you receive from 1 to 5.

3.16 2.81

25. Rate the DELIVERY METHOD used for proficiency training from 1 to 5

2.88 2.46

0.39 0.01

114

Table 5. Facility Types - Presents the data divided by facility types

ARTCC ATCT TRACON

1. The Proficiency Training, I receive as a CPC is overall complete and applicable.

-0.26 0.33 0.38

2. The Recurrent Training, I receive on relevant and timely safety topics is complete and applicable.

-0.17 0.25 0.62

3. The Recurrent Training, I receive on safety alerts and traffic advisories is complete and applicable.

0.36 0.50 0.52

4. The Recurrent Training, I receive on Minimum Safe Altitude Warning (MSAW) procedures is complete and applicable.

-0.21 -0.17 0.24

5. The Recurrent Training, I receive on the relationship between charted minimum altitudes and underlying topography is complete and applicable.

-0.76 0.00 -0.14

6. The Recurrent Training, I receive on Weather and other conditions that affect flight including: icing, thunderstorms, wind shear and VFR aircraft that encounter IFR conditions is complete and applicable.

0.38 0.50 0.48

7. The Recurrent Training, I receive on Bird activity information and dissemination is complete and applicable.

0.10 0.67 0.25

8. The Recurrent Training, I receive on Recovery in ATC Operations is complete and applicable.

-0.28 -0.67 0.10

9. The Recurrent Training, I receive on Fatigue awareness is complete and applicable.

0.03 0.58 0.38

10. The Refresher Training, I receive on Unusual situation is complete and applicable.

-0.21 -0.50 0.43

11. The Refresher Training, I receive on Lost aircraft orientation is complete and applicable.

-0.14 -0.75 0.19

12. The Refresher Training, I receive on Aviation Security Procedures including: interceptor procedures and communications, Domestic Events Network (DEN) notification requirements, hijacking, and airspace intruders is complete and applicable.

-0.24 -0.25 0.29

13. The Refresher Training, I receive on Emergency Situations is based on real-life incidents and aircraft accidents and stresses a lessons-learned approach.

-0.48 0.25 0.14

115

14. The Refresher Training, I receive on Seldom-used procedures, such as transitioning to and applying non-radar separation and procedures for special flight handling is complete and applicable.

-0.83 -0.67 -0.38

15. The Refresher Training, I receive on Air Traffic Organization Operational Contingency Plan is complete and applicable.

-0.97 -0.17 -0.48

16. The Supplemental Training, I receive on new / revised procedures, regulations, or equipment is complete and applicable.

-0.54 -0.50 0.10

17. The Recurrent Training, I receive on Runway Safety is complete and applicable.

0.22 1.00 0.62

18. The Refresher Training, I receive on Facility procedures for Go Arounds and Missed Approaches is complete and applicable.

-0.19 0.27 0.12

19. The Refresher Training, I receive on Local airport de-icing plans (LADP) is complete and applicable.

-0.56 0.38 -0.40

20. The Refresher Training, I receive as a weather observer on ASOS or AWSS and LAWRS is complete and applicable.

-0.20 0.00 -0.09

21. The Refresher Training, I receive on Procedures and responsibilities for special use airspace (SUA) / special Activity airspace (SAA) is complete and applicable.

-0.04 -0.71 0.20

22. The Refresher Training, I receive on Primary Backup Mode is complete and applicable.

-0.78 -0.30 0.00

23. The Refresher Training, I receive on En Route Decision Support Tool (EDST) is complete and applicable.

-0.19 -0.20 0.00

24. Rate the overall CONTENT of the proficiency training you receive from 1 to 5.

2.79 3.00 3.14

25. Rate the DELIVERY METHOD used for proficiency training from 1 to 5

2.55 2.33 2.90

-0.02 0.21 0.38