cognitive models of pilot categorization and prioritization of … · 2013-10-03 · national...

National Aeronautics and Space AdministrationLangley Research Center • Hampton, Virginia 23681-0001

NASA Technical Paper 3528

Cognitive Models of Pilot Categorization andPrioritization of Flight-Deck InformationJon E. JonssonMcDonnell Douglas Aerospace-West • Long Beach, California

Wendell R. RicksLangley Research Center • Hampton, Virginia

August 1995

Printed copies available from the following:

NASA Center for AeroSpace Information National Technical Information Service (NTIS)800 Elkridge Landing Road 5285 Port Royal RoadLinthicum Heights, MD 21090-2934 Springfield, VA 22161-2171(301) 621-0390 (703) 487-4650

The use of trademarks or names of manufacturers in this report is foraccurate reporting and does not constitute an official endorsement,either expressed or implied, of such products or manufacturers by theNational Aeronautics and Space Administration.

Available electronically at the following URL address: http://techreports.larc.nasa.gov/ltrs/ltrs.html

iii

Contents

Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

Background. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Categorization of Information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Prioritization of Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Psychometric Tasks and Analyses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Subjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Stimuli . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Procedure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

How Pilots Cognitively Process Flight-Deck Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8Individual Difference Scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9Hierarchical Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

How Pilots Prioritize Flight-Deck Information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15Multidimensional Preference Analysis of Set A Rankings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Generic condition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16Takeoff condition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Multidimensional Preference Analysis of Set B Rankings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18Generic condition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18Takeoff condition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Cognitive Processing of Information Among Pilots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20Individual Difference Scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21Multidimensional Preference Analyses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Cognitive Processes Across Different Contexts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .24Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Cognitive Processing of Flight-Deck Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Classification of Flight-Deck Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Relative Priority of Flight-Deck Information Categories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Pilot Cognition Across Contexts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Appendix A—Example Application of INDSCAL Dimensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

Appendix B—Categorization and Prioritization Questionnaire. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

Summary

Automated systems on commercial aircraft flightdecks have dramatically increased in number over thepast decade and pilots now regularly interact and sharetasks with these systems. Consequently, this increasedinteraction has led human factors researchers to focusmore attention on the pilot’s cognitive processing offlight-deck information and the pilot’s overall mentalmodel of the information flow that occurs on the flightdeck. The cognitive activities of categorization and prior-itization particularly interest researchers because theseactivities are important and pervasive in managing andprocessing flight-deck information.

The experiment reported herein investigated howpilots categorize and prioritize information typicallyavailable during flight. Fifty-two commercial airlinepilots participated in tasks that required them to providesimilarity ratings for pairs of flight-deck information andthen prioritize this information under two separate con-texts. Such results can be expressed with either spatial orclustering representations. For spatial representations,the dimensions may be considered as representing cogni-tive factors that individuals use to process the informa-tion. For clustering representations, the clusters intowhich information falls represent categories pilots havefor the data.

The results suggested three cognitive dimensionsthat pilots use in categorizing flight-deck information.These dimensions included

1. The flight function that the information is designedto support

2. The strategic or tactical nature of the information

3. The frequency of information referral

The results also suggested four specific high-levelcategories that pilots use in categorizing flight-deckinformation. These information categories included

1. Aviation

2. Navigation

3. Communication

4. Systems administration

Although these high-level categories are sufficientfor classifying the majority of flight-deck information, itwas found that additional differentiation can be made inthe aviation and the navigation categories.

The four information categories were also prominentwhen analyzing how the pilots prioritized flight-deckinformation. Along the primary dimensions in whichpilots prioritized information, elements seemed to cluster

according to the four high-level functions they support.Finally, when asked to prioritize this flight-deck infor-mation, it was discovered that there was both a highdegree of consensus for the importance of the same infor-mation across different flight situations and few individ-ual differences among pilots in their prioritizations.

Introduction

The increase of automation on modern commercialaircraft has made it more challenging for flight-deckdesigners to support the flight crew for two principalreasons.

First, automation has shifted the emphasis on flightcrew tasks from physical actions to the cognitive pro-cesses required to accomplish the tasks (Dornheim 1992;Ricks, Jonsson, and Rogers 1994; Ricks et al. 1991).Two examples are representative of this shift. A trans-continental flight can be completely programmed at thedeparture gate with the flight management computerrather than having the crew make sequential changeswith the mode control panel as the flight progresses, ornow almost never done, handflying the aircraft for theentire flight. On the McDonnell Douglas MD-11, recon-figuration occurs for some failure states without any pilotintervention.

Pilot performance in both of these examples involvesmonitoring the systems and determining whether they arefunctioning correctly. Because of these technologicaladvances, designers need to consider issues such as howthe pilot will monitor the flight progress, the crew’sunderstanding of the flight management system, theinformation required to support the pilots, and the cogni-tive processing and mental models of the informationflow of the pilots. The consideration of pilot cognitionhas thus taken on an added degree of importance in thedesign of modern flight decks. The development of validand reliable methods for addressing these issues willtherefore become an important component from thedesigner’s perspective.

Second, designers are challenged because of thedramatic increase in the information that is or will be pre-sented to pilots (Braune, Hofer, and Dresel 1991; Ricks,Jonsson, and Rogers 1994; Ricks et al. 1991). Part of thischallenge will be to include new sources of informationthat are designed to safely increase traffic flow (such asdata links and global positioning systems), while anotherpart will be to include technological advances in the wayinformation can be stored and displayed to the crew.

The glass cockpit has revolutionized display technol-ogy by allowing the presentation of multiple types ofdata on the same display. This presentation contrastssharply with earlier flight decks where electromechanical

2

displays were dedicated to the presentation of one type ofaircraft information. Proposed information sources suchas an electronic library and an onboard maintenance sys-tems may provide the crew with yet more data. Deter-mining what information to provide and when to provideit for a variety of tasks and functions will becomeincreasingly important.

In addition to the trends previously mentioned, thevery fact that pilots share flight-deck functions with auto-mation adds to the complexity of analyzing crew tasks.While automation frees the crew from many routinetasks, it implicitly adds another requirement becauseshared functions require the crew to have some under-standing of how the automation works, what it is doing,and when it is doing it. Additionally, the crew must beable to ascertain when the automation may be incorrectlyperforming a given task and must be prepared to takeover should the automation fail. In fact, recent researchin this area suggests that, when evaluating performanceeffects of automation on the human operator, issues suchas situation awareness and the pilot’s mental model ofthe automation need to be considered (Sarter and Woods1994, Wiener 1989). Thus, the pilot’s interaction withflight-deck automation implicitly adds another level ofinformation by requiring the pilot to have both anawareness and an understanding of how the automationoperates.

These concerns, frequently referred to as flight-deckinformation management issues, suggest greater atten-tion will need to be given to the pilot’s processing anduse of information. In the development of earlier flightdecks, human factors engineers principally concernedthemselves with form and fit issues (such as whether acontrol was reachable, a display was readable, or theamount of strength required to actuate a control). Thesedevelopments suggest that more attention will need to begiven to the information flow between the pilot and theflight-deck systems in conjunction with the cognitiveprocesses pilots use to process flight-deck information.

While most of the previous human factors researchconcerned features that were directly observable andamenable to traditional measurement techniques, cogni-tive factors lack many of these features. For this reason,rigorous, empirical methods for determining the cogni-tive processes of the flight crew are required. Theresearch program presented herein represents a first steptoward the development of measurement techniques thatcould be used as part of an early flight-deck design pro-cess for assessing the cognitive task loads that are associ-ated with information processing. The availability ofsuch measurement techniques should allow for flight-deck designs that increase the pilot’s situational aware-

ness and lead to an appropriate cognitive workload,thereby resulting in a more effective flight deck.

The project reported herein addresses four principalresearch issues. First is an empirical investigation of howpilots cognitively process flight-deck information. Ofparticular interest is the empirical identification of bothconscious and unconscious cognitive processes. Theseresults will help determine the mental representationpilots have for flight-deck information, which is the ini-tial step for developing cognitive measures of crew per-formance. Such measures would allow researchers toexamine how pilots process the information they use toperform tasks and, if valid measurement techniques canbe developed, to determine whether the crew is subject toover or underloading with respect to their cognitive pro-cessing abilities.

The second research issue concerns how pilots prior-itize flight-deck information. The objectives of this issueare to determine whether a common underlying strategyexists that pilots use to prioritize flight-deck informationand whether a discernible classification scheme exists,that will allow the development of a prioritization systemby category.

A third research issue is the consistency of cognitiveprocessing among pilots. That is, are pilots using similarcognitive processes for managing flight-deck informa-tion. This issue is critical. If significant differences in theway pilots process and categorize information exist, itwould be problematic to extend these findings to otherareas.

The fourth and final research issue examineswhether the cognitive processes used for flight-deckinformation are invariant across different contexts. Thisquestion is concerned with whether the cognitive pro-cesses pilots use change as a function of flight context.

Abbreviations

ANOVA analysis of variance

ATC air traffic control

ATIS Automatic Terminal Information Service

CDU control and display unit

CRT cathode ray tube

DME distance measuring equipment

EPR engine pressure ratio

FMS flight management system

F.O. first officer

ILS instrument landing system

INDSCAL individual differences scaling

3

IRS inertial reference system

L/R left/right

MAU multiattribute utility

MCDU multifunction control and display unit

MDPREF multidimensional preference scaling

MDS multidimensional scaling

ND navigational display

PCPREF personal computer program for MDPREF

PFD primary flight display

SAS statistical analysis system

TCAS traffic alert and collision avoidance system

VHF very high frequency

VOR VHF omnidirectional range

Background

The concerns presented in the Introduction, fre-quently referred to as flight-deck information manage-ment issues, suggest that greater attention will need to begiven to how pilots use and process information. In sodoing, researchers should study cognitive processes rou-tinely engaged in by flight crews. The categorization andprioritization of flight-deck information represent twosuch processes. Because of their ubiquitous nature, itseems appropriate to establish, in an empirical fashion,how pilots categorize flight-deck information and howthey judge the relative importance of that information. Itwould be useful to review briefly both areas from theo-retical and applied perspectives.

Categorization of Information

Humans are able to process and use large amounts ofinformation that are presented in everyday life becausethey can organize the information by relating it to priorexperiences (thereby making it familiar). This organiza-tional process draws heavily on an individual’s long-termmemory and is usually referred to ascategorization.Such grouping considerably simplifies the individual’sprocessing of new information, not only by providing itwith a slot in which to reside, but also by providing somegeneral characteristics of the data once it has beenplaced. This process, along with its benefits, has beennicely summarized by Glass and Holyoak (1986):

Categorization is a fundamental cognitive pro-cess because every experience is in some senseunique. For example, no two apples are entirelyalike. However, if each experience were given aunique mental representation, we would bequickly overwhelmed by the sheer complexity,and we could not apply what we had already

learned to deal with new situations. . . . A cate-gory system allows us to derive further informa-tion about an object that has been assigned to acategory. For example, if you have categorizedsome object as an apple, you can infer how it isexpected to taste, that it has a core, and that itcan be used to fill a pie (p. 149).

Thus, categorization allows individuals to infer agreat deal about a new source of information, because itsuccinctly and implicitly describes characteristics of theinformation. Theories of how individuals categorizeinformation are matters of great debate in the cognitivescience community. Currently, there are two principaltheories of categorization, one employing prototypes,while the other is based on the concept of exemplars.While it is not the intent of this research effort to experi-mentally differentiate between the two theories, it isimportant to briefly review each theory to gain someunderstanding of categorization models of cognitionprior to discussing potential applications for evaluatingpilot cognition. For a detailed analysis of categorizationmodels, see Ashby and Maddox (1993).

Prototype explanations are predicated on the notionthat for any category, individuals develop a mentalinstance, which is most representative of the category.This representation is referred to as the prototype. Newobjects are compared against the prototype of each cate-gory. If a new item is sufficiently similar to the proto-type, the item is assigned to that category. For example,when a person thinks of an aircraft, a prototype theoristwould argue that the individual has a mental representa-tion of what most aircraft look like. Mathematically, thistypicality can be expressed as an averaging process of allfeatures that compose an aircraft (such as wings, tail, andflight deck). Unusual aircraft, such as the Northrop-Grumman B-2 and the Beechcraft Starship, are viewed assuchprecisely because the prototype is violated.

Exemplar models of categorization differ from pro-totype explanations, in that the new object is comparedwith those individual items that compose a category. Themembership of an object in a category is defined byselecting that category whose instances differ the leastfrom the new item. This process essentially takes theform of satisfying a minimization function, whereby anobject to be categorized is compared on all relevant fea-tures to each object in a category. The category selectionrule would be to select the category whose members dif-fer the least from the object to be classified. With thistheory, features of an unusual aircraft, such as the B-2,would be compared to various categories of things whichfly (such as airplanes, helicopters, airships, and rockets).The category selection would be based on an analysis ofthe features of the B-2 and other members of the

4

category. The B-2 would be assigned to the categorywhose individual members had features that were theleast different from it.

Note that the principal difference between the twotheories is that exemplar models compare the item withindividual objects within the category, whereas prototypemodels compare the item with the prototype of thecategory.

Both models have yielded results consistent with thenotion that the presentation of information congruentwith an individual’s mental categorization scheme leadsto superior performance as measured by accuracy andresponse time. For example, Posner and Keele (1970)studied classification performance for visual random dotpatterns that had been memorized by subjects. The mem-orized patterns were variants of prototype patterns thatwere constructed by distorting the prototype by randomlymoving the dots. Subjects were then tested by using thememorized patterns, the prototypes from which the pat-terns were constructed, and two variants of the proto-types that the subjects had not seen. As expected, whentested on classification performance, individuals gave thefastest and most accurate responses to the stimuli theyhad previously memorized. Of particular interest wasthat as distortion from the original prototype increased,more errors and longer response times were observed.

Similar results have been found for meaningful stim-uli such as words and figures (Reed 1972; Rips, Shoben,and Smith 1973). For studies involving semantic stimuli,subjects will respond quicker and/or more accurately toinstances that are closest to the category. For a concrete,everyday example, individuals will take longer to cor-rectly judge that a chicken is a bird, than a robin is a bird.This longer judgement time is due to the fact that for themajority of individuals, “robin” represents a more typicalinstance of a bird than does “chicken.”

The results of these experiments have demonstratedthree major conclusions. First, inclusion of superordinatecategory terms allows researchers to see which stimulimost closely cluster around them. In the above example,“robin” and “bluejay” were much closer to the super-ordinate term “bird” than were “goose” or “duck.” Sec-ond, the time to make category judgments (e.g., is thisinstance a member of categoryx or y?) is directly relatedto the psychological distance subjects have for the givenset of stimuli. Psychological distance can be measuredquantitatively by using psychometric scaling techniques.These techniques will be discussed subsequently. Thefarther apart a pair of items was in the psychologicalspace, the longer it took subjects to make category judg-ments. In the above example with birds, Rips, Shoben,and Smith (1973) discovered that, when analyzing thesedata with techniques that allowed investigators to exam-

ine their spatial positioning (thus indicating their psycho-logical relatedness), “robin” was indeed closer than“chicken” to the category term “bird.” Third, the psycho-metric techniques allow researchers to identify the cogni-tive processes individuals use in categorizing and makingjudgments.

Just as categorization is a pervasive function ineveryday life, it frequently occurs on the flight deck. It ispossible that pilots principally use two characteristics tocategorize and manage flight-deck information. First,both the source (where the information is coming from)and the destination (where the information is displayedor used on the flight deck) provide the pilot with initialcues about content of the information. For example, theflight crew can initially make some inferences about amessage from its source (e.g., ATC, dispatch, or ATIS)prior to interpretation. From the pilot’s perspective, des-tination or location is probably most critical becausewhere information is displayed usually indicates the typeof information. For example, in a glass cockpit, assumingall CRT’s are operative, pilots know what informationthey can expect to see when they view any of the dis-plays. However, because display technology allows forthe presentation of multiple types and layers of informa-tion, location may no longer be as predictive as it hasbeen historically.

Second, the functions that the information supportwill provide an introduction to how the information willbe categorized. Traditionally, pilots are trained to aviate(control of the aircraft), navigate (location and destina-tion), and communicate (communicate intentions toATC). This training provides one high-level frameworkfor categorizing, although further subdivisions may bepossible and even desirable. This issue is a major con-cern of the research presented in this paper.

As automated information management aids areintroduced on the flight deck, the information discussedpreviously suggests that the processing and the categori-zation of flight-deck information should be consistentwith the pilot’s cognitive model of flight-deck informa-tion categories.

Prioritization of Information

Prioritization refers to the process by which informa-tion or actions are ordered along some dimension or by agiven attribute. Single or multiple dimensions and/orattributes may exist, depending on the characteristics ofthe information and the actions. For example, logicaldependence, wherein one source of information isrequired before the next piece of information can effec-tively be used, could be considered an attribute. In thiscase, the first source of information acquired wouldinitially be said to have a higher priority than the

5

subsequent information. While it is important to recog-nize in any prioritization scheme, logical dependenciesare rather trivial. Of considerably more interest are prior-itization dimensions that involve relative attributes ordimensions that change with situations and differ amongpeople. This multidimensional and dynamic nature ofinformation is of primary interest for the current researchplan. As with categorization, such dimensions andattributes can be conceptualized as cognitive variablesthat people use to prioritize the information they use toperform some function.

Prioritization is usually considered as an issue withinthe context of decision theory. With this perspective, newinformation arrives and the user of this informationdecides how to prioritize it in the context of the currentstate. From a decision theory context, this informationcan be analyzed as a specialized, expected utility prob-lem that is expressed through multiattribute utility(MAU) models (Edwards 1987).

In the MAU framework, an attempt is made to deter-mine what variables the decision maker considers, therelative weight given to these variables, and the utilitiesfor the expected outcomes. As Edwards (1987) states

Formal decision theory assumes that the(individual) decision maker has a set ofvalues,and chooses acts that, as he or she sees it, willbest serve them. . . . The consequences of eachact, or of each act-event-act-event . . . sequence,are calledoutcomes. Each outcome is conceivedof as having a subjective value orutility.(pp. 1063–1064)

Once the attributes have been identified, a determi-nation is made of how much weight the decision makerwishes to give each of them. For example, in purchasinga car, a new car buyer may consider reliability, safety,and cost as highly relevant attributes, while color, style,and gas mileage may be considered less relevant. For anindividual making a purchase, each of these variables canbe ordered in terms of its salience for making a purchas-ing decision.

Analogously, pilots must prioritize the attributes ofnew, incoming information relative to that of informationcurrently available and the state of the aircraft. Identifi-cation of how pilots normally process data, with empha-sis on what attributes are considered particularly salient,and the categories into which different data fall, is aprerequisite for understanding how pilots prioritizeinformation.

Currently, pilots are responsible for prioritizingflight-deck information. In some cases (principally emer-gencies) task and information prioritization is done forthe crew by aircraft system controllers or emergency

checklists. For nominal information, however, pilotshave aids to help them determine what information isrequired and when it is needed. Such aids are representedin the form of

1. Flight operations manuals

2. Procedural checklists

3. Training procedures and company policy

4. Operational demands

In general, high-level principles govern prioritiza-tion. Information required to maintain a safe flight willreceive a high priority value. Likewise, because someactivities must be performed in limited amounts of time,information required to support a time critical activitywill be obtained prior to information necessary for lesstime critical activities. Finally, information can be priori-tized on its logical dependency—one piece of informa-tion is necessary before another piece of information canbe used (e.g., one needs to know the ATIS frequencybefore gaining access to that information).

Availability of data must also be considered whendiscussing prioritization. Flight-deck information may becontinuously available (a dedicated display) or the pilotmay have to engage in a series of actions to obtain it (anumber of button presses on the FMS CDU). Alterna-tively, information may be provided to the crew withouta specific request having been made. Primary examplesof this include

1. ATC communications

2. Warnings, alerts, and cautions

3. Sources such as data link and intelligent flight-deckaids

Information competing for available cognitive pro-cessing resources underscores the importance of under-standing pilot expectancy and mental models ofinformation flow on the flight deck.

Psychometric Tasks and Analyses

Recent developments in cognitive research havedemonstrated the usefulness of psychometric techniquesin representing human knowledge and information pro-cessing (Ashby 1992; Nosofsky 1984, 1986, and 1992;Shepard 1987). One method, MDS, calculates a spatialrepresentation among stimuli by using some measure ofhow the stimuli are related to one another. This spatialrepresentation presents the objects in ann-dimensionalspace, with items similar to one another lying closetogether, while dissimilar items lie farther apart in thespace. Another method, cluster analysis also uses a mea-sure of stimulus similarity that identifies items closely

6

associated with one another, groups them, and provides ahierarchical representation of the stimuli, thereby allow-ing the investigator to examine the representation forobvious or intuitive categories. Objects placed in thesame cluster are more similar to one another than objectsplaced in different clusters.

For both methods, when the measures for stimulirelatedness are elicited from human observers, the repre-sentation is said to be cognitive or perceptual. Of particu-lar interest to researchers is the potential for cognitiveinterpretations of the dimensions or clusters emergingfrom such analyses. Although some of the most sophisti-cated work in this area has been done in the last 10 years,MDS had been used as early as 1973 by Rips, Shoben,and Smith.

The assumptions underlying the use of scaling andclustering in cognitive experiments are, in general, quitesimilar. First, the scaling solution for a given set of stim-uli is said to be a cognitive representation for how indi-viduals view the relationship among those items. Forflight-deck information, this would be a representation ofhow related or similar the flight-deck information is per-ceived as being. Second, dimensions extracted from thescaling analyses can be considered as representing thesalient cognitive dimensions along which individualsprocess information. In the present case, such dimensionswould reflect those variables pilots use to process flight-deck information.

Categories or clusters of information tell researchershow pilots define category membership. In the aviationdomain, three of the most frequently mentioned high-level categories are aviate, navigate, and communicate.Of interest here is whether these categories are sufficientto describe all the information pilots use, or whether afiner distinction may be more appropriate.

The potential for discovering the way in which pilotsprocess flight-deck information should not be consideredmerely descriptive. Rather, dimensions found throughscaling analyses and categories that emerge from cluster-ing could be used in a predictive fashion for evaluating apilot’s cognitive processing under different flight scenar-ios and new flight-deck designs. Because the psychomet-ric techniques used herein may prove useful inrepresenting the pilot’s cognitive processing of flight-deck information, it might be possible to use this data inconstructing models for evaluating the cognitivedemands placed on pilots as a result of the tasks they areperforming. High cognitive task loads could indicate asituation where the crew may be unable to adequatelyperform a task; therefore, consideration should be givento modifying the design and/or the operational proce-dure. This consideration could prove especially valuablefor evaluating flight decks early in the design process. If

such valid measurement scales can be developed basedon the results of the study reported herein, this informa-tion could be used early in the flight-deck design processto determine both the potential for information overloadin certain flight phases or the potential for tasks to com-pete for the same set of cognitive resources. Such appli-cations are discussed in appendix A.

Method

The experiment described in this paper was con-ducted at Langley Research Center. Details concerningthe subjects and the procedures are given in the followingsections.

Subjects

Fifty-eight pilots participated in this experiment. Ofthese, six subjects were eliminated from the subsequentdata analysis. Three of these six were removed becauseall commercial airline time was spent as a flight engi-neer, two subjects had no commercial airline experience,and one pilot indicated he had given erroneous responsesbecause of a misinterpretation of one of the stimuli. Theremaining 52 subjects were all commercial airline pilotswho were currently flying or recently retired. The meantotal flying time for these subjects was 11435 hr with arange of 3000 to 27000 hr. Average commercial experi-ence was 15 years with a range of 1.5 to 33 years. Of the52 subjects, all but 1 were male. Eight airlines wererepresented.

Stimuli

Experimental stimuli were obtained from an un-published information analysis performed on theBoeing 747-400 by John Groce of Boeing CommercialAircraft Co.

This analysis consisted of determining all informa-tion elements on the 747-400, each with its associatedsource, destination, display modality, message manage-ment schemes, and control modality. (An informationelement represents a discrete unit of data available to thepilot, usually associated with a specific display or inter-face.) “True Heading,” for example, is an informationelement associated with the navigation display.

This entire list consisted of 396 information ele-ments. To examine information elements applicable toall modern commercial aircraft, terms specific to the747-400 were eliminated by having three commercialpilots (none of whom flew the 747) go through the list ofinformation elements and identify terms unfamiliar tothem. These items were subsequently taken off the list.None of these pilots participated in the experiment. Traf-fic alert and collision avoidance system (TCAS) items

7

were also removed, because at the time of this study ithad not been fully implemented on all commercial air-craft. Elimination of the above items reduced the total setof information elements to 259. The items that remainedon the list were considered to be generic. That is, infor-mation was currently available and understood by anycommercial pilot flying, regardless of aircraft type.

Two stimulus sets, each containing 20 items, wereconstructed by random item selection from the genericlist. Random item selection was used so as not to bias theoutcome of the results. The experimental analysesemployed herein are based on determining the subject’scognitive organization of the stimuli. Had the experi-menters selected the stimuli, it could be argued that thefindings were their mental organization of theinformation.

Although it is desirable to have more items in thestimulus set to enhance the diversity of stimuli, pre-testing revealed that more than 20 items took a consider-able amount of time, which led to subject fatigue.Twenty items represented a balance between having suf-ficient stimuli for analysis, and minimizing subjectfatigue and time required to perform the tasks. The finalsets are hereafter referred to as set A and set B. Fourterms were common to both sets and are presented at thebottom of each list in boldface. The two sets are shown intable 1.

Procedure

Each pilot participated in two separate tasks witheither set A or set B. The first task required subjects to

make similarity judgments between pairs of informationelements for a given set. The second task required sub-jects to rank the terms in their perceived order of priorityunder two different contexts. Each task is described indetail in the following paragraphs.

For the similarity judgment task, two informationelements were presented on a computer screen, oneabove the other. The subject was then asked to provide arating of how similar the items were thought to be. Ascale from 1 (very similar) to 9 (very dissimilar) wasused for the rating. In the lower portion of the display, ascale bar appeared on every trial, which afforded the sub-jects constant access to the rating scale. Subjects couldmake similarity ratings by using the number keys ormoving an “X” along the rating scale bar with the arrowkeys on the computer keyboard and then pressing theenter key once the cursor was at the selected similarityrating. This screen arrangement is shown in figure 1. Acomputer program (Ricks 1994) randomized the order ofthe pairs presented to each subject. Subjects rated thesimilarity of each pair once. This produced a total of 190trials (half the matrix minus the diagonal). Subjects werefree to take a break at any point during the experimentaltrials when they became fatigued.

To acquaint themselves with the computer displayand response arrangement, subjects initially practicedmaking comparisons among six common automobiles.At the end of the practice, subjects were shown a list ofsix information elements similar to those to be evaluatedin the experimental set. Subjects rated the similarity ofthese terms, both immediately before and after the

Table 1. Random Samples of Flight-Deck Information Elements

Set A Flight-Deck Elements Set B Flight-Deck ElementsAlternate flap control status Actual EPRClock time Autothrottle disconnect statusCommanded pitch angle Current bank angleCrew oxygen flow indication Current calibrated (indicated) airspeedCurrent stab trim Engine fire conditionDistance to reach selected altitude Flight path angle (inertial)Flight number ILS tuning data (ident/freq/inbd crs/DME)N1 RPM limit Incoming ATC message (clearances)Passenger oxygen system mode (RESET/NORM/ON) Incoming dispatch message (company comm.)Predicted fuel at waypoints IRS information source (for Captn & F.O. PFD/ND)Relative bearing/distance of waypoints (intersections) Landing reference speedSelected roll mode Oxygen pressure (crew and passenger)Speed range for in-flight engine restart Selected vertical speedTotal air temperature Target N1True heading (current) VOR tuning data (freq/ident and radial/course for L/R VOR’s)Zero fuel weight Wing anti-ice statusPilot-initiated request or message Pilot-initiated request or messagePredicted/estimated wind for descent Predicted/estimated wind for descentSelected altitude Selected altitudeSpeed restriction data (speed limit, altitude) Speed restriction data (speed limit, altitude)

8

experimental trials. This practice rating allowed for thecalculation of a test-retest correlation to determinewhether subjects were using the rating scale consistently.

Prior to making similarity judgments for the experi-mental set, subjects were given a list of the 20 terms theywould be rating. If the subjects had any questions aboutitems on the list, the experimenters showed them the rel-evant section of the Boeing flight-deck operations man-ual (Anon. 1992) for clarification.

Subjects were given no specific definition of similar-ity to perform the comparison task; they were simply toldto use whatever definition of similarity they felt comfort-able with. Following the experimental trials, however,subjects were asked to explicitly state how they judgedthe similarity of the presented items. The entire similarityrating task took approximately 30 min.

In the second part of the experiment, subjects priori-tized the same set of 20 items that was used for the simi-larity judgments. These prioritizations were performedunder two conditions. Under the first condition, subjectswere given the stimuli on 20 separate index cards andtold to order the deck of cards according to their per-ceived priority without regard to a specific phase offlight. This was termed the generic condition. Upon com-pletion of this task, the data were recorded, and the cardswere shuffled and given back to the subjects for a secondprioritization. Under the second condition, subjects weretold to order the items in the context of the takeoff phaseof flight. For both conditions, subjects were told to con-sider all systems as operating normally.

Analogous to the similarity rating task, subjects weretold to order the stimuli in each condition by using what-ever definition of priority they had. After each ordering,subjects were asked to describe how they prioritized the

Figure 1. Experimental screen response arrangement.

Term 1: Landing Reference SpeedTerm 2: Speed Restriction Data

12

34

56

78

9

xVERY SIMILAR VERY DISSIMILAR

How Similar?> 3

data. Subjects were not told of the second prioritizationcondition (takeoff condition) prior to its introduction.The takeoff condition always followed the genericcondition.

Following the experimental tasks, subjects wereasked to fill out a questionnaire for their opinion on sev-eral issues related to the categorization and prioritizationof flight-deck information. A copy of the questionnaireappears in appendix B.

Results and Discussion

Of the 52 subjects retained for analysis, 27 usedset A stimuli while 25 used the set B stimuli. The corre-lations between the sessions preceding and following theexperimental trials (six information elements similar tothose in the experimental trials) were calculated for eachsubject. None of the individual correlations were nega-tive, and the average test-retest correlation was a moder-ate 0.536. Both factors suggest that subjects were usingthe scale consistently. Given this information, data fromall 52 subjects were used.

Upon completion of the experimental trials, datafrom each subject were stored in matrix format on com-puter disk. The individual subject files were analyzedwith personal computer versions of the Individual Differ-ences Scaling (INDSCAL, Carroll and Chang 1970),hierarchical clustering (average linkage method, cf.Romesburg 1984), and multidimensional preferenceanalysis (cf. Carroll 1972 for analysis implemented withPCPREF) programs. These analyses were run separatelyon the set A and the set B respondent data, and will bereported separately. Results have been organized in thissection by the research question being addressed. Analy-sis procedures used to address each issue have been bro-ken out separately within each area.

How Pilots Cognitively Process Flight-DeckInformation

As noted in the section entitled “Background,” cog-nitive researchers have used psychometric techniques,such as MDS and cluster analysis, to infer how individu-als mentally represent information and the associatedcognitive processes they use to act upon their representa-tion. In the current study, these techniques were used toexamine how pilots represent, and therefore process, typ-ical flight-deck information. Similarity ratings, whichwere collected by having subjects evaluate the flight-deck information pairs, served as data for both statisticaltechniques. These data were analyzed by usingINDSCAL and hierarchical clustering techniques. Foreach technique, results will be followed by interpretationand discussion.

9

Individual Difference Scaling

INDSCAL is a statistical technique that uses proxim-ity matrices as input for each subject. Such matrices rep-resent some measurement of stimulus dissimilarity. Inthe present case, this measurement was the pairwisecomparison data from the computer trials. TheINDSCAL model computes a group solution that repre-sents the stimuli spatially inn dimensions. The modelattempts to fit the data so that items perceived as beingvery similar to one another are located in close proximityin then-dimensional space; whereas, items perceived asbeing dissimilar lie far apart in the space. Because thissolution is generated with data from all subjects, thesedimensions are said to be common to all individuals inthe sample.

A unique aspect of the INDSCAL approach, how-ever, is the idea that while individuals may process stim-uli by using the same group dimensions, it is unlikelythat they use the dimensions equally or in the same way.For this reason, INDSCAL also calculates dimensionalweights for each subject, which indicate how relevant orsalient a dimension is for an individual. Thus, the groupspace allows the investigator to examine the commoncognitive processing of stimuli, while the weights allowone to evaluate the degree to which each subject uses thegroup dimensions. The subject weights from theseINDSCAL analyses are reported in the section entitled“Individual Difference Scaling.”

The INDSCAL model was run on the individual sub-ject matrices for set A and set B stimulus sets. Data ineach set were scaled in one, two, three, and four dimen-sions. The dimensional solution was selected based onthe percentage of variance for which it accounted andease of interpretation. Beyond the three-dimensionalsolution for each set, neither the percentage of variancenor the ease of interpretation increased appreciably;hence, the three-dimensional solutions were retained foranalysis. To avoid local minima solutions, multiple runswere performed on each set with five randomly gener-ated starting values. The five random starting values pro-duced virtually identical final configurations within eachset, which suggests that none of the final configurationsarose from a local minimum (differences among solu-tions in terms of variance accounted for were in the1-percent range).

The set A solution selected for interpretationaccounted for 36 percent of the variance and the averagecorrelation between computed and actual subject scoreswas 0.595. The set B solution selected for interpretationaccounted for 42 percent of the variance and the averagecorrelation between computed and actual subject scoreswas 0.635. Given the random stimulus selection, thenumber of subjects within each set, and the unique nature

of some comparisons for the pilots, these fits are good.These three-dimensional solutions, as well as an interpre-tation for sets A and B, are presented in figures 2 and 3,respectively.

For the plots shown in figures 2 and 3, each stimuluselement was plotted in the three-dimensional space andthe viewing angle was changed so that each stimulus wasvisible in the plot. Because this rotation was performedabout the coordinate axis, the relative position of items inthe plot remained identical. These plots were then shown,without the dimensional labels, to several subject matterexperts for interpretation. In addition, the questionnairedata on similarity were used to aid in interpreting thesesolutions. The names assigned to the dimensions hererepresent a consensus of the responses obtained.

For sets A and B, one dimension appears to repre-sent traditional flight functions. This is indicated on bothplots with items on the left side of the dimension relatingto aviate functions, those in the middle relating to navi-gation, while items on the far right related to communi-cation. This dimension has been labeled “Flightfunction.”

The second dimension appears to represent controland planning functions or, as indicated on the plots, whatmight more generically be referred to as a tactical andstrategic continuum. Note that items on one end arerelated to direct, short-term, aircraft control (such as“Commanded Pitch Angle” and “Selected Roll Mode”for set A and “Engine Fire Condition” and “Current Air-speed” for set B), while items on the opposite end of thisdimension support long-term flight activities (such as“Relative Bearing/Distance of Waypoints” and “Pre-dicted Fuel at Waypoints” for set A and “VOR TuningData” and “Predicted/Estimated Wind for Descent” forset B). This latter dimension has been labeled “Flightaction.”

The third dimension for sets A and B might best bedescribed as the number of times the pilot refers to oruses a given piece of information. As indicated on theplots, this dimension has been labeled “Sample Rate.”Items with high values on this scale represent informa-tion referred to more frequently than information withlower values. In set A, the “Relative Bearing/Distance ofWaypoints” has a higher value on this dimension than“Crew Oxygen Flow Information” or “Passenger OxygenSystem Mode.” Analogously, for set B, “Current Air-speed” and “Selected Vertical Speed” have high values(representing frequent referral), while items less oftenused, such as “Engine Fire Condition” and “Wing Anti-Ice Status,” have low values along this dimension.

The INDSCAL dimensions discussed previouslyrepresent how this group of subjects viewed the

10

information they regularly use on the flight deck. Theresults for sets A and B are based on a randomly selectedsampling of information elements (20 in each set) fromthe larger set of 259 items. While it is reassuring that thetwo sets that used these information elements producedsimilar solutions, the intent is to extend these results toinclude flight-deck information that was not part of theoriginal set A or B data. An important issue will be howto extend these results to all data.

To illustrate this issue with a concrete example, con-sider a multidimensional scaling analysis of 10 cities inthe United States, where the data matrix consists of dis-tances between those cities. The analysis produces twodimensions; interpretation of which reveals that they cor-respond to the North/South and East/West map orienta-tions (Kruskal and Wish 1978). Now assume a new cityis presented and it is to be placed among the other 10 inthe spatial plot. With no knowledge of geography of the

United States and no information about this city in rela-tion to the other cities, this task is not possible.

Note that the INDSCAL dimensional solutions areimportant, not because they describe the relationship ofthose items measured in the original analysis; rather, theyare important because they give insight into how individ-uals perceive the relationship of the items and offer pos-sibilities for how this view may be employed in futurestudies. This view, of course, represents the dimensions.Because the dimensions described here may be represen-tative of cognitive processes that pilots use with flight-deck information, the identification of dimensions isextremely useful. As discussed in appendix A, thesedimensions can be quantified and may be employed byhaving pilots scale other flight-deck information notincluded in the study as part of the current stimulus set.(While it is possible with the introduction of new tech-nology that completely novel information may be

Figure 2. Spatial solution and interpretation of set A similarity data.

SelectedRoll Mode

Current Stab Trim

Distance to ReachSelected Altitude

Relative Bearing/Distanceof Waypoints (intersections)

Predicted Fuel at Waypoints

Predicted/EstimatedWind for Descent

Speed Restriction Data

True HeadingSelected Altitude

Speed Range forIn-Flight Engine Restart Total Air Temperature

N1 RPM Limit

Zero Fuel Weight Pilot-InitiatedRequest or Message Clock Time

Flight Number

Alternate FlapControl Status

Crew OxygenFlow Indication

Passenger OxygenSystem Mode

CommandedPitch Angle

aviate

navigate

communicate tactical

strategic

infrequent

frequent

Sample rate

Flight action

Flight function

11

introduced, it would seem that a flight-deck engineercould make a fairly accurate estimate of the characteris-tics of this information.) These data, in turn, could beused as part of a cognitive task analysis for future flight-deck designs. Development and validation of such scaleswill be, of course, a necessary first step.

Hierarchical Clustering

The MDS analyses provide spatial solutions for theproximity data. When viewing these plots, the dimen-sions are interpreted primarily by examining where infor-mation elements lie along the dimensions. In doing thisexamination, researchers will often notice items that tendto lie close to one another in the space. Such groups ofpoints, while potentially aiding interpretation of the spa-tial plots, may also be indicative of a natural category.Such categories emerge from the classifications individu-als have learned to make of events or objects encountered

in the world. For example, individuals have well-developed categories into which they place animals regu-larly encountered. These categories could include “cats,”“dogs,” or “birds.” As noted in the “Background” sec-tion, this categorizing considerably simplifies an individ-ual’s processing of new events and information.Analogously, it seems plausible to assume pilots havehigh-level categories for flight-deck information theyregularly encounter.

This analogy is precisely the reason for exploringthose categories into which pilots place flight-deck infor-mation. Recall from the “Background” section that stud-ies have shown that presentation of informationconsistent with an individual’s cognitive categorizationscheme may lead to superior performance, as indicatedby measures for accuracy and response speed. For thisreason, identification of the natural categories that areused by pilots (if they indeed exist and can be adequately

Figure 3. Spatial solution and interpretation of set B similarity data.

Selected Vertical Speed

Flight PathAngle

Speed RestrictionData

CurrentBank Angle

Predicted/EstimatedWind for DescentCurrent Calibrated

Airspeed

Selected Altitude

Landing Reference Speed

Target N1

Actual EPR

AutothrottleDisconnect Status

Wing Anti-Ice Status

Engine Fire Condition

Oxygen Pressure

IRS InformationSource ILS

Tuning Data

VOR TuningData

Incoming DispatchMessage

Pilot-InitiatedRequest or Message

Incoming ATC Message

aviate

navigate

communicate tactical

strategic

infrequent

frequent

Sample rate

Flight action

Flight function

12

defined) may prove useful in developing new informa-tion presentation formats and information managementsystems.

To address this issue, the original (or raw) proximi-ties for the pairwise comparison data, which were aver-aged across subjects were clustered hierarchically.Analyses were run with the PROC CLUSTER proceduredeveloped by SAS Institute Inc. (Anon. 1988), whichspecified the average linkage method. Cluster analysis isa technique that attempts to group objects togetheraccording to their similarity. Although several variationsare possible, the analysis typically begins by groupingthe most closely related objects (based on proximity),treating this cluster as a new individual object, and thenrecalculating its proximities with the remaining objects.This iterative approach continues until only two clustersremain. (For a full description of the most widely usedclustering approaches, see Romesburg (1984).) Theresulting output is often presented in the form of a den-drogram (or hierarchical tree). When presented laterally,as is the case here, closely related items cluster togetherat the far right of the dendrogram, while the further oneproceeds toward the left (up the plot), the less similarityis perceived among individual items and the fewer dis-tinctions (in the form of clusters) can be made.

Cuts in the dendrogram were made where there wasa high level of stability in the obtained clusters. This cri-terion means that the cut point could be moved across arelatively wide range of the dendrogram without signifi-cantly affecting the number of clusters (for informationon how to make dendrogram cuts, see Romesburg 1984).The dendrograms for the raw similarity ratings of sets Aand B are presented in figures 4 and 5, respectively.Based on where the cut was made, boxes have beenplaced around the clusters. Interpretations have also beenassigned to these clusters and are so identified. As withthe INDSCAL plots, the clusters were examined by sub-ject matter experts and names were assigned to the clus-ters after review.

Although some differences occur between the spatialposition of the information elements from the INDSCALanalyses and the clustering results, in general the patternof results proved quite similar. That is, items that werelocated close together in the INDSCAL plot also tendedto group together in the cluster analyses.

Based on the dendrogram cuts, four high-level clus-ters emerged for the set A raw proximities data and havebeen labeled in figure 4. These clusters include

1. Aviation

2. Navigation

3. Communication


Each of these categories may be defined by using amodified series of definitions described in Abbott(1993). Aviation may be defined as the process “ofadjusting or maintaining the flight path, attitude, andspeed of the [aircraft] relative to flight guidance require-ments.” Navigation may be defined as the process of“developing the desired plan of flight. . . and monitoringits progress.” Communications can be defined as involv-ing the transfer of information between the crew andATC, the airline company, and flight crew members.Finally, systems administration may be defined as theprocess of monitoring the state of aircraft systems, iden-tifying when actions may need to be taken to return asystem to nominal status (if possible) and making suchchanges.

Closer inspection of the large navigational clustersuggests a further distinction could be made within thatgrouping. A differentiation could be made between purenavigational information (“Relative Bearing/Distance ofWaypoints” and “Distance to Selected Altitude”) andinformation traditionally considered as being referencedata (“Speed Range for In-Flight Engine Restart” and“N1 RPM Limit”). While it is useful to recognize thatsuch a distinction can be made, for purposes of parsi-mony, the four-cluster solution was retained. Based onthe dendrogram cuts for the raw proximities data, thesame four high-level clusters emerged in set B asemerged in set A.

Results from the cluster analyses suggest pilots con-ceptualize flight-deck information into at least four dis-tinct categories. These categories are probably bestcaptured by the descriptors (1) aviation, (2) navigation,(3) communication, and (4) systems administration.Relating these results to the prior INDSCAL solutions,these categories appear most strongly related to the flightfunction dimension. While three distinct activitiesappeared along that dimension for the INDSCAL analy-ses, the cluster results suggest that systems administra-tion would most closely align itself with communication.Finally, the fact that similar clusters emerged across twoseparate data sets, with each having substantially differ-ent items, is reassuring because it suggests the same clus-ters generalize across two data sets.

While three of the four information elements com-mon to both sets appeared in different clusters, it isimportant to note this occurrence was most likelybecause of the random stimulus selection employed. Forexample, a cluster, most appropriately considered as nav-igational, emerges in both sets A and B. However,

13

Figure 4. Clustering dendrogram of set A similarity ratings.

PASSENGER OXYGEN SYSTEM MODE

CREW OXYGEN FLOW INDICATION

ALTERNATE FLAP CONTROL STATUS

PILOT INITIATED REQUEST OR MESSAGE

FLIGHT NUMBER

ZERO FUEL WEIGHT

CLOCK TIME

RELATIVE BEARING/DISTANCE OF WAYPOINTS

PREDICTED FUEL AT WAYPOINTS

PREDICTED WIND FOR DESCENT

DISTANCE TO SELECTED ALTITUDE

SELECTED ALTITUDE

TOTAL AIR TEMPERATURE

SPEED RESTRICTION DATA

SPEED RANGE FOR IN-FLIGHT ENGINE RESTART

N1 RPM LIMIT

TRUE HEADING

SELECTED ROLL MODE

CURRENT STAB TRIM

COMMANDED PITCH ANGLE Aviation

Navigation

Communication

Systemsadministration

14

Figure 5. Clustering dendrogram of set B similarity ratings.

OXYGEN PRESSURE

INCOMING DISPATCH MESSAGE

INCOMING ATC MESSAGES

PILOT INITIATED REQUEST OR MESSAGE

IRS INFORMATION SOURCE

ILS TUNING DATA

VOR TUNING DATA

WING ANTI-ICE STATUS

ENGINE FIRE CONDITION

AUTOTHROTTLE DISCONNECT STATUS

ACTUAL EPR

TARGET N1

PREDICTED WIND FOR DESCENT

SELECTED VERTICAL SPEED

SELECTED ALTITUDE

LANDING REFERENCE SPEED

SPEED RESTRICTION DATA

CURRENT CALIBRATED AIRSPEED

FLIGHT PATH ANGLE

CURRENT BANK ANGLE Aviation

Navigation

Communication


15

because the stimulus selection procedure selects ran-domly, set B had three information elements, that wereall strongly related to flight navigation. These relation-ships had the effect of forming an extremely tight naviga-tion cluster that contained three navigation items andforced the other items, that were related to navigation,albeit less so, into the larger aviation cluster. Likewise inset A, four information elements, strongly related to avi-ation, formed a tight cluster that left a large cluster thatcontained items principally related to navigation. Thisfailure to find the identical information elements in thesame clusters is not surprising because the sets were ran-domly formed, and the experimenters had no controlover the combination of stimuli appearing in each set.When examining clustering results, the important thingto note is that the cluster names are consistent with theitems they comprise.

The cluster analyses also provide insight into howpilots categorize regularly used flight-deck information.While most anecdotal evidence from pilots suggests thatthey divide information into three categories (aviate,navigate, and communicate), the current findings suggestthat a richer categorization scheme may be operating,with pilots willing (either consciously or not) to makefiner gradations to the information they process. Thesegradations are seen here as systems administrationemerging as a fourth category in the analyses and withincategories, such as navigation, the emergence of furtherdifferentiation in the form of navigational informationand reference information. (An alternative distinctionwithin this category would be dynamic and static infor-mation, because items such as “N1 RPM Limit” and“Speed Restriction Data” involve fixed values, while“Predicted Wind for Descent,” and “Predicted Fuel atWaypoints” involve dynamic or changing information.)

In determining how pilots process (and prioritize)information, it would be tremendously useful to workwith high-level functionally related categories, such asthose empirically found herein instead of working withthe individual information elements. As will be discussedin the section “How Pilots Prioritize Flight-Deck Infor-mation,” the clusters emerging from these analyses mayprove useful in developing models for the prioritizationof flight-deck information.

How Pilots Prioritize Flight-Deck Information

How pilots prioritize flight-deck information was asecond concern of this research project. While the proce-dures already described (INDSCAL and hierarchicalclustering) provide the researcher with insight into howpilots perceive the flight-deck information elements to berelated, these procedures do not directly address theunderlying method that is used to prioritize the informa-

tion. To explore this, a statistical technique called multi-dimensional preference analysis was employed. As notedearlier, to address the dynamic nature of flight-deckinformation usage, pilots were asked to prioritize undertwo separate conditions. One prioritization occurredwithout regard to any specific phase of flight (genericcondition), while the other was in the context of the take-off condition of a flight mission. The results for each ofthese experimental conditions will be presented sepa-rately for sets A and B.

Multidimensional Preference Analysis (MDPREF) isa technique that takes ranked data and provides a jointrepresentation of the data and the subject preferences.The analysis presents the stimuli spatially, analogous tomultidimensional scaling. The MDPREF analysis wasconducted with the PCPREF algorithm, which is a per-sonal computer version of MDPREF. (See Carroll 1972).This implementation accepts data ordered by subjects asinput. The program then computes principal componentsfrom these data. The number of components that areretained is determined by (1) examining the additionalvariance that is explained by adding a new componentand (2) the overall interpretability of the solution.

MDPREF analyses were conducted on each set forboth generic and takeoff conditions. Across all four runs,the dimensionalities were extremely similar. Each solu-tion generated one principal component that accountedfor over 40 percent of the variance and a second compo-nent that accounted for approximately 10 percent of thevariance. These variance percentages indicate the extentto which each component provides information about thedata. When the components are viewed in the context ofuncovering information in the raw data matrix, large per-centage variances could be expected to account for moreinformation. As successive components are extracted, thevariance accounted for by each component will decline,and begin to represent random variation. Here, the firstcomponents accounted for a large amount of the variancefor each solution, and the variance of componentsbeyond the first two dropped off sharply. For these rea-sons, and to achieve maximum interpretability, two-dimensional solutions were selected for all runs. Thepercentage variance accounted for by each factor is pre-sented in table 2 for sets A and B for the generic and thetakeoff conditions.

An overview of the results for these four factorialconditions will be presented because the same pattern ofresults emerges across all four conditions. Specificresults for each of the four conditions will be discussedin further detail in the following sections. Along the firstdimension, high values were found for aviation informa-tion, moderate values were observed for navigationalinformation, and system status information exhibited the

16

lowest values along this dimension. This latter finding isnot surprising, given that status data in these sets dealtprimarily with emergency information and subjects wereinstructed under both conditions (generic and takeoff) toconsider all systems as operating normally. The seconddimension appears to be capturing communicationsinformation because these items define the extreme endof the dimension.

Multidimensional Preference Analysis of Set ARankings

Average rank of set A data under both the genericand the takeoff conditions are provided in table 3.

The correlation between the generic and the takeoffrankings was 0.701, which indicates that the same infor-mation judged to be important during takeoff is also gen-erally seen as important in the generic phase of flight. Tomore fully investigate how the information elementswere judged for importance, the ranked data were sub-mitted to an MDPREF program for a spatial analysis.These results are given for each condition in the follow-ing sections.

Table 3. Average Rank of Set A Data for Generic and TakeoffConditions

Set A data Generic condition Takeoff condition

Alt 1 4ComPitch 2 1TrueHead 3 5RollMode 4 7N1 RPM Limit 5 2SpdRestriction 6 11RelBearing 7 14PredFuel 8 17CurrStabTrim 9 3DistToAlt 10 12ZeroFuelWt 11 6Plt Reqt/Mess 12 9Flt # 13 8CrewOxygen 14 16PredWind 15 20TAT 16 10SpdRange 17 18PassOxygen 18 15Clock 19 13AltFlapStatus 20 19

Generic condition.Two principal components ac-counted for a total of 57 percent of the variance. The spa-tial plot is shown in figure 6. To aid with interpretationof the plot, those clusters obtained from the similaritydata for set A (reported in the “Hierarchical Clustering”section) have been labeled with the high-level descriptors(aviation, navigation, communication, and systemsadministration) used previously. This technique is usedbecause the cluster names superimposed on the plotswere derived from a completely independent analysis.Interpretation of spatial solutions is achieved by definingdimensional endpoints and by identifying those itemsthat tend to group together. The cluster analysis providesthis information and allows any structure in the solutionto be observed.

Items in the high end of the first dimension princi-pally include aviation information. Navigational infor-mation falls toward the middle of this dimension, whilesystems administration information is found in the lowerend. The second dimension has communications infor-mation at the high end, while systems administration dataaccount for some items at the lower end.

While the dimensions from MDPREF plots oftenlead to obvious interpretation, the solutions obtained hereare more complex. One reasonable interpretation, whichappears across all the MDPREF runs, would be a primaryimportance dimension and a secondary importancedimension. Several factors argue for this interpretation.First, the primary dimension (X-axis) captures a largeamount of the variance. This occurrence could be takenas suggesting a unidimensional solution is sufficient tocapture the information contained in the ordering data.Because the first dimension in an MDPREF solution typ-ically approximates a consensus of the agreement of thestimulus ranking across subjects, this solution wouldsuggest that the first dimension is a good representationof what these pilots consider to be the most importantinformation for safe flight.

Second, if the first dimension corresponds to thepilot’s perception of overall information importance,then one could expect to find a high correlation betweenthe average rank of items in the generic condition (fromtable 3) and their coordinate values along the firstdimension. This was in fact the case, with a correlation of−0.94. Because of this high correlation and because

Table 2. Proportion of Variance Accounted for by Each Factor in the MDPREF Analyses

Set A Set B

Condition Dimension 1 Dimension 2 Dimension 1 Dimension 2

Generic 0.434 0.138 0.478 0.094Takeoff 0.504 0.130 0.595 0.109

17

subjects were told to rank the information in terms ofperceived priority, the first dimension (X-axis) appears tobe measuring an overall information importance factor.

As noted, while the use of previously defined clus-ters represents a useful method for interpreting MDPREFspatial plots, it is also informative to analyze those clus-ters that emerge with the MDPREF stimulus coordinatesas input data for the analysis. This analysis determineswhether similar clusters are derived from entirely differ-ent experimental techniques. If so, the relative prioritiza-tion of classes of flight-deck information as revealed inthe clusters can be examined. In the present case, threedistinct clusters emerged that were essentially the sameas those found in the set A cluster analyses of the similar-ity proximity data. The primary difference is that onelarge cluster emerged that encompassed the items previ-ously appearing under the aviation and navigationclusters.

Takeoff condition.Two principal componentsaccounted for 63 percent of the variance. The spatial plotis shown in figure 7. As in the generic condition, itemsshowing high to moderate values along the first dimen-sion are again included in the aviation category. Naviga-tion elements once again lie in the middle, while systemsadministration items define the lower end. As before,communications items have large values on the seconddimension. Note, however, that in comparing figure 6

with figure 7, certain elements have shifted because oftheir particular importance for takeoff. This shift is mosteasily seen with the increasing importance of “Zero FuelWeight,” “N1 RPM Limit,” and “Current Stab Trim.”Likewise, note that information relevant for in-flightactions moves down in the first dimension (e.g., “Pre-dicted Fuel at Waypoints” and “Predicted/EstimatedWind for Descent”). Finally, the full dimensionality ofthe data can be exploited by noting, in a two-dimensionalsolution, information with high values on both dimen-sions will appear in the upper right hand quadrant of theplot. In this case, those items particularly relevant fortakeoff shifted into the upper right quadrant of figure 7.This shift is as expected, given the logic behind theMDPREF analysis.

As in the generic condition, a cluster analysis wasperformed on the coordinate data obtained from theMDPREF analysis for the takeoff condition. This analy-sis produced a somewhat different result from thatobtained in the generic condition. While the same num-ber of clusters emerged, items appearing within the clus-ters were more diffuse. Examination of these clustersrevealed that they might represent a straightforwardprioritization of information required for safe takeoff(analogous to a checklist) with one group containing themost important information, another group containingsecondary information, and a third group containing theleast important information.

Figure 6. Spatial solution and interpretation of set A ranking data for generic condition.


Speed Range forIn-Flight Engine Restart




Total AirTemperature

Clock Time Flight Number Pilot-InitiatedRequest or Message

Zero Fuel Weight

Current Stab Trim

Predicted Fuelat Waypoints


Relative Bearing/Distance of Waypoints


N1 RPM Limit

Selected Roll Mode

True Heading

Selected Altitude

Commanded Pitch Angle


Communications

Aviation

Navigation

18

Multidimensional Preference Analysis of Set BRankings

The average rank of set B data are presented intable 4.

Table 4. Average Rank of Set B Data for Generic and TakeoffConditions

Set B data Generic condition Takeoff condition

Airspeed 1 2BankAng 2 5Alt 3 6.5Actual EPR 4 3Flight Path Angle 5 8Target N1 6 1EngFireCon 7 6.5SpdRestriction 8 10Sel Vertical Spd 9 9LndRefSpd 10 17Inc ATC Msgs 11 11ILS Tuning Data 12 16AThrottle Data 13 4VOR Tuning Data 14 14Plt Reqt/Msg 15 12IRS Info Source 16 15PredWind 17 19Anti-Ice 18 13OxyPressure 19 20Inc Dispatch Msg 20 18

There was relatively good agreement on the impor-tance of this information, which is indicated by a correla-tion of 0.811 between the two rankings. This correlationindicates that information deemed to be important duringtakeoff is also seen as being important during the genericphase of flight. As with the set A data, the subject rank-ings were analyzed with MDPREF to more fully explorethe data structure. These results will be presented sepa-rately for each of the two conditions in the followingsections.

Generic condition.Two components accounted for57 percent of the variance. The spatial plot is shown infigure 8. To aid with interpretation of the plot, high-leveldescriptors for clusters obtained from the similaritiesproximity data for set B (reported earlier in “HierarchicalClustering” section) are identified on the spatial plot.Note that the aviation cluster captures the majority ofinformation elements with high values on the first dimen-sion. Navigation and communications data show lowervalues on this dimension, while the one systems adminis-tration item (“Oxygen Pressure”) shows an extremelylow value. The second dimension is defined almostexclusively at the high end by “Engine Fire Condition.”Navigation elements occupy the lower end of this seconddimension. As with the set A data, the two dimensionswould again appear to represent a distinction betweeninformation of primary and secondary importance. The

Figure 7. Spatial solution and interpretation of set A ranking data for takeoff condition.


Speed Range forIn-Flight Engine Restart




Total AirTemperature

Clock Time

Flight Number


Zero Fuel Weight

Current Stab Trim

Predicted Fuelat Waypoints


Relative Bearing/Distance of Waypoints


N1 RPM Limit

Selected Roll Mode

True Heading

Selected Altitude

Commanded Pitch Angle


Communications

Aviation

Navigation

19

correlation between the average rank in the generic con-dition for each information element and their coordinatevalues along the first dimension was−0.98.

To more fully analyze these data, a cluster analysiswas run on the stimulus coordinates from the MDPREFanalysis. Four distinct clusters emerged. Interestingly,this cluster analysis proved to be different from thatfound for the raw similarity data. (See fig. 5.) Here, theclusters were less well defined, and in some cases, con-tained different information. These clusters may haveresulted from the nature of the primary task. Again, theranking may become synonymous with relative priority.

Takeoff condition. Two principal components ac-counted for 70 percent of the variance. The spatial plot ispresented in figure 9. The clusters derived from the set Braw proximities have again been identified on the plot.As noted in the generic condition, the aviation clustercontains the majority of items in the stimulus set andagain exhibits high values on this first dimension. Asbefore, navigational items lie along the middle of thedimension. The second dimension is defined at the highend by communications information, while navigationaldata occupies the other end.

This particular data set produced interesting resultsacross the two contextual conditions. First, note that for

the analysis of the takeoff condition with MDPREF,within the aviation cluster, several information elementsshifted in perceived importance. Most notably, propul-sion information (“Target N1,” “Engine Fire Condition,”and “Actual EPR”) moved toward the high end of thefirst dimension. Additionally, communication informa-tion moved to the top of the second dimension.

To more fully analyze these data, a cluster analysiswas run on the stimulus coordinates from the MDPREFanalysis. Four distinct clusters again emerged. As in thegeneric condition for this data set, the clusters obtainedhere contain some analogous items leading to less welldefined categories. As before, this may be due to the par-ticular set of stimulus items and the task requirements.

Discussion

In the MDPREF framework, the content of the firstdimension is that which is most important to the subjects;while the second dimension represents the next mostimportant content that is not correlated with the first. Toname these dimensions on the spatial plot, an analysis isnormally made of those items on the extremes of eachdimension of any groups of items clustering together thatshare obvious similarities. Although a spatial plot is gen-erated, there is no guarantee that a meaningful interpreta-tion will emerge for the dimensions.

Figure 8. Spatial solution and interpretation of Set B ranking data for generic condition.


Incoming ATC MesssageAutothrottle

Disconnect StatusWing Anti-Ice StatusTarget N1


Actual EPR

Selected Altitude Current CalibratedAirspeed

Current Bank AngleSpeed Restriction Data

Landing Reference Speed

Flight Path Angle


IRS Information Source


ILS Tuning Data

VOR Tuning DataOxygen Pressure

Incoming Dispatch Message


Communications

AviationNavigation

20

For both sets, functionally related flight-deck infor-mation appears to be collocated in the MDPREF plots.This collocation is independently confirmed by the clus-ters derived from the similarity data. Information ele-ments in close proximity to one another on these spatialplots (figs. 6 to 9) generally fell within the same fourclusters (aviation, navigation, communication, and sys-tems administration) described earlier. These clusternames have been placed in the spatial plots to indicate,generally, where the individual elements composingthem are located.

As discussed above, the first, or consensus dimen-sion (X-axis) in these analyses, could simply be labeledas a primary importance dimension. The second dimen-sion might represent information considered to be of sec-ondary importance. If interpreted as such, the relativeposition of items forming clusters directly indicates theinformation pilots consider to be most critical for safeflight. This relative position of clusters in the MDPREFplots is, perhaps, one of the more important aspects ofthis analysis. Just as the information elements can beranked along the MDPREF prioritization dimensions, sotoo can clusters of information, if the items within eachcluster share some logical similarity.

Several additional issues emerge from these analy-ses. First, it is important to note that the relative lack ofimportance assigned to the emergency information ele-

ments for the systems administration cluster is almostcertainly due to the experimental instructions. Recall thatsubjects were told to consider all systems as operatingnormally. In addition to this, those emergency itemsappearing in sets A and B would not seem particularlyrelevant for a takeoff emergency situation. Second, notethat, for both sets, the first dimension in the contextdependent condition accounts for almost as much vari-ance by itself as the two dimensions in the context inde-pendent (generic) condition. One explanation for thisoverwhelming difference in variance is that by providinga specific context in which to prioritize, the experimenterimplicitly requires that the pilot give a ranked checklistof that information necessary for a safe takeoff. Thisranking can most easily be captured in a single dimen-sion (from most to least important).

Cognitive Processing of Information AmongPilots

Of interest in the study presented herein was theextent to which the cognitive processing of stimuli wassimilar among pilots. This similarity can be evaluated forboth the cognitive representation pilots have for theinformation elements (as analyzed with the INDSCALmethodology) and the prioritization of flight-deck infor-mation (as analyzed with the MDPREF methodology).Results addressing each of these areas will be presentedseparately.

Figure 9. Spatial solution and interpretation of Set B ranking data for takeoff condition.


Incoming ATC Messsage

AutothrottleDisconnect Status

Wing Anti-Ice Status

Target N1


Actual EPR

Selected Altitude

Current CalibratedAirspeed

Current Bank Angle

Speed Restriction DataLanding Reference Speed

Flight Path Angle


IRS Information Source


ILS Tuning Data VOR Tuning Data

Oxygen Pressure

Incoming Dispatch Message


Communications

Aviation

Navigation

21

Individual Difference Scaling

For the similarity judgments, INDSCAL provides asubject weight space that indicates how salient each ofthe dimensions extracted from the analysis is to an indi-vidual subject. For dimensions interpretable as reflectingcognitive processes, salience can be seen as equivalent tothe use of the dimensions. If a subject is using all dimen-sions equally, then the weights given to each dimensionshould be about equal. If, however, one (or more) of thedimensions is being used more than the others, thereshould be a difference among the weights. Three-dimensional plots for the sets A and B subject weightspaces are shown in figures 10 and 11, respectively.

The subject weights indicate that for set A, pilotswere using all dimensions about equally. For set B, theytended to use dimensions 2 and 3 (flight action and sam-ple rate) less than dimension 1 (flight function). Theseresults were confirmed statistically by analyzing theINDSCAL subject weights in a one-way ANOVA, witheach of the three dimensions representing a treatmentlevel. For set A, there was no statistically significant dif-ference among the three dimensions (F(2, 52) = 1.73,p > 0.05). However, for set B, there was a statistically

significant effect (F(2, 48) = 15.07,p < 0.01). In thiscase, subjects showed higher weights on the first dimen-sion (flight function) than on either of the other twodimensions. These higher weights were confirmed statis-tically by using Bonferroni post hoc comparisons (seeCliff 1987). While dimension 1 was significantly differ-ent from dimensions 2 and 3 (F(1, 24) = 19.28,p < 0.01and F(1, 24) = 22.82,p < 0.01, respectively), no statisti-cally significant difference was found between dimen-sions 2 and 3 (F(1, 24) = 4.33,p > 0.05). Average subjectweights for each dimension and associated standard devi-ations are shown in table 5.

Multidimensional Preference Analyses

In an MDPREF analysis, preference of each subjectfor the stimuli is indicated by a vector in the spatial plot.Projecting the stimulus points onto the vector would rep-resent the relative preference for the stimuli. Further-more, the cosine of the angle subtended between a vectorand a dimension will provide the correlation of a sub-ject’s preferences with that particular dimension. If thespatial dimensions for the group data are interpretable, asthey frequently are, then differences among subjects in

Figure 10. Subject plot for set A similarity ratings.

Flight function Flight action

Samplerate

22

their use of dimensions can be inferred by the location ofsubject vectors within the space.

For the MDPREF information prioritization analy-sis, substantial agreement existed among the subjects.For each of the four solutions considered here, over70 percent of the subject vectors fell between 45° aboveand below the first dimension. This position indicates arelatively high degree of agreement among subjectresponses. Figure 12 plots the set A data for the genericand takeoff conditions. In these plots, each subject is rep-resented as a point. Figure 13 shows the analogous datafor the set B stimuli. One can readily observe that thespread of points is tightly arranged around the firstdimension for both experimental conditions. The

abscissa may best be thought of as a consensus dimen-sion, which reflects an average ranking by a subject ofinformation elements.

Discussion

In examining the INDSCAL subject weights, differ-ences between sets A and B in the relative use of dimen-sions by pilots is not entirely clear. However, it isimportant to note that because the stimulus sets wereconstructed by using random selection, the particularcombination of items in set B may have led to a situationwhere one dimension became more relevant, interpret-able, or salient to the subjects. The anomalous results ofsets A and B may be due to the vagaries of the stimuli

Figure 11. Subject plot for set B similarity ratings.

Table 5. Averages and Standard Deviation of Subject Weights for Sets A and B

Dimension 1 Dimension 2 Dimension 3

Standard Standard StandardSet Mean deviation Mean deviation Mean deviation

A 0.295 0.075 0.333 0.107 0.354 0.122

B 0.433 0.121 0.338 0.094 0.271 0.107

Samplerate

Flight actionFlight function

23

selected for use. Furthermore, although the set B dimen-sions were statistically significant from one another, themain effect accounted for 38.6 percent of the variance.While this percentage is good, it is not overwhelming.Finally, it perhaps should not be surprising that the flightfunction dimension tended to dominate the other two forthe set B results, given the typical pilot training thatemphasizes aviation, navigation, and communication.

The MDPREF results confirm the earlier findingsthat (1) pilots are in good agreement about the relativepriority of the information elements they were asked tojudge and (2) the high percentage of variance in the first

(a) Generic.

(b) Takeoff.

Figure 12. Subject plots for set A ranking for generic and takeoffconditions.

dimension in the earlier MDPREF analyses is reflectedby the relatively tight spread of points around it in thisanalysis. These results, however, do underscore thepotential usefulness of a second dimension, in that agood portion of subjects are using both dimensions.

Also worth noting is the consistent ratings of the fouritems common to both data sets. Those items were“Pilot-Initiated Request or Message,” “Predicted/Estimated Wind for Descent,” “Selected Altitude,” and“Speed Restriction Data.” As can be seen in figures 2and 3, these items appear in the solution space atapproximately the same locations. This near identical

(a) Generic.

(b) Takeoff.

Figure 13. Subject plots for set B ranking for generic and takeoffconditions.

24

location would indicate that the pilots used the sameattributes when measuring information similarity andthat they ranked the common attributes consistently withthese attributes. Likewise, these terms were similarlyranked in the prioritization task. This similarity is appar-ent in their common placement and shift in the spatialplots from the MDPREF analyses. These findings againunderscore the utility of employing two separate data setsfor validation purposes.

Cognitive Processes Across Different Contexts

One hypothesis of interest in the study reportedherein was the potential for invariance of cognitive pro-cessing dimensions in different conditions. Althoughtakeoff was the specific condition selected here andclearly differs from other conditions (such as climb,descent, and approach) in terms of the information andthe actions required, it is representative of other flightconditions in the sense that specific actions need to beperformed at certain times and in a certain order. Thesame would hold true for any other selected flightcondition.

By invariance it is meant that the underlying cogni-tive process that is used by the pilot in accomplishing thetask does not change as a function of the condition inwhich it occurs. This is not to say that in the case of pri-oritization the relative order of information cannotchange as a function of context, but rather that the under-lying process generating that order does not changecontextually.

If, as discussed earlier, the dimensions obtained fromthe MDPREF analyses are representative of the pilot’scognitive processing, then the same dimensions shouldcorrelate closely across different conditions. Alterna-tively, dimensions reflecting different cognitive pro-cesses should still be expected to show less correlationsacross different conditions. Different cognitive processesshould also show a dissociation within a given contextualcondition. However, because the MDPREF analysis cal-culates principal components, and these are orthogonal,the correlation between dimensions is, by definition,zero.

This pattern of correlation was, in fact, observed inthe study reported herein. The average correlation for thesame dimensions between the generic and the takeoffconditions was 0.663. The average correlation for differ-ent dimensions between the generic and the takeoff con-ditions was−0.209. These results indicate that regardlessof the particular context in which prioritization occurred,the same cognitive processes (represented here by thetwo dimensions) operated in a similar fashion. The factthat the different dimensions show less correlation across

contexts lends additional credence to the conditionsbeing separate processes.

It is also important to note that, of the four stimuluselements common to both sets A and B, the relative loca-tion of three of them (“Pilot-Initiated Request or Mes-sage,” “Selected Altitude,” and “Speed RestrictionData”) showed only minor changes across the genericand the takeoff conditions, which can be seen in figures 6and 7. The item which showed the largest shift was“Predicted/Estimated Wind for Descent.” Here the infor-mation moved toward the far left (lower perceivedimportance) as would be expected because it has little, ifany, relevance for the takeoff condition.

Conclusions

The experiment reported herein used psychometricscaling and cluster analyses to determine how pilotsmentally categorize and prioritize flight-deck informa-tion. The techniques used herein are particularly robustin that they give subjects a great degree of freedom inresponding. This freedom has a clear advantage over tra-ditional experimental designs in that subjects are allowedto provide data as they perceive the situation, without theusual constraints placed upon them in a typical experi-mental task (e.g., only correct or incorrect responses). Itwas hoped that this would afford the maximum opportu-nity for exploring how pilots process flight-deck infor-mation with a minimum of artificial restrictions.

A disadvantage to the methods used herein is that byproviding the subjects with this freedom, individual dif-ferences among subjects may completely obscure theoverall results, which was an initial concern. As pointedout in the “Results” section, it did not prove to be a prob-lem. The results obtained for both the scaling and theclustering analyses were remarkably similar, and whilethere were some minor individual differences amongsubjects, the overall response patterns were quite similar.

Cognitive Processing of Flight-Deck Information

Results from the individual differences scaling anal-ysis (INDSCAL) revealed three dimensions along whichpilots categorized and prioritized flight-deck informa-tion. These included (1) the flight function that theinformation supports, (2) the perceived strategic and tac-tical nature of the information (referred to herein as flightaction), and (3) how frequently the pilot refers to the data(referred to herein as the sample rate). These same threedimensions were observed with different subjects anddifferent stimulus sets, which lends support for their sta-tistical stability.

These results provide important insight into thosecognitive factors involved in a pilot’s processing of

25

flight-deck data and suggest additional information thatmay be collected when evaluating a new flight deck orinterface. Such information could be gathered by stan-dardizing the obtained dimensions and treating them asadditional variables for early design analyses. Asdescribed in appendix A, a variety of options for usingthese dimensions exist.

Classification of Flight-Deck Information

Cluster results support the existence of at least fourdistinct categories of information. These categoriesinclude (1) aviation, (2) navigation, (3) communication,and (4) systems administration. These findings wereobserved across two essentially different stimulus setsand with two groups of subjects.

Based on the research discussed in the “Back-ground” section of the paper that demonstrates that infor-mation consistent with an individual’s categorizationscheme is responded to quicker and more accurately, theresults presented herein could have direct implicationsfor methods of presenting flight-deck information. Mostobvious is the possibility for separately displaying infor-mation within these observed categories. Such anapproach may lead to performance improvements bymaking the information cognitively consistent by pre-senting it according to the mental models the crew hasfor the information flow. While this presentation is donefor most displays on modern generation flight decks,these results could have significant bearing on how infor-mation is presented on new displays where a variety ofinformation could be presented (such as an electroniclibrary display). These issues will require further empiri-cal analysis to determine precisely how such presenta-tional issues may affect pilot performance.

Relative Priority of Flight-Deck InformationCategories

The results of the MDPREF (multidimensional pref-erence scaling) analyses provide insight into how pilotsprioritize categories of information. Although, as notedin the “Background” section, most prioritization is cur-rently done with procedural and emergency checklists,

the findings presented herein may help in understandingwhat categories of information pilots consider to beimportant under nominal conditions. The current resultsshow that the relative priorities pilots assign to variouscategories of information (for either generic or takeoffconditions) are

1. Aviation

2. Navigation

3. Communication


While this can provide a starting point for determin-ing how pilots view the relative priorities of differentinformation at a global level, it should not be taken as anabsolute ordering of flight-deck information categoriesbecause of the obvious effect context can have on the rel-ative priority of information.

Pilot Cognition Across Contexts

A central finding of the study presented herein, how-ever, is that while the relative priorities of different infor-mation may change, depending upon the situation, thecognitive processes determining how the information isordered are reasonably invariant with respect to context.This invariance was indicated by high positive correla-tions between the same MDPREF dimensions (whichpresumably reflect the same or similar underlying cogni-tive processes) and low correlations between the differ-ent MDPREF dimensions (which presumably reflectdifferent underlying cognitive processes) across the twocontextual situations.

While the dimensions obtained from the MDPREFresults address relative prioritization, particularly as itrelates to categories of information, further work needsto be done to determine and quantify the particularcomponents that these prioritization dimensions arecomprised.

NASA Langley Research CenterHampton, VA 23681-0001June 13, 1995

26

Appendix A

Example Application of INDSCAL Dimensions

The INDSCAL dimensions obtained in the experi-ment discussed herein may also be used in a predictivefashion for future applications. There are two possiblestrategies for developing scales for research, each ofwhich will be described below. Prior to this, however, itis useful to present the logical background for using theINDSCAL dimensions for cognitive analyses. For thefollowing example, refer to figure A1.

Consider a pilot’s cognitive state at timet1 to be rep-resented at some point P1 in the three-dimensionalINDSCAL plot as shown in figure A1. For example, apilot may be engaged in a tactical navigation activity thatrequires relatively frequent use of data. The pilot’s cog-nitive state at this particular point in time has been repre-sented in figure A1 as P1.

Assume now that a tactical aviation activity arises(such as a request from ATC to climb to a flight level of35000 ft). The question becomes, can the pilot engagedin the long-term navigation task effectively deal with thisnew task, or will imposition of the new task create a dis-

ruption in performance? This question bears on a centralissue in multiple-task performance research—namely,the extent to which different tasks may interfere, or con-versely, facilitate each other.

In the case described above, the issue of whatresources tasks require for adequate performanceemerges. First, the addition of a task from the same high-level category (e.g., an aviation task with an aviationtask) may lead to superior performance, if it is assumedthat the subject is primed for performing such a taskbecause they are currently performing tasks from thesame area. Alternatively, the addition of a task fromwithin the same category might lead to an overload situa-tion where performance actually declines because bothtasks could be assumed to require the same resources.Both empirical questions require further analysis.

An analogous series of questions emerges when theaddition of tasks from outside the same category is con-sidered. One question is, can a subject adequately time-share two tasks from different functions (e.g., an aviationtask with a navigation task) if they may be presumed torequire different resources? Another question is, mightthe tasks interfere with one another?

Figure A1. Hypothetical example of pilot’s cognitive state using INDSCAL dimensions.

P1

Samplerate

Flight actionFlight function

communicationcommunicationcommunicationcommunicationcommunicationcommunication

navigationnavigationnavigation

aviationaviationaviation

tacticaltactical

strategic

frequentfrequentfrequentfrequent

infrequentinfrequentinfrequentinfrequent

27

This example illustrates how the INDSCAL dimen-sions may be used for evaluating flight-deck designs.Because the pilot may be represented at any point in theflight by three INDSCAL dimensions and these dimen-sions relate to the pilot’s processing of flight-deck infor-mation, the issue of using the dimensions to determinewhether subsequent tasks will be compatible with on-going activities becomes quantifiable. Once appropriatevalidation has taken place, such scales could becomeextremely useful for flight-deck design. At least two pos-sibilities for validation present themselves.

First, the dimensional scales could be part of a tradi-tional task analysis, wherein a record of the informationthe operator used to accomplish the task could be accu-mulated on each of the three dimensions. Typically, taskanalyses examine the amount of time required to performa task and the total amount of time available for the givenperiod. While overload (the individual has too little timeto perform too many actions) can be detected in such asituation, and represents a proxy for workload, littleinformation is conveyed about the pilot’s cognitive pro-cessing from this methodology. Dimensional informationwould provide, at a particular point in the timeline, a rat-ing for the pilot. The only requirement would be that theanalyst have knowledge of the flight-deck informationused by the pilot for a given task. Currently, this is not aproblem because as part of any detailed task, the actions

(including display monitoring and MCDU interactions)are explicitly spelled out through scenario developmentwith experienced pilots. If scale measures could be cali-brated to identify high and low cognitive workload lev-els, such scale values could be used to estimate potentialcognitive over or underload situations.

In a second method, these scales could be usedjointly. This approach would be the more difficult of thetwo. Here, instead of presenting a task by task estimatealong each scale, the task loads could be legitimatelycombined mathematically and theoretically. This com-bining would probably be best accomplished by usingconjoint measurement techniques (Louviere 1988). Withthis approach, the analyst would be able to examine thedimensional scores for (1) a particular task or seriesthereof, (2) a defined mission segment (such as takeoff),and/or (3) the entire mission.

The above conditions suggest that the most logicalapproach for using the derived INDSCAL dimensions isas scales for pilots to evaluate and rate flight-deck infor-mation. Three INDSCAL dimensions (flight function,flight action, and sample rate) and their operational defi-nitions could be provided to pilots. Once becomingfamiliar with these scales, pilots could then rank a com-plete list of flight-deck information (such as the list usedto construct the data sets in the study presented herein)along those three dimensions.

28

Appendix B

Categorization and PrioritizationQuestionnaire

I. During the course of a flight you receive a variety ofdifferent types of information. Can you provide a listof general categories into which you place thesesources of flight deck information?

II. What factors would you consider to be of the mostimportance for prioritizing flight deck informationand of the least importance?

III. Are there any types of flight deck information whichyou would consider to have absolute levels of prior-ity (that is, you would attend to these things over anyother information or, conversely, ignore)?

IV. In normal flight, how do you go about prioritizinginformation?

29

References

Abbott, Terence S. 1993:Functional Categories for Future FlightDeck Designs. NASA TM-109005.

Anon. 1988: SAS/STAT® User’s Guide, Release 6.03 Edition,SAS Institute Inc.

Anon. 1992: Boeing Model 747-400—Operations Manual, Vol-ume 2. No. D6-30151-401, Boeing Commercial Airplanes.

Ashby, F. Gregory, ed. 1992:Multidimensional Models of Percep-tion and Cognition. Lawrence Erlbaum.

Ashby, F. Gregory; and Maddox, W. Todd 1993: RelationsBetween Prototype, Exemplar, and Decision Bound Models ofCategorization.J. Math. Psychol., vol. 37, pp. 372–400.

Braune, Rolf J.; Hofer, Elfie F.; and Dresel, K. Michael 1991:Flight Deck Information Management—A Challenge to Com-mercial Transport Aviation. Proceedings of the SixthInternational Symposium on Aviation Psychology, Volume 1,R. S. Jensen, ed., Ohio State Univ., pp. 78–84.

Carroll, J. Douglas 1972: Individual Differences and Multi-dimensional Scaling. Multidimensional Scaling: Theory andApplications in the Behavioral Sciences, Roger N. Shepard,A. Kimball Romney, and Sara Beth Nerlove, eds., SeminarPress, pp. 105–155.

Carroll, J. Douglas; and Chang, Jih-Jie 1970: Analysis of Individ-ual Differences in Multidimensional Scaling Via an N-WayGeneralization of Eckart-Young Decomposition.Psychom.,vol. 35, no. 3, pp. 283–319.

Cliff, Norman 1987:Analyzing Multivariate Data. Harcourt BraceJavanovich, Inc.

Dornheim, Michael A. 1992: Changes Hike MD-11 Training Suc-cess. Aviation Week & Space Technol., vol. 137, no. 21,pp. 93–95.

Edwards, Ward 1987: Decision Making.Handbook ofHuman Factors, Gavriel Salvendy, ed., John Wiley & Sons,pp. 1061–1104.

Glass, Arnold Lewis; and Holyoak, Keith James 1986:Cognition.Second ed., Newberry Award Records, Inc.

Kruskal, Joseph B; and Wish, Myron 1978:MultidimensionalScaling. Sage Publ.

Louviere, Jordan J. 1988:Analyzing Decision Making—MetricConjoint Analysis. Sage Publ.

Nosofsky, R. M. 1984: Choice, Similarity, and the Context Theoryof Classification.J. Exp. Psychol.: Learn, Mem., & Cognit.,vol. 10, pp. 104–114.

Nosofsky, R. M. 1986: Attention, Similarity, and theIdentification-Categorization Relationship.J. Exp. Psychol.:Gen., vol. 115, pp. 39–57.

Nosofsky, Robert M. 1992: Similarity Scaling and Cognitive Pro-cess Models.Annu. Rev. Psychol., vol. 43, pp. 25–53.

Posner, M. I.; and Keele, S. W. 1970: Retention of Abstract Ideas.J. Exp. Psychol., vol. 83, no. 2, pp. 304–308.

Reed, Stephen K. 1972: Pattern Recognition and Categorization.Cogn. Psychol., vol. 3, pp. 382–407.

Ricks, W. R. 1994:PWC—Pairwise Comparison Software: Soft-ware Program for Pairwise Comparison Task for Psychomet-ric Scaling and Cognitive Research. LAR-15143.

Ricks, Wendell R.; Jonsson, Jon E.; and Rogers, William H. 1994:Cognitive Representations of Flight-Deck InformationAttributes.Int. J. Aviat. Psychol., vol. 4, no. 1, pp. 65–83.

Ricks, Wendell R.; Abbott, Kathy H.; Boucek, George; Jonsson,Jon; and Rogers, William H. 1991: Information Managementfor Commercial Aviation—A Research Perspective.Proceed-ings of the Sixth International Symposium on Aviation Psychol-ogy, R. S. Jensen, ed., Ohio State Univ., pp. 60–65.

Rips, L. J.; Shoben, E. J.; and Smith, E. E. 1973: Semantic Dis-tance and Verification of Semantic Relations.J. Verbal Learn.& Verb. Behav., vol. 12, no. 1, pp. 1–20.

Romesburg, H. Charles 1984:Cluster Analysis for Researchers.Wadsworth, Inc.

Sarter, Nadine B.; and Woods, David D. 1994: Pilot InteractionWith Cockpit Automation II: An Experimental Study of Pilots’Model and Awareness of the Flight Management System.Int.J. Aviat. Psychol., vol. 4, no. 1, pp. 1–28.

Shepard, Roger N. 1987: Toward a Universal Law of Gener-alization for Psychological Science.Science, vol. 237,pp. 1317–1323.

Wiener, Earl L. 1989:Human Factors of Advanced Technology(Glass Cockpit) Transport Aircraft. NASA CR-177528.

Form ApprovedOMB No. 0704-0188

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources,gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of thiscollection of information, including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 JeffersonDavis Highway, Suite 1204, Arlington, VA 22202-4302, and to the Office of Management and Budget, Paperwork Reduction Project (0704-0188), Washington, DC 20503.

1. AGENCY USE ONLY (Leave blank) 2. REPORT DATE 3. REPORT TYPE AND DATES COVERED

4. TITLE AND SUBTITLE 5. FUNDING NUMBERS

6. AUTHOR(S)

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES)

11. SUPPLEMENTARY NOTES

8. PERFORMING ORGANIZATIONREPORT NUMBER

10. SPONSORING/MONITORINGAGENCY REPORT NUMBER

12a. DISTRIBUTION/AVAILABILITY STATEMENT 12b. DISTRIBUTION CODE

13. ABSTRACT (Maximum 200 words)

14. SUBJECT TERMS

17. SECURITY CLASSIFICATIONOF REPORT

18. SECURITY CLASSIFICATIONOF THIS PAGE

19. SECURITY CLASSIFICATIONOF ABSTRACT

20. LIMITATIONOF ABSTRACT

15. NUMBER OF PAGES

16. PRICE CODE

NSN 7540-01-280-5500 Standard Form 298 (Rev. 2-89)Prescribed by ANSI Std. Z39-18298-102

REPORT DOCUMENTATION PAGE

August 1995 Technical Paper

Cognitive Models of Pilot Categorization and Prioritization of Flight-DeckInformation WU 505-64-13-22

Jon E. Jonsson and Wendell R. Ricks

L-17463

NASA TP-3528

Jonsson: McDonnell Douglas Aerospace-West, Long Beach, CA; Ricks: Langley Research Center, Hampton, VA.

In the past decade, automated systems on modern commercial flight decks have increased dramatically. Pilots nowregularly interact and share tasks with these systems. This interaction has led human factors research to direct moreattention to the pilot’s cognitive processing and mental model of the information flow occurring on the flight deck.The experiment reported herein investigated how pilots mentally represent and process information typically avail-able during flight. Fifty-two commercial pilots participated in tasks that required them to provide similarity ratingsfor pairs of flight-deck information and to prioritize this information under two contextual conditions. Pilots pro-cessed the information along three cognitive dimensions. These dimensions included the flight function and theflight action that the information supported and how frequently pilots refer to the information. Pilots classified theinformation as aviation, navigation, communications, or systems administration information. Prioritization resultsindicated a high degree of consensus among pilots; while scaling results revealed two dimensions along whichinformation is prioritized. Pilot cognitive workload for flight-deck tasks and the potential for using these findings tooperationalize cognitive metrics are evaluated. Such measures may be useful additions for flight-deck human per-formance evaluation.

Human factors; Cognition; Pilot performance; Categorization 32

A03

NASA Langley Research CenterHampton, VA 23681-0001

National Aeronautics and Space AdministrationWashington, DC 20546-0001

Unclassified–UnlimitedSubject Category 01Availability: NASA CASI (301) 621-0390

Unclassified Unclassified Unclassified

cognitive models of pilot categorization and prioritization of … · 2013-10-03 · national...

Documents