Environmental Research Section A 85, 53}57 (2001)doi:10.1006/enrs.2000.4071, available online at http://www.idealibrary.com on

Feasibility of the Use of Eye Movement Data in the Analysis ofNeurobehavioral Test Performance

Richard Stephens

Psychology Department, Keele University, Staffordshire, ST5 5BG, United Kingdom

Received August 9, 1999; published online December 4, 2000

Better understanding of the functions and abil-ities underlying some neurobehavioral tests wouldassist the assessment of the consequences to thehealth or well-being of exposed populations forwhich decrements in such tests have been found.Eye movements, which can be useful in evaluatingspeciAc theories of visual and cognitive processes,were measured while a single participant com-pleted the NES Symbol-Digit Substitution test.Relationships between the durations and thefrequencies of eye Axations within different parts ofthe test material and overall test performance wereexamined. The overall pattern of eye-movement be-havior observed was as predicted: Axations withinthree regions of visual interest were made, usuallyin the order lower matrix, upper matrix, keyboard.Both the upper matrixAxation latency and the num-ber of regional Axations made per item were foundto be strongly associated with overall item responselatency. This suggests that speed and efAciency ofsearching and the ability to maintain concentrationappeared to be key attributes related to test perfor-mance for this participant. For suitable tests, eye-movement data may be useful for assessing andmeasuring the latencies of subtask components ina real test situation. Monitoring eye movement rep-resents an alternative methodology to using subtest composites, or factor analysis, as a means oftapping into the functions underlying neuro-behavioral test performance. ( 2000 Academic Press

INTRODUCTION

Psychological performance tests have been usedsince the mid-1960s in occupational and environ-mental health toxicology (Anger et al., 1993), in whathas become known as neurobehavioral toxicology.The mental functions underlying many of thetests favored in neurobehavioral studies are not wellunderstood, presenting dif7culties to the interpreta-

53

tion of measured de7cits in such tests (Stephens andBarker, 1998). Recently developed test batteriesbased on cognitive psychological theory may beeasier to interpret because they speci7cally concen-trate more on underlying functions. Nevertheless,moves during the 1980s to standardize the tests usedin neurobehavioral toxicology studies (Johnson,1987) have led to a glut of published studies thatemploy ‘‘traditional’’ tests. Better understanding ofthe functions and abilities underlying these neuro-behavioral tests may therefore be useful for the pur-poses of interpreting many neurobehavioral studiesin the literature.

Substitution tests, derivatives of the WeschlerAdult Intelligence Scale Digit Symbol Test (dis-cussed in Lezak, 1995), are recommended for thestandardized World Health Organization Neur-obehavioral Core Test Battery (Johnson, 1987) andappear extensively in the neurobehavioral litera-ture. A commonly used example of such a test is theSymbol-Digit Substitution test of the Neuro-behavioral Evaluation System (NES) (Baker et al.,1985). In this computer-administered test two ma-trices are presented on screen. An upper matrixcontains nine symbols, each of which is paired witha digit between one and nine. The participant mustcomplete (temporally from left to right) the emptyboxes in the bottom row of a lower matrix with thecorrect paired digit for each symbol. These testsrequire several stages of processing to complete eachitem, but these stages are not re8ected in the tests’standard outcomes, which are expressed as latenciesper item, or number of items completed withina time limit. This is likely to be a factor contributingto the ambiguity over what these tests reallymeasure, illustrated by the various terms usedto describe performance on substitution tests:psychomotor performance, motor persistence,sustained attention, response speed, visuomotor

0013-9351/00 $35.00Copyright ( 2000 by Academic Press

All rights of reproduction in any form reserved.

54 RICHARD STEPHENS

coordination, and perceptual organization (Lezak,1995). In the light of such variety it is easy to seehow interpretative dif7culties may arise for thoseattempting to assess the consequences to the healthor well-being of exposed populations showing decre-ments in tests of this type. Hence, from an inter-pretative standpoint, further understanding of theunderlying functions of substitution tests would beuseful.

Laux and Lane (1985) investigated the under-lying functions of substitution tests by splittingthe task down into a series of discrete, self-contained, subtests, increasing in complexity fromsimple reaction time to the full test. Relationshipsbetween latencies to complete the subtests andthe full test were examined. Though useful, thisapproach is not altogether satisfactory. One reasonis the assumption of ‘‘perfect’’ performance; whathappens if or when participants deviate from thesubtask sequence, for example, through loosingconcentration or making an error? A second reasonis that no matter how well de7ned and executedthe process of breaking down the task into smallertests, performance on the subtests will only ever bean approximation of 8uid performance during thefull test, as it is administered during neurobehavioralstudies.

A different approach that may allow monitoring ofsubtasks in a real test situation is to record partici-pants’ eye movements during substitution test com-pletion. Eye movements (saccades) and pauses(7xations) have been studied for over 100 years, andit is believed that these data can, under certainconditions, be useful in evaluating speci7c theoriesof visual and cognitive processes (Viviani, 1990).Just and Carpenter (1976) suggest three conditionsunder which eye 7xations may be an accurate re8ec-tion of what is being processed: (1) that a participantconcurrently completes a task requiring informationfrom the visual environment to be encoded and pro-cessed; (2) that the task sets goals for the subject(i.e., requires responses to be made by the subjectthroughout); and (3) that the task is speeded, todiscourage extraneous processing and concomitantextraneous 7xations. Substitution tests appear toful7ll these conditions, since they involve a continu-ing process of searching for, and encoding, informa-tion from different parts of the visual 7eld andrequire responses to be made throughout the testand since participants are instructed to go as quicklyas possible.

A study was therefore conducted measuring eyemovements during completion of the computer-ad-ministered NES Symbol-Digit Substitution test. The

aim of the study was to assess whether eye move-ment data may be used to infer information concer-ning substages of processing during neurobehavioraltest performance. Of particular interest waswhether the pattern of the participant’s eye move-ments during the test corresponded with the subtaskstructure put forward by Laux and Lane (1985). Ifthis was shown to be the case, then relationshipsbetween the latencies and serial order of completingthese subtasks and overall item response latenciescould be examined.

METHODS

One participant, a psychologist from the labora-tory, completed the NES Symbol-Digit Substitutiontest in conjunction with the Applied Science Labor-atories Series 4000 Head Mounted Eye TrackingSystem, with control unit 4000 SU (ASL, 1993). Thisequipment accurately measures a freely moving par-ticipant’s line of gaze in relation to what they arelooking at, using the double Purkinje method. Theleft eye is illuminated by a beam of near infraredlight and 7lmed with a headband-mounted camera.A computer-controlled control unit processes the eyecamera signal to extract the positions of the pupiland the re8ection of the light source on the corneaand, from these, computes the line of gaze. Line ofgaze is output as a set of cross hairs superimposed onthe image from a second camera mounted on theheadband, which portrays the scene that the partici-pant is viewing. This output was videotaped andsubsequently analyzed at slowed speed. Figure 1shows a video still from the output of a participantcompleting the computerized NES Symbol-DigitSubstitution test.

The eye-movement data were classi7ed in terms of7xations within three regions of interest within thetest material: the lower matrix containing the ninetarget symbols and spaces to key in responses; theupper matrix containing the same nine symbols andtheir paired digits; and the keyboard of the test-administering computer. The onset latencies of eachregional 7xation and motor response were recordedusing ‘‘The Observer’’ (Noldus Information Techno-logy, 1995) software. Frequencies and durations ofthe regional 7xations were calculated, and thesedata were analyzed using the software package‘‘SPSS for Windows.’’

RESULTS

Table 1 shows the participant’s test score, themean durations of regional 7xations within each of

FIG. 1. Video still from the eye-tracker headband-mounted‘‘scene camera’’ showing a user’s eye view of the NES Symbol-Digit Substitution test with line of gaze superimposed as cross-hairs.

USE OF EYE MOVEMENT DATA IN NEUROBEHAVIORAL TESTING 55

the three regions, and the mean frequency of re-gional 7xations per item. The participant looked no-where other than at elements of the test material(i.e., the computer screen and keyboard) throughoutthe test and made no erroneous key presses during

TABLE 1NES Symbol-Digit Substitution Test Score, Eye-Move-

ment Data, and Pearson Correlation CoefAcient for Rela-tionship Between These

ValueVariable (SD; range) r

Test scoreMean latency per item (s)a 1.75 ;

(0.3; 1.6}2.2)

Fixation within lower matrix(encoding target)

Mean latency per item (s) 0.45 0.28(0.15; 0.08}0.84)

Fixation within upper matrix(target search/encode matched digit)

Mean latency per item (s) 0.72 0.78*(0.26; 0.12}1.20)

Fixation within keyboard(locating digit key on keyboard)

Mean latency per item (s) 0.51 0.58*(0.15; 0.20}1.00)

Frequency of regional 7xationsMean frequency per item 3.33 0.56*

(0.85; 3}7)

aFor the 7nal eight items in the best two trials (of nine items).*P(0.05.

the test. The pattern of regional 7xations was asexpected: for most items, the participant 7xated 7rstwithin the lower matrix (to encode the target sym-bol), then 7xated within the upper matrix (to searchfor the encoded target symbol and encode its paireddigit), and 7nally 7xated upon the computer key-board to locate the correct key to make the response.There was no statistically signi7cant difference be-tween the times spent 7xating within the differentparts of the test material, though the participanttended to spend more time 7xating within the uppermatrix. Correlation analyses found statistically sig-ni7cant positive relationships between both uppermatrix 7xation latency and keyboard 7xationlatency and overall item response latency (seeTable 1). The correlations were positive; that is, thelonger the regional 7xation latency, the longer theoverall latency for that item. There was no statist-ically signi7cant correlation between lower matrix7xation latency and item response latency. Thehighest correlation was between item responselatency and upper matrix 7xation latency. For justunder one-7fth of items (8 of 45 items), more thanthree regional 7xations were required. There wasa statistically signi7cant positive correlation be-tween the frequency of regional 7xations andthe response latency per item. This suggests thatthe more 7xations, the longer the response latencyto complete an item. The mean difference betweenthe key-press latencies obtained from the analysisof the eye-tracker scene camera video and thekey-press latencies recorded by the test-administer-ing computer was !0.0002 s. Differences werewithin one frame for 73% of items (33 of 45), thoughdifferences in excess of two frames occurred for twoitems.

DISCUSSION

The overall pattern of eye-movement behavior ob-served was as predicted by Laux and Lane’s (1985)model. Hence 7xations within the three regions ofvisual interest were usually made in the order lowermatrix; upper matrix; keyboard. While 7xatingwithin these regions, it may be inferred that theparticipant was carrying out the following respectivecognitive tasks: target symbol encoding; encodedsymbol search and encoding the matched digit; re-sponse key search.

Searching for the encoded symbol/encoding thepaired digit took up the most time and was foundto be strongly associated with overall item responselatency. Unfortunately, the eye-tracking equipmentlacked the sensitivity to allow the recording of

56 RICHARD STEPHENS

separate times for searching for the encoded symboland encoding the matched digit. However, the SD ofthe mean of the upper matrix 7xation latency wasapproximately double that of the respective SDs forlower matrix 7xation latency and keyboard 7xationlatency. This extra variance occurs because the sym-bols in the upper matrix are arranged in a (pseudo)random order, which means that chance causessome symbols to be found more quickly than others.The strong in8uence of the search step of the processon the overall variance of upper matrix 7xationlatency indicates a similar in8uence on the positivecorrelation between upper matrix 7xation latencyand overall item response latency. This suggeststhat speed and ef7ciency of searching are importantunderlying functions of this test, for this participant.The corresponding subtask in Laux and Lane(1985) was also found to show a statistically signi7-cant association with overall response latency,though with a lower correlation coef7cient than thatobtained in the current study. Time to locate theresponse key was associated less strongly with itemresponse latency, while the target symbol encodingtime was not associated with item completionlatency.

Not all responses followed the lower matrix/uppermatrix/keyboard sequence. Sometimes the partici-pant would recheck the lower matrix before respon-ding or 7xate within the upper matrix, rather thanthe lower matrix, immediately after responding.Such nonsequential regional 7xations occurred in17.8% of items, and the number of regional 7xationsmade per item was strongly associated with theoverall latency to complete items. Hence, for thisparticipant, it appears that the NES Symbol DigitSubstitution test also tests the ability to maintainconcentration, keeping up an ef7cient rhythm ofworking, over a period of 2}3 min. Use of subtestcomposites in Laux and Lane (1985) precluded ex-amination of the effect of subtask sequence uponoverall test performance in that study. This is a sig-ni7cant advantage of using eye-movement data overthe previous methodology.

Some error was anticipated in latencies measuredfrom the eye-movement video, due to the interactionof the different sample rates of the technology used(test-computer screen refresh rate 50 ms; camerasample rate 20 ms). In comparing response latenciesmeasured using The Observer software with thosemeasured with the software on board the test-ad-ministering computer, there was found a verysmall mean difference between these, less than one-thousandth of a second. This suggests that latenciesmeasured with the eye-tracking technology were

reliable and suf7ciently accurate for the purposes ofthese analyses.

In conclusion, eye movement data allowed insightinto the subtask structure of the NES Symbol-DigitSubstitution test, enabling the monitoring of testperformance in a real test situation. This is an ad-vantage over a previous attempt to investigate theprocesses underlying substitution test performance,where subtest composites were used (Laux andLane, 1985). Measuring the duration and frequencyof 7xations within different regions of the testmaterial allowed conclusions to be drawn aboutthe relative importance of different subprocessesto overall test score. Speed and ef7ciency ofsearching and the ability to maintain concentrationappeared to be key attributes related to test perfor-mance for this participant. For suitable tests, usingeye-movement data appears to provide a useful,noninvasive tool for assessing and measuring thelatencies of subtask components in a real testsituation and represents an alternative methodologyto using subtest composites or factor analysis asa means of tapping into the functions underlyingtest performance.

The next step should be to conduct a groupstudy to test both the validity of these conclusionsand the logistics of measuring eye movements inan applied setting. Testing a control group, withoutthe eye tracker, would reveal whether wearingthe eye-tracking equipment disrupts test perfor-mance. Adopting frame by frame, rather than slowmotion analysis, may improve accuracy of recordingeye-movement events and may facilitate the record-ing of saccadic movements between the 7xationregions.

REFERENCES

Anger, W. K., Cassitto, M. G., Liang, Y. X., Amador, R.,Hooisma, J., Chrislip, D. W., et al. (1993). Comparison ofperformance from three continents on the WHO-recommendedNeurobehavioral Core Test Battery. Environ. Res. 62,125}147.

Applied Science Laboratories (ASL) (1993). ‘‘Head Mounted EyeTracking System Instruction Manual, Model 4000SU HMO,Manual Version 4.2.’’ Bedford, MS, MA.

Baker, E. L., Letz, R., and Fidler, A. T. (1985). A computer-administered neurobehavioral evaluation system foroccupational and environmental epidemiology. Rationale,methodology and pilot study results. J. Occup. Med. 27,206}212.

Johnson, B. L. (Ed.) (1987). ‘‘Prevention of Neurotoxic Illness InWorking Populations.’’ World Health Organization and Wiley,Chichester.

Just, M. A., and Carpenter, P. A. (1976). Eye 7xations and cogni-tive processes. Cogn. Psychol. 8, 441}480.

USE OF EYE MOVEMENT DATA IN NEUROBEHAVIORAL TESTING 57

Laux, L. F., and Lane, D. M. (1985). Information processing compo-nents of substitution test performance. Intelligence 9(2), 111}136.

Lezak, M. D. (1995). ‘‘Neuropsychological Assessment,’’ 3rd ed.Oxford Univ. Press, London.

Noldus Information Technology BV (1995). ‘‘The ObserverSystem for Collection and Analysis of Observational Data.Reference Manual.’’ Wageningen, The Netherlands.

Stephens, R., and Barker, P. (1998). The role of human neuro-behavioral tests in regulatory activity on chemicals. Occup.Environ. Med. 55(3), 210}214.

Viviani, P. (1990). Eye movements in visual search: Cognitive,perceptual and motor control aspects. In ‘‘Eye Movements andTheir Role in Visual and Cognitive Processes’’ (E. Kowler, Ed.),Elsevier, Amsterdam/New York.