Anthony J. HornofTim Halverson
University of OregonSponsored by ONR
Cognitive Strategies andEye Movements for Searching
Hierarchical Displays
2
• A hierarchical display motivates specific patterns of perceptual, cognitive, and motor processing.
• Cognitive modeling can be used to reveal and explain the cognitive processing and strategies that people use when searching a visual hierarchy.
• Eye tracking can be used to evaluate, validate, and refine cognitive models.
Three Main Points
3
A lack of a visual hierarchymakes it harder to find thingsAn example of a layout with no visual hierarchy:
4
A visual hierarchy aids visual search
5
Fleet Area Control and Surveillance FacilitySan Diego http://www.facsfacsd.navy.mil/
6
Cognitive modeling in HCI
eye movement processor
perceptualprocessing - foveal - peripheraleye
muscles
sensoryinfo
pattern matching - object features - visual structure
cognitiveprocessor
visual search strategies - expert vs. novice - hierarchical vs.
nonhierarchal - purely random
• Two main goals:1. Explain user behavior.2. Predict user behavior.
• Simulates perceptual-motor processes and cognitive strategies.
• A generic cognitive architecture for visual search:
7
• EPIC: Executive Process-Interactive Control• Kieras and Meyer (1997)• Captures human perceptual, cognitive, and motor
processing into a simulation framework• Constrains the models that can be built• Inputs into the architecture:• Task environment• Visual-perceptual features• Cognitive strategies
• Outputs from the running model:• Execution times• Trace of the processing• Eye movements
••
The EPIC Cognitive Architecture
8
The Experimental Task
Experimental Design• 2 X 3 design. Layouts were labeled or unlabeled.
Layouts had 2, 4, or 6 groups. Blocked by layout type.• Procedure: Study precue, click on precue, find target, click on target.• 16 participants, motivated to search quickly• Search and selection time recorded separately
1° ofvisual angle
A
B
C
D
E
F
9
Two search strategiesUnlabeledNoisy-systematic search
LabeledTwo-tiered search strategy
0
1000
2000
3000
4000
Sear
ch T
ime
(ms)
A B C D E FTarget Group
Unlabeled
1 group2 groups4 groups6 groups
Sear
ch T
ime
(ms)
A B C D E FTarget Group
Labeled
2 groups4 groups6 groups
Observed
Predicted
10
The experiment run with eye tracking• Identical design, but eye movements were recorded• LC Technologies Eyegaze System (60 Hz, pupil-
center and corneal-reflection)• Dispersion-based fixation identification (minimum
fixation 100 ms, deviation threshold 0.5° visual angle)• Cleaned up data using required fixation locations
(Hornof and Halverson, 2002)
Eye Tracking in HCITwo general uses:
Input device Retrospective analysis
(Jacob and Karn, 2003)
11
Eye movements for a couple trialsUnlabeled Layout
Visualized with VizFix (Google search on “VizFix”)
12
Eye movements for a couple trialsLabeled Layout
Visualized with VizFix (Google search on “VizFix”)
13
Eye movementsAcross All Layouts Observed PredictedFixations per trial (+) 7.4 7.9Fixation duration (+) 264 ms 228 msNumber of scan paths Many OneAnticipatory fixations (+) Yes YesRespond to layout onset (+) Yes YesIgnore white space (+) Yes YesIgnore text shape (+) Yes YesOvershoot the target Rarely Yes
Unlabeled LayoutsFixations per group 2.1 1.1Groups revisited per trial 0.69 4.4Items examined per fixation (+) 2.4 2.6
Labeled LayoutsUse group labels (+) Yes YesGroups revisited per trial 0.29 1.2
14
Across All Layouts Observed PredictedFixations per trial (+) 7.4 7.9Fixation duration (+) 264 ms 228 msNumber of scan paths Many OneAnticipatory fixations (+) Yes YesRespond to layout onset (+) Yes YesIgnore white space (+) Yes YesIgnore text shape (+) Yes YesOvershoot the target Rarely Yes
Eye movements
15
0
5
10
15Fi
xatio
ns P
er T
rial
A B C D E FTarget Group
Unlabeled
1 group2 groups4 groups6 groups
Labe
led,
6 G
roup
s
A B C D E FTarget Group
Labeled
Observed
Predicted
Fixations Per Trial
16
Scan PathsA
B
C
D
E
F
A
B
C
D
E
F
A
B
C
D
E
F
Labeled:Unlabeled:
19%30%
12%12% 17%
Observed
A
B
C
D
E
F
Labeled:Unlabeled:Predicted
12%75%70%
6%5%7%
0%0%0%
0%1%
17
How People Started the Search
0
50
100
150
200
Coun
t
P 1 2 3 4 5 6 7 8 9Layout Position
Anticipatory Fixations
0 100 200 300 400 500 600Time after precue appeared (ms)
400
300
200
100
0
Second FixationsRespond to layout onsetAnticipatory
Fixations
18
Unlabeled Layouts Observed PredictedFixations per group 2.1 1.1Groups revisited per trial 0.69 4.4Items examined per fixation (+) 2.4 2.6
Labeled LayoutsUse group labels (+) Yes YesGroups revisited per trial 0.29 1.2
Eye Movements
19
• Hierarchical Displays• Cognitive Modeling• Eye Tracking
What Have We Learned About...
20
Hierarchical DisplaysPatterns of Behavior
Use a visual hierarchy
Examine multiple items with a single fixation
Jump over white space
Anticipate visual locations
Prime for onset
Design Recommendations
Support a multitiered search
Facilitate simultaneous foveal coverage
Use white space
Provide consistent layouts
Very fast response times
21
• Many aspects of the strategies, models, and architecture appear to be correct:• More than one item with each fixation• Global search strategies• Ignore shape• Anticipatory fixations
• The data also identify a number of improvements to made:• Wider range of scanpaths• Stopping on the target
Cognitive Modeling
22
• Examines specific details of the models:• Strategies• Perceptual-motor processes
• Better than open-ended questions such as:“How do people move their eyes in Task X?”
• Synergy between eye tracking and modeling• A good way to proceed:• Identify theories established with standard
measures (speed, accuracy, clickstreams)• Evaluate the theory with eye tracking.
Eye tracking
23
• Evaluate aspects of the Cognitive Walkthrough for the Web (Blackmon, et al. 2003) using eye tracking.
• “Give meaning” to cognitive modeling of visual search.
• Predictive visual search tool
Future Work