eye tracking and performance evaluation · – important area of hci research where eye tracking...
TRANSCRIPT
Eye Tracking and
Performance Evaluation
Automatic Detection of User Outcomes
Allen Harper Proposal Talk
Department of Computer Science City University of New York Graduate Center
August 22, 2014
Outline • Part 1: Novel Framework for Eye Tracking • Part 2: Introduce Research Questions • Part 3: Pilot Study • Part 4: Machine Learning Analysis • Part 5: Proposed Work and Timeline • Part 6: Discussion
2
Wouldn’t it be nice if computer systems could…
• Automatically and unobtrusively determine if a user belongs to a low or high task performance group
• Using only eye-movement data
3
Why Eye Movement Data? • Eye movements are the overt expression of
our visual attention system – and therefore related to covert cognitive
activities • Eye tracking metrics can then be seen as
proxy variables – from which cognitive qualities can be inferred
4
Traditional Uses of Eye Tracking • HCI-Eye Tracking (HET) perspective • Descriptive Tool • Reports percentage and distribution of
visual attention across a visual stimuli
5
Novel Use for Eye Tracking • Eye Tracking-Performance Connection
(EPC) perspective • Predictive Role • Combined with machine learning techniques
in order to classify users into performance groups
6
Ideal EPC Framework
• Given eye movement data of an unknown subject – accurately predict their
performance level
7
?
Level of performance
Eye
Met
rics
D
Goal of Literature Survey • What can we learn about conducting eye
tracking experiments – Which reveal strong relationships between eye
movement patterns and user performance • “Metric hunt”
– Locate candidate eye tracking measures consistently correlated with user performance
• Discovering which task types improved the strength of the connection between eye movement patterns and performance
8
Experimental Issues
9
Level of performance
Eye
Met
ric
A
Level of performance
Eye
Met
ric
B
Level of performance
Eye
Met
ric
C
Level of performance
Eye
Met
ric
D
Low dispersion Task too easy
Eye movement not necessary
EPC
Hints of EPC • Eye movements Performance
• When… – Users viewed consistent information content – Visual stimuli unaltered during the experiment – Eye tracking aligned with task execution – Performance measure aligned with task execution – Tasks required “full” attention of users
10
Literature Survey Summary
11
Lessons Learned • EPC experiments must have controls for:
– Content Homogeneity – Visual Homogeneity
• And respect the alignments of task execution and performance measurement
• Provide an appropriately difficult user task
12
Take Home • Eye tracking community has found it difficult
to consistently and reliably connect eye movements with user performance
• First step: apply EPC concepts to develop a baseline experiment—EPC verification
• Second step: is to check the validity of each EPC component individually
13
Outline • Part 1: Novel Framework for Eye Tracking • Part 2: Introduce Research Questions • Part 3: Pilot Study • Part 4: Machine Learning Analysis • Part 5: Proposed Work and Timeline • Part 6: Discussion
14
RQ1: Establishing EPC Baseline • Control for:
– Content homogeneity – Visual homogeneity – Performance and task alignments
• Predicted Outcome: – Successful classification of subjects
into performance groups
15
RQ2: Content Homogeneity • Relax control of:
– Content homogeneity restrictions
• While still controlling for: – Visual homogeneity – Performance and task alignments
• Predicted Outcome: – Decline in model accuracy
16
RQ3: Visual Homogeneity • Relax control of:
– Visual homogeneity • While still controlling for:
– Content homogeneity – Task and performance alignments
• Predicted Outcome: – Decline in model accuracy
17
RQ4: Alignments • Relax control of:
– Alignment restrictions • While still controlling for:
– Content homogeneity – Visual Stimuli
• Predicted Outcome: – Decline in model accuracy
18
Outline • Part 1: Novel Framework for Eye Tracking • Part 2: Introduce Research Questions • Part 3: Pilot Study • Part 4: Machine Learning Analysis • Part 5: Proposed Work and Timeline • Part 6: Discussion
19
Pilot Study • Goal was to establish an EPC baseline • Operationalize our EPC concepts • Select application domain • Choose user interface style
– Correct stimuli for visual homogeneity • Produce information content
– Correct for content homogeneity • Develop performance measure
20
Application Domain • Education Learning Systems
– Following the current trend towards increased use of online learning materials both within academia and from external services
– Important area of HCI research where eye tracking can have a significant impact on the improvement of educational methodologies
21
Interface Style • Special characteristics
– Multiple regions • Visual dispersion
– Complex information • Foveal Stress
– Dynamic information • Speed stress
22
Educational Learning System
Correct for Visual Homogeneity • All slides have one line titles and three
bullets • Fonts stay the same in each AOI • No use of color, markup (e.g., italics, bold) • No use of animations • Stationary speaker with identical
background for all stories
23
Developing Information Content • Obscure Wikipedia biographies
– In order to avoid pre-exposure to material • Limited to five minutes in duration
– Both IRB constraints and user fatigue • Each story contains ten PowerPoint slides
– 30s timing intervals per slide and each slide contains four 7.5s subintervals
24
Correct for Content Homogeneity • Discovered significant differences in the
distribution of information across slides • Developed a normalization rubric using
2:1:1:1 ratios for names, dates, numbers, ideas
• Applying a Latin Square design to the set – {Name, Name, Number, Date, Idea}
25
Normalization Rubric
26
Performance Measure Design • Information-recall questionnaire • Engineered to match the same patterns as
found in information content • Limited to 25 items
– Both IRB constraints and subject fatigue
27
Design of Performance Measure
28
Camtasia Studio Layout
29
Outline • Part 1: Novel Framework for Eye Tracking • Part 2: Introduce Research Questions • Part 3: Pilot Study • Part 4: Machine Learning Analysis • Part 5: Proposed Work and Timeline • Part 6: Discussion
30
Machine Learning Analysis • Due to a lack of success found in eye
tracking literature using traditional statistical approaches we adopted a machine learning approach
31
Research Areas • Interactive Information Retrieval (IIR)
– Eye movements indicate relevance of items to the search query term
• Content-Based Image Retrieval – Similar to IIR, but in terms of image search
• Object Relevance in 3-D environments – Search occurs in more naturalistic settings
32
Machine Learning Literature
33
Feature Extraction
34
Feature Groups • Content-dependent Fixation-based • Content-dependent Dwell-based • Content-independent Fixation-based • Content-independent Dwell-based • Distance measures • Completeness of Scan • Eye shape characteristics
35
Data Processing Steps • Eye tracker produces proprietary text format • Python scripts read and convert proprietary
file structure to Excel spreadsheet • 1,197 features constructed
36
Machine Learning Platform • Waikato Environment for Knowledge
Analysis (WEKA) • Availability of current algorithms • Ease of use either from GUI or command
line interface
37
Machine Learning Tools • Machine Learning Algorithms
– Naïve Bayes – Logistic Regression – Support Vector Machine (SVM) – J48 – Random Forest
• Attribute Selection Methods – Best First Forward – Linear Forward Selection
38
Evaluation Metrics • Accuracy: The percentage of predictions
that are correct • Precision: The percentage of positive
predictions that are correct • Recall: The percentage of positive labeled
instances that were predicted as positive • F-Measure: Harmonic mean of precision
and recall
39
Pilot Study Results
40
Outline • Part 1: Novel Framework for Eye Tracking • Part 2: Introduce Research Questions • Part 3: Pilot Study • Part 4: Machine Learning Analysis • Part 5: Proposed Work and Timeline • Part 6: Discussion
41
Proposed Work • Having established EPC baseline result • Test the limits of
– Content Homogeneity – Visual Homogeneity – Alignments of performance and task
42
RQ2: Testing Content Homogeneity
• Relax control of: – Content homogeneity restrictions
• While still controlling for: – Visual homogeneity – Performance and task alignments
• Predicted Outcome: – Decline in model accuracy
43
RQ3: Testing Visual Homogeneity • Relax control of:
– Visual homogeneity • While still controlling for:
– Content homogeneity – Task and performance alignments
• Predicted Outcome: – Decline in model accuracy
44
RQ4: Testing Alignments • Relax control of:
– Alignment restrictions • While still controlling for:
– Content homogeneity – Visual Stimuli
• Predicted Outcome: – Decline in model accuracy
45
Timeline
46
Outline • Part 1: Novel Framework for Eye Tracking • Part 2: Introduce Research Questions • Part 3: Pilot Study • Part 4: Machine Learning Analysis • Part 5: Proposed Work and Timeline • Part 6: Discussion
47
Expected Contributions • Central goal of our research
– Advance the understanding of how eye movement patterns are related to user performance during task execution
48
Contribution 1 • Preliminary Result (RQ1)
– Pilot study indicates that in a rigorously designed EPC experiment verification experiment it is possible to use eye movement metrics in order to classify users into performance groups
– This result opens new possibilities for the types of questions that can be addressed within the field of eye tracking
49
Contribution 2 • EPC Framework
– Provides guidance for eye tracking researchers in the design and implementation of experiments attempting to connect eye movements and performance
– In addition, RQ2-4 will provide additional data on this relationship is impacted by changes in experimental design
50
Contribution 3 • Machine Learning Results
– Provide guidance for best practices in terms of data handling as well as comparative data on model accuracies
51
Contribution 4 • Availability of experimental materials
– By making the visual stimuli, code, and all auxiliary research materials available to the research community this project will facilitate similar research
52
Future Applications • Improving Usability Evaluation
– Automating evaluations – Removing the time consuming – Reducing the need for expert reviews
• Adaptive User Interfaces – Interfaces could employ EPC to provide a
greater user personalization based on the detected performance level
53
THANK YOU!
QUESTIONS?