2016-04-27 research seminar, 2nd presenter
TRANSCRIPT
Avar Pentel PhD student, Tallinn University, School of Digital TechnologiesSupervisor: Tobias Ley
• Area of reasearch:– User profiling– Detecting users motor behaviour via
standard input devices such as keyboard and mouse
– and connecting it to users demographic data, emotions, etc
Title of Presentation
Employing Think-Aloud Protocol to Connect User Emotions and Mouse
Movements*
* Based on the paper (2015) with the same titleavailable at IEEE Xplore digital library: http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=7387970
Goal
To predict user’s emotional state by analyzing mouse movements logs
Outline of the Presentation
• Related work• Experimental setup
– Data collection procedure– Associating tasks to emotional states– Features– Machine learning
• Results• Conclusion
Related work
• Special equipment• Small samples• Specific tasks, no general link between
emotion and mouse movement is studied
Data Collection Procedure
Idea from Christmas Calendar
Data Collection Procedure
Data collection
• The game has collected data about:–Each mouse click.–Time when button was clicked.–All mouse movements with
timestamps
Data Collection
Example of mouse log (x,y, timestamp)
70,34,1365354712662,74,34,1365354713453,78,36,1365354713488,81,38,1365354713517,85,44,1365354713537,87,50,1365354713560,89,53,1365354713573,90,58,1365354713598,91,63,1365354713622,92,59,1365354713903,95,53,1365354713927,97,49,1365354713942,100,45,1365354713954,103,40,1365354713976,106,36,1365354714001,110,34,1365354714049,105,33,1365354714390,100,30,1365354714414,96,27,1365354714439,93,25,1365354714475,93,21,1365354714561,95,18,1365354714598,98,15,1365354714622
Data
• 916 game sessions played by 282 individual users. Participants were between 12 and 52 years old.
• As each game session consisted of 24 searching tasks, we had all together 21984 comparable (standardized session-wise)records, each of them presenting mouse movement logs between two button clicks.
Connecting Emotions with Tasks
Old and New approach
1) Retrospective feedback2) Concurrent Think-Aloud protocol
Old Approach – retrospective feedbackFirst pilot:
There was no room of variety of emotions
Self-Reports on Russel’s Model
Self-Reports on Likert Scale• Interviews with selected particiapants
(N=44)• Right after game session• Still image of the game session was shown
Content Confused
Binary mapping: (1-3) content, (5-7) confused, (4) neutral
Self-Reports on Likert Scale• Emotion data about 44*24=1056 tasks
• All target finging times standardized session-wise
• Pearson correlation between self reports and standardized finding time was found (r = 0.86)
Content Confused
Self-Reports on Likert Scale
• Tasks reported as confused had finding speed 0.5 standard deviation below mean
• Tasks reported as content had finding speed 0.5 above mean
Content Confused
Binary mapping: (1-3) content, (5-7) confused, (4) left out a neutral
Separation of Classes
Standardized item finding speed
Second half ofeach of these
logs counted as characterizing non confusion
First half of each of these logs counted
as characterizing
confused state
Using Think-Aloud Protocol
Think-Aloud Protocol• Users reported five kinds of emotions -
confusion, frustration, shame, content and flow. Strongest emotions were confusion and frustration. Here is an example how users were expressing themselves during states of confusion and frustration:
• “Where is number x, where is number x, it is not there, it is impossible, I looked everywhere, it is missing. “
• “It can’t be, you hide a button, it is not there.”• “I saw it before, but now it is not there any
more.”
Russel’s Model
Think-Aloud protocol
Using Think-Aloud Protocol
• 400 sessions (20 users * 20)• 400*24 = 9600 comparable tasks with
emotion data
Separation of Classes
Final Datasets
• Confused class with 3170 examplesand all the rest with 18814 examples.
• In the case with separation gap between classes, the second class had 12381 examples.
• Before applying classification algorithms, we balanced our datasets.
Features
Distance (curvature)
1
23
4
1
2 3
12
3
45
Ratio between the 3-6 movements length and shortest path between the beginning and end point
Speed (σ of the speed)
s4s5
s1
s2 s3
s6s7
s8
s9s10
Speed is measured for each 10px movement separately
N
S
W O
Direction
Angle based features• Sum of consecutive turns greater than an angle A (A counted by
45-degree step), normalized by number of movements.
• 18 features representing turns from 0 to 180 degrees’ by 10-degreestep. Counted results were normalized by the number of movements.
• Sum of all angles divided by number of movements – 1.
• σ of angles.
A
Bα
Feature selection
10
Machine learning
• Logistic Regression• Support Vector Machine• Random Forest• C4.5
– Motivation based on literature. – Java implementations of data analysis
package Weka.– 10-fold cross validation
Results (with separation gap)
Results (without separation gap)
Conclusion
• Mouse movements reveal users emotsional states
• But is the confusion and frustration in current study comparable with confusion and frustration caused by solving mathematical equation or some other cognitively more demanding task?
Conclusion
However, if we relay on 2D Circumplex Model ofEmotion, then all kind of confusion and frustration is located in the sameplace.
Thank You!
Q&A