sensing valence and confusion with facial emg phil davis and hsin-ni ho
TRANSCRIPT
![Page 1: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/1.jpg)
Sensing Valence and Confusion with Facial EMG
Phil Davis and Hsin-Ni Ho
![Page 2: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/2.jpg)
OutlineI. Experimental SetupII. Valence
A. Problems AddressedB. Experimental ProcedureC. ResultsD. Discussion
III. Confusion A. Problems AddressedB. Experimental ProcedureC. ResultsD. Discussion
IV. Conclusions
![Page 3: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/3.jpg)
OutlineI. Experimental SetupII. Valence
A. Problems AddressedB. Experimental ProcedureC. ResultsD. Discussion
III. Confusion A. Problems AddressedB. Experimental ProcedureC. ResultsD. Discussion
IV. Conclusions
![Page 4: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/4.jpg)
Experimental Setup
Electrodes attached to: Corrugator supercilii (“brow”) Zygomaticus major (“cheek”)
Grounded on arm
![Page 5: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/5.jpg)
OutlineI. Experimental SetupII. Valence
A. Problems AddressedB. Experimental ProcedureC. ResultsD. Discussion
III. Confusion A. Problems AddressedB. Experimental ProcedureC. ResultsD. Discussion
IV. Conclusions
![Page 6: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/6.jpg)
OutlineI. Experimental SetupII. Valence
A. Problems AddressedB. Experimental ProcedureC. ResultsD. Discussion
III. Confusion A. Problems AddressedB. Experimental ProcedureC. ResultsD. Discussion
IV. Conclusions
![Page 7: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/7.jpg)
Problems Addressed Can we sense valence from facial
electromyographic (EMG) activity? Intensity?
Can we sense valence from mouse behavior (pressure in particular)?
In general, what types of activity can we sense with facial EMG?
![Page 8: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/8.jpg)
Relevant Prior Work Recent study by Larsen, Norris, Cacioppo Change in mean “cheek” EMG activity
correlated positively with valence Change in mean “brow” EMG activity
negatively correlated with valence
(CS = “brow” muscle, ZM = “cheek” muscle)
![Page 9: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/9.jpg)
OutlineI. Experimental SetupII. Valence
A. Problems AddressedB. Experimental ProcedureC. ResultsD. Discussion
III. Confusion A. Problems AddressedB. Experimental ProcedureC. ResultsD. Discussion
IV. Conclusions
![Page 10: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/10.jpg)
Experimental Procedure
6 subjects General procedure:
1. Subject watches a short film clip2. Subject answers questions about film
content and experienced affective state Subject uses pressure sensitive mouse
EMG output is recorded during all stages
![Page 11: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/11.jpg)
Film Clip Sequence Each film clip was roughly 3 minutes
1. Golf Instruction (Neutral)2. Clip from “The Champ” (Negative)
Sadness in 94.2% (Gross & Levenson)
3. Golf Instruction (Neutral)4. Clip from “Robin Williams Live” (Pos)
Amusement in 84.1% (Gross & Levenson)
Did not vary order
![Page 12: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/12.jpg)
OutlineI. Experimental SetupII. Valence
A. Problems AddressedB. Experimental ProcedureC. ResultsD. Discussion
III. Confusion A. Problems AddressedB. Experimental ProcedureC. ResultsD. Discussion
IV. Conclusions
![Page 13: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/13.jpg)
Self-reported Valence Consistent with Expectations Neutral Valence: 4 subjects indicated
“neutral” affect, 2 subjects indicated low intensity positive affect
Negative Valence: All 6 reported sadness
Positive Valence: 5 of 6 reported amusement The sixth subject did not like Robin
Williams
![Page 14: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/14.jpg)
Example of Output
Output normalized due to large scale differences
![Page 15: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/15.jpg)
Mean Output Consistent with Prior Work
![Page 16: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/16.jpg)
Individual Results Varied
![Page 17: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/17.jpg)
Other Observations
For some subjects, brow activity was a good indicator of web form activity
![Page 18: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/18.jpg)
OutlineI. Experimental SetupII. Valence
A. Problems AddressedB. Experimental ProcedureC. ResultsD. Discussion
III. Confusion A. Problems AddressedB. Experimental ProcedureC. ResultsD. Discussion
IV. Conclusions
![Page 19: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/19.jpg)
Discussion Highly sensitive to individual subject differences and
electrode positioning Mean output consistent with Larsen, Norris, Cacioppo Do results extend to other positive/negative states? Unable to correlate EMG output with self-reported mood
intensity EMG output may be useful as an activity recognition
sensor Unable to correlate mouse pressure or velocity with
valence Dynamic model of mouse behavior may produce better
results?
![Page 20: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/20.jpg)
OutlineI. Experimental SetupII. Valence
A. Problems AddressedB. Experimental ProcedureC. ResultsD. Discussion
III. Confusion A. Problems AddressedB. Experimental ProcedureC. ResultsD. Discussion
IV. Conclusions
![Page 21: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/21.jpg)
OutlineI. Introduction and Experimental SetupII. Valence
A. Problems AddressedB. Experimental ProcedureC. ResultsD. Discussion
III. Confusion A. Problems AddressedB. Experimental ProcedureC. ResultsD. Discussion
IV. Conclusions
![Page 22: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/22.jpg)
Problem Addressed
Does the facial expression ‘frowning’ represent the feeling of confusion?
Can we recognize the feeling of confusion with the EMG outputs?
Does the importance of understanding influence the feeling of confusion?
![Page 23: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/23.jpg)
OutlineI. Introduction and Experimental SetupII. Valence
A. Problems AddressedB. Experimental ProcedureC. ResultsD. Discussion
III. Confusion A. Problems AddressedB. Experimental ProcedureC. ResultsD. Discussion
IV. Conclusions
![Page 24: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/24.jpg)
Experimental Procedure
6 subjects General procedure:
1. Subject listens to an audio clip2. Subject answers questions about audio
content and experienced affective state EMG output is recorded during all
stages
![Page 25: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/25.jpg)
Eliciting Confusion with Audio Recordings
A B
Mood control 40 s classic music
Control spoken with standard American English
Confused by accent spoken with accent
Confused by meaning
A brief paragraph with confusing content spoken with American English
Two levels of the importance of understanding:Low level importance of understanding. High level importance of understanding.
Audio clips to induce different levels of confusion:
![Page 26: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/26.jpg)
Ratings and Measurements for feeling of Confusion
Subjective rating 5 scale Self-report confusion rating
Objective rating Test of understanding
Measurement EMG responses
![Page 27: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/27.jpg)
OutlineI. Introduction and Experimental SetupII. Valence
A. Problems AddressedB. Experimental ProcedureC. ResultsD. Discussion
III. Confusion A. Problems AddressedB. Experimental ProcedureC. ResultsD. Discussion
IV. Conclusions
![Page 28: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/28.jpg)
Result: EMG responses
Brow Cheek
Mean 210.14 2654
Std 214.44 5876
Difference in importance of understanding
NO DF=1, P=0.28
NO DF=1, P=0.24
![Page 29: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/29.jpg)
Result: Average EMG responsesBrow
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
important not important
No
rmal
ized
EM
G
mood control
1L
1W
2L
2W
3L
3W
For brow
More EMG activity when
filling the web form
listening to clips with accent
EMG in ‘web-filling’ parts increase with level of confusion in high importance of understanding
For cheek
EMG activity increase with degree of confusion ->Subject started to laugh
Cheek
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
important not important
No
rmal
ized
EM
G
0
1L
1W
2L
2W
3L
3W
![Page 30: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/30.jpg)
Result:Self confusion rating and test score
create the feeling of confusion consistently
Self rating confusion
Test score
Mean 3.81 2.38
Std 1.51 1.87
Difference for 6 clips
YesDF=5, P < 0.0001
YesDF=5, P < 0.0001
difference among subjects
NODF=5, P=0.42
NODF=5, P=0.33
Confusion and Test Score
0.001.002.003.004.005.006.00
1 2 3 4 5 6
Audio section
Rat
ing
and
Scor
e
Confusion
Test
Average plot
![Page 31: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/31.jpg)
OutlineI. Introduction and Experimental SetupII. Valence
A. Problems AddressedB. Experimental ProcedureC. ResultsD. Discussion
III. Confusion A. Problems AddressedB. Experimental ProcedureC. ResultsD. Discussion
IV. Conclusions
![Page 32: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/32.jpg)
Discussion
Huge individual difference in EMG responses -> normalize the data
More Brow activity when filling web forms -> It is more correlated to ‘the process of thinking about confusion thing’
More Brow activity when listening to clip with accent
Cheek activity increase with level of confusion -> So confused that subjects were giving up
Difference in importance of understanding
![Page 33: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/33.jpg)
OutlineI. Introduction and Experimental SetupII. Valence
A. Problems AddressedB. Experimental ProcedureC. ResultsD. Discussion
III. Confusion A. Problems AddressedB. Experimental ProcedureC. ResultsD. Discussion
IV. Conclusions
![Page 34: Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho](https://reader030.vdocuments.us/reader030/viewer/2022032517/56649cb25503460f94977598/html5/thumbnails/34.jpg)
Conclusions
Baseline difference in EMG for different muscles
Individual difference in EMG response EMG response sensitive to electrode positioning Brow EMG is negatively correlated with valence
and positively correlated with feeling of confusion
Cheek EMG is positively correlated with valence Facial EMG may be useful for activity
recognition