emotions: the art of visual communication for virtual agents
DESCRIPTION
Emotions: The Art of Visual Communication for Virtual Agents. Emmanuel Tanguy Philip Willis Joanna J. Bryson University of Bath Department of Computer Science. Reseach funded by a studentship from the Department of Computer Science, University of Bath and the ESPRC grant GR/S/79299/01. - PowerPoint PPT PresentationTRANSCRIPT
Emotions: Emotions: The Art of Visual Communication The Art of Visual Communication for Virtual Agents.for Virtual Agents.
Emmanuel Tanguy
Philip Willis
Joanna J. Bryson
University of Bath
Department of Computer Science
Reseach funded by a studentship from the Department of Computer Science, University of Bath and the ESPRC grant GR/S/79299/01.
19/01/2006 2
Virtual Actors: Text To AnimationVirtual Actors: Text To Animation
In video games, films, educational software,…
Virtual Actor aims to communicate with humans
Democratise the creation of facial animation Text to speech software + facial animation system
Emotionally Expressive Facial Animation System (EE-FAS) producing visual speech from tagged text
19/01/2006 3
Marvin, the depressed robotMarvin, the depressed robot
The Hitchhiker's Guide to the Galaxy:– Marvin: You can blame the Sirius Cybernetics Corporation for making
androids with GPP...– Arthur: Um... what's GPP? – Marvin: Genuine People Personalities. I'm a personality prototype. You
can tell, can't you...?
<comm_func name="personal reaction" max_nb_fd="2" intensity="60">
You can blame Sirius Cybernetics Corporation for making androids <comm_func name="Emphasizer" intensity="30" >
with GPP...
</comm_func>
</comm_func>
Marvin, depressed?
19/01/2006 4
Levels of Control (LoC)Levels of Control (LoC)
What kind of facial representation can be added to the text?
Levels of Control:
1. Static Graphical Representation (LoC1)
2. Graphical Deformation Commands (LoC2)
3. Facial Movements or Facial Signals (LoC3)
4. Facial Meanings (LoC4) To be introduced in texts It is more intuitive than other LoCs To create visual speech
19/01/2006 5
Facial MeaningsFacial Meanings
Two types of facial meaning:– Emotional (ex: Happy, Sad, etc)– Communicative (question mark, emphasise, etc)
A facial meaning has physical implementations, e.g. facial signals
30 facial meanings and 50,000 facial signals (Terzopoulos and Waters, 1990).
One communicative function can have several facial signals. (Bavelas and Chovil, 2000; Pelachaud and Bilvi, 2003)
How to map a small number of facial meanings onto a large number of facial signals?
19/01/2006 6
Emotional ContextEmotional Context
We decided to use emotional contexts
Emotion models are widely used
In emotion models we distinguish two parts: – Emotion elicitation mechanisms and – Emotion representation
The Dynamic Emotion Representation (DER):– Why only the DER?– What for?
19/01/2006 7
Dynamic Emotion RepresentationDynamic Emotion Representation
Foundations of the DER:– Definitions of emotion types– Sloman model of mind processes– Picard’s description of
• emotion impulses
• emotion intensity
19/01/2006 8
Dynamic Emotion RepresentationDynamic Emotion RepresentationConfiguration File
EmotionalImpulse
EmotionIntensity
19/01/2006 10
Emotional Context for AnimationsEmotional Context for Animations
• 1.30s: <comm_func name="personal reaction" intensity="60" >
• 5.30s: <comm_func name=“emphasizer" intensity="30">
• 10.00s: <comm_func name="personal reaction"
sec_emo_context="happiness" intensity="60" >
• 13.20s: <comm_func name="question marker" intensity="60" >
Context of happiness
Context of sadness
19/01/2006 11
Meanings of Facial ComponentsMeanings of Facial Components
Purely categorical or componential approach? (Smith and Scott (1997)).
An experiment showing videos with individual facial movements:– Eyebrows frown; Lip corners raise; Eyebrows oblique
People’s perception measured on the following dimensions (60 subjects):– Happy; Angry; Sad; – Pleased/Displeased; Energetic/Lethargic; – Friendly/Unfriendly; Sincere/Insincere
19/01/2006 12
Happy
-3
-2
-1
0
1
2
3
0% 40% 80%
Frown
frown+Smile
smile
smile+em
oblique
oblique+smile
Dimension HappyDimension Happy
T-Test: Paired Samples Test (Happy)
-.66667 .89569 .11563 -.89805 -.43529 -5.765 59 .000
-1.40000 .97772 .12622 -1.65257 -1.14743 -11.091 59 .000
-1.33333 1.01958 .13163 -1.59672 -1.06995 -10.130 59 .000
-.73333 1.11791 .14432 -1.02212 -.44455 -5.081 59 .000
1.46667 1.01625 .13120 1.20414 1.72919 11.179 59 .000
1.16667 1.01124 .13055 .90544 1.42790 8.937 59 .000
Neutral - Frown_SmilePair 1
Neutral - SmilePair 2
Frown - SmilePair 3
Frown_Smile - SmilePair 4
Smile - ObliquePair 5
Smile - Oblique_SmilePair 6
Mean Std. DeviationStd. Error
Mean Lower Upper
95% ConfidenceInterval of the
Difference
Paired Differences
t df Sig. (2-tailed)
Lip corners raised involved in happiness expressionSmith and Scott (1997)
19/01/2006 13
Angry
-3
-2
-1
0
1
2
3
0% 40% 80%
Intensity of the expression
Sco
re
Frown
frown+Smile
smile
oblique
oblique+smile
Dimension AngryDimension Angry
T-Test: Paired Samples Test (Angry)
-1.43333 .88999 .11490 -1.66324 -1.20342 -12.475 59 .000
-1.01667 1.22808 .15855 -1.33391 -.69942 -6.412 59 .000
.41667 1.16868 .15088 .11476 .71857 2.762 59 .008
1.56667 .87074 .11241 1.34173 1.79160 13.937 59 .000
1.50000 .91132 .11765 1.26458 1.73542 12.750 59 .000
1.50000 .85371 .11021 1.27946 1.72054 13.610 59 .000
1.15000 1.10200 .14227 .86532 1.43468 8.083 59 .000
1.08333 1.13931 .14708 .78902 1.37765 7.365 59 .000
1.08333 1.07816 .13919 .80482 1.36185 7.783 59 .000
Neutral - FrownPair 1
Neutral - Frown_SmilePair 2
Frown - Frown_SmilePair 3
Frown - SmilePair 4
Frown - ObliquePair 5
Frown - Oblique_SmilePair 6
Frown_Smile - SmilePair 7
Frown_Smile - ObliquePair 8
Frown_Smile -Oblique_Smile
Pair 9
Mean Std. DeviationStd. Error
Mean Lower Upper
95% ConfidenceInterval of the
Difference
Paired Differences
t df Sig. (2-tailed)
Frown involved in anger expressionSmith and Scott (1997)
19/01/2006 14
Sad
-3
-2
-1
0
1
2
3
0% 40% 80%
Intensity of the expression
Sco
re
Frown
frown+Smile
smile
oblique
oblique+smile
Dimension SadDimension Sad
T-Test: Paired Samples Test (Sad)
-1.76667 1.24010 .16010 -2.08702 -1.44631 -11.035 59 .000
-.78333 1.30308 .16823 -1.11996 -.44671 -4.656 59 .000
.98333 1.38383 .17865 .62585 1.34081 5.504 59 .000
-1.95000 1.18501 .15298 -2.25612 -1.64388 -12.746 59 .000
-2.03333 1.11942 .14452 -2.32251 -1.74416 -14.070 59 .000
-2.08333 1.13931 .14708 -2.37765 -1.78902 -14.164 59 .000
Neutral - ObliquePair 1
Neutral - Oblique_SmilePair 2
Oblique - Oblique_SmilePair 3
Frown - ObliquePair 4
Smile - ObliquePair 5
Frown_Smile - ObliquePair 6
Mean Std. DeviationStd. Error
Mean Lower Upper
95% ConfidenceInterval of the
Difference
Paired Differences
t df Sig. (2-tailed)
19/01/2006 15
Dimensions Energy/PleasedDimensions Energy/Pleased
From Russel, J. A. and J. M. Fernandez-Dols (1997). The Psychology of Facial Expression.Cambridge University Press.
Emotional Disque
-2
-1,5
-1
-0,5
0
0,5
1
1,5
2
-2 -1,5 -1 -0,5 0 0,5 1 1,5 2
Unpleaseed/PleasedLet
har
gic
/Ener
get
ic
Frown
Smile
Oblique
neutral
19/01/2006 16
Friendly/Unfriendly
-3
-2
-1
0
1
2
3
0% 40% 80%
Intensity of the expression
Sco
re
Frown
frown+Smile
smile
oblique
oblique+smile
Dimension FriendlyDimension Friendly
T-Test: Paired Samples Test (Friendly/Unfriendly)
-2.55000 1.38301 .17855 -2.90727 -2.19273 -14.282 59 .000
-1.56667 1.29362 .16701 -1.90084 -1.23249 -9.381 59 .000
-1.91667 1.16868 .15088 -2.21857 -1.61476 -12.704 59 .000
-2.20000 1.69546 .21888 -2.63798 -1.76202 -10.051 59 .000
-1.21667 1.82350 .23541 -1.68773 -.74561 -5.168 59 .000
-1.56667 1.92545 .24857 -2.06406 -1.06927 -6.303 59 .000
Frown - SmilePair 1
Frown - ObliquePair 2
Frown - Oblique_SmilePair 3
Frown_Smile - SmilePair 4
Frown_Smile - ObliquePair 5
Frown_Smile -Oblique_Smile
Pair 6
Mean Std. DeviationStd. Error
Mean Lower Upper
95% ConfidenceInterval of the
Difference
Paired Differences
t df Sig. (2-tailed)
19/01/2006 17
Sincere/Insincere
-3
-2
-1
0
1
2
3
0% 40% 80%
Intensity of the expression
Sco
re
Frown
frown+Smile
smile
oblique
oblique+smile
Dimension SincereDimension Sincere
T-Test: Paired Samples Test (Sincere)
-.13333 1.68208 .21716 -.56786 .30119 -.614 59 .542
1.01667 1.92655 .24872 .51899 1.51435 4.088 59 .000
.61667 2.09997 .27111 .07419 1.15915 2.275 59 .027
-.76667 1.56624 .20220 -1.17127 -.36206 -3.792 59 .000
.03333 1.65669 .21388 -.39464 .46130 .156 59 .877
.80000 1.64471 .21233 .37513 1.22487 3.768 59 .000
1.15000 1.72543 .22275 .70427 1.59573 5.163 59 .000
-.40000 2.29369 .29611 -.99252 .19252 -1.351 59 .182
-.58333 2.21162 .28552 -1.15465 -.01201 -2.043 59 .046
Neutral - FrownPair 1
Neutral - Frown_SmilePair 2
Neutral - SmilePair 3
Neutral - ObliquePair 4
Neutral - Oblique_SmilePair 5
Oblique - Oblique_SmilePair 6
Frown - Frown_SmilePair 7
Frown_Smile - SmilePair 8
Smile - Oblique_SmilePair 9
Mean Std. DeviationStd. Error
Mean Lower Upper
95% ConfidenceInterval of the
Difference
Paired Differences
t df Sig. (2-tailed)
19/01/2006 18
EE-FAS ArchitectureEE-FAS Architecture Modular Customisable DER Message passing mechanism Implementation of all LoCs:
– LoC1: 3D Mesh– LoC2: Abstract Muscles– LoC3: Facial Signals– LoC4: Emotions and
Communicative functions
Customisable transformations from one LoC to another
Loose relation between LoCs
19/01/2006 19
Conclusion (1)Conclusion (1)
We developed a Dynamic Emotion Representation (DER) model that enables users– To create their own Emotion Representation based on
different emotion theories.– To represent any interacting variable, such as drives
We integrated an instance of the DER model within the EE-FAS
Technical Report on the DER:“A Dynamic Emotion Representation Model Within a Facial Animation System”, 2005, Emmanuel Tanguy, Joanna Bryson, Philip Willis, University of Bath
19/01/2006 20
Conclusion (2)Conclusion (2)
Developed an Emotionally Expressive Facial Animation System (EE-FAS), which:– Enables non-specialist users to create facial animations– Is customisable through XML dictionaries
EE-FAS extends the number of facial expressions that a such system can produce by displaying:– Emotional expressions– Multiple facial signals for each communicative function– Fake or genuine facial expressions