International Conference on Multimedia and Expo (July 2000)

Tutorial on Nonverbal Behavior and Paralinguistic Communication

Emotion and Paralinguistic Communication

• Jeffrey Cohn, Facial Expression in Human-and Human-machine Communication

• Francis Quek, Gesture, Speech, and Gaze in Human Conversational Interaction

• Ryohei Nakatsu, Recent Developments in Multi-Modal Interfaces

• Sidney Fels, Intimacy and embodiment at the interface

Facial Expression in Human- and Human-Machine Interaction

Jeffrey CohnAffect Analysis Group

Department of PsychologyRobotics Institute

Carnegie Mellon UniversityUniversity of Pittsburgh

Pittsburgh, PA 15260 USAhttp://www.pitt.edu/~jeffcohn/jfc.html

http://www.cs.cmu.edu/~face

Overview• What does facial expression express?• Factors that influence production and

interpretation of facial expression• Measurement

– Manual: Facial Action Coding System– Automatic

• Relation of facial to other modes of emotion expression

• Interpretation and user-interface

What Does Facial Expression Express?• Emotion

• Readiness for action

• Social signal (e.g., emblems)

• Meta-communication

• Cognitive activity

• Self-presentation

• Physiology

Basic Emotions

Fear Anger DisgustSadness SurpriseJoy

Others? interest, embarrassment, shame, contempt . . .

Cohn et al. (2000)

Defining Features (From Ekman, 1992)

• Distinctive universal signals • Distinctive universals in antecedent events• Distinctive physiology• Presence in other primates• Coherence among emotional response• Quick onset• Brief duration• Automatic appraisal• Unbidden occurrence

– Posture < face < voice < autonomic response

From Ekman (1972)

Pygmy chimpanzee (bonobo)

Silent Bared-Teeth Smile

‘Glare’

From Eibl-Eibesfeldt(1989)

Emotions Emerge asThey Become Adaptive

• Newborn– Disgust in response to sour

taste– Pain/Distress– Joy

• (Social smile and directed gaze at about 6 wks)

• 6-12 months– Surprise– Sadness– Anger– Fear

Dimensional Approaches

• Emotion circumplex• Valence and arousal

• Positive Affect (PA) and Negative Affect (NA)

• Approach/withdrawal

Circumplex: Self-Reported Emotion

Adapted from Lang et al. (199-) and Watson & Clark, 1985.

Excited

Pleasant

PANA

Unpleasant

Calm

Circumplex: Advantages

• Economical representation

• Consistent with both self-report and facial expression data

• Emphasizes importance of arousal

• Explicitly recognizes commonalities

Circumplex: Disadvantages

• Includes non-emotions, e.g., sleepy

• Assumes relationships among emotions are invariant

• De-emphasizes what is unique

• Puts emphasis on phenomenology of emotion to exclusion of other aspects, such as function

Emotions Have Multiple Components

Prosody

Kinesics

Congruence?

ANS

HR, BP, …

(Arousal)

(Specificity)

FaceCNS

Emotion

Valence

Action Readiness (from Fridlund, 1994)

Readiness to continueContentment

Declaration of superiorityContempt

Readiness to submit or flee

Fear

Readiness to attackAnger

Recruitment of succorSad

Appeasement“Non-Duchenne” smile

Readiness to play, affiliate

“Duchenne” smile

Social Displays: Brow Raise in Greeting

From Eibl-Eibesfeldt (1989)

What Else Does Facial Expression Express?

• Meta-communication– Turn-taking

– Back channeling

– Relationship

• Cognitive activity

• Self-presentation

Factors That Influence Production and Interpretation

• Presence of other people and who they are• Display rules

– Minimize– Neutralize– Exaggerate– Mask

• Cultural differences• Gender• Individual differences in

– facial appearance and functional anatomy– Hx of facial expression

Functional Anatomy of Facial Expression

From Duchenne as reported by Darwin (1872)

Clemente (1987)

Clemente(1987)

•Facial nerve (7th cranial nerve)

•Ipsalateral versus contralateral control

•Pyramidal versus extrapyramidal motor track

Motor Control of the Facial Muscles

From Rinn (199-)

Description of Facial Expression

• Basic-emotions– E.g., EMFACS, AFFEX

• Valence– E.g., Cohn et al. (1990)

• Sign-based approaches– Facial Action Coding System (FACS)

Asymmetry

From Wachtman & Cohn et al. (in press).

• Faces are asymmetric

• ‘True’ versus ‘false’ expressions

• Some expressions are typically asymmetric

• Neuropathology

Voluntary SpontaneousFrom Rinn (1985)

Dissociations of Spontaneous and Voluntary Smiles

Facial Action Coding System* Action Units (FACS AUs)

• Upper Face AUs: 12• Lower Face AUs: 18• Eye Position: 6• Head Position: 8• Miscellaneous: 14• Combinations

– Additive– Non-Additive * FACS: Ekman & Friesen

(1978)

AU 12

AU 20

AU 9

AU 10

AU 25

AU 26

AU 27

AU 17

AU 15

Adapted from Farkas & Munro, 1994, 1987

Selected FACS Action UnitsAU 1

AU 2

AU 4

AU 5

AU 6

AU 7

Upper Face AUsAU1 AU2 AU4

Brows lowered & drawn together

Inner portions of the brows are raised

Outer portions of the brows are raised

AU5 AU1+4 AU4+5

Upper eyelids are raised Medial portion of the brow is raised & pulled together

Brows lowered & drawn together & upper eyelids

are raised

AU6 AU7 AU1+2+4+5+7

AU1+2+5+6+7 AU1+2+4+5+6 AU1+2+4+5+6+7

Cheeks are raised & eye opening is narrowed

Lower eyelids are raised

Upper Face AUs

Appendix: Summary of FACS AUs and Identifying Features

Voluntary SpontaneousFrom Rinn (1985)

Dissociations of Spontaneous and Voluntary Smiles

12R 6+12+25

Automated Analysis

• Feature Extraction– Holistic

– Feature based

• Static versus dynamic face image data

• Rigid motion

Dense Optical Flow

Feature-Based Approach

lip width

39642baby

frame number

6561575349454137332925211713951

change fro

m fra

me 1

.3

.2

.1

0.0

-.1

right lip corner height

39642 baby

frame number

6561575349454137332925211713951

cha

ng

e f

rom

fra

me

1

.2

.1

0.0

-.1

Prosody:Not what you say but how you say

it

ProsodyLoudness ~ intensity Rhythm ~ temporal characteristicsFrequency

Prosodic Features Vary With

• Discrete Emotion– latency longer and more variable in depression– pitch increased in joy, decreased in disgust

• Arousal– speech rate in boredom versus excitement

• Communicative Intent– “Bev loves BOB?”– “Bev LOVES Bob”

• Interpersonal synchrony and attraction– matching of turn-taking duration

• Individual, gender, and regional differences

Summary

• System performance likely to vary with– Quality of ground truth training data

– Arousal

– Emotion • dimensions

• discrete emotion

– Communicative intent

– Presence of other people &their relationship to target person

– Individual differences: gender, culture, situation

Summary cont.

• Convergent measures increase accuracy

• Multi-modal information may yield the more robust recognition

• Mapping of measurements to interpretation will require multi-disciplinary efforts

Further Reading

Bachorowski, J.A. (1999). Vocal perception and expression of emotion. Current Directions in Psychological Science, 8, 53-56.

Clark, M.S. (Ed.) (1992). Review of Personality and Social Psychology, Vol. 13: Emotion and Vol. 14: Emotion and Social Behavior. Newbury Park, CA: Sage.

Cohn, J.F. & Katz, G.S. (September 1998). Bimodal expression of emotion by face and voice. ACM and ATR Workshop on Face/Gesture Recognition and Their Applications, pp. 41-44. Bristol, United Kingdom. http://www.cs.cmu.edu/~face

Damasio, A.R. (1994). Descartes' error: Emotion, reason, and the human brain. NY:Grosset/Putnam.

Darwin, C. (1872/1998). The expression of the emotions in man and animals. Third Edition. NY: Oxford

Ekman, P. (1985). Telling lies. NY: Basic Books.

Ekman, P. (1992). An argument for basic emotions. In N. Stein & K. Oatley, K. (Eds.) Basic emotions. Cognition and Emotion, 6, Nos. 3-4, 161-168. Hillsdale, NJ: LEA.

Ekman, P. & Friesen, W. (1978). Facial action coding system. Palo Alto, CA: Consulting Psychologists. Available from P. Ekman, Human Interaction Laboratory, University of California at San Francisco.

Eibl-Eibesfeldt, I. (1989). Human ethology. NY: Aldine de Gruyter.

Frick, R.W. (1985). Communicating emotion: The role of prosodic features. Psychological Bulletin, 97, 412-429.

Fridlund, A.J. (1994). Human facial expression: An evolutionary view. NY: Academic.

Kanade, T., Cohn, J.F., & Tian, Y. (March 2000). Comprehensive Database for Facial Expression Analysis. Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition (FG'00), pp. 46-53. Grenoble, France.

Lien, J.J.J., Kanade, T., Cohn, J.F., & Li, C.C. (2000). Detection, tracking, and classification of subtle changes in facial expression. Journal of Robotics and Autonomous Systems, 31,131-146. . http://www.cs.cmu.edu/~face

Matias, R. and Cohn, J.F. (1993). Are MAX-Specified Infant Facial Expressions During Face-to-Face Interaction Consistent with Differential Emotions Theory? Developmental Psychology, 29, 524-531.

Tian, Y.L, Kanade, T., & Cohn, J.F. (Submitted). Recognizing action units for facial expression analysis. .

Tian, Y., Kanade, T., and Cohn, J.F. (June 2000). Recognizing upper face action units for facial expression analysis. Proceedings of the IEEE Sponsored Conference in Computer Vision and Pattern Recognition, pp. xxx-xxx. Hilton Head, Georgia.http://www.cs.cmu.edu/~face

Additional readings on the topics of facial and prosodic expression and emotion can be found at http://www.pitt.edu/~jeffcohn/emot99.pdf