recognizing facial expressions
TRANSCRIPT
![Page 1: Recognizing Facial Expressions](https://reader030.vdocuments.us/reader030/viewer/2022021009/6203a7fada24ad121e4befdc/html5/thumbnails/1.jpg)
1
Recognizing Facial Expressions
• Facial expressions reflect the emotional stage of a person.
• Recognizing facial expression from video sequences is a challenging problem.
• Applications– Perceptual user interface– Video compression (MPEG-4)– Synthesis of facial expression
![Page 2: Recognizing Facial Expressions](https://reader030.vdocuments.us/reader030/viewer/2022021009/6203a7fada24ad121e4befdc/html5/thumbnails/2.jpg)
2
Facial Expressions• Joy
– The eyebrows are relaxed. The mouth is open, and mouth corners pulled back toward ears.
• Sadness– The inner eyebrows are bent upward. The eyes
are slightly closed. The mouth is relaxed.
• Anger– The inner eyebrows are pulled downward and
together. The eyes are wide open. The lips are pressed against each other or opened to expose teeth.
Facial Expressions
• Fear– The eyebrows are raised and pulled together.
The inner eyebrows are bent upward. The eyes are tense and alert.
• Disgust– The eyebrows and eyelids are relaxed. The upper
lip is raised and curled, often asymmetrically.
• Surprise– The eyebrows are raised. The upper eyelids are
wide open, the lower relaxed. The jaw is open.
![Page 3: Recognizing Facial Expressions](https://reader030.vdocuments.us/reader030/viewer/2022021009/6203a7fada24ad121e4befdc/html5/thumbnails/3.jpg)
3
FACIAL EXPRESSIONS
RAISE EYE BROWS SMILE
FACIAL EXPRESSIONS
DISGUSTANGER
![Page 4: Recognizing Facial Expressions](https://reader030.vdocuments.us/reader030/viewer/2022021009/6203a7fada24ad121e4befdc/html5/thumbnails/4.jpg)
4
Black and Yacoob Algorithm
• Given the location of the face, eyes, brows, and mouth estimate the rigid motion of the face using pseudo perspective motion model.
• Use the face motion to register images through warping.
• Estimate relative motion of face features (eyes, mouth, brows).
• The estimated feature motions are used to predict locations of features in the next frame, and the process is repeated.
• The estimated motion is used to classify the facial expressions.
![Page 5: Recognizing Facial Expressions](https://reader030.vdocuments.us/reader030/viewer/2022021009/6203a7fada24ad121e4befdc/html5/thumbnails/5.jpg)
5
Affine
243
121
),(
),(
byaxayxv
byaxayxu
++=++=
=
2
4
3
1
2
1
10000001
),(),(
b
a
a
b
a
a
yx
yx
yxv
yxu
Affine
)()(
)()(
41
32
41
aavundeformatio
aavucurl
aavudivergence
yx
xy
yx
−=−=
−−=−−=
+=+=
243
121
),(
),(
byaxayxv
byaxayxu
++=++=
Expansion orcontraction
Rotation around Z
Squashing orstretching
![Page 6: Recognizing Facial Expressions](https://reader030.vdocuments.us/reader030/viewer/2022021009/6203a7fada24ad121e4befdc/html5/thumbnails/6.jpg)
6
Pseudo Perspective
a4=yawa5=pitch
=
8
7
6
5
4
3
2
1
2
2
1000
0001),(
),(
a
a
a
a
a
a
a
a
yxyxy
xyxyx
yxv
yxu
254876
52
4321
),(
),(
yaxyayaxaayxv
xyaxayaxaayxu
++++=
++++=
Pseudo Perspective
a4=yawa5=pitch
254876
52
4321
),(
),(
yaxyayaxaayxv
xyaxayaxaayxu
++++=
++++=
![Page 7: Recognizing Facial Expressions](https://reader030.vdocuments.us/reader030/viewer/2022021009/6203a7fada24ad121e4befdc/html5/thumbnails/7.jpg)
7
Affine with Curvature
2243
121
),(
),(
cxbyaxayxv
byaxayxu
+++=
++=
=
c
b
a
a
b
a
a
xyx
yx
yxv
yxu
2
4
3
1
2
1
2100000001
),(),(
Rules for Classifying Expressions
• Anger– B: inward lowering of brows and mouth contraction– E: outward raising of brows and mouth expansion
• Disgust– B: mouth horizontal expansion and lowering of brows– E: mouth contraction and raising of brows
• Happiness– B: upward curving of mouth and expansion or horizontal
deformation– E: downward curving of mouth and contraction or horizontal
deformation
![Page 8: Recognizing Facial Expressions](https://reader030.vdocuments.us/reader030/viewer/2022021009/6203a7fada24ad121e4befdc/html5/thumbnails/8.jpg)
8
Rules for Classifying Expressions
• Surprise– B: raising brows and vertical expansion of mouth– E: lowering brows and vertical contraction of mouth
• Sadness– B: downward curving of mouth and upward-inward
motion in the inner parts of brows– E: upward curving of mouth and downward-outward
motion in inner parts of brows
• Fear– B: expansion of mouth and raising-inwards inner parts of
brows– E: contraction of mouth and lowering inner parts of brows
Smile Expression
![Page 9: Recognizing Facial Expressions](https://reader030.vdocuments.us/reader030/viewer/2022021009/6203a7fada24ad121e4befdc/html5/thumbnails/9.jpg)
9
Smile
Smile Mouth Parameters
![Page 10: Recognizing Facial Expressions](https://reader030.vdocuments.us/reader030/viewer/2022021009/6203a7fada24ad121e4befdc/html5/thumbnails/10.jpg)
10
Anger
Anger Motion Parameters
![Page 11: Recognizing Facial Expressions](https://reader030.vdocuments.us/reader030/viewer/2022021009/6203a7fada24ad121e4befdc/html5/thumbnails/11.jpg)
11
Surprise
Surprise Motion Parameters
![Page 12: Recognizing Facial Expressions](https://reader030.vdocuments.us/reader030/viewer/2022021009/6203a7fada24ad121e4befdc/html5/thumbnails/12.jpg)
12
Blinking
Blinking Motion Parameters for Eyes
![Page 13: Recognizing Facial Expressions](https://reader030.vdocuments.us/reader030/viewer/2022021009/6203a7fada24ad121e4befdc/html5/thumbnails/13.jpg)
13
Rotation
Rotate Face motion parameters
P0 rot yP1 rot X
![Page 14: Recognizing Facial Expressions](https://reader030.vdocuments.us/reader030/viewer/2022021009/6203a7fada24ad121e4befdc/html5/thumbnails/14.jpg)
14
Rotation Motion Parameters
Mid-level predicates for Mouth
![Page 15: Recognizing Facial Expressions](https://reader030.vdocuments.us/reader030/viewer/2022021009/6203a7fada24ad121e4befdc/html5/thumbnails/15.jpg)
15
Mid-level predicates for Head
Parameter values used for classifying expressions
![Page 16: Recognizing Facial Expressions](https://reader030.vdocuments.us/reader030/viewer/2022021009/6203a7fada24ad121e4befdc/html5/thumbnails/16.jpg)
16
Forty Test Subjects
Results
Expression RateSurprise 91%Happiness 95%Anger 90%Disgust 93%Fear 83%Sadness 100%
![Page 17: Recognizing Facial Expressions](https://reader030.vdocuments.us/reader030/viewer/2022021009/6203a7fada24ad121e4befdc/html5/thumbnails/17.jpg)
17
Beginning of Anger Expression
![Page 18: Recognizing Facial Expressions](https://reader030.vdocuments.us/reader030/viewer/2022021009/6203a7fada24ad121e4befdc/html5/thumbnails/18.jpg)
18
Frames from 10 Video Clips
Results
Expression RateSurprise 86%Happiness 95%Anger 80%Disgust 50%Fear 100%Sadness 60%