recognizing action units for facial expression analysis (ppt).pdf

7/28/2019 Recognizing Action Units for Facial Expression Analysis (ppt).pdf

1/26

Recognizing Action Units forFacial Expression Analysis

Y.-l. Tian, T. Kanade, and J. F.

Cohn, PAMI 23(2), 2001Presented by Wei-Kai Liao


2/26


3/26

Automatic Facial Expression

Recognition

Key issues:

Facial features extraction & representation Appearance-based, geometric feature-based, or hybrid

Facial expression classification NN, SVM, BN, HMM, rule-based,

Description of facial expressions Expression prototypes Facial Action Coding System


4/26

Expression Prototypes

6 universal expression prototype (emotion

prototype): disgust, fear, joy, surprise,sadness, anger


5/26

Facial Action Coding System

Defined Action Unit (AU), which is a description of avisible facial change, in the face

Totally 44 AUs in the face: 30 are related to the contractions of specific facial muscles

12 for upper face and 18 for lower face

Initially, it is designed for trained human expert todetect the change of facial appearance

AUs could be additive or non-additive In this paper, the combined AUs are treated as a new AU

AUs in upper and lower face are relativelyindependent


6/26

Upper Face AU


7/26

Lower Face AU


8/26

System Overview Face and facial features are

automatically detected and thenmanually adjusted in the firstframe

Detect and track the facialfeatures.

Map the extracted facialfeatures into 2 sets ofparameters These parameters are

geometrically normalized tocompensate for image scaleand in-plane head motion

Feed these 2 sets of parametersinto 2 NN-based classifiers


9/26

Multistate Facial Component

Models Detect and track facial

components in near frontalfaces

Facial features could bedivided into 2 classes: Permanent: brow, cheek,

lip, eye Transient

Facial features have severalstates

Brow and cheek Modeled by a triangular

template with 6 parameters Use Lucas-Kanade algorithm

to track these templates


10/26

3-State Lip Model Use L-K algorithm to track

points of lip template For open and tightly

closed, there are non-lippixels inside the lip contour

Use the Gaussian mixturemodel to represent the colordistribution of the pixelsinside the lip contour

Determine the state basedon the shape and the color


11/26

Dual-State Eye Model The state is controlled by iris

Iris detection:

Edge maps: Canny edge operator

Fit the edge maps with iris mask

Eye corners Inner corners: LK tracking

Outer corners: determined by inner

corners


12/26

Transient Features Nasolabial furrows,

crows-feet wrinkles,Nose wrinkles

These areas are locatedusing the trackedlocations of thecorrespondingpermanent features

A Canny edge detectorto quantify the amountand orientation offurrows


13/26

Permanent Features Tracking

Results


14/26

Permanent Features Tracking

Results


15/26

Transient Features Tracking

Results


16/26

Upper Face Feature

Representation 15 parameters for

upper face features

12 for motion and shapeof the eyes, brows, and

cheek 2 for crows-feet wrinkles

1 for distance betweenthe brows

Computed as ratios tothe first frame


17/26

Lower Face Feature

Representation 9 parameters for lower face

features:

6 for lip shape, state, motion

3 for furrows in the nasolabial

and nasal root regions.

Normalized to the neutral


18/26

Neural Network Classifier A 3 layer neural

networks with onehidden layer

Separate NNs are usedfor upper and lowerface

6 12 hidden units are

used

Back-propagationalgorithm is used


19/26

Experimental Results

Left: a same subject could appear in both

training and test sets Right: no subject appears in both training and

test sets


20/26

Combined AUs Recognition

Result: Upper Face


21/26

Combined AUs Recognition

Result: Lower Face


22/26

Generalized to Different

Databases

Train on one database and test on the other

database


23/26


24/26

Problems (1) Only handling limited head motion (in-plane

motion) This is not sufficient for a practical application

Robustness issues: Complex environment Various lighting conditions Occlusions

Image resolution Not fully automatic need to manually adjust

the detected features in the first frame


25/26


26/26

Further References

CMU Automated Face Analysis website: http://www-2.cs.cmu.edu/~face/index2.htm http://www-2.cs.cmu.edu/afs/cs/project/face/www/Facial.htm

Survey paper Y.-l. Tian, T. Kanade, and J. F. Cohn, Facial Expression

Analysis, Handbook of Face Recognition, S. Z. Li and A. K.Jain, ed., Springer, October 2003.

FACS P. Ekman and W. Friesen, The Facial Action Coding System:

A Technique for the Measurement of Facial Movement,Consulting Psychologists Press, San Francisco, 1978
http://www-2.cs.cmu.edu/~face/index2.htmhttp://www-2.cs.cmu.edu/afs/cs/project/face/www/Facial.htmhttp://www-2.cs.cmu.edu/afs/cs/project/face/www/Facial.htmhttp://www-2.cs.cmu.edu/afs/cs/project/face/www/Facial.htmhttp://www-2.cs.cmu.edu/~face/index2.htmhttp://www-2.cs.cmu.edu/~face/index2.htm

recognizing action units for facial expression analysis (ppt).pdf

Documents