TechnischeUniversitätMünchen

Face Model Fitting based on Machine Learning from Multi-band Images of Facial Components

Institute for Informatics

Technische Universität München

Germany

Matthias Wimmer

Christoph Mayer

Freek Stulp

Bernd Radig

2/22

Face Model Fitting based on Machine Learning from Multi-band Images of Facial ComponentsTechnische Universität München – Matthias Wimmer

Outline of this Presentation

Infer semantic information

Model fitting with learned objective functions

Compute multi-band image representation

image

facial expression, gaze, identity, gender, age,…

Part 2:

Part 1:

facial component

correctly fitted face model

3/22

Face Model Fitting based on Machine Learning from Multi-band Images of Facial ComponentsTechnische Universität München – Matthias Wimmer

Part 1:Compute Multi-band Image Representing Facial Components

4/22

Face Model Fitting based on Machine Learning from Multi-band Images of Facial ComponentsTechnische Universität München – Matthias Wimmer

Motivation and Related Work

Descriptive feature representation of input image Input for the subsequent process of model fitting Quick computation

Similar approaches: Stegman et al. (IVC2003)

Stegman et al. Image and Vision Computing (2003)

5/22

Face Model Fitting based on Machine Learning from Multi-band Images of Facial ComponentsTechnische Universität München – Matthias Wimmer

Our Idea

Multi-band image contains the location of facial components

Use classifier for this task. Provide a multitude of features

Classifier decides which ones are relevant (→ quick) Consider pixel features only (→ quick) Pre-compute image characteristics

and adjust pixel features (→ accurate)

skin lips teeth brows pupils

6/22

Face Model Fitting based on Machine Learning from Multi-band Images of Facial ComponentsTechnische Universität München – Matthias Wimmer

Prerequisites

Image data base (from Web; 500 images) Face Locator: e.g. Viola and Jones

Computes rectangular regions around human faces

Manual annotations

7/22

Face Model Fitting based on Machine Learning from Multi-band Images of Facial ComponentsTechnische Universität München – Matthias Wimmer

Probability Matrices

Indicate the probability of the pixels to denote a certain facial component.

Relative to the face rectangle Learned offline

skin brows pupils lower lip teeth

8/22

Face Model Fitting based on Machine Learning from Multi-band Images of Facial ComponentsTechnische Universität München – Matthias Wimmer

Image Characteristics Describe characteristics of the entire image Computed by applying probability mask to face rectangle

color Distribution of color

of all facial components Gaussian distribution of color

(Mean, covariance matrix)

space Distribution of pixel locations

of all facial components Gaussian distribution of locations

(Mean, covariance matrix)

example image spatial distribution of skin colorskin color distribution

9/22

Face Model Fitting based on Machine Learning from Multi-band Images of Facial ComponentsTechnische Universität München – Matthias Wimmer

Multitude of Pixel Features color

Static pixel features Color

(RGB, NRGB, HSV, YCbCr)

16 features

Adjusted pixel features Color relative to mean color

(Euclidean, Mahalanobis)

~ 90 features

space

Coordinates (Cartesian, Polar)

Coordinates relative to mean location of facial components (Euclidean, Mahalanobis)

10/22

Face Model Fitting based on Machine Learning from Multi-band Images of Facial ComponentsTechnische Universität München – Matthias Wimmer

Evaluation: Train various Classifiers

4 Classifiers for each facial component C1: static feature only C2: adjusted color features only C3: adjusted location features only C4: all features

11/22

Face Model Fitting based on Machine Learning from Multi-band Images of Facial ComponentsTechnische Universität München – Matthias Wimmer

Classifiers for Lips and TeethC1 C2 C3 C4

77.3% 94.3% 90.4% 97.7%

C1 C2 C3 C4

74.3% 66.2% 87.9% 95.0%

12/22

Face Model Fitting based on Machine Learning from Multi-band Images of Facial ComponentsTechnische Universität München – Matthias Wimmer

Part 2:Model Fitting with Learned Objective Functions

13/22

Face Model Fitting based on Machine Learning from Multi-band Images of Facial ComponentsTechnische Universität München – Matthias Wimmer

Model-based Image Interpretation The model

The model contains a parameter vector that represents the model’s configuration.

The objective function Calculates a value that indicates how accurately a parameterized model matches an image.

The fitting algorithm Searches for the model parameters that describe the image best, i.e. it minimizes the objective function.

14/22

Face Model Fitting based on Machine Learning from Multi-band Images of Facial ComponentsTechnische Universität München – Matthias Wimmer

Local Objective Functions

15/22

Face Model Fitting based on Machine Learning from Multi-band Images of Facial ComponentsTechnische Universität München – Matthias Wimmer

Ideal Objective FunctionsP1: Correctness property:

Global minimum corresponds to the best fit.P2: Uni-modality property:

The objective function has no local extrema. ¬ P1 P1

¬P2

P2

Don’t exist for real-world images

Only for annotated images: fn( I , x ) = | cn – x |

16/22

Face Model Fitting based on Machine Learning from Multi-band Images of Facial ComponentsTechnische Universität München – Matthias Wimmer

Learning the Objective Function

x x x xx

xxx x xxx x x x

x x xx x

x xx x x x x

x xxx x

Ideal objective function generates training data Machine Learning technique generates calculation rules

ideal objective function

training data

learned objective function

17/22

Face Model Fitting based on Machine Learning from Multi-band Images of Facial ComponentsTechnische Universität München – Matthias Wimmer

Benefits of the Machine Learning Approach Accurate and robust calculation rules

Locally customized calculation rules Generalization from many images

Simple job for the designer Critical decisions are automated No domain-dependent knowledge required No loops

18/22

Face Model Fitting based on Machine Learning from Multi-band Images of Facial ComponentsTechnische Universität München – Matthias Wimmer

Evaluation 1: Displacing the Correct Model

statistics-based

objective function

ideal

objective function

learned

objective function

19/22

Face Model Fitting based on Machine Learning from Multi-band Images of Facial ComponentsTechnische Universität München – Matthias Wimmer

Evaluation 2: Selected Features

contour point 116

20/22

Face Model Fitting based on Machine Learning from Multi-band Images of Facial ComponentsTechnische Universität München – Matthias Wimmer

Conclusion

Crucial decisions within Computer Vision algorithms Don’t solve by trial and error

→ Learn from training data Example 1: Learned classifiers for facial components Example 2: Learned objective functions

21/22

Face Model Fitting based on Machine Learning from Multi-band Images of Facial ComponentsTechnische Universität München – Matthias Wimmer

Outlook

More features for learning objective functions Higher number of features Other kinds of features: SIFT, LBP, …

Learn with better classifiers Relevance Vector Machines Boosted regressors

Training images: render faces with AAM Exact ground truth (no manual work required) Many images

Learn global objective function

Learn rules to directly update model parameter

22/22

Face Model Fitting based on Machine Learning from Multi-band Images of Facial ComponentsTechnische Universität München – Matthias Wimmer

Thank you!

Online-Demonstration: http://www9.cs.tum.edu/people/wimmerm