1

Last tuesday, you talked about active shape models

• Data set of 1,500 hand-labeled faces• 20 facial features (eyes, eye brows, nose, mouth, chin)• Train 40 individual regressors (x and y positions for each

facial feature)

Problems/Solutions

• Our regressors aren’t perfect, but they’re often wrong in conflicting ways

• Facial feature positions aren’t independent of each other– eg. knowing something about the

position of the eyes ought to provide a clue about the position of the nose

• Face alignment needs to be more than the sum of its parts– Active shape model

3

Shape model

• Given many examples of face shapes (and ignoring the image data)

• Find a lower dimensional model that can still represent the high dimensional examples

4

The Active Shape Model framework

• Build a model of face shapes to go along with our facial feature models, then combine the two into one output

Shape Features Alignment

Shape model

• Problem: We tried to describe our face using a 40-dimensional vector. That gives us a little bit too much flexibility:

• Solution: If 40 dimensions are too many, let’s build a low-dimensional model

Face Dog (?)

Learning a shape model

• Use Principal Component Analysis (PCA) on the feature positions– Represent each example as a 40-dimensional

vector, (x1, y1, x2, y2, …, x20, y20)

– Subtract the mean from each vector– Find the most important (principal) axes

• PCA gives us a total of 40 axes, but if we only select a few of them (say, the top six), we’ll get a low dimensional parameterization

Learning a shape model

• Compute mean face shape

8

Learning a shape model

• Compute principal components

ACTIVE APPEARANCE MODELS

• “Interpretation through synthesis”

• Form a model of the object/image (Learnt from the training dataset)

I. Matthews and S. Baker, "Active Appearance Models Revisited," International Journal of Computer Vision, Vol. 60, No. 2, November, 2004, pp. 135 - 164.

Face appearance has two components:

Model

“Shape” “texture”

Active Shape Models

training set

(Hard to automate)

• Shape Vector provides alignment

=

42

Alexei (Alyosha) Efros, 15-463 (15-862): Computational Photography, http://graphics.cs.cmu.edu/courses/15-463/2005_fall/www/Lectures/faces.ppt

Random Aside (can’t escape structure!)

Alexei (Alyosha) Efros, 15-463 (15-862): Computational Photography, http://graphics.cs.cmu.edu/courses/15-463/2005_fall/www/Lectures/faces.ppt

Antonio Torralba & Aude Oliva (2002)

Averages: Hundreds of images containing a person are

averaged to reveal regularities in the intensity patterns across

all the images.

Random Aside (can’t escape structure!)

Tomasz Malisiewicz, http://www.cs.cmu.edu/~tmalisie/pascal/trainval_mean_large.png

“100 Special Moments” by Jason Salavon

Jason Salavon, http://salavon.com/PlayboyDecades/PlayboyDecades.shtml

Given Shape Vectors (positions of set of keypoints)

• Shape Vector provides alignment

=

42Alexei (Alyosha) Efros, 15-463 (15-862): Computational Photography, http://graphics.cs.cmu.edu/courses/15-463/2005_fall/www/Lectures/faces.ppt

Collect keypoints from training data in Ms whose columns are training shapes.

Subtract mean shape (s), then run SVD to get (Ms = Us S Vs’).

si = s + Usai

“Shape free” Texture Models

warp to mean shape

Once you have the shape model for each image, warp the model to the mean configuration:

Now, all faces are the same size and shape and place. So you can create a column vector out of the relevant pixels. Put all those columns together (one for each training image) into Mt

Subtract mean texture (t), then run SVD to get Mt = Ut S Vt’.

ti = t + Utbi

Face model

• All faces parameterized by mean texture t and mean shape s.

• Each face has shape parameter ai, and texture paramter bi.

• Given a shape and a texture, we construct a face as:– Texture = t + Utb

– Shape = s + Usa– Make the texture, then warp it onto the new shape.

– Awkward Mathematical notion.– Warp(Image, shape parameters)– W(t + Utb, s + Usa)

PCA Craziness

• The complete description of the face is [ai,bi].• Let’s put those into one column, make a matrix

M for all those columns and run PCA.

• Why the heck would we do this?

Average faces.

• Alignment is the key!

1. Warp to mean shape

2. Average pixels

Alexei (Alyosha) Efros, 15-463 (15-862): Computational Photography, http://graphics.cs.cmu.edu/courses/15-463/2005_fall/www/Lectures/faces.ppt

Random Aside

• Enhancing Gender

more same original androgynous more opposite

D. Rowland, D. Perrett. “Manipulating Facial Appearance through Shape and Color”, IEEE Computer Graphics and Applications, Vol. 15, No. 5: September 1995, pp. 70-76

Playing with the Parameters

First two modes of shape variation First two modes of gray-level variation

First four modes of appearance variation

Essence of the Idea (cont.)

• Explain a new example in terms of the model parameters

Active Appearance Model Search (Results)

Final Projects

• 1 person: Implement any recent paper from CVPR/ ECCV/ ICCV, and apply to some new data.

• 2-3 people, small research project; perhaps starting with recent paper and substantial extension, or one of the following ideas

• Single Image Haze Removal Using Dark Channel Prior – http://research.microsoft.com/en-

us/um/people/jiansun/papers/dehaze_cvpr2009.pdf

Example 1 person projects…

Example 2-3 person projects

Matching historical and modern photos• http://www.thirdview.org/3v/rephotos/index.

html

What year was this photo taken?