automatic face feature localization for face recognition christopher i. fallin honors thesis...
TRANSCRIPT
![Page 1: Automatic Face Feature Localization for Face Recognition Christopher I. Fallin honors thesis defense: May 1, 2009 advisor: Dr. Patrick J. Flynn May 1,](https://reader033.vdocuments.us/reader033/viewer/2022061306/551488c6550346f06e8b4da2/html5/thumbnails/1.jpg)
Automatic Face Feature Localization for Face Recognition
Christopher I. Fallinhonors thesis defense: May 1, 2009
advisor: Dr. Patrick J. Flynn
May 1, 2009 1Chris Fallin - thesis defense
![Page 2: Automatic Face Feature Localization for Face Recognition Christopher I. Fallin honors thesis defense: May 1, 2009 advisor: Dr. Patrick J. Flynn May 1,](https://reader033.vdocuments.us/reader033/viewer/2022061306/551488c6550346f06e8b4da2/html5/thumbnails/2.jpg)
Outline
• Face recognition– Methods of evaluation
• Elastic Bunch Graph Matching– Gabor Jets– Bunch Graphs and Feature Localization
• My Contributions: Automatic Fiducial Points– Information Content model– Fiducial point placement– results
![Page 3: Automatic Face Feature Localization for Face Recognition Christopher I. Fallin honors thesis defense: May 1, 2009 advisor: Dr. Patrick J. Flynn May 1,](https://reader033.vdocuments.us/reader033/viewer/2022061306/551488c6550346f06e8b4da2/html5/thumbnails/3.jpg)
Face Recognition
May 1, 2009 3Chris Fallin - thesis defense
• Subfield of biometrics:– life (bio)– measure (metric)– Extract identifying
information from measures of human traits
• Face recognition: digital images of face
• 2D, 3D, infrared, multimodal, …
http://www-users.cs.york.ac.uk/~nep/research/3Dface/tomh/3DFaces.jpg
![Page 4: Automatic Face Feature Localization for Face Recognition Christopher I. Fallin honors thesis defense: May 1, 2009 advisor: Dr. Patrick J. Flynn May 1,](https://reader033.vdocuments.us/reader033/viewer/2022061306/551488c6550346f06e8b4da2/html5/thumbnails/4.jpg)
May 1, 2009 Chris Fallin - thesis defense 4
Image Set
Face Recognition Evaluation
![Page 5: Automatic Face Feature Localization for Face Recognition Christopher I. Fallin honors thesis defense: May 1, 2009 advisor: Dr. Patrick J. Flynn May 1,](https://reader033.vdocuments.us/reader033/viewer/2022061306/551488c6550346f06e8b4da2/html5/thumbnails/5.jpg)
May 1, 2009 Chris Fallin - thesis defense 5http://en.wikipedia.org/wiki/File:Roc-general.png – used under terms of GNU FDL
System Decision
ROC curvesY N
Actual
Y
N
![Page 6: Automatic Face Feature Localization for Face Recognition Christopher I. Fallin honors thesis defense: May 1, 2009 advisor: Dr. Patrick J. Flynn May 1,](https://reader033.vdocuments.us/reader033/viewer/2022061306/551488c6550346f06e8b4da2/html5/thumbnails/6.jpg)
May 1, 2009 Chris Fallin - thesis defense 6
EER = 11.1%
![Page 7: Automatic Face Feature Localization for Face Recognition Christopher I. Fallin honors thesis defense: May 1, 2009 advisor: Dr. Patrick J. Flynn May 1,](https://reader033.vdocuments.us/reader033/viewer/2022061306/551488c6550346f06e8b4da2/html5/thumbnails/7.jpg)
Rank-one Score
May 1, 2009 Chris Fallin - thesis defense 7
Gallery A B C D E F GA 0.89 0.70 0.10 0.52 0.34 0.48 0.37B 0.70 0.73 0.45 0.82 0.12 0.43 0.44
Probe C 0.10 0.45 0.92 0.89 0.23 0.82 0.13D 0.52 0.82 0.89 0.56 0.20 0.38 0.14E 0.34 0.12 0.23 0.20 0.82 0.52 0.23F 0.48 0.43 0.82 0.38 0.52 0.84 0.11G 0.37 0.44 0.13 0.14 0.23 0.11 0.99
Rank-one: 5/7 = 71.4%
![Page 8: Automatic Face Feature Localization for Face Recognition Christopher I. Fallin honors thesis defense: May 1, 2009 advisor: Dr. Patrick J. Flynn May 1,](https://reader033.vdocuments.us/reader033/viewer/2022061306/551488c6550346f06e8b4da2/html5/thumbnails/8.jpg)
Elastic Bunch Graph Matching (EBGM)
• Wiskott et al., USC/Bochum, mid-90s• Basis of ZN-Face, successful commercial system• We use Face Identity Evaluation System, from
Colorado State• Face features represented by Gabor filter
responses• Features are localized
– Fit an elastic graph onto the features by localization: local optimization problems
May 1, 2009 8Chris Fallin - thesis defense
![Page 9: Automatic Face Feature Localization for Face Recognition Christopher I. Fallin honors thesis defense: May 1, 2009 advisor: Dr. Patrick J. Flynn May 1,](https://reader033.vdocuments.us/reader033/viewer/2022061306/551488c6550346f06e8b4da2/html5/thumbnails/9.jpg)
A Face Graph
May 1, 2009 Chris Fallin - thesis defense 9
[Wiskott99]
![Page 10: Automatic Face Feature Localization for Face Recognition Christopher I. Fallin honors thesis defense: May 1, 2009 advisor: Dr. Patrick J. Flynn May 1,](https://reader033.vdocuments.us/reader033/viewer/2022061306/551488c6550346f06e8b4da2/html5/thumbnails/10.jpg)
Gabor Jets
• Vector of filter responses to 40 Gabor kernels– 5 wavelengths– 8 orientations– Each is complex-valued
• Gabor jets capture information well: Gokberk et al. get 91% rank-one with fixed grid– On FERET: 78.5% max, with
12 grid points
May 1, 2009 Chris Fallin - thesis defense 10
![Page 11: Automatic Face Feature Localization for Face Recognition Christopher I. Fallin honors thesis defense: May 1, 2009 advisor: Dr. Patrick J. Flynn May 1,](https://reader033.vdocuments.us/reader033/viewer/2022061306/551488c6550346f06e8b4da2/html5/thumbnails/11.jpg)
Bunch Graphs
• Each feature has a bunch of canonical jets
• Represents typical features
• Best-match at each feature point for novel images
May 1, 2009Chris Fallin - thesis defense
11
[Wiskott99]
![Page 12: Automatic Face Feature Localization for Face Recognition Christopher I. Fallin honors thesis defense: May 1, 2009 advisor: Dr. Patrick J. Flynn May 1,](https://reader033.vdocuments.us/reader033/viewer/2022061306/551488c6550346f06e8b4da2/html5/thumbnails/12.jpg)
Feature Localization
• Initial alignment: eye locations known a-priori• Overlay bunch graph with average edge
lengths• Take Gabor jets; pick best match in each
bunch• Localize based on displacement estimation
(local optimization problem)
May 1, 2009 12Chris Fallin - thesis defense
![Page 13: Automatic Face Feature Localization for Face Recognition Christopher I. Fallin honors thesis defense: May 1, 2009 advisor: Dr. Patrick J. Flynn May 1,](https://reader033.vdocuments.us/reader033/viewer/2022061306/551488c6550346f06e8b4da2/html5/thumbnails/13.jpg)
The Idea: Automatic Fiducial Point Placement
• Bunch graph training requires manual fiducial point placement– 70 images, 25 points
• Why not statistically determine optimal features to match on?
• We can align/normalize all faces and take some statistical measure at each point in “face space” to determine goodness
• Replaces training step; back-end algorithm is identical
May 1, 2009 13Chris Fallin - thesis defense
![Page 14: Automatic Face Feature Localization for Face Recognition Christopher I. Fallin honors thesis defense: May 1, 2009 advisor: Dr. Patrick J. Flynn May 1,](https://reader033.vdocuments.us/reader033/viewer/2022061306/551488c6550346f06e8b4da2/html5/thumbnails/14.jpg)
Related Work
• Gokberk et al.: choosing fiducial points with genetic algorithms– But their chosen points are global– Same goal as our system, excluding prelocalization
• Salient Points– Wavelet-based approach to image retrieval– Choras et al., 2006: similar approach with
goodness function, but no EBGM
May 1, 2009 14Chris Fallin - thesis defense
![Page 15: Automatic Face Feature Localization for Face Recognition Christopher I. Fallin honors thesis defense: May 1, 2009 advisor: Dr. Patrick J. Flynn May 1,](https://reader033.vdocuments.us/reader033/viewer/2022061306/551488c6550346f06e8b4da2/html5/thumbnails/15.jpg)
Information Content: Variance Model
• Compute goodness function over face-space
• Inter-subject variance over intra-subject variance
• Self-normalizing unitless measure
• Requires multiple images per subject
May 1, 2009 Chris Fallin - thesis defense 15
![Page 16: Automatic Face Feature Localization for Face Recognition Christopher I. Fallin honors thesis defense: May 1, 2009 advisor: Dr. Patrick J. Flynn May 1,](https://reader033.vdocuments.us/reader033/viewer/2022061306/551488c6550346f06e8b4da2/html5/thumbnails/16.jpg)
Computing the goodness function
• FRGC: 5404 images – 700 MB, 128x128 grayscale (7 GB before normalization)
• Each pixel: 12 seconds, on fast Athlon 64• Split into 128 Condor jobs
– Each pixel is independent: easy• Pre-normalize image set, dump to fast-loading
binary format (single file)• Run Condor jobs: three hours• Post-processing to reassemble results
May 1, 2009 Chris Fallin - thesis defense 16
![Page 17: Automatic Face Feature Localization for Face Recognition Christopher I. Fallin honors thesis defense: May 1, 2009 advisor: Dr. Patrick J. Flynn May 1,](https://reader033.vdocuments.us/reader033/viewer/2022061306/551488c6550346f06e8b4da2/html5/thumbnails/17.jpg)
Fiducial Point Placement• Random placement with
probability density• Compute gradient of
goodness function• Probability is product of
gradient and goodness• Place points sequentially,
decay probability around points exponentially
• Mirror-point constraint: mirror placements across centerline, or snap to center
May 1, 2009 Chris Fallin - thesis defense 17
![Page 18: Automatic Face Feature Localization for Face Recognition Christopher I. Fallin honors thesis defense: May 1, 2009 advisor: Dr. Patrick J. Flynn May 1,](https://reader033.vdocuments.us/reader033/viewer/2022061306/551488c6550346f06e8b4da2/html5/thumbnails/18.jpg)
Prelocalization: Pseudo-Bunches
• Displacement estimation requires canonical feature jet from bunch
• We can’t provide this if we have no knowledge of feature
• Solution: fake a jet bunch– Make educated guess with K-means clustering on
jets from all images at given point• Then, run displacement estimation to
prelocalize points on each image
May 1, 2009 18Chris Fallin - thesis defense
![Page 19: Automatic Face Feature Localization for Face Recognition Christopher I. Fallin honors thesis defense: May 1, 2009 advisor: Dr. Patrick J. Flynn May 1,](https://reader033.vdocuments.us/reader033/viewer/2022061306/551488c6550346f06e8b4da2/html5/thumbnails/19.jpg)
Results
• Competitive with original, manual points– In both cases, automatic
training points yield only ~1% performance drop
– With no human training!
• Prelocalization did not work as intended
• Success without this suggested by Gokberk’s results
May 1, 2009 Chris Fallin - thesis defense 19
FERET
FRGC
![Page 20: Automatic Face Feature Localization for Face Recognition Christopher I. Fallin honors thesis defense: May 1, 2009 advisor: Dr. Patrick J. Flynn May 1,](https://reader033.vdocuments.us/reader033/viewer/2022061306/551488c6550346f06e8b4da2/html5/thumbnails/20.jpg)
ROC curves
FERET FRGC
May 1, 2009 Chris Fallin - thesis defense 20
(EER = 11.4%)(orig: 11.1%)
(EER = 31.4%)(orig: 34.8%)
![Page 21: Automatic Face Feature Localization for Face Recognition Christopher I. Fallin honors thesis defense: May 1, 2009 advisor: Dr. Patrick J. Flynn May 1,](https://reader033.vdocuments.us/reader033/viewer/2022061306/551488c6550346f06e8b4da2/html5/thumbnails/21.jpg)
Prelocalization: causes for failure
• Poor pseudo-bunch clustering: K-means often found optimal clustering at self-imposed cap of N/10 clusters– Likely because initial jets are too far off
• Naïve localization: single-step– Bolme thesis compares several optimization
algorithms• Average displacement of 2.628 pixels: larger
than 2.021 pixels in manual points
May 1, 2009 21Chris Fallin - thesis defense
![Page 22: Automatic Face Feature Localization for Face Recognition Christopher I. Fallin honors thesis defense: May 1, 2009 advisor: Dr. Patrick J. Flynn May 1,](https://reader033.vdocuments.us/reader033/viewer/2022061306/551488c6550346f06e8b4da2/html5/thumbnails/22.jpg)
Future Work
• More sophisticated prelocalization• Look at pseudo-bunch statistics to determine
failure mode in more detail• Look at per-fiducial point statistics to
determine where performance is weak• Investigate: are manual pts a theoretical limit,
or can we exceed them?• Try new image classes – test claim of
genericism
May 1, 2009 22Chris Fallin - thesis defense
![Page 23: Automatic Face Feature Localization for Face Recognition Christopher I. Fallin honors thesis defense: May 1, 2009 advisor: Dr. Patrick J. Flynn May 1,](https://reader033.vdocuments.us/reader033/viewer/2022061306/551488c6550346f06e8b4da2/html5/thumbnails/23.jpg)
Questions?
• Email [email protected]• Full thesis and source code will be posted
online: http://c1f.net/research/mark5/
May 1, 2009 23Chris Fallin - thesis defense
![Page 24: Automatic Face Feature Localization for Face Recognition Christopher I. Fallin honors thesis defense: May 1, 2009 advisor: Dr. Patrick J. Flynn May 1,](https://reader033.vdocuments.us/reader033/viewer/2022061306/551488c6550346f06e8b4da2/html5/thumbnails/24.jpg)
Distance Metrics on Jets
• Phase-insensitive: magnitude only– Selects best jet in bunch
• Phase-sensitive– Can solve for
displacement vector: basis of localization
• Displacement estimation
May 1, 2009 Chris Fallin - thesis defense 24