aam based face tracking with temporal matching and face segmentation mingcai zhou 1 、 lin liang 2...
TRANSCRIPT
![Page 1: AAM based Face Tracking with Temporal Matching and Face Segmentation Mingcai Zhou 1 、 Lin Liang 2 、 Jian Sun 2 、 Yangsheng Wang 1 1 Institute of Automation](https://reader033.vdocuments.us/reader033/viewer/2022051622/5697bffb1a28abf838cc0eb0/html5/thumbnails/1.jpg)
AAM based Face Tracking with Temporal Matching and Face SegmentationMingcai Zhou1 、 Lin Liang2 、 Jian Sun2 、 Yangsheng Wang11Institute of Automation Chinese Academy of Sciences, Beijing, China2Microsoft Research Asia Beijing, China
CVPR 2010
![Page 2: AAM based Face Tracking with Temporal Matching and Face Segmentation Mingcai Zhou 1 、 Lin Liang 2 、 Jian Sun 2 、 Yangsheng Wang 1 1 Institute of Automation](https://reader033.vdocuments.us/reader033/viewer/2022051622/5697bffb1a28abf838cc0eb0/html5/thumbnails/2.jpg)
2
Outline• AAM Introduction• Related Work• Method and Theory• Experiment
![Page 3: AAM based Face Tracking with Temporal Matching and Face Segmentation Mingcai Zhou 1 、 Lin Liang 2 、 Jian Sun 2 、 Yangsheng Wang 1 1 Institute of Automation](https://reader033.vdocuments.us/reader033/viewer/2022051622/5697bffb1a28abf838cc0eb0/html5/thumbnails/3.jpg)
3
AAM Introduction• A statistical model of shape and grey-level appearance
Shape model Appearance model
![Page 4: AAM based Face Tracking with Temporal Matching and Face Segmentation Mingcai Zhou 1 、 Lin Liang 2 、 Jian Sun 2 、 Yangsheng Wang 1 1 Institute of Automation](https://reader033.vdocuments.us/reader033/viewer/2022051622/5697bffb1a28abf838cc0eb0/html5/thumbnails/4.jpg)
4
Shape Model Building:mean shape
:shape bases
,shape parameters
learn by PCAgenerate mean shape 、shape bases
![Page 5: AAM based Face Tracking with Temporal Matching and Face Segmentation Mingcai Zhou 1 、 Lin Liang 2 、 Jian Sun 2 、 Yangsheng Wang 1 1 Institute of Automation](https://reader033.vdocuments.us/reader033/viewer/2022051622/5697bffb1a28abf838cc0eb0/html5/thumbnails/5.jpg)
5
Texture Model Building:mean appearance
:appearance bases
:appearance parameters
W(x)
灰階值
Shape-free patchMean shape
![Page 6: AAM based Face Tracking with Temporal Matching and Face Segmentation Mingcai Zhou 1 、 Lin Liang 2 、 Jian Sun 2 、 Yangsheng Wang 1 1 Institute of Automation](https://reader033.vdocuments.us/reader033/viewer/2022051622/5697bffb1a28abf838cc0eb0/html5/thumbnails/6.jpg)
6
AAM Model Building
![Page 7: AAM based Face Tracking with Temporal Matching and Face Segmentation Mingcai Zhou 1 、 Lin Liang 2 、 Jian Sun 2 、 Yangsheng Wang 1 1 Institute of Automation](https://reader033.vdocuments.us/reader033/viewer/2022051622/5697bffb1a28abf838cc0eb0/html5/thumbnails/7.jpg)
7
AAM Model Search• Find the optimal shape parameters and appearance parameters to minimize the difference between the warped-back appearance and synthesized appearance
( , )W x p
p
( ( , ))I W x p
map every pixel x in the model coordinate to its corresponding image point( , )W x p
0s
![Page 8: AAM based Face Tracking with Temporal Matching and Face Segmentation Mingcai Zhou 1 、 Lin Liang 2 、 Jian Sun 2 、 Yangsheng Wang 1 1 Institute of Automation](https://reader033.vdocuments.us/reader033/viewer/2022051622/5697bffb1a28abf838cc0eb0/html5/thumbnails/8.jpg)
8
Problems- AAM tracker• Difficultly generalize to unseen images• Clutterd backgrounds
![Page 9: AAM based Face Tracking with Temporal Matching and Face Segmentation Mingcai Zhou 1 、 Lin Liang 2 、 Jian Sun 2 、 Yangsheng Wang 1 1 Institute of Automation](https://reader033.vdocuments.us/reader033/viewer/2022051622/5697bffb1a28abf838cc0eb0/html5/thumbnails/9.jpg)
9
How to do?• A temporal matching constraint in AAM fitting -Enforce an inter-frame local appearance constraint between frames• Introduce color-based face segmentation as a soft constraint
![Page 10: AAM based Face Tracking with Temporal Matching and Face Segmentation Mingcai Zhou 1 、 Lin Liang 2 、 Jian Sun 2 、 Yangsheng Wang 1 1 Institute of Automation](https://reader033.vdocuments.us/reader033/viewer/2022051622/5697bffb1a28abf838cc0eb0/html5/thumbnails/10.jpg)
10
Related Work -feature-based (mismatched local feature) Integrating multiple visual cues for robust real-time 3d face tracking, W.-K. Liao, D. Fidaleo, and G. G. Medioni. 2007 -intensity-based (fast illumination changes) Improved face model fitting on video sequences, X. Liu, F. Wheeler, and P. Tu. 2007
temporal matching constraint
![Page 11: AAM based Face Tracking with Temporal Matching and Face Segmentation Mingcai Zhou 1 、 Lin Liang 2 、 Jian Sun 2 、 Yangsheng Wang 1 1 Institute of Automation](https://reader033.vdocuments.us/reader033/viewer/2022051622/5697bffb1a28abf838cc0eb0/html5/thumbnails/11.jpg)
11
Method and Theory• Extend basic AAM to Multi-band AAM– The texture(appearance) is a concatenation of three texture band values• The intensity (b)• X-direction gradient strength (c)• Y-direction gradient strength (d)
![Page 12: AAM based Face Tracking with Temporal Matching and Face Segmentation Mingcai Zhou 1 、 Lin Liang 2 、 Jian Sun 2 、 Yangsheng Wang 1 1 Institute of Automation](https://reader033.vdocuments.us/reader033/viewer/2022051622/5697bffb1a28abf838cc0eb0/html5/thumbnails/12.jpg)
12
1. Select feature points with salient local appearances at previous frame2. I(t−1) to the Model coordinate and get the appearance A(t-1)
3. Use warping function W(x;pt) maps R(t-1) to a patch R(t) at frame t
Temporal Matching Constraint
![Page 13: AAM based Face Tracking with Temporal Matching and Face Segmentation Mingcai Zhou 1 、 Lin Liang 2 、 Jian Sun 2 、 Yangsheng Wang 1 1 Institute of Automation](https://reader033.vdocuments.us/reader033/viewer/2022051622/5697bffb1a28abf838cc0eb0/html5/thumbnails/13.jpg)
13
Shape parameter Initialization
,
Face Motion Direction
![Page 14: AAM based Face Tracking with Temporal Matching and Face Segmentation Mingcai Zhou 1 、 Lin Liang 2 、 Jian Sun 2 、 Yangsheng Wang 1 1 Institute of Automation](https://reader033.vdocuments.us/reader033/viewer/2022051622/5697bffb1a28abf838cc0eb0/html5/thumbnails/14.jpg)
14
Shape parameter Initialization
When r reaches the noise level expected in the correspondences, the algorithm stops
![Page 15: AAM based Face Tracking with Temporal Matching and Face Segmentation Mingcai Zhou 1 、 Lin Liang 2 、 Jian Sun 2 、 Yangsheng Wang 1 1 Institute of Automation](https://reader033.vdocuments.us/reader033/viewer/2022051622/5697bffb1a28abf838cc0eb0/html5/thumbnails/15.jpg)
15
Shape parameter Initialization-Comparison
Motion direction
Feature matching Previous frame’s shape
![Page 16: AAM based Face Tracking with Temporal Matching and Face Segmentation Mingcai Zhou 1 、 Lin Liang 2 、 Jian Sun 2 、 Yangsheng Wang 1 1 Institute of Automation](https://reader033.vdocuments.us/reader033/viewer/2022051622/5697bffb1a28abf838cc0eb0/html5/thumbnails/16.jpg)
16
Face Segmentation ConstraintWhere are the locations of the selected outline points in the model coordinate
{ }kx
![Page 17: AAM based Face Tracking with Temporal Matching and Face Segmentation Mingcai Zhou 1 、 Lin Liang 2 、 Jian Sun 2 、 Yangsheng Wang 1 1 Institute of Automation](https://reader033.vdocuments.us/reader033/viewer/2022051622/5697bffb1a28abf838cc0eb0/html5/thumbnails/17.jpg)
17
Face Segmentation Constraint-Face Segmentation
![Page 18: AAM based Face Tracking with Temporal Matching and Face Segmentation Mingcai Zhou 1 、 Lin Liang 2 、 Jian Sun 2 、 Yangsheng Wang 1 1 Institute of Automation](https://reader033.vdocuments.us/reader033/viewer/2022051622/5697bffb1a28abf838cc0eb0/html5/thumbnails/18.jpg)
18
Face Segmentation Constraint
![Page 19: AAM based Face Tracking with Temporal Matching and Face Segmentation Mingcai Zhou 1 、 Lin Liang 2 、 Jian Sun 2 、 Yangsheng Wang 1 1 Institute of Automation](https://reader033.vdocuments.us/reader033/viewer/2022051622/5697bffb1a28abf838cc0eb0/html5/thumbnails/19.jpg)
19
Experiments
Lost frame num
![Page 20: AAM based Face Tracking with Temporal Matching and Face Segmentation Mingcai Zhou 1 、 Lin Liang 2 、 Jian Sun 2 、 Yangsheng Wang 1 1 Institute of Automation](https://reader033.vdocuments.us/reader033/viewer/2022051622/5697bffb1a28abf838cc0eb0/html5/thumbnails/20.jpg)
20
Experiments
![Page 21: AAM based Face Tracking with Temporal Matching and Face Segmentation Mingcai Zhou 1 、 Lin Liang 2 、 Jian Sun 2 、 Yangsheng Wang 1 1 Institute of Automation](https://reader033.vdocuments.us/reader033/viewer/2022051622/5697bffb1a28abf838cc0eb0/html5/thumbnails/21.jpg)
21
Conclusion─ Our tracking algorithm accurately localizes the facial components, such as eyes, brows, noses and mouths, under illumination changes as well as large expression and pose variations.
─ Our tracking algorithm runs in real-time . On a Pentium-4 3.0G computer, the algorithm’s speed is about 50 fps for the video with 320 × 240 resolution
![Page 22: AAM based Face Tracking with Temporal Matching and Face Segmentation Mingcai Zhou 1 、 Lin Liang 2 、 Jian Sun 2 、 Yangsheng Wang 1 1 Institute of Automation](https://reader033.vdocuments.us/reader033/viewer/2022051622/5697bffb1a28abf838cc0eb0/html5/thumbnails/22.jpg)
22
Future Work─ Our tracker cannot robustly track profile views with large angles
─ The tracker’s ability to handle large occlusion also needs to be improved