recent advances in face analysis: database, methods, and software
TRANSCRIPT
![Page 1: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/1.jpg)
Face Analysis: database, methods, and software Yue Wu Rensselaer Polytechnic Institute 06/30/2015
![Page 2: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/2.jpg)
Outline
• Overview
• Face detection
• Face alignment/tracking
• Face recognition
![Page 3: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/3.jpg)
1. Overview
• Major topics:
• Face detection
• Face alignment/tracking
• Face recognition
• Discussion:
• Database
• Methods
• Software
• Highlight:
• Recent works
• Deep-learning based works
• My works
![Page 4: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/4.jpg)
• Face analysis
1. Overview
Face detection
Face alignment /tracking
Face recognition
Head Pose Estimation
Facial expression analysis
*
![Page 5: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/5.jpg)
Outline
• Overview
• Face detection
• Face alignment/tracking
• Face recognition
![Page 6: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/6.jpg)
2. Face detection
• Overview:
• Even though face detection on up-right images is generally considered as solved problem, face detection on challenging conditions remains difficult.
• Pose
• Occlusion
• Low-resolution
• Illumination
• Chronology
• Expression
• Decoration
![Page 7: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/7.jpg)
2. Face detection: database
• Overview:
• Face detection database include images with face location annotations.
• Recently, there are more “in-the-wild” databases.
![Page 8: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/8.jpg)
2.1 Face detection: database
• Databases
Title # images Conditions Availability
1. Face Detection and Data Set Benchmark (FDDB)
5K In-the-wild Public
2. Annotated Facial landmarks in the wild (AFLW)
25K In-the-wild Public
3. Annotated Faces in the Wild (AFW)
~1K In-the-wild Public
* Some databases for face recognition and landmark detections can also be used for face detection.
** Please see the last slides of the chapter for links to the database
![Page 9: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/9.jpg)
2.2 Face detection: methods
• Method overview:
• Viola-Jones face detectors: Cascade framework, effective features.
• Deformable Part Model (DPM): Part based method, tree-structure spatial constraints among parts.
• Exemplar based methods: Select the training face as exemplars.
• Deep learning based methods: Learn deep CNN for face detection
Viola-Jones Face detector
Deformable Part Model
Exemplar based model
Deep learning based methods
![Page 10: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/10.jpg)
2.2 Face detection: methods(1)
• Deformable Part Model (DPM):
• X. Zhu and D. Ramanan. “Face detection, pose estimation and landmark localization in the wild”, CVPR 2012
• Keys:
• Pose-dependent models.
• Template based appearance model
• Tree-structure model to captures the spatial relationships among parts.
• Discriminative training with SVM.
![Page 11: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/11.jpg)
2.2 Face detection: method (1)
• Deep learning based method:
• H. Li, Z. Lin, X. Shen, J. Brandt, G. Hua, “A convolutional neural network cascade for face detection”, CVPR 2015
• Keys:
• Cascade manner + multi-resolution inputs: reject the negative samples quickly, focus on the remaining patches in the later cascade stage.
• Classifier: shallow CNN
• CNN calibration net for box modification: change the scale, locations of the currently predicted boxes.
![Page 12: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/12.jpg)
2.2 Face detection: method (1)
• CNN classifier net: shallow
![Page 13: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/13.jpg)
2.2 Face detection: method (1)
• Results
• Top performance on FDDB database
• Real-time
![Page 14: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/14.jpg)
2.3 Face detection: discussion
Discussion:
Deep learning could be the new star in this area. If more data is provided, it should achieve better performance.
If we consider the face as a special class of objects, we can learn from other object detection methods.
![Page 15: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/15.jpg)
2.4 Face detection: resources
• Databases
• FDDB: http://vis-www.cs.umass.edu/fddb/
• AFLW: https://lrs.icg.tugraz.at/research/aflw/
• AFW: http://www.ics.uci.edu/~xzhu/face/
• Software:
• Viola-Jones face detector: opencv, matlab
• Face detection without bells and whistles: http://markusmathias.bitbucket.org/2014_eccv_face_detection/
• More commercial software: refer to section 3.3
![Page 16: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/16.jpg)
2.4 Face detection: resources
• If we consider the face as a special class of objects. Then, we have more resources:
• Database:
• Pascal VOC http://host.robots.ox.ac.uk/pascal/VOC/
• ImageNet http://www.image-net.org/
• Good methods:
• R-CNN: R. Girshick, J. Donahue, T. Darrell, J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation”, arXiv
• Software: https://github.com/rbgirshick/rcnn
![Page 17: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/17.jpg)
Outline
• Overview
• Face detection
• Face alignment/tracking
• Face recognition
• Other topics
![Page 18: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/18.jpg)
3. Face alignment/tracking
• Overview:
• Facial alignment is to identify the locations of facial key points.
• Face alignment is usually the basic step for face recognition
• Challenge:
• Pose
• Occlusion
• Low-resolution
• Illumination
• Chronology
• Expression
• Decoration
• There are some deep learning based methods for face alignment
![Page 19: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/19.jpg)
3.1 Face alignment/tracking: database • Overview
• Facial alignment database include facial images and landmark annotations.
Title Conditions
1. MultiPie Lab, poses, expressions, illuminations
2. BioID Lab
3. CK+ Lab, expression
4. XM2VT Lab
5. AFLW In-the-wild
6. LFPW In-the-wild
7. Helen In-the-wild
8. AFW In-the-wild
9. COFW In-the-wild, occlusion
![Page 20: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/20.jpg)
3.2 Face alignment/tracking: method • Overview:
Holistic methods
Constrained local methods
Regression based method
appearance Shape model performance speed
Holistic Whole face Explicit Poor generalization
slow
CLM Local patch Explicit good Slow/fast
Regression-based *
Local patch Implicit Very good fast
![Page 21: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/21.jpg)
3.2 Face alignment/tracking: method (1) • Constrained local methods:
• Yue Wu and Qiang Ji “Discriminative deep face shape model for facial point detection”, IJCV 2015.
• Keys:
• Discriminative deep face shape models based on Restricted Boltzmann Machine that decomposes the shape variations into pose related and expression related parts.
![Page 22: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/22.jpg)
3.2 Face alignment/tracking: method (2) • Regression based method:
• X. Xiong and F. De la Torre, “Supervised Descent Method and its applications to face alignemnt”, CVPR 2013.
• Keys:
• Iteratively estimate the locations of facial landmarks, by starting from a mean face.
• Predict the difference vector between the current landmark locations and the target ground truth locations with linear regression methods.
𝑥0 𝑥𝑡
……
∆𝑥𝑡 = 𝑥∗ − 𝑥𝑡
∆𝑥𝑡 = 𝑅Φ(𝐼, 𝑥𝑡)
![Page 23: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/23.jpg)
3.2 Face alignment/tracking: method (3) • Regression based method:
• J. Zhange, S. Shan, M. Kan, and X. Chen, “Coarse-to-fine Auto-encoder network for real-time face alignment”, ECCV 2014.
• Key: Coarse-to-fine search with multi-resolution inputs.
![Page 24: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/24.jpg)
3.3 Face alignment/tracking: Discussion • Discussion:
• Regression based methods become the popular techniques
• Real-time face alignment and tracking is feasible.
• Facial alignment/tracking remain challenging with strong illumination changes, occlusions and large head poses.
![Page 25: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/25.jpg)
3.4 Face alignment/tracking: resource • Lab conditions:
• MultiPie:http://www.multipie.org/
• BioID: https://www.bioid.com/About/BioID-Face-Database
• CK+: http://www.consortium.ri.cmu.edu/ckagree/
• XM2VT: http://www.ee.surrey.ac.uk/CVSSP/xm2vtsdb/
![Page 26: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/26.jpg)
3.4 Face alignment/tracking: resource • “In-the-wild” databases:
• Annotated Facial Landmarks in the Wild (AFLW): https://lrs.icg.tugraz.at/research/aflw/
• Labeled Face Parts In the Wild (LFPW): http://neerajkumar.org/databases/lfpw/
• Helen: http://www.ifp.illinois.edu/~vuongle2/helen/
• Annotated Faces in the Wild (AFW): http://www.ics.uci.edu/~xzhu/face/
• Caltech Occluded Faces in the Wild (COFW): http://www.vision.caltech.edu/xpburgos/ICCV13/
• Additional annotations: http://ibug.doc.ic.ac.uk/resources/300-W/
![Page 27: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/27.jpg)
Outline
• Overview
• Face detection
• Face alignment/tracking
• Face recognition
![Page 28: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/28.jpg)
4. Face recognition
• Overview:
• Challenges:
• Pose
• Occlusion
• Low-resolution
• Illumination
• Chronology
• Expression
• Decoration
• Cross modality (e.g. sketch to image, infrared to visible)
• Large database + deep learning techniques
= close to human face recognition performance ?
![Page 29: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/29.jpg)
4.1 Face recognition: database • Overview
• Face recognition database include facial images and identity labels.
• Public available database is relative small, while there exist very large private databases.
Data set # images # subjects Availability
1. LFW 13K 1.6K Public
2. Youtube face 4K video 1.6K Public
3. MSRA-CFW 202K 1.6K Public
4. CASIA-WebFace 494K 11K Public
5. WDRef 100K 3K Public (feature only)
6. CACD 163K 2K Public (partial annotation)
7. PubFig 59K 200 Public
8. Janus (CVPR15) 6K 500 Public
9. People In Photo Albums (PIPA) 63K 2K Public
![Page 30: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/30.jpg)
4.1 Face recognition: database
• Overview (continue)
• Private databases
Data set # images # subjects Availability
10. CelebFaces (CUHK) 202K 10K Private
11. SCF (Facebook) 4.4M 4K Private
12. SCF2 (Facebook) 500M 10M Private
13. Google database 200M 8M Private
![Page 31: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/31.jpg)
4.1 Face recognition: database
• Two more new databases (public):
• Janus Benchmark: • Major novelty: full poses
• 6K images of 500 people
• Annotations: face locations, identity, landmarks, meta-data
• Reference: B. Klare, et al. “Pushing the frontiers of unconstrained face detection and recognition: IARPA Janus benchmark”, CVPR15
• Link: http://www.nist.gov/itl/iad/ig/face.cfm
![Page 32: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/32.jpg)
4.1 Face recognition: database
• People In Photo Albums (PIPA) • Major novelty: person recognition based on other attributes. Variations
in poses, illuminations, etc.
• 63K faces of 2K people
• DeepFace method only achieves ~ 50% accuracy on the new dataset.
• Reference: N. Zhang, et al (Berkeley and Facebook) “Beyond frontal faces: improving person recognition using multiple cues”, CVPR 2015.
• Link: http://www.eecs.berkeley.edu/~nzhang/piper.html
![Page 33: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/33.jpg)
4.2 Face recognition: method
• Overview:
• Deep learning based methods:
• Going deeper
• More data
Hand crafted features + SVM classifier
Deep learning based methods
![Page 34: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/34.jpg)
4.2 Face recognition: method(1)
• Y. Taigman, M. Yang, M. Ranzato, and L. Wolf, “DeepFace: closing the gap to human-level performance in face verification”, CVPR 2014
• Keys:
• 3D face alignment
• Deep neutral network (convolution & local).
• 97.35% accuracy on LFW comparing to 97.53% of human.
![Page 35: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/35.jpg)
4.2 Face recognition: method(2)
• Y. Taigman, M. Yang, M. Ranzato, and L. Wolf , “Web-scalde training for face identification”, CVPR 2015
• Key: • Structure: similar as DeepFace.
• Very large database: 500M images of 10M people.
• Size of “fc7” determines the generalization performance of the features.
• Random sampling is bad, and they use bootstrapping to select the hard samples.
• Performance: 98% on LFW
![Page 36: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/36.jpg)
4.2 Face recognition: method(3)
• Xiaogang Wang and Xiaoou Tang's group (CUHK):
• DeepID[1]: loose face alignment, several NNs of face parts.
• DeepID2[2]: joint face verification and identification.
• DeepID2+[3]: supervision in early layer, unshared weights in last few layers.
• DeepID3[4]: deeper
• Reference:
• [1] Y. Sun, X. Wang and X. Tang, “Deep learning face representation from predicting 10,000 classes”, CVPR 2014.
• [2] Y. Sun et al, “Deep learning face representation by joint identification-verification”, NIPS 2014
• [3] Y. Sun, X. Wang and X. Wang, “Deeply learned face representations are sparse, selective and robust”, CVPR 2015.
• [4] Y. Sun et al, “DeepID3: face recognition with very deep neural network”, arXiv 2015
![Page 37: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/37.jpg)
4.2 Face recognition: method(3)
• DeepID3:
• Loose face alignment
• Models for facial parts
• Deeper model
• Supervision in early layer
• Joint verification and identification
• Ensemble models
• ~200K training images
• 99.53% on LFW
![Page 38: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/38.jpg)
4.2 Face recognition: method(4)
• F. Schroff, D. Kalenichenko, J. Philbin (google), “FaceNet: a unified embedding for face recognition and clustering”, CVPR 2015.
• Keys: • Goal: learn a mapping from face images to a compact Euclidean space
where distances directly correspond to a measure of face similarity.
• Loss: triplet distance loss
• Structure: 20+ layers, loose face alignment
• Very large database: 200M images of 8M people
• Training time: 1000~2000 hours
• Performance: 99.63% on LFW
![Page 39: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/39.jpg)
4.3 Face recognition: Discussion
• Discussion:
• Deep learning techniques are the popular learning method for face recognition.
• The training set would takes up to millions of labeled images.
• Big training set is not shared.
• Face alignment may not be the necessary step.
![Page 40: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/40.jpg)
4.4 Face recognition: resources
Software overview (mainly commercial software):
title d: detection, r: recognition
Available
1. Lambda Lab d & r Free for limited usage
2. Animetrics Face Recognition
d & r
Sign up for API
3. SkyBiometry d & r Free for limited usage
4. Face++ d & r Sign up for SDK
5. FaceMark d free
6. EYERIS d & r Sign up for SDK
7. ReKognition d & r SDK
8. Betaface d & r Free for limited usage
9. Eyefdea recognition d & r Free for limited usage
10. Kairos r Sign up for SDK
![Page 41: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/41.jpg)
4.4 Face recognition: resources
Public available database:
LFW: http://vis-www.cs.umass.edu/lfw/
Youtube face database: http://www.cs.tau.ac.il/~wolf/ytfaces/
Data Set of Celebrity Faces on the Web (MSRA-CFW): http://research.microsoft.com/en-us/projects/msra-cfw/
Cross-Age Celebrity Dataset (CACD): http://bcsiriuschen.github.io/CARC/
WDRef database: http://home.ustc.edu.cn/~chendong/JointBayesian/
CASIA-WebFace: http://www.cbsr.ia.ac.cn/english/CASIA-WebFace-Database.html
PubFig: http://www.cs.columbia.edu/CAVE/databases/pubfig/
![Page 42: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/42.jpg)
4.4 Face recognition: resources
Software:
Lambda's lab: https://lambdal.com/face-recognition-api
FaceR API: http://api.animetrics.com/
SkyBiometry: https://www.skybiometry.com/
Face++: http://www.faceplusplus.com/api-overview/
FaceMark: http://apicloud.me/apis/
EYERIS: http://emovu.com/e/
ReKognition.com: https://rekognition.com/
Betaface: http://betafaceapi.com/
Eyedea recognition: http://www.eyedea.cz/eyeface-sdk/
Kairos: https://www.kairos.com/
![Page 43: Recent Advances in Face Analysis: database, methods, and software](https://reader033.vdocuments.us/reader033/viewer/2022052606/5884f1f71a28abf76f8b614d/html5/thumbnails/43.jpg)
Summary
• Face detection
• New deep learning based method outperform traditional methods.
• Facial alignment
• Regression based methods, including deep learning methods, are the popular techniques.
• Face recognition
• Based on very large databases, deep learning methods become the leading techniques in this area.
• Over-saturate performance on some databases. New challenging database shows that face recognition is still unsolved problem.