an illumination invariant face recognition system for access control using video
DESCRIPTION
An Illumination Invariant Face Recognition System for Access Control using Video. Ognjen Arandjelovi ć Roberto Cipolla. Funded by Toshiba Corp. and Trinity College, Cambridge. Eigenfaces. 3D Morphable Models. Wavelet methods. Face Recognition. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: An Illumination Invariant Face Recognition System for Access Control using Video](https://reader035.vdocuments.us/reader035/viewer/2022070502/56813ff2550346895dab0b1b/html5/thumbnails/1.jpg)
An Illumination Invariant Face An Illumination Invariant Face Recognition System for Access Control Recognition System for Access Control using Videousing Video
Ognjen ArandjelovićRoberto Cipolla
Funded by Toshiba Corp. and Trinity College, Cambridge
![Page 2: An Illumination Invariant Face Recognition System for Access Control using Video](https://reader035.vdocuments.us/reader035/viewer/2022070502/56813ff2550346895dab0b1b/html5/thumbnails/2.jpg)
Face RecognitionFace Recognition
• Single-shot recognition – a popular area of research since 1970s
• Many methods have been developed
• Bad performance in presence of:
– Illumination variation
– Pose variation
– Facial expression
– Occlusions (glasses, hair etc.)
Eigenfaces
Wavelet methods3D MorphableModels
![Page 3: An Illumination Invariant Face Recognition System for Access Control using Video](https://reader035.vdocuments.us/reader035/viewer/2022070502/56813ff2550346895dab0b1b/html5/thumbnails/3.jpg)
Face Recognition from VideoFace Recognition from Video
• Face motion helps resolve ambiguities of single shot recognition – implicit 3D
• Video information often available (surveillance, authentication etc.)
Recognition setup Training stream Novel stream
![Page 4: An Illumination Invariant Face Recognition System for Access Control using Video](https://reader035.vdocuments.us/reader035/viewer/2022070502/56813ff2550346895dab0b1b/html5/thumbnails/4.jpg)
Face ManifoldsFace Manifolds• Face patterns describe manifolds which are:
– Highly nonlinear, and
– Noisy, but
– Smooth
Facial features Face pattern manifold Face region
![Page 5: An Illumination Invariant Face Recognition System for Access Control using Video](https://reader035.vdocuments.us/reader035/viewer/2022070502/56813ff2550346895dab0b1b/html5/thumbnails/5.jpg)
Limitations of Previous WorkLimitations of Previous Work
• In this work we address 3 fundamental questions:
– How to model nonlinear manifolds of face motion
– How to achieve illumination and pose robustness
– How to choose the distance measure
?
![Page 6: An Illumination Invariant Face Recognition System for Access Control using Video](https://reader035.vdocuments.us/reader035/viewer/2022070502/56813ff2550346895dab0b1b/html5/thumbnails/6.jpg)
Face Motion Manifolds: RevisitedFace Motion Manifolds: Revisited
Unchanging identity, changing illumination
Changing identity, unchanging illumination
• Motivation: How can we use the prior knowledge on the shape of the manifolds?
![Page 7: An Illumination Invariant Face Recognition System for Access Control using Video](https://reader035.vdocuments.us/reader035/viewer/2022070502/56813ff2550346895dab0b1b/html5/thumbnails/7.jpg)
Pose ClustersPose Clusters• Face motion manifolds are nonlinear, but:
– Low-dimensional (c.f. registration for the reduction of the dimensionality), and
– Key observation: can be described well using only 3 linear pose clusters
Colour-coded pose clusters for 3 manifolds
![Page 8: An Illumination Invariant Face Recognition System for Access Control using Video](https://reader035.vdocuments.us/reader035/viewer/2022070502/56813ff2550346895dab0b1b/html5/thumbnails/8.jpg)
Determining Pose ClustersDetermining Pose Clusters• Pose clusters are semantic clusters:
– K-means and similar algorithms are unsuitable
– We are using a simple method based on the motion parallax
– Membership decided based on Maximum Likelihood
0.5 reye leye rnostril lnostril
reye leye
x x x xx x
Pupils
Discrepancy η
Image plane
Yaw measure
Distribution for 3 clusters
![Page 9: An Illumination Invariant Face Recognition System for Access Control using Video](https://reader035.vdocuments.us/reader035/viewer/2022070502/56813ff2550346895dab0b1b/html5/thumbnails/9.jpg)
Pose Clusters: ExamplePose Clusters: Example
Input manifold and colour-coded pose clusters
Sample frames fromthe 3 pose clusters
![Page 10: An Illumination Invariant Face Recognition System for Access Control using Video](https://reader035.vdocuments.us/reader035/viewer/2022070502/56813ff2550346895dab0b1b/html5/thumbnails/10.jpg)
Illumination compensationIllumination compensation• Performed in two stages:
– Coarse illumination compensation (exploiting face smoothness)
– Fine illumination compensation (exploiting low dimensionality of the face illumination subspace)
RGIC Optimization
Reference Cluster
Illumination Subspace
Input Output
![Page 11: An Illumination Invariant Face Recognition System for Access Control using Video](https://reader035.vdocuments.us/reader035/viewer/2022070502/56813ff2550346895dab0b1b/html5/thumbnails/11.jpg)
Region-based GICRegion-based GIC
*
2*
,
*
arg min ( , ) ( , )
( , )
Cx yI x y I x y
I I x y
Gamma Intensity Correction (GIC)
Canonical image
• Region-based GIC (RGIC): faces are (roughly) divided into regions with smoothly varying surface normal
Solved by 1D non-linear optimization
1 2
3 4
Face regions
Varying Gamma
![Page 12: An Illumination Invariant Face Recognition System for Access Control using Video](https://reader035.vdocuments.us/reader035/viewer/2022070502/56813ff2550346895dab0b1b/html5/thumbnails/12.jpg)
Region-based GIC: ArtefactsRegion-based GIC: Artefacts
• Region-based GIC suffers from artefacts at region boundaries
Mean face γ value map Smoothed γ map
Input face RGIC face Our method
Boundary artefacts
Artefactsremoved
![Page 13: An Illumination Invariant Face Recognition System for Access Control using Video](https://reader035.vdocuments.us/reader035/viewer/2022070502/56813ff2550346895dab0b1b/html5/thumbnails/13.jpg)
Illumination SubspaceIllumination Subspace• Each input frame corrected for a linear Pose Illumination Subspace
component to match the reference distribution of the same pose
– Illumination subspace is high-dimensional
– Constrained to expected variations by Mahalanobis distance
Input manifold
Reference manifold
* *
*
Subject to:
arg min
Where ... is the Mahanalobis distance
in the reference Gaussian
I
I M
M
I I B a
a I B a
Illumination Subspace
![Page 14: An Illumination Invariant Face Recognition System for Access Control using Video](https://reader035.vdocuments.us/reader035/viewer/2022070502/56813ff2550346895dab0b1b/html5/thumbnails/14.jpg)
Illumination Compensation ResultsIllumination Compensation Results
Original/input frames
Illumination-correctedframes
Reference frames
Strong side lighting
And in face pattern space…
![Page 15: An Illumination Invariant Face Recognition System for Access Control using Video](https://reader035.vdocuments.us/reader035/viewer/2022070502/56813ff2550346895dab0b1b/html5/thumbnails/15.jpg)
Comparing Pose ClustersComparing Pose Clusters
• “Distribution-based” distances (Kullback-Leibler divergence, Resistor Average Distance etc.) unsuitable
• We use the simple Euclidean distance between cluster centres
Reference cluster
Novel cluster
Reduced spread
Clustercentres
![Page 16: An Illumination Invariant Face Recognition System for Access Control using Video](https://reader035.vdocuments.us/reader035/viewer/2022070502/56813ff2550346895dab0b1b/html5/thumbnails/16.jpg)
Unified Manifold SimilarityUnified Manifold Similarity• Recognition based based on the likelihood ratio:
1,2,3
1,2,3
( | )( | )P D sP D s
Manifolds belong to the same person
Distances between pose clusters• Learn likelihoods from ground truth training data
Likelihoodhistogram
Undefined value regions
RBF-interpolatedlikelihood
Two-pose interpolatedlikelihood
Likelihood nowmonotonically decreasing
![Page 17: An Illumination Invariant Face Recognition System for Access Control using Video](https://reader035.vdocuments.us/reader035/viewer/2022070502/56813ff2550346895dab0b1b/html5/thumbnails/17.jpg)
Face Video Database RevisitedFace Video Database Revisited
• Testing performed under extreme, varying illuminations
10 illumination conditions used (random 5 for training, others for testing)
![Page 18: An Illumination Invariant Face Recognition System for Access Control using Video](https://reader035.vdocuments.us/reader035/viewer/2022070502/56813ff2550346895dab0b1b/html5/thumbnails/18.jpg)
RegistrationRegistration• Linear operations on images are highly nonlinear in the pattern space
• Translation/rotation and weak perspective can be easily corrected for directly from point correspondences
– We use the locations of pupils and nostrils to robustly estimate the optimal affine registration parameters
Translationmanifold
Skewmanifold
Rotationmanifold
![Page 19: An Illumination Invariant Face Recognition System for Access Control using Video](https://reader035.vdocuments.us/reader035/viewer/2022070502/56813ff2550346895dab0b1b/html5/thumbnails/19.jpg)
Registration Method UsedRegistration Method Used• Feature localization based on the combination of shape and pattern matching (Fukui et al.
1998)
Detect features
Crop & affineregister faces
![Page 20: An Illumination Invariant Face Recognition System for Access Control using Video](https://reader035.vdocuments.us/reader035/viewer/2022070502/56813ff2550346895dab0b1b/html5/thumbnails/20.jpg)
ResultsResults
• Very high recognition rates attainted (96% average) under extreme variations in illumination
• Other methods showed little to no illumination invariance
![Page 21: An Illumination Invariant Face Recognition System for Access Control using Video](https://reader035.vdocuments.us/reader035/viewer/2022070502/56813ff2550346895dab0b1b/html5/thumbnails/21.jpg)
Results, continuedResults, continued• The method was shown to give promising results for authentication uses:
– Good separability of inter- and intra- class manifold distances was found
– It can provide a secure system with only 0.1% false positive rate and 8% false negative rate
Cumulative distributionsof inter- and intra- class
manifold distances
The ROC curve forthe proposed method
![Page 22: An Illumination Invariant Face Recognition System for Access Control using Video](https://reader035.vdocuments.us/reader035/viewer/2022070502/56813ff2550346895dab0b1b/html5/thumbnails/22.jpg)
Future ResearchFuture Research
• Non-constant illumination within a single sequence causes problems
• Illumination compensation is still not perfect – pose illumination subspaces have unnecessarily high dimensions
• Pose estimation is too primitive – outliers cause problems in estimation of linear subspaces
• Complete pose invariance is still not achieved (what if there are no corresponding pose clusters?)
For suggestions, questions etc. please contact me at: [email protected]