a review of face recognition...– face recognition is a challenging problem because of the...

2010-04-10 Department of Electronics and Information Engineering, HUST

1

A Review of Face Recognition

problem-oriented analysis of face recognition techniques

Qiang [email protected]


2

Outline

• Introduction• Face recognition across pose• Illumination invariant face recognition• Face recognition from a single image per person• Appendix• Reference


3

Introduction

Automaticface analysis

Face detection Face recognition Facial expressionrecognition


4

Introduction• Backgrounds of face recognition

– A key issue in face analysis is finding efficient descriptors for face appearance [11].

– Face recognition is one of the most important biometric techniques.

– Characteristics:• Easily collectable, universal and non-intrusive.• It is natural and passive over other biometric techniques

requiring cooperative subjects such as fingerprint recognition and iris recognition. It is ideal for applications in scenarios where fingerprinting or iris scanning are impractical (for example, surveillance) or undesirable due to problems of social acceptance.


5

Introduction

• Basic concepts– Biometrics are physiological (for example, fingerprints

and face) and behavioral (for example, voice and gait) characteristics used to determine or verify an individual’s identity.

– Verification (or Authentication) is performed by matching an individual’s biometric with the template of the claimed identity only.

– Identification (or Recognition), on the other hand, is performed by matching an individual’s biometric with the template of every identity in the database [16].


6

Introduction

• Problems involved with face recognition– Face recognition is a challenging problem because of

the diversity in faces and variations caused by pose, illumination, expressions, occlusions, gender, and makeup [16].

– This report provided some recent advances in face recognition across pose and illumination. In addition, face recognition from a single image per person is also reviewed.


7

Face recognition across pose

• Backgrounds– Though many face recognition approaches reported

satisfactory performances, their successes are limited to the conditions of controlled environment, which are unrealistic in many real applications.

– In recent surveys of face recognition techniques, pose variation was defined one of the prominent unsolved problems in the research of face recognition and it gains great interest in the computer vision and pattern recognition research community [2].


8


• Problem definition, challenges– Face recognition across pose refers to recognising face images

in different poses by computers. – It is of great interest in many face recognition applications, most

notably those using indifferent or uncooperative subjects, such as surveillance systems. For example, face recognition is appealing in airport security to recognise terrorists and keep them from boarding plane.

– The most natural solution for this task might be to collect multiple gallery images in all possible poses to cover the pose variationsin the captured images, which requires a fairly easy face recognition algorithm.

– In many real situations, however, it is tedious and/or difficult to collect these multiple gallery images in different poses and therefore the ability of face recognition algorithm to tolerate pose variations is desirable.


9


• Problem definition, challenges– If a face recognition does not have a good pose tolerance, given

a frontal passport photo, the system appears to require cooperative subjects who look directly at the camera and face recognition is no longer passive and non-intrusive.

– Therefore, pose invariance or tolerance is a key ability for face recognition to achieve its advantages of being non-intrusive over other biometric techniques requiring cooperative subjects such as fingerprint recognition and iris recognition.

• Basic concepts about pose variation– Tilt: vertical rotations– Yaw: horizontal rotations, i.e., to turn about the vertical axis.


10


• Pose variation in the PIE database


11

Face recognition across pose• Evaluations

– As many pose-invariant face recognition approaches have been proposed recently, the need of evaluating different algorithms on a fair basis increased. A number of face image database have been established for the purpose to compare performances of different face recognition algorithms across pose.


12



13



14


• Techniques Categorization


15



16



17



18



19



20



21

Face recognition across pose• Conclusions and further discussion

– Prior knowledge• Techniques which learn image variations caused by pose

changes often require extensive training and the performance is dependent on training data.

• Techniques without prior knowledge usually require more than one gallery image to successfully compensate pose variations.

– The 3D face recognition approaches can generally handle larger pose variations than 2D techniques.

• The existing 3D face reconstruction methods made suboptimal surface assumptions on human faces, which affects the reconstruction results. (e.g. Lambertian assumption)

2010-04-10 22

Face recognition across pose• Conclusions and further discussion

– Comprehensive consideration• On the other hand, a comprehensive consideration of the

complicated face surface reflection mechanism and external lighting parameters brings serious ill-conditions to 3D face modeling, because the number of unknown parameters is excessive and the problems are intractable.

– Non-linear mapping • The strategy of non-linear mapping has the promise to find a

feature space best suitable to pose variations, while the current research stage is preliminarily limited to fundamental mapping functions (e.g., radial basis functions).

• The question of whether there is a feature space where rotated faces are separable is still open. An answer to this question may lead to a clearer understanding of pose-invariant face recognition problem, similar to the findings of linear subspaces in illumination-invariant face recognition.

Department of Electronics and Information Engineering, HUST


23


• Eigen light field (ELF) [12]– Method introduction

• This method unified all possible appearances of faces in different poses within a framework of light field, which is in a 4D space (two viewing directions and two pixel positions).

• Assuming human faces as convex Lambertian objects, this light-field was highly redundant and consequently the light field coefficients were associated in different poses for the same identity.


24

Face recognition across pose• Eigen light field (ELF) [12]

– In training stage• PCA was performed on those concatenated

vectors from different training identities. • Because of the redundancy of the light-field, face

images in different poses were represented using a single set of eigen vectors and eigen values to capture the variations due to identity changes.

warped Pose variant imagesrepresentation

A set of face images in different poses of different IDs

A uniform shape:feature points correspond to the light field

Concatenated vector (one per ID)

PCA


25


warped Projection ontothe eigen space

Recognition byComparing the projected

Eigen coefficients

Input image

gallery and/or probe image

• Eigen light field (ELF) [12]– In recognition stage

• The recognition procedure is as below.• The dimensionality of input images is usually

smaller than that of the light field (image dimension times number of poses).


26


• Locally linear regression [13]– Method introduction

• The basic idea lies in generating virtual frontal view from any given nonfrontal view to obtain a virtual gallery/probe face in order to solve pose problem.

• Following this idea, this paper proposed a simple, but efficient, novel locally linear regression (LLR) method, which generates the virtual frontal view from a given nonfrontal face image.


27


• Locally linear regression [13]– Assumption and idea

• The basic assumption of the paper is there exists an approximate linear mapping between a nonfrontal face image and its frontal counterpart.

• With the assistance of a generic cylindrical face model, the author proposed to generate virtual frontal views from single horizontally rotated views through local linear regression (LLR).


28


• Locally linear regression [13]– Virtual frontal view generation

• The author first perform dense sampling in the nonfrontal face image to obtain many overlapped local patches.

• Then, the linear regression technique is applied to each small patch for the prediction of its virtual frontal patch.

• Through the combination of all these patches, the virtual frontal view is generated.


29

Face recognition across pose• Locally linear regression [13]

– Procedure• In training stage, the face image was first divided into 10–30

evenly distributed patches in terms of an average cylindrical face model. In each patch, linear regression was performed to minimize the sum-square of image differences between frontal and non-frontal face images under a linear transformation.

• Then in testing stage, the input non-frontal image was also divided into patches in the same manner and each patch was transformed using the trained linear transformation matrix to form the appearance in the frontal view.

• Finally, all reconstructed patches were combined with a intensity averaging of overlapped pixels to form holistic frontal virtual views for recognition.


30


• Tied factor analysis (TFA) [14]– Introduction

• The algorithm has the following distinctive characteristics:• It is probabilistic and provides a posterior probability for the

matching to a gallery (identification) or for whether the two faces match or belong to different people (verification).

• The algorithm is based on a generative model that describes how an underlying pose-invariant representation created the (pose-varying) observed data (see Fig. 1). This is in contrast to most existing algorithms, where the direction of information flow is from the observed image to the pose-invariant representation.


31


• Tied factor analysis (TFA) [14]– Introduction

• In matching, one may think about the question: “What is the probability that two images were created from the same underlying representation?” However, the fact is this underlying representation is uncertain and never form an explicit point estimate.

• It is admitted that modeling the relationship between the entire frontal and nonfrontal faces (the global approach) is too challenging. Instead, this paper builds several local models describing how each individual facial feature (nose, eye, etc.) changes with pose. Then, the information from each model are combined by using naïve Bayes to make a final recognition decision.


32


• Tied factor analysis (TFA) [14]


33


• Tied factor analysis (TFA) [14]– Observed and Identity spaces

• Observed Image Data– It is assumed that the observed data is vectorized to form

an observed data vector.– The subspace to which faces commonly project is termed

the face manifold. – The manifold has two key characteristics that must be

captured by the proposed model. First, the mean positionin the manifold changes systematically with the pose of the face. Second, for a given individual, the position of the observation vector, relative to this mean, varies.


34



• Observed Image Data


35



• Identity Space Representation– At the core of the algorithm is the notion that there

genuinely exists a multidimensional variable h that represents the identity of the individual, regardless of the pose. We term the space of possible values for this variable as identity space, and the variable itself is termed a latent identity variable.

– Latent identity variables (LIVs) have this key property: If two LIVs take the same value, they represent the same person. If they take different values, they represent different people. LIV may be discrete or continuous.


36



• From Identity Space to Observed Space– 1. Choose the point in the identity space that

corresponds to the individual for which we create image data from some prior distribution.

– 2. Choose a pose (also from a prior distribution).– 3. Transform this identity variable to the observation

space by using a deterministic function. This function depends on the pose.

– 4. Add noise to the resulting observation vector.


37

Face recognition across pose• Tied factor analysis (TFA) [14]

– Observed and Identity spaces• Tied Factor Analysis

– We denote the jth image of individual i in the kth pose by xijk. We assume that this data was generated from an underlying latent identity variable, which we denote hi. The deterministic mapping between the identity and the observed spaces is affine. It comprises a set of offsets m1,...,K and a set of linear functions (matrices) F1,...,K. The generative process can be described as follows

– Formally write the model in terms of conditional probabilities as below


38



• Tied Factor Analysis– To complete the definition of the generative model, a

prior on the latent identity variables h is defined. – The prior is assumed to be a zero-mean Gaussian with

identity covariance I. – This is required to ensure that the learning process (see

next section) converges.


39




40


• Tied factor analysis (TFA) [14]– Learning System Parameters

• Learning parameters• To maximize


41



• E-step


42




43



• Considering (2) and (3), since all terms on the right-hand side of (5) are normally distributed, the left-hand side is also normally distributed (with a mean vector and a covariance matrix). The first two moments of this distribution equal


44



• M-step (GP prior: )


45



• M-step– These derivative expressions are equated to zero and

are rearranged to provide the following update rules:


46


• Tied factor analysis (TFA) [14]– Learning Results

• Data extraction from FERET for training and testing;• Face image segmentation from the background;• Image registration using 21 image features on the face;• Concatenation of the pixel values from the red, green, and

blue (RGB) channels to form a long observation vector.• Then build six models, each describing the variation between

one of the six non-frontal poses and the frontal pose. (according to the specific experiment)

• Use the model to predict how a face will look at a different pose. (uncertainty!)


47


• Tied factor analysis (TFA) [14]– Recognition

• The model is fundamentally Bayesian in nature and describes a probability distribution over the predicted images. It is not clear how we can exploit knowledge about the uncertainty in the predicted image.

• This paper employed a probabilistic approach to face identification.


48



• Characteristics of the probabilistic approach:• The criteria for a gallery and probe face matching are that the

observed data vectors are explained by exactly the same value of the identity variable.

• Since our observations are noisy, we can never be sure which value the identity variable takes. Hence, we integrate out (evidence framework) the hidden identity variable to give a final formulation that does not depend on an estimate of h.

• The final decision is based on a calculation of the relative likelihood that the observed vectors were explained by different configurations of the underlying set of identity variables.


49



In order to calculate this integral, we reformulate the generative equation as a standard factor analyzer, for which the solution is known:


50



• Note that the terms Pr(Mn) are the prior probability for each model. In the specific experiments, this is set to the uniformvalue of 1/N for each model.

• The final recognition decision is made by choosing the maximum a posteriori model (MAP).


51



52




53



54




55


• Tied factor analysis (TFA) [14]– Experiments

• Experiment 1: Face Identification Using Raw Pixel Data • Experiment 2: Face Identification with Local Gabor Data• Experiment 3: Face Verification• Experiment 4: Approximation of Evidence Term• Experiment 5: Automated versus Manual Keypoint Detection


56


• Tensor-Based AAM with Continuous Variation Estimation[15]


57

Illumination invariant face recognition

• Background– The illumination problem is one of the well-known

problems in face recognition in uncontrolled environment [3].

• Techniques Categorization– Passive approaches

• Based on given intensity images

– Active approaches• Using additional devices to get more information


58


• Passive approaches– Illumination Variation Modeling (based on a statistical

or physical model)• Linear Subspaces

– Photometric alignment, 3D linear subspace, segment linear subspace

• Illumination Cone– Cone-attached, Cone-casted

• Spherical Harmonics– Recent advances of 3D spherical harmonic basis morphable

model (SHBMM) in [9].• Nine point lights• Generalized Photometric Stereo


59


• Passive approaches– Illumination Invariant Features

• Features Derived from Image Derivatives– Gradient angle, and recent advance of Gradientfaces [7]– Symmetric shape from shading (A prototype image with

normalized illumination can be obtained from a single training image under unknown illumination.)

– Statistical shape from shading model (to recover face shape from a single image and to synthesize the same face under new illumination)

• Quotient Image• Retinex approach• Transformation domain features• Local Binary Pattern


60


• Passive approaches– Illumination Invariant Features

• Features Derived from Image Derivatives• Quotient Image

– Quotient illumination relighting– Generic intrinsic illumination subspace

• Retinex approach (luminance estimation by smoothed image)– A single Gaussian function is applied to smooth the image in

the single scale retinex approach, – multi-scale retinex (MSR) approach (using isotropic smoothing)– Self-Quotient Image (essentially a MSR, anisotropic smoothing)– Morphological Quotient Image (use mathematical morphology

operation to smooth the original image)• Transformation domain features• Local Binary Pattern


61

Illumination invariant face recognition• Passive approaches

– Illumination Invariant Features• Features Derived from Image Derivatives• Quotient Image• Retinex approach• Transformation domain features (frequency domain)

– Eigenphase approach (perform PCA in the phase domain)– Apply SVM directly on phase– Quaternion correlation method in a wavelet domain– A series of work based on advance correlation filters

• Local Binary Pattern– LBP is unaffected by any monotonic grayscale transformation

in that the pixel intensity order is not changed after such a transformation. Recent advance LGBPHS [27], HGPP [28].


62

Illumination invariant face recognition• Passive approaches

– Photometric Normalization• Histogram equalization• Gamma Intensity Correction for illumination normalization• Homomorphic filtering• Illumination normalization in the wavelet domain

– Histogram equalization is applied to subband image.• Local Normalization• DCT to compensate for illumination variation

– 3D Morphable Model• 3D Morphable Model describes the shape and texture

separately (PCA analysis on database of 3D scans); To fit a face image under unknown illumination to the model, an optimization process is needed to optimize shape and texture coefficients along with 22 rendering parameters.


63



64


• Active approaches– 3D information

• 3D information can be represented as range image, profile, surface curvature, Extended Gaussian Image (EGI), Point Signature, and etc.

• The 3D modality can be fused with 2D modality, i.e. texture, to achieve better performance.

• Attention: 2D face images used above are captured in a controlled environment.


65


• Active approaches– Infrared

• The infrared spectrum ranges from 0.7�m -10mm. It can be divided into 5 bands, namely:

• Near-Infrared (Near-IR) (0.7-0.9�m), the Short-Wave Infrared (SWIR) (0.9-2.4�m), the Mid-Wave Infrared (MWIR) (3.0-8.0�m), the Long-Wave Infrared (LWIR) (8.0-14.0�m), and Far-Infrared (FIR) (14.0�m-10mm). Near-IR and SWIR belong to reflected infrared (0.7-2.4�m), while MWIR and LWIR belong to thermal infrared (2.4�m-14mm).

• Thermal Infrared– A survey on visual and infrared face recognition in [32].

• Active Near-IR Illumination• Active Differential Imaging


66


• Gradientfaces [7]– Reflectance Model

• where I(x,y) is image pixel value, R(x,y) is the reflectance and L(x,y) is the illuminance at each point (x,y) .

• A “common” assumption is that L varies very slowly while Rcan change abruptly, which means L is approximately smooth.

• It should be pointed out that the conclusion (L be approximately smooth) may be violated in shadow boundaries.


67


• Gradientfaces [7]– Gradient Domain Versus Pixel Domain

• the pixel points are not completely independent of each other, there are some relationships between neighboring pixel points.

• the conventional face recognition methods, such as PCA and LDA, which are implemented in pixel domain such that they ignore the underlying relationships between neighboring pixel points.

• While the gradient domain explicitly considers such relationships between neighboring pixel points such that it is able to reveal underlying inherent structure of image data.


68


• Gradientfaces [7]– Gradientfaces


69



70



71


• Narrowband Spectral Range Selection [8]– Chang et al. [8] introduced a distribution separation

measure and the selection of the optimal spectral range by ranking these separation values. The fused images from these chosen spectral ranges are verified to outperform the conventional broadband images by 3%–20%.

– They proposed using narrowband subspectral images instead of conventional broadband images to improve recognition performance. A spectral range selection algorithm was developed to choose the optimal band images under given illumination conditions.


72

Face recognition from a single image per person

• Introduction [4]– Many research efforts [1] have been focused on how

to improve the accuracy of a recognition system.– Most of them ignore the potential problem when the

face database at hand has only one sample image per person stored.

– Under this condition, most of the traditional methods such as eigenface and fisherface will suffer serious performance drop or even fail to work.


73


• The one sample per person problem [4]– Backgrounds

• Geometric-based methods (template matching): geometrical feature extraction. One image per person is not a problem at all for these methods.

• Appearance-based techniques: vectorlized representation of face image. These techniques greatly improved the effectiveness and efficiency of face recognition systems.

• However, one of the key components of appearance-based methods is their learning mechanism, whose performance is heavily affected by the number of training samples for each face.


74


• The one sample per person problem [4]– The challenges of one sample problem

• Broadly speaking, one sample problem is directly related to the small sample size problem in statistics and pattern recognition.

• Dimensionality reduction techniques such as principal component analysis (PCA).

• The covariance matrix for the N training samples C:


75


• The one sample per person problem [4]– The challenges of one sample problem

• Various extended algorithms during the last decades based on the standard eigenface technique, including

– probabilistic-based eigenface, – linear discriminative analysis (LDA) based subspace

algorithms, – support vector machine (SVM) based method, – feature line method,– Evolution pursuit, and – Laplacianfaces.

• If only one training sample per person is available, most of them will either reduce to the basic eigenface approach or simply fail to work in that case.


76


• The one sample per person problem [4]– The significance of the one sample problem

• The extreme case of one sample per person really commonly happens in real scenarios.

• On the other hand, storing only one sample per person in the database has several advantages desired by most real world applications:

– Easy to collect samples, either directly or indirectly– Save storage cost– Save computational cost


77


• Techniques Categorization [4]


78



79


• Recent advances [33, 18]


80



81

Appendix

• Categorization of face recognition techniques– These algorithms are categorized from two different

perspectives [16].– Perspective of the type of data.

• 1) 2D face recognition (which use 2D grayscale or color images),

• 2) 3D face recognition (which use 3D range images or pointclouds of faces), and

• 3) multimodal face recognition algorithms (which use both 2D and 3D facial data).


82

Appendix

• Categorization of face recognition techniques– Perspective of the type of approach those algorithms

use. (Zhao et al. [1] reviewed appearance-based (2D) face recognition algorithms from this perspective.)

• 1) holistic• 2) feature-based (or referred to as region based)• 3) hybrid-matching face recognition algorithms.


83

Appendix

• Categorization of face recognition techniques– Holistic algorithms match the faces as a whole for

recognition. • the eigenfaces of Turk and Pentland [19] that use the

Principal Component Analysis (PCA), • Fisherfaces [20] that use Linear Discriminant Analysis (LDA), • methods based on the Independent Component Analysis

(ICA) [21], • Bayesian methods [22], and • Support Vector Machine (SVM) methods [23]. • Neural networks [24] have also been used for holistic face

recognition.


84

Appendix

• Categorization of face recognition techniques– Feature-based structural matching approaches

• HMM [25]. • Mixture Distance [26].• The graph-matching approach [10] is one of the most

successful region-based approaches [1]. • LBP [11], LGBPHS [27], HGPP [28].• Region-based methods can prove useful in case of variations

(for example, illumination and expression) in the images [1].


85

Appendix

• Categorization of face recognition techniques– Hybrid approaches

• the modular eigenfaces approach [29] uses both global eigenfaces and local eigenfeatures.

• Other examples include the flexible appearance model based method in [30] and [31].


86

Reference• [1] Face Recognition A Literature Survey [Zhao_ACMCS2003]• [2] Face recognition across pose: A review [Xiaozheng

Zhang_PR200911]• [3] Illumination invariant face recognition: A survey [Xuan

Zou_IEEEConf2007]• [4] Face Recognition from a Single Image per Person: A Survey

[Xiaoyang Tan_PR2006]• [5] Recent advances in face biometrics with Gabor wavelets: A

review [Angel Serrano_PRL2009]• [6] A review on Gabor wavelets for face recognition [Linlin

Shen_PAA2006]• [7] Face Recognition Under Varying Illumination Using Gradientfaces

[TP Zhang_TIP200911]• [8] Improving Face Recognition via Narrowband Spectral Range

Selection Using Jeffrey Divergence [Hong Chang_TIFS200903]• [9] Face Relighting from a Single Image under Arbitrary Unknown

Lighting Conditions [Yang Wang_TPAMI200911]


87

Reference• [10] Face Recognition by Elastic Bunch Graph Matching

[WisFelKrue_TPAMI1997]• [11] Face description with local binary patterns: Application to face

recognition [Timo Ahonen_TPAMI2006]• [12] Appearance-Based Face Recognition and Light-fields [Ralph

Gross_TPAMI200404]• [13] Locally linear regression for pose-invariant face recognition [X.

Chai_S.Shan_TIP2007]• [14] Tied Factor Analysis for Face Recognition across Large Pose

Differences [Simon J.D. Prince_TPAMI2008] • [15] Tensor-Based AAM with Continuous Variation Estimation:

Application to Variation-Robust Face Recognition [TPAMI200906]• [16] An efficient multimodal 2D-3D hybrid approach to automatic

face recognition [Ajmal S. Mian_TPAMI07]• [17] Making FLDA applicable to face recognition with one sample

per person [SC Chen_PR2004]• [18] Face Recognition Using Combined Multiple Feature Extraction

Based on Fourier-Mellin Approach for Single Example image per Person [Yee Ming ChenPR2010M]


88

Reference• [19] Eigenfaces for Recognition [JCN1994]• [20] Eigenfaces vs. Fisherfaces: Recognition Using Class Specific

Linear Projection [TPAMI1997]• [21] Independent Component Representation for Face Recognition

[Proc. SPIE1998]• [22] Probabilistic Visual Learning for Object Representation

[TPAMI1997]• [23] Support Vector Machines Applied to Face Recognition

[NIPS1998]• [24] Evolutionary Pursuit and Its Application to Face

Recognition[TPAMI2000]• [25] HMM Based Architecture for Face Identification [IJIVC1994]• [26] Feature-Based Face Recognition Using Mixture Distance

[CVPR1996]


89

Reference• [27] Local Gabor binary pattern histogram sequence (LGBPHS): a

novel non-statistical model for face representation and recognition [Zhang_Shan_ICCV2005]

• [28] Histogram of Gabor phase patterns (HGPP): a novel object representation approach for face recognition [Zhang_Shan_TIP2007]

• [29] View-Based and Modular Eigenspaces for Face Recognition [Pentland_CVPR1994]

• [30] Automatic Face Identification System Using Flexible Appearance Models [Lanitis_IVC1995]

• [31] Component-Based Face Recognition with 3D Morphable Models [Huang_AVBPA2003]

• [32] Recent advances in visual and infrared face recognition: a review [Seong G. Kong_CVIU2005]

• [33] Sparsity preserving discriminant analysis for single training image face recognition [SC Chen_PRL2010]


90

a review of face recognition...– face recognition is a challenging problem because of the...

Documents