a comparison of photometric normalisation … comparison of photometric normalisation algorithms for...

A Comparison of Photometric Normalisation Algorithms for Face Verification

James Short, Josef Kittler and Kieron MesserCentre for Vision, Speech and Signal Processing

University of SurreyGuildford, Surrey, GU2 7XH, UK

�j.short,j.kittler,k.messer�@eim.surrey.ac.uk

Abstract

The variation of illumination conditions of an object canproduce large changes in the image plane, significantly im-pairing the performance of face verification algorithms. Wepresent a comparison of five photometric normalisation al-gorithms for use in pre-processing face images for the pur-pose of verification. The algorithms are tested on variousconfigurations of three contrasting databases, namely theYale B database, the XM2VTS database and the BANCAdatabase.

1. Introduction

In general the variation between images of different facesis smaller than that of the same face taken in a variety of en-vironments. External factors such as pose and illuminationcan cause significant changes in the image plane. It has beenshown that illumination causes larger variation in face im-ages than pose [2]. The importance of illumination is furtherillustrated by examination of the eigenface method [19].Belhumeur improved the accuracy of a recognition systembased on eigenfaces, by removing the first three principalcomponents [4].

Several methods have been proposed to compensate forillumination changes. Kee used shape from shading meth-ods to generate a map of surface normals and reflectancesfor each subject. These were then used to synthesize an im-age under the same illumination conditions as the probe im-age. The two images were then compared directly [13]. Bel-humeur analytically investigated the subspace generated byvarying the illumination of an object, showing that it formeda convex cone [5]. This method requires a large amount oftraining data, but Lee showed that the subspace could begenerated using only nine images captured under a partic-ular set of illumination conditions [15]. Recognition is car-ried out by finding the distance of the probe image to the il-

lumination cone. These algorithms work well, but are com-putationally expensive.

In this paper we compare five algorithms for photomet-ric normalisation. A method based on principal compo-nent analysis, multiscale retinex [18], homomorphic filter-ing [10], a method using isotropic smoothing to estimate theluminance function and a method using anisotropic smooth-ing [11]. Three contrasting databases are used in the ex-periment. The Yale B database has only ten subjects butcontains a large range of illumination conditions [9]. Theimages in the XM2VTS database were all captured un-der a controlled environment in which illumination vari-ation is minimised [16]. The BANCA database containsmuch more realistic illumination conditions [3]. The meth-ods were tested extensively on the three databases usingnumerous protocols. We show that the homomorphic filterand the anisotropic method yield the most consistent resultsacross all three databases.

In the next section, we outline each of the normaliza-tion algorithms. Section 3 details the experimental proce-dures used to compare the methods. The results of the ex-periment are presented in Section 4 and we conclude in Sec-tion 5.

2. Methods

In this section we describe five methods of photometri-cally normalising images. Histogram equalization was usedwith each method.

2.1. Principal Component Analysis Method

Principal component analysis is a popular method of re-ducing the dimensionality of a set of data. This is carriedout using eigen analysis of the data covariance matrix tofind an ordered set of orthogonal basis vectors that best de-fine the directions of greatest variance. When applied to thetask of face verification, these vectors are known as eigen-faces [19].

Proceedings of the Sixth IEEE International Conference on Automatic Face and Gesture Recognition (FGR’04) 0-7695-2122-3/04 $ 20.00 © 2004 IEEE

A face image �� can be represented as a set of coeffi-cients �� where each coefficient corresponds toan eigenface �� so that the face image is equalto the weighted sum of the eigenfaces.

��

��

�� (1)

Bischof and Leonardis [6] take advantage of the linearity ofEquation 1 and convolve both sides with a kernel �.

��

��

�� (2)

where � denotes convolution with a kernel �.This equation holds true, irrespective of the convolution

kernel. As the nature of illumination variation is of low fre-quency, we can choose a kernel designed to remove low fre-quency information. The input image of a face verificationsystem can be filtered and the resulting image decomposedinto a set of coefficients corresponding to a similarly fil-tered set of eigenfaces. These coefficients can then be usedwith the original set of eigenfaces to reconstruct the image.

2.2. Multiscale Retinex Method

An image acquired by a camera is the product of twocomponents, reflectance �� and illumination �� [12]

�� (3)

Illumination is the amount of light falling on the object dueto the light source. Reflectance is the amount of light re-flected from the surface of the object.

Land [14] decomposed the image into reflectance and il-lumination, by estimating the illumination as a low pass ver-sion of the original image, thus finding the reflectance by di-viding the image by the illuminance function.

Rahman [18] improved upon Lands work by estimatingillumination as a combination of images generated by lowpass filtering the original image with Gaussians of varyingwidths.

��

��

�� (4)

where � is a weighting term and the Gaussian �� has a width �� at a scale as defined by

��

��

�� (5)

2.3. Homomorphic Filtering

A homomorphic filter separates illumination and re-flectance by taking the logarithm of Equation 3 [10].

�� (6)

The Fourier transform of the result gives

�� (7)

which can be written as the sum of two functions in the fre-quency domain

�� (8)

�� is composed of mostly high frequency components and�� of mostly low frequency components. � can be con-volved with a filter of transfer function�� that reducesthe low frequencies and amplifies high frequencies, thus im-proving contrast and compressing dynamic range.

�� (9)

The processed image can be found by inverse Fourier trans-forming Equation 9 and taking the exponential.

� �� (10)

2.4. Isotropic Smoothing

Illumination can be estimated as a blurred version of theoriginal image. This blurring can be implemented by con-structing an illumination function that is similar to theoriginal image � but contains a smoothing constraint. Theillumination function can be constructed by minimizing thefollowing cost function.

��

��

�

��

��

�

��

�� (11)

where parameter � controls the relative importance of thesmoothness constraint. The problem in Equation 11 can besolved by a Euler-Lagrange diffusion process which can bediscretized as follows

��

�� (12)

where �� refers to the derivative with respect to each ofthe four adjacent neighbouring pixels, i.e. the subscripts � ,�, � and � define the direction of the neighbouring pixel,so that �� is the value of the �� pixel minus thevalue of the �� pixel. The amount of smoothing iscontrolled by the parameter c. This Equation can be quicklysolved using multigrid methods [7].


2.5. Anisotropic Smoothing

The cost function in Equation 11 can be generalized byincorporating a weight, �� , mimicking the perceptiongain [1, 11], in the term maximising goodness of fit of thesolution to data, i.e.

��

��

��

��

��

��

��

��

��

(13)Gross suggests that the coefficient should be a function oflocal image contrast, so as to enhance the image contrast ina manner which is insensitive to illumination. Using the re-ciprocal of Weber’s contrast as the smoothing parameter,the equation becomes

��

��

��

��

��

�

��

��

��

�(14)

where

� ��

�

��

�� (15)

is known as Weber’s contrast. As with isotropic diffusion,multigrid methods are used to solve the equation.

3. Comparison of the Methods

For a thorough evaluation of the effectiveness of the pho-tometric normalisation algorithms, we investigated their ef-fect on the accuracy of a face verification system. The sys-tem was tested using the various normalisation algorithmson the Yale B database [9], the XM2VTS database [16]and the BANCA database [3]. A benchmark set of im-ages, without illumination normalisation, denoted Geomet-ric in the results, were also tested. The Yale B database con-tains images under widely varying illumination conditionsand poses of ten subjects. Tests were carried out using thefrontal pose set of images with varying illumination. Boththe XM2VTS database and the BANCA database have de-tailed evaluation protocols.

The face verification software is based on linear discrim-inant analysis. It uses three sets of data, namely training,evaluation and testing. For each claim, the system gener-ates a score reflecting how well the probe image matchesthe claimed identity. These scores are generated for theevaluation set in order to find a threshold correspondingto the equal error rate i.e. the point at which the false ac-ceptance rate (FAR) equals the false rejection rate (FRR).

Figure 1. Examples of the YaleB (top),XM2VTS (middle) and BANCA (bottom)database images

This threshold is then used for the test set, from which theFAR and FRR can be calculated. The final result is gener-ated by averaging FAR and FRR, i.e. the half total error rate(HTER).

The Yale B database contains 64 different illuminationconditions for 10 subjects. The illumination conditions area single light source, the position of which varies horizon-tally (from -130Æ to 130Æ) and vertically (from –40Æ to 65Æ).Due to the limited size of the Yale B database, the test wascarried out with the face verification system having beentrained on the XM2VTS database. The ROC curve was thengenerated from the resulting scores.

The XM2VTS database contains images of 295 subjects,captured over 4 sessions in a controlled environment. Thedatabase uses a standard protocol. The Lausanne protocolsplits the database randomly into training, evaluation andtest groups [16]. The training group contains 200 subjectsas clients, the evaluation group an additional 25 subjects asimpostors and the testing group another 70 subjects as im-postors.

The BANCA database was captured over twelve sessionsin three different scenarios and has a population of 52 sub-jects (26 male and 26 female). Sessions 1–4 were capturedin a controlled scenario, sessions 5–8 were captured in adegraded scenario which was captured using a simple webcam and session 9–12 were captured in an adverse scenario.

The BANCA database has seven configurations of train-ing and testing data incorporating different permutations ofdata from the twelve sessions.

The seven configurations are Matched Controlled (MC),


Matched Degraded (MD), Matched Adverse (MA), Un-matched Degraded (UD), Unmatched Adverse (UA),Pooled test (P), and Grand test (G). The content ofeach configuration is described by table 1. T repre-sents clients for training, I impostors for testing and Crepresents clients for testing.

Session MC MD MA UD UA P G1 TI T T TI TI2 CI CI CI3 CI CI CI4 CI CI CI5 TI I I TI6 CI CI CI CI7 CI CI CI CI8 CI CI CI CI9 TI I I TI

10 CI CI CI CI11 CI CI CI CI12 CI CI CI CI

Table 1. How different sessions are used forthe protocols of the BANCA database

4. Experimental Results

This section presents a summary of the results of testingthe various algorithms on the three databases.

Firstly, the results of the Yale B database experimentare presented in Figure 2. The homomorphic filter shows asmall improvement in performance. The anisotropic methodachieves a much better performance and the value of the lo-cal contrast coefficients is illustrated by the huge improve-ment in accuracy over the isotropic method. By far the bestperformance is obtained by the retinex method.

The second experiment was carried out using the twoconfigurations of the XM2VTS database. The results ob-tained from configuration one of the database show the ho-momorphic filter achieving the highest accuracy, narrowlyoutperforming the anisotropic method. In contrast, the re-sults from configuration two of the database show supe-rior results for the anisotropic method. When applied to theXM2VTS database, where the illumination is controlled,the retinex method actually shows a considerable drop inperformance. The filtering process therefore removes valu-able discriminatory information in addition to the illumina-tion information.

The third experiment was carried out on the BANCAdatabase. The results are summarised in table 3 whichshows the half total error rates.

The seven configurations of the BANCA database showdiffering results. The principal component analysis method

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

False Acceptance

Tru

e A

ccepta

nce

geometrichomomorphicisotropicanisotropicretinexpca

Figure 2. ROC curve for the Yale B database

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.20.8

0.82

0.84

0.86

0.88

0.9

0.92

0.94

0.96

0.98

1

False Acceptance

Tru

e A

ccepta

nce


Figure 3. ROC curve for configuration 1 of theXM2VTS database

improves performance on the matched and grand configu-rations, but degrades performance on the unmatched andpooled configurations. The anisotropic method yields thebest results by a large margin over all but the matched ad-verse and unmatched adverse configurations. In the caseof the unmatched adverse configuration, the anisotropicmethod is narrowly better than the homomorphic filter andin the matched adverse case it performs significantly worse.The test sets in these configurations are formed exclu-sively from sessions 10, 11 and 12 which correspond tothe adverse scenarios. The isotropic method outperforms


0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.20.8

0.82

0.84

0.86

0.88

0.9

0.92

0.94

0.96

0.98

1

False Acceptance

Tru

e A

ccepta

nce


Figure 4. ROC curve for configuration 2 of theXM2VTS database

ProtocolConfiguration 1 Configuration 2

Method FAR FRR HTER FAR FRR HTERGeometric 6.06 5.50 5.78 4.71 3.75 4.23PCA 6.22 6.00 6.11 4.75 4.75 4.75Retinex 19.27 12.00 15.64 8.87 9.50 9.18Homomorphic 4.43 3.25 3.84 2.81 2.25 2.53Isotropic 6.58 5.50 6.04 4.21 4.50 4.35Anisotropic 4.91 4.00 4.45 3.53 1.50 2.51

Table 2. Performance on both protocols ofthe XM2VTS database. Values shown are theFAR: false acceptance rate, FRR: false rejec-tion rate, and HTER: half total error rate

the anisotropic method on the matched adverse configura-tion, i.e. the local contrast coefficients lead to a reduction inperformance in this case.

5. Conclusions

The performance of various photometric normalisa-tion algorithms has been compared on three very differentdatabases and over numerous configurations.

The PCA method shows inconsistent improvements.The illumination variation in the Yale B database is

vast and the retinex method proves to be superior, how-ever as a photometric normalisation algorithm it per-forms badly on the realistically illuminated XM2VTS andBANCA databases.

ProtocolMethod MC MD MA UD UA P GGeometric 14.5 14.1 13.3 20.1 28.7 22.8 11.1PCA 9.0 10.7 13.0 25.4 29.5 24.5 8.7Retinex 8.0 5.9 13.6 11.1 26.8 17.2 6.3Homomorphic 6.1 8.3 7.6 16.9 21.5 18.2 6.6Isotropic 6.1 6.0 8.9 14.1 22.5 17.2 5.3Anisotropic 4.2 3.7 9.6 8.6 20.7 13.4 3.6

Table 3. Performance on all protocols ofthe BANCA database. Values shown are theHTER: half total error rate

The XM2VTS database in contrast with the BANCAdatabase has excellent illumination conditions. There are noexamples of shadowing and all images are captured in thesame environment. The difference between the homomor-phic filter and the anisotropic methods demonstrated by theXM2VTS database is vary small and based entirely on theselection of training and testing data. However when illu-mination conditions are degraded, such as in the BANCAdatabase, the anisotropic method is clearly shown to be su-perior.

References

[1] S. Acton, “Multigrid Anisotropic Diffusion” IEEE Trans. Im-age Processing, vol. 7, 1998

[2] Y. Adini, Y. Moses, and S. Ullman, “Face recognition: theproblem of compensating for illumination changes.” IEEETrans. Pattern Anal. Mach. Intelligence, vol. 19(7), pp721–732, 1997

[3] E Bailly-Bailliere, et al., “The BANCA Database and Evalua-tion Protocol” AVBPA, 2003

[4] P. Belhumeur, J. Hespanha, D. Kriegman, “Eigenfaces vs.Fisherfaces: Recognition using Class Specific Linear Projec-tion” IEEE Trans. Pattern Anal. Mach. Intelligence, vol. 19,pp711-720, 1997

[5] P. Belhumeur, D. Kriegman, ”What is the Set of Images of anObject Under All Possible Lighting Conditions?” IEEE Proc.Conf. Computer Vision and Pattern Recognition, 1996

[6] H. Bischof, A. Leonardis, “Illumination InsensitiveEigenspaces” Proc. ICCV, pp233-238, 2001

[7] W. Briggs, V. Henson, S. McCormick, “A Multigrid Tutorial”Siam, Second ed.

[8] B. Funt, et al., “Luminance Based Multiscale Retinex” Proc.AIC, 1997

[9] A. Georghiades, P. Belhumeur, D. Kriegman, “From Few toMany: Illumination Cone Models for Face Recognition underVariable Lighting and Pose” IEEE Trans. Pattern Anal. Mach.Intelligence, vol. 23, pp643-660, 2001

[10] R. Gonzalez, R. Woods, “Digital Image Processing” Pren-tice Hall, Second ed.

[11] R. Gross, V. Brajovic, “An Image Preprocessing Algorithm”AVBPA, pp10-18, 2003


[12] B. Horn, “Robot Vision” MIT Press, 1998[13] S. Kee, K. Lee and S. Lee, “Illumination Invariant Face

Recognition Using Photometric Stereo” IEICE Trans. Inf &Syst, Vol.E83-D, No.7, 2000

[14] E. Land, J. McCann, “Lightness and Retinex Theory” Jour-nal of the Optical Society of America, vol. 61, pp1-11, 1971

[15] K. Lee, J. Ho, D. Kriegman, “9 Points of Light: AquiringSubspaces for Face Recognition Under Variable Lighting”IEEE Proc. Conf. Computer Vision and Pattern Recognition,2001

[16] K. Messer, J. Matas, J. Kittler, “XM2VTSDB: The extendedM2VTS Database” AVBPA, 1999

[17] P. Perona, J. Malik, “Scale-Space and Edge Detection Us-ing Anisotropic Diffusion” IEEE Trans. Pattern Anal. Mach.Intelligence, vol.12(7), 1990

[18] Z. Rahman, G. Woodell, D. Jobson, “A Comparison ofthe Multiscale Retinex with other Image Enhancement Tech-niques” Proceedings of the IS&T 50th Anniversary Confer-ence, 1997

[19] M. Turk, A. Pentland, “Eigenfaces for Recognition” J. Con-gitive Neuroscience, vol. 3, pp 71-86, 1991


a comparison of photometric normalisation … comparison of photometric normalisation algorithms for...

Documents