biometric recognition in spatial frequency domain · recognition --- correlation filters face...

1

Vijayakumar Bhagavatula


Biometric Recognition in Spatial Frequency Domain

2


Acknowledgments

Dr. Marios SavvidesDr. Chunyan XieDr. Jason ThorntonDr. Krithika VenkataramaniPablo HenningsJingu HeoSungwon ParkRamzi AbiantonRamamurthy Bhagavatula

Technology Support Working Group (TSWG)CyLab

3


Outline

Use of spatial frequency domain for image recognition --- Correlation filtersFace recognitionIris recognitionFingerprint recognitionFace tracking examplesSummary

5


Challenge: Pattern VariabilityChallenge: To tolerate pattern variability (some times called distortions) while maintaining discriminationFacial appearance changes due to illuminationFingerprint image changes due to plastic deformation

6


Pattern Recognition ApproachesFeatureComputation

Feature Extraction

Classification

PriorKnowledge

Transform-ations

Statistics/Training Data

INPUT

CLASS

Statistical pattern recognition (e.g., Minimum error rate methods)Nonparametric methods (e.g., Nearest-neighbor methods)Discriminant methods (e.g., linear discriminant functions, artificial neural networks, support vector machines, etc.)Correlation filters

7


Correlation Pattern Recognition

Determine the cross-correlation between the reference and test images for all possible shifts.

When the target image matches the reference image exactly, output is the autocorrelation of the reference image; autocorrelation peaks at origin.

If the input r(x) contains a shifted version s(x-x0) of the reference signal, the correlator will exhibit a peak at x=x0.

If the input does not contain the reference signal s(x), the correlator output will be low .

If the input contains multiple replicas of the reference signal, resulting cross-correlation contains multiple peaks at locations corresponding to input positions.

( ) ( ) ( )c r x s x dxτ τ= −∫

Ref: B.V.K. Vijaya Kumar, A. Mahalanobis and R. Juday, Correlation Pattern Recognition, Cambridge University Press, UK, November 2005.

8


Shift-InvarianceDesired pattern can be anywhere in the input scene.Multiple patterns can appear in the scene.Simple matched filters unacceptably sensitive to rotations, scale changes, etc.

9


Correlation Plane Contour Map Correlation Plane Contour Map

Correlation Plane SurfaceCorrelation Plane Surface

M1A1 in the open M1A1 near tree line

SAIP ATR SDF Correlation Performance for Extended Operating

Conditions

Courtesy: Northrop Grumman

Adjacent trees cause some correlation noise

10


Correlation Filters

MatchNo Match

DecisionTest Image IFFT Analyze

Correlation output

FFT

Correlation Filter

Filter Design . . .Training Images

TrainingRecognition

11


Peak to Sidelobe Ratio (PSR)

σmeanPeakPSR −

=

1. Locate peak1. Locate peak

2. Mask a small 2. Mask a small pixel regionpixel region

3. Compute the mean and 3. Compute the mean and in a in a bigger region centered at the peakbigger region centered at the peak

PSR invariant to constant illumination changes

Match declared when PSR is large, i.e., peak must not only be large, but sidelobes must be small.

12


CMU PIE Database

13 cameras21 Flashes

Ref: T. Sim, S. Baker, and M. Bsat, “The CMU pose, illumination, and expression (PIE) database,” Proc. of the 5th IEEE Intl. Conf. on Automatic Face and Gesture Recognition, May 2002.

13


PIE Database, one face under 21 illuminations65 subjects

14


Train on 3, 7, 16, Train on 3, 7, 16, --> Test on 10.> Test on 10.

Match Quality = 40.95

15


Using the same filter as before, Match Quality = 30.60

Occlusion of Eyes

16


Match Quality = 22.38

Uncentered Images

17


ImpostorUsing someone else’s filterPSR = 4.77

18


Filter h

The center of correlation output

An image in the frequency domain in vector form xi

N images

Distortion-tolerant Correlation Filter Design

xd×d Xd×d

… …

xi

FFT

2 1 2[ , ,..., ]Nd N×=X x x x

……

hd2×1 d2×1 Hd×d

(0,0)i i ic + += =h x x h

19


Minimum Average Correlation Energy Filter

Minimizing average correlation energy can be done directly in the frequency domain by averaging the correlation plane energies Ei as follows.

hDhhXXh iii

d

kid

d

pidi kXkHpcE ++

==

==== ∑∑ *

1

221

1

21 )()()(

⎥⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢⎢

⎣

⎡

)(.

)3()2(

)1(

dX

XX

X

i

i

i

i

DhhhXXh +

=

+

=

=⎥⎥⎦

⎤

⎢⎢⎣

⎡== ∑∑

N

iiiN

N

iiNave EE

1

*1

1

1

h = D-1 X (X+ D-1 X)-1 c

Minimizing average correlation energy h+Dh subject to the constraints X+h=c leads to the MACE filter solution

Ref: A. Mahalanobis, B.V.K. Vijaya Kumar, and D. Casasent, “Minimum average correlation energy filters,” Appl. Opt. 26, pp. 3633-3630, 1987.

20


2D Impulse Response of MACE Filter

22


Features of Correlation Filters

Shift-invariant; no need for centering the test imageGraceful degradationCan handle multiple appearances of the reference image in the test imageClosed-form solutions based on well-defined metrics

Ref: B.V.K. Vijaya Kumar, A. Mahalanobis and Richard D. Juday, Correlation Pattern Recognition,Cambridge University Press, UK, November 2005.

23


PCA in Frequency DomainIs there any advantage to carrying out PCA on the Fourier transforms of training images?Since FT is a unitary transformation, apparently no differences (except for sign changes) in the eigenfaces obtained via space domain or frequency domain.

24


Phase retains more image information

27


EigenphasesPCA carried out on the phase-only versions of training image Fourier transforms

Frequency DomainFrequency Domain

Space DomainSpace Domain

28


Experiments

5-10,18,19,208

18,19,20153,16,207

8,9,10143,10,166

6,7,8131,2,7,165

7,10,19124,7,134

5,7,9,10112,7,163

5-10101,10,162

5-1293,7,161

29


Test Images

Training Images used were full imagesTraining Images used were full images

30


Recognition Rates (Full Test Images)

Ref: M. Savvides, and B.V.K. Vijaya Kumar, “Eigenphases Vs. Eigenfaces,” Intl. Conf. on Pattern Recognition (ICPR), Cambridge, England, 2004.

31


Recognition of Left-half Blocked Test Images

32


Recognition of Test Images with Eye Region

33


Face Recognition Grand Challenge (FRGC)

Ver 2.0625 Subjects; 50,000 Recordings; 70 Gbytes

To facilitate the advancement of face recognition research, FRGChas been organized by NIST

* P. J. Phillips, P. J. Flynn, T. Scruggs, K. W. Bowyer, J. Chang, K. Hoffman, J. Marques, J. Min, and W. Worek, "Overview of the Face Recognition Grand Challenge," In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2005

34


FRGC Dataset: Experiment 4

Generic Training Set consisting of 222 people with a total of 12,776 images

Gallery Set of 466 people (16,028) images total

Feature extraction Feature space generation

Reduced Dimensionality Feature Representation of Gallery Set

16,028

Probe Set of 466 people (8,014) images total

Reduced Dimensionality Feature Representation of Probe Set

8,014

Similarity Matching

Reduced Dimensional Feature Space

project project

35


FRGC “Gallery” Images

Controlled (Indoor) Controlled (Indoor)

16,028 gallery images of 466 people

36


FRGC “Probe” Images

Uncontrolled (Indoor)

37


FRGC “Probe” Images

Uncontrolled (Outdoor)

Outdoor illumination images are very

challenging due to harsh cast shadows

38


FRGC Baseline Results

The verification rate of PCA is about 12% at False Accept Rate 0.1%.

ROC curve from P. Jonathan Phillips et al (CVPR 2005)

39


Correlation Filters for Face Verification

Enrollment

Similarity Score

Verification

Only 1

Low performance

256 million correlations

Long time

40


Class-dependence Feature Analysis (CFA)

MotivationImprove the recognition rate by using the generic training setReduce the processing time by extracting features using inner products

Class-dependence Feature AnalysisModel a population for recognition using a set of people.

41


CFA

Extract features using correlation filtersRedundant class-dependence feature analysis (CFA)

– Train a correlation filter for each subject from the generic training set

– Extract a feature vector for each image by inner product

– Calculate the similarity score

42


Class Dependent Feature Analysis (CFA)

2y

ymace-2

T hy

Class1

….

-mace-2h

Class2 Class 222

…

1y 3y

uX)DX(XDh 111mace

−−+−=

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

=

n

2

1

u

uu

u

T0] 0 0 0[u1 =T1111][u2 = T0] 0 0 0[u222 =

In this figure, we’re building the MACE filter for class 2

43


CFA - 2

CFA basis vectors

-1maceh 2-maceh 3-maceh 222-maceh

…

]...[x]h...hh[xy 22221k222-mace2-mace1-macekk cccH TT ===kx

TN21 uuuu ],...,,[=

T]N11 1 .. 1 1 [1u =hmace-1 : all zeros except u1

……… T]N222222 1 .. 1 1 [1u =hmace-222 : all zeros except u222

Test input with no trained filters

uX)DX(XDh 111mace

−−+−=

]...[),(...),(,([ 22221 cccKKK == k222-macek2-macek-1macek xhxh ),xhy

Linear CFA

Nonlinear CFA using Kernels

44


KCFA Timeline

0

0 . 2

0 . 4

0 . 6

0 . 8

1

Ex p 4

Verif

icat

ion

Rat

es

P CAGS LDA

CFAKCFA- v 1KCFA- v 3

KCFA- v 5

82.4% @ 0.1 % FAR

(Latest Performance)

57


Iris Pattern

Detail from biological tissue:

Muscle ligaments

Connective tissue of stromal layer

Anterior folds, furrows, crypts

Random, detailed arrangement of tissue

58


Iris Segmentation

Eye image Bounded iris region

“Unwrapped” iris pattern

Standard iris segmentation: commonly used, proposed by Daugman

1 J.G. Daugman, “High Confidence Visual Recognition of Persons by a Test of Statistical Independence,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 15, no. 11, pp. 1148-61, Nov. 1993.

1

59


Iris Pattern “Unwrapping”

60


Iris Recognition: Correlation FiltersWe use correlation filters for iris recognition. We design a filter for each iris class using a set of training images.

match

no match

FFT-1x

Correlation filter

FFT

Determining an iris match with a correlation filter

Segmented iris pattern

64


Iris Pattern Deformation

Video Example

Landmark points for all images within one class

Clear deformation from:

Tissue changes AND/OR

Deviations in iris boundaries.

65


Eyelid Occlusion

Example : Eyelid artifacts in segmented pattern.

upper eyelid

lower eyelid

lower eyelid

Example: match comparison

For significant portion of area, similarity is lost.

66


ApproachProblem summary

Accurate pattern matching when patterns experience

relative nonlinear deformations

and partial occlusions

in addition to blurring and observation noise.

PATTERN SAMPLE

PATTERN TEMPLATE

GENERATE EVIDENCE

ESTIMATE STATES

MATCH SCORE

ApproachPROBABILISTIC MODEL:

DEFORM & OCCLUSION STATES

67


Probabilistic Method

Model :

Set of observed variables O (similarity cues, eyelid cues).

Set of hidden variables H (relative transformation states).

Given : Pair of iris patterns to compare.

Estimate : Conditional distribution

Prior probability

Rewards reasonable transformations

Observation Likelihood

Likelihood of observed data, given transformation

68


Hidden Variables : Deformation

Iris plane partitioned into 2D field:

Deformation described by vector field:

69


Hidden Variables : Occlusion

Occlusion described by binary field:

Hidden vars:

Field resolution tradeoff

> representative power

> computation

70


Deformation Evidence

Evidence about local similarities between template and image:

Correlation filter output

Produces similarity cues across local alignments.

Robust to pattern corruption by noise.

71


Eyelid Evidence

Classifier : Fisher Linear Discriminant in statistical feature space

-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.80

1

2

3

4x 10-3

Marginal separation, but enough for useful cues.

non-occluded

occluded

Example classifier output

This yields local occlusion cues .

72


Probabilistic Model Dependencies

Local observed variables: Local hidden variables:

73


Goal : Infer posterior distribution on hidden states:

Matching Process

Template

New pattern

Similarity evidence

Eyelid evidence

Inference technique :Loopy belief propagation

74


After Inference : Match Score Computation

Deterministic match score

Average similarity across unoccluded iris regions:

Better : Probabilistic match score

Expected score with respect to estimated posterior:

Advantage : accounts for uncertainty in the posterior.

75


Example : Match Comparisons

Template

Authentic

Match score:

0.28

Impostor

Match score:

0.01

77


78


ICE Phase I : Performance

Verification Rate at FAR = 0.1%

Experiment 1: 99.63 % Experiment 2: 99.04 %

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.80

0.02

0.04

0.06

0.08

0.1

0.12

0.14

non-match scores

match scores

Experiment 1 score distribution

84


86


NIST 24 DatabaseDigital Video of Live-scan Fingerprint Data Optical sensor DFR-90 from Identicator technology of 500 dpi resolutionChosen data set - plastic distortion set

The finger is rolled and twisted10 fingers of 10 people ( 5 female and 5 male )10 secs of MPEG2 movie per finger

300 images of size 448x478 pixels (padded to 512x512 pixels)Chosen subset - 10 thumb prints

Class 3 Class 10

87


Evaluation ProtocolTraining - uniformly sampled images from the 300 images of a classTest - correlate filters of each class against 300 images of all classes

Images of the same class as the filter – 300 authentics per filterImages of a different class from the filter – 2700 impostors per filte

88


Resolution Effect

128x128 64x64256x256512x512 32x32

K. Venkataramani, B.V.K. Vijaya Kumar, “Fingerprint verification using correlation filters”, Audio-and Video- based Biometric Person Authentication (AVBPA), UK, 2003 .

0.6%

89


Palmprints

Palmprints have a conglomerate of features.These include principal lines, smaller creases or wrinkles, fingerprint-like ridges and textures.Palmprints can be easily aligned about fiducial points of the hand’s geometry or shape.

90


Palmprint Verification

Design ProcessDesign Process

Training ImagesTraining Images

CorrelationCorrelationFilterFilter

FFTFFT IFFTIFFT VerificationVerification

AuthenticAuthentic ImposterImposter

91


Experiment SpecificationsPolyU Palmprint Database100 palms (classes)Left hands flipped to look as right hands3 images per class for training3 images per class for testing5 different experiments using region sizes with sides of 64, 80, 96, 112, and 128 pixels

Left handLeft hand Left hand flippedLeft hand flipped Right handRight handDatasetDataset

usedused

92


Palmprint Verification Results

104


SummaryCorrelation filters

Achieved excellent performance in face recognition grand challenge (FRGC)Performed very well in iris challenge evaluation (ICE)Also successful in fingerprint recognition and palmprintrecognition

Correlation filters provide a single matching engine for a variety of image biometrics --- making multi-biometric approaches feasible.

biometric recognition in spatial frequency domain · recognition --- correlation filters face...

Documents