face image analysis: recognition and presentation attack

Face Image Analysis:Recognition and Presentation Attack Detection

Shervin R. Arashloo

March 12, 2021

Shervin R. Arashloo Face Image Analysis March 12, 2021 1 / 41

Outline

1 Face RecognitionGraph MatchingStatistical Shape PriorMultiresolution and parallel optimisationClass-specific Discriminant Analysis

2 Face Presentation Attack DetectionOne-Class FormulationClient-specific one-class learningClassifier FusionOne-Class Fisher Discriminant AnalysisKernel Fusion

Face Recognition Major Challenges

Pose (out-of-plane rotation)

Illumination

Expression

Deformable Matching

Object Recognition and Matching

Bag of words

Discarding relational information between object primitives

Graph matching

Incorporating structural information for matching

Graphs and Hypergraphs

Nodes represent object primitives

Edges/hyperedges encode their dependencies

Deformation Model

E (x ; θ) =∑u∈V

θu(xu) +∑

(u,v)∈E

θuv (xu, xv )

labels are 2D displacements

θu measures the degree of similarity between graylevel contents of twoblocks

θuv enforces smoothness over deformation map

Decomposed Model

Complexity: O(νL2)L: Cardinality of the label set for the horizontal and vertical displacementsν: Number of nodesThe complexity scales quadratically in number of labels!

Decomposed model: modelling horizontal and vertical labels separately

Complexity of the decomposed model: O(νL)scales linearly in number of labels.

Decomposed Model

Complexity: O(νL2)L: Cardinality of the label set for the horizontal and vertical displacementsν: Number of nodesThe complexity scales quadratically in number of labels!

Decomposed model: modelling horizontal and vertical labels separately

Complexity of the decomposed model: O(νL)scales linearly in number of labels.

Statistical Shape Prior

Deformable models are broadly classified into two categories:

Free-form

Only general continuity and smoothness constraints are considered; can bematched to an arbitrary shape (e.g. Snake Model)

Parametric

Incorporate a general shape of the object of interest and encode specialattributes of an object and its variations-are more robust to occlusions andspurious structures (e.g. Active Shape Model)

Deformation Energy

The updated objective function:

E (x ; θ) =∑s∈V

θs(xs) +∑

(s,u)∈E

θsu(xs , xu)+θg (x)

θg (x) measures deviation from mean shape in the PCA space

Multiresolution Optimisation: Supercoupling Approach

Moving from coarser towardsfiner scales

Challenge: maintaining aconsistency between energyfunctions at different scales

The Supercoupling algorithm consists of two main steps:renormalisation and processing.

Renormalisation

Iteratively constructing coarser and coarser grids of nodes and acorresponding sequence of energy functions

Processing

Performing a multi-scale coarse-to-fine optimisation starting from thecoarsest scale moving towards the finest one

Multiresolution Optimisation: Supercoupling Approach

Moving from coarser towardsfiner scales

Challenge: maintaining aconsistency between energyfunctions at different scales

The Supercoupling algorithm consists of two main steps:renormalisation and processing.

Renormalisation

Iteratively constructing coarser and coarser grids of nodes and acorresponding sequence of energy functions

Processing

Performing a multi-scale coarse-to-fine optimisation starting from thecoarsest scale moving towards the finest one

Optimisation: Dual Decomposition

Two steps:

Decompose the problem into anumber of subproblems andsolve each one separately

Enforce consistency betweensubproblems

A large number of independently solvable subproblems motivate a parallelprocessing!

Graphical Processing Units

Array of highly threaded streaming multiprocessors

High speed shared memory visible to all processing elements as wellas a number of other types of memory

Two steps:

Speed-up Gains

Parallel Processing ∼ 24x

Multiresolution Analysis ∼ 5x

Other Techniques ∼ 1.8x

Overall ∼ 218x

Some Matching Results

Class-specific Discriminant Analysis

Subspace methods:

PCA, LDA, etc. (insufficiency of linear models)

Kernel methods: KPCA, KDA, etc.

Class-Specific Kernel Discriminant Analysis

Learns a class from a single labelled example (one-shot learning)

Results in subject-specific projections

Class-specific Discriminant Analysis

Subspace methods:

PCA, LDA, etc. (insufficiency of linear models)

Kernel methods: KPCA, KDA, etc.

Class-Specific Kernel Discriminant Analysis

Learns a class from a single labelled example (one-shot learning)

Results in subject-specific projections

Classification

Conventional hand-crafted descriptors used:

Local Binary Pattern (LBP)

Local Phase Quantisation (LPQ)

Binarised Statistical Image Features (BSIF)

For classification:

Kernel fusion over the three descriptors

Correspondences are taken into account

Labelled Faces in the Wild (LFW) Dataset

Real world variations of facial images such as pose, illumination,expression, occlusion, low resolution, blur, etc.Contains 13,233 images of 5,749 subjectsPair-matching problem

Figure: Sample images from the LFW dataset.

Evaluation Protocols

Results: Unsupervised

Results: Image-Restricted, No Outside Data

Observations

In the most relaxed scenario, state-of-the-art deep methods achievemore than 98% accuracy

Very large labelled data required to exploit the potential of deepnetworks

Graph-based methods very efficient in terms of the number of trainingdata

Typically high performance computing resources required to traindeep nets

Computationally demanding MAP inference in MRF vs. relatively fastoutput generation in deep networks

Journal Articles Relevant to Face Recognition

Arashloo, S.R. and Kittler, J., ”Energy Normalization for Pose-Invariant Face Recognition Based on MRF Model ImageMatching”, Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 33, no. 6, pp. 1274-1280, Jun. 2011.

Arashloo, S.R., Kittler, J. and Christmas, W.J., ”Pose-Invariant Face Recognition by Matching on MultiresolutionMRFs Linked by Super-coupling Transform”, Computer Vision and Image Understanding , Elsevier, special issue ongraph-based representations in computer vision, vol. 115, issue 7, pp. 1073-1083, July 2011.

Arashloo, S.R. and Kittler, J., ”Fast Pose Invariant Face Recognition Using Supercoupled Multi-resolution MarkovRandom Fields on a GPU”, Pattern Recognition Letters, Elsevier, special issue on celebrating the life and work of MariaPetrou, vol. 48, pp. 49-59, Oct. 2014.

Arashloo, S.R. and Kittler, J., ”Class-Specific Kernel Fusion of Multiple Descriptors for Face Verification UsingMultiscale Binarised Statistical Image Features”, Information Forensics and Security, IEEE Transactions on, special issueon facial biometrics in the wild, vol. 9, no.12, pp. 2100-2109, Dec. 2014.

Arashloo, S.R., ”Incorporating Point Distribution Model Priors into MRFs Using Convex Quadratic Programming”,Machine Vision and Applications, Springer, vol. 27, no. 6, pp. 821-832, Aug. 2016.

Arashloo, S.R., ”A Comparison of Deep Multilayer Networks and MRF Matching Models for Face Recognition in theWild”, Computer Vision, IET , vol. 10, no. 6, pp. 466-474, Sep. 2016.

Arashloo, S.R., ”Multiscale Binarised Statistical Image Features for Symmetric Face Matching Using MultipleDescriptor Fusion Based on Class-Specific LDA”, Pattern Analysis and Applications, Springer, pp. 1-14, May 2015.

Face Presentation Attack Detection

Problem:An unauthorised subject tries to get illegitimate access to a facerecognition system by presenting fake biometrics traits

Typical presentation attack instruments:

Video Replay

Face Presentation Attack Detection

Problem:An unauthorised subject tries to get illegitimate access to a facerecognition system by presenting fake biometrics traits

Typical presentation attack instruments:

Video Replay

Points of Attack to a Biometrics System

Samples captured by a recognition system

(a) corresponds to genuine (bona fide) samples(b),(c) and (d) represent presentation attacks 1

1images from the ”MSU Mobile Face Spoofing Database (MSU MFSD)” dataset.

Samples captured by a recognition system

(a) corresponds to genuine (bona fide) samples(b),(c) and (d) represent presentation attacks 1

1images from the ”MSU Mobile Face Spoofing Database (MSU MFSD)” dataset.

Conventional approach

The Conventional approach is Two-Class Classification:

Training samples include both bona-fide (genuine) and attack samples

A binary classifier is trained to classify an image (sequence) as eitherbona-fide or attack

Drawbacks:

High cost of collecting attack samples

Low generalisation

Different imaging conditionsNovel attack types unseen during training!

Conventional approach

The Conventional approach is Two-Class Classification:

Training samples include both bona-fide (genuine) and attack samples

A binary classifier is trained to classify an image (sequence) as eitherbona-fide or attack

Drawbacks:

High cost of collecting attack samples

Low generalisation

Different imaging conditionsNovel attack types unseen during training!

One-Class Formulation of Face PAD problem

Genuine samples considered as target observations and attacks asanomalies

Our approach learns from genuine data only: not biased towardsspecific attack types!

Goal: Characterise the support domain of probability density function ofgenuine samples

One-Class Formulation of Face PAD problem

Genuine samples considered as target observations and attacks asanomalies

Our approach learns from genuine data only: not biased towardsspecific attack types!

Goal: Characterise the support domain of probability density function ofgenuine samples

Presentation Attack Detection: Common approach

The common approach is Subject-Independent Detection:

A common classifier is trained to detect PA w.r.t. all subjects

Drawback:

Ignores any class-specific information useful for PAD

Client-specific modelling

Deploying client-specific information for face spoofing detection

Subject-specific score distributions motivate a distinct threshold foreach client

Classifier Fusion

Motivation: Different one-class learners+diverse representations

2J. Kittler, M. Hatef, R. P. W. Duin and J. Matas, ”On combining classifiers,” in IEEE Transactions on Pattern Analysis

and Machine Intelligence, vol. 20, no. 3, pp. 226-239, March 1998, doi: 10.1109/34.667881.

Classifier Fusion

Motivation: Different one-class learners+diverse representations

2J. Kittler, M. Hatef, R. P. W. Duin and J. Matas, ”On combining classifiers,” in IEEE Transactions on Pattern Analysis

and Machine Intelligence, vol. 20, no. 3, pp. 226-239, March 1998, doi: 10.1109/34.667881.

Diversity in Experts

Multiple regions:

Multiple Deep CNN’s:

GoogleNet

ResNet50

Multiple One-class learners:

One-class Support Vector Data Description

Mahalanobis distance

Gaussian mixture model

The Impact of Classifier Fusion

One-Class Fisher Discriminant Analysis

The Fisher classifier:

F(β) =β>Σbβ

β>Σwβ

Σb: between-class scatter matrixΣw : within-class scatter matrixβ: the Fisher discriminant

Originally developed for two-classclassification but can be adapted to aone-class setting!

F(β) =β>Σbβ

β>Σwβ

F(β) =β>Σbβ

β>Σwβ

Regression-Based Formulation

Solving for the Fisher discriminant requires costly eigendecompositionof dense matrices

Not convenient to impose regularisation on the discriminant forimproved generalisation performance

Regularised regression-based reformulation in the Hilbert space

minθ‖θ‖2

n∑i=1

(1− θ>υ(xi ))2

Tikhonov regularisation

Or its dual problem as

maxω−ω>Kω − σω>ω + 2ω>1

σ = n/δK: kernel matrix1: denotes an n-dimensional vector of ones

minθ‖θ‖2

n∑i=1

(1− θ>υ(xi ))2

minθ‖θ‖2

n∑i=1

(1− θ>υ(xi ))2

minθ‖θ‖2

n∑i=1

(1− θ>υ(xi ))2

minθ‖θ‖2

n∑i=1

(1− θ>υ(xi ))2

Kernel Fusion

Fusing multiple representations via a sum rule:K = K1 + K2 + · · ·+ KJ

Diversity in the representationsMultiple Regions

Different Deep CNN’s

GoogleNetResNet50VGG16

Kernel Fusion Evaluation Results

Unseen attack evaluation protocol

Multiple Kernel Learning

The ideaInstead of using fixed combination rules, learn linear combination weights

Objective function

−α>(∑J

j=1 βjKj)α− δα>α+ 2α>1

s.t. β ≥ 0,R(β)

kernel weights

Different possibilities for regularisation R(β):

`p-norm ‖β‖pp ≤ 1; p ≥ 1

induces sparsity

mixed (r , p)-norm∥∥ββ>∥∥

r ,p≤ 1; r , p ≥ 1

induces sparsityenables interaction between kernels

Both regularisations lead to convex optimisation problems!

Objective function

−α>(∑J

s.t. β ≥ 0,R(β)

kernel weights

`p-norm ‖β‖pp ≤ 1; p ≥ 1

induces sparsity

r ,p≤ 1; r , p ≥ 1

Objective function

−α>(∑J

s.t. β ≥ 0,R(β)

kernel weights

`p-norm ‖β‖pp ≤ 1; p ≥ 1

induces sparsity

r ,p≤ 1; r , p ≥ 1

Objective function

−α>(∑J

s.t. β ≥ 0,R(β)

kernel weights

`p-norm ‖β‖pp ≤ 1; p ≥ 1

induces sparsity

r ,p≤ 1; r , p ≥ 1

Objective function

−α>(∑J

s.t. β ≥ 0,R(β)

kernel weights

`p-norm ‖β‖pp ≤ 1; p ≥ 1

induces sparsity

r ,p≤ 1; r , p ≥ 1

Objective function

−α>(∑J

s.t. β ≥ 0,R(β)

kernel weights

`p-norm ‖β‖pp ≤ 1; p ≥ 1

induces sparsity

r ,p≤ 1; r , p ≥ 1

Abnormality and Novelty Detection

Abnormality Detectiondetect abnormal observations whenthe classifier is trained using a set ofnormal samples of the correspondingclass

Novelty Detectionassess the novelty of a new samplebased on previously observed samples

Figure: (a) Abnormal image detection: Top three rows arenormal images from the PASCAL dataset. Bottom three rowsare abnormal images from the Abnormal 1001 dataset. (b)Novelty detection: images from the Caltech256 dataset.

`p-norm Evaluation Results

(r , p)-norm Evaluation Results

Journal Articles Relevant to Presentation Attack Detection

Arashloo, S.R. and Kittler, J., ”An Anomaly Detection Approach to Face Spoofing Detection: A New Formulation andEvaluation Protocol”, IEEE Access, vol. 5, pp. 13868-13882, 2017.

Arashloo, S.R. and Kittler, J., ”Robust One-Class Kernel Spectral Regression”,Neural Networks and Learning Systems, IEEE Transactions on, vol. 32, no. 3, pp. 999-1013, March 2021, doi:

10.1109/TNNLS.2020.2979823.

Fatemifar, S., Arashloo, S.R., Awais, M., Kittler, J., ”Client-Specific Anomaly Detection for Face Presentation AttackDetection”, Pattern Recognition, Elsevier, vol. 112, 107696, 2021.

Arashloo, S.R., ”Unseen Face Presentation Attack Detection Using Sparse One-Class Multiple Kernel FusionRegression”, Circuits and Systems for Video Technology, IEEE Transactions on, doi: 10.1109/TCSVT.2020.3046505,2020.

Arashloo, S.R., ”`p -Norm Multiple Kernel One-Class Fisher Null-Space”, under review.

Arashloo, S.R., ”Mixed (r, p)-Norm One-Class Multiple Kernel Fisher Null-Space”, in preparation.

face image analysis: recognition and presentation attack

Documents

gradient-orientation-based pca subspace for novel face...

open research onlineoro.open.ac.uk/58380/8/58380.pdf · •...

face recognition using pca and eigen face...

face recognition & biometric systems, 2005/2006 face...

face recognition by independent component analysis …...

face recognition accuracy of forensic examiners,...

face recognition

face recognition

a summary of deep models for face recognition - wellesley...

deep face recognition - nvidia · herta deep face...

cross-domain face presentation attack detection via multi...

face recognition cpsc 601 biometric course. topics...

face detection and recognition - computer...

face recognition cpsc 4600/5600 @ utc/cse. 2 face...

face recognition face identification