Federated Test-Time Adaptive Face Presentation Attack Detection withDual-Phase Privacy Preservation

Rui Shao1 Bochao Zhang1 Pong C. Yuen1 Vishal M. Patel21 Department of Computer Science, Hong Kong Baptist University, Hong Kong

2 Department of Electrical and Computer Engineering, Johns Hopkins University, USA

Abstract— Face presentation attack detection (fPAD) playsa critical role in the modern face recognition pipeline. Thegeneralization ability of face presentation attack detectionmodels to unseen attacks has become a key issue for real-worlddeployment, which can be improved when models are trainedwith face images from different input distributions and differenttypes of spoof attacks. In reality, due to legal and privacy issues,training data (both real face images and spoof images) are notallowed to be directly shared between different data sources. Inthis paper, to circumvent this challenge, we propose a FederatedTest-Time Adaptive Face Presentation Attack Detection withDual-Phase Privacy Preservation framework, with the aim ofenhancing the generalization ability of fPAD models in bothtraining and testing phase while preserving data privacy. In thetraining phase, the proposed framework exploits the federatedlearning technique, which simultaneously takes advantage ofrich fPAD information available at different data sources byaggregating model updates from them without accessing theirprivate data. To further boost the generalization ability, in thetesting phase, we explore test-time adaptation by minimizing theentropy of fPAD model prediction on the testing data, whichalleviates the domain gap between training and testing dataand thus reduces the generalization error of a fPAD model. Weintroduce the experimental setting to evaluate the proposedframework and carry out extensive experiments to providevarious insights about the proposed method for fPAD.

I. INTRODUCTION

Recent advances in face recognition methods haveprompted many real-world applications, such as automatedteller machines (ATMs), mobile devices, and entrance guardsystems, to deploy this technique as an authenticationmethod. Wide usage of this technology is due to both highaccuracy and convenience it provides. However, many recentworks [14], [3], [28], [22], [12], [21], [23], [24] have foundthat this technique is vulnerable to various face presentationattacks such as print attacks, video-replay attacks [2], [30],[4], [28], [12] and 3D mask attacks [10], [11]. Therefore, de-veloping face presentation attack detection (fPAD) methodsthat make current face recognition systems robust to facepresentation attacks has become a topic of interest in thebiometrics community.

In this paper, we consider the deployment of a fPADsystem in the real-world scenario. We identify two typesof stakeholders in this scenario – data centers and users.Data centers are entities that design and collect fPADdatasets and propose fPAD solutions. Typically data centersinclude research institutions and companies that carry outthe research and development of fPAD. These entities haveaccess to both real data and spoof data and therefore are

Server

Client #2 Client #3 Client #K

Model UploadModel Download

Inference

DataCenter #2

User #1 User #2 User #3 User #4

Traditional Face Presentation Attack Detection

. . .

User #5 User #6 User #7 User #8 User #9

User #1 User #2 User #3 User #4

Federated Test-Time Adaptive Face Presentation AttackDetection with Dual-Phase Privacy Preservation

User #5 User #6 User #7 User #8 User #9

Client #1 . . .

Federated Learning

Server

Model Updating

DataCenter #1

DataCenter #3

DataCenter #K

DataCenter #2

. . .DataCenter #1

DataCenter #3

DataCenter #K

Training Phase

Testing Phase

Fig. 1. Comparison between fPAD (top), traditional federated learning(middle) and the proposed framework (bottom).

able to train fPAD models. Different data centers maycontain images of different identities and different typesof spoof data. However, each data center has limited dataavailability. Real face images are obtained from a small setof identities and spoof attacks are likely to be from a fewknown types of attacks. Therefore, these fPAD models havepoor generalization ability [21], [24] and are likely to bevulnerable against attacks unseen during training.

On the other hand, users are individuals or entities thatmake use of fPAD solutions. For example, when a fPADalgorithm is introduced in mobile devices, mobile devicecustomers are identified as users of the fPAD system. Usershave access only to real data collected from local devices.Due to the absence of spoof data, they cannot locallytrain fPAD models. Therefore, each user relies on a modeldeveloped by a data center for fPAD as shown in Figure 1(top). Since data center models lack generalization ability,inferencing with these models are likely to result in erroneouspredictions.

It has been shown that utilizing real data from differ-ent input distributions and spoof data from different typesof spoof attacks through domain generalization and meta-learning techniques can significantly improve the general-

arX

iv:2

110.

1261

3v1

[cs

.CV

] 2

5 O

ct 2

021

ization ability of fPAD models [21], [24]. Therefore, theperformance of fPAD models, shown in Figure 1 (top), canbe improved if data from all data centers can be exploitedcollaboratively. In reality, due to data sharing agreementsand privacy policies, data centers are not allowed to sharecollected fPAD data with each other. For example, when adata center collects face images from individuals using asocial media platform, it is agreed not to share collecteddata with third parties.

In this paper, we present a framework called FederatedTest-Time Adaptive Face Presentation Attack Detection withDual-Phase Privacy Preservation based on the principles ofFederated Learning (FL) and test-time adaptation targetingfPAD. Federate learning is a distributed and privacy preserv-ing machine learning technique [15], [9], [26], [18], [16]. FLtraining paradigm defines two types of roles named serverand client. Clients contain training data and the capacity totrain a model. As shown in Fig. 1 (middle), each client trainsits own model locally and uploads them to the server at theend of each training iteration. Server aggregates local updatesand produces a global model. This global model is thenshared with all clients which will be used in their subsequenttraining iteration. This process is continued until the globalmodel is converged. During the training process, data of eachclient is kept private. Collaborative FL training allows theglobal model to exploit rich local clients information whilepreserving data privacy.

In the context of FL for fPAD, both data centers and userscan be identified as clients. However, roles of data centersand users are different from conventional clients found inFL. In FL, all clients train models and carry out inferencelocally. In contrast, in FL for fPAD, only data centers carryout local model training. Data centers share their modelswith the server and download the global model during thetraining phase. On the other hand, users download the globalmodel at the end of the training procedure and carry outinference in the testing phase as shown in Figure 1 (bottom).

Although domain gaps between training and testing datacan be alleviated by the above federated learning processcarried out in the training phase, fPAD models are still hardto generalize well to unseen attacks with significant domainvariations in the real-world deployment. To further improvethe generalization ability of fPAD model, we incorporatetest-time adaptation into our framework by exploiting hintsprovided by the testing data. Specifically, such hints canbe unveiled from the entropy of model predictions on thetesting data and generalization error of fPAD models can bereduced via minimization of entropy of model predictions.As illustrated in Figure 1 (bottom), after downloading thetrained fPAD models from the training phase, users furtheradapt the fPAD model in the testing phase with the helpof entropy minimization of model predictions before thefinal classification. Note that test-time adaptation is sensitiveto the initial parameters of a pre-trained model. Federatedlearning can produce a well pre-trained model with privacypreservation which provides a good starting point for thefollowing test-time adaptation. Two phases are compatible

with each other and together improve the generalizationability of fPAD without accessing private training data frommultiple data centers.

Data Center #1

Server

...

Model

Data

Model

Data

Data Center #2

Model

Data

Data Center #3

Model

Data

Data Center #K

...

User

Model Upload

Model Download

Classificatoin

Model

Model

Training Phase

Model Updating

Testing Phase

Testing data

TestingTime

Adaptation

Fig. 2. Overview of the proposed framework. In the training phase,through several rounds of communications between data centers and server,the collaborated trained global fPAD model parameterized by Wt can beobtained in a data privacy preserving way. In the testing phase, usersdownload this global model from the server to their device and furthercarry out testing time adaptation given the testing data at hand. The finalclassification for the testing data can be carried out based on the updatedfPAD model.

II. RELATED WORK

A. Face Presentation Attack Detection

Current fPAD methods can be categorized under single-domain and multi-domain approaches. Single-domain ap-proaches focus on extracting discriminative cues betweenreal and spoof samples from a single dataset, which can befurther divided into appearance-based methods and temporal-based methods. Appearance-based methods focus on ex-tracting various discriminative appearance cues for detectingface presentation attacks. Multi-scale LBP [14] and colortextures [3] methods are two texture-based methods thatextract various LBP descriptors in various color spacesfor the differentiation between real/spoof samples. Imagedistortion analysis [28] aims to detect the surface distortionsas the discriminative cue.

On the other hand, temporal-based methods extract dif-ferent discriminative temporal cues through multiple framesbetween real and spoof samples. Various dynamic texturesare exploited in [17], [23], [22] to extract discriminativefacial motions. rPPG signals are exploited by Liu et al. [11],[10] to capture discriminative heartbeat information from realand spoof videos. [12] learns a CNN-RNN model to estimatedifferent face depth and rPPG signals between the real andspoof samples.

Various fPAD datasets are introduced recently that exploredifferent characteristics and scenarios of face presentationattacks. Multi-domain approach is proposed in order toimprove the generalization ability of the fPAD model tounseen attacks. Recent work [21] casts fPAD as a domaingeneralization problem and proposes a multi-adversarial dis-criminative deep domain generalization framework to searchgeneralized differentiation cues in a shared and discrimina-tive feature space among multiple fPAD datasets. [13] treats

fPAD as a zero-shot problem and proposes a Deep TreeNetwork to partition the spoof samples into multiple sub-groups of attacks. [24] addresses fPAD with a meta-learningframework and enables the model learn to generalize wellthrough simulated train-test splits among multiple datasets.These multi-domain approaches have access to data frommultiple datasets or multiple spoof sub-groups that enablethem to obtain generalized models. In this paper, we studythe scenario in which each data center contains data from asingle domain. Due to data privacy issues, we assume thatthey do not have access to data from other data centers. Thispaper aims to exploit multi-domain information in a privacypreserving manner.

B. Federated Learning

Federated learning is a decentralized machine learningapproach that enables multiple local clients to collabora-tively learn a global model with the help of a server whilepreserving data privacy of local clients. Federated averaging(FedAvg) [15], one of the fundamental frameworks for FL,learns a global model by averaging model parameters fromlocal clients. FedProx [18] and Agnostic Federated Learning(AFL) [16] are two variants of FedAvg which aim to addressthe bias issue of the learned global model towards differentclients. These two methods achieve better models by addingproximal term to the cost functions and optimizing a central-ized distribution mixed with client distributions, respectively.A recent work FedPAD [25] also exploits federated learningfor the task of fPAD. However, when encountering unseenattacks with large domain gap, the performance is stilldegraded. Comparatively, the proposed method integrates thetesting-time adaptation with federated learning, which furtherfacilities the generalization ability of a fPAD model duringtesting.

III. PROPOSED METHOD

A. Training Time Federated Learning

The proposed Test-Time Adaptive Face Presentation At-tack Detection framework is summarized in Fig. 2 andAlgorithm 1. Suppose that K data enters collect theirown fPAD datasets designed for different characteristics andscenarios of face presentation attacks. The correspondingcollected fPAD datasets are denoted as D1,D2, ...,DK withdata provided with image and label pairs denoted as x and y.y denotes the ground-truth with binary class labels (y= 0/1are the labels of spoof/real samples). Based on the collectedfPAD data, each data center can train its own fPAD model byiteratively minimizing the cross-entropy loss as follows:

L(Wk) =∑

(x,y)∼Dk

y logFk(x) + (1− y) log(1−Fk(x)),

where the fPAD model Fk of the k-th data enter is param-eterized by Wk (k = 1, 2, 3, ...,K). After optimization withseveral local epochs via

Wk ←Wk − η∇L(Wk),each data enter can obtain the trained fPAD model with theupdated model parameters.

Algorithm 1 Federated Test-Time Adaptive Face Presenta-tion Attack Detection with Dual-Phase Privacy PreservationRequire:

Input: K Data Centers have K fPAD datasets D1,D2, ...,DK ,Testing data presented to user UInitialization: K Data Centers have K fPAD modelsF1,F2, ...,FK parameterized by W1

0 ,W20 , ...,WK

0 . L is thenumber of local epochs. η is the learning rate. t is the federatedlearning roundsTraining Phase:Server aggregates:initialize W0

for each round t = 0, 1, 2,... dofor each data center k = 1, 2,..., K in parallel doW kt ← DataCenterUpdate(k,Wt)

end forWt =

1K

K∑k=1

Wkt

Download Wt to Data Centersend forTesting Phase:Users Download model Ft parameterized by Wt

H(Ft(x)) =∑x∼UFt(x) logFt(x)+(1−Ft(x)) log(1−Ft(x))

Wt(γ,β) ←Wt(γ,β) − η∇H(Ft(x))Users make final classification

DataCenterUpdate(k,W):for each local epoch i = 1, 2,..., L doL(Wk) =

∑(x,y)∼Dk

y logFk(x) + (1− y) log(1−Fk(x))

Wk ←Wk − η∇L(Wk)end forUpload W k to Server

It should be noted that dataset corresponding to eachdata center is from a specific input distribution and it onlycontains a finite set of known types of spoof attack data.When a model is trained using this data, it focuses on ad-dressing the characteristics and scenarios of face presentationattacks prevalent in the corresponding dataset. However, amodel trained from a specific data center will not generalizewell to unseen face presentation attacks. It is a well knownfact that diverse fPAD training data contributes to a bettergeneralized fPAD model. A straightforward solution is tocollect and combine all the data from K data centers denotedas D = {D1 ∪D2 ∪ ...∪DK} to train a fPAD model. It hasbeen shown that domain generalization and meta-learningbased fPAD methods can further improve the generalizationability with the above combined multi-domain data D [21],[24]. However, when sharing data between different datacenters are prohibited due to the privacy issue, this naivesolution is not practical.

To circumvent this limitation and enable various datacenters to collaboratively train a fPAD model, we proposethe Test-Time Adaptive Face Presentation Attack Detectionframework. Instead of accessing private fPAD data of eachdata center, the proposed framework introduces a serverto exploit the fPAD information of all data centers byaggregating the above model updates (W1,W2, ...,WK) ofall data centers. Inspired by the Federated Averaging [15]

algorithm, in the proposed framework, server carries out theaggregation of model updates via calculating the average ofupdated parameters (W1,W2, ...,WK) in all data centers as

W = 1K

K∑k=1

Wk.

After the aggregation, server produces a global fPADmodel parameterized by W that exploits the fPAD informa-tion of various data centers without accessing the privatefPAD data. We can further extend the above aggregationprocess into t rounds. Server distributes the aggregated modelW to every data center as the initial model parametersfor the next-round updating of local parameters. Thus, datacenters can obtain the t-th round updated parameters denotedas (W1

t ,W2t , ...,WK

t ). The t-th aggregation in the server

can be carried out as Wt = 1K

K∑k=1

Wkt . After t-rounds

of communication between data centers and the server, thetrained global fPAD model parameterized by Wt can beobtained without compromising the private data of individualdata centers. Once training is converged, users will downloadthe trained model from the server to their devices to carryout fPAD locally.

B. Testing-Time AdaptationAlthough federated learning in the training phase can

exploit various fPAD information available from multipledata centers to improve the generalization ability of a fPADmodel, the fPAD model can not generalize well to someunseen attacks when the domain gap between the training andtesting data is large, i.e., significant environmental variations.To address this issue, we further propose the test-timeadaptation. We argue that the data presented to the usersduring testing can provide some hints about their distribution,which can be used to adapt the fPAD model before makingthe final classification. Specifically, as demonstrated in [27],such hints can be unveiled by Shannon entropy [20] of modelpredictions which can indicate the distribution shift betweenseen training data and unseen testing data in an unsupervisedway. Therefore, the domain gap between training and testingdata can be further reduced by minimizing the Shannonentropy of the fPAD model predictions on the testing data. Asillustrated in Fig. 3, given the testing data presented to userU , and the fPAD model Ft downloaded from the trainingphase, we calculate the entropy of fPAD model predictionas follows:

H(Ft(x)) =∑x∼UFt(x) logFt(x)+(1−Ft(x)) log(1−Ft(x)).

To reduce the probability of overfitting during test-timeadaptation given limited testing data, we focus on mini-mizing the above entropy with respect to affine transforma-tion parameters (scale Wt(γ) and shift Wt(β)) of all batchnormalization layers in the fPAD model and keep all theother parameters fixed. This process can be carried out as:Wt(γ,β) ← Wt(γ,β) − η∇H(Ft(x)). After this test timeadaptation, we use the updated fPAD model for the finalreal/fake classification. Same as federated learning carriedout in the training phase, the whole process of the above

fPAD Model Prediction Entropy

Testing DataEntropy Minimization

Affine Parameters Updating

Fig. 3. Details of test-time adaptation during testing. Note that test-timeadaptation updates the parameters of a fPAD model using unlabeled testingdata presented to the users without accessing the training data.

test-time adaptation also does not need to get access to theprivate training data from multiple data centers, and thusthe generalization ability of fPAD model can be furtherameliorated in a privacy preserving manner during testing.Moreover, the above entropy calculation does not requireone to design a specific self-supervision task such as imagerotation for the supervision of adaptation, which significantlyfacilitates its compatibility to the task of fPAD.

On the other hand, we should note that test-time adaptationis sensitive to the quality of initial parameters of a pre-trainedmodel. Federated learning can produce a well pre-trainedmodel in a privacy preserving manner so that a suitablestarting point can be provided for the following optimizationof testing-time adaptation. This further demonstrates thecompatibility of dual-phase privacy preservation for the taskof fPAD.

IV. EXPERIMENTS

To evaluate the performance of the proposed framework,we carry out extensive experiments using five 2D fPADdatasets and two 3D mask fPAD datasets. In this section,we first describe the datasets and the testing protocol used inour experiments. Then we report various experimental resultsbased on multiple fPAD datasets. Discussions and analysisabout the results are carried out to provide various insightsabout FL for fPAD.

A. Experimental Settings

1) Datasets: We evaluate our method using the followingseven publicly available fPAD datasets which contain print,video replay and 3D mask attacks:1) Oulu-NPU [2] (O for short)2) CASIA-MFSD [30] (C for short)3) Idiap Replay-Attack [4] (I for short)4) MSU-MFSD [28] (M for short)5) SiW [12] (S for short)6) 3DMAD [5] (3 for short)7) HKBUMARsV2 [10] (H for short).Table I shows the variations in these seven datasets. Somesample images from these datasets are shown in Fig. 4.From Table I and Fig. 4 it can be seen that different fPADdatasets exploit different characteristics and scenarios offace presentation attacks (i.e. different attack types, displaymaterials and resolution, illumination, background and so

CASIA

Real

Spoof

Idiap MSU OULU SiW 3DMAD HKBUMARsV2Fig. 4. Sample images corresponding to real and attacked faces from CASIA-MFSD [30], Idiap Replay-Attack [4], MSU-MFSD [28], Oulu-NPU [2],SiW [12], 3DMAD [5], and HKBUMARsV2 [10] datasets.

TABLE ICOMPARISON OF SEVEN EXPERIMENTAL DATASETS.

Dataset Extralight

Complexbackground

Attacktype

Displaydevices

C No YesPrinted photo

Cut photoReplayed video

iPad

I Yes YesPrinted photoDisplay photo

Replayed video

iPhone 3GSiPad

M No Yes Printed photoReplayed video

iPad AiriPhone 5S

O Yes NoPrinted photoDisplay photo

Replayed video

Dell 1905FPMacbook Retina

S Yes YesPrinted photoDisplay photo

Replayed video

Dell 1905FPiPad ProiPhone 7

Galaxy S8Asus MB168B

3 No No Thatsmyface 3D mask Kinect

H Yes Yes Thatsmyface 3D maskREAL-f mask MV-U3B

on). Therefore, significant domain shifts exist among thesedatasets.

2) Protocol: The testing protocol used in the paper isdesigned to test the generalization ability of fPAD models.Therefore, in each test, performance of a trained model isevaluated against a dataset that it has not been observedduring training. In particular, we choose one dataset at a timeto emulate the role of users and consider all other datasets asdata centers. Real images and spoof images of data centersare used to train a fPAD model. The trained model is testedconsidering the dataset that emulates the role of users. Weevaluate the performance of the model by considering howwell the model is able to differentiate between real and spoofimages belonging to each user.

3) Evaluation Metrics: Half Total Error Rate (HTER) [1](half of the summation of false acceptance rate and falserejection rate), Equal Error Rates (EER) and Area UnderCurve (AUC) are used as evaluation metrics in our exper-iments, which are three most widely-used metrics for thecross-datasets/cross-domain evaluations. Following [13], inthe absence of a development set, thresholds required forcalculating evaluation metrics are determined based on thedata in all data centers.

4) Implementation Details: Our deep network is imple-mented on the platform of PyTorch. We adopt Resnet-18 [6]as the structure of fPAD models F i(i = 1, 2, 3, ...,K).In the training phase, Adam optimizer [8] is used for theoptimization of federated learning. The learning rate offederated learning is set as 1e-2. The batch size is 64 per

data center. Local optimization epoch L is set equal to 3.In the testing phase, Adam optimizer [8] is used for theoptimization of test time adaptation. The learning rate oftest time adaptation is set as 5e-3.

B. Experimental Results

In this section we demonstrate the practicality and gener-alization ability of the proposed framework in the real-worldscenario. We first compare the performance of the proposedframework with models trained with data from a single datacenter. As mentioned above, due to the limitation of dataprivacy that exists in the real-world, data cannot be sharedamong different data centers. In this case, users will directlyobtain a trained model from one of the data centers. Wereport the performance of this baseline in the Table II underthe label Single. For different choices of user datasets (fromO, C, I, M), we report the performance when the model istrained from the remaining datasets independently.

Rather than obtaining a trained model from a single datacenter, it is possible for users to obtain multiple trainedmodels from several data centers and fuse their predictionscores during inference, which is also privacy preserving.In this case, we fuse the prediction scores of the trainedmodel from various data centers by calculating the average.The results of this baseline are shown in Table II denoted asFused. Please note that it is impossible for users to carry outfeature fusion because a classifier cannot be trained based onthe fused features without accessing to any real/spoof dataduring inference in the users. According to Table II, fusingscores obtained from different data centers improves thefPAD performance on average. However, this would requirehigher inference time and computation complexity (of orderthree for the case considered in this experiment).

FedPAD [25] exploits the federated learning on the taskof fPAD. Table II illustrates that the FedPAD improvesthe performance on most of settings. However, when en-countering large domain gap settings such as O&C&Mto I, FedPAD still cannot achieve very promising perfor-mance. Comparatively, after integrating test-time adaptationwith federated learning, on average the proposed framework(Ours) is able to significantly improve the performance in allsettings and outperform other baselines. Moreover, we plotthe ROC curves in Fig 5, which also shows the effectivenessof the proposed framework. These results demonstrate thatthe proposed method is more effective in facilitating thegeneralization ability of fPAD with the dual-phase privacypreservation.

TABLE IICOMPARISON WITH MODELS TRAINED BY DATA FROM SINGLE DATA CENTER AND VARIOUS DATA CENTERS.

Methods Data Centers User HTER (%) EER (%) AUC (%) Avg. HTER Avg. EER Avg. AUC

Single

O M 41.29 37.42 67.93

41.61 36.66 67.07

C M 27.09 24.69 82.91I M 49.05 20.04 85.89O C 31.33 34.73 73.19M C 39.80 40.67 66.58I C 49.25 47.11 55.41O I 42.21 43.05 54.16C I 45.99 48.55 51.24M I 48.50 33.70 66.29M O 29.80 24.12 84.86C O 33.97 21.24 84.33I O 46.95 35.16 71.58

Fused

O&C&I M 34.42 23.26 81.67

35.75 31.29 73.89O&M&I C 38.32 38.31 67.93O&C&M I 42.21 41.36 59.72I&C&M O 28.04 22.24 86.24

FedPAD

O&C&I M 19.45 17.43 90.24

32.17 28.84 76.51O&M&I C 42.27 36.95 70.49O&C&M I 32.53 26.54 73.58I&C&M O 34.44 34.45 71.74

All

O&C&I M 21.80 17.18 90.96

27.26 25.09 80.42O&M&I C 29.46 31.54 76.29O&C&M I 30.57 25.71 72.21I&C&M O 27.22 25.91 82.21

Ours

O&C&I M 14.70 16.64 90.57

23.18 23.88 83.40O&M&I C 26.33 29.75 77.77O&C&M I 28.61 26.04 82.07I&C&M O 23.09 23.09 83.21

TABLE IIICOMPARISON WITH MODELS TRAINED BY DATA FROM SINGLE DATA CENTER AND VARIOUS DATA CENTERS.

Methods Data Centers User HTER (%) EER (%) AUC (%) Avg. HTER Avg. EER Avg. AUC

Single+Test-Time-Adaptation

O M 28.81 31.35 74.77

35.09 36.43 68.00

C M 34.49 35.64 69.46I M 12.11 16.53 90.23O C 30.37 30.35 74.69M C 41.20 42.10 60.69I C 43.53 42.91 59.21O I 47.88 46.76 56.71C I 60.02 65.26 36.34M I 17.40 17.04 89.65M O 23.24 23.30 83.65C O 31.63 31.08 74.34I O 36.94 37.16 66.60

Ours

O&C&I M 14.70 16.64 90.57

23.18 23.88 83.40O&M&I C 26.33 29.75 77.77O&C&M I 28.61 26.04 82.07I&C&M O 23.09 23.09 83.21

0 0.2 0.4 0.6 0.8 1

APCER

0

0.2

0.4

0.6

0.8

1

BP

CE

R

O&C&I to M

Single (O to M)Single (I to M)Single (C to M)FusedAllFedPADOurs

0 0.2 0.4 0.6 0.8 1

APCER

0

0.2

0.4

0.6

0.8

1

BP

CE

R

O&M&I to C

Single (O to M)Single (I to M)Single (C to M)FusedAllFedPADOurs

0 0.2 0.4 0.6 0.8 1

APCER

0

0.2

0.4

0.6

0.8

1

BP

CE

R

O&C&M to I

Single (O to M)Single (I to M)Single (C to M)FusedAllFedPADOurs

0 0.2 0.4 0.6 0.8 1

APCER

0

0.2

0.4

0.6

0.8

1

BP

CE

R

I&C&M to O

Single (O to M)Single (I to M)Single (C to M)FusedAllFedPADOurs

Fig. 5. ROC curves of models trained by data from single data center and various data centers.

Moreover, we further consider the case where a model istrained with data from all available data centers, which isdenoted as All in Table II. Note that this baseline violates

the assumption of preserving data privacy, and therefore isnot a valid comparison for FL for fPAD. Nevertheless, itindicates the upper bound of performance for the federated

TABLE IVCOMPARISON OF DIFFERENT NUMBER OF DATA CENTERS

Methods Data Centers User HTER EER AUC

FedPADO&I

C

49.22 49.10 51.19O&M&I 42.27 36.95 70.49

O&M&I&S 41.74 29.90 78.47

OursO&I 43.09 42.57 63.21

O&M&I 26.33 29.75 77.77O&M&I&S 22.90 22.39 85.89

TABLE VCOMPARISON OF DIFFERENT NUMBER OF DATA CENTERS.

Methods Data Centers User HTER EER AUC

FedPADC&M

S

25.11 20.64 88.08I&C&M 29.61 14.61 93.30

I&C&M&O 12.45 8.98 97.18

OursC&M 15.45 14.06 92.30

I&C&M 16.45 15.34 92.03I&C&M&O 10.62 9.63 96.35

learning applied in fPAD. From Table II, it can be seenthat the proposed framework is able to perform better thanthis baseline. This shows the proposed framework is able toobtain a privacy persevering fPAD model without sacrificingfPAD performance.

1) Compatibility of Dual Phases: To improve the gener-alization ability of fPAD models, users can also downloadmodels trained with data from a single data center and carryout test-time adaptation. Therefore, a natural choice is tointegrate test-time adaptation with models trained with datafrom a single data center (Single). We tabulate the compar-ison between this baseline (Single+Test Time Adaptation)and the proposed framework in Table III. Table III showsthat the proposed framework integrating test-time adaptationwith federated learning performs better than Single+TestTime Adaptation. fPAD models trained with data from asingle data center (Single) will easily overfit to data ofcorresponding local data center and thus generate a poorfPAD model. Comparatively, by exploiting various fPAD dataavailable from multiple data centers, federated learning canproduce a more suitable fPAD model during the trainingphase with privacy preservation. Since test-time adaptationis very sensitive to the quality of pre-trained model, fPADmodel trained by the proposed method is equipped withbetter initial parameters as a better starting point for thefollowing test-time adaptation. In this way, the correspondingtest-time adaptation can achieve improved performance asshown in Table III. This demonstrates the proposed frame-work is more able to exploit the compatibility between dualphases and thus achieve better fAPD performance.

2) Comparison of different number of data centers:In this section, we investigate the importance of havingmore data centers during training. Different data centersexploit different characteristics of face presentation attacks.Therefore, we expect aggregating information from moredata centers in the proposed framework to produce morerobust models with better generalization. In order to verifythis point, we increase the number of data centers in the

0 0.2 0.4 0.6 0.8 1

False Living Rate

0

0.2

0.4

0.6

0.8

1

Fal

se F

ake

Rat

e

Different types of spoof attacks

Single (I (Print) to M (Print, Video))Single (O (Video) to M (Print, Video))FusedFedPADOurs

Fig. 6. ROC curves of models trained by different types of 2D spoofattacks.

proposed framework and FedPAD method. The comparisonis reported in Tables IV and V. The experiments are carriedout using five datasets (O, M, I, C, S). In Table IV, weselect the dataset C as the data presented to the user and theremaining datasets as the data centers for training the fPADmodel with our framework, where we increase the number ofdata centers from 2 to 4. Another experiment is carried outwith a different combination of the same five datasets and theresults are shown in Table V. From Tables IV and V, it canbe seen that most values of evaluation metrics improve alongwhen the number of data centers increases, and the proposedmethod achieves the best performance in the first settingand comparable results in the second setting compared toFedPAD. This demonstrates that increasing the number ofdata centers in the proposed framework can improve theperformance.

3) Generalization ability to various 2D spoof attacks:In reality, due to limited resources, one data center mayonly be able to collect limited types of 2D attacks. However,various 2D attacks may appear to the users. This sectionsupposes that one data center collects one particular typeof 2D attack such as print attack or video-replay attack.As illustrated in Table VI, first, we select real faces andprint attacks from dataset I and real faces and video-replayattacks from dataset O to train a fPAD model respectively andevaluate them on dataset M (containing both print attacks andvideo-replay attacks). In both considered cases as shown inTable VI, the corresponding trained models cannot generalizewell to dataset M which contains the additional types of2D attacks compared to dataset I and O, respectively. Thistendency can be alleviated when the prediction scores oftwo independently trained models on both types of attacksare fused as shown in Table VI. FedPAD method obtains aperformance gain compared to score fusion. Comparatively,the proposed method further improves the performance,especially with a gain of 12.31% in HTER and 7.47% in EERcompared to FedPAD. We also plot the corresponding ROCcurve for this comparison in Fig 6, which also demonstratethe superior performance of the proposed method. Thisdemonstrate that test-time adaptation can effectively improvethe generalization ability to various 2D attacks when FLmodel is trained with limited types of fPAD data.

TABLE VIEFFECT OF USING DIFFERENT TYPES OF SPOOF ATTACKS

Methods Data Centers User HTER (%) EER (%) AUC (%)

Single I (Print) M (Print, Video) 38.82 33.63 72.46O (Video) M (Print, Video) 35.76 28.55 78.86

Fused I (Print) & O (video) M (Print, Video) 35.22 25.56 81.54FedPAD I (Print) & O (video) M (Print, Video) 30.51 26.10 84.82

Ours I (Print) & O (video) M (Print, Video) 18.20 18.63 87.57

TABLE VIIIMPACT OF ADDING DATA CENTERS WITH DIVERSE ATTACKS

Methods Data Centers User HTER AUC

FedPAD O&C&M (2D)3 (3D)

27.21 76.05O&C&M (2D) &H (3D) 34.70 92.35

Ours O&C&M (2D) &H (3D) 16.97 90.25

4) Generalization ability to 3D mask attacks : In this sec-tion, we investigate the generalization ability of the proposedframework to 3D mask attacks and the comparison is alsoconducted between the proposed framework and FedPADmethod. First, in FedPAD, a fPAD model is trained with datacenters exploiting 2D attacks (data from datasets O, C, I andM). This model is tested with 3D mask attacks (data fromdataset H). Then, we include one more data center containing3D mask attacks (dataset 3) into FedPAD and retrain it.Table VII shows that introducing diversity of data centers(by including a 3D mask attack) can improve performance inall evaluation metrics. This demonstrates that increasing datacenters with 3D mask attacks within the federated learningframework can improve the generalization ability of fPADmodel to the novel 3D mask attacks. We carry out the sameexperiment based on the proposed framework. In Table VII,it can be seen that the proposed method can significantlyimprove HTER by 17.73% compared to FedPAD. Thismeans that after adapted with novel 3D mask attack databy test-time adaptation during testing, fPAD model trainedwith federated learning in the training phase is more able togeneralize well to the novel types of 3D mask attacks, whichforms a better generalized fPAD framework.

V. CONCLUSION

In this paper, we presented a framework based on theprinciples of FL and test-time adaptation, targeting the appli-cation of fPAD with the objective of obtaining generalizedfPAD models while preserving data privacy in both trainingand testing phases. In the training phase, through commu-nications between data centers and the server, a globalfPAD model is obtained by iteratively aggregating the modelupdates from various data centers. In the testing phase,test-time adaptation is further exploited to minimize theprediction entropy of trained fPAD so that the generalizationerror on the unseen face presentation attacks can be furtherreduced. Local private data in the data centers is not accessedduring the whole process. Extensive experiments are carriedout to demonstrate the effectiveness of the proposed frame-work. In the future, we will further investigate the gener-alization improvement among different imbalanced datasets,

like FedMix [29] and Fed-Focal Loss [19]. Moreover, otherapproaches boosting the generalization ability, such as dataaugmentation [7] and robust loss functions [31], will also bestudied in the future work.

REFERENCES

[1] S. Bengio and J. Mariethoz. A statistical significance test for personauthentication. In The Speaker and Language Recognition Workshop,2004.

[2] Z. Boulkenafet and et al. Oulu-npu: A mobile face presentation attackdatabase with real-world variations. In FG, 2017.

[3] Z. Boulkenafet, J. Komulainen, and A. Hadid. Face spoofing detectionusing colour texture analysis. In IEEE Trans. Inf. Forens. Security,11(8): 1818-1830, 2016.

[4] I. Chingovska, A. Anjos, and S. Marcel. On the effectiveness of localbinary patterns in face anti-spoofing. In BIOSIG, 2012.

[5] N. Erdogmus and S. Marcel. Spoofing face recognition with 3D masks.2014. TIFS.

[6] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for imagerecognition. In CVPR, 2016.

[7] D. Hendrycks, N. Mu, E. D. Cubuk, B. Zoph, J. Gilmer, and B. Laksh-minarayanan. Augmix: A simple data processing method to improverobustness and uncertainty. arXiv preprint arXiv:1912.02781, 2019.

[8] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization.In arXiv preprint arXiv:1412.6980, 2014.

[9] T. Li, A. K. Sahu, A. Talwalkar, and V. Smith. Federated learn-ing: Challenges, methods, and future directions. In arXiv preprintarXiv:1908.07873, 2019.

[10] S. Liu, X. Lan, and P. C. Yuen. Remote photoplethysmographycorrespondence feature for 3D mask face presentation attack detection.In ECCV, 2018.

[11] S. Liu, P. C. Yuen, S. Zhang, and G. Zhao. 3D mask face anti-spoofingwith remote photoplethysmography. In ECCV, 2016.

[12] Y. Liu, A. Jourabloo, and X. Liu. Learning deep models for faceanti-spoofing: Binary or auxiliary supervision. In CVPR, 2018.

[13] Y. Liu, J. Stehouwer, A. Jourabloo, and X. Liu. Deep tree learningfor zero-shot face anti-spoofing. In CVPR, 2019.

[14] J. Maatta, A. Hadid, and M. Pietikainen. Face spoofing detection fromsingle images using micro-texture analysis. In IJCB, 2011.

[15] H. B. McMahan, E. Moore, D. Ramage, S. Hampson, et al.Communication-efficient learning of deep networks from decentralizeddata. In AISTATS, 2016.

[16] M. Mohri, G. Sivek, and A. T. Suresh. Agnostic federated learning.2019.

[17] T. F. Pereira and et al. Face liveness detection using dynamic texture.In EURASIP Journal on Image and Video Processing, (1): 1-15, 2014.

[18] A. K. Sahu, T. Li, M. Sanjabi, M. Zaheer, A. Talwalkar, and V. Smith.On the convergence of federated optimization in heterogeneous net-works. In arXiv preprint arXiv:1812.06127, 2018.

[19] D. Sarkar, A. Narang, and S. Rai. Fed-focal loss for imbalanced dataclassification in federated learning. arXiv preprint arXiv:2011.06283,2020.

[20] C. E. Shannon. A mathematical theory of communication. The Bellsystem technical journal, 27(3):379–423, 1948.

[21] R. Shao, X. Lan, J. Li, and P. C. Yuen. Multi-adversarial discriminativedeep domain generalization for face presentation attack detection. InCVPR, 2019.

[22] R. Shao, X. Lan, and P. C. Yuen. Deep convolutional dynamic texturelearning with adaptive channel-discriminability for 3D mask face anti-spoofing. In IJCB, 2017.

[23] R. Shao, X. Lan, and P. C. Yuen. Joint discriminative learning of deepdynamic textures for 3D mask face anti-spoofing. In IEEE Trans. Inf.Forens. Security, 14(4): 923-938, 2019.

[24] R. Shao, X. Lan, and P. C. Yuen. Regularized fine-grained meta faceanti-spoofing. In AAAI, 2020.

[25] R. Shao, P. Perera, P. C. Yuen, and V. M. Patel. Federated facepresentation attack detection. arXiv preprint arXiv:2005.14638, 2020.

[26] V. Smith, C.-K. Chiang, M. Sanjabi, and A. S. Talwalkar. Federatedmulti-task learning. In NIPS, 2017.

[27] D. Wang, E. Shelhamer, S. Liu, B. Olshausen, T. Darrell, U. Berkeley,and A. Research. tent: fully test-time adaptation by entropy minimiza-tion. In International Conference on Learning Representations, 2021.

[28] D. Wen, H. Han, and A. K. Jain. Face spoof detection with imagedistortion analysis. In IEEE Trans. Inf. Forens. Security, 10(4): 746-761, 2015.

[29] T. Yoon, S. Shin, S. J. Hwang, and E. Yang. Fedmix: Approximationof mixup under mean augmented federated learning. arXiv preprintarXiv:2107.00233, 2021.

[30] Z. Zhang and et al. A face antispoofing database with diverse attacks.In ICB, 2012.

[31] Z. Zhang and M. R. Sabuncu. Generalized cross entropy loss fortraining deep neural networks with noisy labels. In 32nd Conferenceon Neural Information Processing Systems (NeurIPS), 2018.