siamese neural network based gait recognition for human ... · network based gait recognition for...

SiameseNeuralNetworkbasedGaitRecognitionforHumanIdentification

ChengZhang,WuLiu,Huadong Ma,Huiyuan FuBeijingUniversityofPostsandTelecommunications

ICASSP2016

Outline

• Introduction• Proposedmethod– ConventionalCNNbasedGaitRecognition– SiameseNetworkbasedGaitRecognition

• Experiments• Conclusions

Definition• Gait analysis is the systematic study of animal locomotion,

more specifically the study of human motion, using the eyeand the brain of observers, augmented by instrumentation formeasuring body movements, body mechanics, and theactivity of the muscles.

Background

• Socialsecurity– Videobigdataandcameranetwork– Remotesurveillance– Identificationandattributeclassification

• Biometricauthenticationtechniques– Facialrecognition– Irisrecognition– Fingerprinttechnologies– Voiceverification– Handgeometry

Characteristics

• Remoteaccessed– Itcanidentifysubjectsfromadistancewithoutinterruptingthesubject

• Robust– Eveninlowresolutionvideos,thegaitstillworkswell

• Security– Itisdifficulttoimitateorcamouflagehumangait

iris fingerprintface voice gait

Whygaitworks?

• A plethora of technique and data continue to showthat a person’s walking is indeed unique

MurrayMP,DroughtAB,KoryRC.Walkingpatternsofnormalmen[J].TheJournalofBone&JointSurgery,1964,46(2):335-360.JohanssonG.Visualperceptionofbiologicalmotionandamodelforitsanalysis[J].Attention,Perception,&Psychophysics,1973,14(2):201-211.

Challenges• Inconspicuousinter-classdifferencefromthedifferentpeople

• Thelargeintra-classvariationsfromthesameperson– Walkingspeeds– Viewpoints– Clothing– Belongings– Occasion normal clothes backpack

Gaitsilhouettesofdifferentsubject

RecentEffortsandMajorDrawback• Model-basedmethods

– Extractinghumanbodystructurefromtheimages– Requiringahighresolutionaswellashighercomputationalcostand

arenotyetsuitableforoutdoorsurveillance

• Model-freemethods– Usingthewholemotionpattern/featuresofthehumanbody,and

performingrecognitionatlowerresolutions– Human-craftedgaitfeaturescanextremelyhardtobreakthrough

featurerepresentationbottleneckwhenfacingwiththegaitandappearancechanges

GeneralStepsofOurSystem

GaitEnergyImage

• Averagingofsilhouetteoveronegaitcycle– Representahumanmotionsequenceinasingleimagewhilepreservingtemporalinformation

– Robusttoincidentalsilhouetteerrorsinindividualimage

ConventionalCNNbasedGR

• RetraintheCNNsonthegaitdataset– CNNsareabletolearndiscriminativefeatures– Fine-tuningfromapre-trainedmodel(e.g.,AlexNet)isagoodsolutiontosolvethedatalimitationproblemandspeeduptheconvergenceofnewmodel

– EmploytheAlexNet andonlychangethe1,000labeloutputtothenumberofsubjectsingaitdataset

Problems

• Datalimitation– Tolearnsufficientfeatures,theCNNrequiresamassoftrainingdataforallcategories

– Forgaitrecognition,thenumberofsubjectscanbelarge,whilewithonlyafewexamplespersubjectinpublicdatabase

• Domaingap– Gaitrecognitionforhumanidentificationisessentiallyasearchproblembutnotclassification

MetricLearning

＝ ≠

ProposedFramework

• SiameseNeuralNetworkbasedgaitrecognition

Sampling

• Trainingdataishighlyunbalanced– Usingasamplertogenerateequalnumberofpositiveandnegativeineachmini-batch,avoidoverlybiased towardstonegativedecisions

– Usingasamplertoenforcevarietytopreventoverfitting toalimitednegativeset

• Specially,thetrainingsetisselectedfromOULP-C1V1-A-Gallery dataset,with20,000similarGEIpairsandrandomlyselected20,000dissimilarpairs

LossFunction

• ThedistancebetweenapairofGEIscanbemeasuredby:

• Wecandefinethecontrastivefunctionasfollows:

TrainingandFeatureExtraction

• Supervisedsetting• MinimizedthecontrastivelossfunctionoveratrainingsetofNpatchpairsusingstochasticgradientdescent

• Experimentedwithdifferentparametersandgavethebestperformanceoffeaturerepresentation

Experiments• Database:OU-ISIRLargePopulation

• Evaluation:Rank-1andRank-5identificationrates

• Baselines:STOAgaitrecognitionmethods,i.e.,GEI,FDF,HWLD,VTM,andRankSVM

• Pipeline:Backgroundsegmentation->Periodicidentification->GEIsgeneration->DNNtraining->DNNfeatureextraction->K-Nearest-Neighborsearching

Database• OU-ISIRLargePopulationGaitDatabase

– Containstheworld’slargestnumberofsubjects– Recordstwosequencesforeachsubject:probe(query)andgallery

(source)sequence,offersfaircomparisontestbed

Intra-viewrecognition

[9]H.Iwama,M.Okumura,Y.Makihara,andY.Yagi,“Theou-isirgaitdatabasecomprisingthelargepopulationdatasetandperformanceevaluationofgaitrecognition,”IEEETIFS.[7]Sivapalan,D.Chen,S.Denman,S.Sridharan,andC.Fookes,“Histogramofweightedlocaldirectionsforgaitrecognition,”inCVPRW,2013.

SomeResults

Inter-viewrecognition

D.Muramatsu,A.Shiraishi,Y.Makihara,M.Uddin,andY.Yagi,“Gait-basedpersonrecognitionusingarbitraryviewtransformationmodel,”IEEETIP,2015.R.Mart´ın-F´elez andT.Xiang,“Gaitrecognitionbyranking,”inECCV,2012.

Conclusions• Wepresentoneofthefirstattemptstostudythedeepneural

networkbasedgaitrecognitionforhumanidentificationwithdistancemetriclearning

• Intheend-to-endframework,weleveragethecompetitiveGEIpresentationastheinputofnetworkwhileholisticallyexploittheSiameseneuralnetworktolearneffectivefeaturerepresentationsforhumanidentification

• Thecomprehensiveevaluationsshowthatweimpressivelyoutperformthestate-of-the-artsontheworld’slargestchallengegaitbenchmarkdataset

FutureWorks

• 3-DimensionalSiameseneuralnetwork• Quasi-periodicorsub-framegaitrecognition• Unconstrainedenvironment,likeilluminationchanges,darkillumination,clutteredbackground,motionblur,andimagecompressionnoise

• …

“High’st Queenofstate,GreatJunocomes;Iknowherbyhergait”—— TheTempest[Act4Scene1],Shakespeare

Anyquestions?