from lukas-kanade to mnemonic descent and deep dense shape …cmp.felk.cvut.cz › cmp › events...
TRANSCRIPT
StefanosZafeiriouThursday19,January2017
FromLukas-KanadetoMnemonicDescentandDeepDenseShapeRegression:Abriefhistoryof(deformable)imagealignment
Image/Objectalignment
2
Image/Object alignment (or registration) is the process oftransformingdifferentsetsofdataintoonecoordinatesystem
•Globalalignment
• Alignment using object parts (i.e., semantically meaningfulcomponents)
DeformableObjectAlignment
3
DeformableModelsaimtomodeltheshapeandappearancevariationsofanobjectclass.
WhatareDeformableModels?
Trainedusing“In-the-Wild”data(i.e.imagescapturedundertotallyunconstrainedconditions)
4
DeformableModelsaimtomodeltheshapeandappearancevariationsofanobjectclass.
WhatareDeformableModels?
Trainedusing“In-the-Wild”data(i.e.imagescapturedundertotallyunconstrainedconditions)
4
6DeepfacebyFacebook,inCVPR2014
RecentsuccessesinComputerVisionFacerecognition/verification~97%inLFW
•Shouldthissuccessreallyattributedindeeplearning?•Thenetworkwithoutalignmentproduces:87.9%
Trainedon4.4millionlabeledfacialimages
•Averysimpleclassifieronaligneddata:~92%
•
•InthesameyearKitttler’sgroupreported:~96%(MRFalignmentandsimplefeatures)
7
Facebook’sDeepface
The success is mainly due to an elaborate image alignment and imagenormalization(frontalization)procedure
Google’sFaceNet
• Betterperformancewithouttheneedtoapplysuchelaboratealignment(stillalignmentimprovesperformance)
• Butitistrainedontherecordnumberof260,000,000images!!!!
8
Whyaretheseimagesdisturbing?
11/12/169
HolisticObjectalignment
10
Max(minim)ize
W is the motion model, p are the motion parameters
Lukas-Kanade(LK)inanutshell
11
• Findmotionparametersbysolving
• Exampleofwarpsandparameters(n-params):
Affine(6-parameters):
Translation(2-parameters):
• xisthetestimageandmthetemplate(N-pixels)
[1]B.Lucas,andT.Kanade."Aniterativeimageregistrationtechniquewithanapplicationtostereovision."IJCAI.Vol.81.No.1.1981.
LK:Forward-Additive
12
Jacobian: Derivativeofthewarp:
Errorimage:
Step1:Linearisearoundthecurrentestimate
Step2:Solve
Step3:Update
13
JacobianandHessiancanbepre-computed
LK:InverseCompositional(IC)Step1:Linearisearoundzero(parametersinthetemplate)
Step2:Update
Step3:Update
[1]Baker,S.andMatthews,I.."Lucas-kanade20yearson:Aunifyingframework."Internationaljournalofcomputervision56.3(2004):221-255.
LinearComplexity!!!!
FromLKtostatisticaldeformablemodels
14
ActiveAppearanceModels(AAMs)
Timeline
1998
2004
2008
2009
2013
2014
2015
2016
[Cootesetal.,1998]AAMswithsinglestepregressionfitting!
[Matthews&Baker,2004]Project-OutInverseCompositionalSimultaneousInverseCompositional[Papandreou&Maragos,2008]AlternatingInverseCompositional[Ambergetal.,2009][Tzimiropoulosetal.,2013]Project-OutForwardCompositional [Asthana,Zafeiriouetal.,2013]
[Xiong&DelaTorre,2013][Caoetal.,2014]Cascadedregressionfittingframework
[Alabort&Zafeiriou,2014,2016]Unifiedframeworkforcompositionalfitting[Antonakosetal.,2015]ActivePictorialStructures
[Trigeorgisetal.,2016]MnemonicDescentMethod
GenerativeModels DiscriminativeModels
[Antonakosetal.,2016]AdaptiveCascadedRegression 23
Timeline
1998
2004
2008
2009
2013
2014
2015
2016
[Cootesetal.,1998]AAMswithsinglestepregressionfitting!
[Matthews&Baker,2004]Project-OutInverseCompositionalSimultaneousInverseCompositional[Papandreou&Maragos,2008]AlternatingInverseCompositional[Ambergetal.,2009][Tzimiropoulosetal.,2013]Project-OutForwardCompositional [Asthana,Zafeiriouetal.,2013]
[Xiong&DelaTorre,2013][Caoetal.,2014]Cascadedregressionfittingframework
[Alabort&Zafeiriou,2014,2016]Unifiedframeworkforcompositionalfitting[Antonakosetal.,2015]ActivePictorialStructures,FeaturebasedAAMs [Trigeorgisetal.,2016]
MnemonicDescentMethod
GenerativeModels DiscriminativeModels
[Antonakosetal.,2016]AdaptiveCascadedRegression 24
17
MeanShapeShapesafterprocrustes
PCAonthatspace
FromLKtoAAMs
18
Warpusingmeanshape
PCAonthatspace
FromLKtoAAMs
[1]Cootes,T.,Edwards,G.andTaylor,C."Activeappearancemodels."IEEET-PAMI23.6(2001):681-685.
• Thewarpingfunctionisdrivenbytheshape.• Insteadofthetemplateimagewehavealineartexturemodel.
• Theparametersthatdrivethemodelare(a)theweightsofthelinearshape,(b)aglobalsimilaritytransformand(c)theweightsofthelineartexturemodel.
19
FromLKtoAAMs
Statisticalparametricmodeloftheshapeandappearanceofanobject.
FromLKtoAAMs
Recovershapeandappearanceparametersthatminimizethereconstructionerrorofthewarpedimage.
ShapePCA AppearancePCA
[1]I.MatthewsandS.Baker.“ActiveAppearanceModelsRevisited”,IJCV,60(2):135-164,2004. 28
21
• Notrobusttooutliers(occlusions,illumination,castshadowsetc.)• Replacetheleastsquaresfunctionwithrobusterrorfunctions•Noisemodelforoutliershardtodefine-Aretheyalwaysrobust?•Approximationsforefficiency
RobustLK-AAMs
[1]Baker,S.andMatthews,I.."Lucas-kanade20yearson:Aunifyingframework."Internationaljournalofcomputervision56.3(2004):221-255.[2]Dowson,N.,andR.Bowden."Mutualinformationforlucas-kanadetracking(milk):Aninversecompositionalformulation."IEEET-PAMI30.1(2008):180-185.[3]Evangelidis,G.,&Psarakis,E.(2008).Parametricimagealignmentusingenhancedcorrelationcoefficientmaximization.IEEET-PAMI,30(10),1858-1865.[4]Lucey,Simon,etal."Fourierlucas-kanadealgorithm."IEEET-PAMI35.6(2013):1383-1396.
GivenanimageofsizeHxW,thedensefeatureextractionfunctionisdefinedas:
whereDisthenumberofchannels.
Feature-BasedLK-AAMs
[1]Antonakos,Alabort-i-Medina,TzimiropoulosandZafeiriou,“Feature-BasedLucas-KanadeandActiveAppearanceModels”,IEEETIP,2015[2]Tzimiropoulos,Georgios,StefanosZafeiriou,andMajaPantic."Robustandefficientparametricfacealignment.",ICCV,2011(oral)[3]Tzimiropoulos,G.,Argyriou,V.,Zafeiriou,S.,&Stathaki,T.(2010).RobustFFT-basedscale-invariantimageregistrationwithimagegradients.IEEET-PAMI,32(10),1899-1906.
19
HOG,SIFT,...
multi-channelDenseimagefeatures
Patch-basedwarping
patchesextraction multi-
channelpatches
Feature-BasedLK-AAMs
[1]Antonakos,Alabort-i-Medina,TzimiropoulosandZafeiriou,“Feature-BasedLucas-KanadeandActiveAppearanceModels”,IEEETIP,2015
27
Feature-BasedAAMs● Warpthefeaturesimages
● Extractfeaturesonthewarpedimage
20
AAM:Project-OutInverseCompositional
distancewithinA(0forallp)
distancetoA(i.e.distancewithinorthogonalcomplement)
linearize
with and
Project-out
Inverseandlinearization
[1]I.MatthewsandS.Baker.“ActiveAppearanceModelsRevisited”,IJCV,60(2):135-164,2004. 30
AAM:Project-OutInverseCompositional
[1]I.MatthewsandS.Baker.“ActiveAppearanceModelsRevisited”,IJCV,60(2):135-164,2004. 31
AAM:AlternatingInverseCompositional
withand
Inverseandlinearization
Project-Out
[1]G.PapandreouandP.Maragos.“Adaptiveandconstrainedalgorithmsforinversecompositionalactiveappearancemodelfitting”,CVPR,2008.
linearize
32
AAM:Project-OutInverseCompositional
[1]G.PapandreouandP.Maragos.“Adaptiveandconstrainedalgorithmsforinversecompositionalactiveappearancemodelfitting”,CVPR,2008. 33
FromLKtoAAMs:What’snext?
29
AAMs:LKalgorithmwithashapedrivenmodelandaspecialweightedleast-squarescostfunction(akindofMahalanobis)
WhatkindofMahalanobis(whytheaboveweights?)
Howtoachievereal-timeperformance?
ActivePictorialStructures
30[1]Antonakos,AlabortiMedina,Zafeiriou,ActivePictorialStructures,CVPR2015
PictorialStructuresQuickOverview
➜ Gaussiandistributionforthepatch-basedappearanceofeachlandmark
➜ Gaussiandistributionforeachpairwiserelativelocationbetween
landmarksbasedonatree(acyclicgraph).
➜ Costfunction
○ Mahalanobisfortheappearanceofeachlandmark○ Spring-likedeformationcostbetweenpairsoflandmarks
➜ Globaloptimumusinganefficientdynamicprogrammingalgorithm.[1]P.FelzenszwalbandD.Huttenlocher.“PictorialStructuresforobjectrecognition”,IJCV2005.[2]P.FelzenszwalbandD.Huttenlocher.“Distancetransformsofsampledfunctions”,2004.[3]P.Felzenszwalb,etal.“Objectdetectionwithdiscriminativelytrainedpart-basedmodels”,IEEET-PAMI2010.[4]X.ZhuandD.Ramanan.“Facedetection,landmarklocalizationinthewild”,CVPR2012.[5]Fischler,M.A.andElschlager,R.A.1973.Therepresentationand1376matchingofpictorialstructures.IEEETransactionsonComputer,137722(1):67–92.
ActivePictorialStructures
[1]E.Antonakos,J.Alabort-i-Medina,S.Zafeiriou.“ActivePictorialStructures”,CVPR2015.
ShapePCA
Blocksparseprecisionmatrix
AppearanceGMRF
Blocksparseprecisionmatrix
DeformationGMRF
ThecostfunctionconsistsoftheappearanceMahalanobisdistance,thedeformationpriorandaweighthyperparameterbetweenthem.
36
ActivePictorialStructures
32
ActivePictorialStructures
[1]E.AntonakosandS.Zafeiriou.“ActivePictorialStructures”,CVPR2015.
• WeightedInverseCompositionalGauss-NewtonwithfixedJacobianandHessian• Fastestinversecompositionalfittingwithfastestconvergencerate• Deformationpriormakesmodelveryrobust
pre-computed pre-computed
residual
38
ActivePictorialStructures
[1]E.Antonakos,J.Alabort-i-MedinaandS.Zafeiriou.“ActivePictorialStructures”,CVPR,2015. 39
ActivePictorialStructures
DiscriminativeAlignment
41
[1]Xiong,X.,andF.DelaTorre."Superviseddescentmethodanditsapplicationstofacealignment."CVPR,2013.[2]Asthana,A.,Zafeiriou,S.,Cheng,S.,&Pantic,M.Robustdiscriminativeresponsemapfittingwithconstrainedlocalmodels,CVPR,2013[3]Asthana,A.,Zafeiriou,S.,Cheng,S.,&Pantic,M.Incrementalfacealignmentinthewild.InCVPR2014[4]Asthana,A.,Zafeiriou,S.,Tzimiropoulos,G.,Cheng,S.,&Pantic,M.Frompixelstoresponsemaps:Discriminativeimagefilteringforfacealignmentinthewild.IEEET-PAMI,37(6),1312-1320.2015
LearningtheDescentDirections
Gauss-Newton
Newton
InsteadofcomputingtheJacobianfromtheimageateachiterationwhynotpre-computeit?
GeneralDescentDirection
42
38
DiscriminativeAlignment
Groundtruth
Perturbation
Solve:
Learnthemapping
39
Applyingthemapping
45
AdaptiveCascadedRegression
Costfunctionwithrespecttoparametricshapemodel.
Motivation
47
Costfunctionwithrespecttoparametricshapemodel.
Realisticinitializationfromfacedetector.
Motivation
48
Costfunctionwithrespecttoparametricshapemodel.
Realisticinitializationfromfacedetector.
Gauss-Newtonfailsduetobadinitialization.
Motivation
49
Costfunctionwithrespecttoparametricshapemodel.
Realisticinitializationfromfacedetector.
Gauss-Newtonfailsduetobadinitialization.
Cascadedregressionmovestowardsthecorrectdirectionbutnotcloseenough.
Motivation
50
Costfunctionwithrespecttoparametricshapemodel.
Realisticinitializationfromfacedetector.
Gauss-Newtonfailsduetobadinitialization.
Cascadedregressionmovestowardsthecorrectdirectionbutnotcloseenough.
ApplyingGauss-Newtondescentdirectionsrightaftercascadedregressiondirectionsreachesoptimum!
Motivation
51
AdaptiveCascadedRegression
DiscriminativeCascadedRegressiondescentdirections
GenerativeGauss-Newtondescentdirections
AdaptiveCascadedRegression
Linearcombinationofdescentdirectionswithadaptiveweights:
Adaptivedescentsteps
Projected-OutResidual
with
52
Matrixwithgroundtruthshapeparameters
Matrixwithparametersfromnoisyshapes
Matrixwithprojected-outfeaturevectors
[1]E.Antonakos*,P.Snape*,G.TrigeorgisandS.Zafeiriou.“AdaptiveCascadedRegression”,oral,ICIP2016.
AdaptiveCascadedRegression
AdaptiveCascadedRegression
[1]E.Antonakos*,P.Snape*,G.TrigeorgisandS.Zafeiriou.“AdaptiveCascadedRegression”,oral,ICIP2016.54
AdaptiveCascadedRegression
[1]E.Antonakos*,P.Snape*,G.TrigeorgisandS.Zafeiriou.“AdaptiveCascadedRegression”,oral,ICIP2016.
ConsistentlylowererrorcomparedtocascadedregressiondiscriminativeandtoGauss-Newtongenerativemodel.
55
AdaptiveCascadedRegression
[ACR]E.Antonakos*,P.Snape*,G.TrigeorgisandS.Zafeiriou.“AdaptiveCascadedRegression”,oral,ICIP2016.[300W1]E.Zhouetal.“Extensivefaciallandmarklocalizationwithcoarse-to-fineconvolutionalnetworkcascade”,ICCV-W2013.[CFSS]S.Zhuetal.“Facealignmentbycoarse-to-fineshapesearching”,CVPR2015.[PO-CR]G.Tzimiropoulos.“Project-Outcascadedregressionwithanapplicationtofacealignment”,CVPR2015.[300W2]J.Yanetal.“Learntocombinemultiplehypothesesforaccuratefacealignment”,ICCV-W2013.[ERT]V.KazemiandJ.Sullivan.“Onemillisecondfacealignmentwithanensembleofregressiontrees”,CVPR2014.[Intraface]X.XiongandF.DelaTorre.“SupervisedDescentMethodanditsApplicationstoFaceAlignment”,CVPR2013.[Chehra]A.Asthana,S.Zafeiriou,S.Cheng,andM.Pantic.“Incrementalfacealignmentinthewild”,CVPR2014. 56
AdaptiveCascadedRegression
52
MnemonicDescent
End-to-EndTrainingofannon-linearcascaderegressionmethod
54
• Why HoGs and not any other features? (Learn a non-lineartransformforfeatures)
• Theprocessbearssimilaritieswithadynamical system.Whynotmodel itexplicitlymodel? (Learnanon-lineardynamicalsystemthatmodelsthecascade).
MnemonicDescentMethod
55
Trigeorgis,G.,Snape,P.,Nicolaou,M.A.,Antonakos,E.,&Zafeiriou,S.(2016,June).MnemonicDescentMethod:Arecurrentprocessappliedforend-to-endfacealignment.CVPR’16,LasVegas,NV,USA.
MnemonicDescentMethod
56
• Updates
57
MnemonicDescentMethod
Clusteringofthestate
58
MnemonicDescentMethod
3DMM
11/12/1659
3DMM
11/12/1660
3DMM“in-the-wild”
11/12/1661
Fittinga3DMM
11/12/1662
DeepLearning
11/12/1663
IntroducingMenpo• OpenSourcePythonpackageforgenerativeanddiscriminativemodelling
• UnifiedframeworkforAAMs,CLMs,&SDMs
• Underactivedevelopment,MorphableModelscoming
• www.menpo.org
• github.com/menpo76
ThankYou