3D Models for Face Image and Video Processing Gbor SzirtesELTE Dept of Information SystemsNeural Information Processing GroupLrincz-lab

SSIP 2002 Budapest, Hungary

ContentShort IntroductionMotivationsPile of concepts (framework?)Future applicationsOngoing projects and immature ideasWhat next?

SSIP 2002 Budapest, Hungary

A few words about our groupSince 1999Within the Information Systems Department, ELTE5 PhD students and ~20 grad studentsMainly biologically motivated projectsRL, ICA, machine learning, facial expressions, dynamical systems, image processing

SSIP 2002 Budapest, Hungary

MotivationsBeing in quest of the Holy Grail: intelligenceOne working example: our brainEvolutionary concepts, need for adaptationPerception and Action

SSIP 2002 Budapest, Hungary

Conceptual framework I.SYSTEMENVIRONMENTNoisy, stochastic, evolvingPerceptionAction?

SSIP 2002 Budapest, Hungary

Conceptual framework II.Central hypothesis

Internal representation (encoded signals from the environment and the systems state)Reconstruction

SSIP 2002 Budapest, Hungary

Perception Active Not simply feed-forward Feed-back modulated and controlled Modular Component based Adaptive, plastic

SSIP 2002 Budapest, Hungary

Perception II.Active: it is not a passive signal detection process. We need to `foresee` and anticipate the expected changes (prediction).Influenced by higher order modulation (e.g. FOA, focus of attention, conscious and unconscious perception)

SSIP 2002 Budapest, Hungary

Perception III.Several stages of processingNot purely hierarchical (feed-back)Distributed, parallel ways, strong interplayModularity

SSIP 2002 Budapest, Hungary

Perception IV.Components: meaningful (?) building blocks

SSIP 2002 Budapest, Hungary

Perception V.This is what we have seen before?

SSIP 2002 Budapest, Hungary

PartsDrawings of 4 year old healthy children

SSIP 2002 Budapest, Hungary

and the wholeDrawing of a 3 and a half year old child with autism

SSIP 2002 Budapest, Hungary

Beyond the theoryRecognition of faces and facial expressionsTwofold goals: Understand perception Help develop applications forHuman-Computer InteractionPsychiatric analysis and treatment

SSIP 2002 Budapest, Hungary

DatabaseIn collaboration with the Psychiatric Clinic of SOTE (Simon-lab)

SSIP 2002 Budapest, Hungary

A few examples of segmented imagesHappinessDisgust

SSIP 2002 Budapest, Hungary

The architectureM1M2M3M*M**RL containerFACESACTION?

SSIP 2002 Budapest, Hungary

Modules for recognition of facesFinding heads: Skin detectionTracking: particle filteringSegmentation3D model based transformationIdentification, recognition or analysis(back-transformation)

SSIP 2002 Budapest, Hungary

Module 1Face location (fitting)Many heuristics are possibleOne particular choiceskin-detector

SSIP 2002 Budapest, Hungary

Skin detectorrgbSkin color cluster learned by MLP

SSIP 2002 Budapest, Hungary

Module 2+Particle filteringCONDENSATION (Conditional Density Propagation )(Isard and Blake, 1998)

Segmentation Tracking

SSIP 2002 Budapest, Hungary

SegmentationImage basedFeature basedproceduresTwo approaches: approximating contours with splines or snakes (too many degrees of freedom) Template basedA simple template

SSIP 2002 Budapest, Hungary

Segmentation II.More sophisticated manually tuned templateArbitrary spine directions(with positive-negative weights)

SSIP 2002 Budapest, Hungary

Segmentation III.Many concurrent candidates

SSIP 2002 Budapest, Hungary

Segmentation IV.Head-shoulder template for better fitting

SSIP 2002 Budapest, Hungary

Particle filtering in action!Initialization made by hand

SSIP 2002 Budapest, Hungary

Well, there is no perfect methodSometimes even the best choice is far from the face to be tracked

SSIP 2002 Budapest, Hungary

Tracking of fast motion against a cluttered backgroundFrom http://www.robots.ox.ac.uk/~misard/condensation.html

SSIP 2002 Budapest, Hungary

CONDENSATIONKeywords:general,multi-modaldensities,sampling,Discrete-continuousMarkovian

SSIP 2002 Budapest, Hungary

Module 3 (off the stream)Facial expression (display) recognition

SSIP 2002 Budapest, Hungary

Facial expression recognitionHMM on segmented image sequencessurpriseReconstruction errorHMM winner: surpriseHMM emission

SSIP 2002 Budapest, Hungary

Module 43D face modelExtension of the CANDIDE (Rydfalk,1987) modelCompatible with FACS(Ekman and Friesen, 1977)Candide 3 (developed for MPEG4 standard)

SSIP 2002 Budapest, Hungary

How to use the model?Target (synthetic) faceSearching

SSIP 2002 Budapest, Hungary

Such a big space!Reconstruction error based optimization problemToo many local minimaGlobal optimum finding procedure: STAGE (Boyan, 1998)

SSIP 2002 Budapest, Hungary

STAGE Algorithm for finding the global optimum Function approximator learns an evaluation function that predicts the outcome of a local search Experience: it is able to explore the global structureLet us find the minimum of F(x)=(|x|-10)cos(2x)

SSIP 2002 Budapest, Hungary

STAGE IIIt can be combined with any local search method (hillclimbing,WALKSAT,)It works on both the objective and the evaluation function at two stagesSmart restart by a better predictionReal-valued (compared to GA)Easy to implement

SSIP 2002 Budapest, Hungary

SSIP 2002 Budapest, Hungary

What we have got so far?A few working modulesWorking RL architecturesWorking combination in an other problem domain: Internet searchand research is focused on how to link all of our concepts.

SSIP 2002 Budapest, Hungary

What next?Many avenuesOngoing projects with psychiatrists: trajectory analysis with cliplets, transient expressions, depression quantificationDistance learningHuman Computer InteractionVirtual reality

SSIP 2002 Budapest, Hungary

Infos about our research activity

http://people.inf.elte.hu/lorincz/

SSIP 2002 Budapest, Hungary

Thanks for your attention (and patience)!

SSIP 2002 Budapest, Hungary