6/8/2011

1

Laboratoire d'InfoRmatique en Image et Systèmes d'information

UMR 5205 CNRS/INSA de Lyon/Université Claude Bernard Lyon

1/Université Lumière Lyon 2/Ecole Centrale de Lyon

http://liris.cnrs.fr

Ludovic Dutreve Alexandre Meyer Saïda Bouakaz

Université Lyon 1

Easy Acquisition and Real-Time Animation of Facial Wrinkles

Motivation

Facial Animation

o Realism of a virtual character

Real-time applications

o Video games

o Interactive systems

Issues

o High level human analysis

o Variety of deformations

o Tissues/muscles complexity

Fine details

o Wrinkles, bulges, …

o Improve expression recognition [Martin09]

May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 2

Related work: capture

Facial motion capture

o Never simple

• Structured light [Zhang04, Na04, …]

• Stereo with often several pair of cameras [Bickel07,…, Beeler10, …]

• Sphere of cameras/polarised lights (Emily project) [Debevec etal 09]

o Hard to apply only the details on other faces

• Capture the face and details in the same huge mesh

May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 3

[Beeler2010]

[Zhang2004]

[Bickel 07][Debevec etal 09]

Related work: synthesis

Dynamic skin effects (wrinkles)

o Hard to get real-time, control, few manual work and realism

• Procedural modeling [Bando02]

• Simulation [Sifakis05], compression [Volino99,Larboulette04]

• Synthesis from examples: wrinkle maps [Oat07]

�Need simple details acquisition methods compatible with real

time animation system

May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 4

[Sifakis 05][Bando 02] [Larboulette 04]

Overview

Footer

…

Session of capture (preprocessing)

Compute the

normal maps

Bones of

the frame i

Generate the normal map

using the reference maps

according to the bones positions

Set of reference poses

…

Reference

pose 0

Reference

pose 1

Reference

pose N

Easy Acquisition and Real-Time Animation of Facial Wrinkles May 2011

Winkles capture system (preprocessing)

1 camera with flash�photos of several expressions

N reference poses (= bones position + normal map)

8 June 2011 6

…Session of capture

Compute the

normal maps

(preprocessing)

…

Reference pose 0

(neutral)

Reference pose 1

Reference pose N

Easy Acquisition and Real-Time Animation of Facial Wrinkles

6/8/2011

2

Winkles capture system (preprocessing)

May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 7

From the photos to the normal maps

Neutral expressionExpression 1

Expression 1 deformed

by RBF to match the

neutral expression

Image ratio Normal map 1

mask

Landmarks (markers

or manually set)

Wrinkles capture: shape from shading

For each pixel

o Flash of the camera � L=(0,01)

o Ratio and assumption 1: each region = diffuse plane

� Declination of N

o Assumption 2: orientation of the gradient of the image ratio

~ orientation of the normal of the surface

� Orientation of N

May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 8

Image ratio Normal map 1

Wrinkles capture: results

May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 9

Image ratio

Normal map

Photos

During the animation: real-time wrinkles

Reference Poses = position of bones + wrinkles map (normal map)

o Position of bones� large-scale deformations

o Wrinkles map � small-scale deformations

May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 10

Input dataNeutral Reference Pose 0 Reference Poses Detail Maps

Runtime

Reference Pose 1 Reference Pose 2

During the animation: real-time wrinkles

May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 11

Input data

Pose EvaluationRuntimeArbitrary Pose

Neutral Reference Pose 0 Reference PosesReference Pose 1 Detail MapsReference Pose 2

During the animation: real-time wrinkles

May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 12

Input data

Arbitrary Pose New Map RenderingRefPose InfluencesPose Evaluation Runtime

Neutral Reference Pose 0 Reference PosesReference Pose 1 Detail MapsReference Pose 2

6/8/2011

3

Real-time wrinkles: influences at pixel level

To generate a new wrinkle map, needo Reference poses influences at a pixel level

From the bones level to the pixel levelo (1)Pose Evaluation Reference Poses Influences � for each bone

o (2)Skinning Bones � Vertices

o (3)Rasterization Vertices � Pixels

May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 13

Reference Poses Influences� Each pixel

α2

α1

α0

Bone i V2

V1

V0

…

Vn

(2)Skinning(1)Pose Evaluation (3)Rasterization

Real-time wrinkles: pose evaluation

During the animation, normal map generation

o For each bone b, compute the influences Ib of each reference pose i

• Orthogonal projection,

� Ib={ Ib0=c/r, …, Ib

i, …}

o For each vertices

• Use the weight of skinning to compute

the reference poses influences

• Vertex program (GPU)

o For each pixel

• Use barycentric interpolation

• Fragment program (GPU)

May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 14

Bone position of the

reference pose i

Bone position of

the neutral

reference pose 0

Current bone

positionc

r

Real-time wrinkles: non linear function

Introduce a non linear term in the normal map generation

to add control on wrinkles apparition

o For each bone b, compute the influences

Ib of each reference pose I

� Ib={ Ib0=F(c/r), …, Ib

i, …}

May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 15

Bone position of the

reference pose i

Bone position of

the neutral

reference pose 0

Current bone

positionc

r

Results : capture and real time synthesis

Tangent space normal map

Linear Blend Skinning

Reference poses

o Rest pose

o Compressed expression

o Stretched expression

May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 16

Examples of normal map captured

and applied to a 3D model Local influence

Results - video

May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 17

Conclusion

Capture• Easy to setup with a simple camera

• Few seconds to compute the normal maps (preprocessing)

Real-Time wrinkles generation• GPU implementation > 100fps

Easy to add to an existing implementation• Widely used techniques (Skinning, Bump-Mapping)

• No additional rendering pass or pipeline modification

Future worko Improve capture/shape from shading assumptions

o Data comparison with stereo-based capture

o Temporal normal maps capture

May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 18

6/8/2011

4

Questions ?

May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 19

Results - video

May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 20

24 mars 2011 21

Results

Pose neutre Pose référence 1 Pose référence 2

Overview of our system(2): animation

Online: during animation for frame i

May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 22

Bones of the frame i

Generate the normal map

using the reference maps

according to the bones positions

Set of reference poses

…

Reference

pose 0

Reference

pose 1

Reference

pose N

Results : real time synthesis

May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 23

3 references poses:

neutral, compressed

and stretched

Détails fins dynamiques

Acquisition

Adapter la carte de normale obtenue au personnage 3Do Espace photo. � Espace texture

• Points repères + Interpolation

�Carte de normale utilisable en 3D (Bump-Mapping)

24 mars 2011 24

Image ratio

Carte de normales

Adaptation