1. Facial Expression Editing in Video Using a Temporally-Smooth Factorization

2. Face Swapping: Automatically Replacing Faces in Photographs

Facial Expression Editing in Video Using a Temporally-Smooth Factorization

Fei Yang, Lubomir Bourdev, Eli Shechtman, Jue Wang, Dimitris Metaxas

CVPR 2012

Goal

The goal is to allow for semantic-level editing of

expressions in a video:

magnifying an expression

suppressing an expression

replacing by another expressions

3

Example

4

Challenges

Natural expression

Different parts changes accordingly

Unique identity

Temporal coherency

5

Related Work 2D based methods

[Theobald09], [Liu01], [Williams90], …

3D based methods [Blanz03], [Pighin98], …

Expression flow [Yang11]…

Frame reorder method [Bregler98], [Kemelmacher- Shlizerman11]

Tensor factorization methods [Vlasic05], [Dale11]…

6

Algorithm

7

Expression Information

Identity Information

3D Tensor Model - [Vlasic et al siggraph05]

Modify

Mode-n Product

8

Algorithm

goal to identify a and

method 2D v.s. 3D

frame t

Minimize: | – |

=

Weak Projective Matrix Rt

Algorithm

Fitting Error:

Shape Distribution Constraint:

Temporal coherence:

Algorithm

11

Levenberg-Marquardt (Siggraph98)

Algorithm

12

Adjust to achieve expression

modification Dynamic Time Warping (DTW)

[Sakoe78]

Residual Expression Flow

Correcting boundary compatibility

Results

13

Face Swapping: Automatically Replacing Faces in Photographs

Dmitri Bitouk Neeraj Kumar Samreen Dhillon Peter Belhumeur Shree K. Nayar

Siggraph 2008

Examples

15

Goals

16

For an input image:

Automatically find the best candidate

Automatically replace the face

Automatically color and lighting adjustmet

Library Building

17

OKAO face detector to detect face pose [Omron07]

Process

18

Alignment

19

Pose, Resolution, and Image Blur:

Yaw, pitch threshold between two images ( )

Eye distance as a measure of distance (80%)

Similarity of the blur degrees [Kundur and

Hatzinakos 1996; Fergus et al. 2006]

Color and Lighting

20

To ensure the similarity between the replaced and original face, a linear combination of 9 spherical harmonics [Ramamoorthi and Hanrahan 2001; Basri and Jacobs 2003] is used as measure metric:Each pixel I(x, y) can be approximated by:

Distance:

Seam Signature

21

256-by-256 patch from the face is used for replacement.

Unfold:

L2 Norm is used to compute the distance

Appearance Adjustment

Using simple scaling on the Harmonics coefficients

, are the original and replacement images

Scale the replaced image

Results

The End

24

Any Questions

?