Digital Face Digital Face Replacement in Replacement in

PhotographsPhotographs

CSC2530F Project CSC2530F Project PresentationPresentation

By: Shahzad MalikBy: Shahzad Malik

January 28, 2003January 28, 2003

Face Replacement Face Replacement MotivationMotivation

Currently done manually by graphic Currently done manually by graphic artists using photo editing softwareartists using photo editing software

An automatic system has many An automatic system has many potential uses: potential uses: Hollywood special effectsHollywood special effects ““Personalized” moviesPersonalized” movies Framing someone…Framing someone…

Required ComponentsRequired Components

Need the following subsystems:Need the following subsystems: Face detection (and tracking for videos)Face detection (and tracking for videos) Head pose estimatorHead pose estimator Illumination extractor (*)Illumination extractor (*) Facial expression synthesisFacial expression synthesis Merging/replacement algorithm (*)Merging/replacement algorithm (*)

Light EstimationLight Estimation

Assuming a Lambertian reflectance model:Assuming a Lambertian reflectance model:

sn ρ)( ppI Any image can then be represented by:Any image can then be represented by:

]sss[n ρ)( 332211 pS pI

)()()()( 332211 pIpIpIpI S

Approximate Skin ToneApproximate Skin Tone

Cannot assume 3 basis images for Cannot assume 3 basis images for arbitrary photographsarbitrary photographs

Use an approximate image to Use an approximate image to generate basisgenerate basis

Fitting a Generic 3D ModelFitting a Generic 3D Model Need geometry to create basis imagesNeed geometry to create basis images Fit a generic 3D face mesh to imagesFit a generic 3D face mesh to images ““Lift” a texture using planar mappingLift” a texture using planar mapping

Generate Basis ImagesGenerate Basis Images

Set 3 linearly independent light positionsSet 3 linearly independent light positions Relight skin tone model with each lightRelight skin tone model with each light

Determining the Determining the CoefficientsCoefficients

Compute a least squares solution to:Compute a least squares solution to:

3

2

1

321 )()()()(

pIpIpIpI S

Solve separately for each RGB channelSolve separately for each RGB channel

Re-illuminating the Target Re-illuminating the Target FaceFace

Set intensities of the 3 light sources to Set intensities of the 3 light sources to the coefficient valuesthe coefficient values

Render the target face with these lightsRender the target face with these lights]sss[n ρ)( 332211 pR pI

Flesh Pixel DetectionFlesh Pixel Detection Match non-mesh skin pixels to new Match non-mesh skin pixels to new

skin toneskin tone Use a histogram-based skin classifierUse a histogram-based skin classifier

Histogram MatchingHistogram Matching

Generate histograms for newly lit Generate histograms for newly lit faceface

Match the Gaussian distribution from Match the Gaussian distribution from original face to newly lit faceoriginal face to newly lit face

For each flesh pixel in original image, For each flesh pixel in original image, choose a new color with a similar choose a new color with a similar location on the Gaussian bell curvelocation on the Gaussian bell curve

Weighted Color BlendingWeighted Color Blending

ff

fe

f

ff p

B

dp

B

dp )0.1(

Blend converted flesh pixels with Blend converted flesh pixels with face mesh pixelsface mesh pixels

ResultsResults

Results (continued)Results (continued)

Results (continued)Results (continued)

Results (continued)Results (continued)

SummarySummary

Presented a face replacement systemPresented a face replacement system Takes lighting and merging into accountTakes lighting and merging into account Future research areas:Future research areas:

Face detection and tracking (for videos)Face detection and tracking (for videos) Expression synthesisExpression synthesis More sophisticated reflectance modelMore sophisticated reflectance model Automatic and precise model-fittingAutomatic and precise model-fitting