single image portrait relighting™天成.pdf · single image portrait relighting tiancheng sun1,...

Single Image Portrait Relighting

Tiancheng Sun1, Jonathan T. Barron2, Yun-Ta Tsai2, Zexiang Xu1, Xueming Yu3, Graham Fyffe3, Christoph Rhemann3, Jay Busch3,

Paul Debevec3, Ravi Ramamoorthi1

1University of California, San Diego, 2Google Research, 3Google

Photography & RecordingAllowed

Overview

Overview

light from the back

Overview

light from the back shadow on the face

Overview

want to add dramatic lightinglight from the back shadow on the face

Overview

want to add dramatic lightinglight from the back shadow on the face

Change the lighting of any portrait after capture using post-processing algorithm

Overview

Portrait Relighting System

Overview


Overview

another lighting

Overview


Overview


I want to rotate the lighting a little bit.

Previous work

Previous work• Light Stage

Debevec, Paul, et al. "Acquiring the reflectance field of a human face." SIGGRAPH 2000.

Previous work• Light Stage


: capture ~100 images and do image-based relighting

Previous work

• Deep image-based relighting

Xu, Zexiang, et al. "Deep image-based relighting from optimal sparse samples." SIGGRAPH 2018

Previous work

• Deep image-based relighting

Xu, Zexiang, et al. "Deep image-based relighting from optimal sparse samples." SIGGRAPH 2018

capture 5 images and do relighting via neural network

Previous work

Previous work

• Portrait lighting transfer

Shu, Zhixin, et al. "Portrait lighting transfer using a mass transport approach." SIGGRAPH 2018

Previous work

• Portrait lighting transfer

Shu, Zhixin, et al. "Portrait lighting transfer using a mass transport approach." SIGGRAPH 2018

transfer lighting from one portrait to another

Previous work

Previous work

Normal

Shading

Albedo

Relit

• SfSNet

Sengupta, Soumyadip, et al. "SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wild'." CVPR 2018.

Previous work

Normal

Shading

Albedo

Relit

• SfSNet

Sengupta, Soumyadip, et al. "SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wild'." CVPR 2018.

: deep intrinsic decomposition mostly trained on synthetic faces

Previous work

Overview

• Goal: practical relighting on single portrait image

Overview


• Practical in detail:

• Robust to the pose and camera view

• Work well on natural lightings

• Adapt to high-resolution images

• Run at interactive rate

Overview


• Practical in detail:

• Robust to the pose and camera view

• Work well on natural lightings

• Adapt to high-resolution images

• Run at interactive rate

• Solution: Deep Neural Network + Real Face Data.

Overview

Method

Method

portrait under lighting A

Method


(Neural Network)portrait under

lighting A

Method



lighting Alighting B

Method



lighting Aportrait under

lighting Blighting B

Method




lighting Blighting A lighting B

Method




lighting Blighting A lighting B

How can we get the portrait pair for training?

Method: Data

Method: DataLight Stage

Method: DataLight Stage One-Light-At-a-Time scans (OLAT)

Method: Data

Method: Data

captured OLAT captured OLATDebevec, Paul, et al. "Acquiring the reflectance field of a human face." SIGGRAPH 2000.

lighting

Method: Data

captured OLAT captured OLATDebevec, Paul, et al. "Acquiring the reflectance field of a human face." SIGGRAPH 2000.

lighting

Method: Data

captured OLAT captured OLAT

latitude-longitude representation


lighting

x x+ + …… =

Method: Data

captured OLAT captured OLAT



lighting

x x+ + …… =

Method: Data

captured OLAT captured OLAT relit image (background removed)



lighting

x x+ + …… =

Method: Data

captured OLAT captured OLAT relit image (background removed)


Wadhwa, Neal, et al. "Synthetic depth-of-field with a single-camera mobile phone." SIGGRAPH 2018Debevec, Paul, et al. "Acquiring the reflectance field of a human face." SIGGRAPH 2000.

Method: Data

• OLAT images• 22 people (18 training, 4 validation), each 3~5 facial expressions

Method: Data


• Each OLAT captured with 7 cameras in 6 seconds.

Method: Data



• HDR lighting environments

Method: Data



• HDR lighting environments• ~2000 indoor HDR lighting

from Laval Dataset

Method: Data




from Laval Dataset

• ~1000 outdoor HDR lighting from the web

Method: Data




from Laval Dataset

• ~1000 outdoor HDR lighting from the web

Method: Data

• Total: 226,800 portrait and lighting pairs for training

Method: Training

Method: Training

Encoder DecoderBottleneck

Method: Training


• Task 1: Complete relighting

Method: Training


source image


Method: Training


source light

source image


Method: Training


source light target light

source image


Method: Training



source image target image


Method: Training




• Task 1: Complete relighting L1 loss

Log L1 loss

Method: Training


Method: Training• Task 2: Illumination retargeting




source image



source light

source image



source light

source image

Rotate




source image

Rotate





Rotate





Rotate

L1 loss

Log L1 loss

Method: Training

Method: Training• Network structure• U-Net

k x k conv layer concatenation

kweighted average tiling

kk-dimensional

input/label

kk-dimensional

activation

256 x 256 128 x 128 64 x 64 32 x 32 16 x 16Spatial Resolution:

loss

Method: Training• Network structure• U-Net



kk-dimensional

input/label

kk-dimensional

activation


loss

SourceImage 64 643 128 1283 256 2563 512 5123 5123 5123

OutputSourceLight

16 x 32 x 33

297 16 x 32 x 13 3 3 3 33

TrueSourceLight

16x

32x3

16x

32x3

Method: Training• Network structure• U-Net• Predict and feed in light at bottleneck



kk-dimensional

input/label

kk-dimensional

activation


loss

SourceImage 64 643 128 1283 256 2563 512 5123 5123 5123

OutputSourceLight

16 x 32 x 33

297 16 x 32 x 13 3 3 3 33

TrueSourceLight

16x

32x3

16x

32x3


confidence learning module



kk-dimensional

input/label

kk-dimensional

activation


loss

SourceImage 64 643 128 1283 256 2563 512 5123 5123 5123

OutputSourceLight

16 x 32 x 33

297 16 x 32 x 13 3 3 3 33

TrueSourceLight

16x

32x3

16x

32x3


confidence learning module



kk-dimensional

input/label

kk-dimensional

activation


loss

SourceImage 64 643 128 1283 256 2563 512 5123 5123 5123

OutputSourceLight

16 x 32 x 33

297 16 x 32 x 13 3 3 3 33

TrueSourceLight

16x

32x3

16x

32x3

3 512TargetLight3

32

323

512

256

512

3

512

512

256

3

256128

3

12864

3

64

51225625612812864643 3 3 3

16x

32x3

TrueTargetImage

OutputTargetImage

3

3

Method: Training

Method: Training• Confidence learning

Method: Training

Several conv layers

• Confidence learning

Method: Training

Several conv layers

resolution of the light


Method: Training

Several conv layers


Light prediction on each image patch


Method: Training

Several conv layers



Confidence of prediction on each

image patch

• Confidence learning• Predict the confidence of light prediction

Method: Training

Several conv layers




image patch

* =

• Confidence learning• Predict the confidence of light prediction

Reshape

Method: Training

Several conv layers




image patch

* =

• Confidence learning• Predict the confidence of light prediction• Allow network to say “I don’t know”

Reshape

Results: Validation

Results: Validation➤ Relit images for complete relighting


source image


source image target image ours [Barron & Malik 2015][Sengupta et al. 2018] [Li et al. 2018]

Results: Validation

Results: Validation

➤ Relight images with predicted light as target light


Results: Validation


source image


Results: Validation


source image with self-supervision

withoutself-supervision


Results: Validation

Results: Validation➤ Comparison with portrait lighting transfer


reference image


source image

reference image


source image

reference image

Extract light from reference


source image

reference image


Apply to source image


source image

groundtruth oursreference image




source image

groundtruth ours

[Shih et al. 2014] [Shu et al. 2018]

reference image



Results: Validation

Results: Validation➤ Evaluation on lighting prediction


source image

ground truth


source image

ground truth

ours

ours w/o confidence learning


source image

ground truth

ours

ours w/o confidence learning

[Barron & Malik 2015]

[Sengupta et al. 2018]

Results: Images in the wild

Results: Images in the wildInput Image Relit Image

Results: Images in the wildInput Image Relit ImageInput Image Relit Image

Results: Images in the wild

Limitations

LimitationsInput Image Relit Image

• Complex shadows

• Specular highlights

• Overexposed pixels


• Complex shadows



• Over-smoothing


• Complex shadows



• Over-smoothing

• Unseen high-saturation color

Conclusion

Conclusion

• Learn the relighting function on portraits using Light Stage data

Conclusion


• Take home message:

Conclusion



• For human faces, use real data.

Conclusion



• For human faces, use real data.

• End-to-end training vs assuming models.

Acknowledgement• This work was funded in part by a Jacobs Fellowship, the Ronald L.

Graham Chair, and the UC San Diego Center for Visual Computing.

• Thanks to Zhixin Shu and Yichang Shih for the help on baseline algorithms

• Thanks to Jean-François Lalonde for providing the indoor lighting dataset.

• Thanks to Peter Denny for coordinating dataset capture.

• Thanks to all the anonymous volunteers in the dataset from Google, UCSD and UCLA

Also presenting in poster session today at 12:15-1:15.



input imagegenerated portraits under new illuminations

single image portrait relighting™天成.pdf · single image portrait relighting tiancheng sun1,...

Documents