www.debevec.org paul debevec, tim hawkins, chris tchou, h.p. duiker, westley sarokin, and mark sagar...

www.debevec.orgwww.debevec.org

Paul Debevec, Tim Hawkins, Chris Tchou, H.P. Duiker, Westley Sarokin, and Mark Sagar

Paul Debevec, Tim Hawkins, Chris Tchou, H.P. Duiker, Westley Sarokin, and Mark Sagar

Acquiring the Reflectance Fieldof a Human Face

Acquiring the Reflectance Fieldof a Human Face

UC Berkeley / USC Institute for Creative Technologies / LifeF/XUC Berkeley / USC Institute for Creative Technologies / LifeF/X

Related WorkRelated WorkHanrahan and Krueger. Reflection from

Layered Surfaces due to Subsurface Scattering. SIGGRAPH 93

Bregler et al. Video Rewrite. SIGGRAPH 97

Guenter et al. Making Faces. SIGGRAPH 98

Pighin et al. Synthesizing Realistic Facial Expressions from Photographs. SIGGRAPH 98

Sagar et al. The Jester. SIGGRAPH 99 ET

Marschner et al. Reflectance Measurements of Human Skin. 1999

Hanrahan and Krueger. Reflection from Layered Surfaces due to Subsurface Scattering. SIGGRAPH 93

Bregler et al. Video Rewrite. SIGGRAPH 97

Guenter et al. Making Faces. SIGGRAPH 98

Pighin et al. Synthesizing Realistic Facial Expressions from Photographs. SIGGRAPH 98

Sagar et al. The Jester. SIGGRAPH 99 ET

Marschner et al. Reflectance Measurements of Human Skin. 1999

The Reflectance FieldThe Reflectance Field

The Reflectance FieldThe Reflectance Field

The Reflectance FieldThe Reflectance Field

The Reflectance FieldThe Reflectance Field

Ri( ui ,vi ,i ,i )Ri( ui ,vi ,i ,i )incident light fieldincident light field

The Reflectance FieldThe Reflectance Field

Rr ( ur ,vr ,r ,r )Rr ( ur ,vr ,r ,r )Ri( ui ,vi ,i ,i )Ri( ui ,vi ,i ,i )incident light fieldincident light field radiant light fieldradiant light field

The Reflectance FieldThe Reflectance Field

Ri( ui ,vi ,i ,i ; ur ,vr ,r ,r )Ri( ui ,vi ,i ,i ; ur ,vr ,r ,r )8D reflectance field8D reflectance field

Ri( ui ,vi ,i ,i ; ur ,vr ,r ,r )Ri( ui ,vi ,i ,i ; ur ,vr ,r ,r )

4D Slices of the 8D

Reflectance Field

4D Slices of the 8D

Reflectance Field

The Light StageThe Light Stage

The Light Stage:60-second exposure

The Light Stage:60-second exposure

Light Stage DataLight Stage DataOriginal

Resolution: 6432

Original Resolution:

6432

Lighting through image recombination: Haeberli ‘92, Nimeroff ‘94, Wong ‘97Lighting through image recombination: Haeberli ‘92, Nimeroff ‘94, Wong ‘97

Light Stage ResultsLight Stage Results

Environments from the Light Probe Image Gallerywww.debevec.org

Environments from the Light Probe Image Gallerywww.debevec.org

Reflectance FunctionsReflectance Functions

Lighting Reflectance FunctionsLighting Reflectance Functions

normalized light map

normalized light map

reflectance function

reflectance function

lighting productlighting product

rendered pixel

rendered pixel

1

1

DCT BasisDCT BasisSmith and Rowe. Compressed domain processing of JPEG-encoded images. 1996Smith and Rowe. Compressed domain processing of JPEG-encoded images. 1996

Interactive Lighting DemoSIGGRAPH 2000 Creative Applications Laboratory

Interactive Lighting DemoSIGGRAPH 2000 Creative Applications Laboratory

Interactive Lighting DemoSIGGRAPH 2000 Creative Applications Laboratory

Interactive Lighting DemoSIGGRAPH 2000 Creative Applications Laboratory

Changing the ViewpointChanging the Viewpoint

Reflection of Light from SkinReflection of Light from Skin

Specular Component:Color of light, shiny, brighter near grazing, maintains polarization

Subsurface Component:Color of skin, diffuse, desaturated near grazing, scrambles polarization

Specular Component:Color of light, shiny, brighter near grazing, maintains polarization

Subsurface Component:Color of skin, diffuse, desaturated near grazing, scrambles polarization

After Hanrahan ‘93After Hanrahan ‘93

Separating Reflectance Components using Crossed Polarizers

Separating Reflectance Components using Crossed Polarizers

Normal ImageNormal Image Subsurface ComponentSubsurface Component

Specular Component

Specular Component

Colorspace techniques - Sato ‘94, Nayar ‘97Colorspace techniques - Sato ‘94, Nayar ‘97

Transforming a Reflectance Function

Transforming a Reflectance Function

Subsurface ComponentSubsurface Component

Specular ComponentSpecular Component

Final RFFinal RF Comparison RFComparison RF

Shifted and ScaledSpecular

Shifted and ScaledSpecular

=> Torrance-Sparrow microfacet distribution

=> Torrance-Sparrow microfacet distribution

Surface Normal EstimateSurface Normal Estimate

Original RFOriginal RF

Point-Source ComparisonPoint-Source Comparison

Original ImageOriginal Image Novel ViewpointNovel Viewpoint

Spatially-VaryingReflectance Parameters

Spatially-VaryingReflectance Parameters

Surface Normals

n

Surface Normals

n

Diffuse Albedo

d

Diffuse Albedo

d

SpecularIntensity

ks

SpecularIntensity

ks

Specular Roughness

Specular Roughness

Compositing TestCompositing Test

Original ImageOriginal Image Light Probe Light Probe

Rendered FaceRendered Face Composite Composite

4. Ongoing Work4. Ongoing Work

• Animate the faces

• Capture more spectral samples

• Use high-speed cameras to achieve real-time capture

• Animate the faces

• Capture more spectral samples

• Use high-speed cameras to achieve real-time capture

5. Conclusion5. Conclusion

We have presented:

The light stage apparatus for capturing slices of the reflectance field of the human face

A technique for correctly relighting faces and objects with arbitrary illumination

A technique for extrapolating human reflectance to novel viewpoints

We have presented:

The light stage apparatus for capturing slices of the reflectance field of the human face

A technique for correctly relighting faces and objects with arbitrary illumination

A technique for extrapolating human reflectance to novel viewpoints

ThanksThanksDigital Media Innovation Program

Interactive Pictures Corporation

Alias|Wavefront

UCB Digital Digital Media/New Genre Program

ONR/BMDO

Cornell Program of Computer Graphics

Berkeley Millennium Project

Digital Media Innovation Program

Interactive Pictures Corporation

Alias|Wavefront

UCB Digital Digital Media/New Genre Program

ONR/BMDO

Cornell Program of Computer Graphics

Berkeley Millennium Project

and

Shawn Brixey, Bill Buxton, Larry Rowe, Jessica Vallot, Patrick Wilson, Melanie Levine, Eric Paulos, Christine Waggoner, Holly Cim, Eliza Ra, Bryan Musson,

David Altenau, Marc Levoy, Maryann Simmons, Henrik Wann Jensen, Don Greenberg, Pat Hanrahan, Randal Kleiser, Chris Bregler, Michael Naimark,

Dan Maas, Steve Marschner, and Kevin Binkert.

and

Shawn Brixey, Bill Buxton, Larry Rowe, Jessica Vallot, Patrick Wilson, Melanie Levine, Eric Paulos, Christine Waggoner, Holly Cim, Eliza Ra, Bryan Musson,

David Altenau, Marc Levoy, Maryann Simmons, Henrik Wann Jensen, Don Greenberg, Pat Hanrahan, Randal Kleiser, Chris Bregler, Michael Naimark,

Dan Maas, Steve Marschner, and Kevin Binkert.

In-plane Reflectometry Measurements

In-plane Reflectometry Measurements

Subsurface exhibits chromaticity falloff

Specular is monochromatic with Torrance-Sparrow microfacet behavior

Subsurface exhibits chromaticity falloff

Specular is monochromatic with Torrance-Sparrow microfacet behavior

SubsurfaceSubsurface

Subsurface + SpecularSubsurface + Specular

ii

rr

ii

rr

Reflectometry ExperimentReflectometry Experiment

The JesterSIGGRAPH 99 Electronic Theater

Mark Sagar et al. - LifeF/X, Inc.Performance and Text: Jessica Vallot

The JesterSIGGRAPH 99 Electronic Theater

Mark Sagar et al. - LifeF/X, Inc.Performance and Text: Jessica Vallot

Changing the ViewpointChanging the ViewpointModel from Structured Lighting

Traditional ApproachTraditional Approach

Derive reflectance parameters for each point on the face’s surface

Map the parameters onto a geometric model of the face

Render using traditional methods

Derive reflectance parameters for each point on the face’s surface

Map the parameters onto a geometric model of the face

Render using traditional methods

Reflectance Function Mosaic

Reflectance Function Mosaic