Image-based Rendering of Real Objects Image-based Rendering of Real Objects with Complex BRDFswith Complex BRDFs

Intensity of One PixelIntensity of One Pixel

s1(, )

q

p

Consider the measured intensity at one pixelConsider the measured intensity at one pixel

I1(1, 1)

as the isotropic point source is moved over as the isotropic point source is moved over the surface.the surface.

I1(, )

Note similarity to Note similarity to

•(Levoy, Hanrahan, 1996) (Levoy, Hanrahan, 1996)

•(Gortler et al, 1996)(Gortler et al, 1996)

Phong Intensity of One Pixel: Phong Intensity of One Pixel: II11((, , ))

This is effectively a 2-D slice of a point’s BRDF except for This is effectively a 2-D slice of a point’s BRDF except for

• ShadowingShadowing

• 1/r1/r22 falloff from the source falloff from the source

Intensity of One Pixel: Intensity of One Pixel: II11((, , ))

This is effectively a 2-D slice of a point’s BRDF except for This is effectively a 2-D slice of a point’s BRDF except for

• ShadowingShadowing

• 1/r1/r22 falloff from the source falloff from the source

Image AcquisitionImage Acquisition

Intensity Over Second SurfaceIntensity Over Second Surface

s1(, )

s2(, )

p

Now, consider moving an isotropic point Now, consider moving an isotropic point source over a second surface and measuring source over a second surface and measuring the intensity of the same pixel: the intensity of the same pixel:

I2(, )I1(, )

I2(, )

II11((, , ) and I) and I22((, , ))

Inner

Sphere:

I1(, )

Outer

Sphere:

I2(, )

Relation Between Intensity MapsRelation Between Intensity Maps

When the surface point p, When the surface point p, s1() and s2() are collinear (in correspondence), the measured pixel intensities are simply related by the relative 1/r2 losses.

s1(, )

s2(, )

p

Depth EstimationDepth Estimation

s1(, )

s2(, )

p()

This correspondence can be expressed as a This correspondence can be expressed as a change of coordinates change of coordinates 22((; ; )) and and

22((; ; ) parameterized by depth ) parameterized by depth .

We can then estimateWe can then estimate by minimizing by minimizing:

O()= [I2(2(), 2 ()) - r2 I1(1,1) ]2d1d1

A Reconstructed Depth MapA Reconstructed Depth Map

143 Images on 143 Images on each surfaceeach surface

Rendering Synthetic Images: Point SourcesRendering Synthetic Images: Point Sources

New light position

Intersection with the sphere

• Intersect light ray throughIntersect light ray through P P with sphere.with sphere.

• Find triangle of light sources Find triangle of light sources containing containing PP..

• Interpolate pixel intensities of Interpolate pixel intensities of images corresponding to the images corresponding to the triangle vertices.triangle vertices.

• For a given image point, there For a given image point, there is a scene point: is a scene point: PP

PP

Rendered ImagesRendered Images

Rendered Image: A Sea ShellRendered Image: A Sea Shell

Isotropic point light source Isotropic point light source located between acquisition located between acquisition spheres.spheres.

Rendered Image: A PearRendered Image: A Pear

• Two light sourcesTwo light sources• Point source to the leftPoint source to the left• 3 by 5 cm area source3 by 5 cm area source to the rightto the right

Video Compositing of Real ObjectsVideo Compositing of Real Objects

Video Frame #567 Radiance Map Frame #567

Video CompositingVideo Compositing

Background Image #2313 Object Image #2313

Video CompositingVideo Compositing

Composite Frame #567 Composite Frame #567

Lighting Sensitive Displays

Shree Nayar Peter Belhumeur Terry Boult

Columbia Yale Lehigh

Computer Vision Laboratory

Columbia University

Sponsor: NSF ITR

Displays Everywhere

But, Displays are Passive

brightness

contrast

display

content

Lighting Sensitive Display (LSD)

• Senses the Environmental Illumination

• Modifies Displayed Content Accordingly

illumination

: Perception

: Reaction

State of the Art

brightness

contrastphotodetector

adjustment

Heijligers 62 ; Thomas 63; Gibson 64; Korda 65; Biggs 65; Szermy 68Newman 72; Constable 78; Fitzgibbon 82; Antwerp 85; Otenstein 93

Display’s Illumination Field

display

content

display

content

(s,t)(u,v)

L(s,t,u,v,

• Wide Range of Sources: Sunlight, Overcast, Halogen, Fluorescent ...

• Arbitrarily Complex : Point/Extended/Multiple Sources, Scene Radiance ...

(s,t)(u,v)

L(s,t,u,v,

Four-Dimensional Ray Manifold

Methods for Sensing the Illumination Field

photodetectorsoptical fibers

hemispherical camera

??

probe video

Compact Hemispherical Illumination Probe

compact wide angle optics

color video camera

neutral density filters

LSD Prototype

Sony 15” LCDFlat Display

HemisphericalProbe Camera

Matting

Wooden Frame

Content Modification : Rendering

• Power Efficiency

• Brighter in Sunlight

• Dimmer Indoors

• Compensation

• Spatially Varying Brightness

• Spatially Varying Color

• Photorealism

• Consistent Colors and Shadings

• Consistent Highlights and Shadows

All Modifications in Real-Time

Rendering Using Explicit Models: 2D+

v

s1

n

s2

O

viewer

source

source

display

rendered image

content: surface

Algorithms: Ray Tracing, Radiosity

Rendering Using Explicit Models: 3D

v

s1

n

s2

O

viewer

source

source

display

rendered image

content:shape, BRDF

Algorithms: Ray Tracing, Radiosity

Image based Rendering

probe cameracapture camera

Off-line Scene Capture

(with Kudelka and Swaminathan)

Efficient Representation and Rendering

Image Bases E

source directions

40

30

1616

Captured Images

i=1 i=4096

I

40

30

16

16

k=1 k=10

bloc

ks x

bas

is

3

0 x

40 x

10

Lighting Coefficient Vectors L

source directionsi=1 i=4096

SVD

Efficient Representation and Rendering

Compressed Coefficient Vectors

Coeff. Bases U

source directions

Coefficient Vectors L

i=1 i=4096

V

q=1 b=200

bloc

ks x

bas

is

3

0 x

40 x

10

SVD

source directionsi=1 i=4096

Real-Time Rendering

Compressed Coefficient Vector

Coeff Eigenvectors U

Illumination Field

Vs

Coefficient Vector

Image Eigenvectors E

Display

I

Compressed Coefficient Vectors V

s

X X X

UVs

Efficient Representation and RenderingCaptured Data

ComputeLocal Subspaces

Local Bases and Coefficients

source direction

Image Reconstruction

Display Illumination Field

Display

4 Gb

10 Mb 8 fps (laptop)

Efficient Representation and RenderingCaptured Data

ComputeLocal Subspaces

Local Bases and Coefficients

source direction

Image Reconstruction

Display Illumination Field

Display

4 Gb

10 Mb 8 fps (laptop)

Face

Still Life: Scene Capture

Still Life

Summary

• Lighting Sensitive Display:

• Senses Environmental Illumination

• Modifies Displayed Content

• Applications:

• Compensation: Computers, PDA’s, Televisions, Billboards

• Photorealism: Digital Art, E-Commerce, Future Homes

Capturing Scenes for Image based Rendering

probe cameracapture camera