Virtual Environments with Light Fields and

Lumigraphs

CS 497

16 April 2002

Presented by Mike Pilat

2

Three Papers

Light Field RenderingLevoy and HanrahanSigGraph 1996

The LumigraphGortler, Grzeszczuk, Szeliski and CohenSigGraph 1996

Dynamically Reparameterized Light FieldsIsaksen, McMillan and GortlerSigGraph 2000

3

Introduction

Traditional 3D scenes are composed of geometric data and lights.

Can be costly to render new views Use set of images to compute new views of

scene Images are pre-acquired Solution: Image-based rendering

4

Image-Based Rendering

Algorithms for IBR require little computational resources; suitable for PCs and workstations

Cost of interactive viewing independent of scene complexity

Image sources can come from: Real photographs Rendered images Both!

5

A long, long time ago…

Environment maps 1976; Blinn: “Texture and Reflection in Computer

Generated Images” 1986; Greene: “Environment Mapping and Other

Applications of World Projections” They work the other way around too

1995; Chen: “QuickTime VR” Major limitation: Fixed view-point

6

… In a galaxy far, far away

View Interpolation Many people: Chen, Greene, Fuchs, McMillan, Narayanan Not always practical

Requires depth value for each pixel in the environment maps.

Need to fill gaps created by occlusions. Image Interpolation

Laveau, McMillan, Seitz Need to find correspondences in two images

Stereo vision problem == hard Correspondences don’t always exist

7

Light Fields

8

Light Fields

Radiance at a point, in a particular direction Coined by Gershun

1936; Gershun: “The Light Field”

Equivalent to Plenoptic Function 1991; Adelson and Bergen: “The Plenoptic Function and

the Elements of Early Vision”

5D Light Fields 1995; McMillan and Bishop: “Plenoptic Modeling: An

Image-Based Rendering System” Panoramic images in 3D space

9

Light Fields May reduce 5D representation to 4D in free space

(no occluders) Radiance does not change along a line unless blocked Redundant (read: 5D) representation is bad

Increased dataset size Complicates reconstruction of radiance samples

Can interpret 4D light fields as “functions on the space of oriented lines”

This is the representation of Levoy and Hanrahan Choose to parameterize lines by their intersection with two

planes in arbitrary positions. “Light Slab” – Restrict parameters to [0,1]

10

Parameterization

Want parameterization to have three qualities Efficient Calculation

Want to use only view transform and pixel location Control over the set of lines

Only need a finite subset of line space Uniform Sampling

Number of lines between in intervals between samples is uniform

Can generate all views from one light slab But only if set of lines include all lines intersecting the

convex hull of the object Not possible, so need to use multiple slabs

11

Huh?

12

Creating Light Fields from Rendered Images

Render a 2D array of images

Each image is slice of 4D light slab

Place center of projection of camera at locations on uv-plane

Need to perform sheared perspective projection to force xy samples of the image to correspond exactly with the st samples

13

Creating Light Fields from Digitized Images Many problems

Need hundreds or thousands of images Need to control lighting (must be static) Must manage optical constraints

Angle of View Focal Distance Depth of Field Aperture

Viewing Inward Looking

Flying around an object Outward Looking

Looking out from an object (inside a house, e.g.)

14

Creating Light Fields from Digitized Images

Computer-controlled acquisition Spherical motion

Easier to cover entire range of viewing directions Sampling rate is more uniform

Planar motion Easier to build Closer to light slab representation

Constructed planar gantry x,y-movement Pitch and yaw

15

Looking at a Light Field

16

Compression Desirable properties

Data Redundancy Remove redundant info without affecting content Redundancy in all four dimensions

Random Access Samples referenced during rendering are scattered During movement access pattern changes

Asymmetry Assume light fields are pre-computed Long encoding time OK if decode is fast

Computationally Efficient Can’t consume full CPU power Need CPU time for display algorithm as well

Utilize two-stage compression scheme

17

Compression

Vector Quantization (24:1) Vector of samples quantized into a predetermined

reproduction vector Reproduction vector called a “codeword” “codebook” of rep-vectors used to encode data

Entropy Coding (Lempel-Ziv) (5:1) Objects usually imaged on constant-intensity

background Many vectors from VQ occur with high probability

18

Compression Two-stage decompression

Decode entropy coding while loading file End up with codebook and

code indices packed into 16-bit words.

De-quantization Engine requests samples

of light field Subscript index calculated Look up index in codebook Get vector of sample

values Subscript again for

requested sample

Digitization Slabs 4 Images/Slab 32x16 Pixels/Image 256x256 Total Size 402 MB

Compression VQ coding 24:1 Entropy coding 5:1 Total 120:1 Comp. Size 3.4 MB

19

Displaying a Light Field

Step 1 Compute the (u,v,s,t)

parameters for each image ray

Step 2 Resample the radiance

at those line parameters

20

Displaying a Light Field Step 1

Simple projective map from image coords to (u,v) and (s,t)

Can use texture mapping to compute line coords u = uw/w

v = vw/w(uw, vw, w)

Inverse map to (u,v,s,t) only two texcoord calculations per ray

Can be hardware accelerated!

21

Displaying a Light Field Step 2

Reconstruct radiance function from original samples Approximate by

interpolation from nearest sample

Prefilter (4D mip map) if image is small

Quadralinear interpolation on full 4D function best

Apply band pass filter to remove HF noise that may cause aliasing

22

Results

23

Results

24

Limitations

Need high sampling density Prevents excessive blurriness Need large number of images Thanks to coherence compression is good

Observer restricted to free space Can be addressed by stitching together multiple light fields

based on geometry partition

Requires fixed illumination If interreflections are ignore, can address by augmenting

light field with surface normals and optical properties

25

Future Work

Abstract light representations have not received the systematic study as other methods of light representation Re-examine from first principles

Design better instrumentation for image acquisition Parallel array of cameras

Representing light interactions in high-dimension matrices And how to compress ‘em

26

The Lumigraph

27

Lumigraphs

Limit interest to light leaving the convex hull of a bounded object

Only need values of plenoptic function on a surrounding surface

Chose cube for computational simplicity

28

Parameterized Déjà Vu?

29

Parameterization

Use same method as Levoy and Hanrahan

Discrete subdivision of function space M in st, N in uv Basis Functions

Basis Projection

Duals of Basis functions

),,,(),,,(~

,,,0 0 0 0

,,, vutsBxvutsL qpji

M

i

M

j

N

p

N

qqpji

qpjiqpji BLx .,,,,,

~,

30

Using Geometric Information

Use geometric information to learn about coherence of Lumigraph function

Modify basis functions to weight for that coherence

Use intersection depth

zz

issuu 1)('

)',',,(),,,( ,,,'

,,, vutsBvutsB pqjipqji

31

Creating a Lumigraph

32

Capturing from Synthetic Scenes

Capture a single sample per Lumigraph coefficient at each grid point

Place pinhole of camera at grid point Point down z axis Pixel values (p,q) are used in coefficients xi,j,p,q

To integrate against kernel B~, shoot multiple rays, jitter camera and pixel locations, weight each image using B~.

33

Capturing from Real Images

Use motion control platform from L&H Capture images at grid points Would rather use hand-held camera Place calibration marks in scene Obtain silhouette through blue-screening to

get rough geometric shape Use silhouettes to build a volumetric model Guide user interactively in selecting new

positions

34

Calibrating for Real Images

35

Rebinning “The coefficient associated with the basis function B

is defined as the integral of the continuous Lumigraph function multiplied by some kernel function B~”

Three-step approach developed Splat Pull Push

dvdudtdsvutsBvutsLx qpjiqpji ),,,(~

),,,( ,,,,,,

36

Splat, Push, Pull

Splatting – estimates integral Coefficents are computed with Monte-Carlo

integration Pulling – coefficients computed for

hierarchical set of basis functions Linearly sum together higher-resolution kernels

Pushing – information from lower grids “pushed” up to higher-res ones to fill in gaps Up sample and convolve

37

Reconstruction

Ray Tracing Generate ray Calculate (s,t,u,v) coords Interpolate nearby grid

points

Texture Mapping One texture associated

with each st grid point Project pixel onto uv-

plane

38

Results

39

Results

40

Dynamically Reparameterized Light Fields

41

New parameterizations

Interactive Rendering Moderately sampled light fields Significant, unknown depth variation

Passive autostereoscopic viewing “fly’s eye” lens array Computation for synthesizing a new view comes

directly from optics of display device

To achieve desired flexibility, do not perform aperture filter from original paper.

42

Light Slabs (Again)

Two-plane parameterization impacts reconstruction filters

Aperture filtering removes HF data

But if plane is wrong place, only stores blurred version of scene

43

Focal Surface Parameterization

2D array of pinhole cameras, treated as a single optical system

Each data camera Ds,t can have its own internal orientation and parameters

Focal surface F parameterized by (f,g)F

44

Ray Construction

Given a ray r, can find rays (s’,t’,u’,v’) and (s’’,t’’,u’’,v’’) in Ds’,t’ and Ds’’,t’’ that intersect at the same (f,g)F

Combine values with a filter

In practice, use more rays and weight the filter accordingly

45

Variable Aperture

Real camera aperture produces depth-of-field effects

Emulate depth-of-field effects by combining rays from several cameras

Combine samples for each Ds,t in the synthetic aperture to produce image

46

Variable Focus

Can also create variable focus effects

Multiple focal surfaces Can use one or many

simultaneously Similar ray construction

technique

47

Boring Stuff

Ray-Space Analysis Examine the effects of dynamic

reparameterization on light fields when viewed from ray space.

Frequency Domain Analysis Shearing can arbitrarily modify the relative

sampling frequencies between dimensions in ray space

See paper

48

Rendering

Ray Tracing Trace rays like before

Texture Mapping If the desired camera,

data cameras, and focal surface are all planar, can apply texture mapping techniques similar to Lumigraph method.

49

Autostereoscopic Light Fields

Can think of the parameterization as a integral photograph, or the “fly’s eye view”

Each lenslet treated as single pixel

50

View-Dependent Color

Each lenslet can have view dependent color Draw a ray through the principle point of the lenslet Use paraxial approximation to determine which color

will be seen from a direction

51

Results

52

Results

53

Results

54

Results

55

Results

56

Results

57

Future Work

Develop method to exploit redundancy of light fields

Develop algorithm for automatically selecting the focal plane (like camcorders)

Promise for using reparameterization for depth-from-focus and depth-from-defocus problems

58

Fin

Questions?

Thanks!