Lightfields and Lumigraphs

Vision for GraphicsCSE 590SS, Winter 2001

Richard Szeliski

(with lots of slides from Michael Cohen)

3/7/2001 Vision for Graphics 2

Modeling light

How do we generate new scenes and animations from existing ones?

Classic “3D Vision + Graphics”:• take (lots of) pictures• recover camera pose• build 3D model• extract texture maps / BRDFs• synthesize new views

3/7/2001 Vision for Graphics 3

Computer Graphics

Image

Output

ModelSyntheticCamera

3/7/2001 Vision for Graphics 4

Real Scene

Computer Vision

Real Cameras

Model

Output

3/7/2001 Vision for Graphics 5

Combined

Model Real Scene

Real Cameras

Image

Output

SyntheticCamera

3/7/2001 Vision for Graphics 6

But, vision technology falls short

ModelReal Scene

Real Cameras

Image

Output

SyntheticCamera

3/7/2001 Vision for Graphics 7

… and so does graphics.

ModelReal Scene

Real Cameras

Image

Output

SyntheticCamera

3/7/2001 Vision for Graphics 8

Image Based Rendering

Real Scene

Real Cameras-or-

Expensive Image Synthesis

Images+Model

Image

Output

SyntheticCamera

3/7/2001 Vision for Graphics 9

Ray

Constant radiance• time is fixed

5D• 3D position• 2D direction

3/7/2001 Vision for Graphics 10

All Rays

Plenoptic Function:• all possible images• too much stuff!

3/7/2001 Vision for Graphics 11

Line

Infinite line

4D• 2D direction• 2D position• non-dispersive medium

3/7/2001 Vision for Graphics 12

Ray

Discretize, then interpolate

Distance between 2 rays• Which is closer together?

3/7/2001 Vision for Graphics 13

Image

What is an image?

All rays through a point• Panorama?

3/7/2001 Vision for Graphics 14

Image

Image plane

2D• position

3/7/2001 Vision for Graphics 15

Light leaving towards “eye”

2D• just dual of image

Object

3/7/2001 Vision for Graphics 16

Object

All light leaving object

3/7/2001 Vision for Graphics 17

Object

4D• 2D position• 2D direction

3/7/2001 Vision for Graphics 18

Object

All images

3/7/2001 Vision for Graphics 19

Lumigraph / Lightfield

Outside convex space

4D

StuffEmpty

3/7/2001 Vision for Graphics 20

Lumigraph

Inside convex space

4D

EmptyStuff

3/7/2001 Vision for Graphics 21

Lumigraph

How to • organize• capture• render

3/7/2001 Vision for Graphics 22

Lumigraph - Organization

2D position

2D direction

s

3/7/2001 Vision for Graphics 23

Lumigraph - Organization

2D position

2D position

2 plane parameterization

su

3/7/2001 Vision for Graphics 24

Lumigraph - Organization

2D position

2D position

2 plane parameterization

us

t s,tu,v

v

s,t

u,v

3/7/2001 Vision for Graphics 25

Lumigraph - Organization

Hold s,t constant

Let u,v vary

An image

s,t u,v

3/7/2001 Vision for Graphics 26

Lumigraph - Organization

Discretization• higher res near object

– if diffuse– captures texture

• lower res away– captures directions

s,t u,v

3/7/2001 Vision for Graphics 27

Lumigraph - Capture

Idea 1• Move camera carefully over

s,t plane• Gantry

– see Lightfield paper

s,t u,v

3/7/2001 Vision for Graphics 28

Lumigraph - Capture

Idea 2• Move camera anywhere• Rebinning

– see Lumigraph paper

s,t u,v

3/7/2001 Vision for Graphics 29

Lumigraph - Rendering

For each output pixel• determine s,t,u,v• either

– find closest discrete RGB– interpolate near values s,t u,v

3/7/2001 Vision for Graphics 30

Lumigraph - Rendering

For each output pixel• determine s,t,u,v

• either• use closest discrete RGB• interpolate near values

s u

3/7/2001 Vision for Graphics 31

Lumigraph - Rendering

Nearest• closest s• closest u• draw it

Blend 16 nearest• quadrilinear interpolation

s u

3/7/2001 Vision for Graphics 32

Lumigraph - Rendering

Depth Correction• closest s• intersection with “object”• best u• closest u

s u

3/7/2001 Vision for Graphics 33

Lumigraph - Rendering

Depth Correction • quadralinear interpolation• new “closest”• like focus

s u

3/7/2001 Vision for Graphics 34

Lumigraph - Rendering

Fast s,t,u,v finding• scanline interpolate• texture mapping• shear warp

s u

3/7/2001 Vision for Graphics 35

Lumigraph - Ray Space

3/7/2001 Vision for Graphics 36

Lumigraph - Ray Space

Image effects:• parallax• occlusion• transparency• highlights

3/7/2001 Vision for Graphics 37

Lumigraph - Demo

Lumigraph• Lion, Fruit Bowl, Visible Woman, Path Tracing

3/7/2001 Vision for Graphics 38

Lightfield - DemoDigital Michelangelo Project

– Marc Levoy, Stanford University– Lightfield (“night”) assembled by Jon Shade

3/7/2001 Vision for Graphics 39

Surface Lightfields

Turn 4D parameterization around

Leverage coherence

3/7/2001 Vision for Graphics 40

Surface Lightfields

Wood et al, SIGGRAPH 2000

3/7/2001 Vision for Graphics 41

3D Representations

Image is 2D

Lumigraph is 4D

What happened to 3D?• 3D Lumigraph subset• Concentric mosaics

3/7/2001 Vision for Graphics 42

3D Lumigraph

One row of s,t plane• i.e., hold t constant

s,t u,v

3/7/2001 Vision for Graphics 43

3D Lumigraph

One row of s,t plane• i.e., hold t constant• thus s,u,v• a “row of images”

s

u,v

3/7/2001 Vision for Graphics 44

Concentric Mosaics

Replace “row” with “circle” of images

3/7/2001 Vision for Graphics 45

Concentric Mosaics

3/7/2001 Vision for Graphics 46

Concentric Mosaics

From above

3/7/2001 Vision for Graphics 47

Concentric Mosaics

Depth correction

3/7/2001 Vision for Graphics 48

Concentric Mosaics

Panorama

3/7/2001 Vision for Graphics 49

2.5D Representations

Image is 2DLumigraph is 4D3D

• 3D Lumigraph subset• Concentric mosaics

2.5D• Layered Depth Images• View Dependent Surfaces

3/7/2001 Vision for Graphics 50

Layered Depth Image

ImageDepthLayered

2.5 D ?

3/7/2001 Vision for Graphics 51

Layered Depth Image

Rendering from LDI

• Incremental in LDI X and Y • Guaranteed to be in back-to-front order

3/7/2001 Vision for Graphics 52

Layered Depth Image

Rendering from LDI

• Incremental in LDI X and Y • Guaranteed to be in back-to-front order

3/7/2001 Vision for Graphics 53

Layered Depth Image (LDI)

3/7/2001 Vision for Graphics 58

Summary

5D: Plenoptic Function (Ray)

4D: Lumigraph / Lightfield

3D: Lumigraph Subset

3D: Concentric Mosaics

2.5D: Layered Depth Image

2.5D: Image Based Models

2D: Image