computational photography light field rendering

Computational PhotographyLight Field Rendering

Jinxiang Chai

Image-based Modeling: Challenging Scenes

Why will they produce poor results?

- lack of discernible features

- occlusions

- difficult to capture high-level structure

- illumination changes

- specular surfaces

Some Solutions

- Use priors to constrain the modeling space

- Aid modeling process with minimal user interaction

- Combine image-based modeling with other modeling approaches

Videos

Morphable face (click here)

Image-based tree modeling (click here)

Video trace (click here)

3D modeling by ortho-images (Click here)

Spectrum of IBMR

Images user input range

scans

Model

Images

Image based modeling

Image-based renderingGeometry+ Images

Light field

Images + Depth

Geometry+ Materials

Panoroma

Kinematics

Dynamics

Etc.

Camera + geometry

Outline

Light field rendering [Levoy and Hanranhan SIG96]

3D light field (concentric mosaics) [Shum and He Sig99]

Plenoptic Function

Can reconstruct every possible view, at every moment, from every position, at every wavelength

Contains every photograph, every movie, everything that anyone has ever seen! it completely captures our visual reality!

An image is a 2D sample of plenoptic function!

P(x,y,z,θ,φ,λ,t)

Ray

Let’s not worry about time and color:

5D• 3D position

• 2D direction

P(x,y,z,)

Static object Camera

No Change in

Radiance

Static Lighting

How can we use this?

Static object Camera

No Change in

Radiance

Static Lighting

How can we use this?

Ray Reuse

Infinite line• Assume light is constant (vacuum)

4D• 2D direction

• 2D position

• non-dispersive medium

Slide by Rick Szeliski and Michael Cohen

Only need plenoptic surface

Synthesizing novel views

Assume we capture every ray in 3D space!

Synthesizing novel views

Light field / Lumigraph

Outside convex space

4DStuff

Empty

Light Field

How to represent rays?

How to capture rays?

How to use captured rays for rendering

Light Field

How to represent rays?

How to capture rays?

How to use captured rays for rendering

Light field - Organization

2D position

2D direction

s

Light field - Organization

2D position

2D position

2 plane parameterization

su

Light field - Organization

2D position

2D position

2 plane parameterization

us

t s,tu,v

v

s,t

u,v

Light field - Organization

Hold u,v constant

Let s,t vary

What do we get?

s,tu,v

Lumigraph - Organization

Hold s,t constant

Let u,v vary

An image

s,tu,v

Lightfield / Lumigraph

Light field/lumigraph - Capture

Idea 1• Move camera carefully over u,v

plane

• Gantry> see Light field paper

s,tu,v

Stanford multi-camera array

640 × 480 pixels ×30 fps × 128 cameras

synchronized timing

continuous streaming

flexible arrangement

For each output pixel• determine s,t,u,v

• either• use closest discrete RGB• interpolate near values

s u

Light field/lumigraph - rendering

Light field/lumigraph - rendering

Nearest• closest s

• closest u

• draw it

Blend 16 nearest• quadrilinear interpolation

s u

Ray interpolation

s u

Nearest neighbor

Linear interpolation in S-T

Quadrilinear interpolation

Image Plane

Camera Plane

Light FieldLight Field CaptureCapture RenderingRendering

Light Field/Lumigraph Rendering

Light fields

Advantages:• No geometry needed

• Simpler computation vs. traditional CG

• Cost independent of scene complexity

• Cost independent of material properties and other optical effects

Disadvantages:• Static geometry

• Fixed lighting

• High storage cost

3D plenoptic function

Image is 2D

Light field/lumigraph is 4D

What happens to 3D?

- 3D light field subset

- Concentric mosaic [Shum and He]

3D light field

One row of s,t plane• i.e., hold t constant

s,t u,v

3D light field

One row of s,t plane• i.e., hold t constant

• thus s,u,v

• a “row of images”

s

u,v

Concentric mosaics [Shum and He]

Polar coordinate system:

- hold r constant

- thus (θ,u,v)

Concentric mosaics

Why concentric mosaic?

- easy to capture

- relatively small in storage size

Concentric mosaics

From above

How to captured images?

Concentric mosaics

From above

How to render a new image?

Concentric mosaics

From above

How to render a new image?

- for each ray, retrieval the closest captured rays

Concentric mosaics

From above

How to render a new image?

- for each ray, retrieval the closest captured rays

Concentric mosaics

From above

How to render a new image?

- for each ray, retrieval the closest captured rays

Concentric mosaics

From above object

How to retrieval the closest rays?

Concentric mosaics

From above object (s,t) interpolation plane

How to retrieve the closest rays?

Concentric mosaics

From above object (s,t) interpolation plane

How to retrieve the closest rays?

Concentric mosaics

From above object (s,t) interpolation plane

How to retrieve the closest rays?

Concentric mosaics

From above object (s,t) interpolation plane

How to retrieve the closest rays?

Concentric mosaics

From above object (s,t) interpolation plane

How to synthesize the color of rays?

Concentric mosaics

From above object (s,t) interpolation plane

How to synthesize the color of rays? - bilinear interpolation

Concentric mosaics

From above

Concentric mosaics

From above

Concentric mosaics

What are limitations?

Concentric mosaics

What are limitations? - limited rendering region?

- large vertical distortion