Transcript
Page 1: Image-based rendering

Image-based rendering

Michael F. CohenMicrosoft Research

Page 2: Image-based rendering

Computer Graphics

Image

Output

ModelSyntheticCamera

Page 3: Image-based rendering

Real Scene

Computer Vision

Real Cameras

Model

Output

Page 4: Image-based rendering

Combined

Model Real Scene

Real Cameras

Image

Output

SyntheticCamera

Page 5: Image-based rendering

But, vision technology falls short

ModelReal Scene

Real Cameras

Image

Output

SyntheticCamera

Page 6: Image-based rendering

… and so does graphics.

ModelReal Scene

Real Cameras

Image

Output

SyntheticCamera

Page 7: Image-based rendering

Image Based Rendering

Real SceneReal Cameras

-or-Expensive Image Synthesis

Images+Model

Image

Output

SyntheticCamera

Page 8: Image-based rendering

Ray

Constant radiance• time is fixed

5D• 3D position• 2D direction

Page 9: Image-based rendering

All Rays

Plenoptic Function• all possible images• too much stuff!

Page 10: Image-based rendering

Line Infinite line

4D• 2D direction• 2D position

Page 11: Image-based rendering

Ray

Discretize

Distance between 2 rays• Which is closer together?

Page 12: Image-based rendering

Image

What is an image?

All rays through a point• Panorama?

Page 13: Image-based rendering

Image

2D • position of rays has been fixed• direction remains

Page 14: Image-based rendering

Image

Image plane

2D• position

Page 15: Image-based rendering

Image plane

2D• position

Image

Page 16: Image-based rendering

Light leaving towards “eye”

2D• just dual of image

Object

Page 17: Image-based rendering

Object

All light leaving object

Page 18: Image-based rendering

Object

4D• 2D position• 2D direction

Page 19: Image-based rendering

Object

All images

Page 20: Image-based rendering

Lumigraph

How to • organize• capture• render

Page 21: Image-based rendering

Lumigraph - Organization

2D position 2D direction

s

Page 22: Image-based rendering

Lumigraph - Organization

2D position 2D position

2 plane parameterization

su

Page 23: Image-based rendering

Lumigraph - Organization

2D position 2D position

2 plane parameterization us

t s,tu,v

v

s,t

u,v

Page 24: Image-based rendering

Lumigraph - Organization

Hold s,t constant Let u,v vary An image

s,t u,v

Page 25: Image-based rendering

Lumigraph - Organization

Discretization• higher res near object

• if diffuse• captures texture

• lower res away• captures directions

s,t u,v

Page 26: Image-based rendering

Lumigraph - Capture

Idea 1• Move camera carefully

over s,t plane• Gantry

• see Lightfield paper

s,t u,v

Page 27: Image-based rendering

Lumigraph - Capture

Idea 2• Move camera anywhere• Rebinning

• see Lumigraph paper

s,t u,v

Page 28: Image-based rendering

Lumigraph - Rendering

For each output pixel• determine s,t,u,v• either

• find closest discrete RGB• interpolate near values

s,t u,v

Page 29: Image-based rendering

Lumigraph - Rendering

For each output pixel• determine s,t,u,v

• either• use closest discrete RGB• interpolate near values s u

Page 30: Image-based rendering

Lumigraph - Rendering

Nearest• closest s• closest u• draw it

Blend 16 nearest• quadrilinear interpolation s u

Page 31: Image-based rendering

High-Quality Video View Interpolation Using a Layered Representation

Larry ZitnickSing Bing KangMatt UyttendaeleSimon WinderRick Szeliski

Interactive Visual Media GroupMicrosoft Research

Page 32: Image-based rendering

Current practice

Many cameras

Motion Jitter

vs.

free viewpoint video

Page 33: Image-based rendering

Current practice

Many cameras

Motion Jitter

vs.

free viewpoint video

Page 34: Image-based rendering

Video view interpolation

Fewer cameras

Smooth Motion

Automatic

and

Real-time rendering

Page 35: Image-based rendering

Prior work: IBR (static)

The LumigraphGortler et al., SIGGRAPH ‘96

Concentric MosaicsShum & He, SIGGRAPH ‘99

Plenoptic ModelingMcMillan & Bishop, SIGGRAPH ‘95

Light Field RenderingLevoy & Hanrahan, SIGGRAPH ‘96

Page 36: Image-based rendering

Prior work: IBR (dynamic)

Free-viewpoint Video of HumansCarranza et al., SIGGRAPH ‘03

Image-Based Visual HullsMatusik et al., SIGGRAPH ‘00

Virtualized RealityTM

Kanade et al., IEEE Multimedia ‘97Dynamic Light Fields

Goldlucke et al., VMV ‘02

Stanford Multi-Camera Array Project

3D TVMatusik & Pfister,

SIGGRAPH ‘04

Page 37: Image-based rendering

System overview

OFFLINE

ONLINE

Video Capture

Stereo Compression

SelectiveDecompression Render

File

Representation

Video Capture

Page 38: Image-based rendering

concentrators

hard disks controlling

laptop

camerascameras

Page 39: Image-based rendering

Calibration

Zhengyou Zhang, 2000

Page 40: Image-based rendering

Input videos

Page 41: Image-based rendering

Key to view interpolation: Geometry

Stereo Geometry

Camera 1 Camera 2

Image 1 Image 2

Virtual Camera

Page 42: Image-based rendering

Match ScoreMatch ScoreMatch Score Match Score

Good

Bad

Image correspondence

Correct

Image 1 Image 2

Leg

Wall

Incorrect

Page 43: Image-based rendering

Image 1 Image 2

Local matching

Low texture

Page 44: Image-based rendering

Number of states = number of depth levels

Image 2Image 1

Global regularizationCreate MRF (Markov Random Field):

A F

E

DC

B

colorA ≈ colorB → zA ≈ zB

Each segment is a nodezA ≈ zP, zQ, zS

P Q R

S T

U

A

A

Page 45: Image-based rendering

Iteratively solve MRF

Page 46: Image-based rendering

Depth through time

Page 47: Image-based rendering

MattingInterpolated view without matting

Background Surface

Foreground Surface

Camera

Foreground Alpha Background

Bayesian MattingChuang et al. 2001

Strip Width

Background

Foreground

Page 48: Image-based rendering

Rendering with matting

MattingNo Matting

Page 49: Image-based rendering

Representation

Main Layer:

Color

Depth

Main

Boundary

Boundary Layer:

Color

Depth

Alpha

Background

ForegroundStrip Width

Page 50: Image-based rendering

“Massive Arabesque” videoclip


Top Related