image based rendering(ibr) jiao-ying shi state key laboratory of computer aided design and graphics...

Image Based Rendering(IBR)

Jiao-ying ShiState Key laboratory of Computer Aided Design and

Graphics

Zhejiang University, Hangzhou, China

jyshi@cad.zju.edu.cn

http://www.cad.zju.edu.cn/home/jyshi

Survey on Image Based Rendering (IBR)

PART I

Traditional Computer Graphics

Use Geometry and lighting model to simulate the imaging process and generate realistic scene

– No Guarantees for the rightness of the models– A lot of computation time needed

Use of images In Computer Graphics

Texture Mapping Environment map

• How about more images?

Computer vision

Extract Geometry model from real scene(photos)

Combined with Computer Graphics:

Image based renderingBypass the “model”,driectly from real image to s

ynthesized image

Image Based Rendering

Images

Geometry

Images

Computer

vision

analyze

Computer

Graphics:

simulate

Image based

rendering

A Framework of Image Based Rendering

Real

Scene

Sampling System

Data

Storage

System

Data representation

System

Rendering

System

Synthesized view

The Key Part of IBR

The data representation system is the key part of IBR, It determines the other three subsystems.

-A taxonomy based on the data representation system

A Taxonomy of IBR The Geometry based data representation The Image based data representation The plenoptic function based data representation

The Geometry based data representation Geometry elements used as data

representation in IBR:– polyhedra(Debevec, et. al 1996)– layers (Baker, Szeliski and Anandan 1998)– points(Shade et al. 1998)

Similar to Traditional Computer Graphics, except the geometry model comes from images

General working processImage User input

stereo

Geometry

Interactive

modeling

Ranger

3D Warping

Rendering

Image based data representation

data are treated as a series of images with correspondence relations

“optical flow” ”morphing map” “Trifocal/Trilinear tensor” are used to control the generation of novel image

forward/ reverse mapping;morphing

Examples: View interpolation (Chen and William,1993)

View Morphing(Seitz and Dyer 1998)

General working process

Images User input

Correspondence relations

Existing geometry model

(Synthetic images only)

Stereo

2D image warping

rendering

Plenoptic function based data representation Plenoptic function (Adelson and Bergen,1991)

),,,,,,( tVVVPlenoptic zyx

General working process

images

image

processingstereo

resampling

rendering

plenoptic function

User Input

Representative IBR methods based on Plenoptic Functions

Plenoptic Modeling: 5D Light field/Lumigraph: 4D Concentric Mosaics : 3D Panorama: 2D

IBR

The Geometry based data

representation

The Geometry based data

representation

The Plenoptic function based

data representation

IBR data are composed of

geometry elements

IBR data are composed of a

set of images with

correspondence relations

IBR data are composed of

a set of light rays

polyhedra

layers

points

View interpolation[CW93]

View morphing[SD96]]

Transfer mode [LF94]

Plenoptic

modeling[MB95]

Lightgield [LH96]/

Lumigraph[GGSC96]

Concentric

Mosaics[SH99]

Panorama[Chen95],[SS97]

MCOP images[RB98],

LDI[SGHS98]

Depth based [BSA98]、Motion based [LS97]、TIP [HAA97] etc.

Hybrid approach of

geometry and image

[DTM96]

5D plenoptic

function

4D plenoptic

function

3D plenoptic

function

2D plenoptic

function

Conclusion

The progress of IBR technique is also the progress of new data representation method, We treat an image:

– as texture in geometry texture mapping– as images with correspondence relation

view interpolation /morphing– as light beams light field– as slit image concentric mosaics

...

The study on slit images in image based rendering

PART II

The concept of slit images

The slit image is a kind of 1-D image with width only 1 pixel.

An example of slit image

Previous work based on slit images

MCOP images concentric mosaics

The advantage of using slit images

Most computer graphics technology is used to simulate human motion and observing usually only in 3 DOF:

The walk through task in virtual reality applications requires human motion only in 3 DOF:

Left/right, forward/backward and look around.

The representation of slit images

A slit image is identified by the camera 2D position and orientation (azimuth angle)

in polar coordinates

in Cartesian coordinates

S(x,y,θ)

φ

ρ θ

),,( S

(x,y)

θ

Slit image sets(I)

A scene view at position (ρv, φv), with azimuth θv and horizontal FOV ω:

Sv=

Panorama at position (ρp, φp)

Sp =

2/2/,,|),,( vvvvS

ppS ,|),,(

Slit image sets(II) Concentric mosaic with its center at origin ：

Sc ={S(ρ, φ,θ)|θ=-π/2 orθ=π/2 ， ρ≤R}

– (camera alone normal direction)Scn ={S(ρ, φ,θ)|-ω /2<θ< ω/2 ， ρ=R}

– (camera alone tangential direction)Sct ={S(ρ, φ,θ)|-ω /2 ＋ π/2 <θ< ω/2 ＋ π/2 ， ρ=R}

moving straight forward from origin, with horizontal FOV ω

{S(x ， y,θ)|y=0, x>0, -ω /2<θ< ω/2 }

Slit image field

Slit images that captured at any position and any azimuth inside a 2D region.– Inside a circle:

{S(ρ, φ ， θ)|ρ≤R}– Inside a rectangle:

{S(x, y,θ)| x 1≤x≤ x 2 , y1≤y≤ y2}

From the slit image field we can generate the walk-through scenes inside th region just by resampling

Analogical Slit Images

f

hdhn

dddn

H

Cn Cd

f

M

O

md

mn

on od

Object

in scene

hr

dr

Cr

for

mr

Relations between analogical slit images

rcr

crr

dcd

cdd

d

Hf

Z

fYy

d

Hf

Z

fYy

Let hd=| yd| ， hr=| yr|

or

d

r

r

d

d

d

y

y

d

r

r

d

d

d

h

h

r

rd

d

dr

d

dd

h

hh

and

let and

– analogical slit images are highly coherent– slit images can be synthesized by their analogical slit image

Relations between analogical Slit Images

)1)1/(( rd

nddn offset

offsetkhh

d

r

r

dd

d

h

h

d

n

n

dd

d

h

h

dnnd ddoffset drrd ddoffset rd hhk /

Analogical relation of slit images

LR

S1

S2

Analogical relation of slit images is

– reflexive

S1 ~S1

– symmetric

if S1 ~S2 ， there will be S2 ~S1

– transitive

if S1 ~S2 、 S2 ~S3 ， there will be S1 ~S3

Written as S1 ~S2

So analogical relation of slit image is an equivalence relation

Analogical slit image set

Slit images that are analogical each other are consisted to be a analogical slit image set.

Analogical relation is a kind of equivalence relation

an analogical slit image set is a partition of slit image field

A slit image field can be obtained approximately by limited sampling Each analogical slit image set can be approximated by one or a

few its member silt images The set of slit image sets can cover the slit image field. A slit image field can be approximated by limited sampling

Depth correction for Concentric Mosaics

-A slit image segments based approach

Application of analogical slit images

Motivation

In concentric mosaics, only one slit image is captured for every analogical slit image set. And this slit image is simply used as substitution for all its analogical slit images. Distortion caused

find the pixel relations between analogical slit images and correct the distortion of images

Slit image segments

Definition: a slit image segment consists of a series of adjacent pixels in one slit image which have either similar color or similar depth. Segment is used as primitive of image.

Applications: use segment mapping instead of pixel mapping between analogical slit images

Advantages:

– Reduce big amount of data.– Segment is used as basic block in VQ compress

ion

How to segment slit images

Analyze 2 analogical slit images Initial segment

– Find edge point of slit image

Warp slit image segment of one slit image to its analogical slit image, find the best segmentation and correspondence relations between two slit images.

Data slit image and reference slit image

In the 2 slit images:– one is used to synthesize novel view, called

data slit image.– The other is used to find the best segmentation

and define the segment mapping of data slit image, called reference slit image.

Implementation

Capturing Slit Images using normal camera Calibration between concentric mosaics Slit Image Segments Matching Synthesizing novel view

Capturing slit images using normal camera

'

ω /2

R’

Rω /2

R’

R

inward Setup outward Setup

Capturing two set of Concentric Mosaics

θ d

θ r

O

Pr(Rr,φ r)Pd(Rd,φ d)

φ r=φ d=0

θ r

θ d

Pd(R,φ d)

Pr(R,φ r)

φ r=φ d=0

a)Same direction setup b)Opposite direction setup

Possible errors

Δ φ

O φ r=φ d=0

Δ θ

eRd

O

Lead to wrong “analogical” slit images

Calibration between concentric mosaics

– Estimate the errors parameters so that we can find the correct analogical slit images.

smallΔθ is treated asΔφ for simplification. Only consider the relative error e of R 。

Calibration between concentric mosaics Method:– analogical slit images should be alike– select a set of slit images in one CM, calculate their analogical slit images in another CM with the

consideration of introduced error parameters.

Sadjis the set of slit images in one CM for calibration use

Conform() is a likelihood measurement between data and reference slit images.

S

RSReRReRRSconformadjdS

ddddrdddrrdddrrre

)),,()),,,,,,(),,,,,,(,((max,

Calibration in the Same Direction Setup

Two error parameters notice when |θd| is small, e has only small effect to θr andφr .

De-coupling: Select the slit images with small |θd|, estimate Δφ, then estimate e.

))1(

)(arcsin(sinr

ddr

drdr

R

Re

Calibration in the Opposite Direction Setup

dr

drdr

Only need to estimate 1 error parameter

Preprocessing

Edge detection inside slit image: find the initial segment

warp the initial segments to its analogical slit images, find the best segmentation and correspondence relations between two analogical slit image.

Generate corrected image

θ n

θ d

O

Pd(Rd,φ d)Pn(ρ n,φ n)

φ d=0

2/2/,,|),,( vvvvS

||

)sin()sin(sin

nd

dn

d

n

n

d

nddn

PPR

Desired Slit image Set: )1)1/(( rd

nddn offset

offsetkhh

Generate images from known slit image segment relations

between analogical slit images

Result

Panoramic mosaics of slit images with depth

Panorama Method (Chen, 1995)

Only several picture captured at a viewpoint needed, small data size and easy to sampling.

The only off-the-shelf IBR technological for large scene althoughalthough

Fixed viewpoint, can only look around and zoom in / zoom out, or hop between viewpoints

Data size Vs. Motion range in IBR

Small data size very limited DOF of virtual

camera more DOF huge data size

of virtual camera

Slit images with depth

Assume a uniform depth value is

used for every slit image

Panoramaic mosaics of Slit image with depth

recover or assign depth

recover depth from correspondence relations between analogical slit images– search correspondence points– interactive assign correspondence points

recover depth

Depth may be got from a known map

10001

100 ddh

hh

ddd

Interactive Rendering: Finding Slit Images

φ s

φ n

λ

ρ n

ρ sθ

Novel

viewpoint

Slit images with

depth

dn

O

2/2/,,|),,( vvvvS

sn

n

sn

ns d

sin)sin()sin(

Interactive Rendering: Adjusting

Looming effects simulation– scale slit images uniformly

fill holes– fill holes using nearby slit images

s

n

n

s

d

d

l

l

Sample multiple panoramic mosaics of slit image with depth

Join multiple mosaics together

Join multiple mosaics together to achieve a wider motion range of virtual point

Specify reference points

reference circle

reference point

Map slit images to reference point

reference circle reference point

slit images with united depth

Generate novel view

reference

point

virtual camera

postion

Implementation

Sampling– capture slit images– recover or assign depth

Preprocessing– mapping slit images to reference circle

Interactive Rendering

Synthesized view

Move forward and

backward

Move left and right

Advantages and Disadvantages

Advantages– 3 DOF (move left and right, forward and

backward, look around)for the virtual camera with small data size

– multiple mosaics can be joined up smoothly

Disadvantages– only fit for those scene depth variation is small

along the vertical direction scene or for open scene

Conclusion

For 3 DOF motion, slit image is a good data representation for scene

We studied the slit image proprieties and introduced the following 3 concepts:– analogical slit images– analogical slit image set– slit image field

Conclusion

Applications of slit image concepts– Use of slit image segments to correct vertical

distortion of concentric mosaics– A new IBR method: panoramic mosaics of slit

image with depth