csce 641 computer graphics: image-based modeling

CSCE 641 Computer Graphics: Image-based Modeling

Jinxiang Chai

Image-based modeling

Estimating 3D structure

Estimating motion, e.g., camera motion

Estimating lighting

Estimating surface model

Traditional modeling and rendering

User input Texture map survey data

Geometry Reflectance Light source

Camera model

Images modeling rendering

For photorealism: - Modeling is hard

- Rendering is slow

Can we model and render this?

What do we want to do for this model?

Image based modeling and rendering

Images user input range

scansModel Images

Image-based modeling

Image-based rendering

Spectrum of IBMR

Images user input range

scans

Model

Images

Image based modeling

Image-based renderingGeometry+ Images

Geometry+ Materials

Images + Depth

Light field

Panoroma

Kinematics

Dynamics

Etc.

Camera + geometry

Stereo reconstruction

Given two or more images of the same scene or object, compute a representation of its shape

What are some possible applications?

knownknowncameracamera

viewpointsviewpoints

3D modeling

From one stereo pair to a 3D head model

[Frederic Deverney, INRIA]

3D modeling

The Digital Michelangelo Project, Levoy et al.

Optical mocap

Vicon mocap system

Z-keying: mix live and synthetic

Takeo Kanade, CMU (Stereo Machine)

Virtualized RealityTM

[Takeo Kanade et al., CMU]• collect video from 50+ stream• reconstruct 3D model sequences

•

• steerable version used forSuperBowl XXV “eye vision”

http://www.cs.cmu.edu/afs/cs/project/VirtualizedR/www/VirtualizedR.html

View interpolation

input depth image novel view[Szeliski & Kang ‘95]

View morphing

Morph between pair of images using epipolar geometry [Seitz & Dyer, SIGGRAPH’96]

Image warping

Video view interpolation

Performance Interface

Microsoft Natal project

Additional applications?

• Real-time people tracking (systems from Pt. Gray Research and SRI)

• “Gaze” correction for video conferencing [Ott,Lewis,Cox InterChi’93]

• Other ideas?

Stereo matching

Given two or more images of the same scene or object, compute a representation of its shape

What are some possible representations for shapes?• depth maps• volumetric models• 3D surface models• planar (or offset) layers

Outline

Stereo matching - Traditional stereo - Multi-baseline stereo - Active stereo

Volumetric stereo - Visual hull - Voxel coloring - Space carving

Stereo matching• Masatoshi Okutomi and Takeo Kanade. A multiple-baseline stereo. IEEE Trans.

on Pattern Analysis and Machine Intelligence (PAMI), 15(4), 1993, pp. 353--363.• D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame

stereo correspondence algorithms.International Journal of Computer Vision, 47(1/2/3):7-42, April-June 2002.

Visual-hull reconstruction• Szeliski, “Rapid Octree Construction from Image Sequences”, Computer Vision,

Graphics, and Image Processing: Image Understanding, 58(1), 1993, pp. 23-32.• Matusik, Buehler, Raskar, McMillan, and Gortler , “Image-Based Visual Hulls”,

Proc. SIGGRAPH 2000, pp. 369-374.

Photo-hull reconstruction• Seitz & Dyer, “Photorealistic Scene Reconstruction by Voxel Coloring”, Intl.

Journal of Computer Vision (IJCV), 1999, 35(2), pp. 151-173. • Kutulakos & Seitz, “A Theory of Shape by Space Carving”, International Journal of

Computer Vision, 2000, 38(3), pp. 199-218.

Papers

Stereo

scene pointscene point

optical centeroptical center

image planeimage plane

Stereo

Basic Principle: Triangulation• Gives reconstruction as intersection of two rays• Requires

> calibration> point correspondence

Camera calibration

From world coordinate to image coordinate

u0

v0

100-sy0

sx auv1

Perspective projection

View transformation

Viewport projection

);( pxwx sCamera parameters3D points2D projections

Stereo correspondence

Determine Pixel Correspondence• Pairs of points that correspond to same scene point

Epipolar Constraint• Reduces correspondence problem to 1D search along conjugate

epipolar lines• Java demo: http://www.ai.sri.com/~luong/research/Meta3DViewer/EpipolarGeo.html

epipolar lineepipolar lineepipolar lineepipolar lineepipolar plane

Stereo image rectification

Stereo image rectification

• reproject image planes onto a commonplane parallel to the line between optical centers

• pixel motion is horizontal after this transformation• two homographies (3x3 transform), one for each

input image reprojection C. Loop and Z. Zhang. Computing Rectifying Homographies

for Stereo Vision. IEEE Conf. Computer Vision and Pattern Recognition, 1999.

http://research.microsoft.com/~zhang/Papers/TR99-21.pdf



Rectification

Original image pairs

Rectified image pairs

Stereo matching algorithms

Match Pixels in Conjugate Epipolar Lines• Assume brightness constancy• This is a tough problem• Numerous approaches

> A good survey and evaluation: http://www.middlebury.edu/stereo/

Your basic stereo algorithm

For each epipolar lineFor each pixel in the left image

• compare with every pixel on same epipolar line in right image

• pick pixel with minimum matching cost

Improvement: match windows• This should look familiar..• Can use Lukas-Kanade or discrete search (latter more common)

Window size

• Smaller window+ -

• Larger window+ -

W = 3 W = 20

Effect of window size

Stereo results

Ground truthScene

• Data from University of Tsukuba• Similar results on other images without ground truth

Results with window search

Window-based matching(best window size)

Ground truth

Better methods exist...

State of the art methodBoykov et al., Fast Approximate Energy Minimization via Graph Cuts,

International Conference on Computer Vision, September 1999.

Ground truth

Stereo reconstruction pipeline

Steps• Calibrate cameras• Rectify images• Compute disparity• Estimate depth

• Camera calibration errors• Poor image resolution• Occlusions• Violations of brightness constancy (specular reflections)• Large motions• Low-contrast image regions

Stereo reconstruction pipeline

Steps• Calibrate cameras• Rectify images• Compute disparity• Estimate depth

What will cause errors?

Outline



Depth from disparity

f

x x’

baseline

z

C C’

X

f

input image (1 of 2) [Szeliski & Kang ‘95]

disparity map 3D rendering

width of a pixel

Choosing the stereo baseline

What’s the optimal baseline?• Too small: large depth error• Too large: difficult search problem

Large BaselineLarge Baseline Small BaselineSmall Baseline

all of thesepoints projectto the same pair of pixels

The effect of baseline on depth estimation

1/z

width of a pixel

width of a pixel

1/z

pixel matching score

Multi-baseline stereo

Basic Approach• Choose a reference view• Use your favorite stereo algorithm BUT

> replace two-view SSD with SSD over all baselines

Limitations• Must choose a reference view (bad)• Visibility!

CMU’s 3D Room Video

Outline



Active stereo with structured light

Project “structured” light patterns onto the object• simplifies the correspondence problem

camera 2

camera 1

projector

camera 1

projector

Li Zhang’s one-shot stereo

Active stereo with structured light

Laser scanning

Optical triangulation• Project a single stripe of laser light• Scan it across the surface of the object• This is a very precise version of structured light scanning

Digital Michelangelo Projecthttp://graphics.stanford.edu/projects/mich/

Laser scanned models

The Digital Michelangelo Project, Levoy et al.

Desktop scanner

Convenient to use Good quality

Relatively low-cost - next engine (about 2k)

csce 641 computer graphics: image-based modeling

Documents

image sequences

image processing

image understanding

stereo pair

stereo matchinggiven

multiplebaseline stereo

pair of images

computer graphics