3d surface reconstruction from 2d images (survey) 2006. 11. 3 (fri) young ki baik, computer vision...

33
3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab.

Upload: magnus-johnson

Post on 11-Jan-2016

213 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

3D Surface Reconstruction from 2D Images (Survey)

2006. 11. 3 (Fri)Young Ki Baik, Computer Vision Lab.

Page 2: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

2

3D Surface Reconstruction from 2D Images

• References• A Comparison and Evaluation of Multi-View Stereo

Reconstruction Algorithms • Steven M. Seitz, Richard Szeliski et. al. (CVPR 2006)

• A Survey of Methods for Volumetric Scene Reconstruction

• Greg Slabaugh et. al. (VG 2001) : Volume Graphics

Page 3: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

3

3D Surface Reconstruction from 2D Images

• Contents• Introduction

• Camera calibration

• Shape from 2D images, techniques

• Conclusion

Page 4: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

4

3D Surface Reconstruction from 2D Images

• Introduction• Volumetric data representations

• Gaining importance since their introduction in the early 70’s

→ 3D medical imaging [Greenleaf 70]

Page 5: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

5

3D Surface Reconstruction from 2D Images

• Introduction• Volumetric data representation

• The exponential growth of computational storage and processing

→ practical alternatives to surface based geometrical representation for many applications in computer graphics and scientific visualization

Page 6: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

6

3D Surface Reconstruction from 2D Images

• Application• Reverse engineering

• Augmented reality

• Human computer interaction

• Animation, Game

• Etc.

Page 7: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

7

3D Surface Reconstruction from 2D Images

• Methods for volumetric reconstruction• By hand

• 3D tool (3DMAX, MAYA, …)• Limitation

Too much time, tedious

Page 8: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

8

3D Surface Reconstruction from 2D Images

• Methods for volumetric reconstruction• Laser scanner (Range data)

• Advantage High quality and accuracy

• Limitation Too much money Specific configuration

Page 9: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

9

3D Surface Reconstruction from 2D Images

• Methods for volumetric reconstruction• CCD camera (Images)

• Advantage Cheap price Usefulness

Page 10: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

10

3D Surface Reconstruction from 2D Images

• Condition of 3D reconstruction

3D point

Camera

3D object

Camera

mapping

Image plane

Camera system for obtaining images

Page 11: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

11

3D Surface Reconstruction from 2D Images

• Condition of 3D reconstruction

3D point

Camera

3D object

Camera

• Point correspondence

• Camera parameter and motion

3D reconstruction system to make 3D object

Page 12: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

12

3D Surface Reconstruction from 2D Images

• Camera Calibration• Camera parameters:

• Extrinsic: Translation T, Rotation R.• Intrinsic: Focal Length f, image center (ox ,oy),

effective pixel size (sx ,sy), radial distortion k.

• Recover parameters from 3D points and their projections.

Object

Camera Motion

View9

View6

View1

Page 13: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

13

3D Surface Reconstruction from 2D Images

• Camera Calibration• Simple overall flow

• We can acquire 3D volumetric representation by applying various reconstruction algorithm.

• Obtaining camera parameters using projected 2D image and world 3D data with Known plane or 3D rig.

• Camera is set fixed location.

Page 14: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

14

3D Surface Reconstruction from 2D Images

• Camera Calibration• Calibration with pattern:

• Tsai’s method [Tsai87]• Zhang’s method [Zhang00]

• Self-calibration [Maybank92]

• Bundle adjustment [Triggs], evenly distribute errors.

Page 15: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

15

3D Surface Reconstruction from 2D Images

• Shape from 2D Images• Shape from silhouette

• Shape from structured light

• Shape from Illumination• Shape from shading• Photometric stereo

• Shape from Color(or intensity)• Voxel coloring• Stereo vision

• Fusion method

Page 16: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

16

3D Surface Reconstruction from 2D Images

• Shape from Silhouette(SFS) • Early works of vision

• Effective method

• Sculpturing a statue

Page 17: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

17

3D Surface Reconstruction from 2D Images

• Shape from Silhouette(SFS)

O1

O2

O3

- Calibrated cameras and object- Set initial 3D volumetric region including object- Back-project each silhouette along the ray- Obtain 3D volumetric data from intersecting back-projected volume

Page 18: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

18

3D Surface Reconstruction from 2D Images

• Shape from Silhouette(SFS) • Advantages:

• Simple to implement and fairly robust• Fast execution• complete closed surface → commonly used as the effective initial boundary

• Limitations:• only produced line hull• can’t detect non-convex region• sensitive to segmentation result → specific color is used as the background

Page 19: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

19

3D Surface Reconstruction from 2D Images

• Shape from Structured Light• Shape from Structured Light

• Rays coming out of light source hit the object surface and captured by image sensor (usually a camera) in a different angle. [Levoy00, Allen03]

Problem - Optically uncooperative materials - Scanning in the presence of occlusion - Filling holes in dense polygon models

Page 20: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

20

3D Surface Reconstruction from 2D Images

• Shape from Illumination• Shape from Shading

• Assume distance point light source, orthographic projection, local shading and Lambertian surface

• Given image intensity, determine depth by solving reflectance map in the fields of Radiometry.

LambertianA surface point is equally bright from all directions.

Limitation - Do not provide qualified results

Page 21: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

21

3D Surface Reconstruction from 2D Images

• Shape from Illumination• Photometric stereo

• Advanced version of shape from shading• Method to determine surface shape using

multiple images taken by varying illumination direction, while fixed camera position

Advantage- Provide good results relative to shape from shading

Limitation- Have to know the location of light sources

Page 22: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

22

3D Surface Reconstruction from 2D Images

• Shape from Color• Voxel Coloring

• Images can be constraints on 3D scene: a valid 3D scene model projected must produce

synthetic images same as the corresponding real input images.

• SFS+color consistency• Opaque or not

Sees blue Sees blue Sees red Sees green

Page 23: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

23

3D Surface Reconstruction from 2D Images

• Shape from Color• Voxel Coloring (overall flow)

- Set the camera on the fixed location.- Place the object to the fixed location.- Set up voxel region covering object.

Page 24: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

24

3D Surface Reconstruction from 2D Images

• Shape from Color• Voxel Coloring (overall flow)

- Select a voxel and project onto the each image.- Iterate this algorithm about all voxel in the region.- Judge opaqueness by thresholding variance of colors.

2222

222

222

222

)1

(1

1

)1

(1

1

)1

(1

1

BGR

N

i

N

iiiB

N

i

N

iiiG

N

i

N

iiiR

BN

BN

GN

GN

RN

RN

Page 25: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

25

3D Surface Reconstruction from 2D Images

• Voxel Coloring• Advantages:

• simple to implement and fairly robust

• Limitations:• performance depends on voxel and image

resolution. → reconstruct object in small area

→ high computational cost• occlusion and illumination problem

Page 26: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

26

3D Surface Reconstruction from 2D Images

• Voxel coloring• More advanced algorithm

• Space carving• Generalized voxel coloring• Multi-hypothesis voxel coloring

Page 27: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

27

3D Surface Reconstruction from 2D Images

• Shape from color• Stereo vision

• Features popular method pixel based method mimics the behavior of human vision apply feature matching criterion at all pixels

simultaneously search only over epipolar lines (fewer candidate positions)

Scene object point

Left Camera

Left Camera

Optical axes

Epipolar lines

Epipolar plane

Image plane

Page 28: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

28

3D Surface Reconstruction from 2D Images

• Shape from color• Stereo vision

• Matching cost Squared Intensity Differences (SD,SSD). Absolutely Intensity Differences (AD,MSE). Normalized Cross-correlation, normalized SSD.

Page 29: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

29

3D Surface Reconstruction from 2D Images

• Stereo vision• Advantages

• gives detailed surface estimates• covering wide area object

Building, topography • Fast execution• multi-view aggregation improves accuracy

Page 30: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

30

3D Surface Reconstruction from 2D Images

• Stereo vision• Limitation

• narrow baseline give rise to noisy estimates• fails in ~

textureless and occlusion areas sparse, in complete surface

• sensitive to non-Lambertian effects.

• Other effective methods• http://cat.middlebury.edu/stereo/

Page 31: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

31

3D Surface Reconstruction from 2D Images

• Fusion methods

• Silhouette and Stereo Fusion for 3D Object Modeling - Carlos Hernandez Esteban and Francis Schmitt (CVIU 2004)

# SFS +multi +stereo correlation voting +Gradient vector flow +Snake

• High-Fidelity Image-Based Modeling - Yasutaka Furukawa, Jean Ponce (CVR-TR-2006)

# SFS + wide baseline matching + propagation + Energy minimization

• Multi-View Stereo Revisited - Michael Goesele et. al. (CVPR2006)

# SFS +Stereo matching + volumetric method (range data + Level Set)

• MultiView Geometry for Texture mapping 2D Images Onto 3D Range Data

- Lingyoun Liu et. Al. (CVPR2006)# SFS +Stereo matching + volumetric method (range data)

Page 32: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

32

3D Surface Reconstruction from 2D Images

• Trend of 3D reconstruction method

• ~1995 Simple algorithm

• ~1999 VC and variants (treating occlusion), LevelSet, optimization

• ~2001 Probabilistic formulation

• ~2003 Non-Lambatian surface, specular surface, textureless regions

• ~2006 PGM, Fusion method (SFS + Stereo + Level set + …)

Page 33: 3D Surface Reconstruction from 2D Images (Survey) 2006. 11. 3 (Fri) Young Ki Baik, Computer Vision Lab

33

3D Surface Reconstruction from 2D Images

• Conclusion• Survey of methods for volumetric scene

reconstruction from photographs.

• States of the arts shows very good reconstruction results.

• All algorithm do not solve problems yet.• occlusion ,illumination changes • non-Lambatian surface• Real data (no silhouette)

• There is room for improvement