Download - Structure from images
![Page 1: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/1.jpg)
Structure from images
![Page 2: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/2.jpg)
Calibration
![Page 3: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/3.jpg)
Review: Pinhole Camera
![Page 4: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/4.jpg)
Review: Perspective Projection
![Page 5: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/5.jpg)
Review: Perspective Projection
Points go to Points Lines go to Lines Planes go to whole
image or Half-planes
Polygons go to Polygons
![Page 6: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/6.jpg)
Review: Intrinsic Camera Parameters
X
Y
Z C
Image plane
Focal plane
M
m
CCC ZYX ,,
u
v
i
j
I
JJkjIki
v
u
II vu ,
fvu CC ,,
101000000
0
0
C
C
C
v
unew
new
ZYX
vfuf
SVU
vv
uu
fkffkf
![Page 7: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/7.jpg)
Review: Extrinsic Parameters
X
Y
Z C
Image plane
Focal plane
M
m
CCC ZYX ,,
u
v
i
j
I
JJkjIki
v
u
II vu ,
fvu CC ,,Z
Y
X
WWW ZYX ,,
By Rigid Body Transformation:
WC
W
W
W
C
C
C
DMMZYX
TRZYX
110
131
1333
![Page 8: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/8.jpg)
Alper Yilmaz, CAP5415, Fall 2004
8
Estimating Camera Parameters
11, yx 111 ,, ZYX
222 ,, ZYX 333 ,, ZYX
NNN ZYX ,,
22 , yx 33 , yx
NN yx ,
![Page 9: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/9.jpg)
Shape From Images
![Page 10: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/10.jpg)
Perspective cues
![Page 11: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/11.jpg)
Perspective cues
![Page 12: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/12.jpg)
Perspective cues
![Page 13: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/13.jpg)
Ames Room
![Page 14: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/14.jpg)
Ames Room
Video
![Page 15: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/15.jpg)
Recovering 3D from images What cues in the image provide 3D
information?
![Page 16: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/16.jpg)
Shading
Visual cues
Merle Norman Cosmetics, Los Angeles
![Page 17: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/17.jpg)
Visual cues Shading
Texture
The Visual Cliff, by William Vandivert, 1960
![Page 18: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/18.jpg)
Visual cues
From The Art of Photography, Canon
Shading
Texture
Focus
![Page 19: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/19.jpg)
Visual cues Shading
Texture
Focus
Motion
![Page 20: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/20.jpg)
Julesz: had huge impact because it showed that recognition not needed for stereo.
![Page 21: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/21.jpg)
Shape From Multiple Views
![Page 22: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/22.jpg)
Multi-View GeometryRelates
• 3D World Points
• Camera Centers
• Camera Orientations
![Page 23: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/23.jpg)
Multi-View GeometryRelates
• 3D World Points
• Camera Centers
• Camera Orientations
• Camera Intrinsic Parameters
• Image Points
![Page 24: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/24.jpg)
Stereoscene point
optical center
image plane
![Page 25: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/25.jpg)
Stereo
Basic Principle: Triangulation• Gives reconstruction as intersection of two rays
• Requires – calibration– point correspondence
![Page 26: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/26.jpg)
Stereo Constraints
p p’ ?
Given p in left image, where can the corresponding point p’in right image be?
![Page 27: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/27.jpg)
Stereo Constraints
X1
Y1
Z1O1
Image plane
Focal plane
M
p p’Y2
X2
Z2O2
Epipolar Line
Epipole
![Page 28: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/28.jpg)
Epipolar Constraint
![Page 29: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/29.jpg)
From Geometry to Algebra
O O’
P
pp’
![Page 30: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/30.jpg)
From Geometry to Algebra
O O’
P
pp’
![Page 31: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/31.jpg)
Linear Constraint:Should be able to express as matrix multiplication.
![Page 32: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/32.jpg)
The Essential Matrix
![Page 33: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/33.jpg)
Correspondence
![Page 34: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/34.jpg)
Pin Hole Camera Model
ZXfx
![Page 35: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/35.jpg)
Basic Stereo Derivations
Derive expression for Z as a function of x1, x2, f and B
![Page 36: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/36.jpg)
Basic Stereo Derivations
ZXfx 1 Z
BfxZ
BXfx
12
21 xxfBZ
![Page 37: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/37.jpg)
Basic Stereo Derivations
Disparity: 21 xxd
dfBZ
![Page 38: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/38.jpg)
We can always achieve this geometry with image rectification
Image Reprojection reproject image planes onto
common plane parallel to line between optical centers (Seitz)
![Page 39: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/39.jpg)
Rectification example
![Page 40: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/40.jpg)
Correspondence: Epipolar constraint.
![Page 41: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/41.jpg)
Correspondence Problem Two classes of algorithms:
Correlation-based algorithms Produce a DENSE set of correspondences
Feature-based algorithms Produce a SPARSE set of correspondences
![Page 42: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/42.jpg)
Correspondence: Photometric constraint Same world point has same intensity in
both images. Lambertian fronto-parallel Issues:
Noise Specularity Foreshortening
![Page 43: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/43.jpg)
Using these constraints we can use matching for stereo
For each epipolar lineFor each pixel in the left image
• compare with every pixel on same epipolar line in right image• pick pixel with minimum match cost• This will never work, so:
Improvement: match windows
![Page 44: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/44.jpg)
Comparing Windows: =?
f g
Mostpopular
For each window, match to closest window on epipolar line in other image.
![Page 45: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/45.jpg)
It is closely related to the SSD:
Maximize Cross correlation
Minimize Sum of Squared Differences
![Page 46: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/46.jpg)
Matching cost
disparity
Left Right
scanline
Correspondence search
• Slide a window along the right scanline and compare contents of that window with the reference window in the left image
• Matching cost: SSD or normalized correlation
![Page 47: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/47.jpg)
Left Right
scanline
Correspondence search
SSD
![Page 48: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/48.jpg)
Left Right
scanline
Correspondence search
Norm. corr
![Page 49: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/49.jpg)
Effect of window size
W = 3 W = 20
• Smaller window+ More detail– More noise
• Larger window+ Smoother disparity maps– Less detail– Fails near boundaries
![Page 50: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/50.jpg)
Stereo results
Ground truthScene
Data from University of Tsukuba
(Seitz)
![Page 51: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/51.jpg)
Results with window correlation
Window-based matching(best window size)
Ground truth
(Seitz)
![Page 52: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/52.jpg)
Results with better method
State of the art methodBoykov et al., Fast Approximate Energy Minimization via Graph Cuts,
International Conference on Computer Vision, September 1999.
Ground truth
(Seitz)
![Page 53: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/53.jpg)
Failures of correspondence search
Textureless surfaces Occlusions, repetition
Non-Lambertian surfaces, specularities
![Page 54: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/54.jpg)
How can we improve window-based matching?
So far, matches are independent for each point
What constraints or priors can we add?
![Page 55: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/55.jpg)
Stereo constraints/priors• Uniqueness
For any point in one image, there should be at most one matching point in the other image
![Page 56: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/56.jpg)
Stereo constraints/priors• Uniqueness
For any point in one image, there should be at most one matching point in the other image
• Ordering Corresponding points should be in the same order in both views
Ordering constraint doesn’t hold
![Page 57: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/57.jpg)
Priors and constraints
• Uniqueness For any point in one image, there should be at most one
matching point in the other image• Ordering
Corresponding points should be in the same order in both views
• Smoothness We expect disparity values to change slowly (for the
most part)
![Page 58: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/58.jpg)
Stereo matching as energy minimizationI1 I2 D
• Energy functions of this form can be minimized using graph cutsY. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 2001
W1(i ) W2(i+D(i )) D(i )
)(),;( smooth21data DEIIDEE 2
,neighborssmooth )()(
ji
jDiDE 221data ))(()(
i
iDiWiWE
![Page 59: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/59.jpg)
Examples
bread toy apple
![Page 60: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/60.jpg)
Szeliski
Active stereo with structured light
Project “structured” light patterns onto the object simplifies the correspondence problem
camera 2
camera 1
projector
camera 1
projector
Li Zhang’s one-shot stereo
![Page 61: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/61.jpg)
Active stereo with structured light
L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisition Using Color Structured Light and Multi-pass Dynamic Programming. 3DPVT 2002
![Page 62: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/62.jpg)
Kinect
https://www.youtube.com/watch?v=dTKlNGSH9Po
![Page 63: Structure from images](https://reader035.vdocuments.us/reader035/viewer/2022062515/568168f2550346895ddffa3d/html5/thumbnails/63.jpg)
The third view can be used for verification
Beyond two-view stereo