multi-view geometry. multi-view geometry problems structure: given projections of the same 3d point...
TRANSCRIPT
![Page 1: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/1.jpg)
Multi-view geometry
![Page 2: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/2.jpg)
Multi-view geometry problems• Structure: Given projections of the same 3D point in two
or more images, compute the 3D coordinates of that point
Camera 3
R3,t3 Slide credit: Noah Snavely
?
Camera 1Camera 2R1,t1 R2,t2
![Page 3: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/3.jpg)
Multi-view geometry problems• Stereo correspondence: Given a point in one of the
images, where could its corresponding points be in the other images?
Camera 3
R3,t3
Camera 1Camera 2R1,t1 R2,t2
Slide credit: Noah Snavely
![Page 4: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/4.jpg)
Multi-view geometry problems• Motion: Given a set of corresponding points in two or
more images, compute the camera parameters
Camera 1Camera 2 Camera 3
R1,t1 R2,t2R3,t3
? ? ? Slide credit: Noah Snavely
![Page 5: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/5.jpg)
Structure: Triangulation• Given projections of a 3D point in two or more
images (with known camera matrices), find the coordinates of the point
O1O2
x1x2
X?
![Page 6: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/6.jpg)
Structure: Triangulation• We want to intersect the two visual rays
corresponding to x1 and x2, but because of noise and numerical errors, they don’t meet exactly
O1O2
x1x2
X?
![Page 7: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/7.jpg)
Triangulation: Geometric approach• Find shortest segment connecting the two
viewing rays and let X be the midpoint of that segment
O1O2
x1x2
X
![Page 8: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/8.jpg)
Triangulation: Linear approach
baba ][
0
0
0
z
y
x
xy
xz
yz
b
b
b
aa
aa
aa
XPx
XPx
22
11
2
1
0XPx
0XPx
22
11
0X]P[x
0X]P[x
22
11
Cross product as matrix multiplication:
![Page 9: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/9.jpg)
Triangulation: Linear approach
XPx
XPx
22
11
2
1
0XPx
0XPx
22
11
0X]P[x
0X]P[x
22
11
Two independent equations each in terms of three unknown entries of X
![Page 10: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/10.jpg)
Triangulation: Nonlinear approach
Find X that minimizes
O1O2
x1x2
X?
P1X
)()( 22 XP,xXP,x 2211 dd
P2X
![Page 11: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/11.jpg)
Two-view geometry
![Page 12: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/12.jpg)
• Epipolar Plane – plane containing baseline (1D family)
• Epipoles = intersections of baseline with image planes = projections of the other camera center= vanishing points of the baseline (motion direction)
• Baseline – line connecting the two camera centers
Epipolar geometryX
x x’
![Page 13: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/13.jpg)
The Epipole
Photo by Frank Dellaert
![Page 14: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/14.jpg)
• Epipolar Plane – plane containing baseline (1D family)
• Epipoles = intersections of baseline with image planes = projections of the other camera center= vanishing points of the baseline (motion direction)• Epipolar Lines - intersections of epipolar plane with image
planes (always come in corresponding pairs)
• Baseline – line connecting the two camera centers
Epipolar geometryX
x x’
![Page 15: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/15.jpg)
Example: Converging cameras
![Page 16: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/16.jpg)
Example: Motion parallel to image plane
![Page 17: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/17.jpg)
Example: Motion perpendicular to image plane
![Page 18: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/18.jpg)
Example: Motion perpendicular to image plane
• Points move along lines radiating from the epipole: “focus of expansion”• Epipole is the principal point
![Page 19: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/19.jpg)
Epipolar constraint
• If we observe a point x in one image, where can the corresponding point x’ be in the other image?
x x’
X
![Page 20: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/20.jpg)
• Potential matches for x have to lie on the corresponding epipolar line l’.
• Potential matches for x’ have to lie on the corresponding epipolar line l.
Epipolar constraint
x x’
X
x’
X
x’
X
![Page 21: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/21.jpg)
Epipolar constraint example
![Page 22: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/22.jpg)
X
x x’
Epipolar constraint: Calibrated case
• Assume that the intrinsic and extrinsic parameters of the cameras are known
• We can multiply the projection matrix of each camera (and the image points) by the inverse of the calibration matrix to get normalized image coordinates
• We can also set the global coordinate system to the coordinate system of the first camera. Then the projection matrices of the two cameras can be written as [I | 0] and [R | t]
![Page 23: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/23.jpg)
X
x x’ = Rx+t
Epipolar constraint: Calibrated case
R
t
The vectors Rx, t, and x’ are coplanar
= (x,1)T
![Page 24: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/24.jpg)
Essential Matrix(Longuet-Higgins, 1981)
Epipolar constraint: Calibrated case
0])([ xRtx RtExEx T ][with0
X
x x’
The vectors Rx, t, and x’ are coplanar
![Page 25: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/25.jpg)
X
x x’
Epipolar constraint: Calibrated case
• E x is the epipolar line associated with x (l' = E x)• ETx' is the epipolar line associated with x' (l = ETx')• E e = 0 and ETe' = 0• E is singular (rank two)• E has five degrees of freedom
0])([ xRtx RtExEx T ][with0
![Page 26: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/26.jpg)
Epipolar constraint: Uncalibrated case
• The calibration matrices K and K’ of the two cameras are unknown
• We can write the epipolar constraint in terms of unknown normalized coordinates:
X
x x’
0ˆˆ xEx T xKxxKx ˆˆ,ˆ 11
![Page 27: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/27.jpg)
Epipolar constraint: Uncalibrated case
X
x x’
Fundamental Matrix(Faugeras and Luong, 1992)
0ˆˆ xEx T
xKx
xKx
1
1
ˆ
ˆ
1with0 KEKFxFx TT
![Page 28: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/28.jpg)
Epipolar constraint: Uncalibrated case
• F x is the epipolar line associated with x (l' = F x)• FTx' is the epipolar line associated with x' (l' = FTx')• F e = 0 and FTe' = 0• F is singular (rank two)• F has seven degrees of freedom
X
x x’
0ˆˆ xEx T 1with0 KEKFxFx TT
![Page 29: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/29.jpg)
The eight-point algorithm
Minimize:
under the constraint||F||2=1
2
1
)( i
N
i
Ti xFx
0
1
1
333231
232221
131211
v
u
fff
fff
fff
vu 01
33
32
31
23
22
21
13
12
11
f
f
f
f
f
f
f
f
f
vuvvvuvuvuuu
)1,,(,)1,,( vuvu T xx
![Page 30: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/30.jpg)
The eight-point algorithm
• Meaning of error
sum of squared algebraic distances between points x’i and epipolar lines F xi (or points xi and epipolar lines FTx’i)
• Nonlinear approach: minimize sum of squared geometric distances
:)( 2
1i
N
i
Ti xFx
N
iiiii
1
22 ),(d),(d xFxxFx T
![Page 31: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/31.jpg)
Problem with eight-point algorithm
1
32
31
23
22
21
13
12
11
f
f
f
f
f
f
f
f
vuvvvuvuvuuu
![Page 32: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/32.jpg)
1
32
31
23
22
21
13
12
11
f
f
f
f
f
f
f
f
vuvvvuvuvuuu
Problem with eight-point algorithm
Poor numerical conditioning
Can be fixed by rescaling the data
![Page 33: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/33.jpg)
The normalized eight-point algorithm
• Center the image data at the origin, and scale it so the mean squared distance between the origin and the data points is 2 pixels
• Use the eight-point algorithm to compute F from the normalized points
• Enforce the rank-2 constraint (for example, take SVD of F and throw out the smallest singular value)
• Transform fundamental matrix back to original units: if T and T’ are the normalizing transformations in the two images, than the fundamental matrix in original coordinates is T’T F T
(Hartley, 1995)
![Page 34: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/34.jpg)
Comparison of estimation algorithms
8-point Normalized 8-point Nonlinear least squares
Av. Dist. 1 2.33 pixels 0.92 pixel 0.86 pixel
Av. Dist. 2 2.18 pixels 0.85 pixel 0.80 pixel
![Page 35: Multi-view geometry. Multi-view geometry problems Structure: Given projections of the same 3D point in two or more images, compute the 3D coordinates](https://reader030.vdocuments.us/reader030/viewer/2022032600/56649dbd5503460f94ab0399/html5/thumbnails/35.jpg)
From epipolar geometry to camera calibration
• Estimating the fundamental matrix is known as “weak calibration”
• If we know the calibration matrices of the two cameras, we can estimate the essential matrix: E = K’TFK
• The essential matrix gives us the relative rotation and translation between the cameras, or their extrinsic parameters