computing f and rectification class 14
DESCRIPTION
Computing F and rectification class 14. Multiple View Geometry Comp 290-089 Marc Pollefeys. Multiple View Geometry course schedule (subject to change). Two-view geometry. Epipolar geometry 3D reconstruction F-matrix comp. Structure comp. Epipolar geometry: direct computation. - PowerPoint PPT PresentationTRANSCRIPT
Computing F and rectification
class 14
Multiple View GeometryComp 290-089Marc Pollefeys
Multiple View Geometry course schedule(subject to change)
Jan. 7, 9 Intro & motivation Projective 2D Geometry
Jan. 14, 16
(no class) Projective 2D Geometry
Jan. 21, 23
Projective 3D Geometry (no class)
Jan. 28, 30
Parameter Estimation Parameter Estimation
Feb. 4, 6 Algorithm Evaluation Camera Models
Feb. 11, 13
Camera Calibration Single View Geometry
Feb. 18, 20
Epipolar Geometry 3D reconstruction
Feb. 25, 27
Fund. Matrix Comp. Structure Comp.
Mar. 4, 6 Planes & Homographies Trifocal Tensor
Mar. 18, 20
Three View Reconstruction
Multiple View Geometry
Mar. 25, 27
MultipleView Reconstruction
Bundle adjustment
Apr. 1, 3 Auto-Calibration Papers
Apr. 8, 10
Dynamic SfM Papers
Apr. 15, 17
Cheirality Papers
Apr. 22, 24
Duality Project Demos
Two-view geometry
Epipolar geometry
3D reconstruction
F-matrix comp.
Structure comp.
Epipolar geometry: direct computation
0Fxx'T
0f1''''''
1'''''' 111111111111
nnnnnnnnnnnn yxyyyxyxyxxx
yxyyyxyxyxxx
0)λIFFdet(Fdet)λFFdet( 1-12221
Basic equation
8-point algorithm (normalize!)
7-point algorithm (impose rank 2)
Epipolar geometry: iterative computation
Maximum Likelihood Estimation
i
iiii dd 22 'x̂,x'x̂,x
(= least-squares for Gaussian noise)
0x̂F'x̂ subject to T
22
21
2
2T2
1T
2T
FxFxFx'Fx'
Fxx'
2
221
2
2T2
1T
T
FxFx
1
Fx'Fx'
1Fxx'
i
iiii dFd2T2 x'F,xx,x'
Sampson error (first order approx. to MLE)
Symmetric epipolar error
Automatic computation of F
(i) Interest points(ii) Putative correspondences(iii) RANSAC (iv) Non-linear re-estimation of F(v) Guided matching(repeat (iv) and (v) until stable)
• Extract feature points to relate images
• Required properties:• Well-defined
(i.e. neigboring points should all be different)
• Stable across views(i.e. same 3D point should be extracted as feature for neighboring viewpoints)
Feature points
homogeneous
edge
corner
dxdyyxwyI
xI
W yIxI
,
M
M should have large eigenvalues
(e.g.Harris&Stephens´88; Shi&Tomasi´94)
MTSSD
Find points that differ as much as possible from all neighboring points
Feature = local maxima (subpixel) of F(1, 2)
Feature points
Select strongest features (e.g. 1000/image)
Feature points
Evaluate NCC for all features withsimilar coordinates
Keep mutual best matchesKeep mutual best matches
Still many wrong matches!Still many wrong matches!
10101010 ,,´´, e.g. hhww yyxxyx
?
Feature matching
0.96 -0.40 -0.16 -0.39 0.19
-0.05 0.75 -0.47 0.51 0.72
-0.18 -0.39 0.73 0.15 -0.75
-0.27 0.49 0.16 0.79 0.21
0.08 0.50 -0.45 0.28 0.99
1 5
24
3
1 5
24
3
Gives satisfying results for small image motions
Feature example
• Requirement to cope with larger variations between images• Translation, rotation, scaling• Foreshortening• Non-diffuse reflections• Illumination
geometric transformations
photometric changes
Wide-baseline matching…
Wide baseline matching for two different region types
(Tuytelaars and Van Gool BMVC 2000)
Wide-baseline matching…
Step 1. Extract featuresStep 2. Compute a set of potential matchesStep 3. do
Step 3.1 select minimal sample (i.e. 7 matches)
Step 3.2 compute solution(s) for F
Step 3.3 determine inliers
until (#inliers,#samples)<95%
samples#7)1(1
matches#inliers#
#inliers 90%
80%
70% 60%
50%
#samples
5 13 35 106 382
Step 4. Compute F based on all inliersStep 5. Look for additional matchesStep 6. Refine F based on all correct matches
(generate hypothesis)
(verify hypothesis)
RANSAC
restrict search range to neighborhood of epipolar line (1.5 pixels)
relax disparity restriction (along epipolar line)
Finding more matches
• Degenerate cases• Planar scene• Pure rotation
• No unique solution• Remaining DOF filled by noise• Use simpler model (e.g. homography)
• Model selection (Torr et al., ICCV´98, Kanatani, Akaike)
• Compare H and F according to expected residual error (compensate for model complexity)
Degenerate cases:
Model selection
n = number of measurements (inliers+outliers)r = dimension of datak = motion model parametersd = dimension of structure
structure
motion
MLE
Geometric Robust Information Criterion
Model selection
Video tracking
H
F
Dominant planes
• Absence of sufficient features (no texture)• Repeated structure ambiguity
(Schaffalitzky and Zisserman, BMVC‘98)
• Robust matcher also finds Robust matcher also finds support for wrong hypothesissupport for wrong hypothesis• solution: detect repetition solution: detect repetition
More problems:
geometric relations between two views is fully
described by recovered 3x3 matrix F
two-view geometry
Image pair rectification
simplify stereo matching by warping the images
Apply projective transformation so that epipolar linescorrespond to horizontal scanlines
e
e
map epipole e to (1,0,0)
try to minimize image distortion
problem when epipole in (or close to) the image
Planar rectification
Bring two views Bring two views to standard stereo setupto standard stereo setup
(moves epipole to )(not possible when in/close to image)
~ image size
(calibrated)(calibrated)
Distortion minimization(uncalibrated)
(standard approach)
Polar re-parameterization around epipolesRequires only (oriented) epipolar geometryPreserve length of epipolar linesChoose so that no pixels are compressed
original image rectified image
Polar rectification(Pollefeys et al. ICCV’99)
Works for all relative motionsGuarantees minimal image size
polar rectification: example
polar rectification: example
Example: Béguinage of Leuven
Does not work with standard Homography-based approaches
Example: Béguinage of Leuven
Exploiting motion and scene constraints
• Ordering constraint• Uniqueness constraint• Disparity limit• Disparity continuity constraint
• Epipolar constraint Epipolar constraint (through rectification)
Ordering constraint
11 22 33 4,54,5 66 11 2,32,3 44 55 66
2211 33 4,54,5 6611
2,32,3
44
55
66
surface slicesurface slice surface as a pathsurface as a path
occlusion right
occlusion left
Uniqueness constraint
• In an image pair each pixel has at most one corresponding pixel• In general one corresponding pixel• In case of occlusion there is none
Disparity constraint
surface slicesurface slice surface as a pathsurface as a path
bounding box
dispa
rity b
and
use reconsructed features to determine bounding box
constantdisparitysurfaces
Disparity continuity constraint
• Assume piecewise continuous surface
piecewise continuous disparity• In general disparity changes
continuously• discontinuities at occluding
boundaries
Stereo matching
Optimal path(dynamic programming )
Similarity measure(SSD or NCC)
Constraints• epipolar
• ordering
• uniqueness
• disparity limit
• disparity gradient limit
Trade-off
• Matching cost (data)
• Discontinuities (prior)
(Cox et al. CVGIP’96; Koch’96; Falkenhagen´97; Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)
Hierarchical stereo matching
Dow
nsam
plin
g
(Gau
ssia
n p
yra
mid
)
Dis
pari
ty p
rop
ag
ati
on
Allows faster computation
Deals with large disparity ranges
(Falkenhagen´97;Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)
Disparity map
image I(x,y) image I´(x´,y´)Disparity map D(x,y)
(x´,y´)=(x+D(x,y),y)
Example: reconstruct image from neighboring
images
Multi-view depth fusion
• Compute depth for every pixel of reference image• Triangulation• Use multiple views• Up- and down
sequence• Use Kalman filter
(Koch, Pollefeys and Van Gool. ECCV‘98)
Allows to compute robust texture
Assignment 2Compute F automatically from image
pair
(matches, 7-point, RANSAC, 8-point, iterative,more matches, epipolar lines, etc.)
(due by Wednesday 19/03/03)
Next class: reconstructing points and lines