Local Features
Computer Vision
Correspondence Problem & Keypoint
• Correspondence: matching points, patches, edges, or regions across images
≈
Overview of Keypoint Matching
K. Grauman, B. Leibe
Af Bf
A1
A2 A3
Tffd BA ),(
1. Find a set of
distinctive key-
points
3. Extract and
normalize the
region content
2. Define a region
around each
keypoint
4. Compute a local
descriptor from the
normalized region
5. Match local
descriptors
DoG – Efficient Computation
• Computation in Gaussian scale pyramid
K. Grauman, B. Leibe
s
Original image 4
1
2s
Sampling with
step s4 =2
s
s
s
Find local maxima in position-scale space of Difference-of-Gaussian
K. Grauman, B. Leibe
)()( ss yyxx LL
s
s2
s3
s4
s5
List of (x, y, s)
Results: Difference-of-Gaussian
K. Grauman, B. Leibe
SIFT
• Scale-space extrema detection
• Keypoint localization
• Orientation assignment
• Keypoint descriptor
Scale-space extrema detection
• Find the points, whose surrounding patches (with some scale) are distinctive
• An approximation to the scale-normalized Laplacian of Gaussian
Maxima and minima in a
3*3*3 neighborhood
Eliminating edge points
Orientation assignment
• Assign an orientation to each keypoint, the keypoint descriptor can be represented relative to this orientation and therefore achieve invariance to image rotation
• Compute magnitude and orientation on the Gaussian smoothed images
T. Tuytelaars, B. Leibe
Orientation normalization
• Compute orientation histogram
• Select dominant orientation
• Normalize: rotate to fixed orientation
0 2 p
Reduce effect of illumination • 128-dim vector normalized to 1
• Threshold gradient magnitudes to avoid excessive influence of high gradients
– after normalization, clamp gradients >0.2
– renormalize
A result
Structure from Motion
Computer Vision
Depth from disparity
x’ z
f
x
C C’
X
baseline
f
(X – X’) / f = baseline / z X – X’ = (baseline*f) / z z = (baseline*f) / (X – X’)
General case, with calibrated cameras
• The two cameras need not have parallel optical axes.
Vs.
• Given p in left image, where can corresponding point p’ be?
Stereo correspondence constraints
Stereo correspondence constraints
Geometry of two views constrains where the
corresponding pixel for some image point in the first view
must occur in the second view.
• It must be on the line carved out by a plane
connecting the world point and optical centers.
Epipolar constraint
• Epipolar Plane
Epipole
Epipolar Line
Baseline
Epipolar geometry
Epipole
• Baseline: line joining the camera centers
• Epipole: point of intersection of baseline with image plane
• Epipolar plane: plane containing baseline and world point
• Epipolar line: intersection of epipolar plane with the image plane
• All epipolar lines intersect at the epipole
• An epipolar plane intersects the left and right image planes in epipolar lines
Epipolar geometry: terms
Why is the epipolar constraint useful?
Epipolar constraint
This is useful because it reduces the correspondence
problem to a 1D search along an epipolar line.
Image from Andrew Zisserman
Example
What do the epipolar lines look like?
Ol Or
Ol Or
1.
2.
Example: converging cameras
Figure from Hartley & Zisserman
Figure from Hartley & Zisserman
Example: parallel cameras
Where are the
epipoles?
Example: Forward motion
What would the epipolar lines look like if the camera moves directly forward?
e
e’
Example: Forward motion
Epipole has same coordinates in both
images.
Points move along lines radiating from e:
“Focus of expansion”
Fundamental matrix
• Let p be a point in left image, p’ in right image
• Epipolar relation – p maps to epipolar line l’ – p’ maps to epipolar line l
• Epipolar mapping described by a 3x3 matrix F
• It follows that
l’ l
p p’
Fundamental matrix
• This matrix F is called – the “Essential Matrix”
• when image intrinsic parameters are known
– the “Fundamental Matrix” • more generally (uncalibrated case)
• Can solve for F from point correspondences
– Each (p, p’) pair gives one linear equation in entries of F
– F has 9 entries, but really only 7 or 8 degrees of freedom. – With 8 points it is simple to solve for F, but it is also
possible with 7. See Marc Pollefey’s notes for a nice tutorial
Active stereo with structured light
• Project “structured” light patterns onto the object – Simplifies the correspondence problem
– Allows us to use only one camera
camera
projector
L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisition Using Color Structured
Light and Multi-pass Dynamic Programming. 3DPVT 2002
Structured light
https://www.youtube.com/watch?v=jmffVdU
JYv4
Kinect: Structured infrared light
http://bbzippo.wordpress.com/2010/11/28/kinect-in-infrared/
Photometry stereo
https://www.youtube.com/watch?v=qlq3n5r1Xy0
Helmholtz Stereopsis
Structure from Motion
Structure from motion
• Given a set of corresponding points in two or more images, compute the camera parameters and the 3D point coordinates
Camera 1 Camera 2 Camera 3
R1,t1 R2,t2 R3,t3
? ? ? Slide credit:
Noah Snavely
?
https://www.youtube.com/watch?v=i7ierVk
XYa8
Structure from motion ambiguity
• If we scale the entire scene by some factor k and, at the same time, scale the camera matrices by the factor of 1/k, the projections of the scene points in the image remain exactly the same:
It is impossible to recover the absolute scale of the scene!
)(1
XPPXx kk
How do we know the scale of image content?
Bundle adjustment
• Non-linear method for refining structure and motion
• Minimizing reprojection error
2
1 1
,),(
m
i
n
j
jiijDE XPxXP
x1j
x2j
x3j
Xj
P1
P2
P3
P1Xj
P2Xj
P3Xj
Photo synth
Noah Snavely, Steven M. Seitz, Richard Szeliski, "Photo tourism: Exploring
photo collections in 3D," SIGGRAPH 2006
https://www.youtube.com/watch?v=IgBQC
oEfiMs