project 2 results
DESCRIPTION
Project 2 Results. Results Common Problems. 10/17/11. Interest Points: Corners. Computer Vision CS 143, Brown James Hays. Slides from Rick Szeliski , Svetlana Lazebnik , and Kristin Grauman. Feature extraction: Corners. 9300 Harris Corners Pkwy, Charlotte, NC. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/1.jpg)
Project 2 Results
Results
Common Problems
![Page 2: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/2.jpg)
Interest Points: Corners
Computer VisionCS 143, Brown
James Hays
10/17/11
Slides from Rick Szeliski, Svetlana Lazebnik, and Kristin Grauman
![Page 3: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/3.jpg)
Feature extraction: Corners
9300 Harris Corners Pkwy, Charlotte, NC
Slides from Rick Szeliski, Svetlana Lazebnik, and Kristin Grauman
![Page 4: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/4.jpg)
Why extract features?
• Motivation: panorama stitching• We have two images – how do we combine them?
![Page 5: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/5.jpg)
Local features: main components1) Detection: Identify the interest
points
2) Description: Extract vector feature descriptor surrounding each interest point.
3) Matching: Determine correspondence between descriptors in two views
],,[ )1()1(11 dxx x
],,[ )2()2(12 dxx x
Kristen Grauman
![Page 6: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/6.jpg)
Characteristics of good features
• Repeatability• The same feature can be found in several images despite geometric
and photometric transformations
• Saliency• Each feature is distinctive
• Compactness and efficiency• Many fewer features than image pixels
• Locality• A feature occupies a relatively small area of the image; robust to
clutter and occlusion
![Page 7: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/7.jpg)
Goal: interest operator repeatability
• We want to detect (at least some of) the same points in both images.
• Yet we have to be able to run the detection procedure independently per image.
No chance to find true matches!
Kristen Grauman
![Page 8: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/8.jpg)
Goal: descriptor distinctiveness
• We want to be able to reliably determine which point goes with which.
• Must provide some invariance to geometric and photometric differences between the two views.
?
Kristen Grauman
![Page 9: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/9.jpg)
Applications
Feature points are used for:• Image alignment • 3D reconstruction• Motion tracking• Robot navigation• Indexing and database retrieval• Object recognition
![Page 10: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/10.jpg)
Local features: main components1) Detection: Identify the interest
points
2) Description:Extract vector feature descriptor surrounding each interest point.
3) Matching: Determine correspondence between descriptors in two views
![Page 11: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/11.jpg)
Many Existing Detectors Available
K. Grauman, B. Leibe
Hessian & Harris [Beaudet ‘78], [Harris ‘88]Laplacian, DoG [Lindeberg ‘98], [Lowe 1999]Harris-/Hessian-Laplace [Mikolajczyk & Schmid ‘01]Harris-/Hessian-Affine[Mikolajczyk & Schmid ‘04]EBR and IBR [Tuytelaars & Van Gool ‘04] MSER [Matas ‘02]Salient Regions [Kadir & Brady ‘01] Others…
![Page 12: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/12.jpg)
• What points would you choose?
Kristen Grauman
![Page 13: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/13.jpg)
Corner Detection: Basic Idea
• We should easily recognize the point by looking through a small window
• Shifting a window in any direction should give a large change in intensity
“edge”:no change along the edge direction
“corner”:significant change in all directions
“flat” region:no change in all directions
Source: A. Efros
![Page 14: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/14.jpg)
Corner Detection: Mathematics
2
,
( , ) ( , ) ( , ) ( , )x y
E u v w x y I x u y v I x y
Change in appearance of window w(x,y) for the shift [u,v]:
I(x, y)E(u, v)
E(3,2)
w(x, y)
![Page 15: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/15.jpg)
Corner Detection: Mathematics
2
,
( , ) ( , ) ( , ) ( , )x y
E u v w x y I x u y v I x y
I(x, y)E(u, v)
E(0,0)
w(x, y)
Change in appearance of window w(x,y) for the shift [u,v]:
![Page 16: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/16.jpg)
Corner Detection: Mathematics
2
,
( , ) ( , ) ( , ) ( , )x y
E u v w x y I x u y v I x y
IntensityShifted intensity
Window function
orWindow function w(x,y) =
Gaussian1 in window, 0 outside
Source: R. Szeliski
Change in appearance of window w(x,y) for the shift [u,v]:
![Page 17: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/17.jpg)
Corner Detection: Mathematics
2
,
( , ) ( , ) ( , ) ( , )x y
E u v w x y I x u y v I x y
We want to find out how this function behaves for small shifts
Change in appearance of window w(x,y) for the shift [u,v]:
E(u, v)
![Page 18: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/18.jpg)
Corner Detection: Mathematics
2
,
( , ) ( , ) ( , ) ( , )x y
E u v w x y I x u y v I x y
Local quadratic approximation of E(u,v) in the neighborhood of (0,0) is given by the second-order Taylor expansion:
v
u
EE
EEvu
E
EvuEvuE
vvuv
uvuu
v
u
)0,0()0,0(
)0,0()0,0(][
2
1
)0,0(
)0,0(][)0,0(),(
We want to find out how this function behaves for small shifts
Change in appearance of window w(x,y) for the shift [u,v]:
![Page 19: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/19.jpg)
Corner Detection: Mathematics
2
,
( , ) ( , ) ( , ) ( , )x y
E u v w x y I x u y v I x y Second-order Taylor expansion of E(u,v) about (0,0):
v
u
EE
EEvu
E
EvuEvuE
vvuv
uvuu
v
u
)0,0()0,0(
)0,0()0,0(][
2
1
)0,0(
)0,0(][)0,0(),(
),(),(),(),(2
),(),(),(2),(
),(),(),(),(2
),(),(),(2),(
),(),(),(),(2),(
,
,
,
,
,
vyuxIyxIvyuxIyxw
vyuxIvyuxIyxwvuE
vyuxIyxIvyuxIyxw
vyuxIvyuxIyxwvuE
vyuxIyxIvyuxIyxwvuE
xyyx
xyyx
uv
xxyx
xxyx
uu
xyx
u
![Page 20: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/20.jpg)
Corner Detection: Mathematics
2
,
( , ) ( , ) ( , ) ( , )x y
E u v w x y I x u y v I x y Second-order Taylor expansion of E(u,v) about (0,0):
),(),(),(2)0,0(
),(),(),(2)0,0(
),(),(),(2)0,0(
0)0,0(
0)0,0(
0)0,0(
,
,
,
yxIyxIyxwE
yxIyxIyxwE
yxIyxIyxwE
E
E
E
yxyx
uv
yyyx
vv
xxyx
uu
v
u
v
u
EE
EEvu
E
EvuEvuE
vvuv
uvuu
v
u
)0,0()0,0(
)0,0()0,0(][
2
1
)0,0(
)0,0(][)0,0(),(
![Page 21: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/21.jpg)
Corner Detection: Mathematics
2
,
( , ) ( , ) ( , ) ( , )x y
E u v w x y I x u y v I x y Second-order Taylor expansion of E(u,v) about (0,0):
v
u
yxIyxwyxIyxIyxw
yxIyxIyxwyxIyxw
vuvuE
yxy
yxyx
yxyx
yxx
,
2
,
,,
2
),(),(),(),(),(
),(),(),(),(),(
][),(
),(),(),(2)0,0(
),(),(),(2)0,0(
),(),(),(2)0,0(
0)0,0(
0)0,0(
0)0,0(
,
,
,
yxIyxIyxwE
yxIyxIyxwE
yxIyxIyxwE
E
E
E
yxyx
uv
yyyx
vv
xxyx
uu
v
u
![Page 22: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/22.jpg)
Corner Detection: MathematicsThe quadratic approximation simplifies to
2
2,
( , ) x x y
x y x y y
I I IM w x y
I I I
where M is a second moment matrix computed from image derivatives:
v
uMvuvuE ][),(
M
![Page 23: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/23.jpg)
yyyx
yxxx
IIII
IIIIyxwM ),(
x
II x
y
II y
y
I
x
III yx
Corners as distinctive interest points
2 x 2 matrix of image derivatives (averaged in neighborhood of a point).
Notation:
![Page 24: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/24.jpg)
The surface E(u,v) is locally approximated by a quadratic form. Let’s try to understand its shape.
Interpreting the second moment matrix
v
uMvuvuE ][),(
yx yyx
yxx
III
IIIyxwM
,2
2
),(
![Page 25: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/25.jpg)
2
1
,2
2
0
0),(
yx yyx
yxx
III
IIIyxwM
First, consider the axis-aligned case (gradients are either horizontal or vertical)
If either λ is close to 0, then this is not a corner, so look for locations where both are large.
Interpreting the second moment matrix
![Page 26: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/26.jpg)
Consider a horizontal “slice” of E(u, v):
Interpreting the second moment matrix
This is the equation of an ellipse.
const][
v
uMvu
![Page 27: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/27.jpg)
Consider a horizontal “slice” of E(u, v):
Interpreting the second moment matrix
This is the equation of an ellipse.
RRM
2
11
0
0
The axis lengths of the ellipse are determined by the eigenvalues and the orientation is determined by R
direction of the slowest change
direction of the fastest change
(max)-1/2
(min)-1/2
const][
v
uMvu
Diagonalization of M:
![Page 28: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/28.jpg)
Visualization of second moment matrices
![Page 29: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/29.jpg)
Visualization of second moment matrices
![Page 30: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/30.jpg)
Interpreting the eigenvalues
1
2
“Corner”1 and 2 are large,
1 ~ 2;
E increases in all directions
1 and 2 are small;
E is almost constant in all directions
“Edge” 1 >> 2
“Edge” 2 >> 1
“Flat” region
Classification of image points using eigenvalues of M:
![Page 31: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/31.jpg)
Corner response function
“Corner”R > 0
“Edge” R < 0
“Edge” R < 0
“Flat” region
|R| small
22121
2 )()(trace)det( MMR
α: constant (0.04 to 0.06)
![Page 32: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/32.jpg)
Harris corner detector
1) Compute M matrix for each image window to get their cornerness scores.
2) Find points whose surrounding window gave large corner response (f> threshold)
3) Take the points of local maxima, i.e., perform non-maximum suppression
C.Harris and M.Stephens. “A Combined Corner and Edge Detector.” Proceedings of the 4th Alvey Vision Conference: pages 147—151, 1988.
![Page 33: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/33.jpg)
Harris Detector: Steps
![Page 34: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/34.jpg)
Harris Detector: StepsCompute corner response R
![Page 35: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/35.jpg)
Harris Detector: StepsFind points with large corner response: R>threshold
![Page 36: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/36.jpg)
Harris Detector: StepsTake only the points of local maxima of R
![Page 37: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/37.jpg)
Harris Detector: Steps
![Page 38: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/38.jpg)
Invariance and covariance• We want corner locations to be invariant to photometric
transformations and covariant to geometric transformations• Invariance: image is transformed and corner locations do not change• Covariance: if we have two transformed versions of the same image,
features should be detected in corresponding locations
![Page 39: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/39.jpg)
Affine intensity change
• Only derivatives are used => invariance to intensity shift I I + b
• Intensity scaling: I a I
R
x (image coordinate)
threshold
R
x (image coordinate)
Partially invariant to affine intensity change
I a I + b
![Page 40: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/40.jpg)
Image translation
• Derivatives and window function are shift-invariant
Corner location is covariant w.r.t. translation
![Page 41: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/41.jpg)
Image rotation
Second moment ellipse rotates but its shape (i.e. eigenvalues) remains the same
Corner location is covariant w.r.t. rotation
![Page 42: Project 2 Results](https://reader036.vdocuments.us/reader036/viewer/2022062723/56813f52550346895daa100d/html5/thumbnails/42.jpg)
Scaling
All points will be classified as edges
Corner
Corner location is not covariant to scaling!