![Page 1: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/1.jpg)
![Page 2: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/2.jpg)
Last lecture: Edges primer
• Edge detection to identify
visual change in image
• Derivative of Gaussian
and linear combination
of convolutions
• What is an edge?
What is a good edge?
gdx
df
f
gdx
d
![Page 3: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/3.jpg)
Canny edge detector
• Probably the most widely used edge detector in computer vision.
• Theoretical model: step-edges corrupted by additive Gaussian noise.
• Canny showed that first derivative of Gaussian closely approximates the operator that optimizes the product of signal-to-noise ratio and localization.
J. Canny, A Computational Approach To Edge Detection, IEEE
Trans. Pattern Analysis and Machine Intelligence, 8:679-714, 1986.
L. Fei-Fei
22,000 citations!
![Page 4: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/4.jpg)
Examples: Controversy and Appropriateness
‘Lena’ ‘Fabio’
Deanna Needell @ Claremont McKenna, 2012Alexander Sawchuk @ USC, 1973
![Page 5: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/5.jpg)
If it wasn’t clear from class…
Use of Lena is now generally considered
inappropriate, as it is not inclusive.
If you wish to include models, then please
respect diversity.
We will approach diversity issues in more detail
when we start to talk about machine learning
and datasets.
![Page 6: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/6.jpg)
Canny edge detector
1. Filter image with x, y derivatives of Gaussian
Source: D. Lowe, L. Fei-Fei
![Page 7: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/7.jpg)
Derivative of Gaussian filter
x-direction y-direction
![Page 8: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/8.jpg)
Compute Gradients
X-Derivative of Gaussian Y-Derivative of Gaussian
![Page 9: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/9.jpg)
Canny edge detector
1. Filter image with x, y derivatives of Gaussian
2. Find magnitude and orientation of gradient
Source: D. Lowe, L. Fei-Fei
![Page 10: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/10.jpg)
Compute Gradient Magnitude
sqrt( X-Deriv.ofGaussian ^2 + Y-Deriv.ofGaussian ^2 ) = gradient
magnitude
![Page 11: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/11.jpg)
Compute Gradient Orientation
Threshold magnitude at minimum level
Get orientation via theta = atan2(gy, gx)
![Page 12: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/12.jpg)
Canny edge detector
1. Filter image with x, y derivatives of Gaussian
2. Find magnitude and orientation of gradient
3. Non-maximum suppression:
• Thin multi-pixel wide “ridges” to single pixel width
Source: D. Lowe, L. Fei-Fei
![Page 13: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/13.jpg)
Non-maximum suppression for each orientation
At pixel q:
We have a maximum if the
value is larger than those at
both p and at r.
Interpolate along gradient
direction to get these values.
Source: D. Forsyth
![Page 14: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/14.jpg)
Sidebar: Bilinear Interpolation
http://en.wikipedia.org/wiki/Bilinear_interpolation
![Page 15: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/15.jpg)
Sidebar: Interpolation options
• imx2 = imresize(im, 2, interpolation_type)
• ‘nearest’ – Copy value from nearest known
– Very fast but creates blocky edges
• ‘bilinear’– Weighted average from four nearest known
pixels
– Fast and reasonable results
• ‘bicubic’ (default)– Non-linear smoothing over larger area (4x4)
– Slower, visually appealing, may create negative pixel values
Examples from http://en.wikipedia.org/wiki/Bicubic_interpolation
![Page 16: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/16.jpg)
Non-maximum suppression for each orientation
At pixel q:
We have a maximum if the
value is larger than those at
both p and at r.
Interpolate along gradient
direction to get these values.
Source: D. Forsyth
![Page 17: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/17.jpg)
Before Non-max Suppression
Gradient magnitudeJames Hays
![Page 18: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/18.jpg)
After non-max suppression
Gradient magnitudeJames Hays
![Page 19: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/19.jpg)
Canny edge detector
1. Filter image with x, y derivatives of Gaussian
2. Find magnitude and orientation of gradient
3. Non-maximum suppression:
• Thin multi-pixel wide “ridges” to single pixel width
4. ‘Hysteresis’ Thresholding:
• Define two thresholds: low and high
• Use the high threshold to start edge curves and
the low threshold to continue them
• ‘Follow’ edges starting from strong edge pixels– Connected components (Szeliski 3.3.4)
Source: D. Lowe, L. Fei-Fei
![Page 20: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/20.jpg)
‘Hysteresis’ thresholding
• Two thresholds – high and low
• Grad. mag. > high threshold? = strong edge
• Grad. mag. < low threshold? noise
• In between = weak edge
• ‘Follow’ edges starting from strong edge pixels
• Continue them into weak edges– Connected components (Szeliski 3.3.4)
Source: S. Seitz
![Page 21: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/21.jpg)
Final Canny Edges
![Page 22: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/22.jpg)
Effect of (Gaussian kernel spread/size)
Canny with Canny with Original
The choice of depends on desired behavior• large detects large scale edges
• small detects fine features
Source: S. Seitz
![Page 23: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/23.jpg)
Canny edge detector
1. Filter image with x, y derivatives of Gaussian
2. Find magnitude and orientation of gradient
3. Non-maximum suppression:
• Thin multi-pixel wide “ridges” to single pixel width
4. ‘Hysteresis’ Thresholding:
• Define two thresholds: low and high
• Use the high threshold to start edge curves and
the low threshold to continue them
• ‘Follow’ edges starting from strong edge pixels– Connected components (Szeliski 3.3.4)
MATLAB: edge(image, ‘canny’)
Source: D. Lowe, L. Fei-Fei
![Page 24: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/24.jpg)
James Hays
![Page 25: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/25.jpg)
James Hays
![Page 26: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/26.jpg)
Interest Points and Corners
Slides from Rick Szeliski, Svetlana Lazebnik, Derek Hoiem and Grauman&Leibe 2008 AAAI Tutorial
Szeliski 4.1
“Interest points” = “keypoints”
Sometimes called “features”
![Page 27: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/27.jpg)
Correspondence across views
• Correspondence: matching points, patches, edges, or regions across images.
≈
James Hays
![Page 28: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/28.jpg)
Example: estimate “fundamental matrix” that corresponds two views
Slide from Silvio Savarese
![Page 29: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/29.jpg)
Example: structure from motion
![Page 30: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/30.jpg)
Fundamental to Applications
• Feature points are used for:– Image alignment
– 3D reconstruction
– Motion tracking
– Robot navigation
– Indexing and database retrieval
– Object recognition
James Hays
![Page 31: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/31.jpg)
Example application
• Panorama stitching• We have two images – how do we combine them?
![Page 32: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/32.jpg)
Local features: main components
1) Detection:Find a set of distinctive key points.
2) Description: Extract feature descriptor around
each interest point as vector.
3) Matching: Compute distance between feature
vectors to find correspondence.
],,[ )1()1(
11 dxx x
],,[ )2()2(
12 dxx x
Td )x,x( 21
1x
2x
K. Grauman, B. Leibe
![Page 33: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/33.jpg)
Characteristics of good features
• Repeatability• The same feature can be found in several images despite geometric
and photometric transformations
• Saliency• Each feature is distinctive
• Compactness and efficiency• Many fewer features than image pixels
• Locality• A feature occupies a relatively small area of the image; robust to
clutter and occlusion
Kristen Grauman
![Page 34: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/34.jpg)
Goal: interest operator repeatability
• We want to detect (at least some of) the
same points in both images.
• Yet we have to be able to run the detection
procedure independently per image.
No chance to find true matches!
Kristen Grauman
![Page 35: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/35.jpg)
Goal: descriptor distinctiveness
• We want to be able to reliably determine which
point goes with which.
• Must provide some invariance to geometric and
photometric differences between the two views.
?
Kristen Grauman
![Page 36: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/36.jpg)
Local features: main components
1) Detection:Find a set of distinctive key points.
2) Description:Extract feature descriptor around
each interest point as vector.
3) Matching:Compute distance between feature
vectors to find correspondence.
![Page 37: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/37.jpg)
Detection: Basic Idea
• We do not know which other image
locations the feature will end up being
matched against.
• But we can compute how stable a
location is in appearance with respect
to small variations in position u.
• Compare image patch against
local neighbors.
Source: A. Efros
![Page 38: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/38.jpg)
Corner Detection: Mathematics
2
,
( , ) ( , ) ( , ) ( , )x y
E u v w x y I x u y v I x y
IntensityShifted intensity
Window function
orWindow function w(x,y) =
Gaussian1 in window, 0 outside
Source: R. Szeliski
Change in appearance of window w(x,y) for shift [u,v]:
![Page 39: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/39.jpg)
Corner Detection: Mathematics
2
,
( , ) ( , ) ( , ) ( , )x y
E u v w x y I x u y v I x y
I(x, y)E(u, v)
E(0,0)
w(x, y)
Change in appearance of window w(x,y)
for the shift [u,v]:
![Page 40: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/40.jpg)
Corner Detection: Mathematics
2
,
( , ) ( , ) ( , ) ( , )x y
E u v w x y I x u y v I x y
Change in appearance of window w(x,y)
for the shift [u,v]:
I(x, y)E(u, v)
E(3,2)
w(x, y)
![Page 41: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/41.jpg)
Inverted, and in 3D!
TPS Time
2
,
( , ) ( , ) ( , ) ( , )x y
E u v w x y I x u y v I x y
![Page 42: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/42.jpg)
Corner Detection: Basic Idea
• We might recognize the point by looking through a small window.
• We want a window shift in any direction to give a large change in intensity.
“Edge”:
no change
along the edge
direction
“Corner”:
significant
change in all
directions
“Flat” region:
no change in
all directions
Source: A. Efros
![Page 43: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/43.jpg)
Corner Detection: Mathematics
2
,
( , ) ( , ) ( , ) ( , )x y
E u v w x y I x u y v I x y
We want to find out how this function behaves for
small shifts
Change in appearance of window w(x,y)
for the shift [u,v]:
But this is very slow to compute naively.
O(window_width2 * shift_range2 * image_width2)
O( 112 * 112 * 6002 ) = 5.2 billion of these
14.6 thousand per pixel in your image
![Page 44: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/44.jpg)
Corner Detection: Mathematics
2
,
( , ) ( , ) ( , ) ( , )x y
E u v w x y I x u y v I x y
We want to find out how this function behaves for
small shifts.
Change in appearance of window w(x,y)
for the shift [u,v]:
But we know the response in E that we are looking
for – strong peak.
![Page 45: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/45.jpg)
Recall: Taylor series expansion
A function f can be represented by an infinite series
of its derivatives at a single point a:
Approximation of
f(x) = ex
centered at f(0)
Wikipedia
Set a = 0
(MacLaurin series)
as window centered
![Page 46: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/46.jpg)
Taylor expansion in 2D
Local quadratic approximation of E(u,v) in the
neighborhood of (0,0) is given by the second-order
Taylor expansion:
v
u
EE
EEvu
E
EvuEvuE
vvuv
uvuu
v
u
)0,0()0,0(
)0,0()0,0(][
2
1
)0,0(
)0,0(][)0,0(),(
![Page 47: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/47.jpg)
Corner Detection: Mathematics
Local quadratic approximation of E(u,v) in the
neighborhood of (0,0) is given by the second-order
Taylor expansion:
v
u
EE
EEvu
E
EvuEvuE
vvuv
uvuu
v
u
)0,0()0,0(
)0,0()0,0(][
2
1
)0,0(
)0,0(][)0,0(),(
Set to 0First
derivative,
set to 0
![Page 48: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/48.jpg)
Corner Detection: Mathematics
The quadratic approximation simplifies to
2
2,
( , )x x y
x y x y y
I I IM w x y
I I I
where M is a second moment matrix computed from image
derivatives:
v
uMvuvuE ][),(
M
v
u
EE
EEvuvuE
vvuv
uvuu
)0,0()0,0(
)0,0()0,0(][),(
![Page 49: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/49.jpg)
yyyx
yxxx
IIII
IIIIyxwM ),(
x
II x
y
II y
y
I
x
III yx
Corners as distinctive interest points
2 x 2 matrix of image derivatives (averaged in
neighborhood of a point).
Notation:
James Hays
![Page 50: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/50.jpg)
The surface E(u,v) is locally approximated by a
quadratic form. Let’s try to understand its shape.
Interpreting the second moment matrix
v
uMvuvuE ][),(
yx yyx
yxx
III
IIIyxwM
,2
2
),(
James Hays
![Page 51: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/51.jpg)
Consider a horizontal “slice” of E(u, v):
Interpreting the second moment matrix
This is the equation of an ellipse.
const][
v
uMvu
James Hays
![Page 52: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/52.jpg)
Consider a horizontal “slice” of E(u, v):
Interpreting the second moment matrix
This is the equation of an ellipse.
RRM
2
11
0
0
The axis lengths of the ellipse are determined by the
eigenvalues and the orientation is determined by R
direction of the
slowest change
direction of the
fastest change
(max)-1/2
(min)-1/2
const][
v
uMvu
Diagonalization of M:
James Hays
![Page 53: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/53.jpg)
Visualization of second moment matrices
James Hays
![Page 54: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/54.jpg)
Visualization of second moment matrices
James Hays
![Page 55: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/55.jpg)
Interpreting the eigenvalues
1
2
“Corner”
1 and 2 are large,
1 ~ 2;
E increases in all
directions
1 and 2 are small;
E is almost constant
in all directions
“Edge”
1 >> 2
“Edge”
2 >> 1
“Flat”
region
Classification of image points using eigenvalues of M:
![Page 56: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/56.jpg)
Corner response function
“Corner”
R > 0
“Edge”
R < 0
“Edge”
R < 0
“Flat”
region
|R| small
22
2121 )(trace)det()( MMR
α: constant (0.04 to 0.06)
Determinant (det(A)):
Trace (trace(A)):
1
2
![Page 57: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/57.jpg)
Harris corner detector
1) Compute M matrix for each image window to
get their cornerness scores.
2) Find points whose surrounding window gave
large corner response (f > threshold)
3) Take the points of local maxima, i.e., perform
non-maximum suppression
C.Harris and M.Stephens. “A Combined Corner and Edge Detector.” Proceedings of the 4th Alvey Vision Conference: pages 147—151, 1988.
![Page 58: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/58.jpg)
Harris Detector [Harris88]
• Second moment matrix
)()(
)()()(),(
2
2
DyDyx
DyxDx
IDIIII
IIIg
67
1. Image
derivatives
2. Square of
derivatives
3. Gaussian
filter g(I)
Ix Iy
Ix2 Iy
2 IxIy
g(Ix2) g(Iy
2) g(IxIy)
222222 )]()([)]([)()( yxyxyx IgIgIIgIgIg
])),([trace()],(det[ 2
DIDIhar
4. Cornerness function – both eigenvalues are strong
har5. Non-maxima suppression
1 2
1 2
det
trace
M
M
(optionally, blur first)
James Hays
![Page 59: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/59.jpg)
Harris Detector: Steps
![Page 60: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/60.jpg)
Harris Detector: Steps
Compute corner response R
![Page 61: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/61.jpg)
Harris Detector: Steps
Find points with large corner response: R>threshold
![Page 62: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/62.jpg)
Harris Detector: Steps
Take only the points of local maxima of R
![Page 63: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/63.jpg)
Harris Detector: Steps
![Page 64: Last lecture: Edges primer - Brown Universitycs.brown.edu/courses/cs143/2017_Spring/lectures_Spring... · 2017. 2. 13. · Last lecture: Edges primer • Edge detection to identify](https://reader034.vdocuments.us/reader034/viewer/2022051814/603899d78aa9a31d3342e969/html5/thumbnails/64.jpg)
Invariance and covariance
• We want corner locations to be invariant to photometric
transformations and covariant to geometric transformations
• Invariance: image is transformed and corner locations do not change
• Covariance: if we have two transformed versions of the same image,
features should be detected in corresponding locations