scale-space image processing - stanford...
TRANSCRIPT
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 1
Scale-space image processing
Corresponding image features can appear at different scales
Like shift-invariance, scale-invariance of image processing algorithms
is often desirable.
Scale-space representation is useful to process an image in a manner
that is both shift-invariant and scale-invariant
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 2
Scale-space image processing
Scale-space theory
Laplacian of Gaussian (LoG) and Difference of Gaussian (DoG)
Scale-space edge detection
Scale-space keypoint detection
Harris-Laplacian
SIFT detector
SURF detector
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 3
Scale-space representation of a signal
Successive smoothingwith a Gaussian filter
Zero-crossings of 2nd derivative Fewer edges at coarser scales
Parametric family of signals f t (x) where fine-scale information is successively attenuated
f x( ) = f 0 x( )
f t x( ) scale t
f t x
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 4
Scale-space representation of images
Parametric family of images smoothed by Gaussian filter
Shift-invariance
Rotation-invariance
2 2
2 2
1, , * , ; t 0 with , exp
2 2
, , , with , exp2
t t t
t t t
x y x y x y x y x y
x yf x y g x y f x y g x y
t t
tF G F G
, , * , t tf x x y y g x y f x x y y
cos sin , sin cos , * cos sin , sin cost tf x y x y g x y f x y x y
Original
image f (x,y)
Coarser
scalest
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 5
Scale-space representation of images (cont.)
Commutative semigroup property
Separability
2 2 2 2
2 2 2 2
1 1 1, exp exp exp
2 2 2 22 2
, exp exp exp2 2 2
t
t
x y x y x y
x y x yg x y
t t t tt t
t t tG
1 2 1 2 2 1
1 2
, , * , , * ,
, * , * ,
t t t t t t
t t
f x y g x y f x y g x y f x y
g x y g x y f x y
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 6
Scale-space representation of images (cont.)
Non-creation of local extrema (for f (x,y) and all of its partial derivatives)
since and unimodal.
Solution to diffusion equation (heat equation)
, 0tg x y
2 2
2 2 2 2
2 2
, , ,
exp ,2
1exp ,
2 2
1,
2
t t
x y x y x y
x y x y
x y x y x y
t
x y x y
F G Ft t
tF
t
tF
F
21, ,
2
t tf x y f x yt
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 7
21, ,
2
t tf x y f x yt
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 8
LoG vs. DoG
-4
-2
0
2
4
-4
-2
0
2
4
-0.25
-0.2
-0.15
-0.1
-0.05
0
0.05
XY -4
-2
0
2
4
-4
-2
0
2
4
-0.25
-0.2
-0.15
-0.1
-0.05
0
0.05
XY
2 2
2 2
22
1, 1
2
x y
tx y
LoG x y et t
21
, ( , ) ( , )1
k t tDoG x y g x y g x yk t
Laplacian of Gaussian Difference of Gaussians
t = σ2 = 1 t = σ2 = 1, k = 1.1 21
, ,2
t tf x y f x yt
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 9
LoG vs. DoG (cont.)
Laplacian of Gaussian Difference of Gaussians
-2
0
2
-2
0
2
0
0.2
0.4
0.6
0.8
x
y
|H|
-2
0
2
-2
0
2
0
0.2
0.4
0.6
0.8
x
y
|H|
2 2, ,t
x y x y x yH G
21
, , ,1
k t t
x y x y x yH G Gk t
t = σ2 = 1 t = σ2 = 1, k = 1.1
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 10
Scale space: Laplacian images
t = 1 t = 4 t = 16 t = 64
,tf x y
2 ,tt f x y
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 11
Scale space: Binarized Laplacian images
t = 1 t = 4 t = 16 t = 64
,tf x y
2 ,tsign t f x y
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 12
Scale space: edge detection
Zero crossings of Laplacian images
Low-gradient-magnitude edges removed
t = 1 t = 4 t = 16 t = 64
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 13
Laplacian zero-crossings
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 14
Scale-space representation provides all scales;
which scale is best for keypoint detection?
Harris-Laplacian
1. Detect Harris corners at some initial scale
2. For each Harris corner
detect characteristic scale
3. Apply Harris detector in a spatial neighborhood
at scale to refine keypoint location
4. Repeat 2. and 3. until convergence
2arg max ,t
h h ht
t t f x y
,h hx y
ht ,h hx y
scale t
x
y Harris
Harris
Harris
Keypoint detection with automatic scale selection
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 15
Harris-Laplacian example (150 strongest peaks)
Keypoint detection with automatic scale selection
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 16
Keypoint detection with automatic scale selection
Harris-Laplacian example (200 strongest peaks)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 17
SIFT - Scale-Invariant Feature Transform
Decompose image into DoG scale-space representation
Detect minima and maxima locally and across scales
Fit 3-d quadratic function to localize extrema with sub-pixel/sub-scale accuracy [Brown, Lowe, 2002]
Eliminate edge responses based on Hessian
t 1
t 2
t 2
t 2 2
t 4
t 4
t 4 2
t 8
t 8 2
t 16
Gaussian
Difference of
Gaussian (DoG)
Scale
(first
octave)
Scale
(next
octave)
Scale
…SIFT keypoint detection
[Lowe, 1999, 2004]
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 18
SIFT scale space pyramid: octave 1
-
-
-
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 19
SIFT scale space pyramid: octave 2
-
-
-
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 20
SIFT scale space pyramid: octave 3
-
-
-
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 21
SIFT scale space pyramid: octave 4
-
-
-
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 22
SIFT scale space pyramid: octave 5
-
-
-
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 23
SIFT keypoints
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 24
SIFT keypoints
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 25
Robustness against scaling
[Mikolajczyk, Schmid, 2001]
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 26
Hessian keypoints in scale space
( , )g x y
( , )g x yx
( , )g x y
y
2
2( , )g x y
x
2
2( , )g x y
y
2
( , )g x yx y
2 2
2
2 2
2
,
, ,
, ,
t
t t
t t
x y
f x y f x yx x y
f x y f x yx y y
H
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 27
SURF keypoint detection
SURF – Speeded Up Robust Features [Bay, Tuytelaars, Van Gool, ECCV 2006]
No subsampling – all resolution levels at full spatial resolution
Simple approximation of scale space Gaussian derivatives using integral images
Determinant of Hessian
Non-maximum suppression in 3x3x3 [x,y,t] neighborhood
Interpolation of maximum of det(H) in image space x,y and scale t
t
yyD t
xyD
2
det 0.9t t t t
xx yy xyD D D H
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 28
SURF keypoints
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 29
SIFT keypoints
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 30
SURF keypoints
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 31
SIFT keypoints