spatial filtering dan witzner hansen. recap exercises??? feedback last lectures?
Post on 19-Dec-2015
227 views
TRANSCRIPT
Spatial Filtering
Dan Witzner Hansen
Recap
• Exercises???• Feedback• Last lectures?
Digital images• In computer vision we operate on digital (discrete) images:
– Sample the 2D space on a regular grid– Quantize each sample (round to nearest integer)
• Image thus represented as a matrix of integer values.
Adapted from S. Seitz
2D
1D
What is point processing?
• Only one pixel in the input has an effect on the output• For example:
– Changing the brightness, thresholding, histogram stretching
0 21 2
1 2 12 5 3
1 32 20 1
1 2 02 1 41 0 1
9
12
Input Output
Image neighborhoods
• Q: What happens if we reshuffle all pixels within the images?
• A: Its histogram won’t change. Point-wise processing unaffected.
• Need to measure properties relative to small neighborhoods of pixels
Why use proximity
• Noise reduction• Smoothing• Feature detection• Correlation
What are they good for?• Improve Search
– Search over translations• Classic coarse-to-fine strategy
– Search over scale• Template matching• E.g. find a face at different scales
• Precomputation– Need to access image at different blur levels– Useful for texture mapping at different resolutions (called mip-
mapping) • Image Processing
– Editing frequency bands separately– E.g. image blending…
Motivation: noise reduction
Common types of noise– Salt and pepper noise:
random occurrences of black and white pixels
– Impulse noise: random occurrences of white pixels
– Gaussian (additive) noise: variations in intensity drawn from a Gaussian normal distribution
Source: S. Seitz
Gaussian noise
Fig: M. Hebert
>> noise = randn(size(im)).*sigma;
>> output = im + noise;
),(),(),( yxNyxFyxI
IdealImage
Noise process
),(),( GyxN
Changing Sigma and zero mean
First attempt at a solution
Replace each pixel with an average of all the values in its neighborhood
Assumptions: – Expect pixels to be like their neighbors– Expect noise processes to be independent from
pixel to pixel
First attempt at a solution
• Let’s replace each pixel with an average of all the values in its neighborhood
• Moving average in 1D:
Source: S. Marschner
Weighted Moving Average
• Can add weights to our moving average• Weights [1, 1, 1, 1, 1] / 5
Source: S. Marschner
Weighted Moving Average
• Non-uniform weights [1, 4, 6, 4, 1] / 16
Source: S. Marschner
Neighborhood processing
Several pixels in the input has an effect on the output
0 21 2
1 2 12 5 3
1 32 20 1
1 2 02 1 41 0 1
9
12
Input Output
Moving Average In 2D
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
Source: S. Seitz
Moving Average In 2D
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
Source: S. Seitz
Moving Average In 2D
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10 20
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
Source: S. Seitz
Moving Average In 2D
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10 20 30
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
Source: S. Seitz
Moving Average In 2D
0 10 20 30 30
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
Source: S. Seitz
Moving Average In 2D
0 10 20 30 30 30 20 10
0 20 40 60 60 60 40 20
0 30 60 90 90 90 60 30
0 30 50 80 80 90 60 30
0 30 50 80 80 90 60 30
0 20 30 50 50 60 40 20
10 20 30 30 30 30 20 10
10 10 10 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
Source: S. Seitz
Correlation filteringSay the averaging window size is 2k+1 x 2k+1:
Loop over all pixels in neighborhood around image pixel F[i,j]
Attribute uniform weight to each pixel
Now generalize to allow different weights depending on neighboring pixel’s relative position:
Non-uniform weights
Correlation filtering
Filtering an image: replace each pixel with a linear combination of its neighbors.
The filter “kernel” or “mask” H[u,v] is the prescription for the weights in the linear combination.
Assume the kernel H has weights 0 ‘outside’ the finite region
This is called cross-correlation, denoted
Averaging filter
What values belong in the kernel H for the moving average example?
0 10 20 30 30
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
111
111
111
“box filter”
?
Smoothing by averagingdepicts box filter: white = high value, black = low value
original filtered
Gaussian filterWhat if we want nearest neighboring pixels to
have the most influence on the output?
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
1 2 1
2 4 2
1 2 1
This kernel is an approximation of a Gaussian function:
Source: S. Seitz
Smoothing with a Gaussian
Gaussian filtersWhat parameters matter here?Size of kernel or mask
– Note, Gaussian function has infinite support, but discrete filters use finite kernels
σ = 5 with 10 x 10 kernel
σ = 5 with 30 x 30 kernel
Gaussian filters
Variance of Gaussian: determines extent of smoothing
σ = 2 with 30 x 30 kernel
σ = 5 with 30 x 30 kernel
Matlab>> hsize = 10;>> sigma = 5;>> h = fspecial(‘gaussian’ hsize, sigma);>> mesh(h);
>> imagesc(h);
>> outim = imfilter(im, h);>> imshow(outim);
The effect of smoothing and noise
More noise
Wider sm
oothing kernel
Boundary issues
• What is the size of the output?• MATLAB: filter2(g, f, shape) / conv2
– shape = ‘full’: output size is sum of sizes of f and g– shape = ‘same’: output size is same as f– shape = ‘valid’: output size is the difference of
sizes of f and g
f
gg
gg
f
gg
gg
f
gg
gg
full same valid
Source: S. Lazebnik
Boundary issues
• What about near the edge?– the filter window falls off the boundaries of the
image– methods:
clip filter (black)wrap aroundcopy edgereflect across edge
Source: S. Marschner
Boundary issues in Matlab
clip filter (black): imfilter(f, g, 0)wrap around: imfilter(f, g, ‘circular’)copy edge: imfilter(f, g, ‘replicate’)reflect across edge: imfilter(f, g, ‘symmetric’)
Source: S. Marschner
Filtering an impulse signal
What is the result of filtering the impulse signal (image) F with the arbitrary kernel H?
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 1 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
a b c
d e f
g h i ?
Convolution
• Convolution: – Flip the filter in both dimensions (bottom to top, right to left)– Then apply correlation
Notation for convolution operator
F
H
Convolution vs. correlationConvolution
Cross-correlation
Shift invariant linear system
• Shift invariant: Operator behaves the same everywhere, i.e. the
value of the output depends on the pattern in the image neighborhood, not the position of the neighborhood.
• Linear: Superposition: h * (f1 + f2) = (h * f1) + (h * f2) Scaling: h * (k f) = k (h * f)
Properties of convolution
• Linear & shift invariant• Commutative:
f * g = g * f• Associative
(f * g) * h = f * (g * h)• Identity:
unit impulse e = […, 0, 0, 1, 0, 0, …]. f * e = f
• Differentiation:
How to do color blurring?
Predict the filtered outputs
000010000
* = ?000100000
* = ?
111111111
000020000-* = ?
Practice with linear filters
000010000
Original
?
Source: D. Lowe
Practice with linear filters
000010000
Original Filtered (no change)
Source: D. Lowe
Practice with linear filters
000100000
Original
?
Source: D. Lowe
Practice with linear filters
000100000
Original Shifted leftby 1 pixel with correlation
Source: D. Lowe
Practice with linear filters
Original
?111111111
Source: D. Lowe
Practice with linear filters
Original
111111111
Blur (with abox filter)
Source: D. Lowe
Practice with linear filters
Original
111111111
000020000
- ?
Source: D. Lowe
Practice with linear filters
Original
111111111
000020000
-
Sharpening filter- Accentuates differences with
local average
Source: D. Lowe
Filtering examples: sharpening
Convolution with special filters for iris recognition
53
Template MatchingThe filter is called a template or a mask
• The brighter the value in the output, the better the match• Implemented as
– the correlation coefficient– Sums of squared differences (SSD) - sum(sum( (I-F).^2))
Input image
Template Output
Output as 3D
54
Template Matching
Output
Template matching
Scene
Template
What if the template is not identical to some subimage in the scene?
Template matching
Detected template
Template
Match can be meaningful, if scale, orientation, and general appearance is right.
So, what scale to choose?It depends what we’re looking for.
Too fine of a scale…can’t see the forest for the trees.Too coarse of a scale…can’t tell the maple grain from the cherry.
Smoothing with a Gaussian
for sigma=1:3:10 h = fspecial('gaussian‘, fsize, sigma);out = imfilter(im, h); imshow(out);pause;
end
…
Parameter σ is the “scale” / “width” / “spread” of the Gaussian kernel, and controls the amount of smoothing.
Gaussian pre-filtering
G 1/4
G 1/8
Gaussian 1/2
Solution: filter the image, then subsample• Filter size should double for each ½ size reduction. Why?
Subsampling with Gaussian pre-filtering
G 1/4 G 1/8Gaussian 1/2
Solution: filter the image, then subsample• Filter size should double for each ½ size reduction. Why?• How can we speed this up?
Image Pyramids
Known as a Gaussian Pyramid [Burt and Adelson, 1983]• In computer graphics, a mip map [Williams, 1983]
A bar in the big images is a hair on the zebra’s nose; in smaller images, a stripe; in the smallest, the animal’s nose
Figure from David Forsyth
So what about speed
• Convolution is• What now if we have to use many scales?• Can it be done faster?
– Seperability– Integral images (somewhat opposite of scale-
space / pyramid)– Fourier Transform (brief introduction)
€
O(N 2)
Separability
• In some cases, filter is separable, and we can factor into two steps:– Convolve all rows– Convolve all columns
Image Features
“Rectangle filters”
Value =
∑ (pixels in white area) – ∑ (pixels in black area)
Fast computation with integral images
• The integral image computes a value at each pixel (x,y) that is the sum of the pixel values above and to the left of (x,y), inclusive
• This can quickly be computed in one pass through the image
• Calculated once per image(x,y)
Computing the integral image
Computing the integral image
• Cumulative row sum: s(x, y) = s(x–1, y) + i(x, y) • Integral image: ii(x, y) = ii(x, y−1) + s(x, y)
ii(x, y-1)
s(x-1, y)
i(x, y)
MATLAB: ii = cumsum(cumsum(double(i)), 2);
Computing sum within a rectangle
• Let A,B,C,D be the values of the integral image at the corners of a rectangle
• Then the sum of original image values within the rectangle can be computed as:– sum = A – B – C + D
• Only 3 additions are required for any size of rectangle!
D B
C A
Example
-1 +1+2-1
-2+1
Integral Image
Integral images
Exceptionally fast when using many features at different scales
Notice the difference to image pyramids
Not all filters can be (efficiently) used with Integral images (e.g. Gaussian, but some approximation can)
Feature selection• For a 24x24 detection region, the number of
possible rectangle features is ~160,000!
• Can we make one approximation filter for the eye?
Boosting for face detection• First two features selected by boosting:
This feature combination can yield 100% detection rate and 50% false positive rate
Jean Baptiste Joseph Fourier (1768-1830)
• Had crazy idea (1807):• Any periodic function can be
rewritten as a weighted sum of Sines and Cosines of different frequencies.
• Don’t believe it? – Neither did Lagrange, Laplace,
Poisson and other big wigs– Not translated into English until
1878!• But it’s true!
– called Fourier Series– Possibly the greatest tool – used in Engineering
=
3 sin(x) A
+ 1 sin(3x) B A+B
+ 0.8 sin(5x) CA+B+C
+ 0.4 sin(7x) DA+B+C+D
A sum of sines and cosines
sin(x) A
Time and Frequency
• example : g(t) = sin(2pf t) + (1/3)sin(2p(3f) t)
Time and Frequency
• example : g(t) = sin(2pf t) + (1/3)sin(2p(3f) t)
= +
Higher frequencies dueto sharp image variations
(e.g., edges, noise, etc.)
Frequency Spectra
• example : g(t) = sin(2pf t) + (1/3)sin(2p(3f) t)
= +
Fourier Transform and Convolution
hfg FHG
€
g = fh HFG
Spatial Domain (x) Frequency Domain (u)
So, we can find g(x) by Fourier transform
g f h
G F H
FT FTIFT€
O(N 2)
€
O(NLg(N))
€
O(N)
The Continuous Fourier Transform
u
iuxduxf 2F(u)e)(
)2sin()2cos2 uxiux(iux e
Complex Numbers
Real
ImaginaryZ=(a,b)
a
b
|Z|
)(conjugate
(phase)
spectrum)(Fourier
vector)unit (a
i
i
eZibaZZ
abtg
baZ
ie
ii
*
1
22
2
)/(
sincos
)1(1
ieZ
iba
ZiZZ
)Im()Re(
x
ux2cos
– The wavelength is 1/u .
– The frequency is u .
1
The 1D Basis Functionsiuxe 2
)2sin()Im(
)2cos()Re(2
2
uxe
uxeiux
iux
1/u
The Fourier Transform
))(( xf
x
iuxdxuF 2f(x)e)(
1D Continuous Fourier Transform:
))((1 uFThe InverseFourier Transform
The Continuous Fourier Transform
x y
uxi dxdyvuF vy)(2y)ef(x,),(
2D Continuous Fourier Transform:
u v
uxi dudvyxf vy)(2v)eF(u,),(
u
iuxduxf 2F(u)e)(
The Inverse Transform
The Transform
The wavelength is . The direction is u/v .22/1 vu
The 2D Basis Functions
u=0, v=0 u=1, v=0 u=2, v=0u=-2, v=0 u=-1, v=0
u=0, v=1 u=1, v=1 u=2, v=1u=-2, v=1 u=-1, v=1
u=0, v=2 u=1, v=2 u=2, v=2u=-2, v=2 u=-1, v=2
u=0, v=-1 u=1, v=-1 u=2, v=-1u=-2, v=-1 u=-1, v=-1
u=0, v=-2 u=1, v=-2 u=2, v=-2u=-2, v=-2 u=-1, v=-2
U
V
)(2 vyuxie
1
0
2
)()(N
u
N
iux
euFxf
1
0
2
)(1
)(N
x
N
iux
exfN
uF
(u = 0,..., N-1)
(x = 0,..., N-1)
1D Discrete Fourier Transform:
The Discrete Fourier Transform
1
0
1
0
)(2),(
11),(
N
x
M
y
Mvy
Nux
ieyxf
MNvuF
2D Discrete Fourier Transform:
1
0
1
0
)(2),(),(
N
u
M
v
Mvy
Nux
ievuFyxf
(x = 0,..., N-1; y = 0,…,M-1)
(u = 0,..., N-1; v = 0,…,M-1)
Fourier spectrum log(1 + |F(u,v)|) Image f
The Fourier Image
Fourier spectrum |F(u,v)|
Frequency Bands
Percentage of image power enclosed in circles (small to large) :
90%, 95%, 98%, 99%, 99.5%, 99.9%
Image Fourier Spectrum
Low pass Filtering
90% 95%
98% 99%
99.5% 99.9%
Noise Removal
Noisy image
Fourier Spectrum Noise-cleaned image
High Pass Filtering
Original High Pass Filtered
High Frequency Emphasis
+Original High Pass Filtered
High Frequency EmphasisOriginal High Frequency Emphasis
OriginalHigh Frequency Emphasis
Original High pass Filter
High Frequency Emphasis
High Frequency Emphasis +
Histogram Equalization
High Frequency Emphasis