spatial filtering dan witzner hansen. recap exercises??? feedback last lectures?

Spatial Filtering

Dan Witzner Hansen

Recap

• Exercises???• Feedback• Last lectures?

Digital images• In computer vision we operate on digital (discrete) images:

– Sample the 2D space on a regular grid– Quantize each sample (round to nearest integer)

• Image thus represented as a matrix of integer values.

Adapted from S. Seitz

2D

1D

What is point processing?

• Only one pixel in the input has an effect on the output• For example:

– Changing the brightness, thresholding, histogram stretching

0 21 2

1 2 12 5 3

1 32 20 1

1 2 02 1 41 0 1

9

12

Input Output

Image neighborhoods

• Q: What happens if we reshuffle all pixels within the images?

• A: Its histogram won’t change. Point-wise processing unaffected.

• Need to measure properties relative to small neighborhoods of pixels

Why use proximity

• Noise reduction• Smoothing• Feature detection• Correlation

What are they good for?• Improve Search

– Search over translations• Classic coarse-to-fine strategy

– Search over scale• Template matching• E.g. find a face at different scales

• Precomputation– Need to access image at different blur levels– Useful for texture mapping at different resolutions (called mip-

mapping) • Image Processing

– Editing frequency bands separately– E.g. image blending…

Motivation: noise reduction

Common types of noise– Salt and pepper noise:

random occurrences of black and white pixels

– Impulse noise: random occurrences of white pixels

– Gaussian (additive) noise: variations in intensity drawn from a Gaussian normal distribution

Source: S. Seitz

Gaussian noise

Fig: M. Hebert

>> noise = randn(size(im)).*sigma;

>> output = im + noise;

),(),(),( yxNyxFyxI

IdealImage

Noise process

),(),( GyxN

Changing Sigma and zero mean

First attempt at a solution

Replace each pixel with an average of all the values in its neighborhood

Assumptions: – Expect pixels to be like their neighbors– Expect noise processes to be independent from

pixel to pixel

First attempt at a solution

• Let’s replace each pixel with an average of all the values in its neighborhood

• Moving average in 1D:

Source: S. Marschner

Weighted Moving Average

• Can add weights to our moving average• Weights [1, 1, 1, 1, 1] / 5

Source: S. Marschner

Weighted Moving Average

• Non-uniform weights [1, 4, 6, 4, 1] / 16

Source: S. Marschner

Neighborhood processing

Several pixels in the input has an effect on the output

0 21 2

1 2 12 5 3

1 32 20 1

1 2 02 1 41 0 1

9

12

Input Output

Moving Average In 2D

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

Source: S. Seitz

Moving Average In 2D

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 10

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

Source: S. Seitz

Moving Average In 2D

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 10 20

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

Source: S. Seitz

Moving Average In 2D

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 10 20 30

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

Source: S. Seitz

Moving Average In 2D

0 10 20 30 30

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

Source: S. Seitz

Moving Average In 2D

0 10 20 30 30 30 20 10

0 20 40 60 60 60 40 20

0 30 60 90 90 90 60 30

0 30 50 80 80 90 60 30

0 30 50 80 80 90 60 30

0 20 30 50 50 60 40 20

10 20 30 30 30 30 20 10

10 10 10 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

Source: S. Seitz

Correlation filteringSay the averaging window size is 2k+1 x 2k+1:

Loop over all pixels in neighborhood around image pixel F[i,j]

Attribute uniform weight to each pixel

Now generalize to allow different weights depending on neighboring pixel’s relative position:

Non-uniform weights

Correlation filtering

Filtering an image: replace each pixel with a linear combination of its neighbors.

The filter “kernel” or “mask” H[u,v] is the prescription for the weights in the linear combination.

Assume the kernel H has weights 0 ‘outside’ the finite region

This is called cross-correlation, denoted

Averaging filter

What values belong in the kernel H for the moving average example?

0 10 20 30 30

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

111

111

111

“box filter”

?

Smoothing by averagingdepicts box filter: white = high value, black = low value

original filtered

Gaussian filterWhat if we want nearest neighboring pixels to

have the most influence on the output?

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

1 2 1

2 4 2

1 2 1

This kernel is an approximation of a Gaussian function:

Source: S. Seitz

Smoothing with a Gaussian

Gaussian filtersWhat parameters matter here?Size of kernel or mask

– Note, Gaussian function has infinite support, but discrete filters use finite kernels

σ = 5 with 10 x 10 kernel

σ = 5 with 30 x 30 kernel

Gaussian filters

Variance of Gaussian: determines extent of smoothing

σ = 2 with 30 x 30 kernel

σ = 5 with 30 x 30 kernel

Matlab>> hsize = 10;>> sigma = 5;>> h = fspecial(‘gaussian’ hsize, sigma);>> mesh(h);

>> imagesc(h);

>> outim = imfilter(im, h);>> imshow(outim);

The effect of smoothing and noise

More noise

Wider sm

oothing kernel

Boundary issues

• What is the size of the output?• MATLAB: filter2(g, f, shape) / conv2

– shape = ‘full’: output size is sum of sizes of f and g– shape = ‘same’: output size is same as f– shape = ‘valid’: output size is the difference of

sizes of f and g

f

gg

gg

f

gg

gg

f

gg

gg

full same valid

Source: S. Lazebnik

Boundary issues

• What about near the edge?– the filter window falls off the boundaries of the

image– methods:

clip filter (black)wrap aroundcopy edgereflect across edge

Source: S. Marschner

Boundary issues in Matlab

clip filter (black): imfilter(f, g, 0)wrap around: imfilter(f, g, ‘circular’)copy edge: imfilter(f, g, ‘replicate’)reflect across edge: imfilter(f, g, ‘symmetric’)

Source: S. Marschner

Filtering an impulse signal

What is the result of filtering the impulse signal (image) F with the arbitrary kernel H?

0 0 0 0 0 0 0

0 0 0 0 0 0 0

0 0 0 0 0 0 0

0 0 0 1 0 0 0

0 0 0 0 0 0 0

0 0 0 0 0 0 0

0 0 0 0 0 0 0

a b c

d e f

g h i ?

Convolution

• Convolution: – Flip the filter in both dimensions (bottom to top, right to left)– Then apply correlation

Notation for convolution operator

F

H

Convolution vs. correlationConvolution

Cross-correlation

Shift invariant linear system

• Shift invariant: Operator behaves the same everywhere, i.e. the

value of the output depends on the pattern in the image neighborhood, not the position of the neighborhood.

• Linear: Superposition: h * (f1 + f2) = (h * f1) + (h * f2) Scaling: h * (k f) = k (h * f)

Properties of convolution

• Linear & shift invariant• Commutative:

f * g = g * f• Associative

(f * g) * h = f * (g * h)• Identity:

unit impulse e = […, 0, 0, 1, 0, 0, …]. f * e = f

• Differentiation:

How to do color blurring?

Predict the filtered outputs

000010000

* = ?000100000

* = ?

111111111

000020000-* = ?

Practice with linear filters

000010000

Original

?

Source: D. Lowe

Practice with linear filters

000010000

Original Filtered (no change)

Source: D. Lowe

Practice with linear filters

000100000

Original

?

Source: D. Lowe

Practice with linear filters

000100000

Original Shifted leftby 1 pixel with correlation

Source: D. Lowe

Practice with linear filters

Original

?111111111

Source: D. Lowe

Practice with linear filters

Original

111111111

Blur (with abox filter)

Source: D. Lowe

Practice with linear filters

Original

111111111

000020000

- ?

Source: D. Lowe

Practice with linear filters

Original

111111111

000020000

-

Sharpening filter- Accentuates differences with

local average

Source: D. Lowe

Filtering examples: sharpening

Convolution with special filters for iris recognition

53

Template MatchingThe filter is called a template or a mask

• The brighter the value in the output, the better the match• Implemented as

– the correlation coefficient– Sums of squared differences (SSD) - sum(sum( (I-F).^2))

Input image

Template Output

Output as 3D

54

Template Matching

Output

Template matching

Scene

Template

What if the template is not identical to some subimage in the scene?

Template matching

Detected template

Template

Match can be meaningful, if scale, orientation, and general appearance is right.

So, what scale to choose?It depends what we’re looking for.

Too fine of a scale…can’t see the forest for the trees.Too coarse of a scale…can’t tell the maple grain from the cherry.

Smoothing with a Gaussian

for sigma=1:3:10 h = fspecial('gaussian‘, fsize, sigma);out = imfilter(im, h); imshow(out);pause;

end

…

Parameter σ is the “scale” / “width” / “spread” of the Gaussian kernel, and controls the amount of smoothing.

Gaussian pre-filtering

G 1/4

G 1/8

Gaussian 1/2

Solution: filter the image, then subsample• Filter size should double for each ½ size reduction. Why?

Subsampling with Gaussian pre-filtering

G 1/4 G 1/8Gaussian 1/2

Solution: filter the image, then subsample• Filter size should double for each ½ size reduction. Why?• How can we speed this up?

Image Pyramids

Known as a Gaussian Pyramid [Burt and Adelson, 1983]• In computer graphics, a mip map [Williams, 1983]

A bar in the big images is a hair on the zebra’s nose; in smaller images, a stripe; in the smallest, the animal’s nose

Figure from David Forsyth

So what about speed

• Convolution is• What now if we have to use many scales?• Can it be done faster?

– Seperability– Integral images (somewhat opposite of scale-

space / pyramid)– Fourier Transform (brief introduction)

€

O(N 2)

Separability

• In some cases, filter is separable, and we can factor into two steps:– Convolve all rows– Convolve all columns

Haar-Like Features

Haar-Wavelet

Image Features

“Rectangle filters”

Value =

∑ (pixels in white area) – ∑ (pixels in black area)

Fast computation with integral images

• The integral image computes a value at each pixel (x,y) that is the sum of the pixel values above and to the left of (x,y), inclusive

• This can quickly be computed in one pass through the image

• Calculated once per image(x,y)

Computing the integral image

Computing the integral image

• Cumulative row sum: s(x, y) = s(x–1, y) + i(x, y) • Integral image: ii(x, y) = ii(x, y−1) + s(x, y)

ii(x, y-1)

s(x-1, y)

i(x, y)

MATLAB: ii = cumsum(cumsum(double(i)), 2);

Computing sum within a rectangle

• Let A,B,C,D be the values of the integral image at the corners of a rectangle

• Then the sum of original image values within the rectangle can be computed as:– sum = A – B – C + D

• Only 3 additions are required for any size of rectangle!

D B

C A

Example

-1 +1+2-1

-2+1

Integral Image

Integral images

Exceptionally fast when using many features at different scales

Notice the difference to image pyramids

Not all filters can be (efficiently) used with Integral images (e.g. Gaussian, but some approximation can)

Feature selection• For a 24x24 detection region, the number of

possible rectangle features is ~160,000!

• Can we make one approximation filter for the eye?

Boosting for face detection• First two features selected by boosting:

This feature combination can yield 100% detection rate and 50% false positive rate

Jean Baptiste Joseph Fourier (1768-1830)

• Had crazy idea (1807):• Any periodic function can be

rewritten as a weighted sum of Sines and Cosines of different frequencies.

• Don’t believe it? – Neither did Lagrange, Laplace,

Poisson and other big wigs– Not translated into English until

1878!• But it’s true!

– called Fourier Series– Possibly the greatest tool – used in Engineering

=

3 sin(x) A

+ 1 sin(3x) B A+B

+ 0.8 sin(5x) CA+B+C

+ 0.4 sin(7x) DA+B+C+D

A sum of sines and cosines

sin(x) A

Time and Frequency

• example : g(t) = sin(2pf t) + (1/3)sin(2p(3f) t)

Time and Frequency

• example : g(t) = sin(2pf t) + (1/3)sin(2p(3f) t)

= +

Higher frequencies dueto sharp image variations

(e.g., edges, noise, etc.)

Frequency Spectra

• example : g(t) = sin(2pf t) + (1/3)sin(2p(3f) t)

= +

Fourier Transform and Convolution

hfg FHG

€

g = fh HFG

Spatial Domain (x) Frequency Domain (u)

So, we can find g(x) by Fourier transform

g f h

G F H

FT FTIFT€

O(N 2)

€

O(NLg(N))

€

O(N)

The Continuous Fourier Transform

u

iuxduxf 2F(u)e)(

)2sin()2cos2 uxiux(iux e

Complex Numbers

Real

ImaginaryZ=(a,b)

a

b

|Z|

)(conjugate

(phase)

spectrum)(Fourier

vector)unit (a

i

i

eZibaZZ

abtg

baZ

ie

ii

*

1

22

2

)/(

sincos

)1(1

ieZ

iba

ZiZZ

)Im()Re(

x

ux2cos

– The wavelength is 1/u .

– The frequency is u .

1

The 1D Basis Functionsiuxe 2

)2sin()Im(

)2cos()Re(2

2

uxe

uxeiux

iux

1/u

The Fourier Transform

))(( xf

x

iuxdxuF 2f(x)e)(

1D Continuous Fourier Transform:

))((1 uFThe InverseFourier Transform

The Continuous Fourier Transform

x y

uxi dxdyvuF vy)(2y)ef(x,),(

2D Continuous Fourier Transform:

u v

uxi dudvyxf vy)(2v)eF(u,),(

u

iuxduxf 2F(u)e)(

The Inverse Transform

The Transform

The wavelength is . The direction is u/v .22/1 vu

The 2D Basis Functions

u=0, v=0 u=1, v=0 u=2, v=0u=-2, v=0 u=-1, v=0

u=0, v=1 u=1, v=1 u=2, v=1u=-2, v=1 u=-1, v=1

u=0, v=2 u=1, v=2 u=2, v=2u=-2, v=2 u=-1, v=2

u=0, v=-1 u=1, v=-1 u=2, v=-1u=-2, v=-1 u=-1, v=-1

u=0, v=-2 u=1, v=-2 u=2, v=-2u=-2, v=-2 u=-1, v=-2

U

V

)(2 vyuxie

1

0

2

)()(N

u

N

iux

euFxf

1

0

2

)(1

)(N

x

N

iux

exfN

uF

(u = 0,..., N-1)

(x = 0,..., N-1)

1D Discrete Fourier Transform:

The Discrete Fourier Transform

1

0

1

0

)(2),(

11),(

N

x

M

y

Mvy

Nux

ieyxf

MNvuF

2D Discrete Fourier Transform:

1

0

1

0

)(2),(),(

N

u

M

v

Mvy

Nux

ievuFyxf

(x = 0,..., N-1; y = 0,…,M-1)

(u = 0,..., N-1; v = 0,…,M-1)

Fourier spectrum log(1 + |F(u,v)|) Image f

The Fourier Image

Fourier spectrum |F(u,v)|

Frequency Bands

Percentage of image power enclosed in circles (small to large) :

90%, 95%, 98%, 99%, 99.5%, 99.9%

Image Fourier Spectrum

Low pass Filtering

90% 95%

98% 99%

99.5% 99.9%

Noise Removal

Noisy image

Fourier Spectrum Noise-cleaned image

High Pass Filtering

Original High Pass Filtered

High Frequency Emphasis

+Original High Pass Filtered

High Frequency EmphasisOriginal High Frequency Emphasis

OriginalHigh Frequency Emphasis

Original High pass Filter

High Frequency Emphasis

High Frequency Emphasis +

Histogram Equalization

High Frequency Emphasis