Fourier, filtering, smoothing, and noise

Nuno Vasconcelos ECE Department, UCSD

(with thanks to David Forsyth)

2

Plan for todaywe have talked about some theoretic of 2D DSPtoday we will see examples with real imageswe will:• talk about the discrete-space Fourier transform• sampling in 2D• visualize concepts such as frequency spectra and aliasing• see that very important vision operations are really just filtering,

with some post processing• introduce some important filters that are widely used in practice • talk about the impact of noise and how to deal with it

3

The Discrete-Space Fourier Transformis, once again, a straightforward extension of the 1D Discrete-Time Fourier Transform

properties:• basically the same as in 1D (see table in Lim, page 25)• only novelty is separability (homework)

2121221

2121

2211

2211

1 2

),()2(

1],[

],[),(

ωωωωπ

ωω

ωω

ωω

ddeeXnnx

eennxX

njnj

njnj

n n

∫∫

∑∑

=

= −−

)()(),(][][],[ 221121221121 ωωωω XXXnxnxnnx =↔=

4

Image spectrumtwo images, the magnitude, and phase of their FTs

5

Phase and Magnitudecurious fact• all natural images have about the same magnitude transform• monotonically decaying with frequency

hence, phase seems to matter, but magnitude largely doesn’twe have seen this a little bit in the first problem settook two pictures, swap the phase transforms, compute the inversehere is another example

22

21

211),(ωω

ωω+

∝X

6

The importance of phaseReconstruction with zebra

phase, cheetah magnitudeReconstruction with cheetah

phase, zebra magnitude

7

Sampling in 2Dconsider an analog signal xc(t1,t2) and let its analog Fourier transform be Xc(Ω1,Ω2)• we use capital Ω to emphasize that this is analog frequency

sample with period (T1,T2) to obtain a discrete-space signal

222111 ;2121 ),(],[TntTntc ttxnnx

===

8

Sampling in 2Drelationship between the Discrete-Space FT of x[n1,n2]and the FT of xc(t1,t2) is simple extension of 1D result

DSFT of x[n1,n2] FT of xc(ω1,ω2) “discrete spectrum” “analog spectrum”

Discrete Space spectrum is sum of replicas of analog spectrum• in the “base replica” the analog frequency Ω1 (Ω2) is mapped into

the digital frequency Ω1T1 (Ω2T2)• discrete spectrum has periodicity (2π,2π)

∑ ∑∞

−∞=

∞

−∞=⎟⎟⎠

⎞⎜⎜⎝

⎛ −−=

1 2 1

11

1

11

2121

2,21),(r r

c Tr

TrX

TTX πωπωωω

9

For exampleω2

ω1

Ω’T1

Ω”T2

ω1

ω1

Ω’

2π−Ω”T2

2π2π

2π

2π

2π−Ω’T1

Ω2

Ω1

Ω’

Ω”

2

1

''''''

TT

Ω=→ΩΩ=→Ω

βα

no aliasing if

⇔⎩⎨⎧

Ω−≤ΩΩ−≤Ω

22

11

'2'''2'TTTT

ππ

⎩⎨⎧

Ω≤Ω≤

⇔''/'/

2

1

ππ

TT

10

Aliasingthe frequency (Ω’/π,Ω’’/π)is the critical sampling frequencybelow it we havealiasing

this is just like the 1D case, but now there are more possibilities for overlap

ω2

ω1

11

Reconstructionif there is no aliasing we can recover the signal in a way similar to the 1D case

note: in 2D there are many more possibilities than in 1D• e.g. the sampling grid does not have to be rectangular, e.g.

hexagonal sampling when T2 = T1/sqrt(3) and

• in practice, however, one usually adopts the rectangular grid

( )

( )

( )

( )∑ ∑∞

−∞=

∞

−∞= −

−

−

−=

1 2222

2

2222

1111

1111

2121

sinsin],[),(

n nc

TntT

TntT

TntT

TntTnnxtty π

π

π

π

⎩⎨⎧

= ==

otherwise0

odd or even both21;2121

,),(],[ 222111

nnttxnnx TntTntc

12

Sampling and aliasing in 2Din summary, sampling is not very different from the 1D case

but aliasing is a lot more fun to look at in imagesit shows up in video too (the wagon wheel effect)

13

a sequence of images obtained by down-sampling without any filteringaliasing: the low-frequency parts are replicated throughout the low-res image

14

The role of smoothing

none

some

a lot

too little leads to aliasingtoo much leads to loss of information

15

Linear Filteringsmoothing is implemented with linear filtersgiven an image x(n1,n2), filtering is the process ofconvolving it with a kernel h(n1,n2)

some very common operations in image processing are nothing but filtering, e.g.• smoothing an image by low-pass filtering• contrast enhancement by high pass filtering• finding image derivatives• noise reduction

),(),(),( 2211212121

knknhkkxnnykk

−−= ∑

16

Popular filtersbox function

Fourier transform of a box is the sinc, low-pass filter

side-lobes produce artifacts, smoothed image does not look like the result of defocusing

⎩⎨⎧ −≤≤−≤≤

=otherwise

NnNnnnR xNN 0

10,10,1),( 2211

2121

F

17

Example: Smoothing by Averaging

18

Camera defocusingif you point an out-of-focus camera at a very small white light (e.g. a light-bulb) at night, you get something like thisthe light can be thought of as an impulsethis must be the impulse responsewell approximated by a Gaussianmore natural filter for image blur than the box

⎟⎟⎠

⎞⎜⎜⎝

⎛ +−= 2

22

2 2exp

21),(

σπσyxyxh

19

The Gaussianthe discrete space version is

obviously separable

h(n1,n2) has Fourier transform

⎟⎟⎠

⎞⎜⎜⎝

⎛ +−= 2

22

21

221 2exp

21),(

σπσnnnnh

44344214434421)(

2

)(

221

2

2

22

1

2

21

21

21),(

nh

n

nh

n

eennh σσ

σπσπ

−−×=

⎟⎟⎠

⎞⎜⎜⎝

⎛ +−=

2)(exp),(

22

21

2

21ϖϖσϖϖH

20

The Gaussian filterthe Fourier transform of a Gaussian is a Gaussian(σx,σy) ∝ (1/σw1,1/σw2)

F

note that there are no annoying side-lobes

21

Smoothing with a Gaussian

22

Role of the variancethe variance controls the amount of smoothingeach column shows different realizations of an image of gaussian noiseeach row shows smoothing with gaussians of different σ

23

Gradients and edgesfor image understanding, one of the problems is that there is too much information in an image just smoothing is not good enoughhow to detect important (most informative) image points?note that derivatives are large at points of great change• changes in reflectance (e.g. checkerboard pattern)• change in object (an object boundary is different from background)• change in illumination (the boundary of a shadow)

these are usually called edge pointsdetecting them could be useful for various problems• segmentation: we want to know what are object boundaries• recognition: cartoons are easy to recognize and terribly efficient to

transmit

24

The importance of edgeslight

reflectance

boundary

25

Gradientsfor a 2D function, f(x,y) the gradient at a point (x0,y0)

is the direction of greatest increase at that pointthe gradient magnitude

measures the rate of changeit is large at edges!

T

yxyfyx

xfyxf ⎟⎟

⎠

⎞⎜⎜⎝

⎛∂∂

∂∂

=∇ ),(),,(),( 000000

( )Tyx yxfyxf ),(),,( 0000=

f(x,y)

),( 00 yxf∇

),( 11 yxf∇

– large gradient magnitude

– small gradient magnitude

2

)⎟⎟⎠

⎞00

2

002

00 ,(),(),( ⎜⎜⎝

⎛∂∂

+⎟⎠⎞

⎜⎝⎛∂∂

=∇ yxyfyx

xfyxf

26

Derivatives and convolutionrecall that a derivative is defined as

linear and shift invariant, so must be the result of a convolution.we could approximate as

where the derivative kernel is

( ) ( ) ( )x

xfxxfxxf

x ∆−∆+

=∂

∂→∆ 0

lim

( ) ( ) ( ) ( ) ( ) )(*11

1 nhfnfnfnfnfnnf

=−+=−+

=∂

∂

)()1()( nnnh δδ −+=

27

Finite difference kernelsin two dimensions we have various possible kernelse.g. , N1=2, N2=3, derivative along n1, (line n2=k)(horizontal)

derivative along n2, (line n1=k) (vertical)

derivative along line n1=n2 (diagonal)

n1

n21 -11 -11 -1

0 01 -10 0

n1

n2

0 -1 00 1 0

-1 -1 -11 1 1

n1

n20 0 -10 0 01 0 0

1 -1 00 1 -10 0 1

28

Finite difference kernelsnote that, when

h(n1,n2)=

we have

derivative is a high-pass filterhw: check that this holds for all othersintuitive, because a derivative is a measure of the rate of change of a function

222111 ϖϖϖ jjj

eee ⎟⎟⎠

⎞⎜⎜⎝

⎛−=

−

0 01 -10 0

211

2sin2

ϖϖ je⎟⎠⎞

⎜⎝⎛=

( ) 1, 121 −= ϖϖϖ jeH

29

Finite differencesQ: which one do we have here? (gray=0,white=+,dark=-)

30

Finite differences and noisebecause they perform high-pass filtering, finite difference filters respond strongly to noisegenerally, the larger the noise the stronger the responsefor noisy images it is usually best to apply some smoothing before computing derivativeswhat do mean by noise? we only consider the simplest model• independent stationary additive Gaussian noise• the noise value at each pixel is given by an independent draw

from the same normal probability distribution

),0(~),,(),(),( 2212121 σεε Nnnnnxnny +=

31

sigma=1

32

sigma=16

33

Finite differences responding to noise

increasing noise variance

note that as the noise variance increases the estimates of the image derivative are also very noisyQ: would a larger filter do better?

34

The response of a linear filter to noisesuppose we apply filter h(n1,n2) to

by linearity

consider the noise term

by linearity of expectation

),0(~),,(),(),( 2212121 σεε Nnnnnxnny +=

noiseoutputsignaloutput

hxhyh

ε*** +=

),(),(),( 2211212121

knknhkknnkk

−−= ∑εν

[ ] [ ] 0),(),(),( 2211212121

=−−= ∑ knknhkkEnnEkk

εν

35

The response of a linear filter to noiseand since the ε(n1,n2) are independent

output error has• zero mean• variance that increases with the size of the filter and the input

variance

increasing the filter size would definitely not help!

[ ] [ ] ),(),(var),(var 22112

212121

knknhkknnkk

−−=∑ εν

),( 221122

21

knknhkk

−−= ∑σ

36

Smoothing reduces noisenoise has a lot of high-frequenciesstrategy:1. start by low-pass filtering, to suppress

noise2. compute derivative on smoothed image

i.e. for a smoothing filter g(n1,n2)compute

note that, by associativity of convolution, this is equal to

i.e. filter the image with the filter h*g

g(n1,n2)

h(n1,n2)

( )xgh **

( ) xgh **smoothed derivatives

37

The derivative of a GaussianDoG along n1let’s consider, for example,

in which case

is a difference of two Gaussiansthis is the derivative of a Gaussian (DoG) filterfor other definitions of h we have a similar result

),(),1(),( 212121 nnnnnnh δδ −+=

),(),1(),(* 212121 nngnngnngh −+=DoG along n2

38

Smoothed derivativesno smoothing

with smoothing

39

Choosing the right scalethe scale of the smoothing filter affects • derivative estimates, • the semantics of the derivative image

trade-off between noise and ability to detect detail

1 pixel 3 pixels 7 pixels

40

41

Gradients and edgesby now we have a good idea of how to differentiate imagesremember that edges are points of large gradient magnitudeedge detection strategy1. determine magnitude of image gradient

2. mark points where gradient magnitude is particularly large wrt neighbours(ideally, curves of such points)

– large gradient magnitude

– small gradient magnitude

2

00

2

002

00 ),(),(),( ⎟⎟⎠

⎞⎜⎜⎝

⎛∂∂

+⎟⎠⎞

⎜⎝⎛∂∂

=∇ yxyfyx

xfyxf

42

Looks easy butthree major issues (to discuss next class):• 1) gradient magnitude at different scales is different (see below);

which should we choose?• 2) gradient magnitude is large along thick trail; what are the

significant points?• 3) how do we link the relevant points up into curves?