digital image processing: introduction -...

University of Ioannina - Department of Computer Science

Spatial Filtering

Christophoros [email protected]

Digital Image Processing

2

C. Nikou – Digital Image Processing (E12)

Contents

In this lecture we will look at spatial filtering techniques:

– Neighbourhood operations– What is spatial filtering?– Smoothing operations– What happens at the edges?– Correlation and convolution– Sharpening filters– Combining filtering techniques

3


Neighbourhood Operations

Neighbourhood operations simply operate on a larger neighbourhood of pixels than point operationsNeighbourhoods are mostly a rectangle around a central pixelAny size rectangle and any shape filter are possible

Origin x

y Image f (x, y)

(x, y)Neighbourhood

4


Simple Neighbourhood Operations

Some simple neighbourhood operations include:

– Min: Set the pixel value to the minimum in the neighbourhood

– Max: Set the pixel value to the maximum in the neighbourhood

– Median: The median value of a set of numbers is the midpoint value in that set (e.g. from the set [1, 7, 15, 18, 24] 15 is the median). Sometimes the median works better than the average

5


Simple Neighbourhood Operations Example

123 127 128 119 115 130

140 145 148 153 167 172

133 154 183 192 194 191

194 199 207 210 198 195

164 170 175 162 173 151

Original Image x

y

Enhanced Image x

y

6


The Spatial Filtering Process

r s t

u v w

x y z

Origin x

y Image f (x, y)

eprocessed = v*e + r*a + s*b + t*c + u*d + w*f + x*g + y*h + z*i

FilterSimple 3*3

Neighbourhood e 3*3 Filter

a b c

d e f

g h iOriginal Image

Pixels

*

The above is repeated for every pixel in the original image to generate the filtered image

7


Spatial Filtering: Equation Form

∑∑−= −=

++=a

as

b

bttysxftswyxg ),(),(),(

Filtering can be given in equation form as shown aboveNotations are based on the image shown to the left

Imag

es ta

ken

from

Gon

zale

z &

Woo

ds, D

igita

l Im

age

Pro

cess

ing

(200

2)

8


Smoothing Spatial Filters

One of the simplest spatial filtering operations we can perform is a smoothing operation

– Simply average all of the pixels in a neighbourhood around a central value

– Especially useful in removing noise from images

– Also useful for highlighting gross detail

1/9 1/9 1/9

1/9 1/9 1/9

1/9 1/9 1/9

Simple averaging filter

9


Smoothing Spatial Filtering

1/9 1/9 1/91/9 1/9 1/91/9 1/9 1/9

Origin x

y Image f (x, y)

e = 1/9*106 + 1/9*104 + 1/9*100 + 1/9*108 + 1/9*99 + 1/9*98 + 1/9*95 + 1/9*90 + 1/9*85

= 98.3333

FilterSimple 3*3

Neighbourhood106

104

99

95

100 108

98

90 85

1/9 1/9 1/91/9 1/9 1/91/9 1/9 1/9

3*3 SmoothingFilter

104 100 108

99 106 98

95 90 85

Original Image Pixels

*

The above is repeated for every pixel in the original image to generate the smoothed image.

10


Image Smoothing Example

The image at the top left is an original image of size 500*500 pixelsThe subsequent images show the image after filtering with an averaging filter of increasing sizes

– 3, 5, 9, 15 and 35Notice how detail begins to disappear

Imag

es ta

ken

from

Gon

zale

z &

Woo

ds, D

igita

l Im

age

Pro

cess

ing

(200

2)

11


Weighted Smoothing Filters

More effective smoothing filters can be generated by allowing different pixels in the neighbourhood different weights in the averaging function

– Pixels closer to the central pixel are more important

– Often referred to as a weighted averaging

1/162/16

1/16

2/164/16

2/16

1/162/16

1/16

Weighted averaging filter

12


Another Smoothing Example

By smoothing the original image we get rid of lots of the finer detail which leaves only the gross features for thresholding

Imag

es ta

ken

from

Gon

zale

z &

Woo

ds, D

igita

l Im

age

Pro

cess

ing

(200

2)

Original Image Smoothed Image Thresholded Image

13


Averaging Filter Vs. Median Filter Example

Filtering is often used to remove noise from imagesSometimes a median filter works better than an averaging filter

Original ImageWith Noise

Image AfterAveraging Filter

Image AfterMedian Filter

Imag

es ta

ken

from

Gon

zale

z &

Woo

ds, D

igita

l Im

age

Pro

cess

ing

(200

2)

14


Spatial smoothing and image approximation

Spatial smoothing may be viewed as a process for estimating the value of a pixel from its neighbours.

What is the value that “best” approximates the intensity of a given pixel given the intensities of its neighbours?

We have to define “best” by establishing a criterion.Im

ages

take

n fro

m G

onza

lez

& W

oods

, Dig

ital I

mag

e P

roce

ssin

g (2

002)

15


Spatial smoothing and image approximation (cont...)

Imag

es ta

ken

from

Gon

zale

z &

Woo

ds, D

igita

l Im

age

Pro

cess

ing

(200

2)

[ ]2

1

( )N

i

E x i m=

= −∑ [ ]2

1arg min ( )

N

m im x i m

=

⎧ ⎫⇔ = −⎨ ⎬⎩ ⎭∑

0Em∂

=∂

( )1

2 ( ) 0N

ix i m

=

⇔ − − =∑1 1

( )N N

i ix i m

= =

⇔ =∑ ∑

1( )

N

ix i Nm

=

⇔ =∑1

1 ( )N

im x i

N =

⇔ = ∑

A standard criterion is the the sum of squares differences.

The average value

16



Imag

es ta

ken

from

Gon

zale

z &

Woo

ds, D

igita

l Im

age

Pro

cess

ing

(200

2)

1

( )N

i

E x i m=

= −∑1

arg min ( )N

m im x i m

=

⎧ ⎫⇔ = −⎨ ⎬⎩ ⎭∑

0Em∂

=∂

( )1

( ) 0,N

isgn x i m

=

⇔ − − =∑1 0

( ) 0 01 0

xsign x x

x

>⎧⎪= =⎨⎪− <⎩

Another criterion is the the sum of absolute differences.

There must be equal in quantity positive and negative values.

median{ ( )}m x i=

17



Imag

es ta

ken

from

Gon

zale

z &

Woo

ds, D

igita

l Im

age

Pro

cess

ing

(200

2)

– The median filter is non linear:

– It works well for impulse noise (e.g. salt and pepper).

– It requires sorting of the image values.– It preserves the edges better than an average

filter in the case of impulse noise.– It is robust to impulse noise at 50%.

median{ } median{ } median{ }x y x y+ ≠ +

18



Imag

es ta

ken

from

Gon

zale

z &

Woo

ds, D

igita

l Im

age

Pro

cess

ing

(200

2)

Example x[n] 1 1 1 1 1 2 2 2 2 2

Impulse noise x[n] 1 3 1 1 1 2 3 2 2 3

Median(N=3) x[n] - 1 1 1 1 2 2 2 2 -

Average (N=3)

x[n] - 1.7 1.7 1 1.3 2 2.3 2.3 2.2 -

edge

The edge is smoothed

19


Strange Things Happen At The Edges!

Origin x

y Image f (x, y)

e

e

e

e

At the edges of an image we are missing pixels to form a neighbourhood

e e

e

20


Strange Things Happen At The Edges! (cont…)

There are a few approaches to dealing with missing edge pixels:

– Omit missing pixels• Only works with some filters• Can add extra code and slow down processing

– Pad the image • Typically with either all white or all black pixels

– Replicate border pixels– Truncate the image– Allow pixels wrap around the image

• Can cause some strange image artefacts

21


Strange Things Happen At The Edges! (cont…)

OriginalImage

Filtered Image: Zero Padding

Filtered Image: Replicate Edge Pixels

Filtered Image: Wrap Around Edge Pixels

Imag

es ta

ken

from

Gon

zale

z &

Woo

ds, D

igita

l Im

age

Pro

cess

ing

(200

2)

22


Correlation & Convolution

The filtering we have been talking about so far is referred to as correlation with the filter itself referred to as the correlation kernelConvolution is a similar operation, with just one subtle difference

For symmetric filters it makes no difference.

eprocessed = v*e + z*a + y*b + x*c + w*d + u*e + t*f + s*g + r*h

r s t

u v w

x y zFilter

a b c

d e e

f g hOriginal Image

Pixels

*

23


Effect of Low Pass Filtering on White Noise

Let f be an observed instance of the image f0corrupted by noise w:

with noise samples having mean value E[w(n)]=0 and being uncorrelated with respect to location:

0f f w= +

2 ,[ ( ) ( )]

0,m n

E w m w nm n

σ⎧ == ⎨

≠⎩

24


Effect of Low Pass Filtering on White Noise (cont...)

Applying a low pass filter h (e.g. an average filter) by convolution to the degraded image:

The expected value of the output is:

The noise is removed in average.

*g h f= 0*( )h f w= + 0* *h f h w= +

0[ ] [ * ] [ * ]E g E h f E h w= + 0* * [ ]h f h E w= +

0* *0h f h= + 0*h f=

25



What happens to the standard deviation of g?Let where the bar represents filtered versions of the signals, then

( )22 2[ ] [ ]g E g E gσ = −

0 0* *g h f h w f w= + = +

2 20 0[( ) ] ( )f w f= Ε + −

2 2 20 0 0[( ) ( ) 2 ] ( )f w f w f= Ε + + −

20[( ) ] 2 [ ] [ ]E w E f E w= + 2[( ) ]E w=

26



Considering that h is an average filter, we have at pixel n:

Therefore,

2[( ( )) ]E w n

( ) ( * )( )w n h w n=( )

1 ( )k n

w kN ∈Γ

= ∑

2

( )

1 ( )k n

E w kN ∈Γ

⎡ ⎤⎛ ⎞⎢ ⎥= ⎜ ⎟⎢ ⎥⎝ ⎠⎣ ⎦

∑

27



2

( )

1 ( )k n

E w kN ∈Γ

⎡ ⎤⎛ ⎞⎢ ⎥⎜ ⎟⎢ ⎥⎝ ⎠⎣ ⎦

∑

{ }22

( )

1 ( )k n

E w kN ∈Γ

⎡ ⎤= ⎣ ⎦∑

[ ]2( ) ( )

2 ( ) ( )l n m n

m l

E w n l w n mN ∈Γ ∈Γ

≠

+ − −∑ ∑

Sum of squares

Cross products

28



{ }2 22 2

( ) ( )

1 1( )k n k n

E w kN N

σ∈Γ ∈Γ

⎡ ⎤ =⎣ ⎦∑ ∑

[ ]2( ) ( )

2 ( ) ( ) 0l n m n

m l

E w n l w n mN ∈Γ ∈Γ

≠

+ − − =∑ ∑

Sum of squares

Cross products (uncorrelated as m∫l)

29



Finally, substituting the partial results:

The effect of the noise is reduced.This processing is not optimal as it also smoothes image edges.

2

2

( )

1 ( )gk n

E w kN

σ∈Γ

⎡ ⎤⎛ ⎞⎢ ⎥= ⎜ ⎟⎢ ⎥⎝ ⎠⎣ ⎦

∑ 22

( )

1k nN

σ∈Γ

= ∑

22

1 NN

σ=2

Nσ

=

30


Sharpening Spatial Filters

Previously we have looked at smoothing filters which remove fine detailSharpening spatial filters seek to highlight fine detail

– Remove blurring from images– Highlight edges

Sharpening filters are based on spatial differentiation

31


Spatial Differentiation

Differentiation measures the rate of change of a functionLet’s consider a simple 1 dimensional example

Imag

es ta

ken

from

Gon

zale

z &

Woo

ds, D

igita

l Im

age

Pro

cess

ing

(200

2)

32


Spatial DifferentiationIm

ages

take

n fro

m G

onza

lez

& W

oods

, Dig

ital I

mag

e P

roce

ssin

g (2

002)

A B

33


Derivative Filters Requirements

First derivative filter output– Zero at constant intensities– Non zero at the onset of a step or ramp– Non zero along ramps

•Second derivative filter output– Zero at constant intensities– Non zero at the onset and end of a step or ramp– Zero along ramps of constant slope

34


1st Derivative

The formula for the 1st derivative of a function is as follows:

It’s just the difference between subsequent values and measures the rate of change of the function

)()1( xfxfxf

−+=∂∂

35


1st Derivative (cont.)

The gradient points in the direction of most rapid increase in intensity.

• The gradient of an image:

•

Gradient direction

The edge strength is given by the gradient magnitude

Source: Steve Seitz

36


1st Derivative (cont.)

fx∂∂

fy∂∂

f∇

37


1st Derivative (cont…)

0

1

2

3

4

5

6

7

8

-8

-6

-4

-2

0

2

4

6

8

5 5 4 3 2 1 0 0 0 6 0 0 0 0 1 3 1 0 0 0 0 7 7 7 7

0 -1 -1 -1 -1 -1 0 0 6 -6 0 0 0 1 2 -2 -1 0 0 0 7 0 0 0

38


2nd Derivative

The formula for the 2nd derivative of a function is as follows:

Simply takes into account the values both before and after the current value

)(2)1()1(2

2

xfxfxfxf

−−++=∂∂

39


2nd Derivative (cont…)

0

1

2

3

4

5

6

7

8

5 5 4 3 2 1 0 0 0 6 0 0 0 0 1 3 1 0 0 0 0 7 7 7 7

-15

-10

-5

0

5

10

-1 0 0 0 0 1 0 6 -12 6 0 0 1 1 -4 1 1 0 0 7 -7 0 0

40


Using Second Derivatives For Image Enhancement

Edges in images are often ramp-like transitions– 1st derivative is constant and produces thick

edges – 2nd derivative zero crosses the edge (double

response at the onset and end with opposite signs)

A common sharpening filter is the Laplacian– Isotropic– One of the simplest sharpening filters– We will look at a digital implementation

41


The Laplacian

The Laplacian is defined as follows:

where the partial 1st order derivative in the xdirection is defined as follows:

and in the y direction as follows:

yf

xff 2

2

2

22

∂∂

+∂∂

=∇

),(2),1(),1(2

2

yxfyxfyxfxf

−−++=∂∂

),(2)1,()1,(2

2

yxfyxfyxfyf

−−++=∂∂

42


The Laplacian (cont…)

So, the Laplacian can be given as follows:

We can easily build a filter based on this

),1(),1([2 yxfyxff −++=∇)]1,()1,( −+++ yxfyxf

),(4 yxf−

0 1 0

1 -4 1

0 1 0

43


The Laplacian (cont…)

Applying the Laplacian to an image we get a new image that highlights edges and other discontinuities

Imag

es ta

ken

from

Gon

zale

z &

Woo

ds, D

igita

l Im

age

Pro

cess

ing

(200

2)

OriginalImage

LaplacianFiltered Image


Scaled for Display

44


But That Is Not Very Enhanced!

The result of a Laplacian filtering is not an enhanced imageWe have to do more work in order to get our final imageSubtract the Laplacian result from the original image to generate our final sharpened enhanced image


Scaled for Display

Imag

es ta

ken

from

Gon

zale

z &

Woo

ds, D

igita

l Im

age

Pro

cess

ing

(200

2)

fyxfyxg 2),(),( ∇−=

45


Laplacian Image Enhancement

In the final sharpened image edges and fine detail are much more obvious

Imag

es ta

ken

from

Gon

zale

z &

Woo

ds, D

igita

l Im

age

Pro

cess

ing

(200

2)

- =

OriginalImage


SharpenedImage

46


Laplacian Image EnhancementIm

ages

take

n fro

m G

onza

lez

& W

oods

, Dig

ital I

mag

e P

roce

ssin

g (2

002)

47


Simplified Image Enhancement

The entire enhancement can be combined into a single filtering operation

),1(),1([),( yxfyxfyxf −++−=)1,()1,( −+++ yxfyxf

)],(4 yxf−

fyxfyxg 2),(),( ∇−=

),1(),1(),(5 yxfyxfyxf −−+−=)1,()1,( −−+− yxfyxf

48


Simplified Image Enhancement (cont…)

This gives us a new filter which does the whole job for us in one step

0 -1 0

-1 5 -1

0 -1 0

Imag

es ta

ken

from

Gon

zale

z &

Woo

ds, D

igita

l Im

age

Pro

cess

ing

(200

2)

49


Simplified Image Enhancement (cont…)Im

ages

take

n fro

m G

onza

lez

& W

oods

, Dig

ital I

mag

e P

roce

ssin

g (2

002)

50


Variants On The Simple Laplacian

There are lots of slightly different versions of the Laplacian that can be used:

0 1 0

1 -4 1

0 1 0

1 1 1

1 -8 1

1 1 1

-1 -1 -1

-1 9 -1

-1 -1 -1

SimpleLaplacian

Variant ofLaplacian

Imag

es ta

ken

from

Gon

zale

z &

Woo

ds, D

igita

l Im

age

Pro

cess

ing

(200

2)

51


Unsharp masking

Used by the printing industrySubtracts an unsharped (smooth) image from the original image f(x,y).

–Blur the image b(x,y)=Blur{f(x,y)}

–Subtract the blurred image from the original (the result is called the mask)

gmask(x,y)=f(x,y)-b(x,y)–Add the mask to the original

g(x,y)=f(x,y)+k gmask(x,y) with k non negative

Imag

es ta

ken

from

Gon

zale

z &

Woo

ds, D

igita

l Im

age

Pro

cess

ing

(200

2)

52


Unsharp masking (cont...)Im

ages

take

n fro

m G

onza

lez

& W

oods

, Dig

ital I

mag

e P

roce

ssin

g (2

002) Sharping mechanism

When k>1 the process is referred to as highboost filtering

53


Unsharp masking (cont...)Im

ages

take

n fro

m G

onza

lez

& W

oods

, Dig

ital I

mag

e P

roce

ssin

g (2

002) Original image

Blurred image

Mask

Unsharp masking

Highboost filtering (k=4.5)

54


1st Derivative Filtering

Implementing 1st derivative filters is difficult in practiceFor a function f(x, y) the gradient of f at coordinates (x, y) is given as the column vector:

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

∂∂∂∂

=⎥⎦

⎤⎢⎣

⎡=∇

yfxf

GG

y

xf

55


1st Derivative Filtering (cont…)

The magnitude of this vector is given by:

For practical reasons this can be simplified as:

)f(∇=∇ magf

[ ] 2122

yx GG +=

21

22

⎥⎥⎦

⎤

⎢⎢⎣

⎡⎟⎟⎠

⎞⎜⎜⎝

⎛∂∂

+⎟⎠⎞

⎜⎝⎛∂∂

=yf

xf

yx GGf +≈∇

56


1st Derivative Filtering (cont…)

There is some debate as to how best to calculate these gradients but we will use:

which is based on these coordinates

( ) ( )321987 22 zzzzzzf ++−++≈∇

( ) ( )741963 22 zzzzzz ++−+++

z1 z2 z3

z4 z5 z6

z7 z8 z9

57


Sobel Operators

Based on the previous equations we can derive the Sobel Operators

To filter an image it is filtered using both operators the results of which are added together

-1 -2 -1

0 0 0

1 2 1

-1 0 1

-2 0 2

-1 0 1

58


Sobel Example

Sobel filters are typically used for edge detection

Imag

es ta

ken

from

Gon

zale

z &

Woo

ds, D

igita

l Im

age

Pro

cess

ing

(200

2) An image of a contact lens which is enhanced in order to make defects (at four and five o’clock in the image) more obvious

59


1st & 2nd Derivatives

Comparing the 1st and 2nd derivatives we can conclude the following:

– 1st order derivatives generally produce thicker edges (if thresholded at ramp edges)

– 2nd order derivatives have a stronger response to fine detail e.g. thin lines

– 1st order derivatives have stronger response to grey level step

– 2nd order derivatives produce a double response at step changes in grey level (which helps in detecting zero crossings)

60


Combining Spatial Enhancement Methods

Successful image enhancement is typically not achieved using a single operationRather we combine a range of techniques in order to achieve a final resultThis example will focus on enhancing the bone scan to the right

Imag

es ta

ken

from

Gon

zale

z &

Woo

ds, D

igita

l Im

age

Pro

cess

ing

(200

2)

61


Combining Spatial Enhancement Methods (cont…)

Imag

es ta

ken

from

Gon

zale

z &

Woo

ds, D

igita

l Im

age

Pro

cess

ing

(200

2)

Laplacian filter ofbone scan (a)

Sharpened version ofbone scan achievedby subtracting (a)and (b) Sobel filter of bone

scan (a)

(a)

(b)

(c)

(d)

62



Imag

es ta

ken

from

Gon

zale

z &

Woo

ds, D

igita

l Im

age

Pro

cess

ing

(200

2)

The product of (c)and (e) which will beused as a mask

Sharpened imagewhich is sum of (a)and (f)

Result of applying apower-law trans. to(g)

(e)

(f)

(g)

(h)

Image (d) smoothed witha 5*5 averaging filter

63



Compare the original and final images

Imag

es ta

ken

from

Gon

zale

z &

Woo

ds, D

igita

l Im

age

Pro

cess

ing

(200

2)

64


Summary

In this lecture we have looked at the idea of spatial filtering and in particular:

– Neighbourhood operations– The filtering process– Smoothing filters– Dealing with problems at image edges when

using filtering– Correlation and convolution– Sharpening filters– Combining filtering techniques

digital image processing: introduction -...

Documents