cvision 3 image pre-processing: filtering & enhancementmcoimbra/lectures/mapi... · median...

Cvision – 3

Image Pre-processing:

Filtering & Enhancement

João Paulo Silva Cunha(jcunha@det.ua.pt)

IEETA / Universidade de Aveiro

jcunha@det.ua.pt 2

Outline

• Frequency Space

• Spatial Convolution

• Spatial filters

– Gaussian; Oriented Gaussian; Mean; Median

• Frequency domain filtering

• Edge detection

Acknowledgements: Most of this course is based on the excellent courses offered by Prof. Shree Nayar at

Columbia University, USA and by Prof. Srinivasa Narasimhan at CMU, USA. This was also based on Prof.

Miguel Coimbra’s slides. Please acknowledge the original source when reusing these slides for academic

purposes.

jcunha@det.ua.pt 3

Topic: Frequency Space

• Frequency Space

• Spatial Convolution

• Spatial filters

• Frequency domain filtering

• Edge detection

jcunha@det.ua.pt 4

How does this apply to images?

• We have defined the

Fourier Transform as

• But images are:

– Discrete.

– Two-dimensional.

What a computer sees

dxexfuF iux

jcunha@det.ua.pt 5

2D Discrete FT

• In a 2-variable case, the discrete FT pair

is:1

0

1

0

)]//(2exp[),(1

),(M

x

N

y

NvyMuxjyxfMN

vuF

1

0

1

0

)]//(2exp[),(),(M

u

N

v

NvyMuxjvuFyxf

For u=0,1,2,…,M-1 and v=0,1,2,…,N-1

For x=0,1,2,…,M-1 and y=0,1,2,…,N-1

AND:

New matrix

with the

same size!

jcunha@det.ua.pt 6

Frequency Space

• Image Space

– f(x,y)

– Intuitive

• Frequency Space

– F(u,v)

– What does this mean?

jcunha@det.ua.pt 7

Power distribution

An image (500x500 pixels) and its Fourier spectrum. The super-imposed circles have

radii values of 5, 15, 30, 80, and 230, which respectively enclose 92.0, 94.6, 96.4, 98.0,

and 99.5% of the image power.

jcunha@det.ua.pt 8

Power distribution

• Most power is in low frequencies.

• Means we are using more of this:

And less of this:

To represent our signal.

• Why?

jcunha@det.ua.pt 9What does this mean??

jcunha@det.ua.pt 10

Horizontal and Vertical Frequency

• Frequencies:

– Horizontal frequencies

correspond to

horizontal gradients.

– Vertical frequencies

correspond to vertical

gradients.

• What about diagonal

lines?

jcunha@det.ua.pt 11

jcunha@det.ua.pt 12If I discard high-frequencies, I get a blurred image...

Why?

jcunha@det.ua.pt 13

Why bother with FT?

• Great for filtering.

• Great for compression.

• In some situations: Much faster than

operating in the spatial domain.

• Convolutions are simple multiplications in

Frequency space!

• ...

jcunha@det.ua.pt 14

Topic: Spatial Convolution

• Frequency Space

• Spatial Convolution

• Spatial filters

• Frequency domain filtering

• Edge detection

jcunha@det.ua.pt 15

Convolution

fg

gf Eric Weinstein’s Math World

jcunha@det.ua.pt 16

Convolution

1 2-1-2

xc-1 1

1

xb

-1 1

1xa

bac

1

hfgdxhfxg

jcunha@det.ua.pt 17

Convolution Kernel – Impulse Response

• What h will give us g = f ?

f gh hfg

Dirac Delta Function (Unit Impulse)

x

21

0

jcunha@det.ua.pt 18

Point Spread Function

• Ideally, the optical system should be a Dirac delta function.

• However, optical systems are never ideal.

• Point spread function of Human Eyes.

Optical

Systemscene image

x xPSFOptical

Systempoint source point spread function

x

21

0

jcunha@det.ua.pt 19

Point Spread Function

normal vision myopia hyperopia

Images by Richmond Eye Associates

jcunha@det.ua.pt 20

Properties of Convolution

• Commutative

• Associative

• Cascade system

abba

cbacba

f g1h 2h

f g21 hh

f g12 hh

jcunha@det.ua.pt 21

Fourier Transform and Convolution

hfg dxexguG uxi2

dxdexhf uxi2

dxexhdef xuiui 22

'' '22 dxexhdef uxiui

Let Then

uHuF

Convolution in spatial domain

Multiplication in frequency domain

jcunha@det.ua.pt 22

Fourier Transform and Convolution

hfg FHG

fhg HFG

Spatial Domain (x) Frequency Domain (u)

So, we can find g(x) by Fourier transform

g f h

G F H

FT FTIFT

jcunha@det.ua.pt 23

Topic: Spatial filters

• Frequency Space

• Spatial Convolution

• Spatial filters

• Frequency domain filtering

• Edge detection

jcunha@det.ua.pt 24

Images are Discrete and Finite

yxf , yxg ,yxh ,

f

MN

i

j

h

ConvolutionM

m

N

n

njmihnmfjig1 1

,,,

Fourier Transform

M

m

N

n

N

nv

M

mui

enmfvuF1 1

2

,,

Inverse Fourier Transform

M

u

N

v

N

lv

M

kui

evuFMN

lkf1 1

2

,1

,

jcunha@det.ua.pt 25

Spatial Mask

• Simple way to

process an image.

• Mask defines the

processing function.

• Represents a spatial

convolution that

corresponds to a

multiplication in

frequency domain.

Convolution – Mask

‘slides’ over the image

Mask Image

jcunha@det.ua.pt 26

Example (1 pixel)

• Each mask position

has weight w.

• The result of the

operation for each

pixel is given by:

1 2 1

0 0 0

-1 -2 -1

2 2 2

4 4 4

4 5 6

Mask Image

=1*2+2*2+1*2+…

=8+0-20

=-12

a

as

b

bt

tysxftswyxg ),(),(),(

jcunha@det.ua.pt 27

Definitions

• Spatial filters

– Use a mask (kernel) over an image region.

– Work directly with pixels.

– As opposed to: Frequency filters.

• Advantages

– Simple implementation: convolution with the kernel function.

– Different masks offer a large variety of functionalities.

jcunha@det.ua.pt 28

Example (360x500 image)

jcunha@det.ua.pt 29

Example (360x500 image)

jcunha@det.ua.pt 30

Example (360x500 image)

jcunha@det.ua.pt 31

Example (360x500 image)

jcunha@det.ua.pt 32

Convolution Border Issues

jcunha@det.ua.pt 33

• A Gaussian kernel gives less weight to pixels further from the center of the window

• This kernel is an approximation of a Gaussian function:

Gaussian Smoothing

1 2 1

2 4 2

1 2 1

jcunha@det.ua.pt 34

2

8.2 4

original

jcunha@det.ua.pt 35

Gaussian Smoothing (maths)

2

22

2

1

22

1,

ji

ejihGaussian

kernel

N pixels

Filter size N …can be very large

(truncate, if necessary)

1 1

2

1

2,

2

1,

2

22

m n

nm

njmifejig

2D Gaussian is separable!

1 1

2

1

2

1

2,

2

1,

2

2

2

2

m n

nm

njmifeejig

Use two 1D

Gaussian

Filters!

jcunha@det.ua.pt 36

Finite kernel support

jcunha@det.ua.pt 37

Oriented Gaussian Filters

jcunha@det.ua.pt 38

Oriented Gaussian Filters

jcunha@det.ua.pt 39

Oriented Gaussian Filters

jcunha@det.ua.pt 40

jcunha@det.ua.pt 41

Mean Filtering

• We are degrading the

energy of the high spatial

frequencies of an image

(low-pass filtering).

– Makes the image

‘smoother’.

– Used in noise reduction.

• Can be implemented with

spatial masks or in the

frequency domain. 1 1 1

1 1 1

1 1 1

1/9 1/9 1/9

1/9 1/9 1/9

1/9 1/9 1/9

jcunha@det.ua.pt 42Mean filter Gaussian filter

jcunha@det.ua.pt 43

http://www.michaelbach.de/ot/cog_blureffects/index.html

by Charles Allen Gillbert by Harmon & Julesz

jcunha@det.ua.pt 44

http://www.michaelbach.de/ot/cog_blureffects/index.html

jcunha@det.ua.pt 45

Median Filter

• Smoothing is averaging

(a) Blurs edges

(b) Sensitive to outliers

(a)

(b)

– Sort values around the pixel

– Select middle value (median)

– Non-linear (Cannot be implemented with convolution)

• Median filtering

12N

sort median

jcunha@det.ua.pt 46

3x3

5x5

7x7

Salt and pepper noise Gaussian noise

jcunha@det.ua.pt 47

Topic: Frequency domain filtering

• Frequency Space

• Spatial Convolution

• Spatial filters

• Frequency domain filtering

• Edge detection

jcunha@det.ua.pt 48

Image Processing in the Fourier

Domain

Does not look anything like what we have seen

Magnitude of the FT

jcunha@det.ua.pt 49

Lets the low frequencies

pass and eliminates the

high frequencies.

Generates image with overall

shading, but not much detail

Low-pass Filtering

jcunha@det.ua.pt 50

Lets through the high

frequencies (the detail),

but eliminates the low

frequencies (the overall

shape). It acts like an

edge enhancer.

High-pass Filtering

jcunha@det.ua.pt 51

Boosting High Frequencies

jcunha@det.ua.pt 52

Topic: Edge detection

• Frequency Space

• Spatial Convolution

• Spatial filters

• Frequency domain filtering

• Oriented Gaussian filters

• Edge detection

jcunha@det.ua.pt 53

Edge Detection

• Convert a 2D image into a set of curves– Extracts

salient features of the scene

– More compact than pixels

jcunha@det.ua.pt 54

Origin of Edges

• Edges are caused by a variety of factors

depth discontinuity

surface color discontinuity

illumination discontinuity

surface normal discontinuity

jcunha@det.ua.pt 55

How can you tell that a pixel is

on an edge?

jcunha@det.ua.pt 56

Image derivatives

jcunha@det.ua.pt 57

Image derivatives

jcunha@det.ua.pt 58

Smooth derivative

jcunha@det.ua.pt 59

Smooth derivative

jcunha@det.ua.pt 60

Smooth derivative

jcunha@det.ua.pt 61

Smooth but …

jcunha@det.ua.pt 62

Gradient

• Gradient equation:

• The edge strength is given

by the gradient magnitude

jcunha@det.ua.pt 63

Gradient

jcunha@det.ua.pt 64

Directional derivatives

jcunha@det.ua.pt 65

Directional derivatives

jcunha@det.ua.pt 66

Laplacian Operator

jcunha@det.ua.pt 67

Operators for edge detection

jcunha@det.ua.pt 68

What is an edge ?

jcunha@det.ua.pt 69

Gradient of an edge

jcunha@det.ua.pt 70

Gradient operator for edge

detection

jcunha@det.ua.pt 71

Gradient operator for edge

detection

jcunha@det.ua.pt 72

Discrete Edge Operators

• How can we differentiate a discrete image?

Finite difference approximations:

1, jiI 1,1 jiI

jiI , jiI ,1

jijijiji IIIIx

I,,11,1,1

2

1

jijijiji IIIIy

I,1,,11,1

2

1

1 1

1 12

1

x

I 1 1

1 12

1

y

I

Convolution masks :

jcunha@det.ua.pt 73

1, jiI 1,1 jiI

jiI , jiI ,1

1,1 jiI

jiI ,1

1,1 jiI 1, jiI 1,1 jiI

• Second order partial derivatives:

jijiji IIIx

I,1,,122

2

21

1,,1,22

2

21

jijiji IIIy

I• Laplacian :

2

2

2

22

y

I

x

II

2

2 1I

Convolution masks :

1 0

4 1

0

1

0 1 0

or26

14 1

20 4

1

4

1 4 1

Discrete Edge Operators

(more accurate)

jcunha@det.ua.pt 74

The Sobel Operators

• Better approximations of the gradients exist

– The Sobel operators below are commonly used

-1 0 1

-2 0 2

-1 0 1

1 2 1

0 0 0

-1 -2 -1

jcunha@det.ua.pt 75

Comparing Edge Operators

-1 0 1

-1 0 1

-1 0 1

1 1 1

0 0 0

-1 -1 1

Gradient:

Roberts (2 x 2):

Sobel (3 x 3):

Sobel (5 x 5):-1 -2 0 2 1

-2 -3 0 3 2

-3 -5 0 5 3

-2 -3 0 3 2

-1 -2 0 2 1

1 2 3 2 1

2 3 5 3 2

0 0 0 0 0

-2 -3 -5 -3 -2

-1 -2 -3 -2 -1

0 1

-1 0

1 0

0 -1

Good Localization

Noise Sensitive

Poor Detection

Poor Localization

Less Noise Sensitive

Good Detection

jcunha@det.ua.pt 76

Effects of Noise

• Consider a single row or column of the image

– Plotting intensity as a function of position gives a signal

Where is

the edge??

jcunha@det.ua.pt 77Where is the edge?

Solution: Smooth First

Look for peaks in

jcunha@det.ua.pt 78

Derivative Theorem of Convolution

…saves us one operation.

jcunha@det.ua.pt 79

Laplacian of Gaussian (LoG)

Laplacian of Gaussian operator

Where is the edge? Zero-crossings of bottom graph !

fhx

fhx 2

2

2

2

Laplacian of Gaussian

jcunha@det.ua.pt 80

2D Gaussian Edge Operators

Laplacian of GaussianGaussian

Derivative of Gaussian (DoG)

Mexican Hat (Sombrero)

• is the Laplacian operator:

jcunha@det.ua.pt 81

Canny Edge Operator

• Smooth image I with 2D Gaussian:

• Find local edge normal directions for each pixel

• Compute edge magnitudes

• Locate edges by finding zero-crossings along the edge normal

directions (non-maximum suppression)

IG

IGn

IG

02

2

n

IG

IG

jcunha@det.ua.pt 82

Canny Edge Operator

jcunha@det.ua.pt 83

Canny Edge Operator

jcunha@det.ua.pt 84

Non-local maxima suppression

jcunha@det.ua.pt 85

Non-local maxima suppression

jcunha@det.ua.pt 86

Hysteresis thresholding

jcunha@det.ua.pt 87

Resources

• Russ – Chapter 8, 9

• Gonzalez & Woods – Chapter 4

• http://homepages.inf.ed.ac.uk/rbf/HIPR2/c

anny.htm