1

Maria Magnusson, Computer Vision Lab., Dept. of Electrical Engineering, Linköping University

Digital Image Processing Lecture 8 Thresholding with hysteres Matched filters and pattern recognition

Different types of correlation Pattern recognition with correlation Point and blob detection with Laplacian or matched filter

The Structure tensor and the Harris operator Gonzales & Woods:

Lacks much material, except from: Correlation in Chapter 12, pp. 915-917, Laplace in Chapter 3, pp. 179 Point detection in Chapter 10, pp. 701-708 Harris operator in Chapter 11, pp. 869-875

p. 1

Thresholding with hysteres Useful when... Threshold T1 keeps only parts of the object Threshold T2 keeps parts of the background

a: Gray scaleimage

b: Thresholdedat T1

c: Thresholdedat T2

d: Result

Image b is dilated with the condition that expansion is not allowed outside the objects in image c.

In G&W: Only as a part of Canny’s edge (Chapter 10)

Lab 6

p. 2

Flowchart for thresholding with hysteres

c:=(a>T2)

b:=d mult c

b = e? The final resultis image byesno

b:=(a>T1)

Load image a

d:=dilate(b)

e:=b

Not in G&W!

Lab 6

p. 3

Correlation is the same as convolution with a 180o rotated filter. (The convolution procedure without the 180o rotation of the filter.)

Correlation:

Convolution:

𝑓 ∗ 𝑔 𝑥, 𝑦 𝑓 𝛼, 𝛽 ⋅ 𝑔 𝑥 𝛼, 𝑦 𝛽  𝑑𝛼𝑑𝛽

𝑓 𝑔 𝑥, 𝑦 𝑓 𝛼, 𝛽 ⋅ 𝑔 𝑥 𝛼, 𝑦 𝛽  𝑑𝛼𝑑𝛽

2D Convolution and correlation

Also called cross correlation, which definition contains the expectation value E( ), when it is defined for stochastic signals.

p. 4

2

The convolutionresult for comparison

𝑓 ∗ 𝑔 𝑥, 𝑦 𝑓 𝛼, 𝛽 ⋅ 𝑔 𝑥 𝛼, 𝑦 𝛽

𝑓 𝑔 𝑥, 𝑦 𝑓 𝛼, 𝛽 ⋅ 𝑔 𝑥 𝛼, 𝑦 𝛽

𝑓 𝑥, 𝑦 𝑔 𝑥, 𝑦Note the symmetryand that there is amax in the origin

00 -2

2

1

000

0 00 -2

2

1

000

000

0 0

0

2 -4

-2-200 0

-4 2

00

0 0

0

4

400 0

4 -4

0

0

1

9 -8

2D discreteconvolution and correlation

(12-9)

p. 5

The proofs are a nice exercise!

Correlation does not commutate:

Convolution commutates:𝑓 ∗ 𝑔 𝑥, 𝑦 𝑔 ∗ 𝑓 𝑥, 𝑦

𝑓 𝑔 𝑥, 𝑦 𝑔 𝑓 𝑥, 𝑦

Convolution in the Fourier domain:ℱ 𝑓 ∗ 𝑔 𝑥, 𝑦 𝐹 𝑢, 𝑣 ⋅ 𝐺 𝑢, 𝑣

Correlation in the Fourier domain, f and g real:ℱ 𝑓  𝑔 𝑥, 𝑦 𝐹∗ 𝑢, 𝑣 ⋅ 𝐺 𝑢, 𝑣

Theorems for convolutionand correlation

Table 4.4

Why is the Sobel filters in Maria’s slides rotated 180º

compared to G&W and previously Wikipedia?

Maria convolve with the Sobel filters, whereas

G&W correlate with the Sobel filters. From G&W,

2013: “This imprecise terminology [in the

literature] is a frequent source of confusion”.

Maria strongly believes in using convolution since then the Sobel filtering process corresponds to what is performed in a

linear system.

p. 6

Correlation in the Fourier domain:ℱ 𝑓  𝑔 𝑥, 𝑦 𝐹 𝑢, 𝑣 ⋅ 𝐺 𝑢, 𝑣

Computation in the spatial or the Fourier domain Convolution/correlation with a filter consisting of n x n

pixels performed in the spatial domain would require n2 x 5122 = n2 106 /4 multiplications and (n2-1) 106 /4 additions. This suggests that correlation with filters smaller than about 7 x 7 pixels is preferably executed in the spatial domain and not in Fourier domain.

It can be calculated that convolution/correlation on a 512 x 512 image costs 10 million multiplications and 15 million additions, provided that the Fourier transform is implemented with FFT.

We will check this on the next 3 slides!We presuppose that a 2D FFT of an NxN real image

takes N2logN2 multiplications and 1.5N2logN2 additions.

p. 7 Computation inthe spatial domain

p. 8

𝑛 · 𝑁 𝑛 · 512 𝑛 ·

· 10 𝑛 7

12 milj. MULT

𝑛 1 · 𝑁 𝑛 1 · 512𝑛 7 12 milj. ADD

3

𝐶 𝑢, 𝑣 𝐴∗ 𝑢, 𝑣 ⋅ 𝐵 𝑢, 𝑣

Computation in the Fourier domain

0 0 00 00 0 0

A(u,v) A*(u,v)2DFFT conj.

can be pre-computed

zero-paddedpattern a(x,y) C(u,v) c(x,y)

2DIFFT

Image:b(x,y) B(u,v)

correlationresult2D

FFT

p. 9 Computation in the Fourier domain

p. 10

2D FFT: 𝑁 log 𝑁 MULT, 1.5𝑁 log 𝑁 ADD

2𝑁 log 𝑁 2 · 512 · 2 · 9 10 milj. MULT

Mult: 𝑁 2⁄ komplex MULT

2D IFFT: 𝑁 log 𝑁 MULT, 1.5𝑁 log 𝑁 ADD

3𝑁 log 𝑁 3 · 512 · 2 · 9 15 milj. ADD

In total:

Pattern recognitionwith correlation

p. 11

𝑐 𝑥, 𝑦 𝑎 𝛼, 𝛽 ⋅ 𝑏 𝛼 𝑥, 𝛽 𝑦

𝑐 𝑥, 𝑦 ℱ 𝐶 𝑢, 𝑣 ℱ 𝐴∗ 𝑢, 𝑣 ⋅ 𝐵 𝑢, 𝑣

Disadvantage: A positive pattern and high positive values in the image can provide high correlation results even if the pattern and image are not similar.

𝑏

𝑎

𝑐

Pattern recognition with common correlation

(12-9)

Lab 6

p. 12

4

𝑏

𝑎

𝑎

𝑎 𝑥, 𝑦∑ ∑ 𝑎 𝛼, 𝛽 ⋅ 𝑏 𝛼 𝑥, 𝛽 𝑦

∑ ∑ 𝑎 𝛼, 𝛽 ∑ ∑ 𝑏 𝛼 𝑥, 𝛽 𝑦, ∈

Pattern recognition with normalized correlation Lab 6

p. 13 Pattern recognition with normalized correlation

Has maximum value (= 1) when the imageand pattern match perfectly, i.e. when a = b.

Compare with the Cachy-Schwarz's inequality.

𝑎∑ ∑ 𝑎 𝛼, 𝛽 ⋅ 𝑏 𝛼 𝑥, 𝛽 𝑦

∑ ∑ 𝑎 𝛼, 𝛽 ∑ ∑ 𝑏 𝛼 𝑥, 𝛽 𝑦, ∈

Lab 6

p. 14

𝑏

𝑎

𝑑

𝑎 𝑥, 𝑦 𝑎 𝛼, 𝛽 𝜇 ⋅ 𝑏 𝛼 𝑥, 𝛽 𝑦 𝜇

𝑎 𝛼, 𝛽 𝜇 ⋅ 𝑏 𝛼 𝑥, 𝛽 𝑦 𝑎 𝛼, 𝛽 ⋅ 𝑏 𝛼 𝑥, 𝛽 𝑦 𝜇

≈ crosscovariance

Lab 6

Pattern recognition with corre-lation without local DC-level

p. 15

b is the mean in the local neighborhood.

The proofs

are a nice exercise!

𝑎 𝑥, 𝑦∑ ∑ 𝑎 𝛼, 𝛽 𝜇 ⋅ 𝑏 𝛼 𝑥, 𝛽 𝑦 𝜇

∑ ∑ 𝑎 𝛼, 𝛽 𝜇   ∑ ∑ 𝑏 𝛼 𝑥, 𝛽 𝑦 𝜇, ∈

Pattern recognition with normalized correlation without local DC-level

See example on next slide from G&W 2:nd edition, Fig.12.9.G&W 4:th edition Fig.12.13 contains another example.

Lab 6

(12-10)

p. 16

b is the mean in the local neighborhood.

a is the mean of the pattern.

It is also called the correlation coefficient.

5

In-image Pattern Correlation result

Maximum value here

Pattern recognition with normalized correlation without local DC-level

p. 17

The choice of correlation method depends on what you want

a) 1D pattern b) found by

correlation without local DC-level

c) found by normalized correlation

d) found by normalized correlation without local DC-level

Common correlation, calculated via the Fourier domain:

Phase filter variant 1 divides with the amplitudes in the Fourier domain (”remove amplitude, keep phase”):

Suppose that the image b is translated in relation to the pattern a:

The correlation peak then becomes a dirac impulse at the desired position:

𝑐 𝑥, 𝑦 ℱ 𝐶 𝑢, 𝑣 ℱ 𝐴∗ 𝑢, 𝑣 ⋅ 𝐵 𝑢, 𝑣

Pattern recognition with phase filters

p. 19

𝑝 𝑥, 𝑦 ℱ∗ , ⋅ ,∗ , ⋅ ,

ℱ∗ , ⋅ ,

, ,

𝑏 𝑥, 𝑦 𝑎 𝑥 𝑥 , 𝑦 𝑦ℑ

𝐵 𝑢, 𝑣 𝑒 𝐴 𝑢, 𝑣

𝑝 𝑥, 𝑦 𝛿 𝑥 𝑥 , 𝑦 𝑦

Pattern recognition with phase filters Phase filter variant 2 divides with the pattern energy and

multiplies with a Gaussian G(u,v) in in the Fourier domain:

This way, the correlation peak becomes a Gaussian. This phase filter is less sensitive to noise and small

pattern variations.

p. 20

𝑝 𝑥, 𝑦 ℱ∗ , ⋅ , ⋅ ,

∗ , ⋅ ,ℱ

∗ , ⋅ , ⋅ ,,

• Our guest lecturer Bertil Grelsson uses a similar filter in his work.• Similar to the MOSSE correlation filter that can be used for

visual object tracking in images.• Martin Danelljan (former employee at CVL) uses a similar filter in his PhD thesis about visual tracking. His thesis has been assessed as the best Nordic PhD Thesis 2017/2018.

6

Point detection with a negative Laplacian filter

Fig. 10.4

Can be viewed as a high pass

filter or a correlation

pattern without DC-level.

X-ra

y im

age

of

turb

ine

blad

e

p. 21

Matches a single point

0 000 0

00 01 1 1

11 1

11 11 -1 -1

-1-1 -1

-1-1 -18

Calculate local DC-level

Do not match a smooth surface

/9/9

A negative Laplacian filter also works as line and edge detection

Fig. 10.5

Imag

e

Afte

r filt

erin

g

Abso

lute

val

ues

Posi

tive

valu

es

Note that edges gives one positive line close to one negative

line(Lab1)

p. 22

Alternative NegativeLaplacian filters

Fig. 3.45

Match points and lines with dif-ferent thickness. Also perform various amount of smoothing.

Also respond to edges.

Fig. 10.21

Lab5!

∇ sobelx ∗ sobelx sobely ∗ sobely

2 4 4 4 24 0 8 0 44 8 24 8 44 0 8 0 42 4 4 4 2

/64

∇ 𝑒 ⁄

(10.2-21)(10.2-22)

Laplacian of Gaussian

(LoG)

p. 23

The sum of the filter coefficients = 0 Constant areas give zero response.

Symmetric Does not care about direction.

Positive values in the center and negative values around.

What characterize a Negative Laplacian filter?

p. 24

7

𝑐 𝑥, 𝑦 𝐚′ 𝛼, 𝛽 • 𝐛′ 𝛼 𝑥, 𝛽 𝑦

    𝑎 𝛼, 𝛽 ⋅ 𝑏 𝛼 𝑥, 𝛽 𝑦

    𝑎 𝛼, 𝛽 ⋅ 𝑏 𝛼 𝑥, 𝛽 𝑦

där 𝐚′𝑎𝑎

𝜕𝑎𝜕𝑥𝜕𝑎𝜕𝑦

och 𝐛′𝑏𝑏

𝜕𝑏𝜕𝑥𝜕𝑏𝜕𝑦

Pattern recognition with correlation of the gradient

Not in G&W!

p. 25

Found insome smart

cameras.Master

thesis at SICK IVP

2007.

Pattern recognition with correlation of the gradient

In the pattern, the vectors may be normalized.

In the image, the vectors can possibly be un-normalized.

p. 26

Not in G&W!

Very computationally

intensive!

Translation-, scaling- and rotation-invariant pattern recognition

p. 27

Methods to speed up the pattern recognition on the previous slide To find the rotation (and possibly

scaling) with a scale pyramid. Make a scale pyramid of the image. Start at a rough scale and find

translation, scale and rotation. Refine the search at finer scales in the

scale pyramid. To find translation and rotation

(next Lecture) Fourier Descriptors (see also PPT-

presentation FourierDescr at home page)

Using eigenvectors

Found insome smart

cameras.Master

thesis at SICK IVP

2007.

p. 28

8

Estimation of local orientation using the

structure tensor The structure tensor is a little 2x2-matrix that can

be computed in every pixel of the image. It contains combinations of 1D derivatives. It gives information about the structure in a

neighborhood of every pixel. It can be used in many different applications. Taught more theoretical in the Computer Vision

course (TSBB15).

Partl

y in

G&W

!H

arris

: pp.

869

-875

p. 29

Lab 6

Test image: The camera man slightly modified The image

contain edges and line-like structures and also an artificial pattern in the upper right part of the image.

p. 30

Estimation of local orientation• The gradient ∇f(x,y) can be computed with two convolutions:

gx and gy are derivative filters, e.g. gx= sobel‐x and gy = sobel‐y, or Gaussian derivatives.

• A very simple description of local orienta‐tion is given by the direction of ∇f(x,y) :

p. 31

∇𝑓 𝑥, 𝑦𝑓𝑓

𝜕𝑓 𝑥, 𝑦𝜕𝑥

𝜕𝑓 𝑥, 𝑦𝜕𝑦

𝑔 ∗ 𝑓 𝑥, 𝑦𝑔 ∗ 𝑓 𝑥, 𝑦

𝐧∇𝑓 𝑥, 𝑦∇𝑓 𝑥, 𝑦

Estimation of local orientation• Problem 1: The angle of ∇f(x,y) changes 180° across a line. 

• Problem 2: ∇f(x,y) may be zero, even though there is a well‐defined orientation.

p. 32

9

The gradient components, fxand fy, as two images Negative and positive values are found close

together in the line-like structures.

p. 33

Estimation of local orientation• Form the outer product of the gradient with itself:

• This is a symmetric 2×2 matrix (a tensor):

• Note: This is a preliminary stage of T. A neigh-borhood is needed to estimate local orientation. Therefore LP-filtering is necessary to form the true structure tensor T.

p. 34

∇𝑓 ∇𝑓𝑓𝑓 𝑓 𝑓

𝑓 𝑓 𝑓

𝑓 𝑓 𝑓𝑇 𝑇𝑇 𝑇

𝑇 𝑇𝑇 𝑇 𝑇 𝑇 𝑇 𝑇

𝑇 𝑇 𝐓

Estimation of local orientation• A complex representation of the gradient is:

• Multiply this with itself and note that it can be written by using the tensor elements:

• Note that the angle is doubled:

p. 35

𝐳 𝑓 𝑗𝑓 abs 𝐳 exp 𝑗 arg 𝐳

𝐳 𝐳 ⋅ 𝐳 abs 𝐳 exp 𝑗2 ⋅ arg 𝐳

𝐳 𝐳 ⋅ 𝐳 𝑓 𝑓 𝑗2 𝑓 𝑓 𝑇 𝑇 𝑗2𝑇

Estimation of local orientation• Problem 1 solved: The angle of z do not change across a line.

• Solution to Problem 2: Perform low‐pass filtering of the individual elements, see 2‐4 slides ahead.

p. 36

angle for zg angle for z

10

Color coding of“the double angle representation”

angle 0

angle 𝜋 2⁄

angle 𝜋

angle 3 𝜋 2⁄

p. 37

T11 and T22 before LP-filtering• Line‐like objects in the original image give gaps in T11 and T22 (also in T12).

p. 38

LP-filtering As mentioned earlier, a neighborhood is needed

to estimate local orientation. Hence LP-filtering is necessary to form a true structure tensor T.

The LP-filter can be a Gaussian. It is separable and can be performed first in the x-direction and then in the y-direction.

The standard deviation sigma can be adjusted to a suitable size. Matlab example:>> sigma = 2;>> gauss = exp(-0.5*([-9:9]/sigma).^2);>> lp = gauss/sum(gauss);>> imageLP = conv2(image,lp,'same');>> imageLP = conv2(imageLP,lp','same');

p. 39

T11 and T22 after LP-filtering• The gaps have been filled.

p. 40

11

Re(z)=T11-T22 and Im(z)=2ꞏT12

p. 41

abs(z) and arg(z) (its angle)

p. 42

Left:abs(z) and arg(z) combined in one image. Right: color coding

p. 43

Information in the eigenvalues Let λ1 and λ2 be the eigenvalues of T,

where λ1 ≥ λ2 It can be shown that for local regions

containing only 1D variation λ1≥0, λ2=0 It can be shown that for a local isotropic

region, λ1 = λ2

p. 44

12

λ1- and λ2-images

p. 45 Confidence measuresC1, C2 and C• From the identities:

• we can define three confidence measures:

• For regions containing only 1D variation: C2=0• For a local isotropic region: C1=0 • For regions with weak structures: C=“small”• For regions with strong structures: C=“large”

tr 𝐓 𝑇 𝑇 𝜆 𝜆det 𝐓 𝑇 𝑇 𝑇 𝜆 𝜆

𝐶 tr 𝐓 4 det 𝐓 𝜆 𝜆𝐶 2 det 𝐓 2𝜆 𝜆𝐶 𝐶 𝐶 tr 𝐓 2 det 𝐓 𝜆 𝜆

p. 46

C1- and C2-images

p. 47

10 10

To increase the visibility, we can use: sqrt(C1)- and sqrt(C2)-images

p. 48

13

Conclusion about 1D and 2D structures C1, C2 or sqrt(C1), sqrt(C2 ), are slightly better to

discriminate 1D or 2D structures than λ1- and λ2-images

C=C1+C2 can discriminate weak structures from strong structures

p. 49 Confidence measure:the Harris corner detector Compute the corner response function:

A) Find points with large values of cHarris

(thresholding).

B) Find local maxima points of cHarris

Multiply A) and B) => Harris points!

𝐂Harris det 𝐓 𝑘 tr 𝐓 , 𝑘 0.05

p. 50

(11-63)

cHarris and original image with overlaid Harris points in red

p. 51

10

Example) Automatic reading of barcodes on packages The first part

of automatic bar-code reading is to detect the positions of the barcodes.

p. 52

14

Sqrt(C1)- and Sqrt(C2)-images

p. 53 Classification of structures:1D and 2D, weak and strong C1 and C2 (or sqrt(C1) and sqrt(C2)) can

discriminate 1D and 2D structures. C=C1+C2 can discriminate weak structures from

strong structures.

p. 54

After classification Strong structures classified in 1D and 2D.

p. 55 1D structures after some morphology: Final result! Apply opening, closing and removal of small

objects and holes.

p. 56