texture segmentation - biomedical engineering · introduction to signal and image processing may...

TextureSegmentation

Introduction to Signal andImage Processing

Prof. Dr. Philippe Cattin

MIAC, University of Basel

May 24th/31st, 2016

May 24th/31st, 2016Introduction to Signal and Image Processing

1 of 48 22.02.2016 09:20

Contents

2

4

5

6

7

8

10

12

13

14

15

17

18

19

20

21

22

23

25

26

27

28

30

32

Ph. Cattin: Texture Segmentation

Contents

Abstract

1 Introduction

What is Texture?

What do we need it for?

Characteristics of Textures

Principle of Texture Segmentation Methods

Problems with Texture Segmentation is Scale

2 Feature Extraction

Feature Extraction

2.1 First-Order Histogram Based Features

First-Order Histogram Based Features

Histogram Features

Properties of these Histogram Features

First-Order Texture Feature Example

2.2 Cooccurrence Matrices

Cooccurrence Matrices

The Cooccurrence Matrix

Cooccurrence Matrix Examples

Cooccurrence Matrix Examples (2)

Extensions to the Cooccurrence Matrix

Cooccurrence Matrix Features

Haralick Feature Examples

2.3 Fourier Features

Fourier Features

Fourier Features: Entropy Example

Fourier Features (2)

Disadvantage of the Fourier Features

2.4 Filter Banks

Filter Banks

2.4.1 Laws Filters

Laws Filters May 24th/31st, 2016Introduction to Signal and Image Processing

2 of 48 22.02.2016 09:20

34

35

36

37

38

40

41

42

43

44

45

46

47

48

50

52

53

55

57

59

61

2.4.2 Gabor Filters

Gabor Filters

Gabor Filter Power Spectrum

Gabor Filter in the Frequency Domain

Gabor Filter Example

2.4.3 Eigenfilters

Eigenfilters

Generalisation of Eigenfilters

Eigenfilter Example

Eigenfilter Example - Periodicity of theTexture

Eigenfilter Example - Extract the SampleVectors

Eigenfilter Example - Calculation of theEigenfilters

Eigenfilter Example - Eigenfilter the FlawedTextile

Eigenfilter Example - Smooth the FilterExample

Eigenfilter Example - Mahalanobis Distance

3 Texture Discrimination

Texture Discrimination

3.1 Region Growing

Region Growing Methods

Region Growing by Pixel Aggregation

3.2 Other Methods

Other Methods

3.3 Reduction of Dimensionality

Reduction of Dimensionality

4 Texture Classification

Texture Classification

5 References

References


3 of 48 22.02.2016 09:20


(2)


Abstract

4 of 48 22.02.2016 09:20

Introduction


(4)What is Texture?

The term texture is difficult to define, but it represents aspects of thesurface pattern, such as

coarseness,

directionality,

brightness,

colour,

regularity, and so forth.

Texture is a phenomenon that is widespread, easy torecognise and hard to define. Forsyth/Ponce, 2002.

5 of 48 22.02.2016 09:20


Introduction

(5)


What do we need it for?

Texture features can be used to segmentscenes into parts. In remote sensing it isquite usual to segment scenes intodifferent types of land, such as

forrest,

vineyards,

urban,

streets,

lakes,...

Fig 8.1: Segmentation example

(forrest, cultural land, other),

ETH Zurich

6 of 48 22.02.2016 09:20


Introduction

(6)


Characteristics of Textures

Already form the small set oftextures some differences can beobserved.

(a)+(b) are regular and

contain some basic patter

(texel) that is repeated in

some way (placement rule)

1.

(a)'s texel is larger than (b)'s.

(a) is thus said to be coarser

2.

It is even more subtle, as

texture (a) has subpatterns

(micro-texture) → hierarchical

organisation

3.

Texture (a) has a dominant

diagonal organisation, (b) has

a vertical orientation, (c)+(d)

have no apparent orientation

4.

(a) (b)

(c) (d)

Fig 8.2: Sample textures

7 of 48 22.02.2016 09:20


Introduction

(7)


Principle of TextureSegmentation Methods

Texture analysis methods typically probe for theaforementioned characteristics like regularity, coarseness,and directionality.

Texture analysis can be split in three fundamental steps as shown inFig 8.3.

Feature extraction: Extract the texture measures, that

numerically describe the texture properties

1.

Texture discrimination: partition a textured image into regions,

each corresponding to a perceptually homogeneous texture

2.

Texture classification: determine to which of a finite number of

physically defined classes a homogeneous textured region

belongs

3.

Fig 8.3: Fundamentals steps in texture analysis

8 of 48 22.02.2016 09:20


Introduction

(8)


Problems with TextureSegmentation is Scale

(a) Segment forest, water,...(b) Segment fields, forest,

streets,...

(c) Segment small groups of

trees, grass,...

(d) Segment the leafs(e) Segment structure of

the leaf(f) Segment the cells

9 of 48 22.02.2016 09:20

Feature Extraction


(10)Feature Extraction

The approaches to extract the texture measures are typicallycategorised into

statistical,1.

structural,2.

transform, and3.

model-based approaches.4.

Fig 8.4: Typology of the texture measures commonly used in texture analysis

10 of 48 22.02.2016 09:20

First-Order HistogramBased Features


(12)First-Order Histogram BasedFeatures

Given an image of two space variables

and . The function can take the discrete values ,

where is the number of intensity levels in theimage. The intensity level histogram is a

function showing the number of pixels in the wholeimage with this intensity level:

(8.1)

where is the Kronecker delta function

(8.2)

(a)

(b)

Fig 8.5: (a) Texture,

(b) corresponding

histogram

The probability density function (PDF) can be approximated bydeviding by the total number of pixels , thus

11 of 48 22.02.2016 09:20



(13)


Histogram Features

The shape of the histogram can subsequently be used to characterisethe image. For example,

a narrowly distributed histogram indicates a low-contrast image,

and

a bimodal histogram often suggests a low contrast object against

a differing background.

To quantitatively describe the image, different parameters (imagefeatures) can be extracted from the histogram.

Mean: Skewness: Energy:

Variance: Kurtosis: Entropy:

12 of 48 22.02.2016 09:20



(14)


Properties of these HistogramFeatures

The mean takes the average level of intensity in the image

The variance describes the variation of the intensity around the

mean → is a measure of contrast

The energy is a measure of peakness

The skewness is zero if the histogram is symmetrical about the

mean → indicator of symmetry

The kurtosis is a measure of flatness of the histogram → for

Gaussian shaped histograms

The entropy is a measure of histogram uniformity

The mean and variance do not actually carry directinformation about the texture. They rather depend on theimage acquisition process.

One often gets better texture segmentation performance if theimages are normalised to a mean of and a standarddeviation of .

13 of 48 22.02.2016 09:20



(15)


First-Order Texture FeatureExample

(a)

Mean

Variance

Skewness

Kurtosis

Energy

Entropy

(b)

(c) (d)

Fig 8.6: (a) Texture 1, (b) Texture 2, (c) Histogram of texture 1, (d) Histogram of

texture 2

14 of 48 22.02.2016 09:20

CooccurrenceMatrices


(17)Cooccurrence Matrices

The histogram as used in theprevious section capturesintensity information in a highlycompact, but also highlyincomplete way.

As Fig. 8.7 shows, two textureswith a completely different visualappearance can have the samegray-scale histogram.

All information aboutconfigurational aspects of theintensity distribution is lost.

The Cooccurrence matrix,in contrast, is a relateddata structure that baseson second-order statisticsand preserves someaspects of the spatialconfigurations.

(a) Texture 1 (b) Histogram of

texture 1

(c) Texture 2 (d) Histogram of

texture 2

Fig 8.7: Completely different textures can

have the same gray-scale histogram

15 of 48 22.02.2016 09:20



(18)


The Cooccurrence Matrix

The Cooccurrence matrix is in the literaturealso known as the GLCM (Gray-LevelCo-occurrence Matrices), or the Spatialdependence matrix.

Mathematically, the cooccurrence matrix is defined over an image ,parametrised by an offset as

(8.3)

The cooccurrence matrix is extracted for aspecific vector . Fig 8.8: Construction of the

cooccurrence matrix

16 of 48 22.02.2016 09:20



(19)


Cooccurrence MatrixExamples

Fig 8.9: Cooccurrence matrix sample of a texture with a vector ( ) that

does correspond to a period of the texture.

The high values in the cooccurrence matrix are concentrated in acentral location.

17 of 48 22.02.2016 09:20



(20)


Cooccurrence MatrixExamples (2)

Fig 8.10: Cooccurrence matrix sample of a texture with a vector ( )

that does not correspond to a period of the texture.

In contrast to the previous example, the values in the cooccurrencematrix are more smeared out.

18 of 48 22.02.2016 09:20



(21)


Extensions to theCooccurrence Matrix

Several intuitive extension of theCooccurrence matrix are commonly usedin pactice:

To analyse a texture, one can build

several such matrices , each for a

different choice of vector → if the

length of the vector is varied, a

certain degree of scale invariance

could be achieved.

1.

One can also use several vectors at

once and put the results into a single

matrix → vectors of the same length

but with a different orientation

combined results in a rotation

invariant cooccurrence matrix.

2.

Fig 8.11: Rotation invariant

cooccurrence matrix

19 of 48 22.02.2016 09:20



(22)


Cooccurrence Matrix Features

Retaining the entire cooccurrence matrix is not efficient. typically, thewidely spread Haralick features are extracted for later classification:

Energy:

Entropy:

Contrast:

Homogeneity:

Maxprobability:

Energy and entropy are both measures of how concentrated theentries occur. Energy is high if has strong peaks, and the entropyreaches a max if all entries are equal probable.

Contrast and homogeneity are more specifically oriented towards theentries being concentrated near the diagonal (then contrast is smalland homogeneity is large).

20 of 48 22.02.2016 09:20



(23)


Haralick Feature Examples

(a)

Energy

Entropy

Contrast

Homogeneity

Max. Probability

(b)

(d) (e)

21 of 48 22.02.2016 09:20

Fourier Features


(25)Fourier Features

Frequency related concepts like a texture beingperiodic or coarse-grained led to features extractedfrom the Fourier power spectrum.

If a texture is highly regular (periodic) it can beexpected to have dominant peaks in its powerspectrum.

To evaluate how strongly peaked a spectrum is, onecan calculate its entropy. The term Entropyoriginates from Physics as was later adapted toInformation Theory and is a measure for theamount of information stored in a particular signal.It is defined as

(8.4)

where is the probability density function (PDF)

estimated with the histogram of thepowerspectrum, is usually 2 yielding the unit bitfor the entropy.

Fig 8.12: Sample

texture

Fig 8.13:

Corresponding

Fourier power

spectrum

22 of 48 22.02.2016 09:20


Fourier Features

(26)


Fourier Features: EntropyExample

Example of the Fourier power spectra and its entropy of fourdifferent textures.

Entropy: Entropy: Entropy: Entropy:

Fig 8.14 Power spectrum and entropy example

23 of 48 22.02.2016 09:20


Fourier Features

(27)


Fourier Features (2)

Other commonly used Fourier features use integratedpower values to assess the textures coarseness and itsdirectionality.

Integration is performed over two different kinds ofregions in the Fourier domain, namely

concentric rings around the origin to assess the

coarseness, and

(8.5)

sectors from the apex to assess the directionality

(8.6)

Fig 8.15:

Fig 8.16:

24 of 48 22.02.2016 09:20


Fourier Features

(28)


Disadvantage of the FourierFeatures

As the Fourier power spectrum is typically calculated over theentire image, these features collect global information insteadof localised texture properties.

25 of 48 22.02.2016 09:20

Filter Banks


(30)Filter Banks

A simple but very effective method for extracting texture informationis to convolve the image with a series of convolution filters. Theoutput of these Filter banks are then combined into a feature vectorused for classification.

Fig 8.17: Principle of filter banks

26 of 48 22.02.2016 09:20

Laws Filters


(32)Laws Filters

The Laws Filters use a fixed set ofad hoc filters of size , ,and , created by theconvolution of row and columnfilters with coefficients specifiedin Table 8.1.

Usually, the output of these filtersis non-linearly transformed andthen averaged over larger regionsin order to get an energy.

The energies for the differentfilters are then combined into afeature vector, that can be usedfor segmentation or classification.

Tab 8.1: 1D Laws Filter

27 of 48 22.02.2016 09:20

Gabor Filters


(34)Gabor Filters

In contrast to the ad hoc chosen Lawfilters, the Gabor filters offer a moreprincipled alternative with a solidmathematical background. The Gaborfilters are constructed by modulating aGaussian curve with a sine function. Thisyields for vertically oriented Gaborfilters:

(8.7)

where the parameter determines

the width of the Gaussian envelope, specifies the frequency of the modulatingcosine, while specifies the phase.

:

Fig 8.18: A Gabor filter profile (in

the spatial domain) with

, and

Filters with other orientations can be easily obtained by applying arigid rotation of the -plane.

28 of 48 22.02.2016 09:20


Gabor Filters

(35)


Gabor Filters

Guesswhat the

PowerSpectrum

of theGaborfilter

looks like

29 of 48 22.02.2016 09:20


Gabor Filters

(36)


Gabor Filter Power Spectrum

The Gabor filter samples the Fourier domain as can be seen from itsFourier transform

(8.8)

where . The

corresponding power spectrum plot isdepicted in Fig 8.19.

The Gabor filter being a product of aGaussian and a cosine, it's Fouriertransform is known to be the convolutionof a Gaussian and a pair of Diracimpulses.

If we rotate the filter, thepositions of these two lobes willundergo the same rotation in thefrequency domain.

Fig 8.19: Fourier power spectrum

of the Gabor filter

30 of 48 22.02.2016 09:20


Gabor Filters

(37)


Gabor Filter in the FrequencyDomain

By increasing the number of parameters, theGaussian lobes can be made elliptical. Fig 8.20shows how the Gaussian profile can be madeelliptical to optimally tessellate the frequencydomain.

Gabor filters provide the best possiblesimultaneous localisation in the spatialand the frequency domain, i.e.

.

Fig 8.20: Gabor filter in

the frequency domain

Fig 8.21: Gabor filter in

the frequency domain

31 of 48 22.02.2016 09:20


Gabor Filters

(38)


Gabor Filter Example

(a) (b) (c)

Fig 8.22: (a) Input texture, (b) output of a Gabor filter tuned to detect horizontal

structures, (c) same for vertical structures.

32 of 48 22.02.2016 09:20

Eigenfilters


(40)Eigenfilters

The Laws and Gabor filter banks have fixed coefficients. If a specifictexture should be analysed, it can be interesting to finetune the filterset for this texture.

Basic Idea

Consider a small neighbourhood e.g.

pixels

1.

All 9 pixels in this mask are put in a

9-vector

2.

When shifting the mask over the

image many such samples of this

9-vector are acquired

3.

Calculate the corresponding

covariance matrix

4.

The principal components yield a

new orthogonal basis for these

9-vectors

5.

Arranging the components of all

eigenvectors (principal components)

yields in total 9 convolution filters of

size

6.

Fig 8.23: Principle of the

eigenfilter design

33 of 48 22.02.2016 09:20


Eigenfilters

(41)


Generalisation of Eigenfilters

In most practical cases it is useful to alsotake the scale of the texture intoaccount. This is done by extracting theperiodicity of the pattern with e.g.autocorrelation and then adapting thesize of the filters accordingly.

This will generally result in large andtherefore unpractical convolution masks.A way out is to consider sparse masks,i.e. masks where most of the elementsare set to zero, see Fig 8.24.

Fig 8.24: Structure of the

eigenfilter convolution mask. The

black squares indicate positions

in the convolution mask

that have non-zero coefficients

34 of 48 22.02.2016 09:20


Eigenfilters

(42)


Eigenfilter Example

TaskDevelop a computationally efficient filter that can reliably detectsuch flaws.

Fig 8.25: Textile sample with a flaw

35 of 48 22.02.2016 09:20


Eigenfilters

(43)


Eigenfilter Example -Periodicity of the Texture

%load the flawless image

img=imread('textile.png');

img=double(img(:,:,1));

imshow(img,[]);

% Autocorrelation the image

% with a small patch of itself

r=imfilter(img,img(1:30,1:30));

% plot the 1D graphs in x- and

% y-direction

plot(r(:,100));

plot(r(100,:));

% read out the periodicity

dy = 7; dx = 9;

As the horizontal periodicity wasfound to be pixels and the verticalperiodicity was found as , aconvolution mask of size isused.

To simplify the calculations we try asparse convolution mask with thearrangement as in Fig 8.24.

Fig 8.26: Correct textile sample

Fig 8.27: Autocorrelation result in

x-direction

Fig 8.28: Autocorrelation result in

y-direction

36 of 48 22.02.2016 09:20


Eigenfilters

(44)


Eigenfilter Example - Extractthe Sample Vectors

In the second step we shift the sparsemask over the entire image of theflawless textile image and extract the9-element vectors at each location. Thesesample vectors are then stored in thematrix .

[m n] = size(img);

X=zeros((n-dx)*(m-dy),9);

w=[1 0 0 0 1 0 0 0 1;

0 0 0 0 0 0 0 0 0;

0 0 0 0 0 0 0 0 0;

1 0 0 0 1 0 0 0 1;

0 0 0 0 0 0 0 0 0;

0 0 0 0 0 0 0 0 0;

1 0 0 0 1 0 0 0 1];

w=find(w>0);

[m n] = size(img);

pos=1;

for x=1:n-dx

for y=1:m-dy

tmp=img(y:y+dy-1,x:x+dx-1);

X(pos,:)=tmp(w)';

pos=pos+1;

end

end

Fig 8.29: Extraction of the

sample vectors

37 of 48 22.02.2016 09:20


Eigenfilters

(45)


Eigenfilter Example -Calculation of the Eigenfilters

The Eigenfilters can the easilycalculated with the MATLABfunction princomp.

[pc,score,lambda] =

princomp(X,'econ');

Fig 8.30: Construction of the eigenfilters

38 of 48 22.02.2016 09:20


Eigenfilters

(46)


Eigenfilter Example -Eigenfilter the Flawed Textile

In this step we filter the flawedimage with all the eigenfilterswe previously found.

flawed =

imread('textile_flaw.png');

flawed =

double(flawed(:,:,1));

flawed_flt = zeros(m,n,9);

for i = 1:9

kern = zeros(7,9);

kern(w) = pc(:,i);

flawed_flt(:,:,i) =

imfilter(flawed,kern);

figure,

imshow(flawed_flt(:,:,i),

[]);

end

It can be clearly seen, that theflaw shows up in several of theeigenfiltred images.

Filtered with PC 1

Filtered with PC 2

Filtered with PC 3

Filtered with PC 4

Filtered with PC 5

Filtered with PC 6

Filtered with PC 7

Filtered with PC 8

Filtered with PC 9

39 of 48 22.02.2016 09:20


Eigenfilters

(47)


Eigenfilter Example - Smooththe Filter Example

The aim of this step is to generateimages with more homogeneousintensities.

This is achieved by squaring allvalues and boxfiltering the imagewith a kernel of size .

flawed_pwr = zeros(m,n,9);

for i = 1:9

flawed_pwr(:,:,i) =

imfilter(flawed_flt(:,:,i).^2,ones(5));

figure, imshow(flawed_pwr(:,:,i),[]);

end

Local energy filtered

with PC 1


with PC 2


with PC 3


with PC 4


with PC 5


with PC 6


with PC 7


with PC 8


with PC 9

40 of 48 22.02.2016 09:20


Eigenfilters

(48)


Eigenfilter Example -Mahalanobis Distance

In this last step, the energy values arecombined into a single distance(Mahalanobis) to the typical energies.For this we need the energies of theflawless textile:

intact_pwr = zeros(m,n,9);

for i = 1:9

kern = zeros(7,9);

kern(w) = pc(:,i);

tmp=imfilter(imfilter(img,kern).^2,ones(5));

figure, imshow(tmp,[]);

intact_pwr(:,:,i) = tmp;

end

then we calculate the Mahalanobisdistance of every point of the defectivetextile to the energy values seen in theflawless textile image.

intact_pwr2 = reshape(intact_pwr,n*m,9);

flawed_pwr2 = reshape(flawed_pwr,n*m,9);

r = mahal(flawed_pwr2,intact_pwr2);

out = reshape(r,m,n);

figure, imshow(out,[]);

Fig 8.31: Mahalanobis distance for

the image in Fig 8.25

41 of 48 22.02.2016 09:20

TextureDiscrimination


(50)Texture Discrimination

In Texture discrimination the extractedtexture measures are analysed and theimage split into regions, eachcorresponding to a perceptuallyhomogeneous texture.

In texture segmentation wealways face the trade-off betweensample size and accuracy. Thebigger the sample size, the betterthe accuracy of featureestimation; however, this allows acoarse segmentation only.

Fig 8.32: Texture segmentation principle: texture

discrimination

Fig 8.33: Texture measures for

small regions or even for

individual pixels are known

Fig 8.34: Regions with similar

texture measures are grouped

together

42 of 48 22.02.2016 09:20

Region Growing


(52)Region Growing Methods

Segmentation may be viewed as a process that partitions into subregions, , such that

(a)

(b) is a connected region,

(c) for all and ,

(d) for , and

(e) for

where is a logical predicate over points in the set .

43 of 48 22.02.2016 09:20


Region Growing

(53)


Region Growing by PixelAggregation

The simplest Region growing approach is pixel aggregation, whichstarts with a set of seed points and from these grows regions byappending to each seed point those neighbouring pixels that havesimilar properties i.e. texture measures.

(a) Step 1 (b) Step 2 (c) Step 3 (d) Step 4

Fig 8.35: Region growing by pixel aggregation using an absolute difference of less

than 3 between intensity levels and 4-connectivity

44 of 48 22.02.2016 09:20

Other Methods


(55)Other Methods

Below is a list other commonly used texture discrimination methods

Watershed

k-Means clustering

Bayesian classification

Artificial neural networks (ANN)

Estimation Theory - Maximum likelihood

Split-and-Merge

45 of 48 22.02.2016 09:20

Reduction ofDimensionality


(57)Reduction of Dimensionality

In many cases it is desirable to prior reduce the dimensionality of theextracted texture measures. PCA is the method of choice as can beseen in Fig 8.36.

Fig 8.36: Successfull dimensionality

reduction with PCA

Fig 8.37: Failed PCA dimensionality

reduction

Beware: in some special cases dimensionality reduction withPCA fails, see Fig 8.37.

46 of 48 22.02.2016 09:20

Texture Classification


(59)Texture Classification

Once the texture features have been extracted and regions withsimilar properties grouped, all that remains is to classify the differentregions into known textures. The following methods are given as areference but will not be discussed in this lecture

Bayesian Decision Theory

Artificial Neural Networks (ANN)

Support Vector Machines (SVM)

Fig 8.38: Texture segmentation principle: Texture classification

47 of 48 22.02.2016 09:20

References


(61)References

Robert M Haralick, K Shanmugam, Its'hak Dinstein (1973).

"Textural Features for Image Classification". IEEE Transactions

on Systems, Man, and Cybernetics SMC-3 (6): 610-621.

48 of 48 22.02.2016 09:20

texture segmentation - biomedical engineering · introduction to signal and image processing may...

Documents