1

Pattern Recognition:Statistical and Neural

Lonnie C. Ludeman

Lecture 29

Nov 11, 2005

Nanjing University of Science & Technology

2

Lecture 29 Topics

1. Review Fuzzy Sets and Fuzzy Partitions

2. Fuzzy C- Means Clustering Algorithm Preliminaries

3. Fuzzy C-Means Clustering Algorithm Details

4. Fuzzy C-Means Clustering Algorithm Example

5. Comments about Fuzzy Clustering

3

Example of a Fuzzy Set:

Define a Fuzzy set A by the following membership function

Or equivalently

Example

Review

4

S

Function defined on S

Review

5Membership Functions for Fuzzy Clusters

Domain Pattern Vectors

Domain Pattern Vectors

Membership Function for F1

Membership Function for F2

EXAMPLE

Review

6

A Fuzzy Partition F, of a set S, is defined by its membership functions for the fuzzy sets Fk : k =1, 2, ... , K

) ]

Fuzzy Partition

Review

7

where

Each value bounded by 0 and 1

Sum of each columns values =1

Sum of each row is less than n

Review

8

Conversion of Fuzzy Clustering into Crisp Clustering

) ]

1 1 1

2 2 2

C C C

Fuzzy Clusters

Crisp Clustering

All membership values are equal to 1 or 0

Only one 1 in each column where membership value for a Fuzzy cluster is a maximum

9

Fuzzy Clusters

Crisp Clusters from Fuzzy Clusters

Cl1 : [ 0.6 0.7 0.3 0.1 0.4 0.2 0.1 ] Cl2 : [ 0.1 0.1 0.3 0.5 0.1 0.7 0.1 ] Cl3 : [ 0.3 0.2 0.4 0.4 0.5 0.1 0.8 ]

Cl1 : [ 1 1 0 0 0 0 0 ] Cl2 : [ 0 0 0 1 0 1 0 ] Cl3 : [ 0 0 1 0 1 0 1 ]

Example: Given Fuzzy clusters below convert to Crisp Clusters

10

Cl1 : [ 1 1 0 0 0 0 0 ] Cl2 : [ 0 0 0 1 0 1 0 ] Cl3 : [ 0 0 1 0 1 0 1 ]

Cl1 : { x1 , x2 }

Cl2 : { x4 , x6 }

Cl3 : { x3 , x5 , x7 }

Cl1 : { x1 x2 }

Cl2 : { x4 x6 }

Cl3 : { x3 x5 x7 }

Answer: Crisp Clusters

11

Fuzzy C-Means Clustering Preliminary

Given a set S composed of pattern vectors which we wish to cluster

) ]

1 1 1

2 2 2

C C C

S = { x1, x2, ... , xN}

Define C Cluster Membership Functions

... ...

12

Define C Cluster Centroids as follows

Let Vi be the Cluster Centroid for Fuzzy

Cluster Cli , i = 1, 2, …, C

Define a Performance Objective Jm as

where

13

The Fuzzy C-Means Algorithm minimizes Jm

by selecting Vi and i , i =1, 2, … , C by an

alternating iterative procedure as described in the algorithm’s details

m = Fuzziness Index (m >1 ) Higher numbers being more fuzzy

A is a symmetric positive definite matrix

Ns is total number of pattern vectors

Definitions

14

Fuzzy C-Means Clustering Algorithm (a) Flow Diagram

No Yes

15

Fuzzy C-Means Clustering Algorithm

(b) Details of Steps in the Algorithm

16

Step 1: Initialization

Select initial membership functions such that

This is equivalent to specifying fuzzy clusters F1, F2, … , FC

17

One Method to accomplish this selection is

to chose rki randomly from the open

interval (0, 1) and then Normalize

18

Step 2: Computation of Fuzzy Centroids

Compute the Fuzzy Centroids as

19

Step 3: Compute New Fuzzy Membership Functions

Using the Vi, i = 1, 2, … , C from step 2 compute i(k)

20

Alternative method for computing the Membership Functions

where

21

If membership functions do not change convergence has occurred

If algorithm converges then the i

represent the fuzzy clusters and we Stop

If Convergence has not occurred and the number of iterations is less than some preassigned maximum value (MAXIT) then return to step 2

If otherwise then Stop with no solution.

Step 4: Check for Convergence

22

Usually convergences

Answer not unique as depends upon initial conditions.

Can converge to a local minimum

Can be used to produce hard clusters

Properties of Fuzzy C-Means Algorithm

23

Example – Application of Fuzzy Clustering Algorithm

Given the following set of data vectors

(a) Perform a Fuzzy Clustering of the data using Fuzzy C-Means Algorithm to obtain two fuzzy clusters. Try several initial conditions. Is the result unique? (use MAXIT=1000)

(b) Using the results of (a) give a crisp clustering of the data.

(c) Repeat (a) and (b) for three Fuzzy Clusters.

24

Plot of Data for Fuzzy clustering example

25

(a) Solution for two clusters

Fuzzy cluster membership functions randomly selected

Calculation of Fuzzy Centroids

26

Calculation of New Membership Functions

Calculation of Performance

Not the same as preceding iteration Membership function (No convergence)

Number of iterations not greater than 1000 therefore the iterations continue.

27

Results Converge at Iteration 17

Cluster membership Functions

Performance Measure

Cluster Centroids

28

(a) Final Cluster membership Functions

Cl2 :

Cl1 :

29

(b) Solution Crisp Clustering

Cl1 = { x1, x2, x3 } Cl2 = { x4 , x5 }

Crisp Membership functions

Set Assignment

Cl2 :

Cl1 :

Fuzzy Membership functions

30

Jm = 0.9337

(c) Solution for three fuzzy clustersApplying the Fuzzy Clustering Algorithm convergence was obtained in ?? iterations as

Final Cluster membership functions

Cl1 :

Cl2 :

Cl3 :

31

(c) Solution Crisp Clustering

Cl1 = { x1 , x2 } Cl2 = { x4 , x5 } Cl3 = { x3 }

Membership functions

Set Assignment

F1 : [ 1, 1, 0, 0, 0 ] F2 : [ 0, 0, 0, 1, 1 ] F3 : [ 0, 0, 1, 0, 0 ]

“Crisp Clusters”

32

Comments: Fuzzy Clustering Can be used to

Produce Hard Clustering

The larger the value of m the fuzzier the clusters

The Fuzzy algorithm is relatively stable and usually converges in a reasonable number of iterations

The Fuzzy algorithm is relatively insensitve to initial conditions

33

Of the two different fuzzy clusterings given

below, which clustering is the Fuzzier ???

Cl1 : [ 0.52 0.51 0.04 0.47 0.46 ] Cl2 : [ 0.48 0.49 0.96 0.5 0.53 ]

Cl1 : [ 0.89 0.85 0.04 0.26 0.15 ] Cl2 : [ 0.11 0.15 0.96 0.74 0.85 ]

# 1

# 2

or

34

Cl1 : [ 0.52 0.51 0.04 0.47 0.46 ] Cl2 : [ 0.48 0.49 0.96 0.53 0.53 ]

Cl1 : [ 0.89 0.85 0.04 0.26 0.15 ] Cl2 : [ 0.11 0.15 0.96 0.74 0.85 ]

# 1

# 2

ANSWER: #1 is the fuzzier of the two different clusterings

35

Why is #1 the Fuzzier of the two ???

36

Why is #1 the Fuzzier of the two ???

ANSWER: Because the cluster membership functions contain many entries close to 0.5 ( for the two class case) as opposed to values close to 0 and 1.

For the M class case values close to 1/M would indicate most fuzziness..

*

*

37

Lecture 29 Topics

1. Reviewed Fuzzy Sets and Fuzzy Partitions

2. Presented Fuzzy C- Means Clustering Algorithm Preliminaries

3. Gave Fuzzy C-Means Clustering Algorithm Details

4. Showed an Example of the Fuzzy C-Means Clustering Algorithm

5. Made a few comments about Fuzzy Clustering in General.

38

End of Lecture 29