Fuzzy Clustering with Improved Artificial Fish Swarm Algorithm

Si He Nabil Belacel Habib Hamam Yassine Bouslimani Electrical Engineering

Department,Université de

Moncton, N.B.,Canada

esh0581@umoncton.ca

Institute for Information

Technology, National

Research Council of

Canada, N.B., Canada

Electrical Engineering

Department,Université

de Moncton,

N.B.,Canada

Electrical Engineering

Department,Université

de Moncton,

N.B.,Canada

Abstract

This paper applies the artificial fish swarm

algorithm (AFSA) to fuzzy clustering. An improved

AFSA with adaptive Visual and adaptive step is

proposed. AFSA enhances the performance of the fuzzy

C-Means (FCM) algorithm. A computational

experiment shows that AFSA improved FCM

outperforms both the conventional FCM algorithm and

the Genetic Algorithm (GA) improved FCM.

1. Introduction

Clustering is an unsupervised process through

which data are classified into homogenous groups.

These groups have well-defined boundaries. However,

some data may fit more than a one single cluster. They

may belong to several clusters to different extends at

the same time. Thus, other fuzzy clustering methods

have been studied to improve assessments of datasets.

The fuzzy clustering problem (F-CP) is one of the

most important topics in pattern recognition. Yang [3]

mentions that F-CP can be solved by either: 1) fuzzy

clustering based on fuzzy relation, 2) fuzzy clustering

based on generalized k-nearest neighbor rules, or 3)

fuzzy clustering based on optimization of an objective

function. This paper focuses on the third method.

Fuzzy C-Means (FCM) is the most frequently used

fuzzy clustering method that follows method 3. FCM

alternately optimizing membership degrees and

centroids until the best clusters are found. However,

FCM is not a perfect measure, its major weakness is

that the optimal solution relies on initial conditions.

This paper proposes an novel fuzzy clustering

algorithm, which combines AFSA and conventional

FCM, to enhance the performance of FCM.

2. Fuzzy clustering problems

The primary task of fuzzy clustering problems is to

define clusters such that each data item may belong to

more than one cluster. Each data will have different

degrees of membership in different clusters. Suppose

X is the set of data patterns with n patterns in d-

dimensional, � = {��, ��, … , �} , where �� =(���, ���, … , �� ) , � = 1,2, … , � . Matrix � = [���]∗�

denotes the unknown cluster membership degrees for

pattern i with respect to cluster j, where c is the number

of clusters, ��� ∈ [0,1] and ∑ ��� = 1���� ( � =

1,2, … , �, � = 1,2, … , �). = {!�, !�, … , !�} is the set

of c centroids where !� = (!��, !��, … , !��). The F-CP

problems can be formulated as follows:

#�� $ (�, ) = ∑ ∑ (��,� )%&'!� , ��(�����

��� (1)

(p) ∑ ��� = 1���� )*+ ,-- � = 1, … , � (2)

��� ∈ [0,1] )*+ ,-- � = 1, … , �, � = 1, … , � (3) where: 1) m is the fuzziness exponent which is

usually set at 2 in literature. 2) &'!� , ��( is the

distance norm representing the dissimilarity between

any measured data pattern and the centroid, for

example, this distance can be the Euclidean distance,

the Hamming distance , the cosine distance, etc.

In the non-linear programming problem (p), U and

V are decision variables. For any U, the problem (p)

can be solved to find V. This leads to heuristics where

it can be solved by optimizing U and V alternately.

FCM is the most common approach to solve this

problem. The steps are summarized as follows:

Step 1: Initialize a membership matrix � =[���]∗� , ∑ ��� = 1�

��� , � = 1,2, … , �, � = 1,2, … , �

Fix m and ɛ>0 (ɛ is a small positive constant), set

iteration counter k=1.

Step 2: at k-step, calculate cluster centroids .

with �. using Eq. (4) . = {!�., !�., … , !�.}

!�. = ∑ (/012)3405

067∑ (/01

2)35067

(4)

Step 3: update �.8�using Eq. (5)

���.8� = �

∑ (90190:

);

3<7 =:67

(5)

where &�� = &(�� , !�.)

Step 4: if >�.8� − �.>< ɛ, calculate the value of

objective function J using Eq. (1) and stop the iterative

process. Otherwise, set k=k+1 and go back to step 2.

2009 International Joint Conference on Computational Sciences and Optimization

978-0-7695-3605-7/09 $25.00 © 2009 IEEE

DOI 10.1109/CSO.2009.367

317

2009 International Joint Conference on Computational Sciences and Optimization

978-0-7695-3605-7/09 $25.00 © 2009 IEEE

DOI 10.1109/CSO.2009.367

317

2009 International Joint Conference on Computational Sciences and Optimization

978-0-7695-3605-7/09 $25.00 © 2009 IEEE

DOI 10.1109/CSO.2009.367

317

2009 International Joint Conference on Computational Sciences and Optimization

978-0-7695-3605-7/09 $25.00 © 2009 IEEE

DOI 10.1109/CSO.2009.367

317

FCM is a calculus-based optimization approach.

The convergence depends on the initial conditions.

Usually, unless the objective function is strictly convex

or strictly concave, the local optimum will not be

unique, and these local optimums can prevent FCM to

move to the global optimum. In this paper, swarm

intelligence (SI), which has the ability for searching the

global optimum, is introduced to overcome this

problem.

SI is an artificial intelligence technique based on

collective behaviors in any decentralized systems. SI

systems are composed of agents whom interact locally

with one another and the environment. In this paper, a

novel swarm intelligent algorithm, which is known as

the artificial fish swarm algorithm, is used to improve

the performance of FCM.

3. A brief introduction to Artificial Fish

Swarm Algorithm

AFSA is developed by Xiaolei Li in 2001. Fish

swims towards locations where food concentration is

highest. As a typical application of behaviorism in

artificial intelligence, AFSA can search for the global

optimum. The behaviors of artificial fish (AF) are

assumed as follows:

1) AF_Prey: Preys are born with instincts to find

food. The movement of fishes is driven by food

concentration. The current position of @A� is �� , then

within the visual area S, it randomly chooses another

position �� , @A� will swim towards �� for a random

distance if B� > B� (Y represents the food

concentration). Otherwise @A� will continue searching

within S and choose another position �D , @A� will

swim towards �D if BD > B� . If @A� cannot find a

position that have higher food concentrations

after E+F��#GH+’ times, @A� will swim some distance

in a random direction.

2) AF_Swarm: Fish usually assemble in groups to

capture colonies and/or to avoid dangers. There are �I

neighbors within the visualarea S. �� is the centre of

those neighbors. @A� will swim a random distance

towards �� if JK < δ (δ represents the crowd factor)

and B� > B� . Otherwise, @A� will resume the

behavior of Prey as mention in 1).

3) AF_Follow: When some fish find food through

the moving process of swarm, their neighborhood

partners tends to follow them to the best food-

concentration location. The current position of @A� is

�� and there are �I neighbors within the visual area S,

@A� will swim for a random distance to �%M4 if

JK < δ and B%M4 > B� . Otherwise, @A� will resume

the behavior of Prey.

The collective behaviors of AF_Follow and

AF_Swarm of all fish are simulated in every iteration.

The fishes will choose the behavior that has the best

position. AFSA is independent on the initial condition

[1][2]. A termination criterion can be added for each

specific problem. In basic AFSA models, the iterations

terminate when either 1) the estimated standard

deviation in two successive iterations is less than user-

set delta, or 2) the maximum number of iterations has

been reached, whichever occurs first.

AFSA has been applied to solve optimization

problems like signal processing [12], neural network

classifier [13] and other complex function optimization.

The performance of AFSA has been encouraging and it

has gradually become a prospective method in solving

the optimization problems.

4. An improved AFSA with adaptive visual

and adaptive step

In AFSA, many parameters can affect the result.

This paper proposes an improved AFSA that combines

an adaptive visual and adaptive step. This step

accelerates convergence speed, enhances global

stability and increases the precision of the optimization

process.

1) Adaptive Visual: Visual has a complex impact on

the artificial fish behaviors. Large values of Visual

reduce the chance of an AF being trapped in the local

optimum, especially during the early iterative process.

However, this algorithm may end prematurely.

Adaptive Visual may increase convergence in later

rounds of iterations and thus enhances the precision.

Some adaptive Visual methods are shown below: (t

denotes the current iteration, N is total iteration).

a) �N�,-O8� = P ∗ �N�,-O ,

where P = ( �N�,-K �N�,-�Q )

RS

�N�,-� is the initial Visual value, �N�,-K is the

pre-defined value of Visual at the last iteration, this

value affects the precision of the result.

b) �N�,-O8� = TU − OKV ∗ �N�,-I�4W

where U is a constant and it depends on the

precision of the result. It is suggested U ∈ [1.1,1.5] When iteration is within an optimal range, method

(a) and (b) ensure results with precision. They are

associated with iteration times, and gradually decrease

the value of Visual. For example, the authors use

method (a) and (b) to cluster iris dataset using Eq. (1)

as the objective function. Figure 1 shows the

comparative result.

318318318318

Figure 1. Comparative result between fixed Visual

and adaptive Visual

Fixed Visual has a faster convergence in early

iterations but adaptive Visual gives better precision and

increases the convergence speed in later iterations.

However, both method (a) and (b) might be trapped in

local optimum if the value of Visual decreases before

any AFs reaches the neighborhood of the global

optimum. Thus, a newly define method (c) is proposed:

c) Fixed Visual is used in early iterations. When no

improvements are possible, apply method (a) or (b).

Using fixed Visual in early stages of the iteration,

global stability can be increased. By changing to the

adaptive Visual in later rounds, convergence speed

increases and also, precision can be guaranteed.

2) Adaptive step: the standard AFSA can increase

convergence speed by increasing step. However, if the

increase of step is beyond certain ranges, convergence

speed could be reduced. Unexpected vibrations can

dramatically decelerate the convergence rate. This

problem is magnified if adaptive Visual is applied.

When Visual is smaller than fixed step, AF might go

beyond the target. This paper proposes a method using

adaptive step to overcome this problem:

�� (O8�) = �� (O) + N[H\ ∗ (�� (O) − �� (O))

where t represents the current iteration. The

current position of @A� is at �� , and the target @A� is at

�� . step is a positive constant (N[H\ ∈ (0,1) ). This

adaptive step can prevent unexpected vibrations.

In order to compare the improved AFSA with

conventional AFSA, we use both approaches to cluster

the iris dataset under the same initial conditions. Figure

2 shows the simulated comparative results.

From Figure 2, it can be seen that the improved

AFSA has a higher convergence speed in the later

rounds of iterations. It can also give a more accurate

optimization result. Besides, the improved AFSA can

effectively prevent premature convergence which

happens often in the conventional AFSA.

Figure 2. Comparative result between improved

AFSA and conventional AFSA

5. ASFA fuzzy clustering approach A new fuzzy clustering algorithm based on FCM

and AFSA is proposed here. The algorithm has the

following steps:

Step 1(Determine parameter encoding)

= '!�, !�, … !D … . !�( represents the centroids of

the clusters. It is considered to be one AF. !D is the

centroid of the pth

cluster ( 1 ≤ \ ≤ � ), where c is the

number of clusters. V is a c*n dimension-vector.

Step 2(Initialization)

Define the clusters number c, the population of AF

N, fuzziness exponent m, termination criterion, visual

distance of AF, step of AF, crowd factor and

Trynumber. Determine maximum iteration time K for

AFSA, set iteration counter k=1; initialize the first AF

population: @A. = { �. , �

. , … ̂ ., … K.} where ̂ .

represents the position of the qth

AF at the kth

iteration.

1 ≤ _ ≤ ` , N is the population of AF.

Step 3(Global search)

a) According to ̂ . , calculate membership matrix

�.̂ = [��,� . ]�∗ using the Eq. (5). Execute improved

AFSA using Eq. (1) as the objective function.

b) Go to step 4 when the result satisfies the

termination criterion, otherwise, increment k (k: =k+1)

and go back to step 3(a).

Step 4 (Local search)

a) Find the best individual AF: aWbO.

b) Calculate �.8� using Eq. (5).

c) Update .8�using Eq. (4)

d) Stop iteration if the result satisfies termination

criterion, or, increment k and return to step 4(b).

This algorithm is used to search for cluster centroids

so that the objective function f is minimized. After

each iteration, AFs swim to better locations. This

enables convergence to the global optimum.

319319319319

6. Experimental result

To evaluate effectiveness of the proposed method,

we compare conventional FCM and the GA improved

FCM using a test-suit of two real datasets along with

two artificial ones. The real datasets used in the

experiments are the Iris and Glass data set obtained

from UCI [11]. The clustering results from all three

methods were assessed using Eq. (1). Smaller values of

J indicate better results. Both the GA-FCM and AFSA-

FCM algorithms started on identical initial

populations, the values of fuzziness exponent m are set

at 2 in all algorithms. Parameter settings are tabulated

in Table 1. These algorithms were executed 10 times.

The average and the best values of J were collected.

Table 2 summarizes the results of the comparisons

between FCM, GA-FCM and AFSA-FCM methods

under random initial solutions. The first column shows

the number of clusters and the second column contains

the best values obtained by the heuristics compared.

The next columns show deviation percentages

calculated by[($ − $aWbO)/$aWbO] ∗ 100 .

The following findings are observed: 1) when c is

small, the performances of all methods are similar. 2)

AFSA-FCM and GA-FCM have better performances

than FCM when the number of cluster is large. 3)

Compared with GA-FCM, the proposed method has

better global stability.

Table 1. Parameter settings for different algorithms

Algorithm Parameters Values

FCM StopErr 10-4

GA-FCM Population Size

Crossover probability

Mutation Probability

Iterations

StopErr

30

0.8

0.2

200

10-4

AFSA-FCM Population of AF

Visual

Crowd Factor

Step

Trynumber

Iterations

StopErr

30

method (c)

0.6

0.9

10

200

10-4

Table 2. Comparison of FCM, AFSA-FCM and GA-FCM

% Deviation value from the best know solution Average in 10 runs Best in 10 runs

C Best known FCM AFSA-FCM GA-FCM FCM AFSA-FCM GA-FCM

Iris n=150, 4 features 3 60.5760 0.0002 0.0000 0.0000 0.0000 0.0000 0.0000

4 41.6887 3.7522 0.0000 0.0000 0.0000 0.0000 0.0000

5 32.8058 4.1149 0.0329 0.0753 0.0000 0.0000 0.0000

6 24.6758 4.3410 0.0769 0.1450 0.0000 0.0000 0.0000 7 20.3375 5.7382 0.0811 0.0562 0.0000 0.0000 0.0000

8 17.4790 8.5539 0.2350 0.3421 0.0610 0.0000 0.0000

9 15.6229 3.5229 0.0397 0.2190 0.0000 0.0000 0.0000

10 13.4852 9.0480 0.0801 0.1165 0.0890 0.0000 0.0000 Average 4.8839 0.0682 0.1193 0.0187 0.0000 0.0000

Glass n=214, 9 features 3 353.112 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

4 241.783 0.0008 0.0000 0.0000 0.0000 0.0000 0.0000

5 190.751 2.0189 0.1329 0.0753 0.0000 0.0000 0.0000

6 154.146 0.5640 0.0000 0.0000 0.0000 0.0000 0.0000

7 130.261 1.5385 0.0181 0.0062 0.0190 0.0000 0.0000

8 112.501 1.1760 0.0035 0.0421 0.0000 0.0000 0.0000

9 98.4343 2.6590 0.0129 0.0079 0.0021 0.0000 0.0000

10 87.5819 2.2591 0.1610 0.2105 0.0090 0.0000 0.0000 Average 1.2770 0.0410 0.4270 0.0038 0.0000 0.0000

Synthetic 1 n=2400, 2 features 3 62.2011 0.1170 0.0000 0.0020 0.0000 0.0000 0.0000

4 17.7462 0.0003 0.0000 0.0000 0.0000 0.0000 0.0000

5 8.3687 18.7650 3.1721 7.8753 0.1300 0.0000 0.8630

10 4.0096 21.3006 2.7010 0.9380 0.9700 0.0140 0.0000

15 2.7130 15.6579 0.0290 0.0262 0.7800 0.0000 0.0000

20 1.9920 3.1016 0.2369 0.0410 0.0000 0.0000 0.0000

30 1.2716 5.0126 1.0478 2.3001 0.0078 0.0000 0.0000

40 0.9475 2.2591 0.1987 0.4789 0.0061 0.0000 0.0250 Average 8.2767 0.9231 1.4576 0.2367 0.0017 0.1110

320320320320

Table 2 (Con't). Comparison of FCM, AFSA-FCM and GA-FCM

% Deviation value from the best know solution Average in 10 runs Best in 10 runs

C Best known FCM AFSA-FCM GA-FCM FCM AFSA-FCM GA-FCM

Synthetic 2 n=1500,3 features 3 32.3648 0.1660 0.0000 0.0000 0.0000 0.0000 0.0000

4 11.5460 0.0000 0.0017 0.0000 0.0000 0.0000 0.0000

5 9.7146 0.1064 0.1329 0.0753 0.0020 0.0042 0.0000

10 4.1205 9.9612 0.7960 1.5870 3.4150 0.0000 0.0340

15 2.4772 19.100 4.7851 3.8952 2.7344 0.0045 0.0000

20 1.7624 5.7653 0.7035 1.6420 0.9050 0.0000 0.0071

30 1.0660 7.8601 0.0129 0.9060 3.4521 0.0000 0.0000

40 0.7625 5.1257 0.5700 0.8005 1.0190 0.0000 0.0018 Average 6.1370 0.9188 1.1579 1.5012 0.0012 0.0058

6. Conclusion

In this paper, we have proposed a novel fuzzy

clustering algorithm which uses Artificial Fish Swarm

Algorithm that can enhance the performance of the

FCM. An improved ASFA with adaptive Visual and

adaptive step is proposed. The improved Algorithm

exploits the search capability of AFSA to overcome the

local optimal problem of the FCM algorithm. To the

best of our knowledge, it is the first time that AFSA

heuristics are applied as a method for solving fuzzy

clustering problems. Experimental results show that

the proposed method outperformed both the

conventional FCM and the genetic algorithm improved

FCM using the test datasets. These results also indicate

that the proposed method is particularly effective for

fuzzy clustering problems that have many entities and

cluster centroids.

Future developments of the proposed method may

include the following research directions: 1) Study the

interrelationship among parameters of AFSA and

define adaptive method of other parameter 2) Study the

interrelationships among the AF behaviors to improve

computational time; 3) Develop AF_delete and

AF_insert behavior to accelerate convergence speed 4)

Develop an automatic clustering method using AFSA.

Acknowledgement

The authors wish to thank the Natural Sciences and

Engineering Research Council of Canada for the

financial support.

References [1] X.L. Li, Z.J. Shao and J.X. Qian, “ An optimizing method based on autonomous animate: fish swarm algorithm”, System Engineering Theory and Practice, 11: 32-38, 2002.

[2] S. Gao and J.Y. Yang, “Swarm intelligence algorithms and applications” China Waterpower Press, Beijing, 2006 [3] M.-S, Yang, “A survey of Fuzzy clustering”,Mathl. Comput.Modellling, 1993, 18(11) 1-16 [4] Nabil Beelacel, Peirre Hansen, Menad Mladenovie, “Fuzzy J-Means: a new heuristic for fuzzing clustering”, Pattern Fecognition, 35, 2002, 2193-2200 [5] S. Gao and J.Y. Yang, “Swarm intelligence algorithms and applications” China Waterpower Press, Beijing, 2006 [6] Li, X. L., “A New Intelligent Optimization-Artificial Fish Swarm Algorithm.” Ph.D. thesis, Zhejiang University, 2003 [7] Jiang, M. Y., Wang, Y., Pfletschinger, S.Lagunas, M.A., “Optimal Multiuser Detection with Artificial Fish Swarm Algorithm.” ICIC 2007. CCIS 2, Springer-Verlag Berlin Heidelberg, 2007, 1084-1093 [8] J. C. Bezdek, R. J. Hathaway, “Optimization of Fuzzy criteria using genetic algorithms”, Proceedings of the first IEEE Conference on Evolutionary Computation, 1994 [9] K. H. Al-Suan, C. A. Fedjki, “A tabu search-based algorithm for the fuzzy clustering problems”, Pattern Recognition 30(12), 1997, 2023-2030 [10] W.Pedryez, “Conditional fuzzy clustering in the design of radial basis function neural networks”, IEEE trans Neural networks 9(4), 1998, 601-602 [11] University of California, Machine Learning Repository, http://www.ics.uci.edu/~mlearn/MLRepository.html [12]Jiang, M.Y., Yuan, D.F. “Wavelet Threshold Optimization with Artificial Fish Swarm Algorithm” Proc. of IEEE International Conference on Neural Networks and Brain, Beijing China, 2005, 569 -572 [13]Zhang, M.F., Cheng, S., Li, F.C. “Evolving Neural Network Classifiers and Feature Subset Using Artificial Fish Swarm. “ Proc. of IEEE International Conference on Mechatronics and Automation, Luoyang China, 2006, 1598-1602

321321321321