towards information-theoretic k-means clustering …jselab.njue.edu.cn/download/journal/towards...

Contents lists available at SciVerse ScienceDirect

Signal Processing

Signal Processing ] (]]]]) ]]]–]]]

0165-16

http://d

n Corr

E-m

PleasProc

journal homepage: www.elsevier.com/locate/sigpro

Towards information-theoretic K-means clusteringfor image indexing

Jie Cao a, Zhiang Wu a, Junjie Wu b,n, Wenjie Liu c

a Jiangsu Provincial Key Laboratory of E-Business, Nanjing University of Finance and Economics, Nanjing, Chinab Department of Information Systems, School of Economics and Management, Beihang University, Beijing, Chinac Department of Computer Science & Technology, Nanjing University, Nanjing, China

a r t i c l e i n f o

Article history:

Received 3 March 2012

Received in revised form

30 June 2012

Accepted 25 July 2012

Keywords:

Information-theoretic clustering

K-means

KL-divergence

Variable Neighborhood Search (VNS)

84/$ - see front matter & 2012 Elsevier B.V. A

x.doi.org/10.1016/j.sigpro.2012.07.030

esponding author.

ail address: [email protected] (J. Wu).

e cite this article as: J. Cao, et al., Tessing (2012), http://dx.doi.org/10.1

a b s t r a c t

Information-theoretic K-means (Info-Kmeans) aims to cluster high-dimensional data,

such as images featured by the bag-of-features (BOF) model, using K-means algorithm

with KL-divergence as the distance. While research efforts along this line have shown

promising results, a remaining challenge is to deal with the high sparsity of image data.

Indeed, the centroids may contain many zero-value features that create a dilemma in

assigning objects to centroids during the iterative process of Info-Kmeans. To meet this

challenge, we propose a Summation-bAsed Incremental Learning (SAIL) algorithm for

Info-Kmeans clustering in this paper. Specifically, SAIL can avoid the zero-feature

dilemma by replacing the computation of KL-divergence between instances and

centroids, by the computation of centroid entropies only. To further improve the

clustering quality, we also introduce the Variable Neighborhood Search (VNS) meta-

heuristic and propose the V-SAIL algorithm. Experimental results on various benchmark

data sets clearly demonstrate the effectiveness of SAIL and V-SAIL. In particular, they

help to successfully recognize nine out of 11 landmarks from extremely high-dimen-

sional and sparse image vectors, with the presence of severe noise.

& 2012 Elsevier B.V. All rights reserved.

1. Introduction

Image indexing aims to identify images based on theirattributes and provide access to useful groupings ofimages. Without image indexing, most of the images willremain buried in the databases, never seen by the users[25]. In general, there are two types of approaches forimage indexing: (1) The concept-based approaches, inwhich image attributes and semantic content are identi-fied and described verbally by human indexers. (2) Thecontent-based approaches, in which features of imagesare automatically identified and extracted by computersoftware. Cluster analysis, which provides insight into the

ll rights reserved.

owards information-016/j.sigpro.2012.07.0

data by dividing the objects into groups (clusters) ofobjects, shows great potential in grouping and summariz-ing images for content-based image indexing, and thereforeabsorbs much research attention [19]. Indeed, clusteringsimilar images is equivalent to looking for those graphrepresentations that are similar to each other in a database.Fig. 1 shows such an example, where cluster analysis helpsto retrieve three types of Oxford landmarks. Nevertheless,clustering a large set of images is still widely unexploredand remains one of the most challenging problems instructural pattern recognition [19,12,24,6].

Recent years have witnessed an increasing interest ininformation-theoretic clustering [3,4,28,10], as informa-tion theory [2] can be naturally adopted as the guidancefor the clustering process. This type of algorithms usuallytreats clustering as the iterative process of finding thebest partitioning on data in a way such that the loss of

theoretic K-means clustering for image indexing, Signal30

www.elsevier.com/locate/sigpro

www.elsevier.com/locate/sigpro

dx.doi.org/10.1016/j.sigpro.2012.07.030



mailto:[email protected]





Fig. 1. Cluster analysis for image indexing.

J. Cao et al. / Signal Processing ] (]]]]) ]]]–]]]2

mutual information due to the partitioning is the least [3].Our study in this paper is focused on employing Info-Kmeans,a K-means clustering algorithm [5] with KL-divergence [13]as the distance function, for the clustering of high-dimen-sional sparse data extracted from images, texts, genes, etc.For instance, images are often preprocessed by the popularbag-of-features (BOF) model [26], which treats images ascollections of unordered appearance descriptors extractedfrom local patches, and quantizes these descriptors intodiscrete ‘‘visual words’’. By doing this, an image can becharacterized by a vector of visual words, which is usually inhigh dimensionality and extremely sparse [18,7,8].

While being rooted in information theory, Info-Kmeansstill faces some challenging issues from a practical view-point. For example, research on text clustering has shownthat, the performance of Info-Kmeans is poorer than that ofthe spherical K-means, which takes the cosine similarity asthe proximity function and becomes the defacto benchmarkof text clustering [27]. We argue that, however, this largelyattributes to the zero-feature dilemma of Info-Kmeans. Thatis, for high-dimensional sparse data, the centroids of Info-Kmeans usually contain many zero-value features. Thiscreates infinite KL-divergence values and leads to a chal-lenge in assigning objects to the centroids during theiterative process of Info-Kmeans. A traditional solution isto smooth the sparse data by adding a small value to everydata object. But this smoothing technique can also degradethe clustering performance of Info-Kmeans, since the truevalues and data sparseness have been changed severely.

In light of this, we propose a Summation-bAsed Incre-mental Learning (SAIL) algorithm for Info-Kmeans clusteringin this paper. Our main contributions lie in the followingaspects. First, we identify the zero-feature dilemma of Info-Kmeans, and point out that this dilemma is the primaryreason that degrades the performance of Info-Kmeans onhigh-dimensional data. Second, we propose an objective-equivalent algorithm SAIL for Info-Kmeans clustering, whichcan avoid the zero-feature dilemma by replacing the com-putation of KL-divergence between instances and centroidsby the computation of the centroid entropies only. Third, theentropy computation in SAIL is further refined to make use ofthe data sparseness to improve the efficiency of SAIL. Finally,

Please cite this article as: J. Cao, et al., Towards information-Processing (2012), http://dx.doi.org/10.1016/j.sigpro.2012.07.0

the Variable Neighborhood Search (VNS) meta-heuristic [9] isintroduced to improve the clustering quality of SAIL, pro-vided that the clustering efficiency is not a major concern.

Our experimental results on various benchmark textcorpora have shown that, with SAIL as a booster, theclustering performance of Info-Kmeans can be significantlyimproved. Indeed, for most of the text collections, SAILproduces clustering results competitive to or even slightlybetter than the results by the state-of-the-art sphericalK-means algorithm: CLUTO. Some settings such as featureweighting, instance weighting and bisecting, have beenshown to have varied effects on SAIL, but SAIL withoutthese settings shows more robust results. Moreover, withthe help of VNS, V-SAIL further improves the clusteringquality of SAIL by searching around the neighborhood ofthe solution. Finally, SAIL and V-SAIL are employed for thereal-world high-dimensional, noisy image data clustering.Experimental results show that after noise removal usingthe CosMiner algorithm [22], SAIL and V-SAIL can success-fully recognize nine out of 11 landmarks from the extre-mely sparse Oxford_5K data set.

2. Problem definition

K-means [14] is a prototype-based, simple partitionalclustering algorithm, which attempts to find the user-specified K clusters via a two-phase iterative process.K-means has an objective function:

obj : minfckg

Xk

Xx2ck

px distðx,mkÞ, ð1Þ

where ck denotes cluster k, mk denotes the centroid of ck

(a centroid is the arithmetic mean of the cluster members),distð�Þ is the distance function, and px40 is the weight of x.

2.1. Information-theoretic K-means

As we know, different distance functions lead todifferent types of K-means. Let DðxJyÞ denote theKL-divergence [13] between two discrete distributions





Table 1Four cases in KL-divergence computation.

Case i ii iii iv

xi 40 ¼0 ¼0 40

mk,i 40 ¼0 40 ¼0

Di ð�1,þ1Þ ¼0 ¼0 þ1

J. Cao et al. / Signal Processing ] (]]]]) ]]]–]]] 3

x and y, we have

DðxJyÞ ¼X

i

xi logxi

yi

: ð2Þ

It is easy to note that in most cases DðxJyÞaDðyJxÞ, andthat DðxJyÞþDðyJzÞZDðxJzÞ cannot be guaranteed. So D isnot a metric. If we let ‘‘dist’’ be D in Eq. (1), we have theobjective function of information-theoretic K-means clus-tering (Info-Kmeans) as follows:

O1 : minfckg

Xk

Xx2ck

pxDðxJmkÞ, ð3Þ

where each instance x is normalized to a discrete dis-tribution, and mk ¼

Px2ck

pxx=P

x2ckpx is the arithmetic

mean of instances assigned to cluster ck. Let 9 � 9 denotethe sum of feature values of a vector. Since 9x9¼ 1, wehave 9mk9¼ 1, 8k.

It has been pointed out that Info-Kmeans actually aimsto minimize the loss of mutual information between theinstance variable and the feature variable after the clus-tering [3]. This illustrates why Info-Kmeans belongs to thefamily of information-theoretic clustering.

2.2. The problem of Info-Kmeans

Though having clear physical meaning, Info-Kmeans haslong been criticized for performing relatively poorly onhigh-dimensional sparse data [28]. In this section, however,we highlight an implementation challenge of Info-Kmeans.We believe this challenge is one of the major factors thatdegrade the performance of Info-Kmeans.

Assume that we use Info-Kmeans to cluster a textcorpus. To this end, we must compute the KL-divergencebetween each text vector x and each centroid mk. Inpractice, we usually let x¼ x=9x9, and then computeDðxJmkÞ by Eq. (2). Note that Eq. (2) implies that all thefeature values of x are positive real numbers. Unfortu-nately, however, this is not the case for high-dimensionaldata, which are usually very sparse in their feature space.

To illustrate this, we observe the computation of KL-divergence in the ith dimension. Let Di denote xi logðxi=mk,iÞ,we have four scenarios as follows:

1.

PP

Case 1: xi40 and mk,i40. In this case, the computa-tion of Di is straightforward, and the result can be anyreal number.

2.
Case 2: xi¼0 and mk,i ¼ 0. In this case, we can simplyomit this feature, or equivalently let Di¼0.
3.
Case 3: xi¼0 and mk,i40. In this case, logðxi=mk,iÞ ¼
log 0¼�1, which implies that the direct computa-tion is infeasible. However, by the L’ Hospital’s rule [1],limx-0þ x logðx=aÞ ¼ 0 (a40). So we can let x6xi anda6mk,i, and thus have Di¼0.

4.
Case 4: xi40 and mk,i ¼ 0. In this case, Di ¼ þ1, whichis hard to handle in practice.
We summarize the four cases in Table 1. In general, forCases 1 and 2, the computation of Di is logically reasonable.However, the computation of Di in Case 3 is somewhatweird; that is, it cannot reveal any difference between xi

and mk,i, although mk,i may deviate heavily from zero.

lease cite this article as: J. Cao, et al., Towards information-rocessing (2012), http://dx.doi.org/10.1016/j.sigpro.2012.07.0

Nevertheless, the most difficult case is Case 4. It will leadto infinite D and hinder the instance from being properlyassigned. This is particularly true for high-dimensionalsparse data, since the centroids of such data typicallycontain many zero-value features. We call this problemthe ‘‘zero-feature dilemma’’.

Problem definition: Design a new information-theoreticK-means algorithm, which can avoid the zero-featuredilemma and is particularly suitable for high-dimensionalsparse-data clustering.

3. The SAIL algorithm

In this section, we propose a new algorithm calledSummation-bAsed Incremental Learning (SAIL), for Info-Kmeans clustering.

3.1. SAIL: theoretical foundation

Let H(x) denote the Shannon entropy of a discretedistribution x. We first have the following lemma:

Lemma 1.

DðxJyÞ ¼ �HðxÞþHðyÞþðx�yÞtrHðyÞ: ð4Þ

Proof. Since HðyÞ ¼�Pd

i ¼ 1 yi log yi, it is easy to have

rHðyÞ ¼ �ðlog y1, . . . ,log ydÞT�log e� ð1, . . . ,1ÞT :

Accordingly, we have

�HðxÞþHðyÞþðx�yÞTrHðyÞ

¼Xd

i ¼ 1

xi logðxi�yiÞ|fflfflfflfflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflfflfflfflffl}ðaÞ

�log e�Xd

i ¼ 1

ðxi�yiÞ|fflfflfflfflfflfflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl}ðbÞ

:

Since ðaÞ ¼DðxJyÞ and (b)¼0 provided thatPd

i ¼ 1 xi ¼Pdi ¼ 1 yi ¼ 1, the lemma follows. &

Based on DðxJyÞ in Eq. (4), we now derive SAIL, a newvariant of Info-Kmeans. Specifically, we have the follow-ing theorem:

Theorem 1. Given pck¼P

x2ckpx, the objective function of

Info-Kmeans O1 in Eq. (3) is equivalent to

O2 : minfckg

Xk

pckHðmkÞ: ð5Þ

Proof. By Eq. (4), we have DðxJmkÞ ¼�HðxÞþHðmkÞþ

ðx�mkÞtrHðmkÞ. As a result,X

k

Xx2ck

pxDðxJmkÞ ¼X

k

pckHðmkÞ|fflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflffl}ðaÞ

�X

x

pxHðxÞ|fflfflfflfflfflfflffl{zfflfflfflfflfflfflffl}ðbÞ






�X

k

Xx2ck

pxðx�mkÞtrHðmkÞ|fflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl}

ðcÞ

:

Since mk ¼P

x2ckpxx=pck

, we haveP

x2ckpxðx�mkÞ ¼ 0,

which implies that (c)¼0. Moreover, (b) is a constantgiven the data set and the weights of the instances. Thus,the goal of O1 is equivalent to minimize (a), which com-pletes the proof. &

O2 in Eq. (5) gives the objective function of SAIL. That is,by replacing the computations of KL-divergence betweeninstances and centroids by the computations of centroidentropies only, SAIL can avoid the zero-feature dilemma ininformation-theoretic clustering of highly sparse data.

3.2. SAIL: computational issues

Here we specify the computational details of SAIL. Themajor concern is the efficiency issue.

Generally speaking, SAIL is a greedy scheme whichupdates the objective-function value ‘‘instance by instance’’.That is, SAIL firstly selects an instance from data and assignsit to the most suitable cluster. Then the objective-functionvalue and other related variables are updated immediatelyafter the assignment. The process will be repeated untilsome stopping criterion is met.

Apparently, to find the suitable cluster is the criticalpoint of SAIL. To illustrate this, suppose SAIL randomlyselects x0 from a cluster ck0 . Then, if we assign x0 to clusterck, the change of the objective-function value will be

Dk ¼O2ðnewÞ�O2ðoldÞ

¼ ðpck0�px0 ÞH

Px2ck0

pxx�px0x0

pck0�px0

!|fflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl}

ðaÞ

�pck0H

Px2ck0

pxx

pck0

!|fflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl}

ðbÞ

þðpckþpx0 ÞH

Px2ck

pxxþpx0x0

pckþpx0

� �|fflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl}

ðcÞ

�pckH

Px2ck

pxx

pck

� �|fflfflfflfflfflfflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl}

ðdÞ

, ð6Þ

where (a), (b) and (c), (d) represent the two parts ofchanges of the objective-function value due to the move-ment of x0 from cluster ck0 to cluster ck. Then x0 will beassigned to the cluster c with the smallest D.

The computation of Dk in Eq. (6) has two appealingproperties. First, it only depends on the changes occurredwithin the two involving clusters ck and ck0 . Other clustersremain unchanged and thus have no contribution to Dk.Second, both

Px2cpx and

Px2cpxx have additivity, which

can be incrementally updated for the computation of Dk.The computation of Dk in Eq. (6), however, is still time-

consuming when computing Shannon entropy in (a) or (c).Let us take (a) for example. Since the denominator changesfrom pck0

to pck0�px0 , every non-zero feature in the numera-

torP

x2ck0pxx�px0x0 will have a new value, which requires us

to update the logarithm of all these features, and thus leadsto a huge cost for data instances in high dimensionality. Todeal with this, in what follows, we refine the computationalscheme of SAIL.


Recall SAIL’s objective function O2 in Eq. (5). Let Sk

denoteP

x2ckpxx, and Sk,i the ith dimension of Sk. Since

mk ¼P

x2ckpxx=pck

, we have

Xk

pckHðmkÞ ¼ �

Xk

Xi

Xx2ck

pxxi logXx2ck

pxxi�log pck

!

¼X

k

Xx2ck

px

Xi

xi

!log pck

�X

k

Xi

Sk,i log Sk,i

¼X

k

pcklog pck

�X

k

Xi

Sk,i log Sk,i: ð7Þ

Accordingly, if we move x0 from cluster ck0 to cluster ck,the change of the objective-function value will be

Dk ¼ ðpckþp0xÞlogðpck

þp0xÞ�pcklog pck

þðpck0�p0xÞlogðpck0

�p0xÞ�pck0log pck0

þXfi9x0

ia0g

Sk,i log Sk,i�Sþk,i log Sþk,i

|fflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl}ðaÞ

þXfi9x0

ia0g

Sk0 ,i log Sk0 ,i�S�k0 ,i log S�k0 ,i

|fflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl}ðbÞ

, ð8Þ

where Sþk,i ¼ Sk,iþpx0x0i and S�k0 ,i ¼ Sk0 ,i�px0x0i.According to Eq. (8), only the non-zero features of x0

have contributions to Dk in (a) and (b), and thus willtrigger the expensive computations of logarithm. Consid-ering that a high-dimensional vector x0 is often verysparse, the computational saving due to Eq. (8) will besignificant. As a result, we adopt Eq. (8) rather than Eq. (6)for the computation of Dk in SAIL.

3.3. SAIL: algorithmic details

Here we present the implementation details of SAIL.Figs. 2 and 3 show the pseudocodes of SAIL.

Lines 1–3 in Fig. 2 are for data initialization. Twomethods, i.e., px ¼ 9x9=

Px9x9, or simply px ¼ 1, are provided

to assign weights to instances. The second one is muchsimpler and is the default setting in our experiments. Thepreprocessing of D in line 2 includes the row and columnmodeling, e.g., tf-idf, to initialize the instances and assignweights to the features. Then in line 3, we normalize theinstance x to x=9x9.

Lines 4–12 in Fig. 2 show the clustering process. Line 5 isfor initialization, where labeln�1 contains the cluster labelsof all instances, and cluSumK�ðdþ1Þ stores the summations ofthe weights and weighted instances in each cluster. That is,for k¼1,y,K, cluSumðk,1 : dÞ ¼ Sk, and cluSumðk,dþ1Þ ¼ pck

,where d and K are the numbers of features and clusters,respectively. Two initialization modes are employed in ourimplementation, i.e., ‘‘random label’’ and ‘‘random center’’(the default one).

The LocalSearch subroutine in Line 7 performs clusteringfor each instance. As shown in Fig. 3, it traverses all instancesat random as a round. In each round, it assigns each instanceto the cluster with the heaviest drop of the objective-functionvalue, and then updates the values of the relevant variables.The computational details have been given in Section 3.2.





Fig. 2. The pseudocodes of SAIL.

Fig. 3. The LocalSearch subroutine.

Fig. 4. Illustration of VNS.


Lines 8–10 show the stopping criterion in addition tomaxIter; that is, if no instance changes its label after a round,we stop the clustering. Finally, Lines 13 and 14 choose andreturn the best clustering result among the reps clusterings.

Next, we briefly discuss the convergence and complex-ity of SAIL. Since the objective-function value decreasescontinuously after reassigning each instance, and thecombinations of the labels assigned to all instances arelimited, SAIL is guaranteed to converge after finite itera-tions. The time complexity of SAIL is OðIKndÞ, where I isthe number of iterations for the convergence, K and n arethe numbers of clusters and instances, respectively, and d

is the average number of dimensions. As K is often small,and I is typically not beyond 20, the complexity of SAIL isoften very low—just as the classic K-means algorithm.However, due to the complexity of the feasible region,SAIL may converge to a local minima or a saddle point.


That is why we usually do multiple clusterings in SAIL andchoose the one with a lowest objective-function value.

4. The V-SAIL algorithm

As SAIL is apt to converge to some local minima orsaddle points, we here incorporate the Variable Neighbor-hood Search (VNS) scheme [9] to SAIL, and establish theVNS-enabled SAIL algorithm: V-SAIL.

VNS is a meta-heuristic for solving combinatorial andglobal optimization problems. The idea of VNS is to conduct asystematic change of neighborhood within the search [9].Fig. 4 schematically illustrates the search process of VNS.That is, VNS first finds an initial solution x, and then shakes inthe k-th neighborhood N k to obtain x0. Then VNS centers thesearch around x0 and finds the local minimum x00. If x00 isbetter than x, x is replaced by x00, and VNS starts to shake inthe first neighborhood of x00. Otherwise, VNS continues tosearch in the (kþ1)-th neighborhood N kþ1 of x. The set ofneighborhoods are often defined by metric function in thesolution space, e.g., Hamming distance, Euler distance,k-OPT operator, etc. The stopping condition for VNS can bethe size of the neighborhood set, the maximum CPU time,and/or the maximum number of iterations.

Fig. 5 shows the pseudocodes of V-SAIL. Generallyspeaking, V-SAIL is a two-stage algorithm employing aclustering step and a refinement step. In c-step, the SAILalgorithm is called to generate a clustering result. Thisresult serves as the departure point of the subsequentr-step, which employs the VNS scheme to refine theclustering result to a more accurate one. It is interestingto note that the ‘‘LocalSearch’’ subroutine of SAIL is alsocalled in VNS as a heuristic method to search for localminima. Some important details are as follows.

Lines 3–18 in Fig. 5 describe the r-step of V-SAIL. In Line4, for the ith neighborhood N i, the ‘‘Shaking’’ function iscalled to generate a solution label0 in N i that has a Hammingdistance Hi ¼ i� 9D9=kmax to the current solution label. Morespecifically, ‘‘Shaking’’ first randomly selects Hi instances, andthen changes their labels at random in label, which results ina new solution label0. Apparently, label0 tends to deviate label

more heavily as the increase of i. The idea is that once thebest solution in a large region has been found, it is necessaryto explore an improved one far from the incumbent solution.

In Lines 6–11, the ‘‘LocalSearch’’ subroutine of SAIL iscalled to search for a local minimum initialized on label0.





Fig. 5. The pseudocodes of V-SAIL.

Table 2Experimental text data.

ID Data set #Instance #Feature #Class CV Density

1 k1b 2340 21 839 6 1.316 0.0068

2 la1 3204 21 604 6 0.493 0.0048

3 sports 8580 126 373 7 1.022 0.0010

4 tr11 414 6429 9 0.882 0.0438

5 tr12 313 5804 8 0.638 0.0471

6 tr23 204 5832 6 0.935 0.0661

7 tr31 927 10 128 7 0.936 0.0265

8 tr41 878 7454 10 0.913 0.0262

9 tr45 690 8261 10 0.669 0.0340

10 wap 1560 8460 20 1.040 0.0167

1 http://www.mathworks.cn/help/toolbox/stats/kmeans.html.2 http://glaros.dtc.umn.edu/gkhome/views/cluto.


This implies that SAIL is not only a clustering algorithm,but also a combinatorial optimization method. Then in Lines12–14, if the minimum is smaller than the current objVal,we update the related variables, and set the solution as thenew starting point of VNS. Line 15 stops VNS either if calling‘‘Shaking’’ too many times or when a given CPU time is due.

In summary, V-SAIL well combines the complementaladvantages of SAIL and VNS. That is, VNS can help SAILavoid inferior solutions, while SAIL can help VNS fastlocate a good local minimum. However, it is worth notingthat VNS is often very time-consuming, since it does nothave a convergence strategy. As a result, V-SAIL is a betterchoice than SAIL only if the quality rather than runtime isthe major concern of the clustering.

5. Experimental validation

In this section, we demonstrate the effectiveness of SAILand its variants on high-dimensional sparse data clustering.

5.1. Experimental setup

Data sets: Ten real-world text data sets are used in ourexperiments, as listed in Table 2. ‘‘CV’’ is the coefficient ofvariation [11] used to characterize the class imbalance of datasets, and ‘‘Density’’ is the ratio of non-zero feature-values in


each text collection. A large CV indicates a severe classimbalance, and a small density indicates a high sparsity.

Clustering tools: In the experiments, we employ threetypes of clustering tools. The first one is SAIL and itsvariant V-SAIL, coded by ourselves in Cþþ. The other twoare well-known software packages for K-means cluster-ing, i.e., MATLAB v7.11 and CLUTO v2.1.1.2

The MATLAB implementation of K-means is a batch-learning version that computes the distances betweeninstances and centroids. We extend it to include KL-divergence, and use it as the traditional Info-Kmeans thatcomputes KL-divergence directly.

CLUTO is a software package for clustering high-dimensional data sets. Its K-means implementation withcosine similarity as the proximity function shows superiorperformances in text clustering [27].

The parameters of the three K-means implementationsare set to match one another for the purpose of compar-ison, and the cluster number K is set to match the numberof true classes of each data set.

Validation measures: Many recent studies on clusteringuse the Normalized Mutual Information (NMI) to evaluatethe clustering performance [28]. For consistency, we alsouse NMI in our experiments, which can be computed as:

NMI¼ IðX,YÞ=ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiHðXÞHðYÞ

p, where the random variables X

and Y denote the cluster and class sizes, respectively,IðX,YÞ is the mutual information between X and Y, andH(X) and H(Y) are the Shannon entropies of X and Y,

respectively. Note thatffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiHðXÞHðYÞ

phere serves as the

normalization factor for I. The value of NMI is in theinterval: [0,1], and a larger value indicates a betterclustering result. More computational details of NMI canbe found in [21] from a contingency-table perspective.

5.2. The impact of zero-value dilemma

Here, we demonstrate the negative impact of the zero-feature dilemma to Info-Kmeans. Since the MATLAB imple-mentation of Info-Kmeans can handle infinity (denoted asINF), we select tr23 and tr45 as the test data sets andemploy MATLAB Info-Kmeans on them. The clusteringresults are shown in Table 3, where CV0 and CV1 representthe distributions of the class and cluster sizes, respectively.


http://www.mathworks.cn/help/toolbox/stats/kmeans.html

http://glaros.dtc.umn.edu/gkhome/views/cluto





As indicated by the close-to-zero NMI values, the cluster-ing performance of MATLAB Info-Kmeans without smoothingis extremely poor. Also, by comparing the CV0 and CV1

values, we find that the distributions of the resulting clustersizes are much more skewed than the distributions of theclass sizes. In fact, for both data sets, nearly all the textvectors have been assigned to ONE cluster!

This experimental result well confirms our analysis inSection 2; that is, Info-Kmeans will face the serious zero-feature dilemma when clustering highly sparse data sets.

5.3. The comparison of SAIL and the smoothing technique

Here we illustrate the effect of the smoothing techni-que on sparse data sets. We use MATLAB Info-Kmeans andtake seven text collections for illustration. For thesmoothing method, we try a series of added values, i.e.,10�5, 10�4, 10�3, 10�2, 10�1, 1, and select the best onefor the comparison.

tr11 tr12 tr23 tr0

0.1

0.2

0.3

0.4

NMI

0.5

0.6

0.7SmootSAIL

Fig. 6. SAIL versus

Table 3Clustering results of Info-Kmeans in MATLAB.

Data set NMI CV0 CV1

tr23 0.035 0.935 2.435

tr45 0.022 0.669 3.157

k1b la1 sports tr11 tr120

0.1

0.2

0.3

0.4

0.5

0.6

0.7

NMI

Fig. 7. SAIL vers


We also test SAIL on these data sets for the comparisonpurpose. The parameters are set as follows: (1) px ¼ 1,8x 2 D; (2) no row or column modeling; (3) the initializationmode is ‘‘random center’’; (4) reps¼1, but we repeat SAIL 10times for each data set and return the average NMI value.Unless otherwise stated, these are the default settings of SAILin our experiments.

Fig. 6 shows the comparison results. As can be seen,the clustering results of SAIL are consistently superior tothe ones using the smoothing technique. For some datasets such as tr11, tr12, tr41 and tr45, SAIL takes thelead with a very wide margin.

5.4. The comparison of SAIL and spherical K-means

In this subsection, we compare the clustering perfor-mances between SAIL and the spherical K-means [27]. Inthe literature, people have shown that spherical K-meansusually produces better clustering results than Info-Kmeans [28]. However, we would like to show in thisexperiment that the performance of SAIL is comparable toor even slightly better than the benchmark spherical K-means algorithm: CLUTO.

In this experiment, the parameter setting of CLUTO is:clmethod¼direct, crfun¼ i2, sim¼cosine, colmodel¼none,ntrials¼10. For SAIL, we use the default setting as for theprevious experiment. Fig. 7 shows the clustering results,

31 tr41 tr45 wap

hing

Smoothing.

tr23 tr31 tr41 tr45 wap

CLUTO (no−IDF)CLUTO (IDF)SAIL

us CLUTO.





k1b la1 sports tr11 tr12 tr23 tr31 tr41 tr45 wap0.35

0.4

0.45

0.5

0.55

0.6

0.65

0.7

0.75SAILV−SAIL

NMI

Fig. 8. SAIL versus V-SAIL.


where ‘‘CLUTO (no IDF)’’ indicates the spherical K-meansclustering without using the IDF (Inverse-Document-Fre-quency) weighting [20]. As can be seen, SAIL beats CLUTO(no IDF) at all 10 data sets. For some data sets, e.g., la1,sports and tr31, SAIL even shows dominant advantages.This result demonstrates that SAIL is particularly suitable forhigh-dimensional data clustering, even compared with thestate-of-the-art methods.

As it is reported that feature weighting often makes greatimpact to the spherical K-means, we also compare theclustering performances of SAIL and CLUTO with IDF. Fig. 7shows the comparison result. Two observations are notableas follows. First, although CLUTO with IDF improves theclustering quality of CLUTO without IDF in five out of tendata sets, it still shows poorer performance than SAIL ineight out of ten data sets. Second, for some data sets, such astr11, tr12, tr41 and tr45, the IDF scheme actuallyseriously degrades the clustering performance of CLUTO.These observations imply that feature weighting is an x-factor for spherical K-means without the guidance of exter-nal information. In contrast, SAIL with default setting showsstable clustering quality and therefore is more robust inpractice.

5.5. The comparison of SAIL and V-SAIL

In this subsection, we take a further step to explore theproperties of SAIL, and compare the performance of SAILand V-SAIL.

We firstly investigate some factors that may impact theperformance of SAIL. Specifically, we introduce the featureweighting, instance weighting and bisecting schemes forSAIL, and observe the variation of clustering performance.Table 4 shows the results. Note that ‘‘Default’’ representsSAIL with defaulting settings, ‘‘IDF’’ represents SAIL usingthe IDF scheme, and ‘‘Prob.’’ represents SAIL with textsweighted by 9x9=

Px9x9. As to ‘‘Bisecting’’, it represents a

top-down divisive variant of SAIL. That is, it first divides allinstances into two clusters using SAIL; and then repeatedlyselects one cluster according to a certain criterion, anddivides that cluster into two sub-clusters using SAIL again;the procedure will continue unless the desired K clustersare found. In our experiment, we use two cluster-selection

Table 4Impact of parameter settings of SAIL.

ID Data set Default IDF Prob. B-Size B-Obj

1 k1b 0.648 0.568 0.607 0.538 0.554

2 la1 0.595 0.571 0.559 0.578 0.574

3 sports 0.685 0.650 0.533 0.641 0:002

4 tr11 0.640 0.584 0.578 0.642 0.6585 tr12 0.645 0.556 0:447 0.553 0.545

6 tr23 0.385 0.403 0:147 0.364 0.327

7 tr31 0.545 0.525 0.550 0.616 0.5508 tr41 0.645 0.623 0.684 0.585 0.589

9 tr45 0.618 0.685 0.527 0.630 0.602

10 wap 0.584 0.543 0.594 0.558 0.548

Note: (1) Measured by average NMI.

(2) The value larger than the one in ‘‘Default’’ is in bold.

(3) The value far smaller than the one in ‘‘Default’’ is underlined.


criteria for bisecting SAIL, where ‘‘B-Size’’ chooses thelargest cluster, and ‘‘B-Obj’’ chooses the one with thelargest objective-function value.

As can be seen in Table 4, one observation is that‘‘Prob.’’ and ‘‘B-Obj’’ may produce very poor clusteringresults (scores underlined), and therefore cannot be usedfor SAIL. In contrast, ‘‘IDF’’ and ‘‘B-Size’’ produce relativelyrobust results, and in some cases even improve SAILslightly (scores in bold), thus can be considered as valu-able supplements to SAIL. Nonetheless, SAIL with defaultsettings produces competitive clustering results in mostcases, and therefore is the most robust one amongdifferent settings.

For the experiments of V-SAIL, we set kmax ¼ 12, andreturn the average NMI values of 10 repetitions. The stop-ping criterion for V-SAIL is the maximum times for calling‘‘LocalSearch’’, which we set to 500. Fig. 8 shows thecomparison results of V-SAIL and SAIL. As can be seen fromthe figure, V-SAIL generally achieves higher NMI scores thanSAIL. For some data sets, such as tr31, tr41 and tr45, theimprovements are quite considerable. Nonetheless, it isworthy of noting that V-SAIL often consumes much moretime than SAIL for the iterations of VNS.

6. Application to image data clustering

In this section, we exploit SAIL and V-SAIL algorithmsfor high-dimensional image clustering. We begin bybriefly introducing the data set and the clustering proce-dure. Then we present the results and evaluations.

6.1. The image data

Our target is the Oxford_5K data set,3 which is an imagecollection retrieved from Flickr,4 using 17 queries includingseveral specific queries such as ‘‘All Souls Oxford’’, ‘‘BalliolOxford’’ and ‘‘Jesus Oxford’’, and some general queries suchas ‘‘Oxford’’ and ‘‘New Oxford’’. In total, it contains 5060images of 11 different Oxford landmarks—a landmark heremeans a particular part of a building.

Each image is then scanned for ‘‘salient’’ regions, and ahigh-dimensional descriptor is computed for each region.These descriptors are then quantized into a vocabulary of

3 http://www.robots.ox.ac.uk/�vgg/data/oxbuildings/index.html4 http://www.flickr.com


http://www.robots.ox.ac.uk/~vgg/data/oxbuildings/index.html

http://www.robots.ox.ac.uk/~vgg/data/oxbuildings/index.html

http://www.flickr.com





visual words. As a result, an image is represented by a bag of1M visual words, with feature values being the numbers ofword occurrences. Table 5 shows some characteristics ofthis image collection. Note that the features occurring lessthan 3 times across the entire data set are deleted.

As can be seen from Table 5, Oxford_5K has twonotable features. First, it is a very sparse high-dimensionaldata set, whose density (the ratio of non-zero entities) isonly slightly higher than 0.02%. Second, since the imagesare retrieved by simple queries, Oxford_5K contains alarge amount of vague or even irrelevant images. Indeed,as pointed out in [15], if we take the images with morethan 25% of the object clearly visible as normal instances,there are only 568 normal instances in Oxford_5K, orequivalently, roughly 88.77% instances are noise!

6.2. The clustering procedure

Given the Oxford_5K data set in Table 5, our task is todo image clustering, and then identify the 11 Oxford land-marks from the clusters. To this end, we must deal with twointractable challenges, i.e., the high sparsity and the vastnoise. Fig. 9 shows some normal and particularly noise

Fig. 9. Two landmark

Table 5Some characteristics of image data.

Name #Instances #Features #Class Density Noise

Oxford_5K 5060 658 346 11 0.0228% 88.22%


images for the two landmarks: All Souls and Ashmolean,which may seriously degrade the clustering quality. Toaddress these challenges, we employ a three-step procedureas follows (the whole procedure is summarized in Fig. 10):

�

s an

the30

Step 1: Noise removal. Since there are huge volume ofnoise in the data, we first launch a noise-filtering stepbased on the CosMiner algorithm proposed in [22]. Thatis, we first transform Oxford_5K into a transaction data(or equivalently, the market-basket data), and thenemploy CosMiner on the new data for the so-called‘‘cosine interesting patterns’’ (CIPs), i.e., itemsets that havesupport and cosine values larger than the user-specifiedsupport and cosine thresholds, respectively [22,23]. In ourcase, a CIP is a set of words that occurs simultaneously insome images. Images that cover no CIP are treated asnoise, and therefore removed from the data.
� Step 2: Image clustering. In this step, we employ SAIL
and V-SAIL on the data after noise removal. Thenumber of clusters K is set to 11.
� Step 3: Landmark recognition. In this step, we try to
recognize the 11 landmarks from the resultant 11clusters. Since the noise may not be fully removed inthe first step, we manually remove the remainingnoise in each cluster, and have the landmark inmajority as the landmark of that cluster.

6.3. The results

Table 6 shows the parameter setting and the results ofnoise removal using CosMiner, where min_supp and min_

d the noise.

oretic K-means clustering for image indexing, Signal




Oxford_5KData

TransactionData

Patterndiscovery

CosMiner

Noiseremoval

Imageclustering

Landmarkrecognition

SAIL &V-SAIL

CIPs

Fig. 10. Overview of the clustering procedure.

Table 6Parameter settings and results of noise removal.

Case Parameter setting #Patterns #Images

min_supp min_cos

1 – – – 5060

2 0.09% 0.48 12 675 1001

3 0.10% 0.48 1327 559


cos are the minimum support and cosine thresholdsrespectively for cosine interesting pattern mining.We consider three cases for the purpose of comparisonin the subsequent image clustering. In Case 1, we do notemploy noise removal, and therefore have all the 5060instances remained. In Case 2, we set a relatively lowermin_supp and generate over 12 000 patterns, which leadto the remaining 1001 instances. Finally, in Case 3, wegenerate less patterns and therefore have only 559instances remained. It is noteworthy that less instancescan lower the manual labor for noise removal, but mayincrease the risk of missing too many normal imagesremoved by CosMiner as noise.

We then employ SAIL and V-SAIL for the remainingimages, remove the noise in each cluster, and recognizethe landmarks from the resultant clusters. Results returnedby V-SAIL in Case 2 indicate that nine out of 11 landmarksare successfully found. Fig. 11 shows 10 images for each ofthe four sample landmarks identified. Among them, AllSouls and Radcliffe Cam are recognized easily. Forinstance, in the cluster for All Souls, 62 out of 63 instancesare the correct images, which comes near to perfection. Incontrast, the other two landmarks, i.e., Magdalen andAshmolean, are much harder to recognize. For instance,the cluster for Magdalen contains 12 instances, but onlyhalf of them are the correct images (the images in therectangle are the incorrect ones in the cluster). Finally, it isnoteworthy that the two landmarks, i.e., Cornmarket andKeble, cannot be recognized from the clusters due to therareness of the samples—there are only one and threeimages for these two landmarks in the data.

Table 7 lists the results of clustering evaluation for thethree cases in terms of the NMI measure. For each case, we


do clustering 5 times and compute the average score aftermanual noise-removal. One observation is that, withoutnoise removal in the first step, the clustering quality of Case1 is much poorer than the ones of the other cases. This isbecause vast noise will exert very bad influence on thecentroid computation of SAIL, and eventually bias theclustering result. Moreover, although the clustering qualityof Case 3 is the best, it indeed misses many normal imagessuch that four landmarks totally disappear in the clusters.This implies that we should find a good balance betweennoise removal and image completeness. That is also why weregard Case 2 as the best one among the three cases.

In summary, both SAIL and V-SAIL show excellentclustering performance in the application of landmarkimage clustering, even with the presence of vast noise.

7. Related work

In the literature, great research efforts have been takento incorporate information theoretic measures into existingclustering algorithms, such as K-means [3,4,28,10]. How-ever, the zero-feature dilemma remains a critical challenge.

For instance, Dhillon et al. proposed information-theoretic K-means, which used the KL-divergence as theproximity function [3]. While the authors noticed the‘‘infinity’’ values when computing the KL-divergence, theydid not provide specific solutions to this dilemma. Inaddition, Dhillon et al. further extended information-theoretic K-means to the so-called information-theoreticco-clustering [4]. This algorithm is the two-dimensionalversion of information-theoretic K-means which mono-tonically increases the preserved mutual information byinterwinding both the row and column clusterings at allstages. Again, however, there is no solution provided forhandling the zero-feature dilemma when computing theKL-divergence.

Since many other proximity functions such as thesquared Euclidean distance and the cosine similarity canalso be used in K-means clustering [17], a natural idea isto compare their performances in practice [28] is oneof such studies. The authors argued that the sphericalK-means produces better clustering results than Info-Kmeans [16,27] also showed the merits of sphericalK-means in text clustering. However, these studies did





Fig. 11. Four sample landmarks recognized using V-SAIL.

Please cite this article as: J. Cao, et al., Towards information-theoretic K-means clustering for image indexing, SignalProcessing (2012), http://dx.doi.org/10.1016/j.sigpro.2012.07.030





Table 7Performance of image clustering (measured by NMI).

Case i ii iii iv v Avg.

1 SAIL 0.105 0.095 0.083 0.090 0.079 0.091

V-SAIL 0.115 0.106 0.097 0.084 0.091 0.099

2 SAIL 0.601 0.633 0.576 0.541 0.523 0.575

V-SAIL 0.643 0.627 0.646 0.620 0.599 0.627

3 SAIL 0.652 0.632 0.550 0.651 0.595 0.616

V-SAIL 0.702 0.616 0.652 0.642 0.561 0.635


not tell us why Info-Kmeans shows inferior performanceto spherical K-means and how to enhance Info-Kmeans.

Our paper indeed fills this crucial void by proposingSAIL and V-SAIL to handle the zero-feature dilemma, andimproving the clustering performance of Info-Kmeans to alevel competitive to the spherical K-means.

8. Concluding remarks

This paper studied the problem of information-theoreticK-means clustering (Info-Kmeans) for high-dimensionalsparse data, such as images and texts featured by the bag-of-words model. In particular, we revealed the zero-featuredilemma of Info-Kmeans in assigning objects to the cen-troids. To deal with it, we developed a Summation-basedIncremental Learning (SAIL) algorithm, which can avoid thezero-feature dilemma by computing the Shannon entropyinstead of the KL-divergence. The effectiveness of thisreplacement is guaranteed by an equivalent mathematicaltransformation in the objective function of Info-Kmeans.Moreover, we proposed the V-SAIL algorithm to furtherenhance the clustering ability of SAIL. Finally, as demon-strated by extensive benchmark data sets in the experimentand a challenging image data set in the application, SAIL andV-SAIL can greatly improve the performance of Info-Kmeanson high-dimensional sparse data, even with a huge volumeof noise inside.

Acknowledgments

This research was partially supported by the NationalNatural Science Foundation of China (Nos. 71072172,61103229), Jiangsu Provincial Colleges and UniversitiesOutstanding S&T Innovation Team Fund (No. 2001013),and Key Project of Natural Science Research in JiangsuProvincial Colleges and Universities (No. 12KJA520001).The third author was also supported in part by the NationalNatural Science Foundation of China (Nos. 70901002,71171007, 71031001, 70890080, 90924020).

References

[1] L. Brand, Advanced Calculus: An Introduction to Classical Analysis,Dover Publications, 2006.

[2] T.M. Cover, J.A. Thomas, Elements of Information Theory, 2ndedition, Wiley-Interscience, 2006.


[3] I.S. Dhillon, S. Mallela, R. Kumar, A divisive information-theoreticfeature clustering algorithm for text classification, Journal ofMachine Learning Research 3 (2003) 1265–1287.

[4] I.S. Dhillon, S. Mallela, D.S. Modha, Information-theoretic co-clus-tering, in: Proceedings of the 9th International Conference onKnowledge Discovery and Data Mining, 2003, pp. 89–98.

[5] L. Galluccio, O. Michel, P. Comon, A.O. Hero III, Graph based k-means clustering, Signal Processing 92 (9) (2012) 1970–1984.

[6] N. Guan, D. Tao, Z. Luo, B. Yuan, Non-negative patch alignmentframework, IEEE Transactions on Neural Networks 22 (8) (2011)1218–1230.

[7] N. Guan, D. Tao, Z. Luo, B. Yuan, Non-negative patch alignmentframework, IEEE Transactions on Neural Networks 22 (8) (2011)1218–1230.

[8] N. Guan, D. Tao, Z. Luo, B. Yuan, NeNMF: an optimal gradientmethod for nonnegative matrix factorization, IEEE Transactions onImage Processing 60 (6) (2012) 2882–2898.

[9] P. Hansen, N. Mladenovic, Variable neighborhood search: principlesand applications, European Journal of Operational Research 130(2001) 449–467.

[10] Z. He, A. Cichocki, Y. Li, S. Xie, S. Sanei, K-hyperline clusteringlearning for sparse component analysis, Signal Processing 89 (June(6)) (2009) 1011–1022.

[11] W.A. Hendricks, K.W. Robey, The sampling distribution of thecoefficient of variation, Annals of Mathematical Statistics 7 (3)(1936) 129–132.

[12] M. Kotti, V. Moschou, C. Kotropoulos, Speaker segmentation andclustering, Signal Processing 88 (May (8)) (2008) 1091–1124.

[13] S. Kullback, R.A. Leibler, On information and sufficiency, Annals ofMathematical Statistics 22 (1) (1951) 79–86.

[14] J. MacQueen, Some methods for classification and analysis ofmultivariate observations, in: Proceedings of the 7th BerkeleySymposium on Mathematical Statistics and Probability, 1967,pp. 281–297.

[15] J. Philbin, O. Chum, M. Isard, J. Sivic, A. Zisserman, Object retrievalwith large vocabularies and fast spatial matching, in: Proceedingsof the IEEE Conference on Computer Vision and Pattern Recogni-tion, 2007.

[16] M. Steinbach, G. Karypis, V. Kumar, A comparison of documentclustering techniques, in: Proceedings of the 6th InternationalConference on Knowledge Discovery and Data Mining, World TextMining Conference, 2000.

[17] P.N. Tan, M. Steinbach, V. Kumar, Introduction to Data Mining,Addison-Wesley, 2005.

[18] X. Tian, D. Tao, Y. Rui, Sparse transfer learning for interactive videosearch reranking, ACM Transactions on Multimedia Computing,Communications and Applications (8) .

[19] M. Wang, Z. Ye, Y. Wang, S. Wang, Dominant sets clustering forimage retrieval, Signal Processing 88 (11) (2008) 2843–2849.

[20] H.C. Wu, R.W.P. Luk, K.F. Wong, K.L. Kwok, Interpreting tf-idf termweights as making relevance decisions, ACM Transactions onInformation Systems 26 (3) (2008) 1–37.

[21] J. Wu, H. Xiong, J. Chen, Adapting the right measures for k-meansclustering, in: Proceedings of the 15th International Conference onKnowledge Discovery and Data Mining, World Text Mining Con-ference, 2009.

[22] J. Wu, S. Zhu, H. Liu, G. Xia, Cosine interesting pattern discovery,Information Sciences 184 (1) (2012) 176–195.

[23] J. Wu, S. Zhu, H. Xiong, J. Chen, Adapting the right measures forinteresting pattern discovery: a unified view, IEEE Transactions onSystems, Man and Cybernetics: Part B 42 (4) (2012) 1203–1214.

[24] T. Xia, D. Tao, T. Mei, Y. Zhang, Multiview spectral embedding, IEEETransactions on Systems, Man, and Cybernetics, Part B 40 (6)(2010) 1438–1446.

[25] C.W. Yang, J.J. Shen, Recover the tampered image based on VQindexing, Signal Processing 90 (1) (2010) 331–343.

[26] J. Yang, K. Yu, Y. Gong, T. Huang, Linear spatial pyramid matchingusing space coding for image classification, in: Proceedings of theIEEE International Conference on Computer Vision and PatternRecognition, 2009.

[27] Y. Zhao, G. Karypis, Criterion functions for document clustering:experiments and analysis, Machine Learning 55 (3) (2004) 311–331.

[28] S. Zhong, J. Ghosh, Generative model-based document clustering: acomparative study, Knowledge and Information Systems 8 (3)(2005) 374–384.





towards information-theoretic k-means clustering …jselab.njue.edu.cn/download/journal/towards...

Documents