Learning Topics in Short Texts by Non-negative Matrix Factorization

on Term Correlation Matrix∗

Xiaohui Yan†, Jiafeng Guo†, Shenghua Liu†, Xueqi Cheng†, Yanfeng Wang‡

Abstract

Nowadays, short texts are very prevalent in various web

applications, such as microblogs, instant messages. The

severe sparsity of short texts hinders existing topic models

to learn reliable topics. In this paper, we propose a novel

way to tackle this problem. The key idea is to learn

topics by exploring term correlation data, rather than the

high-dimensional and sparse term occurrence information in

documents. Such term correlation data is less sparse and

more stable with the increase of the collection size, and can

well capture the necessary information for topic learning.

To obtain reliable topics from term correlation data, we

first introduce a novel way to compute term correlation in

short texts by representing each term with its co-occurred

terms. Then we formulated the topic learning problem as

symmetric non-negative matrix factorization on the term

correlation matrix. After learning the topics, we can easily

infer the topics of documents. Experimental results on three

data sets show that our method provides substantially better

performance than the baseline methods.

1 Introduction

Nowadays, short texts are very prevalent in various webapplications, such as microblogs, SNS statuses, instantmessages, video titles, and so on. These type of data al-ways summarizes all types of information, like the mostrecent personal information, or news events. Therefore,it is increasingly important to understand and representshort texts properly for many text processing tasks, likeemerging topics discovery[4] in social media, efficient in-dex and retrieval[21], personalized recommendation[17],etc.

Topic models, like PLSA[11] and LDA[2], provide aprincipled way to represent and analyze text collectionby uncovering the hidden thematic structure of it auto-

∗This work is funded by the National Natural Science Foun-dation of China under Grant No. 61202213, No. 60933005, No.

61173008, No. 61003166, and 973 Program of China under GrantsNo. 2012CB316303.

†Institute of Computing Technology of the Chinese Academyof Sciences, Beijing. Email: yanxiaohui@software.ict.ac.cn,

guojiafeng,liushenghua,cxq@ict.ac.cn‡Sogou Inc, Beijing. Email: wangyanfeng@sogou-inc.com

X UV

S U

X U

=

=

=

×

×

×

UT

V

(a)

(b)

(c)

Figure 1: (a) The conventional topic models decomposethe extremely sparse term-document matrix X into theterm-topic matrix U and the topic-document matrix V ;Our approach (b) learns U by symmetric factorizationon a dense term correlation matrix S, and then (c) solvesthe topic-document matrix V with U learned in hand.

matically. However, conventional topic models usuallytarget at normal text, whose effectiveness will be highlyinfluenced when the document length reduces, as the ex-periments of [12] shown. An intuitive explanation is likefollows. Most conventional topic models, like PLSA andNMF[16], learn topics by decomposing the the so-calledterm-document matrix into two low-rank matrices, il-lustrating in Figure 1(a). However, the term-documentmatrix which represents the term occurrence informa-tion in documents, is extremely sparse as for short texts.More formally, Let M , N , K be the number of distinctterms, documents, and topics, respectively. We haveMK +NK latent variables to be estimated for the twolow-rank matrices U and V , where on average MK

N +Klatent variables for each document. When the docu-ments are very short, e.g. with 10 or less terms, theproblem becomes highly underdetermined for a large K.In other words, we may not be able to learn the topicsreliably for such sparse data.

To overcome this problem, we propose a novel wayto learn topics from short texts. The key idea isthat since the term-document matrix is too sparse toestimate reliable topics, we shall turn to more stableand dense data for this purpose. As we know, topicsare mainly uncovered based on the correlations betweenterms. For instance, if the terms “President” and“Obama” co-occur frequently, they might talk aboutthe same topic. Meanwhile, we observe that when the

0 2 4 6 8 10

x 104

0

2

4

6

8

10x 10

4

Collection size

Voca

bu

lary

siz

e

without removing rare terms

Rare terms removed

Figure 2: The Vocabulary size grows slowly as thecollection size increasing on 10w twitter posts, whenrare terms with document frequency < 4 are removed.

size of the short text corpus becomes larger and larger,the size of distinct terms usually keeps relative smalland stable. Figure 2 illustrates this phenomenon, asthe size of Twitter posts increases from 104 to 105, thevocabulary size almost keeps the same in Twitter2011data set, which will be detailed in the Experimentssection. Therefore, it is feasible to directly estimatetopics from term correlation data rather than the sparseterm-document matrix.

The consequent question is how to obtain the termcorrelation data from short texts. A straightforwardway is to directly use the term co-occurrences. It isequivalent to represent each distinct term by a docu-ment vector, in which each entry indicates the occur-rence of the term in a document, and then measuringcorrelation between terms via similarity measures likecosine similarity, or Pearson correlation. However, sucha way also suffers from the sparsity problem in shorttexts, since the dimension of document vector could bevery high for a large corpus, the term-document vec-tor turns out to be very sparse. Instead, we employ analternative term correlation measure for short texts byrepresenting each term by a vector of co-occurred termsrather than documents, and then compute correlation ofthese vectors. Since the vocabulary size is much smallerand more stable than collection size in short texts, thiscorrelation measure does not suffer from the sparsityproblem.

Specifically, our approach for topic learning in shorttexts consists of two steps, shown in Figures 1 (b)and (c). First, we construct a term correlation matrixS based on the proposed term correlation measure.And then we apply symmetric non-negative matrixfactorization on the correlation matrix to learn thetopics, which result in the term-topic matrix U , asshown in Figure 1(b). Second, we infer the topics ofdocuments by solving the topic-document matrix V ,

according to the original term-document matrix X andthe term-topic matrix U in hand, as shown in Figure1(c).

We develop efficient algorithms for topic learningand inference in short texts, and test our approach onthree real-world short text data sets. Experimentalresults demonstrate the effectiveness of the proposedtopic learning method in topic visualization, documentclustering, and document classification.

The major contributions of our approach on shorttext topic learning are as follows:

• So far as we know, it is the first attempt to learntopics directly from the term correlation matrixrather then term-document matrix. In this way,we can largely alleviate the data sparsity problemwhen applying topic models on large scale shorttext collections.

• We propose to measure the correlation betweenterms by representing each term with its co-occurred terms rather than the documents it oc-curred. Hence, we can avoid the sparsity problemin term representation and better measure the termcorrelation.

• We developed efficient algorithms for topic learningand inference in our approach.

• We conduct extensive experiments to demonstratethe effectiveness and efficiency of the proposedapproach.

The rest of the paper is organized as follows. Sec-tion 2 introduces related work. Section 3 describes oursolution to short texts topic model. Section 4 discussesthe algorithms for topic learning and inference. Exper-imental results are presented in Section 5. Conclusionsare made in the last section.

2 Related Work

There is considerable amount of literature on topiclearning for text data in the past decade. From theview of methodology, they fall into two groups: non-probabilistic approaches and probabilistic approaches.

Most of non-probabilistic approaches are based onmatrix factorization techniques, which project the term-document matrix into a K-dimensional topic space. Forexample, the early work LSI [6] employed SVD to iden-tify latent semantic factors with orthogonal constraints.Some recent works utilized the sparse constraint to sub-stitute the orthogonal ones, like the regularized LSI [22]and sparse LSA [5]. However, these methods lack an in-tuitive interpretation for the negative values in results[24]. A more popular way is NMF [16, 24, 19], which

introduced non-negative constraint instead. With thenon-negative constraints, only additive operator is al-lowed in factorization. Therefore, NMF is believed tolearn part-based representations of dataset. From theview of topic modeling, NMF decomposes the term-document matrix into two low-rank non-negative ma-trices: a term-topic matrix, each column represents atopic as a convex combination of terms; and a topic-document matrix, each column represents a documentas a convex combination of topics. Our work falls intothis group.

Probabilistic topic models are also very popular.The representative models are the probabilistic LatentSemantic Analysis (PLSA) of Hofmann [11], and the La-tent Dirichlet Allocation (LDA) of Blei[2]. PLSA mod-els each document as a mixture of topics, which equalsto NMF with KL divergence[10]. LDA incorporates theDirichlet priors for topic mixtures, thus it is less proneto over-fitting and capable of inferring topics for unob-served documents.

While all of the above topic models deal with nor-mal texts, there are some recent works advocating forshort text medias, such as Twitter posts. For example,[23, 12] proposed to train LDA on “fake” documentsby aggregating tweets of users. Ramage et.al[18] devel-oped a partially supervised learning method (LabeledLDA) to model Twitter posts with the help of hashtaglabels. Yan et.al[25] proposed the Ncut-weighted NMFfor short text clustering by utilizing term correlationinformation, too. Different from them, we studied theproblem of topic learning for general short texts whichis domain-independent.

3 Our Approach

In our work, we learn topics from term correlations,rather than term-documents matrix, which is extremelysparse for short texts. At first, a term correlationmatrix is constructed. And then, symmetric non-negative matrix factorization is performed to learn thetopic representation of the terms. Finally, by projectingdocuments into the low-dimensional topic space, weare capable to infer the topical representation of eachdocument.

3.1 Term Correlation Matrix ConstructionConventionally, documents are presented via the well-known Vector Space Model, which models a collectionof documents by a term-document matrix X, with eachentry xij indicating the weight (typically by TFIDF)of the term ti in document dj . In other words, a termti can be viewed as a document vector (xi,1, ..., xi,N ),where N is the number of documents in the collection.To measure the correlation of two terms, a common way

(a)500 1000 1500 2000 2500

500

1000

1500

2000

2500

(b)500 1000 1500 2000 2500

500

1000

1500

2000

2500

Figure 3: Visualization of Term correlation matrix onTweets corpus computed via (a) document vector repre-sentation of terms, density=0.0415; (b) via co-occurredterm vector representation of terms, density=0.8835.

is to calculate the similarity of their vector representa-tions. However, the document vector representation stillsuffers from the extreme sparsity problem in short textdata, because of the shortness of short texts limits eachterm occurs in a relative small part of documents in acollection. Figure 3(a) visualizes the term correlationmatrix computed via document vector represenation onthe Tweets corpus. We can see the result matrix ishighly sparse, only with denstiy 0.0415.

Instead, here we suggest to represent a term by aterm co-occurrence vector, rather than the documentvector. The original idea comes from the famousdictum—“You shall know a word by the company itkeeps!”[9], in natural language processing field, whichtells us that the meaning of a word can be decidedby the distribution of words around it. For example,considering the word “xbox” occurring in the followingshort text snippets: “xbox live game downloading”,“can xbox play Left 4 Dead”, and “xbox 360 vs. ps3”,it is not difficult to deduce that “xbox” refers to a gamemachine on the basis of the co-occurred words “game”,“play” and “ps3”.

After then, we can also measure the similarity of twoterms according to their co-occurred words distribution.In detail, a term ti is presented by a term occurrencevector (wi,1, ..., wi,M ), where wi,m is decided by the co-occurrence of terms ti and tm. In this work, we applythe positive point mutual information (PPMI) to assesswi,m:

wi,m = PPMI(ti, tm) = max(logP (ti, tm)

P (ti)P (tm), 0),

where the probabilities P (ti, tm) and P (ti) are esti-mated empirically:

P (ti, tm) =n(ti, tm)∑j,l n(tj , tl)

, P (ti) =

∑m n(ti, tm)∑j,l n(tj , tl)

,

where n(ti, tm) is the times of terms ti and tm co-occurred. After representing each term by the term co-

occurrence vector, we then apply the common vector-similarity measures, like cosine coeffcient, to computethe correlation between any two terms, resulting in thefinal term correlation matrix S.

The above method to estimate the correlation be-tween two terms is known as distributional methods innatural language processing fields[14]. Figure 3(b) vi-sualizes the term correlation matrix computed via termco-occurrence vector representation on the Tweets cor-pus, it is with much higher density(0.8835) than Figure3(a).

3.2 Topics Learning In probabilistic topic models,a “topic” is considered as a distribution over terms[2].Moreover, a meaningful “topic” should be a distributionconcentrated on terms about a particular subject. Ac-cordingly, in non-probabilistic topic models, a “topic”can be viewed as a group of terms with weights indi-cating the importance or significance in some subject.Consequently, to discover the topics equal to cluster theterms into some meaningful groups. On the other hand,terms usually have multiple meanings, and may belongto multiple topics. Therefore, a direct way to performtopic learning is to conduct “soft” clustering based onthe term correlation matrix.

Motivated by previous works about graphclustering[15], we formulate this topic learningproblem as finding a term-topic matrix U to minimizethe following objective function:

(3.1) L(U) = ∥S − UUT ∥2F , s.t. U ≥ 0.,

where each column of the term-topic matrix U repre-sents a topic by a vector of weighted terms. This spe-cial formulation of non-negative matrix factorization isreferred as the symmetric non-negative matrix factor-ization, which is suggested to be equivalent to kernelKmeans clustering and spectral clustering[7].

3.3 Topics Inference for Documents The term-topic matrix U uncovers the latent topic structure of thecollection. Once it obtained, we can subsequently inferthe topic presentations of documents, namely the topic-document matrix V by projecting the documents intothe latent topic space. In the non-negative matrix fac-torization framework, this problem can be formulatedas finding a non-negative matrix V to minimize the dis-tance between the product UV andX, while given term-topic matrix U .

To judge the optimal solution, we need to know thehow well the estimated model fits the observed data.Two popular distance functions are used for measuringthe lost. One is the square of the Euclidean distance:

(3.2) LE(V ) =∥ X − UV ∥2F .

The other one is the generalized I-divergence as thefollowing:

LI(V ) = D(X ∥ UV )

=∑ij

(xij log

xij

(UV )ij− xij + (UV )ij

),(3.3)

which reduces to the Kullback-Leibler divergence if∑ij xij =

∑ij(UV )ij = 1.

In both Eq. (3.2) and Eq. (3.3), we can find thateach column vi in V only dependents on the columnxi in X, suggesting that we can infer the topics of eachdocument independently. Hence, it is easy to parallelizethe topic inference procedure for large-scale data.

The major difference between our approach and thestandard NMF is that we separate the topic modelingprocess in NMF into two sub-tasks: topic learningand topic inference for documents, which are proceededsequentially. In the topic learning stage, we solvesa smaller and denser matrix factorization problem inEq. (3.1). While in the topic inference stage, we solvesa non-negative least squares problem in Eq. (3.2) orEq. (3.3). Both of the two sub-task is much easierto solve than directly factorizing the extremely sparseand large term-document matrix. For convenience,we denote our approach as “TNMF” in the followingdescription.

4 Learning Algorithm

In this section, we will detail the algorithms to learn theterm-topic matrix U and the topic-document matrix V ,respectively.

Algorithm 1: The overall procedure of our approach

Input: the topic number K, the term-documentmatrix X ∈ RM×N

Output: the term-topic matrix U ∈ RM×K , andthe topic-document matrix V ∈ RK×N

compute term correlation matrix S ∈ RM×M

random initialize Urepeat

U ← max(SU(UTU)−1, 0)

until convergence;V = max((UTU)−1UTY, 0)return U, V

4.1 Solving the Term-Topic Matrix U Differentfrom the standard NMF, the objection function ofsymmetric non-negative matrix factorization in Eq.(3.1)is quartic non-convex function. However, we can accessit via conventional NMF solver by treating U and

UT as two different matrices, and then update themalternatively as follows:

(4.4) Ut+1 = argminU≥0

∥ S − UtUT ∥2,

where Ut is the term-topic matrix solved in tth iteration.When this update converge to a stationary point, wehave Ut+1 = Ut. It is easy to see the stationary point isa solution of Eq.(3.1).

Eq.(4.4) is a non-negative least squares (NNLS)problem. If Ut has full rank, the objective function ofthe NNLS problem is strictly convex, which is guar-anteed to have a unique optimal solution. However,exactly solving NNLS is much slower than solving un-constrained least squares problems, especially in highdimensional space[1]. Instead, we try to find an approxi-mate solution by enforcing the non-negativity constraintby setting all the negative elements resulting from theleast squares solution to 0, namely:

(4.5) Ut+1 = max(SUt(UTt Ut)

−1, 0).

Although lack of convergence theory for this ad-hoc en-forcement, this approximation algorithm often convergequickly and give very accurate results in practice[1].

4.2 Solving the Topic-Document Matrix V Af-ter learning matrix U , we then solve the topic-documentmatrix V . If the Euclidean distance used, minimizingthe objective function (3.2) is also a NNLS problem. Asthe same as solving matrix U , we can obtain the follow-ing update rule:

(4.6) V = max((UTU)−1UTY, 0).

If the generalized I-divergence used, the objective func-tion (3.3) can be write as minimizing N independentsub-problems as follows:

l(vj) =M∑i=1

(xij log

xij∑Kk=1 uikvkj

− xij +K∑

k=1

uikvkj

).

Unfortunately, this problem has no closed form solution.We then apply the coordinate descent method to solveit. For each variable vkj , we fix all other variablesvk′j(k

′ = k) in vector vj and apply Newton’s methodto update vkj . The first order derivative g(vkj) and thesecond order derivative h(vkj) of l on vkj is

g(vkj) =∂l

∂vkj=

M∑i=1

uik

(1− xij∑K

k=1 uikvkj

),

h(vkj) =∂2l

∂v2kj=

M∑i=1

xiju2ik

(∑K

k=1 uikvkj)2i.

100

101

102

103

104

100

101

102

103

104

105

Document frequency

Nu

mb

er

of

term

s

Tweets

Questions

Titles

Figure 4: Distribution of terms in three short text datasets

And the Newton update rule is:

vkj ← max(vkj −g(vkj)

h(vkj), 0).

Algorithm 1 summarizes the overall procedure ofour approach using the Euclidean distance for illustra-tion.

5 Experiments

In this section, we report empirical experiments onreal-world short text data sets to demonstrate theeffectiveness of our approach.

5.1 Data Sets We carried out experiments on threedata sets. 1) Tweet data, a subset of TREC 2011microblog track 1. 2) Title data, including news titleswith class labels from some news websites, which ispublished by Sogou Lab 2. 3) Question data, containingquestions crawled from a popular Chinese question-and-answer website3. Each question is with a manual classlabel assigned by the questioner.

Figure 4 shows the distributions of terms in thethree sets of short text data. We can see that thereare a large proportion of terms occur in less than 10documents in all the three data sets. With furtherinvestigation, we find more than half of those rare termsare meaningless, e.g. “witit”,“25c3”, etc. Therefore,we preprocessed the raw data by removing words withdocument frequency less than 10.

The data sets after preprocessing are summarizedin Table 1. In the two relative small data sets, i.e. theTweet data and the Title data, the number of docu-ments is only about twice of the number of distinctterms. But in the much larger Question data, the num-ber of documents is more than 7 times of distinct terms.

1http://trec.nist.gov/data/tweets/2http://www.sogou.com/labs/dl/tce.html3http://zhidao.baidu.com

Table 1: Statistics of the three data setsData sets Tweet Title Question

#documents 4520 2630 36219

#words 2502 1403 4956

avg words† 8.5958 5.2684 5.8092

#classes unavailable 9 34† denotes the average number of words in a document

5.2 Baselines Our baseline methods include:

• LDA. We used the Gibbs sampling based LDAimplementation GibbsLDA++4.

• NMF. We compared with NMF with two differentcost functions: “NMF E” denotes the NMF withthe Euclidean distance based cost function; And“NMF I” denotes the NMF with the generalized I-divergence. Note we added the ℓ2 norm regularizerfor “NMF E” to avoid over-fitting as [19] did.However, “NMF I” is without any regularization.

• GNMF(graph regularized NMF)[3]. GNMF di-rectly factorizes the term-document matrix, whileemploying a Laplace regularization to keep the lo-cal neighboring relationship. The neighborhood re-lationship of documents is generated by the cosinesimilarity measure. Here, we use the authors’ code5.

• SymNMF(symmetric NMF)[15]. SymNMF factor-izes a symmetric matrix containing pairwise docu-ment similarity values, measured by cosine similar-ity, too.

All the parameters of baseline methods were tuned tobest manually6. In addition, we use “TNMF E” todenote our TNMF with Euclidean distance in learningdocument-term matrix V , and use “TNMF I” to denotethe generalized I-divergence based counterpart.

5.3 Topic Visualization Interpretability of topicmodels is very important. In most applications, weprefer topic models generating topics with good read-ability and compact representation. For this reason, wecompared the hidden topics discovered by all the testmethods qualitatively. Considering the topics learnedby different methods are very different, we only selectsome similar topics among them for comparison.

4http://gibbslda.sourceforge.net5http://www.cad.zju.edu.cn/home/dengcai/Data/GNMF.html6In GNMF, we find the best value of the regularization

coefficient sometimes is 0, which reduces to the standard NMF.In such case, we set it to 0.01 for the purpose of comparison.

Table 2 shows 4 topics from Tweet data (K=20),and each topic represented by its top 5 weighted terms.It is clear that they talk about “Egyptian revolution”,“business”, “weather”, and “Super Bowl”, respectively,which are hot in the Tweet data. Noting that theoriginal SymNMF does not learn the term-topic matrix,hence we did not compare with it. From Table 2 we canconclude that

• LDA is able to discover meaningful topics, but itassigns high weights to some common terms, suchas “medium” and “change”.

• NMF E generated topics more discriminative thanLDA, for the sake of IDF weighting used to improveterm’s discriminative power. However, some topicsstill need effort for explanations, e.g. the “SuperBowl” and “business” topics.

• NMF I failed in finding meaningful topics. Thepossible reason is over-fitting, since NMF I does notemploy any regularization.

• GNMF showed similar results with NMF E, sug-gesting that the document neighborhood regular-ization plays little effect on short texts.

• TNMF E and TNMF I generated much more com-pact topics with the best readability.

5.4 Document Clustering Document clustering isan important technique to automatically group similardocuments in corpus, and widely used in various appli-cations, such as navigation, retrieval, and summariza-tion of huge volumes of text documents. Topic mod-els also have the effect in grouping similar documents,but they allow documents belonging to multiple groups.To utilize topic models for document clustering, we as-signed each document to the highest weighted topic.

5.4.1 Evaluation Metrics Assuming Ω =ω1, ..., ωK is the set of clusters, each ωk is thedocument set in cluster k. And C = c1, ..., cP is theset of P classes of test documents labeled ahead asthe ground truth. We adopt three standard criteria tomeasure the quality of clusters set Ω:

• Purity[26]. Supposing documents in each clustershould take the dominant class in it, purity is theaccuracy of this assignment measured by countingthe number of correctly assigned documents anddividing by the total number of test doucments.Formally:

purity(Ω,C) =1

n

k∑i=1

maxj|ωi ∩ cj |

Table 2: Top words in 4 topics discovered by different methods on the Tweet data.Topic LDA NMF E NMF I GNMF TNMF E TNMF I

egypt egyptian house egypt egyptian egyptianfood egypt egypt state cairo cairo

Egyptian state mubarak update egyptian protester protesterrevolution egyptian cairo state president mubarak egypt

report protester 100 report egypt mubarak

service market market market market marketmedium business red business debt company

business stand social white social credit financialmarket medium hey medium company businessjob company die online business finance

hot wind snow wind humidity humiditywind humidity fall humidity temperature temperature

weather change temperature wind rain mph mphfall rain street temperature hpa hpa

humidity mph humidity mph pressure wind

super super super super packer packerbowl bowl bowl bowl nfl nfl

Super Bowl green xlv green xlv bay bayteam sunday red packer bowl bowlfan party heart sunday rodgers rodgers

Note that when documents in each cluster are withthe same class label, purity is highest with value of1. Conversely, it is close to 0 for bad clustering.

• Normalized Mutual Information(NMI)[20]. NMImeasures the mutual information I(Ω;C) penalizedby the entropies H(Ω) and H(C):

NMI(Ω,C) =I(Ω;C)

[H(Ω) +H(C)]/2

=

∑i,j

|ωi∩cj |n log

|ωi||cj |n|ωi∩cj |

(∑

i|ωi|n log |ωi|

n +∑

j|cj |n log

|cj |n )/2

NMI is 1 for perfect match of Ω and C, and 0 whenthe clusters are random with respect to the classmembership.

• Adjusted Rand Index(ARI)[13]. The Rand In-dex measures the accuracy of pair-wise decisions,i.e. the ratio of pairs of objects which are both lo-cated in the same cluster and the same class, orboth in different clusters and different classes. Ad-justed Rand Index is the corrected-for-chance ver-sion of the Rand Index, which yield a value be-tween [-1, 1]. The higher the Adjusted Rand In-dex, the more resemblance between the clusteringresults and the labels.

ARI =

∑i,j

(|ωi∩cj |2

)− [∑

i

(|ωi|2

)∑j

(|cj |2

)]/(n2

)12 [∑

i

(|ωi|2

)+∑

j

(|cj |2

)]− [

∑i

(|ωi|2

)∑j

(|cj |2

)]/(n2

)

5.4.2 Quantitative Evaluation Quantitative eval-uation is conducted on the two data sets with label in-formation: the Title data and the Question data, withthe number of clusters ranging from 20 to 100. We haverun 10 times for each evaluation, and the average per-formance scores are reported.

Figure 5 and Figure 6 illustrate the comparisonof each method’s clustering performance on the twodata sets. From the results, we can draw the followingconclusions.

• Overall, TNMF E and TNMF I always outperformall the baseline methods in all three evaluationmetrics, especially in terms of NMI and ARI. Inthe Question data, the improvement is much moresignificant than in the Title data. It demonstratesthat topics can be accurately identified from theterm correlation matrix.

• Among the baseline methods, NMF E is slightlyworse than LDA in most cases, but comparablewith or even better than GNMF and SymNMF. Itsuggests that the document similarity informationdo not benefit the clustering performance in theseshort texts.

• Besides, NMF KL performed very bad, and thepossible reason is that the sparse data make it over-fit the data.

• Surprisingly, the results show that SymNMF andTNMF are not sensitive to over-fitting. The pos-

20 40 60 80 1000

0.2

0.4

0.6

0.8

K

Pu

rity

20 40 60 80 1000

0.1

0.2

0.3

0.4

0.5

K

NM

I

20 40 60 80 1000

0.05

0.1

0.15

0.2

0.25

0.3

K

AR

I

LDA

NMF_E

NMF_I

GNMF

SymNMF

TNMF_E

TNMF_I

Figure 5: Comparison of (a)Purity, (b)NMI, (c)ARI w.r.t the cluster number k on the Title data

20 40 60 80 1000

0.2

0.4

0.6

K

Pu

rity

20 40 60 80 1000

0.2

0.4

0.6

K

NM

I

20 40 60 80 1000

0.1

0.2

0.3

0.4

0.5

K

AR

I

LDA

NMF_E

NMF_I

GNMF

TNMF_E

TNMF_I

Figure 6: Comparison of (a)Purity, (b)NMI, (c)ARI w.r.t the cluster number k on Question data

sible reason is that the symmetric matrix factor-ization has only one sub-matrix to be estimated.The freedom of latent variables are much less thanNMF.

5.5 Document Classification To verify how wellthe documents are represented by the learned topics, wefurther tested the classification accuracy of short textsby representing documents in the latent spaces.

The document classification experiments were con-ducted on both the Title data and the Question data,too. In each data set, documents are randomly split intotraining and testing sub-sets with the ratio 4 : 1, thenclassified by the linear support vector machine classifierLIBLINEAR[8].

Figure 7 and Figure 8 show the classification resultsfor all the test methods with topic number K varyingfrom 20 to 100 on the two data sets. These experimentsreveal several interesting points:

• TNMF E substantially outperforms baseline meth-ods on both data sets, especially in the Questiondata which has much more documents. It suggeststhat TNMF E can capture the topics of documentsmore accurately than other methods.

• The performance of TNMF I is not as good asTNMF E in classification, but still better thanthe baseline methods on the Question data, andcomparable with them on the Title data.

20 40 60 80 10040

45

50

55

60

65

70

75

80

85

90

TNMF_ITNMF_ESymNMFGNMF_DNMF_INMF_ELDA

K

accuracy

Figure 7: Classification accuracy on the Title data

• GNMF and SymNMF do not achieve any improve-ment over NMF on the Title data, while even de-cline in terms of accuracy on the Question data.It implies that the document neighborhood rela-tionship might not be accurate for short texts anymore.

6 Conclusions

Learning topics for short texts is considered to be adifficult problem due to the severe sparsity of term-document co-occurrence data. To tackle this problem,we have presented a novel method based on the non-negative matrix factorization framework. Our approachfirst learns topics from term correlation data using sym-metric non-negative matrix factorization, and then in-fers the topic representations of documents by solving anon-negative non-negative least squares problem. Since

20 40 60 80 10030

40

50

60

70

80

90

TNMF_ITNMF_EGNMF_DNMF_INMF_ELDA

K

accuracy

Figure 8: Classification accuracy on the Question data

the term correlation data is less sparse and more stablewith the increase of the collection size, our approach isable to learn better topics than tranditional methods.Experiments on three short text data sets illustrated su-perior performance of our methods compared with theother baseline methods.

References

[1] R. Albright, J. Cox, D. Duling, A. Langville, andC. Meyer. Algorithms, initializations, and convergencefor the nonnegative matrix factorization. Technicalreport, 2006.

[2] D.M. Blei, A.Y. Ng, and M.I. Jordan. Latent dirichletallocation. The Journal of Machine Learning Research,3:993–1022, 2003.

[3] D. Cai, X. He, J. Han, and T.S. Huang. Graph regu-larized nonnegative matrix factorization for data repre-sentation. IEEE Transactions on Pattern Analysis andMachine Intelligence, 33(8):1548–1560, 2011.

[4] Mario Cataldi, Luigi Di Caro, and Claudio Schifanella.Emerging topic detection on twitter based on temporaland social terms evaluation. In Proceedings of the TenthInternational Workshop on Multimedia Data Mining,MDMKDD ’10, pages 4:1–4:10, New York, NY, USA,2010. ACM.

[5] X. Chen, B. Bai, Y. Qi, Q. Lin, and J. Carbonell.Sparse latent semantic analysis. In NIPS Workshop,2010.

[6] S. Deerwester, S.T. Dumais, G.W. Furnas, T.K. Lan-dauer, and R. Harshman. Indexing by latent semanticanalysis. Journal of the American society for informa-tion science, 41(6):391–407, 1990.

[7] C. Ding, X. He, and H.D. Simon. On the equivalence ofnonnegative matrix factorization and spectral cluster-ing. In Proc. SIAM Data Mining Conf, pages 606–610,2005.

[8] R.E. Fan, K.W. Chang, C.J. Hsieh, X.R. Wang, andC.J. Lin. Liblinear: A library for large linear classi-fication. The Journal of Machine Learning Research,9:1871–1874, 2008.

[9] John R. Firth. A Synopsis of Linguistic Theory, 1930-1955. Studies in Linguistic Analysis, pages 1–32, 1957.

[10] E. Gaussier and C. Goutte. Relation between plsa andnmf and implications. In Proceedings of the 28th annualinternational ACM SIGIR conference, pages 601–602.ACM, 2005.

[11] T. Hofmann. Probabilistic latent semantic indexing. InSIGIR, pages 50–57. ACM, 1999.

[12] L. Hong and B.D. Davison. Empirical study of topicmodeling in twitter. In Proceedings of the First Work-shop on Social Media Analytics, pages 80–88. ACM,2010.

[13] L. Hubert and P. Arabie. Comparing partitions. Jour-nal of classification, 2(1):193–218, 1985.

[14] Daniel Jurafsky and James H. Martin. Speech andLanguage Processing: An Introduction to NaturalLanguage Processing, Computational Linguistics andSpeech Recognition. Prentice Hall, second edition,February 2008.

[15] D. Kuang, C. Ding, and H. Park. Symmetric nonnega-tive matrix factorization for graph clustering. In Proc.SIAM Data Mining Conf, 2012.

[16] D.D. Lee, H.S. Seung, et al. Learning the parts ofobjects by non-negative matrix factorization. Nature,401(6755):788–791, 1999.

[17] O. Phelan, K. McCarthy, and B. Smyth. Using twitterto recommend real-time topical news. In Proceedingsof the third ACM conference on Recommender systems,pages 385–388, 2009.

[18] D. Ramage, S. Dumais, and D. Liebling. Character-izing microblogs with topic models. In InternationalAAAI Conference on Weblogs and Social Media. TheAAAI Press, 2010.

[19] F. Shahnaz, M.W. Berry, V.P. Pauca, and R.J. Plem-mons. Document clustering using nonnegative matrixfactorization. Information Processing & Management,42(2):373–386, 2006.

[20] A. Strehl and J. Ghosh. Cluster ensembles—a knowl-edge reuse framework for combining multiple parti-tions. The Journal of Machine Learning Research,3:583–617, 2003.

[21] J. Teevan, D. Ramage, and M.R. Morris. # twit-tersearch: a comparison of microblog search and websearch. In WSDM, pages 35–44. ACM, 2011.

[22] Q. Wang, J. Xu, H. Li, and N. Craswell. Regularizedlatent semantic indexing. In SIGIR, pages 685–694.ACM, 2011.

[23] Jianshu Weng, Ee-Peng Lim, Jing Jiang, and Qi He.Twitterrank: finding topic-sensitive influential twitter-ers. In WSDM, pages 261–270, New York, NY, USA,2010. ACM.

[24] W. Xu, X. Liu, and Y. Gong. Document clusteringbased on non-negative matrix factorization. In SIGIR,pages 267–273. ACM, 2003.

[25] Xiaohui Yan, Jiafeng Guo, Shenghua Liu, Xueqi Cheng,and Yanfeng Wang. Clustering short text using ncut-weighted non-negative matrix factorization. In CIKM,pages 2259–2262, 2012.

[26] Y. Zhao and G. Karypis. Criterion functions fordocument clustering: Experiments and analysis. 2001.