i256: applied natural language processing

1

I256: Applied Natural Language Processing

Marti HearstNov 6, 2006

2

Today

Text ClusteringLatent Semantic Indexing (LSA)

3

Text Clustering

Finds overall similarities among groups of documentsFinds overall similarities among groups of tokensPicks out some themes, ignores others

4

Text ClusteringClustering is

“The art of finding groups in data.” -- Kaufmann and Rousseeu

Term 1

Term 2

5

Text Clustering

Term 1

Term 2

Clustering is“The art of finding groups in data.” -- Kaufmann and Rousseeu

6Slide by Vasileios Hatzivassiloglou

Clustering Applications

Find semantically related words by combining similarity evidence from multiple indicatorsTry to find overall trends or patterns in text collections

7Slide by Vasileios Hatzivassiloglou

“Training” in Clustering

Clustering is an unsupervised learning methodFor each data set, a totally fresh solution is constructedTherefore, there is no trainingHowever, we often use some data for which we have additional information on how it should be partitioned to evaluate the performance of the clustering method

8

Pair-wise Document Similarity

nova galaxy heat h’wood film role diet fur 1 3 1

5 2 2 1 5 4 1

ABCD

How to compute document similarity?

9

Pair-wise Document Similarity(no normalization for simplicity)

nova galaxy heat h’wood film role diet fur 1 3 1

5 2 2 1 5 4 1

ABCD

t

iii

t

t

wwDDsim

wwwD

wwwD

12121

2,22212

1,12111

),(

...,,

...,,

9)11()42(),(0),(0),(0),(0),(

11)32()51(),(

DCsimDBsimCBsimDAsimCAsimBAsim

10

Pair-wise Document Similarity(cosine normalization)

normalized cosine )()(

),(

edunnormaliz ),(

...,,

...,,

1

22

1

21

121

21

12121

2,22212

1,12111

t

ii

t

ii

t

iii

t

iii

t

t

ww

wwDDsim

wwDDsim

wwwD

wwwD

11

Document/Document Matrix

....

.....

................

21

2212

1121

21

nnn

t

t

t

ddD

ddDddDDDD

jiij DDd to of similarity

12Slide by Vasileios Hatzivassiloglou

Hierarchical clustering methods

Agglomerative or bottom-up:Start with each sample in its own clusterMerge the two closest clustersRepeat until one cluster is left

Divisive or top-down:Start with all elements in one clusterPartition one of the current clusters in twoRepeat until all samples are in singleton clusters

13

Agglomerative Clustering

A B C D E F G HI

14

Agglomerative Clustering

A B C D E F G HI

15

AgglomerativeClustering

A B C D E F G HI

16Slide by Vasileios Hatzivassiloglou

Merging NodesEach node is a combination of the documents combined below itWe represent the merged nodes as a vector of term weightsThis vector is referred to as the cluster centroid

17Slide by Vasileios Hatzivassiloglou

Merging criteriaWe need to extend the distance measure from samples to sets of samplesThe complete linkage method

The single linkage method

The average linkage method

),(max),(,

yxdBAdByAx

),(min),(,

yxdBAdByAx

ByAx

yxdBA

BAd,

),(1),(

18

Single-link merging criteria

Merge closest pair of clusters:Single-link: clusters are close if any of their points are

dist(A,B) = min dist(a,b) for aA, bB

each word type isa single-point cluster

merge

19

Fast, but tend to get long, stringy, meandering clusters ...Bottom-Up Clustering – Single-Link

20

Bottom-Up Clustering – Complete-Link

Again, merge closest pair of clusters:Complete-link: clusters are close only if all of their points are dist(A,B) = max dist(a,b) for aA, bB

distancebetweenclusters

21

Bottom-Up Clustering – Complete-Link

distancebetweenclusters

Slow to find closest pair – need quadratically many distances

22

K-Means Clustering1 Decide on a pair-wise similarity measure2 Find K centers using agglomerative clustering

take a small sample group bottom up until K groups found

3 Assign each document to nearest center, forming new clusters4 Repeat 3 as necessary

23Slide by Vasileios Hatzivassiloglou

k-Medians

Similar to k-means but instead of calculating the means across features, it selects as ci the sample in cluster Ci that minimizes (the median)Advantages

Does not require feature vectorsDistance between samples is always availableStatistics with medians are more robust than statistics with means

iCx

icxd ),(

24Slide by Vasileios Hatzivassiloglou

Choosing k

In both hierarchical and k-means/medians, we need to be told where to stop, i.e., how many clusters to formThis is partially alleviated by visual inspection of the hierarchical tree (the dendrogram)It would be nice if we could find an optimal k from the dataWe can do this by trying different values of k and seeing which produces the best separation among the resulting clusters.

25

Scatter/Gather: Clustering a Large Text Collection

Cutting, Pedersen, Tukey & Karger 92, 93Hearst & Pedersen 95

Cluster sets of documents into general “themes”, like a table of contents

Display the contents of the clusters by showing topical terms and typical titles

User chooses subsets of the clusters and re-clusters the documents within

Resulting new groups have different “themes”

26

S/G Example: query on “star”Encyclopedia text

14 sports 8 symbols 47 film, tv 68 film, tv (p) 7 music97 astrophysics 67 astronomy(p) 12 steller phenomena 10 flora/fauna 49 galaxies, stars

29 constellations 7 miscelleneous

Clustering and re-clustering is entirely automated

27

28

29

30

Clustering Retrieval Results

Tends to place similar docs togetherSo can be used as a step in relevance rankingBut not great for showing to users

Exception: good for showing what to throw out!

31

Another use of clustering

Use clustering to map the entire huge multidimensional document space into a huge number of small clusters.“Project” these onto a 2D graphical representationLooks neat, but doesn’t work well as an information retrieval interface.

32

Clustering Multi-Dimensional Document Space(image from Wise et al 95)

33

How to evaluate clusters?

In practice, it’s hard to doDifferent algorithms’ results look good and bad in different waysIt’s difficult to distinguish their outcomes

In theory, define an evaluation functionTypically choose something easy to measure (e.g., the sum of the average distance in each class)

34Slide by Inderjit S. Dhillon

Two Types of Document Clustering

Grouping together of “similar” objectsHard Clustering -- Each object belongs to a single cluster

Soft Clustering -- Each object is probabilistically assigned to clusters

35Slide by Vasileios Hatzivassiloglou

Soft clustering

A variation of many clustering methodsInstead of assigning each data sample to one and only one cluster, it calculates probabilities of membership for all clustersSo, a sample might belong to cluster A with probability 0.4 and to cluster B with probability 0.6

36Slide by Vasileios Hatzivassiloglou

Application: Clustering of adjectivesCluster adjectives based on the nouns they modifyMultiple syntactic clues for modificationThe similarity measure is Kendall’s τ, a robust measure of similarityClustering is done via a hill-climbing method that minimizes the combined average dissimilarity

Predicting the semantic orientation of adjectives,

V Hatzivassiloglou, KR McKeown, EACL 1997

37Slide by Vasileios Hatzivassiloglou

Clustering of nouns

Work by Pereira, Tishby, and LeeDissimilarity is KL divergenceAsymmetric relationship: nouns are clustered, verbs which have the nouns as objects serve as indicatorsSoft, hierarchical clustering

38

Distributional Clustering of English Words - Pereira, Tishby and Lee, ACL 93

39

Distributional Clustering of English Words - Pereira, Tishby and Lee, ACL 93

40Slide by Kostas Kleisouris

Latent Semantic AnalysisMathematical/statistical technique for extracting and representing the similarity of meaning of wordsRepresents word and passage meaning as high-dimensional vectors in the semantic spaceUses Singular Value Decomposition (SVD) to simulate human learning of word and passage meaningIts success depends on:

Sufficient scale and sampling of the data it is givenChoosing the right number of dimensions to extract

41Slide by Schone, Jurafsky, and Stenchikova

LSA CharacteristicsWhy is reducing dimensionality beneficial?

Some words with similar occurrence patterns are projected onto the same dimensionClosely mimics human judgments of meaning similarity

42Slide by Kostas Kleisouris

Sample Applications of LSAEssay Grading

LSA is trained on a large sample of text from the same domain as the topic of the essayEach essay is compared to a large set of essays scored by experts and a subset of the most similar identified by LSAThe target essay is assigned a score consisting of a weighted combination of the scores for the comparison essays

43Slide by Kostas Kleisouris

Sample Applications of LSA Prediction of differences in comprehensibility of texts

By using conceptual similarity measures between successive sentencesLSA has predicted comprehension test results with students

Evaluate and give advice to students as they write and revise summaries of texts they have readAssess psychiatric status

By representing the semantic content of answers to psychiatric interview questions

44Slide by Kostas Kleisouris

Sample Applications of LSAImproving Information Retrieval

Use LSA to match users’ queries with documents that have the desired conceptual meaning Not used in practice – doesn’t help much when you have large corpora to match against, but maybe helpful for a few difficult queries and for term expansion

45Slide by Kostas Kleisouris

LSA intuitionsImplements the idea that the meaning of a passage is the sum of the meanings of its words: meaning of word1 + meaning of word2 + … + meaning of wordn = meaning of passage

This “bag of words” function shows that a passage is considered to be an unordered set of word tokens and the meanings are additive.By creating an equation of this kind for every passage of language that a learner observes, we get a large system of linear equations.

46Slide by Kostas Kleisouris

LSA IntuitionsHowever

Too few equations to specify the values of the variablesDifferent values for the same variable (natural since meanings are vague or multiple)

Instead of finding absolute values for the meanings, they are represented in a richer form (vectors)

Use of SVD (reduces the linear system into multidimensional vectors)

47Slide by Jason Eisner

Latent Semantic AnalysisA trick from Information Retrieval

Each document in corpus is a length-k vector– Or each paragraph, or whatever

(0, 3, 3, 1, 0, 7, . . . 1, 0) aardvark

abacusabbot

abductabove

zygotezymurgy

abandoned

a single document

48Slide by Jason Eisner

Latent Semantic AnalysisA trick from Information Retrieval

Each document in corpus is a length-k vectorPlot all documents in corpus

True plot in k dimensionsReduced-dimensionality plot

49Slide by Jason Eisner

Latent Semantic AnalysisReduced plot is a perspective drawing of true plotIt projects true plot onto a few axes a best choice of axes – shows most variation in the data.

Found by linear algebra: “Singular Value Decomposition” (SVD)

True plot in k dimensionsReduced-dimensionality plot

50Slide by Jason Eisner

Latent Semantic AnalysisSVD plot allows best possible reconstruction of true plot

(i.e., can recover 3-D coordinates with minimal distortion)Ignores variation in the axes that it didn’t pick outHope that variation’s just noise and we want to ignore it

True plot in k dimensionsReduced-dimensionality plot

word 1

word

2

word 3

theme Athem

e B

theme A

them

e B

51Slide by Jason Eisner

Latent Semantic AnalysisSVD finds a small number of theme vectorsApproximates each doc as linear combination of themesCoordinates in reduced plot = linear coefficients

How much of theme A in this document? How much of theme B?Each theme is a collection of words that tend to appear together

True plot in k dimensionsReduced-dimensionality plot

theme Athem

e B

theme A

them

e B

52Slide by Jason Eisner

Latent Semantic AnalysisAnother perspective (similar to neural networks):

documents1 2 3 4 5 6 7

1 2 3 4 5 6 7 8 9terms

matrix of strengths(how strong is each

term in each document?)

Each connection has aweight given by the matrix.

53Slide by Jason Eisner

Latent Semantic AnalysisWhich documents is term 5 strong in?

docs 2, 5, 6 light up strongest.

documents1 2 3 4 5 6 7

1 2 3 4 5 6 7 8 9terms

54Slide by Jason Eisner

This answers a query consisting of terms 5 and 8!

really just matrix multiplication:term vector (query) x strength matrix = doc

vector .

Latent Semantic AnalysisWhich documents are terms 5 and 8 strong in?

documents1 2 3 4 5 6 7

1 2 3 4 5 6 7 8 9terms

55Slide by Jason Eisner

Latent Semantic AnalysisConversely, what terms are strong in document 5?

gives doc 5’s coordinates!

documents1 2 3 4 5 6 7

1 2 3 4 5 6 7 8 9terms

56Slide by Jason Eisner

Latent Semantic AnalysisSVD approximates by smaller 3-layer network

Forces sparse data through a bottleneck, smoothing it

documents1 2 3 4 5 6 7

1 2 3 4 5 6 7 8 9terms

documents1 2 3 4 5 6 7

1 2 3 4 5 6 7 8 9terms

themes

57Slide by Jason Eisner

Latent Semantic AnalysisI.e., smooth sparse data by matrix approx: M A B

A encodes camera angle, B gives each doc’s new coords

documents1 2 3 4 5 6 7

1 2 3 4 5 6 7 8 9terms

matrixM

A

B

documents1 2 3 4 5 6 7

1 2 3 4 5 6 7 8 9terms

themes

58Slide by Kostas Kleisouris

How LSA worksTakes as input a corpus of natural languageThe corpus is parsed into meaningful passages (such as paragraphs)A matrix is formed with passages as rows and words as columns. Cells contain the number of times that a given word is used in a given passageThe cell values are transformed into a measure of the information about the passage identity the carrySVD is applied to represent the words and passages as vectors in a high dimensional semantic space

59Slide by Schone, Jurafsky, and Stenchikova

Represent text as a matrix

d1 d2 d3 d4 d5 d6cosmonaut 1 0 1 0 0 0astronaut 0 1 0 0 0 0moon 1 1 0 0 0 0car 1 0 0 1 1 0truck 0 0 0 1 0 1

documents

word

s

A[i,j] = number of of occurrence of a word i in document j

60Slide by Schone, Jurafsky, and Stenchikova

SVD {A}={T}{S}{D}’

An

m

n

m Min(n,m)n m

= x xTS D’

Reduce dimensionality to k and compute A1:k

A1n

m

n

kk

kk

m

= x xT1S D’

A1 is the best least square approximation of A by a matrix in rank k

61Slide by Schone, Jurafsky, and Stenchikova

Matrix T

dim1 dim2 dim3 dim4 dim5

cosmonaut -0.44 -0.30 0.57 0.58 0.25

astronaut -0.13 -0.33 -0.59 0.00 0.73

moon -0.48 -0.51 -0.37 0.00 -0.61

car -0.70 0.35 0.15 -0.58 0.16

truck -0.26 0.65 -0.41 0.58 -0.09

dimensions

word

s

T- term matrix. Rows of T correspond toRows of original matrix A

Dim2 directly reflects the different co-occurrence patterns

62Slide by Schone, Jurafsky, and Stenchikova

Matrix D’

d1 d2 d3 d4 d5 d6

Dimension1 -0.75 -0.28 -0.20 -0.45 -0.33 -0.12

Dimension2 -0.29 -053 -0.19 0.63 0.22 0.41

Dimension3 0.28 -0.75 0.45 -0.20 0.12 -0.33

Dimension4 -0.00 0.00 0.58 0.00 -0.58 0.58

Dimension5 -0.53 0.29 0.63 0.19 0.41 -0.22

documents

D- document matrix. Columns of D’ (rows of D) correspond to rows of original matrix A

Dim2 directly reflects the different co-occurrence patterns

dim

ensio

ns

63Slide by Schone, Jurafsky, and Stenchikova

Reevaluating document similarities

B = S1 x D1Matrix B is a dimensionality reduction of the original matrix A Compute document correlation B’*B

d1 d2 d3 d4 d5 d6

d1 1

d2 0.78 1

d3 0.40 0.88 1

d4 0.47 -0.18 -0.62 1

d5 0.74 0.16 -0.32 0.94 1

d6 0.10 -0.54 -0.87 0.93 0.74 1

64Slide by Schone, Jurafsky, and Stenchikova

Unfolding new documentsGiven a new document, how to determine which documents it is similar to?

A = T S D’T’A = T’ T S D’T’ A = S D’

q a new vector in the space of Aq in reduced space = T’ * q

65

Next Time

Several takes on blog analysisSentiment classification