word sense disambiguation ling571 deep processing techniques for nlp february 28, 2011
Post on 19-Dec-2015
215 views
TRANSCRIPT
![Page 1: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/1.jpg)
Word Sense Disambiguation
Ling571Deep Processing Techniques for NLP
February 28, 2011
![Page 2: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/2.jpg)
Word Sense Disambiguation
Robust Approaches Similarity-based approaches
Thesaurus-based techniques Resnik/Lin similarity
Unsupervised, distributional approaches Word-space (Infomap)
Why they work Why they don’t
![Page 3: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/3.jpg)
Resnik’s Similarity Measure
Information content of node: IC(c) = -log P(c)
Least common subsumer (LCS):Lowest node in hierarchy subsuming 2 nodes
Similarity measure:simRESNIK(c1,c2) = - log P(LCS(c1,c2))
Issue:Not content, but difference between node & LCS
![Page 4: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/4.jpg)
Application to WSD Calculate Informativeness
For Each Node in WordNet:
![Page 5: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/5.jpg)
Application to WSD Calculate Informativeness
For Each Node in WordNet:Sum occurrences of concept and all children
Compute IC
![Page 6: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/6.jpg)
Application to WSD Calculate Informativeness
For Each Node in WordNet:Sum occurrences of concept and all children
Compute IC
Disambiguate with WordNet
![Page 7: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/7.jpg)
Application to WSD Calculate Informativeness
For Each Node in WordNet:Sum occurrences of concept and all children
Compute IC
Disambiguate with WordNetAssume set of words in context
E.g. {plants, animals, rainforest, species} from article
![Page 8: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/8.jpg)
Application to WSD Calculate Informativeness
For Each Node in WordNet:Sum occurrences of concept and all children
Compute IC
Disambiguate with WordNetAssume set of words in context
E.g. {plants, animals, rainforest, species} from article Find Most Informative Subsumer for each pair, I
Find LCS for each pair of senses, pick highest similarity
![Page 9: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/9.jpg)
Application to WSD Calculate Informativeness
For Each Node in WordNet:Sum occurrences of concept and all children
Compute IC
Disambiguate with WordNetAssume set of words in context
E.g. {plants, animals, rainforest, species} from article Find Most Informative Subsumer for each pair, I
Find LCS for each pair of senses, pick highest similarity
For each subsumed sense, Vote += I
![Page 10: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/10.jpg)
Application to WSD Calculate Informativeness
For Each Node in WordNet:Sum occurrences of concept and all children
Compute IC
Disambiguate with WordNetAssume set of words in context
E.g. {plants, animals, rainforest, species} from article Find Most Informative Subsumer for each pair, I
Find LCS for each pair of senses, pick highest similarity
For each subsumed sense, Vote += I Select Sense with Highest Vote
![Page 11: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/11.jpg)
IC Example
![Page 12: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/12.jpg)
There are more kinds of plants and animals in the rainforests than anywhere else on Earth. Over half of the millions of known species of plants and animals live in the rainforest. Many are found nowhere else. There are even plants and animals in therainforest that we have not yet discovered.Biological Example
The Paulus company was founded in 1938. Since those days the product range has been the subject of constant expansions and is brought up continuously to correspond with the state of the art. We’re engineering, manufacturing and commissioning world-wide ready-to-run plants packed with our comprehensive know-how. Our Product Range includes pneumatic conveying systems for carbon, carbide, sand, lime andmany others. We use reagent injection in molten metal for the…Industrial Example
Label the First Use of “Plant”
![Page 13: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/13.jpg)
Sense Labeling Under WordNet
Use Local Content Words as ClustersBiology: Plants, Animals, Rainforests, species… Industry: Company, Products, Range, Systems…
![Page 14: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/14.jpg)
Sense Labeling Under WordNet
Use Local Content Words as ClustersBiology: Plants, Animals, Rainforests, species… Industry: Company, Products, Range, Systems…
Find Common Ancestors in WordNet
![Page 15: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/15.jpg)
Sense Labeling Under WordNet
Use Local Content Words as ClustersBiology: Plants, Animals, Rainforests, species… Industry: Company, Products, Range, Systems…
Find Common Ancestors in WordNetBiology: Plants & Animals isa Living Thing
![Page 16: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/16.jpg)
Sense Labeling Under WordNet
Use Local Content Words as ClustersBiology: Plants, Animals, Rainforests, species… Industry: Company, Products, Range, Systems…
Find Common Ancestors in WordNetBiology: Plants & Animals isa Living Thing Industry: Product & Plant isa Artifact isa Entity
![Page 17: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/17.jpg)
Sense Labeling Under WordNet
Use Local Content Words as ClustersBiology: Plants, Animals, Rainforests, species… Industry: Company, Products, Range, Systems…
Find Common Ancestors in WordNetBiology: Plants & Animals isa Living Thing Industry: Product & Plant isa Artifact isa EntityUse Most Informative
Result: Correct Selection
![Page 18: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/18.jpg)
Thesaurus Similarity Issues
![Page 19: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/19.jpg)
Thesaurus Similarity Issues
Coverage:Few languages have large thesauri
![Page 20: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/20.jpg)
Thesaurus Similarity Issues
Coverage:Few languages have large thesauriFew languages have large sense tagged corpora
![Page 21: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/21.jpg)
Thesaurus Similarity Issues
Coverage:Few languages have large thesauriFew languages have large sense tagged corpora
Thesaurus design:Works well for noun IS-A hierarchy
![Page 22: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/22.jpg)
Thesaurus Similarity Issues
Coverage:Few languages have large thesauriFew languages have large sense tagged corpora
Thesaurus design:Works well for noun IS-A hierarchyVerb hierarchy shallow, bushy, less informative
![Page 23: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/23.jpg)
Distributional Similarity Unsupervised approach:
Clustering, WSD, automatic thesaurus enrichment
![Page 24: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/24.jpg)
Distributional Similarity Unsupervised approach:
Clustering, WSD, automatic thesaurus enrichment
Insight:“You shall know a word by the company it keeps!”
(Firth, 1957)
![Page 25: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/25.jpg)
Distributional Similarity Unsupervised approach:
Clustering, WSD, automatic thesaurus enrichment
Insight:“You shall know a word by the company it keeps!”
(Firth, 1957)A bottle of tezguino is on the table.Everybody likes tezguino.Tezguino makes you drunk.We make tezguino from corn.
![Page 26: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/26.jpg)
Distributional Similarity Unsupervised approach:
Clustering, WSD, automatic thesaurus enrichment
Insight:“You shall know a word by the company it keeps!”
(Firth, 1957)A bottle of tezguino is on the table.Everybody likes tezguino.Tezguino makes you drunk.We make tezguino from corn.
Tezguino: corn-based, alcoholic beverage
![Page 27: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/27.jpg)
Distributional SimilarityRepresent ‘company’ of word such that similar
words will have similar representations ‘Company’ = context
![Page 28: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/28.jpg)
Distributional SimilarityRepresent ‘company’ of word such that similar
words will have similar representations ‘Company’ = context
Word represented by context feature vectorMany alternatives for vector
![Page 29: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/29.jpg)
Distributional SimilarityRepresent ‘company’ of word such that similar
words will have similar representations ‘Company’ = context
Word represented by context feature vectorMany alternatives for vector
Initial representation: ‘Bag of words’ binary feature vector
![Page 30: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/30.jpg)
Distributional SimilarityRepresent ‘company’ of word such that similar
words will have similar representations ‘Company’ = context
Word represented by context feature vectorMany alternatives for vector
Initial representation: ‘Bag of words’ binary feature vectorFeature vector length N, where N is of vocabulary
fi= 1 if wordi within window of w, 0 o.w.
![Page 31: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/31.jpg)
Binary Feature Vector
![Page 32: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/32.jpg)
Distributional Similarity Questions
![Page 33: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/33.jpg)
Distributional Similarity Questions
What is the right neighborhood?What is the context?
![Page 34: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/34.jpg)
Distributional Similarity Questions
What is the right neighborhood?What is the context?
How should we weight the features?
![Page 35: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/35.jpg)
Distributional Similarity Questions
What is the right neighborhood?What is the context?
How should we weight the features?
How can we compute similarity between vectors?
![Page 36: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/36.jpg)
Feature Vector DesignWindow size:
How many words in the neighborhood?Tradeoff:
![Page 37: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/37.jpg)
Feature Vector DesignWindow size:
How many words in the neighborhood?Tradeoff:
+/- 500 words
![Page 38: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/38.jpg)
Feature Vector DesignWindow size:
How many words in the neighborhood?Tradeoff:
+/- 500 words: ‘topical context’
![Page 39: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/39.jpg)
Feature Vector DesignWindow size:
How many words in the neighborhood?Tradeoff:
+/- 500 words: ‘topical context’
+/- 1 or 2 words:
![Page 40: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/40.jpg)
Feature Vector DesignWindow size:
How many words in the neighborhood?Tradeoff:
+/- 500 words: ‘topical context’
+/- 1 or 2 words: collocations, predicate-argument
![Page 41: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/41.jpg)
Feature Vector DesignWindow size:
How many words in the neighborhood?Tradeoff:
+/- 500 words: ‘topical context’
+/- 1 or 2 words: collocations, predicate-argument
Only words in some grammatical relationParse text (dependency) Include subj-verb; verb-obj; adj-mod
NxR vector: word x relation
![Page 42: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/42.jpg)
Example Lin Relation Vector
![Page 43: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/43.jpg)
Weighting FeaturesBaseline: Binary (0/1)
![Page 44: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/44.jpg)
Weighting FeaturesBaseline: Binary (0/1)
Minimally informativeCan’t capture intuition that frequent features
informative
![Page 45: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/45.jpg)
Weighting FeaturesBaseline: Binary (0/1)
Minimally informativeCan’t capture intuition that frequent features
informative
Frequency or Probability:
![Page 46: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/46.jpg)
Weighting FeaturesBaseline: Binary (0/1)
Minimally informativeCan’t capture intuition that frequent features
informative
Frequency or Probability:
Better but,
![Page 47: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/47.jpg)
Weighting FeaturesBaseline: Binary (0/1)
Minimally informativeCan’t capture intuition that frequent features
informative
Frequency or Probability:
Better but,Can overweight a priori frequent features
Chance cooccurrence
![Page 48: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/48.jpg)
Pointwise Mutual Information
PMI: - Contrasts observed cooccurrence - With that expected by chance (if independent)
![Page 49: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/49.jpg)
Pointwise Mutual Information
PMI: - Contrasts observed cooccurrence - With that expected by chance (if independent)- Generally only use positive values - Negatives inaccurate unless corpus huge
![Page 50: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/50.jpg)
Vector SimilarityEuclidean or Manhattan distances:
![Page 51: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/51.jpg)
Vector SimilarityEuclidean or Manhattan distances:
Too sensitive to extreme values
![Page 52: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/52.jpg)
Vector SimilarityEuclidean or Manhattan distances:
Too sensitive to extreme values
Dot product:
![Page 53: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/53.jpg)
Vector SimilarityEuclidean or Manhattan distances:
Too sensitive to extreme values
Dot product:Favors long vectors:
More features or higher values
![Page 54: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/54.jpg)
Vector SimilarityEuclidean or Manhattan distances:
Too sensitive to extreme values
Dot product:Favors long vectors:
More features or higher values
Cosine:
![Page 55: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/55.jpg)
Schutze’s Word SpaceBuild a co-occurrence matrix
![Page 56: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/56.jpg)
Schutze’s Word SpaceBuild a co-occurrence matrix
Restrict Vocabulary to 4 letter sequences
![Page 57: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/57.jpg)
Schutze’s Word SpaceBuild a co-occurrence matrix
Restrict Vocabulary to 4 letter sequencesSimilar effect to stemmingExclude Very Frequent - Articles, Affixes
![Page 58: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/58.jpg)
Schutze’s Word SpaceBuild a co-occurrence matrix
Restrict Vocabulary to 4 letter sequencesSimilar effect to stemmingExclude Very Frequent - Articles, Affixes
Entries in 5000-5000 MatrixApply Singular Value Decomposition (SVD)Reduce to 97 dimensions
![Page 59: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/59.jpg)
Schutze’s Word SpaceBuild a co-occurrence matrix
Restrict Vocabulary to 4 letter sequencesSimilar effect to stemmingExclude Very Frequent - Articles, Affixes
Entries in 5000-5000 MatrixApply Singular Value Decomposition (SVD)Reduce to 97 dimensions
Word Context4grams within 1001 CharactersSum & Normalize Vectors for each 4gramDistances between Vectors by dot product
![Page 60: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/60.jpg)
Schutze’s Word SpaceWord Sense Disambiguation
![Page 61: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/61.jpg)
Schutze’s Word SpaceWord Sense Disambiguation
Context Vectors of All Instances of Word
![Page 62: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/62.jpg)
Schutze’s Word SpaceWord Sense Disambiguation
Context Vectors of All Instances of Word
Automatically Cluster Context Vectors
![Page 63: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/63.jpg)
Schutze’s Word SpaceWord Sense Disambiguation
Context Vectors of All Instances of Word
Automatically Cluster Context Vectors
Hand-label Clusters with Sense Tag
![Page 64: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/64.jpg)
Schutze’s Word SpaceWord Sense Disambiguation
Context Vectors of All Instances of Word
Automatically Cluster Context Vectors
Hand-label Clusters with Sense Tag
Tag New Instance with Nearest Cluster
![Page 65: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/65.jpg)
There are more kinds of plants and animals in the rainforests than anywhere else on Earth. Over half of the millions of known species of plants and animals live in the rainforest. Many are found nowhere else. There are even plants and animals in therainforest that we have not yet discovered.Biological Example
The Paulus company was founded in 1938. Since those days the product range has been the subject of constant expansions and is brought up continuously to correspond with the state of the art. We’re engineering, manufacturing and commissioning world-wide ready-to-run plants packed with our comprehensive know-how. Our Product Range includes pneumatic conveying systems for carbon, carbide, sand, lime andmany others. We use reagent injection in molten metal for the…Industrial Example
Label the First Use of “Plant”
![Page 66: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/66.jpg)
Sense Selection in “Word Space”
Build a Context Vector
![Page 67: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/67.jpg)
Sense Selection in “Word Space”
Build a Context Vector1,001 character window - Whole Article
![Page 68: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/68.jpg)
Sense Selection in “Word Space”
Build a Context Vector1,001 character window - Whole Article
Compare Vector Distances to Sense Clusters
![Page 69: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/69.jpg)
Sense Selection in “Word Space”
Build a Context Vector1,001 character window - Whole Article
Compare Vector Distances to Sense ClustersOnly 3 Content Words in CommonDistant Context VectorsClusters - Build Automatically, Label Manually
![Page 70: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/70.jpg)
Sense Selection in “Word Space”
Build a Context Vector1,001 character window - Whole Article
Compare Vector Distances to Sense ClustersOnly 3 Content Words in CommonDistant Context VectorsClusters - Build Automatically, Label Manually
Result: 2 Different, Correct Senses92% on Pair-wise tasks
![Page 71: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/71.jpg)
Odd Cluster ExamplesThe “Ste.” Cluster:
Dry Oyster Whisky Hot Float Ice
![Page 72: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/72.jpg)
Odd Cluster ExamplesThe “Ste.” Cluster:
Dry Oyster Whisky Hot Float IceWhy? – River name
![Page 73: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/73.jpg)
Odd Cluster ExamplesThe “Ste.” Cluster:
Dry Oyster Whisky Hot Float IceWhy? – River name
Learning the Corpus, not the Sense
Keeping cluster: Bring Hoping Wiping Could Should Some Them
Rest
![Page 74: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/74.jpg)
Odd Cluster ExamplesThe “Ste.” Cluster:
Dry Oyster Whisky Hot Float IceWhy? – River name
Learning the Corpus, not the Sense
Keeping cluster: Bring Hoping Wiping Could Should Some Them
RestUninformative: Wide context misses verb sense
![Page 75: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/75.jpg)
The Question of ContextShared Intuition:
Context -> Sense
Area of Disagreement:What is context?
Wide vs Narrow Window
Word Co-occurrences
![Page 76: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/76.jpg)
Taxonomy of Contextual Information
Topical Content
Word Associations
Syntactic Constraints
Selectional Preferences
World Knowledge & Inference
![Page 77: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/77.jpg)
Limits of Wide ContextComparison of Wide-Context Techniques (LTV ‘93)
Neural Net, Context Vector, Bayesian Classifier, Simulated Annealing
![Page 78: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/78.jpg)
Limits of Wide ContextComparison of Wide-Context Techniques (LTV ‘93)
Neural Net, Context Vector, Bayesian Classifier, Simulated AnnealingResults: 2 Senses - 90+%; 3+ senses ~ 70%
![Page 79: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/79.jpg)
Limits of Wide ContextComparison of Wide-Context Techniques (LTV ‘93)
Neural Net, Context Vector, Bayesian Classifier, Simulated AnnealingResults: 2 Senses - 90+%; 3+ senses ~ 70%
Nouns: 92%; Verbs: 69%
![Page 80: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/80.jpg)
Limits of Wide ContextComparison of Wide-Context Techniques (LTV ‘93)
Neural Net, Context Vector, Bayesian Classifier, Simulated AnnealingResults: 2 Senses - 90+%; 3+ senses ~ 70%
Nouns: 92%; Verbs: 69%
People: Sentences ~100%; Bag of Words: ~70%
![Page 81: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/81.jpg)
Limits of Wide ContextComparison of Wide-Context Techniques (LTV ‘93)
Neural Net, Context Vector, Bayesian Classifier, Simulated AnnealingResults: 2 Senses - 90+%; 3+ senses ~ 70%
Nouns: 92%; Verbs: 69%
People: Sentences ~100%; Bag of Words: ~70%
Inadequate Context
![Page 82: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/82.jpg)
Limits of Wide ContextComparison of Wide-Context Techniques (LTV ‘93)
Neural Net, Context Vector, Bayesian Classifier, Simulated AnnealingResults: 2 Senses - 90+%; 3+ senses ~ 70%
Nouns: 92%; Verbs: 69%
People: Sentences ~100%; Bag of Words: ~70%
Inadequate Context
Need Narrow ContextLocal Constraints OverrideRetain Order, Adjacency
![Page 83: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/83.jpg)
Interactions Below the Surface
Constraints Not All Created Equal “The Astronomer Married the Star”
![Page 84: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/84.jpg)
Interactions Below the Surface
Constraints Not All Created Equal “The Astronomer Married the Star”Selectional Restrictions Override Topic
![Page 85: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/85.jpg)
Interactions Below the Surface
Constraints Not All Created Equal “The Astronomer Married the Star”Selectional Restrictions Override Topic
No Surface Regularities “The emigration/immigration bill guaranteed
passports to all Soviet citizens
![Page 86: Word Sense Disambiguation Ling571 Deep Processing Techniques for NLP February 28, 2011](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d2f5503460f94a0668a/html5/thumbnails/86.jpg)
Interactions Below the Surface
Constraints Not All Created Equal “The Astronomer Married the Star”Selectional Restrictions Override Topic
No Surface Regularities “The emigration/immigration bill guaranteed
passports to all Soviet citizensNo Substitute for Understanding