paper presentation: a pendulum swung too far
DESCRIPTION
A paper presentation made by me for the paper 'A Pendulum Swung Too Far' by Kenneth Church at IIT Bombay as a part of preparation for the MTech Seminar. Get the paper on which this presentation is based here: http://languagelog.ldc.upenn.edu/myl/ldc/swung-too-far.pdfTRANSCRIPT
![Page 1: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/1.jpg)
Paper PresentationA Pendulum Swung Too Far (2011) by
Kenneth Church
Sagar Ahire [133050073]
![Page 2: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/2.jpg)
Roadmap● Introduction● History of NLP● Objections to Empiricism
○ Chomsky○ Minsky○ Pierce
● Reasons for the Problem and Solutions
![Page 3: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/3.jpg)
Roadmap: We Are Here● Introduction● History of NLP● Objections to Empiricism
○ Chomsky○ Minsky○ Pierce
● Reasons for the Problem and Solutions
![Page 4: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/4.jpg)
Introduction● The paper deals with the oscillation between
the predominance of theory-driven approaches vs data-driven approaches in the history of NLP and its reasons.
● Specifically, it predicts a surge in rationalism in the 2010s and explains why and how researchers need to be prepared for it.
![Page 5: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/5.jpg)
Rationalism vs. EmpiricismRationalism1. Emphasizes on theory2. Assumes an “innate
language faculty”3. Aims at discovering the
language of the human mind (linguistic competence)
4. Assigns categories to language units
5. Major advocates: Chomsky, Minsky
Empiricism1. Emphasizes on data2. Assumes all knowledge
gathered only via senses3. Aims at analysing
language as it actually occurs (linguistic performance)
4. Assigns probabilities to language units
5. Major advocates: Shannon, Norvig
![Page 6: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/6.jpg)
Rationalism vs. EmpiricismRationalism1. Emphasizes on theory2. Assumes an “innate
language faculty”3. Aims at discovering the
language of the human mind (linguistic competence)
4. Assigns categories to language units
5. Major advocates: Chomsky, Minsky
Empiricism1. Emphasizes on data2. Assumes all knowledge
gathered only via senses3. Aims at analysing
language as it actually occurs (linguistic performance)
4. Assigns probabilities to language units
5. Major advocates: Shannon, Norvig
![Page 7: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/7.jpg)
Rationalism vs. EmpiricismRationalism1. Emphasizes on theory2. Assumes an “innate
language faculty”3. Aims at discovering the
language of the human mind (linguistic competence)
4. Assigns categories to language units
5. Major advocates: Chomsky, Minsky
Empiricism1. Emphasizes on data2. Assumes all knowledge
gathered only via senses3. Aims at analysing
language as it actually occurs (linguistic performance)
4. Assigns probabilities to language units
5. Major advocates: Shannon, Norvig
![Page 8: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/8.jpg)
Rationalism vs. EmpiricismRationalism1. Emphasizes on theory2. Assumes an “innate
language faculty”3. Aims at discovering the
language of the human mind (linguistic competence)
4. Assigns categories to language units
5. Major advocates: Chomsky, Minsky
Empiricism1. Emphasizes on data2. Assumes all knowledge
gathered only via senses3. Aims at analysing
language as it actually occurs (linguistic performance)
4. Assigns probabilities to language units
5. Major advocates: Shannon, Norvig
![Page 9: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/9.jpg)
Rationalism vs. EmpiricismRationalism1. Emphasizes on theory2. Assumes an “innate
language faculty”3. Aims at discovering the
language of the human mind (linguistic competence)
4. Assigns categories to language units
5. Major advocates: Chomsky, Minsky
Empiricism1. Emphasizes on data2. Assumes all knowledge
gathered only via senses3. Aims at analysing
language as it actually occurs (linguistic performance)
4. Assigns probabilities to language units
5. Major advocates: Shannon, Norvig
![Page 10: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/10.jpg)
Roadmap: We Are Here● Introduction● History of NLP● Objections to Empiricism
○ Chomsky○ Minsky○ Pierce
● Reasons for the Problem and Solutions
![Page 11: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/11.jpg)
History of NLP
![Page 12: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/12.jpg)
1950s: Empiricism● Empiricism dominated across several fields● Words were classified on the basis of their
co-occurrence with other words (“You shall know a word by the company it keeps” - Firth, 1957)
![Page 13: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/13.jpg)
1970s: Rationalism● Several authors such as Chomsky, Minsky,
etc criticized the Empirical approach● Failure of the Empirical approach led to
funding cutbacks (“winters”)○ 1966: Machine Translation Failure○ 1970: The abandonment of connectionism○ 1971-75: Speech Recognition Failure
![Page 14: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/14.jpg)
1990s: Empiricism● Large amounts of data became available● Several specialized problems could be
solved by statistical frameworks, without concentration on the general problems
![Page 15: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/15.jpg)
2010s: Rationalism?● Most of the low-hanging fruit has been
picked up● But the original criticisms of the empirical
approach are still as valid
![Page 16: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/16.jpg)
Roadmap: We Are Here● Introduction● History of NLP● Objections to Empiricism
○ Chomsky○ Minsky○ Pierce
● Reasons for the Problem and Solutions
![Page 17: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/17.jpg)
Objections to Empiricism● Several common empirical frameworks were
opposed by rationalists in the 70s, including:○ Linear Separators (Machine Learning)○ Vector Space Model (Information Retrieval)○ n-grams (Language Modeling)○ HMMs (Speech Recognition)
● Many of these are mere approximations of complex phenomena
![Page 18: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/18.jpg)
Chomsky’s Objections● n-gram Language Modeling● Finite State Methods
![Page 19: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/19.jpg)
Chomsky’s Objections:n-gram Language Modeling● Chomsky showed that n-grams cannot learn
long-distance dependencies (dependencies spanning more than n words)
● For practical purposes ‘n’ needs to be a small value (3 or 5)
● However, such small values fail to capture several interesting facts
![Page 20: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/20.jpg)
Chomsky’s Objections:Finite State Methods● Examples of Finite State Methods include
○ Hidden Markov Models (HMMs)○ Conditional Random Fields (CRFs)
● Finite State Methods can capture dependencies beyond n words
● However, they may require infinite memory to process certain sentences
![Page 21: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/21.jpg)
Chomsky’s Objections:Center Embedded Grammars● A center embedded grammar is of the form:
○ A -> x A y● Chomsky proved that a center embedded
grammar will require infinite memory and thus cannot be handled by finite state methods
● Center embedding is common in English, for example:○ A man that a woman that a child that a bird that I
heard saw knows loves
![Page 22: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/22.jpg)
Minsky’s Objection● Linear Separators
![Page 23: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/23.jpg)
Minsky’s Objections:Perceptrons● Minsky showed that perceptrons (and linear
separators in general) cannot learn functions that are not linearly separable such as XOR.
![Page 24: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/24.jpg)
Minsky’s Objections:Perceptrons● This has implications for several tasks
including:○ Word Sense Disambiguation○ Information Retrieval○ Author Identification○ Sentiment Analysis
● For instance, this is the reason why sentiment analysis ignores loaded terms
![Page 25: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/25.jpg)
Minsky’s Objections:Sentiment Analysis● Loaded terms can be either positive or
negative depending on whom it is addressed to. This is an XOR dependency:
Loaded Term Addressed to us Sentiment
Positive Y Positive
Positive N Negative
Negative Y Negative
Negative N Positive
![Page 26: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/26.jpg)
Pierce’s Objections● Evaluation by Demos● Pattern Matching
![Page 27: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/27.jpg)
Pierce’s Objections:Evaluation by Demos● According to Pierce, evaluation of projects
should be based on scientific principles rather than laboratory demos.
● Projects give good results in laboratory conditions, but have much higher error rates in real-world conditions.
![Page 28: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/28.jpg)
Pierce’s Objections:Pattern Matching● Pierce stated that pattern matching is “artful
deception”, i.e. it is based on heuristics rather than scientific theory.
● Examples:○ The ELIZA effect○ The Chinese Room argument
![Page 29: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/29.jpg)
Pierce’s Objections:Pattern Matching● While pattern matching produces better
results in the short term, it does so only by ignoring real scientific questions.
● While ambitious approaches may require time to deliver, they are backed by hard science.
![Page 30: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/30.jpg)
Roadmap: We Are Here● Introduction● History of NLP● Objections to Empiricism
○ Chomsky○ Minsky○ Pierce
● Reasons for the Problem and Solutions
![Page 31: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/31.jpg)
Reason for the Oscillations:Gaps in Teaching● The “losing” side of the debate (currently
Rationalism) is never mentioned in textbooks/courses
● Leads to “reinventing the wheel” by each generation of NLP researchers
![Page 32: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/32.jpg)
Reason for the Oscillations:Gaps in Teaching● Currently most courses concentrate on
Statistical methods, ignoring linguistic and scientific questions
● This prepares students only for “low-hanging fruit” but not the real scientific questions
![Page 33: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/33.jpg)
Solution● Introduce the following in NLP courses:
○ Syntax○ Morphology○ Phonology○ Phonetics○ Historical Linguistics○ Language Universals
● Create parallels between computational linguistics and formal linguistics
![Page 34: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/34.jpg)
Solution● Teach both sides of the rationalism vs.
empiricism debate● Educate students about the challenges
ahead of the “low-hanging fruit”
![Page 35: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/35.jpg)
Major References● A Pendulum Swung Too Far by Kenneth
Church, 2001
![Page 36: Paper Presentation: A Pendulum Swung Too Far](https://reader034.vdocuments.us/reader034/viewer/2022052321/5564af80d8b42a98268b4db4/html5/thumbnails/36.jpg)
Other References● Papers In Linguistics 1934-1951 by JR Firth, 1957● Syntactic Structures by Noam Chomsky, 1957● Whither Speech Recognition by John Pierce, 1969● ELIZA - A Computer Program for the Study of Natural
Language Communication between Man and Machine by Joseph Weizenbaum, 1966
● Minds, Brains and Programs by John Searle, 1980