Download - Paper Presentation: A Pendulum Swung Too Far
Paper PresentationA Pendulum Swung Too Far (2011) by
Kenneth Church
Sagar Ahire [133050073]
Roadmap● Introduction● History of NLP● Objections to Empiricism
○ Chomsky○ Minsky○ Pierce
● Reasons for the Problem and Solutions
Roadmap: We Are Here● Introduction● History of NLP● Objections to Empiricism
○ Chomsky○ Minsky○ Pierce
● Reasons for the Problem and Solutions
Introduction● The paper deals with the oscillation between
the predominance of theory-driven approaches vs data-driven approaches in the history of NLP and its reasons.
● Specifically, it predicts a surge in rationalism in the 2010s and explains why and how researchers need to be prepared for it.
Rationalism vs. EmpiricismRationalism1. Emphasizes on theory2. Assumes an “innate
language faculty”3. Aims at discovering the
language of the human mind (linguistic competence)
4. Assigns categories to language units
5. Major advocates: Chomsky, Minsky
Empiricism1. Emphasizes on data2. Assumes all knowledge
gathered only via senses3. Aims at analysing
language as it actually occurs (linguistic performance)
4. Assigns probabilities to language units
5. Major advocates: Shannon, Norvig
Rationalism vs. EmpiricismRationalism1. Emphasizes on theory2. Assumes an “innate
language faculty”3. Aims at discovering the
language of the human mind (linguistic competence)
4. Assigns categories to language units
5. Major advocates: Chomsky, Minsky
Empiricism1. Emphasizes on data2. Assumes all knowledge
gathered only via senses3. Aims at analysing
language as it actually occurs (linguistic performance)
4. Assigns probabilities to language units
5. Major advocates: Shannon, Norvig
Rationalism vs. EmpiricismRationalism1. Emphasizes on theory2. Assumes an “innate
language faculty”3. Aims at discovering the
language of the human mind (linguistic competence)
4. Assigns categories to language units
5. Major advocates: Chomsky, Minsky
Empiricism1. Emphasizes on data2. Assumes all knowledge
gathered only via senses3. Aims at analysing
language as it actually occurs (linguistic performance)
4. Assigns probabilities to language units
5. Major advocates: Shannon, Norvig
Rationalism vs. EmpiricismRationalism1. Emphasizes on theory2. Assumes an “innate
language faculty”3. Aims at discovering the
language of the human mind (linguistic competence)
4. Assigns categories to language units
5. Major advocates: Chomsky, Minsky
Empiricism1. Emphasizes on data2. Assumes all knowledge
gathered only via senses3. Aims at analysing
language as it actually occurs (linguistic performance)
4. Assigns probabilities to language units
5. Major advocates: Shannon, Norvig
Rationalism vs. EmpiricismRationalism1. Emphasizes on theory2. Assumes an “innate
language faculty”3. Aims at discovering the
language of the human mind (linguistic competence)
4. Assigns categories to language units
5. Major advocates: Chomsky, Minsky
Empiricism1. Emphasizes on data2. Assumes all knowledge
gathered only via senses3. Aims at analysing
language as it actually occurs (linguistic performance)
4. Assigns probabilities to language units
5. Major advocates: Shannon, Norvig
Roadmap: We Are Here● Introduction● History of NLP● Objections to Empiricism
○ Chomsky○ Minsky○ Pierce
● Reasons for the Problem and Solutions
History of NLP
1950s: Empiricism● Empiricism dominated across several fields● Words were classified on the basis of their
co-occurrence with other words (“You shall know a word by the company it keeps” - Firth, 1957)
1970s: Rationalism● Several authors such as Chomsky, Minsky,
etc criticized the Empirical approach● Failure of the Empirical approach led to
funding cutbacks (“winters”)○ 1966: Machine Translation Failure○ 1970: The abandonment of connectionism○ 1971-75: Speech Recognition Failure
1990s: Empiricism● Large amounts of data became available● Several specialized problems could be
solved by statistical frameworks, without concentration on the general problems
2010s: Rationalism?● Most of the low-hanging fruit has been
picked up● But the original criticisms of the empirical
approach are still as valid
Roadmap: We Are Here● Introduction● History of NLP● Objections to Empiricism
○ Chomsky○ Minsky○ Pierce
● Reasons for the Problem and Solutions
Objections to Empiricism● Several common empirical frameworks were
opposed by rationalists in the 70s, including:○ Linear Separators (Machine Learning)○ Vector Space Model (Information Retrieval)○ n-grams (Language Modeling)○ HMMs (Speech Recognition)
● Many of these are mere approximations of complex phenomena
Chomsky’s Objections● n-gram Language Modeling● Finite State Methods
Chomsky’s Objections:n-gram Language Modeling● Chomsky showed that n-grams cannot learn
long-distance dependencies (dependencies spanning more than n words)
● For practical purposes ‘n’ needs to be a small value (3 or 5)
● However, such small values fail to capture several interesting facts
Chomsky’s Objections:Finite State Methods● Examples of Finite State Methods include
○ Hidden Markov Models (HMMs)○ Conditional Random Fields (CRFs)
● Finite State Methods can capture dependencies beyond n words
● However, they may require infinite memory to process certain sentences
Chomsky’s Objections:Center Embedded Grammars● A center embedded grammar is of the form:
○ A -> x A y● Chomsky proved that a center embedded
grammar will require infinite memory and thus cannot be handled by finite state methods
● Center embedding is common in English, for example:○ A man that a woman that a child that a bird that I
heard saw knows loves
Minsky’s Objection● Linear Separators
Minsky’s Objections:Perceptrons● Minsky showed that perceptrons (and linear
separators in general) cannot learn functions that are not linearly separable such as XOR.
Minsky’s Objections:Perceptrons● This has implications for several tasks
including:○ Word Sense Disambiguation○ Information Retrieval○ Author Identification○ Sentiment Analysis
● For instance, this is the reason why sentiment analysis ignores loaded terms
Minsky’s Objections:Sentiment Analysis● Loaded terms can be either positive or
negative depending on whom it is addressed to. This is an XOR dependency:
Loaded Term Addressed to us Sentiment
Positive Y Positive
Positive N Negative
Negative Y Negative
Negative N Positive
Pierce’s Objections● Evaluation by Demos● Pattern Matching
Pierce’s Objections:Evaluation by Demos● According to Pierce, evaluation of projects
should be based on scientific principles rather than laboratory demos.
● Projects give good results in laboratory conditions, but have much higher error rates in real-world conditions.
Pierce’s Objections:Pattern Matching● Pierce stated that pattern matching is “artful
deception”, i.e. it is based on heuristics rather than scientific theory.
● Examples:○ The ELIZA effect○ The Chinese Room argument
Pierce’s Objections:Pattern Matching● While pattern matching produces better
results in the short term, it does so only by ignoring real scientific questions.
● While ambitious approaches may require time to deliver, they are backed by hard science.
Roadmap: We Are Here● Introduction● History of NLP● Objections to Empiricism
○ Chomsky○ Minsky○ Pierce
● Reasons for the Problem and Solutions
Reason for the Oscillations:Gaps in Teaching● The “losing” side of the debate (currently
Rationalism) is never mentioned in textbooks/courses
● Leads to “reinventing the wheel” by each generation of NLP researchers
Reason for the Oscillations:Gaps in Teaching● Currently most courses concentrate on
Statistical methods, ignoring linguistic and scientific questions
● This prepares students only for “low-hanging fruit” but not the real scientific questions
Solution● Introduce the following in NLP courses:
○ Syntax○ Morphology○ Phonology○ Phonetics○ Historical Linguistics○ Language Universals
● Create parallels between computational linguistics and formal linguistics
Solution● Teach both sides of the rationalism vs.
empiricism debate● Educate students about the challenges
ahead of the “low-hanging fruit”
Major References● A Pendulum Swung Too Far by Kenneth
Church, 2001
Other References● Papers In Linguistics 1934-1951 by JR Firth, 1957● Syntactic Structures by Noam Chomsky, 1957● Whither Speech Recognition by John Pierce, 1969● ELIZA - A Computer Program for the Study of Natural
Language Communication between Man and Machine by Joseph Weizenbaum, 1966
● Minds, Brains and Programs by John Searle, 1980