day 8: information extraction and advanced statistical nlp

DAY 8: INFORMATION EXTRACTION ANDADVANCED STATISTICAL NLP

PART II: ADVANCED STATISTICAL NLP

Mark Granroth-Wilding

HELSINGIN YLIOPISTOHELSINGFORS UNIVERSITETUNIVERSITY OF HELSINKI NLP 2019. D8: IE and Advanced Stat NLP Mark Granroth-Wilding 1/44

OUTLINE

Statistical NLP

Advanced Statistical Methods

Closing remarks

OUTLINE

Statistical NLP

Closing remarks

REMINDER: AMBIGUITY OFINTERPRETATION

What is the mean temperature inKumpula?

SELECT day_mean FROM daily_forecast:

WHERE station = ’Helsinki Kumpula’:

AND date = ’2019-05-21’;

SELECT day_mean FROM weekly_forecast

WHERE station = ’Helsinki Kumpula’

AND week = ’w22’;

SELECT MEAN(day_temp)

FROM weather_history

WHERE station = ’Helsinki Kumpula’

AND year = ’2019’;

. . .?• Many forms of ambiguity

• Every level/step of analysis

REMINDER: AMBIGUITY INSYNTAX

telescope

REMINDER: STATISTICAL MODELS

• Statistics over previous analyses can help estimate confidences

• Often use probabilistic models

• Local ambiguity: probabilities/confidences

• Multiple hypotheses about meaning/structure

• Update hypotheses as larger units are combined

STATISTICS IN NLP

• Statistical models hit NLP in 1990s

• Now almost ubiquitous

• Seen already: rule-based systems augmented with statisticsE.g.: PCFG

• Also: derive rules from dataE.g. treebank parser

• So far: carefully defined models for specific sub-tasks

• Derived from linguistic theory

• Some exceptions: word embeddings, topic models

STATISTICS IN NLP

DISAMBIGUATION WITH A PCFG

telescope

1.0 0.7

0.6 0.7

p(t) = 2.07× 10−4

telescope

1.0 1.0 0.7

0.55 0.6 0.7

0.5 1.0

p(t) = 2.96× 10−4

Advantages:

• Linguistic structures (relatively) easy to understand

• Simple statistical model, easy to estimate (treebank)

• Further learning by semi-supervised estimation

• Quite efficient parsing with beam search

Disadvantages:

• Not expressive enough to capture all dependencies

• Not much like syntactic formalisms linguists use

• Statistical model limited• Independence assumptions• Can extend, efficient parsing becomes complicated

Advantages:

• Linguistic structures (relatively) easy to understand

• Simple statistical model, easy to estimate (treebank)

• Further learning by semi-supervised estimation

• Quite efficient parsing with beam search

Disadvantages:

• Not expressive enough to capture all dependencies

• Not much like syntactic formalisms linguists use

• Statistical model limited• Independence assumptions• Can extend, efficient parsing becomes complicated

MORE DATA, LESS LINGUISTICS

• Can use more advanced models• For parsing and other tasks

• More sophisticated modelling/learning/inference

• Use less linguistic motivation and knowledge

• E.g. neural end-to-end NLG architecture

• Potential advantages:• Learn from data instead of specifying by hand

E.g. tools for new languages• Discover new phenomena we didn’t know about• Learn structures too complex to write by hand• Maybe don’t care about linguistics, psycholinguistics

How the brainprocesses language

:just do well on task

• Help with long tail in some cases: e.g. rare words

E.g. tools for new languages• Discover new phenomena we didn’t know about

• Learn structures too complex to write by hand• Maybe don’t care about linguistics, psycholinguistics

DANGERS OF FORGETTINGLINGUISTICS

A little linguistics takes you a long way

• Linguists know a lot about language!

• Easily fall into traps they’ve known about fordecades/centuries

• Or waste valuable insight

• Don’t claim linguistic insight without knowledge of linguistics

• Data helps with some types of long tail, but:• easy to focus too much on common phenomena• get caught out by less common ones – more informative

• Data helps with some types of long tail, but:• easy to focus too much on common phenomena

wordssyntactic structuresword classes. . .

• get caught out by less common ones – more informative

CLASSIFICATION AND REGRESSION

• Many practical NLP tasks• Classification: text classification, sentiment analysis, NER

• Regression: sentiment analysis, other continuous predictions

• Formulate task as one of classification/regression→ apply existing advanced techniques

• SVMs, logistic regression, polynomial regression, neuralnetworks, random forests, . . .

• Often, input features from other NLP components:POS tags, syntactic dependencies, lemmas, word embeddings

• Many practical NLP tasks• Classification: text classification, sentiment analysis, NER• Regression: sentiment analysis, other continuous predictions

SENTIMENT ANALYSIS

• Input: text (article, review, . . . )

• Goal: predict positive/negative sentiment

• E.g. good or bad review of product?

I just saw #CaptainMarveland I really enjoyed it frombeginning to end. I thoughtthe humor was great, andI loved Brie Larson. Surethere were some changesfrom the comic books, butI liked them. The changeswere necessary.

I wanted to like this movie,but it flopped for me... Theyshould have went with a dif-ferent route, instead of start-ing her off as a strong heroalready, just like every hero.Larson’s acting was a bitstale and lacked more per-sonality.

SENTIMENT ANALYSIS

• Standard ML problem

• / labels

• What are input features? Words?

SENTIMENT ANALYSIS

• / labels

SENTIMENT ANALYSIS

• / labels

SENTIMENT ANALYSIS

• / labels

SENTIMENT ANALYSIS

• Apply standard classification (or regression) methods

• NLP-based features

• Word embeddings, POS tags, . . . what else?

• Limitation of BOW model

• Solve with better features? Multi-word features?

• Or better model? Compose word meanings

SENTIMENT ANALYSIS

OUTLINE

Statistical NLP

Closing remarks

ADVANCED STATISTICALMETHODS

• SVMs: another classifier

• Topic modelling: model detail on LDA

• Deep learning: hot topic, example use in NLP

SUPPORT VECTOR MACHINES(SVMs)

• Probably seen before. . . ?

• Commonly used classifier

• Still often best choice for classification tasks

• Well developed, efficient libraries

SUPPORT VECTOR MACHINES(SVMs)

• Probably seen before. . . ?

• Commonly used classifier

• Still often best choice for classification tasks

• Well developed, efficient libraries

• Linearly separable data

• Find hyperplane to separate data with maximum margin

• High dimensional space

Original image: Fabian BurgerHELSINGIN YLIOPISTOHELSINGFORS UNIVERSITETUNIVERSITY OF HELSINKI NLP 2019. D8: IE and Advanced Stat NLP Mark Granroth-Wilding 19/44

• For non-linearly separable data: use kernel trick

• Map space to new dimensions via non-linear kernels

• Find linear separation

• Different kernels: different types of hyperplane

• Capture interactions between (original) dimensions

SVM APPLICATIONS

• Apply to sentiment analysis

• Separate positive examples from negative

• Many dimensions (features):words, word-POSs, embeddings, dependencies, . . .

• With kernels, word features no longer independent

• Overcomes some problems with BOW

SVM APPLICATIONS

• Apply to other classification problems

• E.g. email spam detection: spam vs ham

• Similar features

TOPIC MODELLING

• Topic modelling: LDA

• Example of unsupervised learning in NLP

• Type of clustering of docs

• Based on their words: BOW model

• More detail on modelling now

TOPIC MODELLING

• Latent Dirichlet Allocation (LDA)

• Bayesian modelling: generative probability distribution

• Simplifying assumptions:• Each document generated by simple process• Fixed number of topics in corpus

• Bag of words model• Document covers a few topics• Topic associated (mainly) with small number of words

• Words that often appear in same doc belong to same topic

• Simplifying assumptions:• Each document generated by simple process• Fixed number of topics in corpus

• Bag of words model• Document covers a few topics• Topic associated (mainly) with small number of words

• Simplifying assumptions:• Each document generated by simple process• Fixed number of topics in corpus• Bag of words model

• Document covers a few topics• Topic associated (mainly) with small number of words

• Simplifying assumptions:• Each document generated by simple process• Fixed number of topics in corpus• Bag of words model• Document covers a few topics

• Topic associated (mainly) with small number of words

• Simplifying assumptions:• Each document generated by simple process• Fixed number of topics in corpus• Bag of words model• Document covers a few topics• Topic associated (mainly) with small number of words

Generative process (‘story’):

(1) For each topic t:(1) Choose words associated with t(2) Prob dist over words

(2) For each document d :(1) Choose a distribution over topics

(2) For each word in d :(1) Choose a topic t from d ’s distribution

(2) Choose a word from t’s distribution

• Process assumed by model to underlie data

• Not process followed by inference!

• Only words and documents are observed:topics are latent variables discovered by model

(2) For each document d :(1) Choose a distribution over topics

(2) For each word in d :(1) Choose a topic t from d ’s distribution

(2) For each document d :(1) Choose a distribution over topics(2) For each word in d :

(1) Choose a topic t from d ’s distribution

(1) Choose a topic t from d ’s distribution(2) Choose a word from t’s distribution

REMINDER: PLATE DIAGRAMS

p(t0|Start)

p(t1|t0)

p(t2|t1)

p(t3|t2). . .

p(this|t0)

p(is|t1)

p(a|t2)

p(sentence|t3)

p(t0|Start)

p(t1|t0)

p(t2|t1)

p(t3|t2). . .

p(this|t0)

p(is|t1)

p(a|t2)

p(sentence|t3)

Variables

p(t0|Start)

p(t1|t0)

p(t2|t1)

p(t3|t2). . .

p(this|t0)

p(is|t1)

p(a|t2)

p(sentence|t3)

Variables

Hidden

Observed

p(t0|Start)

p(t1|t0)

p(t2|t1)

p(t3|t2). . .

p(this|t0)

p(is|t1)

p(a|t2)

p(sentence|t3)

Variables

Hidden

Observed

Conditional distributions

ti−1 ti

More compact notation: box around repeated elements

ti−1 ti

Repeat N times:N words

More compact notation: box around repeated elements

(1) For each topic t:

(1) Choose words associatedwith t

(2) Prob dist over words

(2) For each document d :

(1) Choose a distribution overtopics

(2) For each word in d :

(1) Choose a topic t fromd ’s distribution

(2) Choose a word fromt’s distribution

Per-topic distover words

Hyperparameter for βDist over dists

Per-doc distover topics

Hyperparameter for θDist over dists

Topic for each word

Word, drawn from z ’s word dist (βz)HELSINGIN YLIOPISTOHELSINGFORS UNIVERSITETUNIVERSITY OF HELSINKI NLP 2019. D8: IE and Advanced Stat NLP Mark Granroth-Wilding 28/44

TRAINING LDA

• LDA defines prob of words given topics

• Topics are unknown – words are observed

• Can’t estimate p(w |t) from counts, as with HMM POS tagger

• Bayes’ rule: prob of topic given word (+ other topics)

• Rough idea behind training:• See lots of docs (words only)• Try selecting some topics based on words and current guess at

distributions

• Keep iterating – distributions start to look consistent

TRAINING LDA

distributions

TRAINING LDA

distributions

TRAINING LDA

distributions

Initially random

TRAINING LDA

distributions

Initially random

TRAINING LDA

• Over time, discover words that tend to occur together

• Grouped into topics

• E.g. many docs use bank and money→ one topic covers these docs

TRAINING LDA

• Over time, discover words that tend tooccur together

• Key to estimation: α and η specify priorbelief that

• each document talks about few topics• each topic is discussed using few words

• How few depends on α and η

TRAINING LDA

• each document talks about few topics

• each topic is discussed using few words

TRAINING LDA

• How few depends on α and ηD

N Kwd ,n

NEURAL NETWORKS / DEEPLEARNING

• Neural networks: been around a long time

• Recent explosion:

• Clever new training tricks• Lots of data• Faster processors (GPUs)

• Can now train huge networks, many hidden layers→ deep learning

• Loads of applications to NLP

• Couple of examples here

• No details of modelling, training, ML theory, maths, . . . : seeother courses!

• Recent explosion:

• Clever new training tricks• Lots of data• Faster processors (GPUs)

• Recent explosion:• Clever new training tricks

• Lots of data• Faster processors (GPUs)

• Recent explosion:• Clever new training tricks• Lots of data

• Faster processors (GPUs)

• Recent explosion:• Clever new training tricks• Lots of data• Faster processors (GPUs)

SEQUENCE MODELS

• In NLP, lots of sequence processing!

• Words – sentences, POS tags, NER tags, . . .

• Not the full story: syntax, semantics (LR deps)

• Many applications of recurrent neural networks (RNNs)

• Learning problems with traditional RNNs

• Overcome by recent models: LSTMs, GRUs

• But same basic idea

SEQUENCE MODELS

RNN LANGUAGE MODELLING

• One application: language modelling

• Earlier: Markov LMs

, n-grams

• Some problems:• Markov assumption: bad, but practical• Limited n-gram size: data sparsity• Fixed n-gram size: lose LR deps

• Modern RNNs can help! (LSTMs, . . . )

p(w0|START)

p(w1|w0)

p(w2|w1)

• Earlier: Markov LMs

, n-grams

p(w0|START)

p(w1|w0)

p(w2|w1)

• Earlier: Markov LMs, n-grams

p(w0|START)

p(w1|START ,w0)

p(w2|w0,w1)

p(w0|START)

p(w1|START ,w0)

p(w2|w0,w1)

p(w0|START)

p(w1|START ,w0)

p(w2|w0,w1)

h0 h1 h2 h3 . . .

this is a sentence

o0 o1 o2 o3

• Looks rather like HMM

• States: theoretically ‘remember’ distant inputs/states

• Modern RNNs (LSTMs, GRUs) make this work in practice

• Outputs: based on seq so far

h0 h1 h2 h3 . . .

this is a sentence

o0 o1 o2 o3

h0 h1 h2 h3 . . .

this is a sentence

o0 o1 o2 o3

Hidden state(layer/vector)

h0 h1 h2 h3 . . .

this is a sentence

o0 o1 o2 o3

h0 h1 h2 h3 . . .

this is a sentence

o0 o1 o2 o3

h0 h1 h2 h3 . . .

this is a sentence

o0 o1 o2 o3

RNNs FOR LM

h0 h1 h2 h3 . . .

w0 w0 w0 w0

o0 o1 o2 o3

• Inputs: words

• Outputs: prediction of next word

• Softmax output: probability distribution

• Train on sentences

RNNs FOR LM

h0 h1 h2 h3 . . .

w0 w0 w0 w0

o0 o1 o2 o3

• Inputs: words

RNNs FOR LM

h0 h1 h2 h3 . . .

w0 w0 w0 w0

w1 w2 w3 w4

• Inputs: words

RNNs FOR LM

h0 h1 h2 h3 . . .

w0 w0 w0 w0

h′0 h′1 h′2 h′3. . .

w1 w2 w3 w4

‘Deep’ hidden representations

RNNs FOR LM

h0 h1 h2 h3 . . .

w0 w0 w0 w0

h′0 h′1 h′2 h′3. . .

w1 w2 w3 w4

Stack hidden layers(arbitrarily many)

RNNs FOR LM

h0 h1 h2 h3 . . .

w0 w0 w0 w0

h′0 h′1 h′2 h′3. . .

w1 w2 w3 w4

Stack hidden layers(arbitrarily many)

OTHER TASKS

h0 h1 h2 h3 . . .

w0 w0 w0 w0

o0 o1 o2 o3

• Not just language modelling

• Trivial to apply to any supervised labelling task

• E.g. POS tagging

OTHER TASKS

h0 h1 h2 h3 . . .

w0 w0 w0 w0

JJ NNP VBI NNS

• E.g. POS tagging

OTHER TASKS

h0 h1 h2 h3 . . .

w0 w0 w0 w0

O Plc O Prs

• E.g. POS tagging, NER

RNNs FOR SENTIMENT

h0 h1 h2 h3

w0 w0 w0 w0

• Feed in input text

• Only predict at end

RNNs FOR SENTIMENT

h0 h1 h2 h3

w0 w0 w0 w0

RNNs FOR SENTIMENT

h0 h1 h2 h3

w0 w0 w0 w0

RNNs FOR SENTIMENT h0 h1 h2 h3

w0 w0 w0 w0

• RNN sees full input: prediction can depend on any word

• Supervised training: known sentiment scores/classes

• Potential advantages

• No more BOWs! RNN can learn phrases, MWE, . . .• Can generalize based on word similarity:

great → ⇒ awesome →• Might learn LR deps

w0 w0 w0 w0

• Potential advantages• No more BOWs! RNN can learn phrases, MWE, . . .

• Can generalize based on word similarity:

w0 w0 w0 w0

• Potential advantages• No more BOWs! RNN can learn phrases, MWE, . . .• Can generalize based on word similarity:

great → ⇒ awesome →

• Might learn LR deps

w0 w0 w0 w0

• Potential advantages• No more BOWs! RNN can learn phrases, MWE, . . .• Can generalize based on word similarity:

MORE NNs

• Lots more work in this area• Bidirectional RNNs• Convolutional NNs (CNNs)• Context-sensitive word embeddings• Seq2seq models: Machine Translation, etc

• Other courses:• Deep Learning• Deep Learning for NLP (seminar)

PART II SUMMARY

• Statistics in NLP

• Data-driven NLP, less linguistics

• Classification / regression

• Sentiment analysis

• SVMs

• Topic modelling: details of LDA

• Neural networks

• RNNs, applications

READING MATERIAL

• All statistical methods: Eisenstein

• Neural networks: Neural Network Methods in NLPE-book, available through Helka

NEXT UP

After lunch:Practical assignments in BK107

9:15 – 12:00 Lectures12:00 – 13:15 Lunch

13:15 – ∼13:30 Introduction13:30 – 16:00 Practical assignments

• Building an IE system

• Regular expression-based methods

• Building on other NLP components: pipeline

day 8: information extraction and advanced statistical nlp

Documents

part ii. statistical nlp - uni-freiburg.de

statistical nlp winter 2009

nlp in web data extraction (omer gunes)

statistical nlp winter 2009

comp790: statistical nlp

natural language processing (nlp) e information extraction...

information extraction, and partially supervised...

statistical techniques in nlp

statistical nlp: lecture 7

natural language processing (nlp) i. introduction ii. issues...

combining nlp approaches for rule extraction from … ·...

statistical methods for nlp - cs.columbia.edu

statistical nlp: course notes

combining nlp approaches for rule extraction from legal...

november 2003csa4050: information extraction i1 csa4050:...

seminar: statistical nlp

statistical relational learning for nlp

part ii. statistical nlp

combining nlp approaches for rule extraction from legal

introduction to statistical nlp