natural language processingrobotics.cs.tamu.edu/dshell/cs420/nlp.pdf · 2019-12-01 · natural...

NaturalLanguageProcessing

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Unigram, Bigramand N-gram

Natural Language Processing

Nov 19, 2019

1 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

2 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Who wrote the Federalist Papers?

1787-8: anonymous essays try to convince New York toratify U.S Constitution: Jay, Madison, Hamilton.

Authorship of 12 of the letters in dispute.

1963: solved by Mosteller and Wallace using Bayesianmethods.

By the end of this lecture we will see how to do that.

3 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Who wrote the Federalist Papers?

1787-8: anonymous essays try to convince New York toratify U.S Constitution: Jay, Madison, Hamilton.

Authorship of 12 of the letters in dispute.

1963: solved by Mosteller and Wallace using Bayesianmethods.

By the end of this lecture we will see how to do that.

3 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

What makes it hard?

Formal languages are:

unambiguous

Natural languages areambiguous:

“He saw her duck”.“Time flies like an arrow. Fruit flies like a banana”

4 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

By the end of the class

By the end of the class we will see how to do:

1 Text Classification. E.g. Spam detection, Authorshipidentification.

2 Spell Correction. E.g. Auto-correct.

3 Word suggestion.

5 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Regular Expressions

A formal language for specifying text strings.

6 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Notations

• Disjunctions [] :

Pattern Matches[Ww ]oodchuck woodchuck, Woodchuck[0123456789] Any single digit

• Disjunctions |:

Pattern Matchesabc|def Find ‘abc’ or ‘def’.a|b|ab Find ‘a’ or ‘b’ or ‘ab’. Example: ‘abc’

7 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Notations

• Ranges:

Pattern Matches[A− Z ] An uppercase letter.[a− z ] A lowercase letter.[0− 9] A single digit.

• Negation ˆ. (Note: Carat means negation only when its firstin [])

Pattern Matches[ˆA− Z ] Not upper case

[ˆSs] Not ‘S’ nor ‘s’[ˆeˆ] Not ‘e’ nor ‘ˆ’aˆb Search for the pattern‘aˆb’

8 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Notations (? * . + ˆ $)

? 0 or 1 of previous character* 0 or more of previous character+ 1 or more of previous character. Any characterˆ Start anchor$ End anchor\ Escape character

9 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Examples

• Pattern: ˆ[A− Z ].Which of them are matches? “Class”, “cSCE”, “420”.

Class.• Pattern: ˆ[ˆA− Z ].Which of them are matches? “Class”, “cSCE”, “420”.cSCE, 420.• Pattern: .$Which of them are matches? “end”, “end?”, “end!”, “end.”.end. end? end! end.• Pattern: \.$Which of them are matches? “end”, “end?”, “end!”, “end.”.end..

10 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Examples

• Pattern: ˆ[A− Z ].Which of them are matches? “Class”, “cSCE”, “420”.Class.

• Pattern: ˆ[ˆA− Z ].Which of them are matches? “Class”, “cSCE”, “420”.cSCE, 420.• Pattern: .$Which of them are matches? “end”, “end?”, “end!”, “end.”.end. end? end! end.• Pattern: \.$Which of them are matches? “end”, “end?”, “end!”, “end.”.end..

10 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Examples

• Pattern: ˆ[A− Z ].Which of them are matches? “Class”, “cSCE”, “420”.Class.• Pattern: ˆ[ˆA− Z ].Which of them are matches? “Class”, “cSCE”, “420”.

cSCE, 420.• Pattern: .$Which of them are matches? “end”, “end?”, “end!”, “end.”.end. end? end! end.• Pattern: \.$Which of them are matches? “end”, “end?”, “end!”, “end.”.end..

10 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Examples

• Pattern: ˆ[A− Z ].Which of them are matches? “Class”, “cSCE”, “420”.Class.• Pattern: ˆ[ˆA− Z ].Which of them are matches? “Class”, “cSCE”, “420”.cSCE, 420.

• Pattern: .$Which of them are matches? “end”, “end?”, “end!”, “end.”.end. end? end! end.• Pattern: \.$Which of them are matches? “end”, “end?”, “end!”, “end.”.end..

10 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Examples

• Pattern: ˆ[A− Z ].Which of them are matches? “Class”, “cSCE”, “420”.Class.• Pattern: ˆ[ˆA− Z ].Which of them are matches? “Class”, “cSCE”, “420”.cSCE, 420.• Pattern: .$Which of them are matches? “end”, “end?”, “end!”, “end.”.

end. end? end! end.• Pattern: \.$Which of them are matches? “end”, “end?”, “end!”, “end.”.end..

10 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Examples

• Pattern: ˆ[A− Z ].Which of them are matches? “Class”, “cSCE”, “420”.Class.• Pattern: ˆ[ˆA− Z ].Which of them are matches? “Class”, “cSCE”, “420”.cSCE, 420.• Pattern: .$Which of them are matches? “end”, “end?”, “end!”, “end.”.end. end? end! end.

• Pattern: \.$Which of them are matches? “end”, “end?”, “end!”, “end.”.end..

10 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Examples

• Pattern: ˆ[A− Z ].Which of them are matches? “Class”, “cSCE”, “420”.Class.• Pattern: ˆ[ˆA− Z ].Which of them are matches? “Class”, “cSCE”, “420”.cSCE, 420.• Pattern: .$Which of them are matches? “end”, “end?”, “end!”, “end.”.end. end? end! end.• Pattern: \.$Which of them are matches? “end”, “end?”, “end!”, “end.”.

10 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Examples

• Pattern: ˆ[A− Z ].Which of them are matches? “Class”, “cSCE”, “420”.Class.• Pattern: ˆ[ˆA− Z ].Which of them are matches? “Class”, “cSCE”, “420”.cSCE, 420.• Pattern: .$Which of them are matches? “end”, “end?”, “end!”, “end.”.end. end? end! end.• Pattern: \.$Which of them are matches? “end”, “end?”, “end!”, “end.”.end..

10 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Examples

• Pattern: colou?r .Which of them are matches? “color”, “colour”, “colouur”.

color, colour.• Pattern: colou + r .Which of them are matches? “color”, “colour”, “colouur”.colour, colouur.• Pattern: colou ∗ r .Which of them are matches? “color”, “colour”, “colouur”.color, colour, colouur.• Pattern: colou.r .Which of them are matches? “color”, “colour”, “colouur”.colouur.

11 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Examples

• Pattern: colou?r .Which of them are matches? “color”, “colour”, “colouur”.color, colour.

• Pattern: colou + r .Which of them are matches? “color”, “colour”, “colouur”.colour, colouur.• Pattern: colou ∗ r .Which of them are matches? “color”, “colour”, “colouur”.color, colour, colouur.• Pattern: colou.r .Which of them are matches? “color”, “colour”, “colouur”.colouur.

11 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Examples

• Pattern: colou?r .Which of them are matches? “color”, “colour”, “colouur”.color, colour.• Pattern: colou + r .Which of them are matches? “color”, “colour”, “colouur”.

colour, colouur.• Pattern: colou ∗ r .Which of them are matches? “color”, “colour”, “colouur”.color, colour, colouur.• Pattern: colou.r .Which of them are matches? “color”, “colour”, “colouur”.colouur.

11 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Examples

• Pattern: colou?r .Which of them are matches? “color”, “colour”, “colouur”.color, colour.• Pattern: colou + r .Which of them are matches? “color”, “colour”, “colouur”.colour, colouur.

• Pattern: colou ∗ r .Which of them are matches? “color”, “colour”, “colouur”.color, colour, colouur.• Pattern: colou.r .Which of them are matches? “color”, “colour”, “colouur”.colouur.

11 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Examples

• Pattern: colou?r .Which of them are matches? “color”, “colour”, “colouur”.color, colour.• Pattern: colou + r .Which of them are matches? “color”, “colour”, “colouur”.colour, colouur.• Pattern: colou ∗ r .Which of them are matches? “color”, “colour”, “colouur”.

color, colour, colouur.• Pattern: colou.r .Which of them are matches? “color”, “colour”, “colouur”.colouur.

11 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Examples

• Pattern: colou?r .Which of them are matches? “color”, “colour”, “colouur”.color, colour.• Pattern: colou + r .Which of them are matches? “color”, “colour”, “colouur”.colour, colouur.• Pattern: colou ∗ r .Which of them are matches? “color”, “colour”, “colouur”.color, colour, colouur.

• Pattern: colou.r .Which of them are matches? “color”, “colour”, “colouur”.colouur.

11 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Examples

• Pattern: colou?r .Which of them are matches? “color”, “colour”, “colouur”.color, colour.• Pattern: colou + r .Which of them are matches? “color”, “colour”, “colouur”.colour, colouur.• Pattern: colou ∗ r .Which of them are matches? “color”, “colour”, “colouur”.color, colour, colouur.• Pattern: colou.r .Which of them are matches? “color”, “colour”, “colouur”.

colouur.

11 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Examples

• Pattern: colou?r .Which of them are matches? “color”, “colour”, “colouur”.color, colour.• Pattern: colou + r .Which of them are matches? “color”, “colour”, “colouur”.colour, colouur.• Pattern: colou ∗ r .Which of them are matches? “color”, “colour”, “colouur”.color, colour, colouur.• Pattern: colou.r .Which of them are matches? “color”, “colour”, “colouur”.colouur.

11 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Example

We need to find instances of “the” in a text.

the × ‘The’

[Tt]he × ‘Theology’

[ˆA− Za− z ][Tt]he[ˆA− Za− z ]

12 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Example

× ‘The’

12 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Example

the × ‘The’

12 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Example

the × ‘The’

[Tt]he

× ‘Theology’

12 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Example

the × ‘The’

12 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Example

the × ‘The’

12 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Text classification

Assigning subject categories, topics, or genres.

Spam detection.

Authorship identification.

Age/gender identification.

Language Identification.

· · ·

13 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Text Classification

Inputs:

Document d.Fixed set of classes C = {c1, c2, · · · , cn}.

Output:

A predicted class c ∈ C

14 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Naive Bayes

Relies on simple representation of document – Bag of Words.

For a document d and a class c

P(c |d) =P(d |c)P(c)

15 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Naive Bayes

Relies on simple representation of document – Bag of Words.

For a document d and a class c

P(c |d) =P(d |c)P(c)

15 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Naive Bayes Classifier

cMAP = argmaxc∈C

P(c |d)

MAP - Maximum a posteriori (most likely class).

cMAP = argmaxc∈C

P(d |c)P(c)

cMAP = argmaxc∈C

P(d |c)P(c)

16 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Naive Bayes Classifier

cMAP = argmaxc∈C

P(d |c)P(c)

Let’s say that the document is represented by n featuresx1, x2, · · · xn

cMAP = argmaxc∈C

P(x1, x2, · · · xn|c)P(c)

17 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Assumptions

Bag of words: Position of words does not matter.Conditional Independence: The feature probabilities P(xi |c)are independent given the class c .

P(x1, x2, · · · xn|c) =n∏

P(xi |c)

18 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Bag of word representation

19 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Bag of word representation

20 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Naive Bayes: Learning

What do we need?Training set of m hand-labeled documents(d1, c1), · · · , (dm, cm)

21 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Let ND be the number of documents, and Ncj be the numberof documents present in class cj .Let Vcj be the set of all words in the documents of class cjNow we find the maximum likelihood estimates:

P̂(cj) =Ncj

P̂(wi |cj) =count(wi , cj)∑

w∈Vcjcount(w , cj)

Now we can classify a document d by:

cd = argmaxcj∈C

P̂(cj)∏wi∈d

P̂(wi |cj)

22 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

What if we come across an unknown word in the document d .Let wu be the unknown word P̂(wu|cj) = 0,∀cj .

23 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Laplace smoothing

Let V be the set of all words in the test documents, i.e,V = ∪cjVcj

Add one word for the unknown word in the vocabulary.

P̂(wi |cj) =count(wi , cj) + 1∑

w∈Vcjcount(w , cj) + |V |+ 1

So, for all unknown words, we have:

P̂(wu|cj) =1∑

w∈Vcjcount(w , cj) + |V |+ 1

24 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Example

Training set:

# Text Class1 Carla Betty Carla a2 Carla Carla Suzanne a3 Carla Matt a4 Taylor Jessica Carla b

P̂(a) = 3/4P̂(b) = 1/4

25 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Example

Training set:

P̂(a) =

3/4P̂(b) = 1/4

25 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Example

Training set:

P̂(a) = 3/4P̂(b) =

25 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Example

Training set:

P̂(a) = 3/4P̂(b) = 1/4

25 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Example

Training set:

P̂(Carla|a) =(5 + 1)/(8 + 6 + 1) = 6/15P̂(Taylor |a) = (0 + 1)/(8 + 6 + 1) = 1/15

26 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Example

Training set:

P̂(Carla|a) =

(5 + 1)/(8 + 6 + 1) = 6/15P̂(Taylor |a) = (0 + 1)/(8 + 6 + 1) = 1/15

26 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Example

Training set:

P̂(Carla|a) =(5 + 1)/(8 + 6 + 1) = 6/15P̂(Taylor |a) =

(0 + 1)/(8 + 6 + 1) = 1/15

26 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Example

Training set:

P̂(Carla|a) =(5 + 1)/(8 + 6 + 1) = 6/15P̂(Taylor |a) = (0 + 1)/(8 + 6 + 1) = 1/15

26 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Example

Document d5: Carla Carla Carla Taylor Jessica

P(a|d5) = 3/4× (6/15)3 × 1/15× 1/15 ≈ 0.0002P(b|d5) = 1/4× (2/10)3 × 2/10× 2/10 ≈ 0.00008

27 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Example

P(a|d5) = 3/4× (6/15)3 × 1/15× 1/15 ≈ 0.0002P(b|d5) = 1/4× (2/10)3 × 2/10× 2/10 ≈ 0.00008

27 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Example

P(a|d5) =

3/4× (6/15)3 × 1/15× 1/15 ≈ 0.0002P(b|d5) = 1/4× (2/10)3 × 2/10× 2/10 ≈ 0.00008

27 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Example

P(a|d5) = 3/4× (6/15)3 × 1/15× 1/15 ≈ 0.0002P(b|d5) =

1/4× (2/10)3 × 2/10× 2/10 ≈ 0.00008

27 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Example

P(a|d5) = 3/4× (6/15)3 × 1/15× 1/15 ≈ 0.0002P(b|d5) = 1/4× (2/10)3 × 2/10× 2/10 ≈ 0.00008

27 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Naive Bayes

Naive Bayes is not so naive!!

Robust to Irrelevant Features.

Optimal if the independence assumptions hold.

A good dependable baseline for text classification. - Thereexists other classifiers that give better accuracy

28 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Federalist Papers

Discussion: Federalist papers.E.g. What training set do we need?

29 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Language Modelling

Goal: Assign probability to a sentence.

Machine Translation. P(high winds tonight) > P(largewinds tonight)

Spell Correction. P(about fifteen minutes from) >P(about fifteen minuets from)

Speech Recognition. P(I saw a van) > P(eyes awe of an)

30 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Language Modelling

Goal: Assign probability to a sentence.Why?

30 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Language Modelling

30 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Language Modelling

30 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Language Modelling

30 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Language Modelling

Goal: compute the probability of a sentence or sequenceof words. P(W ) = P(w1,w2, · · · ,wn)

Related task: probability of an upcoming word.P(wi |w1,w2, · · · ,wi−1)

Chain rule:

P(x1, x2, · · · , xn)

= P(x1)P(x2|x1)P(x3|x1, x2) · · ·P(xn|x1, x2 · · · , xn−1)

P(xi |x1, x2, · · · xi−1)

31 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Language Modelling

Goal: compute the probability of a sentence or sequenceof words. P(W ) = P(w1,w2, · · · ,wn)

Related task: probability of an upcoming word.P(wi |w1,w2, · · · ,wi−1)

Chain rule:

P(x1, x2, · · · , xn)

= P(x1)P(x2|x1)P(x3|x1, x2) · · ·P(xn|x1, x2 · · · , xn−1)

P(xi |x1, x2, · · · xi−1)

31 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Example

P(its water is so transparent that) = P(its)× P(water |its)×P(so|its,water)× P(transparent|its,water , is, so)×P(that|its,water , is, so, transparent)Can we count?

P(that|its,water , is, so, transparent)

=P(its,water , is, so, transparent, that)

P(its,water , is, so, transparent)

No. Too many possibilities.

32 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Example

P(its water is so transparent that) = P(its)× P(water |its)×P(so|its,water)× P(transparent|its,water , is, so)×P(that|its,water , is, so, transparent)Can we count?

P(that|its,water , is, so, transparent)

=P(its,water , is, so, transparent, that)

P(its,water , is, so, transparent)

No. Too many possibilities.

32 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Markov Assumption

Take only the k words preceding it.

P(wi |w1,w2, · · · ,wi−1) ≈ P(wi |wi−k · · · ,wi−1)

P(that|its,water , is, so, transparent) = P(that|transparent)

P(that|its,water , is, so, transparent) = P(that|so, transparent)

33 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Markov Assumption

Take only the k words preceding it.

P(wi |w1,w2, · · · ,wi−1) ≈ P(wi |wi−k · · · ,wi−1)

P(that|its,water , is, so, transparent) = P(that|transparent)

P(that|its,water , is, so, transparent) = P(that|so, transparent)

33 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Unigram, Bigram and N-gram

Unigram model:

P(w1,w2, · · · ,wn−1,wn) ≈∏i

P(wi )

Bigram model:

P(wi |w1,w2, · · · ,wi−1) ≈ P(wi |wi−1)

N-gram model:Extension to trigram, 4-gram, 5-gram, etc.

34 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Discussion

1. Spell correction

2. Word suggestion.

35 / 35

Introduction

RegularExpression

Notations

Examples

TextClassification

Naive Bayes

Example

LanguageModelling

Discussion

1. Spell correction2. Word suggestion.

35 / 35

natural language processingrobotics.cs.tamu.edu/dshell/cs420/nlp.pdf · 2019-12-01 · natural...

Documents

cs420: operating systems file-system interface20_file... ·...

chapter 3 the relational model and normalization cs420...

cs420 lecture five priority queues, sorting wim bohm cs csu

cs420-2016s-01 java vs. c++ 1 01-0:...

stigmergy, self-organization, and owen holland sorting in...

cs420 lecture twelve shortest paths wim bohm cs csu

cs420: operating systems deadlocks & deadlock...

cs420 machine learning -...

living machines - texas a&m...

cs420 convex#hulls# - university of british...

cs420: operating systems kernel i/o...

cs420 machine learning - weinan...

cs420/520 computer architecture i

unsupervised learning -...

cs420/520 computer architecture...

cs420 finite automata and regular languages

game engineering: 3dgalles/cs420/lecture/spatial... ·...

smc-it 2009 paper id #16 dshell++: a component …...smc-it...

keyframe animation -...

cs420/cse 402/ece 492 introduction to parallel programming...