natural language processing (highlights)

Natural Language Processing(highlights)

Fall 2012 : Chambers

Early NLP• Dave: Open the pod bay doors, HAL. • HAL: I’m sorry Dave. I’m afraid I can’t do that.

Commercial NLP

NLP is hard. (news headlines)1. Minister Accused Of Having 8 Wives In Jail2. Juvenile Court to Try Shooting Defendant3. Teacher Strikes Idle Kids4. Miners refuse to work after death5. Local High School Dropouts Cut in Half6. Red Tape Holds Up New Bridges7. Clinton Wins on Budget, but More Lies Ahead8. Hospitals Are Sued by 7 Foot Doctors9. Police: Crack Found in Man's Buttocks

NLP needs to adapt.

NLP needs to adapt.

http://xkcd.com/1083/

NLP is also a Knowledge Problem

Language Models• Language Modeling

• Build probabilities of words and phrases

• Author Detection• Who wrote this email? (is it spam?)• Historical analysis, who was the author of this book?• Intelligence community, who wrote this incendiary blog?

Language Models: Author IDIt was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday.

Mr. Bennet was among the earliest of those who waited on Mr. Bingley. Hehad always intended to visit him, though to the last always assuringhis wife that he should not go; and till the evening after the visit waspaid she had no knowledge of it.

Baby, baby, baby ooohLike baby, baby, baby noooLike baby, baby, baby ooohI thought you'd always be mine

- Charles Dickens

- Jane Austen

- Justin Bieber

Motivation• We want to predict something.• We have some text related to this something.

• something = target label Y• text = text features X

Given X, what is the most probable Y?

Motivation: Author DetectionAlas the day! take heed of him; he stabbed me inmine own house, and that most beastly: in goodfaith, he cares not what mischief he does. If hisweapon be out: he will foin like any devil; he will

spare neither man, woman, nor child.

X =

Y = { Charles Dickens, William Shakespeare, Herman

Melville, Jane Austin, Homer, Leo Tolstoy }

)|()(maxarg kkyyYXPyYPY

k

N-gram Terminology• Unigrams: single words• Bigrams: pairs of words• Trigrams: three word phrases• 4-grams, 5-grams, 6-grams, etc.

“I saw a lizard yesterday”Unigrams

Isaw

alizard

yesterday</s>

Bigrams<s> II sawsaw a

a lizardlizard yesterdayyesterday </s>

Trigrams<s> <s> I<s> I saw

I saw asaw a lizard

a lizard yesterdaylizard yesterday </s>

Sentiment Analysis

It's about finding out what people think...

Online social media sentiment apps

● Several Sentiment Sites● Twitter sentiment http://twittersentiment.appspot.com/

● Twends: http://twendz.waggeneredstrom.com/

● Twittratr: http://twitrratr.com/

Or was she?

Twitter for Stock Market Prediction

“Hey Jon, Derek in Atlanta is having a bacon and egg, er, sandwich. Is that good for wheat futures?”

Sometimes science is hype• The Bollen paper has since been strongly questioned

by others in the field.

• It contained some overuse of statistical significance tests that could have overestimated how well sentiment actually aligned with market movements.

• Nobody has been able to recreate their findings.

Monitor Real-World Events

Learn a Lexicon1. Find some data that is labeled

• Movie reviews have star ratings• Manually label data yourself • Use a noisy label, such as “#angry” on tweets

2. Learn a model from the labeled data• Naïve Bayes Classifier• MaxEnt Model (you have not yet learned)• Decision Trees• etc.

Try it now!

Track Population Moods

Information Extraction

http://www.youtube.com/watch?v=YLR1byL0U8M

24

Current Examples• Fact extraction about people.

Instant biographies.• Search “tom hanks” on google

• Never-ending Language Learning• http://rtw.ml.cmu.edu/rtw/

Where is the Naval Academy?• The United States Naval Academy (also known as USNA,

Annapolis, or Navy) is a four-year coeducational federal service academy located in Annapolis.

• Start your tour at the Armel-Leftwich Visitor Center of the United States Naval Academy, Annapolis, Md.

• this is a great place to walk around, whether you are a 1st time or frequent visitor to annapolis. the academy's campus is situated along the creek, thus offering beautiful views of the water and horizons.

P(annapolis | sentence) = P(annapolis | features/ngrams/etc.)

Extracting structured knowledge

LLNL EQ Lawrence Livermore National Laboratory LLNL LOC-IN CaliforniaLivermore LOC-IN CaliforniaLLNL IS-A scientific research laboratoryLLNL FOUNDED-BY University of CaliforniaLLNL FOUNDED-IN 1952

Each article can contain hundreds or thousands of items of knowledge...

“The Lawrence Livermore National Laboratory (LLNL) in Livermore, California is a scientific research

laboratory founded by the University of California in 1952.”

Sentence Parsing

28

Sentence Parsing• “Fed raises interest rates”

29

Example 2“I saw the man on the hill with a telescope.”

30

Words barely affect structure.

telescopes planets

Correct!!! Incorrect

Machine TranslationStart at ~6min in.http://www.youtube.com/watch?v=Nu-nlQqFCKg

Machine Translation• Commercial-grade translation

• translate.google.com

Machine Translation• How to model translations?

• Words: P( casa | house )• Spurious words: P( a | null )• Fertility: Pn( 1 | house )

• English word translates to one Spanish word• Distortion: Pd( 5 | 2 )

• The 2nd English word maps to the 5th Spanish word

Distortion• Encourage translations to follow the diagonal…

• P( 4 | 4 ) * P( 5 | 5 ) * …

Learning Translations• Huge corpus of “aligned sentences”.• Europarl

• Corpus of European Parliamant proceedings• The EU is mandated to translate into all 21 official languages• 21 languages, (semi-) aligned to each other

• P( casa | house ) = (count all casa/house pairs!)• Pd( 2 | 5 ) = (count all sentences where 2nd word went

to 5th word)

Machine Translation Technology• Hand-held devices for military

• Speak english -> recognition -> translation -> generate Urdu

• Translate web documents

• Education technology?• Doesn’t yet receive much of a focus

Text Influence

Text Influence• Can text style influence people?• Can a computer learn to adapt language to

accomplish a goal?

• Obama 2012 campaign• Sent emails to people every day asking for donations• Sent variations of email, and learned what features caused

more donations• http://www.businessweek.com/articles/2012-11-29/the-science-behind-those-obama-ca

mpaign-e-mails

Mobile Devices

Mobile Devices• Keystroke prediction has been around for a while

now.

• New idea: learn individual user preferences• New idea: use a user’s social media text to train on

• http://www.youtube.com/watch?v=3hQT-o8ch0o• http://www.youtube.com/watch?v=kA5Horw_SOE