computaonal linguiscs - the stanford natural language

Computa(onalLinguis(cs(akaNaturalLanguageProcessing)

BillMacCartneySymSys100

StanfordUniversity26May2011

(someslidesadaptedfromChrisManning)

xkcdsnarkiness

OK,Randall,it’sfunny…butwrong!

cartoon from xkcd.com

Awordonterminology

Ifyoucallit…

•  Computa(onalLinguis(cs(CL)•  …you’realinguist!•  …youusecomputerstostudylanguage

•  NaturalLanguageProcessing(NLP)•  …you’reacomputerscien(st!

•  …youworkonapplica(onsinvolvinglanguage

Butreally,they’repreQymuchsynonymous

Let’sgetsituated!

Today,weareinthisinters(ce

cartoon from xkcd.com

NLP:TheVision

I’msorry,Dave.Ican’tdothat.

Oh,dear! Thatiscorrect,captain.

Language:theul(mateUI

WhereisABug’sLifeplayinginMountainView?

ABug’sLifeisplayingattheCentury16Theater.

Whenisitplayingthere?

It’splayingat2pm,5pm,and8pm.

OK.I’dlike1adultand2childrenforthefirstshow.Howmuchwouldthatcost?

Butweneeddomainknowledge,discourseknowledge,worldknowledge(Nottomen(onlinguis(cknowledge!)

NLP:Goalsofthefield

•  Fromthelo_y…•  full‐onnaturallanguageunderstanding•  par(cipa(oninspokendialogues•  open‐domainques(onanswering•  real‐(mebi‐direc(onaltransla(on

•  …tothemundane•  iden(fyingspam•  categorizingnewsstories(&otherdocs)•  finding&comparingproductinforma(onontheweb•  assessingsen(menttowardproducts,brands,stocks,…

Predominantinrecentyears

NLPinthecommercialworld

Powerset

Currentmo(va(onsforNLP

•  Theexplosionofmachine‐readablenaturallanguagetext•  Exabytes(1018bytes)oftext,doublingeveryyearortwo•  Webpages,emails,IMs,SMSs,tweets,docs,PDFs,…•  Opportunity—andincreasingnecessity—toextractmeaning

•  Media(onofhumaninterac(onsbycomputers•  Opportunityforthecomputerinthelooptodomuchmore

•  Growingroleoflanguageinhuman‐computerinterac(on

What’sdrivingNLP?Threetrends:

Furthermo(va(onforCL

Onereasonforstudyinglanguage—andformepersonallythemostcompellingreason—isthatitistemp6ngtoregardlanguage,inthetradi6onalphrase,asa“mirrorofmind”.

Chomsky,1975

Forthesamereason,computa(onallinguis(csisacompellingwaytostudypsycholinguis(csandlanguageacquisi(on.

Some(mes,thebestwaytounderstandsomethingistobuildamodelofit.

WhatIcannotcreate,Idonotunderstand.Feynman,1988

Earlyhistory:50sand60s

•  Founda(onalworkonautomata,formallanguages,probabili(s(cmodeling,andinforma(ontheory

•  Firstspeechsystems(Davisetal.,BellLabs)

•  MTheavilyfundedbymilitary—hugeoverconfidence

•  Butusingmachinesdumberthanapocketcalculator

•  LiQleunderstandingofsyntax,seman(cs,pragma(cs

•  ALPACreport(1966):crap,thisisreallyhard!

Refocusing:70sand80s

•  Founda(onalworkonspeechrecogni(on:stochas(cmodeling,hiddenMarkovmodels,the“noisychannel”

•  Ideasfromthisworkwouldlaterrevolu(onizeNLP!

•  Logicprogramming,rules‐drivenAI,determinis(calgorithmsforsyntac(cparsing(e.g.,LFG)

•  Increasinginterestinnaturallanguageunderstanding:SHRDLU,LUNAR,CHAT‐80

•  ButsymbolicAIhitthewall:“AIwinter”

Thesta(s(calrevolu(on:90s

•  InfluxofnewideasfromEE&ASR:probabilis(cmodeling,corpussta(s(cs,supervisedlearning,empiricalevalua(on

•  Newsourcesofdata:explosionofmachine‐readabletext;human‐annotatedtrainingdata(e.g.,thePennTreebank)

•  Availabilityofmuchmorepowerfulmachines

•  Loweredexpecta(ons:forgetfullseman(cunderstanding,let’sdotextcat,part‐of‐speechtagging,NER,andparsing!

Theriseofthemachines:00s

•  Consolida(onofthegainsofthesta(s(calrevolu(on

•  Moresophis(catedsta(s(calmodelingandmachinelearningalgorithms:MaxEnt,SVMs,BayesNets,LDA,etc.

•  Bigbigdata:100xgrowthofweb,massiveserverfarms

•  Focusshi_ingfromsupervised to unsupervisedlearning

•  Revivedinterestinhigher‐levelseman(capplica(ons

Subfieldsandtasks

Textcategoriza(on Coreferenceresolu(on Ques(onanswering(QA)

Part‐of‐speech(POS)tagging Wordsensedisambigua(on(WSD)

Textualinference&paraphrase

Nameden(tyrecogni(on(NER) Syntac(cparsing Summariza(on

Informa(onextrac(on(IE) Machinetransla(on(MT) Discourse&dialog

Sen(mentanalysis

mostlysolved makinggoodprogress s(llreallyhard

Spamdetec(onOK,let’smeetbythebig…

D1cktoosmall?BuyV1AGRA…

✓ ✗

PhilliesshutdownRangers2‐0

Joblessratehitstwo‐yearlow

SPORTS

BUSINESS

Colorlessgreenideassleepfuriously.

ADJADJNOUNVERBADV

ObamametwithUAWleadersinDetroit…

PERSONORGLOC

You’reinvitedtoourbungabungaparty,FridayMay27at8:30pminCorduraHall

PartyMay27add

Thephowasauthen(candyummy.

Waiterignoredusfor20minutes.

ObamatoldMubarakheshouldn’trunagain.

IneednewbaQeriesformymouse.

Ourspecialtyispandafriedrice.

我们的专长是熊猫炒饭

Sheencon(nuesrantagainst…Sheencon(nuesrantagainst…Sheencon(nuesrantagainst…

Sheenisnuts

Q.WhatcurrencyisusedinChina?

A.Theyuan

IcanseeRussiafrommyhouse!

T.Thirteensoldierslosttheirlives…

H.Severaltroopswerekilledinthe… YES

WhereisThorplayinginSF?

Metreonat4:30and7:30

Seman(csearchpeopleprotes(ngglobaliza(on Search

…demonstratorsstormedIMFoffices…

WhyisNLPhard?

Naturallanguageis:•  highlyambiguousatalllevels

•  complex,withrecursivestructuresandcoreference

•  subtle,exploi(ngcontexttoconveymeaning

•  fuzzyandvague•  involvesreasoningabouttheworld•  partofasocialsystem:persuading,insul(ng,amusing,…

(Nevertheless,simplefeatureso_endohalfthejob!)

Meaningsandexpressions

soda

so_drink

pop

beverage

Coke

Onemeaning,manyexpressions

ImageCaptureDevice:1.68millionpixel1/2‐inchCCDsensor

ImageCaptureDeviceTotalPixelsApprox.3.34millionEffec(vePixelsApprox.3.24million

ImagesensorTotalPixels:Approx.2.11million‐pixel

ImagingsensorTotalPixels:Approx.2.11million1,688(H)x1,248(V)

CCDTotalPixels:Approx.3,340,000(2,140[H]x1,560[V])Effec(vePixels:Approx.3,240,000(2,088[H]x1,550[V])RecordingPixels:Approx.3,145,000(2,048[H]x1,536[V])

Theseallcamefromthesamevendor’swebsite!

Tobuildashoppingsearchengine,youneedtoextractproductinforma(onfromvendors’websites:

Onemeaning,manyexpressions

Gazpromconfirmstwo‐foldincreaseingaspriceforGeorgia

GazpromdoublesGeorgia'sgasbill

Russiagasmonopolytodoublepriceofgas

RussiahitsGeorgiawithhugeriseinitsgasbill

RussiaplanstodoubleGeorgiangasprice

RussiaincreasingpriceofgasforGeorgia Search

Russiadoublesgasbillto“punish”neighbourGeorgia

Orconsideraseman(csearchapplica(on:

Oneexpression,manymeanings

cartoon from qwantz.com

Syntac(c&seman(cambiguity

NPNP

VP

S

NPNP

PP

VP

S

photosfromworth1000.com

seman(cambiguity

syntac(cambiguity

FruitflieslikeabananaFruitflieslikeabanana

Ambiguousheadlines

Teacher Strikes Idle Kids China to Orbit Human on Oct. 15 Red Tape Holds Up New Bridges Hospitals Are Sued by 7 Foot Doctors Juvenile Court to Try Shooting Defendant Local High School Dropouts Cut in Half Police: Crack Found in Man's Buttocks

OK,whyelseisNLPhard?Ohsomanyreasons!

non‐standardEnglish

Greatjob@jus(nbieber!WereSOOPROUDofwhatyouveaccomplished!Utaughtus2#neversaynever&youyourselfshouldnevergiveupeither♥

segmenta1onissues idiomsdarkhorsegetcoldfeetloseface

throwinthetowel

neologisms

unfriendretweetbromanceteabagger

gardenpathsentences

Themanwhohuntsducksoutonweekends.

ThecoQonshirtsaremadefromgrowshere.

trickyen1tynames

…amuta(onontheforgene…

WhereisABug’sLifeplaying…

MostofLetItBewasrecorded…

worldknowledge

MaryandSuearesisters.

MaryandSuearemothers.

prosody

Ineversaidshestolemymoney.



lexicalspecificity

Butthat’swhatmakesitfun!

theNewYork‐NewHavenRailroad

theNewYork‐NewHavenRailroad

So,howtomakeprogress?

•  Thetaskisdifficult!Whattoolsdoweneed?•  Knowledgeaboutlanguage•  Knowledgeabouttheworld•  Awaytocombineknowledgesources

•  Theanswerthat’sbeengezngtrac(on:•  probabilis(cmodelsbuiltfromlanguagedata

•  P(“maison”→“house”)high

•  P(“L’avocatgénéral”→“thegeneralavocado”)low

•  Somethinkthisisafancynew“A.I.”idea•  Butreallyit’sanoldideastolenfromtheelectricalengineers…

Machinetransla(on(MT)

美国关岛国际机场及其办公室均接获一名自称沙地阿拉伯富商拉登等发出的电子邮件，威胁将会向机场等公众地方发动生化袭击後，关岛经保持高度戒备。

TheU.S.islandofGuamismaintainingahighstateofalerta_ertheGuamairportanditsofficesbothreceivedane‐mailfromsomeonecallinghimselftheSaudiArabianOsamabinLadenandthreateningabiological/chemicalaQackagainstpublicplacessuchastheairport.

•  Theclassicacidtestfornaturallanguageprocessing.

•  Requirescapabili(esinbothinterpreta(onandgenera(on.

•  About$10billionspentannuallyonhumantransla(on.

Empiricalsolu(on

Hieroglyphs

ParallelTexts:TheRoseQaStone

Demo(c

Greek

Empiricalsolu(on

Hmm,every(meonesees“banco”,transla(onis“bank”or“bench”…Ifit’s“bancode…”,italwaysbecomes“bank”,never“bench”…

slide from Kevin Knight

ParallelTexts:–  HongKongLegisla(on–  MacaoLegisla(on

–  CanadianParliamentHansards

–  UnitedNa(onsReports–  EuropeanParliament

–  Instruc(onManuals–  Mul(na(onalcompany

websites

Sindarin‐English

Iamarprestaraen.Theworldischanged.

Hanmathonnenen.Ifeelitinthewaters.

Hanmathonnechae.Ifeelitintheearth.

Ahannostonned'wilith.Ismellitintheair.

FellowshipoftheRingsmoviescript

slide from Lori Levin

ABayesianapproach

ê=argmaxeP(e|f)

=argmaxeP(f)

P(f|e)P(e)

=argmaxeP(f|e)P(e)

languagemodeltransla(onmodel languagemodel(fluency)

transla(onmodel(fidelity)

The“noisychannel”model

illustration from Jurafsky & Martin

Languagemodels(LMs)

•  NoisychannelmodelrequireslanguagemodelP(e)

•  LMtellsuswhichsentencesseemlikelyor“good”

•  Givensomecandidatetransla(ons,LMhelpswith:•  wordchoice(“shrankfrom”or“shrankof”?)

•  wordordering(“toughdecisions”or“decisionstough”?)

sentence P(e)

Heshrankfromtoughdecisions. 1.89e‐11

Heshrankfromimportantdecisions. 9.46e‐12

Heshrankoftoughdecisions. 7.11e‐16

Heshrankfromdecisionstough. 3.21e‐17

Sta(s(callanguagemodels

•  Wherewillthelanguagemodelcomefrom?

•  We’llbuilditbycoun(ngthingsincorpusdata!

•  Sta(s(cales(ma(onofmodelparameters

•  Butwecan’tjustcountwholesentences

sentence count P(e)

Heshrankfromtoughdecisions. 1/49208 2.03e‐05

Heshrankfromimportantdecisions. 0/49208 0

Heshrankoftoughdecisions. 0/49208 0

Heshrankfromdecisionstough. 0/49208 0

toohigh!

toolow!

N‐gramlanguagemodels

•  Instead,we’llbreakthingsintopieces

•  Thisiscalledabigramlanguagemodel

•  Wecanes(matebigramprobabili(esfromcorpus

P(Heshrankfromtoughdecisions)=P(He|•)×P(shrank|He)×P(from|shrank)×…×P(decisions|tough)

w1 w2 C(w1) C(w1w2) P(w2|w1)

• He 49208 978 0.0199

He shrank 53142 21 0.0004

shrank from 122 17 0.1393

from tough 18777 184 0.0098

Sta(s(caltransla(onmodels

•  Noisychannelalsoneedstransla(onmodelP(f|e)

•  Similarstrategy:breaksentencepairsintophrases

•  Countco‐occurringpairsinalargeparallelcorpus

•  (ButI’llskipthegorydetails…)

e f C(e) C(e,f) P(f|e)

heshrank illuirépugnait 17 6 0.3529

from de 27111 17855 0.6586

from des 27111 6434 0.2373

toughdecisions décisionsdifficiles 98 81 0.8265

Sta(s(calMTSystems

French BrokenEnglish

English

Sta(s(calAnalysis Sta(s(calAnalysis

J’aitrèsfaim Iamsohungry

WhathungerhaveI,HungryIamso,Iamsohungry,HaveIthathunger…

LanguageModelP(e)

Transla1onModelP(f|e)

DecodingalgorithmargmaxeP(f|e)P(e)

French/EnglishParallelTexts

Michellemabellesontlesmotsquivonttrèsbienensemble


Michelle,mabelle,sontlesmotsquivonttrèsbienensemble



Michelle,mybeau(ful,arewordsthatgotogetherwell

EnglishTexts



Manygreattradi(onsinartoriginatedintheartofoneofthefive…

Applica(onsofthenoisychannel

Thismodelcanbeappliedtomanydifferentproblems!

Channelmodelspeechproduc(onOCRtypingwithspellingerrorstransla(ngtoEnglish

LanguagemodelEnglishwordsEnglishwordsEnglishwordsEnglishwords

ê=argmaxeP(x|e)P(e)

(WidelyusedatGoogle,forexample)

IfyoulikeNLP/CompLing…

•  learnJavaorPython(andplaywithJavaNLPorNLTK)•  studylogic,probability,sta(s(cs,linearalgebra•  getsomeexposuretolinguis(cs(LING1,…)•  studyAIandmachinelearning(CS121,CS221,CS229)

•  readJurafsky&Mar(norManning&Schütze

•  CS124:FromLanguagetoInforma(on(Jurafsky)

•  CS224N:NaturalLanguageProcessing(Manning)

•  CS224S:SpeechRecogni(on&Synthesis(Jurafsky)•  CS224U:NaturalLanguageProcessing(MacCartney)

Onemorefortheroad

cartoon from qwantz.com

computaonal linguiscs - the stanford natural language

Documents