Computa(onalLinguis(cs(akaNaturalLanguageProcessing)
BillMacCartneySymSys100
StanfordUniversity26May2011
(someslidesadaptedfromChrisManning)
xkcdsnarkiness
OK,Randall,it’sfunny…butwrong!
cartoon from xkcd.com
Awordonterminology
Ifyoucallit…
• Computa(onalLinguis(cs(CL)• …you’realinguist!• …youusecomputerstostudylanguage
• NaturalLanguageProcessing(NLP)• …you’reacomputerscien(st!
• …youworkonapplica(onsinvolvinglanguage
Butreally,they’repreQymuchsynonymous
Let’sgetsituated!
Today,weareinthisinters(ce
cartoon from xkcd.com
NLP:TheVision
I’msorry,Dave.Ican’tdothat.
Oh,dear! Thatiscorrect,captain.
Language:theul(mateUI
WhereisABug’sLifeplayinginMountainView?
ABug’sLifeisplayingattheCentury16Theater.
Whenisitplayingthere?
It’splayingat2pm,5pm,and8pm.
OK.I’dlike1adultand2childrenforthefirstshow.Howmuchwouldthatcost?
Butweneeddomainknowledge,discourseknowledge,worldknowledge(Nottomen(onlinguis(cknowledge!)
NLP:Goalsofthefield
• Fromthelo_y…• full‐onnaturallanguageunderstanding• par(cipa(oninspokendialogues• open‐domainques(onanswering• real‐(mebi‐direc(onaltransla(on
• …tothemundane• iden(fyingspam• categorizingnewsstories(&otherdocs)• finding&comparingproductinforma(onontheweb• assessingsen(menttowardproducts,brands,stocks,…
Predominantinrecentyears
NLPinthecommercialworld
Powerset
Currentmo(va(onsforNLP
• Theexplosionofmachine‐readablenaturallanguagetext• Exabytes(1018bytes)oftext,doublingeveryyearortwo• Webpages,emails,IMs,SMSs,tweets,docs,PDFs,…• Opportunity—andincreasingnecessity—toextractmeaning
• Media(onofhumaninterac(onsbycomputers• Opportunityforthecomputerinthelooptodomuchmore
• Growingroleoflanguageinhuman‐computerinterac(on
What’sdrivingNLP?Threetrends:
Furthermo(va(onforCL
Onereasonforstudyinglanguage—andformepersonallythemostcompellingreason—isthatitistemp6ngtoregardlanguage,inthetradi6onalphrase,asa“mirrorofmind”.
Chomsky,1975
Forthesamereason,computa(onallinguis(csisacompellingwaytostudypsycholinguis(csandlanguageacquisi(on.
Some(mes,thebestwaytounderstandsomethingistobuildamodelofit.
WhatIcannotcreate,Idonotunderstand.Feynman,1988
Earlyhistory:50sand60s
• Founda(onalworkonautomata,formallanguages,probabili(s(cmodeling,andinforma(ontheory
• Firstspeechsystems(Davisetal.,BellLabs)
• MTheavilyfundedbymilitary—hugeoverconfidence
• Butusingmachinesdumberthanapocketcalculator
• LiQleunderstandingofsyntax,seman(cs,pragma(cs
• ALPACreport(1966):crap,thisisreallyhard!
Refocusing:70sand80s
• Founda(onalworkonspeechrecogni(on:stochas(cmodeling,hiddenMarkovmodels,the“noisychannel”
• Ideasfromthisworkwouldlaterrevolu(onizeNLP!
• Logicprogramming,rules‐drivenAI,determinis(calgorithmsforsyntac(cparsing(e.g.,LFG)
• Increasinginterestinnaturallanguageunderstanding:SHRDLU,LUNAR,CHAT‐80
• ButsymbolicAIhitthewall:“AIwinter”
Thesta(s(calrevolu(on:90s
• InfluxofnewideasfromEE&ASR:probabilis(cmodeling,corpussta(s(cs,supervisedlearning,empiricalevalua(on
• Newsourcesofdata:explosionofmachine‐readabletext;human‐annotatedtrainingdata(e.g.,thePennTreebank)
• Availabilityofmuchmorepowerfulmachines
• Loweredexpecta(ons:forgetfullseman(cunderstanding,let’sdotextcat,part‐of‐speechtagging,NER,andparsing!
Theriseofthemachines:00s
• Consolida(onofthegainsofthesta(s(calrevolu(on
• Moresophis(catedsta(s(calmodelingandmachinelearningalgorithms:MaxEnt,SVMs,BayesNets,LDA,etc.
• Bigbigdata:100xgrowthofweb,massiveserverfarms
• Focusshi_ingfromsupervised to unsupervisedlearning
• Revivedinterestinhigher‐levelseman(capplica(ons
Subfieldsandtasks
Textcategoriza(on Coreferenceresolu(on Ques(onanswering(QA)
Part‐of‐speech(POS)tagging Wordsensedisambigua(on(WSD)
Textualinference¶phrase
Nameden(tyrecogni(on(NER) Syntac(cparsing Summariza(on
Informa(onextrac(on(IE) Machinetransla(on(MT) Discourse&dialog
Sen(mentanalysis
mostlysolved makinggoodprogress s(llreallyhard
Spamdetec(onOK,let’smeetbythebig…
D1cktoosmall?BuyV1AGRA…
✓ ✗
PhilliesshutdownRangers2‐0
Joblessratehitstwo‐yearlow
SPORTS
BUSINESS
Colorlessgreenideassleepfuriously.
ADJADJNOUNVERBADV
ObamametwithUAWleadersinDetroit…
PERSONORGLOC
You’reinvitedtoourbungabungaparty,FridayMay27at8:30pminCorduraHall
PartyMay27add
Thephowasauthen(candyummy.
Waiterignoredusfor20minutes.
ObamatoldMubarakheshouldn’trunagain.
IneednewbaQeriesformymouse.
Ourspecialtyispandafriedrice.
我们的专长是熊猫炒饭
Sheencon(nuesrantagainst…Sheencon(nuesrantagainst…Sheencon(nuesrantagainst…
Sheenisnuts
Q.WhatcurrencyisusedinChina?
A.Theyuan
IcanseeRussiafrommyhouse!
T.Thirteensoldierslosttheirlives…
H.Severaltroopswerekilledinthe… YES
WhereisThorplayinginSF?
Metreonat4:30and7:30
Seman(csearchpeopleprotes(ngglobaliza(on Search
…demonstratorsstormedIMFoffices…
WhyisNLPhard?
Naturallanguageis:• highlyambiguousatalllevels
• complex,withrecursivestructuresandcoreference
• subtle,exploi(ngcontexttoconveymeaning
• fuzzyandvague• involvesreasoningabouttheworld• partofasocialsystem:persuading,insul(ng,amusing,…
(Nevertheless,simplefeatureso_endohalfthejob!)
Meaningsandexpressions
soda
so_drink
pop
beverage
Coke
Onemeaning,manyexpressions
ImageCaptureDevice:1.68millionpixel1/2‐inchCCDsensor
ImageCaptureDeviceTotalPixelsApprox.3.34millionEffec(vePixelsApprox.3.24million
ImagesensorTotalPixels:Approx.2.11million‐pixel
ImagingsensorTotalPixels:Approx.2.11million1,688(H)x1,248(V)
CCDTotalPixels:Approx.3,340,000(2,140[H]x1,560[V])Effec(vePixels:Approx.3,240,000(2,088[H]x1,550[V])RecordingPixels:Approx.3,145,000(2,048[H]x1,536[V])
Theseallcamefromthesamevendor’swebsite!
Tobuildashoppingsearchengine,youneedtoextractproductinforma(onfromvendors’websites:
Onemeaning,manyexpressions
Gazpromconfirmstwo‐foldincreaseingaspriceforGeorgia
GazpromdoublesGeorgia'sgasbill
Russiagasmonopolytodoublepriceofgas
RussiahitsGeorgiawithhugeriseinitsgasbill
RussiaplanstodoubleGeorgiangasprice
RussiaincreasingpriceofgasforGeorgia Search
Russiadoublesgasbillto“punish”neighbourGeorgia
Orconsideraseman(csearchapplica(on:
Oneexpression,manymeanings
cartoon from qwantz.com
Syntac(c&seman(cambiguity
NPNP
VP
S
NPNP
PP
VP
S
photosfromworth1000.com
seman(cambiguity
syntac(cambiguity
FruitflieslikeabananaFruitflieslikeabanana
Ambiguousheadlines
Teacher Strikes Idle Kids China to Orbit Human on Oct. 15 Red Tape Holds Up New Bridges Hospitals Are Sued by 7 Foot Doctors Juvenile Court to Try Shooting Defendant Local High School Dropouts Cut in Half Police: Crack Found in Man's Buttocks
OK,whyelseisNLPhard?Ohsomanyreasons!
non‐standardEnglish
Greatjob@jus(nbieber!WereSOOPROUDofwhatyouveaccomplished!Utaughtus2#neversaynever&youyourselfshouldnevergiveupeither♥
segmenta1onissues idiomsdarkhorsegetcoldfeetloseface
throwinthetowel
neologisms
unfriendretweetbromanceteabagger
gardenpathsentences
Themanwhohuntsducksoutonweekends.
ThecoQonshirtsaremadefromgrowshere.
trickyen1tynames
…amuta(onontheforgene…
WhereisABug’sLifeplaying…
MostofLetItBewasrecorded…
worldknowledge
MaryandSuearesisters.
MaryandSuearemothers.
prosody
Ineversaidshestolemymoney.
Ineversaidshestolemymoney.
Ineversaidshestolemymoney.
lexicalspecificity
Butthat’swhatmakesitfun!
theNewYork‐NewHavenRailroad
theNewYork‐NewHavenRailroad
So,howtomakeprogress?
• Thetaskisdifficult!Whattoolsdoweneed?• Knowledgeaboutlanguage• Knowledgeabouttheworld• Awaytocombineknowledgesources
• Theanswerthat’sbeengezngtrac(on:• probabilis(cmodelsbuiltfromlanguagedata
• P(“maison”→“house”)high
• P(“L’avocatgénéral”→“thegeneralavocado”)low
• Somethinkthisisafancynew“A.I.”idea• Butreallyit’sanoldideastolenfromtheelectricalengineers…
Machinetransla(on(MT)
美国关岛国际机场及其办公室均接获一名自称沙地阿拉伯富商拉登等发出的电子邮件,威胁将会向机场等公众地方发动生化袭击後,关岛经保持高度戒备。
TheU.S.islandofGuamismaintainingahighstateofalerta_ertheGuamairportanditsofficesbothreceivedane‐mailfromsomeonecallinghimselftheSaudiArabianOsamabinLadenandthreateningabiological/chemicalaQackagainstpublicplacessuchastheairport.
• Theclassicacidtestfornaturallanguageprocessing.
• Requirescapabili(esinbothinterpreta(onandgenera(on.
• About$10billionspentannuallyonhumantransla(on.
Empiricalsolu(on
Hieroglyphs
ParallelTexts:TheRoseQaStone
Demo(c
Greek
Empiricalsolu(on
Hmm,every(meonesees“banco”,transla(onis“bank”or“bench”…Ifit’s“bancode…”,italwaysbecomes“bank”,never“bench”…
slide from Kevin Knight
ParallelTexts:– HongKongLegisla(on– MacaoLegisla(on
– CanadianParliamentHansards
– UnitedNa(onsReports– EuropeanParliament
– Instruc(onManuals– Mul(na(onalcompany
websites
Sindarin‐English
Iamarprestaraen.Theworldischanged.
Hanmathonnenen.Ifeelitinthewaters.
Hanmathonnechae.Ifeelitintheearth.
Ahannostonned'wilith.Ismellitintheair.
FellowshipoftheRingsmoviescript
slide from Lori Levin
Sta(s(calMT
Supposewehadaprobabilis(cmodeloftransla(onP(e|f)
Example:supposefisderienP(you’rewelcome|derien) =0.45P(nothing|derien) =0.13P(piddling|derien) =0.01P(underpants|derien) =0.000000001
Thenthebesttransla(onforfisargmaxeP(e|f)
ABayesianapproach
ê=argmaxeP(e|f)
=argmaxeP(f)
P(f|e)P(e)
=argmaxeP(f|e)P(e)
languagemodeltransla(onmodel languagemodel(fluency)
transla(onmodel(fidelity)
The“noisychannel”model
illustration from Jurafsky & Martin
Languagemodels(LMs)
• NoisychannelmodelrequireslanguagemodelP(e)
• LMtellsuswhichsentencesseemlikelyor“good”
• Givensomecandidatetransla(ons,LMhelpswith:• wordchoice(“shrankfrom”or“shrankof”?)
• wordordering(“toughdecisions”or“decisionstough”?)
sentence P(e)
Heshrankfromtoughdecisions. 1.89e‐11
Heshrankfromimportantdecisions. 9.46e‐12
Heshrankoftoughdecisions. 7.11e‐16
Heshrankfromdecisionstough. 3.21e‐17
Sta(s(callanguagemodels
• Wherewillthelanguagemodelcomefrom?
• We’llbuilditbycoun(ngthingsincorpusdata!
• Sta(s(cales(ma(onofmodelparameters
• Butwecan’tjustcountwholesentences
sentence count P(e)
Heshrankfromtoughdecisions. 1/49208 2.03e‐05
Heshrankfromimportantdecisions. 0/49208 0
Heshrankoftoughdecisions. 0/49208 0
Heshrankfromdecisionstough. 0/49208 0
toohigh!
toolow!
N‐gramlanguagemodels
• Instead,we’llbreakthingsintopieces
• Thisiscalledabigramlanguagemodel
• Wecanes(matebigramprobabili(esfromcorpus
P(Heshrankfromtoughdecisions)=P(He|•)×P(shrank|He)×P(from|shrank)×…×P(decisions|tough)
w1 w2 C(w1) C(w1w2) P(w2|w1)
• He 49208 978 0.0199
He shrank 53142 21 0.0004
shrank from 122 17 0.1393
from tough 18777 184 0.0098
Sta(s(caltransla(onmodels
• Noisychannelalsoneedstransla(onmodelP(f|e)
• Similarstrategy:breaksentencepairsintophrases
• Countco‐occurringpairsinalargeparallelcorpus
• (ButI’llskipthegorydetails…)
e f C(e) C(e,f) P(f|e)
heshrank illuirépugnait 17 6 0.3529
from de 27111 17855 0.6586
from des 27111 6434 0.2373
toughdecisions décisionsdifficiles 98 81 0.8265
Sta(s(calMTSystems
French BrokenEnglish
English
Sta(s(calAnalysis Sta(s(calAnalysis
J’aitrèsfaim Iamsohungry
WhathungerhaveI,HungryIamso,Iamsohungry,HaveIthathunger…
LanguageModelP(e)
Transla1onModelP(f|e)
DecodingalgorithmargmaxeP(f|e)P(e)
French/EnglishParallelTexts
Michellemabellesontlesmotsquivonttrèsbienensemble
Michellemabellesontlesmotsquivonttrèsbienensemble
Michelle,mabelle,sontlesmotsquivonttrèsbienensemble
Michellemabellesontlesmotsquivonttrèsbienensemble
Michellemabellesontlesmotsquivonttrèsbienensemble
Michelle,mybeau(ful,arewordsthatgotogetherwell
EnglishTexts
Michellemabellesontlesmotsquivonttrèsbienensemble
Michellemabellesontlesmotsquivonttrèsbienensemble
Manygreattradi(onsinartoriginatedintheartofoneofthefive…
Applica(onsofthenoisychannel
Thismodelcanbeappliedtomanydifferentproblems!
Channelmodelspeechproduc(onOCRtypingwithspellingerrorstransla(ngtoEnglish
LanguagemodelEnglishwordsEnglishwordsEnglishwordsEnglishwords
ê=argmaxeP(x|e)P(e)
(WidelyusedatGoogle,forexample)
IfyoulikeNLP/CompLing…
• learnJavaorPython(andplaywithJavaNLPorNLTK)• studylogic,probability,sta(s(cs,linearalgebra• getsomeexposuretolinguis(cs(LING1,…)• studyAIandmachinelearning(CS121,CS221,CS229)
• readJurafsky&Mar(norManning&Schütze
• CS124:FromLanguagetoInforma(on(Jurafsky)
• CS224N:NaturalLanguageProcessing(Manning)
• CS224S:SpeechRecogni(on&Synthesis(Jurafsky)• CS224U:NaturalLanguageProcessing(MacCartney)
Onemorefortheroad
cartoon from qwantz.com