ttic 31190: natural language processing › ~kgimpel › teaching › 31190 › lectures ›...

TTIC31190:NaturalLanguageProcessing

KevinGimpelWinter2016

Lecture16:MachineTranslation

andotherNLPApplications

Announcements• presentationswillactuallybe9minutesbecausewehavesomanytofitin

• Iwillpostguidelinesonthefinalprojectreport– thinkofitasashort(4-page)paper

• Iwillsendyouyourmidtermandassignment2gradestomorrow

Roadmap• classification• words• lexicalsemantics• languagemodeling• sequencelabeling• neuralnetworkmethodsinNLP• syntaxandsyntacticparsing• computationalsemantics• machinetranslation• otherNLPapplications

model score

BLEUscore

AfricanNationalCongress opposition sanction Zimbabwe

非国大反对制裁津巴布韦

Goldstandard:AfricanNationalCongressopposes

sanctionsagainstZimbabwe

model score

BLEUscore



opposition tosanctionsagainstZimbabweAfricanNationalCongress

predictedtranslation



model score

BLEUscore





learningmovestranslationsleftorrightinthisplot



model score

BLEUscore

“ideal”model



model score

BLEUscore



Issue:goldstandardtranslationisoften

unreachable bythemodel



Why?limitedtranslationrules,

freetranslations,noisydata

model score

BLEUscore

PerceptronLossgoldstandard

model score

BLEUscore

PerceptronLoss

modelprediction

goldstandard

model score

BLEUscore

HingeLossgoldstandard

cost-augmentedprediction

model score

BLEUscore

PerceptronLossforMT?(Collins,2002)

reference

modelprediction

model score

BLEUscore

RampLossMinimization

model score

BLEUscore


modelprediction

model score

BLEUscore


modelprediction

“fear”translation

model score

BLEUscore

“Fear”RampLoss(Doetal.,2008)

modelprediction


model score

BLEUscore

“Hope”RampLoss(McAllester &Keshet,2011; Liangetal.,2006)

modelprediction

model score

BLEUscore

modelprediction

“hope”translation

“Hope”RampLoss(McAllester &Keshet,2011; Liangetal.,2006)

model score

BLEUscore

“Hope-Fear”RampLoss(Chiangetal.,2008;2009;Cherry&Foster,2012;

Chiang,2012;Gimpel &Smith,2012)

“hope”translation


Experiments(Gimpel,2012)

Moses%BLEU Hiero %BLEU

MERT 35.9 37.0

FearRamp(awayfrombad) 34.9 34.2

HopeRamp(towardgood) 35.2 36.0

Hope-FearRamp(towardgood+awayfrombad) 35.7 37.0

averagesover8 testsetsacross3languagepairs

Whydoyouthinkthathoperampworksbetterthanfearramp?

Ithink:goingawayfromsomethingbaddoesnotnecessarilymeanthatyouaregoingtowardsomethinggood.

youmightbegoingtowardsomethingelsethat’sbad!

ClassificationFrameworkforMachineTranslation

• wehavealatentvariable,sothisbecomes:

• wemaximizeoverthelatentvariableANDtheoutput!• h couldbewordalignments,phrasesegmentations/alignments,synchronousCFGderivations,etc.

inference:solve_

• Forphrase-basedtranslation,searchover:– Segmentationsintophrases– Translationsforeachphrase– Orderingsofthetranslatedphrases

zimbabwe african national congresssanctions againstopposition to

ANC opposition sanction Zimbabwe


african national congress opposes sanctions against zimbabweReference:

• Forphrase-basedtranslation,searchover:– Segmentationsintophrases– Translationsforeachphrase– Orderingsofthetranslatedphrases


ANC opposition sanction Zimbabwe


african national congress opposes sanctions against zimbabweReference:

ThissearchproblemisNP-hard(Knight,1999)Approximatebeamsearchisusedinpractice



African National Congress opposessanctions against Zimbabwe

Reference translation:

Koehn et al. (2003)

Phrase-Based Machine Translation





1 非国大 / African National Congress 2 反对 / opposition to 3 反对 / is opposed to 4 制裁 / sanctions 5 制裁津巴布韦 /

sanctions against Zimbabwe...

Phrase TableKoehn et al. (2003)








Phrase Table

opposition to

African National Congress

2

1

Koehn et al. (2003)








Phrase Table

opposition toopposition to sanctions

against Zimbabwe


African National Congressis opposed to

2 5

3

1

Koehn et al. (2003)








Phrase Table

opposition to sanctionsagainst Zimbabwe


1opposition to

opposition to sanctionsagainst Zimbabwe


African National Congressis opposed to

2 5

3

1

Koehn et al. (2003)



zimbabwe african national congresssanctions onopposition to

zimbabwe african national congress sanctionsopposition to

zimbabweafrican national congress sanctions againstopposition to

zimbabweafrican national congress sanctions againstoppose

1

2

3

4

5

otherusefulinferencetasks:• findk-besttranslations

-11.8

-12.1

-12.4

-12.9

-13.5

Rank Score

typicallatticescontainupto1080 paths!(butnotallareuniquetranslations)

zimbabwe

zimbabwe

zimbabwe

african national congress



african national assemblyafricannationalcongress

sanctions against

sanctions on

sanctions against

sanctions

sanctions against

opposition to

opposition to

opposition tozimbabwe

is opposed to

otherusefulinferencetasks:• findphraselatticeoftranslations

NeuralNetworksandMachineTranslation

• currenttrendinMTresearchistouseneuralnetworksforeverything

• “neuralMT”typicallyreferstoapproachesthatonlyuseneuralnetworks

• butmostMTsystemscombinetraditionalphrase-basedmodelswithfeaturesbasedonneuralnetworks

ACL2014(bestpaperaward)

ACL2014

NeuralMT

EMNLP2013

EMNLP2014

NIPS2014

ICLR2015

OtherNLPTasksandApplications• coreference resolution• questionanswering• summarization• dialoguesystems

Coreference Resolution• determinewhichpiecesoftextrefertothesamereferent:– PresidentObamaselectedtendelegatesafterreceivingrecommendationsfromhis cabinetmembers.They spentalldaySaturdayworkingontheir recommendationsforhim.

OtherNLPTasksandApplications• coreference resolution• questionanswering

– factoidquestionanswering– machinecomprehension

• summarization• dialoguesystems

IBM’sWatson

ClassifyingQuestionsinto“LexicalAnswerTypes”

OtherNLPTasksandApplications• coreference resolution• questionanswering• summarization• dialoguesystems

AutomaticSummarization• givenadocument,produceasummaryofaprovidedlength

• vastmajorityofsystemsareextractive:theyextractcontentfromthedocument– thisissafer,sincethedocumentispresumablygrammatical

– butthislimitsapplicability• somework,especiallyrecently,thattriestodoabstractivesummarization– typicallybasedonintermediatesemanticrepresentationsorneuralnetworks

baseline=takefirst100wordsofdocument

regarding thefirsttwoyearsofDUC:

AAAI2005

MachineComprehensionCanamachinereadadocumentand

answerquestionsaboutit?

58

• 660fictionalstories,writtenata4th gradereadinglevel

• 4multiplechoicequestionsperstory

OncetherewasaboynamedFritzwholovedtodraw.Hedreweverything.Inthemorning,hedrewapictureofhiscerealwithmilk.Hispapasaid,“Don’tdrawyourcereal.Eatit!”Afterschool,Fritzdrewapictureofhisbicycle.Hisunclesaid,“Don'tdrawyourbicycle.Rideit!”…

59


WhatdidFritzdrawfirst?A)thetoothpasteB)hismamaC)cerealandmilkD)hisbicycle

60


WhatdidFritzdrawfirst?A)thetoothpasteB)hismamaC)cerealandmilkD)hisbicycle

61


WhatdidFritzdrawfirst?A)thetoothpasteB)hismamaC)cerealandmilkD)hisbicycleE)everything

62

63

JamestheTurtlewasalwaysgettingintrouble.…

Whatisthenameofthetroublemakingturtle?A)FriesB)PuddingC)JamesD)Jane

• Somequestionsaremucheasier

• Simplewordoverlapbaselinegets63%correct

64

institution year accuracy(%)

TTI-Chicago 2015 69.9

CarnegieMellon 2015 67.8

UniversityCollegeLondon 2015 66.0

MIT 2015 63.8

Microsoft Research 2013 63.3

MCTest Leaderboard

Oursystemusesseveraltypesofautomaticlinguisticanalysis:

– dependencyparsing– framesemanticparsing– coreference– wordembeddings

65


– dependencyparsing

66



67

outputofStanforddependencyparser



68

FritzdrawXfirst



69

FritzdrawXfirst

FritzdrawthetoothpastefirstFritz draw his mama firstFritz draw cereal and milk firstFritz draw his bicycle first


– dependencyparsing– framesemanticparsing

70



71

outputofCarnegieMellonframesemanticparser



72



73


– dependencyparsing– framesemanticparsing– coreference

74


– dependencyparsing– framesemanticparsing– coreference

75

outputofStanfordcoreference resolutionsystem


– dependencyparsing– framesemanticparsing– coreference– wordembeddings

76

OncetherewasaboynamedFritz wholovedtodraw.Hedreweverything.Inthemorning,hedrewapictureofhiscerealwithmilk.Hispapasaid,“Don’tdrawyourcereal.Eatit!”…WhatdidFritzdrawfirst?

77

OncetherewasaboynamedFritz wholovedtodraw.Hedreweverything.Inthemorning,hedrewapictureofhiscerealwithmilk.Hispapasaid,“Don’tdrawyourcereal.Eatit!”…WhatdidFritzdrawfirst?

transformedquestion(usingdependencyparsing):Fritzdrawcerealandmilkfirst

Fritz≈he (coreference,framesemantics)draw≈drew (wordembeddings,framesemantics)withmilk≈andmilk (wordembeddings)

78

RemovingFeaturesOneataTime

69.9

67.667.9

68.4 68.3

64

65

66

67

68

69

70

71

72

allfeatures

removedependencyparsing

removeframesemantics

removecoreference

removeembeddings

Accuracy

ttic 31190: natural language processing › ~kgimpel › teaching › 31190 › lectures ›...

Documents