algorithms for nlp - cs.cmu.edutbergkir/11711fa17/fa17 11-711 lecture 20... · under the new...
TRANSCRIPT
MachineTranslationTaylorBerg-Kirkpatrick– CMU
Slides:DanKlein– UCBerkeley
AlgorithmsforNLP
MachineTranslation
MachineTranslation:Examples
LevelsofTransfer
Word-LevelMT:Examples
§ lapolitiquedelahaine. (Foreign Original)§ politicsofhate. (ReferenceTranslation)§ thepolicyofthehatred. (IBM4+N-grams+Stack)
§ nousavonssignéleprotocole. (Foreign Original)§ wedidsignthememorandumofagreement. (ReferenceTranslation)§ wehavesignedtheprotocol. (IBM4+N-grams+Stack)
§ oùétaitleplansolide? (Foreign Original)§ butwherewasthesolidplan? (ReferenceTranslation)§ wherewastheeconomicbase? (IBM4+N-grams+Stack)
PhrasalMT:Examples
Metrics
MT:Evaluation§ Humanevaluations:subjectmeasures,
fluency/adequacy
§ Automaticmeasures:n-grammatchtoreferences§ NISTmeasure:n-gramrecall(workedpoorly)§ BLEU:n-gramprecision(noonereallylikesit,but
everyoneusesit)§ Lotsmore:TER,HTER,METEOR,…
§ BLEU:§ P1=unigramprecision§ P2,P3,P4=bi-,tri-,4-gramprecision§ WeightedgeometricmeanofP1-4§ Brevitypenalty(why?)§ Somewhathardtogame…§ Magnitudeonlymeaningfulonsamelanguage,corpus,
numberofreferences,probablyonlywithinsystemtypes…
AutomaticMetricsWork(?)
SystemsOverview
Corpus-BasedMT
Modeling correspondences between languages
Sentence-aligned parallel corpus:
Yo lo haré mañanaI will do it tomorrow
Hasta prontoSee you soon
Hasta prontoSee you around
Yo lo haré pronto I will do it soon
I will do it around
See you tomorrow
Machine translation system:
Model of translation
Phrase-BasedSystemOverview
Sentence-aligned corpus
cat ||| chat ||| 0.9 the cat ||| le chat ||| 0.8dog ||| chien ||| 0.8 house ||| maison ||| 0.6 my house ||| ma maison ||| 0.9language ||| langue ||| 0.9 …
Phrase table(translation model)Word alignments
Many slides and examples from Philipp Koehn or John DeNero
WordAlignment
WordAlignment
WordAlignment
What is the anticipated cost of collecting fees under the new proposal?
En vertu des nouvelles propositions, quel est le coût prévu de perception des droits?
x zWhat
is the
anticipatedcost
ofcollecting
fees under
the new
proposal?
En vertu delesnouvelles propositions, quel est le coût prévu de perception de les droits?
UnsupervisedWordAlignment§ Input:abitext:pairsoftranslatedsentences
§ Output:alignments:pairsoftranslatedwords
§ Whenwordshaveuniquesources,canrepresentasa(forward)alignmentfunctiona fromFrenchtoEnglishpositions
nous acceptons votre opinion .
we accept your view .
1-to-ManyAlignments
EvaluatingModels§ Howdowemeasurequalityofaword-to-wordmodel?
§ Method1:useinanend-to-endtranslationsystem§ Hardtomeasuretranslationquality§ Option:humanjudges§ Option:referencetranslations(NIST,BLEU)§ Option:combinations(HTER)§ Actually,nooneusesword-to-wordmodelsaloneasTMs
§ Method2:measurequalityofthealignmentsproduced§ Easytomeasure§ Hardtoknowwhatthegoldalignmentsshouldbe§ Oftendoesnotcorrelatewellwithtranslationquality(likeperplexityinLMs)
AlignmentErrorRate§ AlignmentErrorRate
Sure align.
Possible align.
Predicted align.
=
=
=
IBMModel1:Allocation
IBMModel1(Brown93)§ Alignments:ahiddenvectorcalledanalignment specifieswhichEnglish
sourceisresponsibleforeachFrenchtargetword.
A:
IBMModels1/2
Thank you , I shall do so gladly .
1 3 7 6 9
1 2 3 4 5 76 8 9
Model ParametersTransitions: P( A2 = 3)Emissions: P( F1 = Gracias | EA1 = Thank )
Gracias , lo haré de muy buen grado .
8 8 88
E:
F:
ProblemswithModel1
§ There’sareasontheydesignedmodels2-5!
§ Problems:alignmentsjumparound,aligneverythingtorarewords
§ Experimentalsetup:§ Trainingdata:1.1Msentences
ofFrench-Englishtext,CanadianHansards
§ Evaluationmetric:alignmenterrorRate(AER)
§ Evaluationdata:447hand-alignedsentences
IntersectedModel1
§ Post-intersection:standardpracticetotrainmodelsineachdirectionthenintersecttheirpredictions[OchandNey,03]
§ Secondmodelisbasicallyafilteronthefirst§ Precisionjumps,recalldrops§ Endupnotguessinghard
alignments
Model P/R AERModel 1 E®F 82/58 30.6Model 1 F®E 85/58 28.7Model 1 AND 96/46 34.8
JointTraining?§ Overall:
§ Similarhighprecisiontopost-intersection§ Butrecallismuchhigher§ Moreconfidentaboutpositingnon-nullalignments
Model P/R AERModel 1 E®F 82/58 30.6Model 1 F®E 85/58 28.7Model 1 AND 96/46 34.8Model 1 INT 93/69 19.5
IBMModel2:GlobalMonotonicity
MonotonicTranslation
Le Japon secoué par deux nouveaux séismes
Japan shaken by two new quakes
LocalOrderChange
Le Japon est au confluent de quatre plaques tectoniques
Japan is at the junction of four tectonic plates
IBMModel2§ Alignmentstendtothediagonal(broadlyatleast)
§ Otherschemesforbiasingalignmentstowardsthediagonal:§ Relativevsabsolutealignment§ Asymmetricdistances§ Learningafullmultinomialoverdistances
EMforModels1/2
§ Model1Parameters:Translationprobabilities(1+2)Distortionparameters(2only)
§ Startwith uniform,including§ Foreachsentence:
§ ForeachFrenchpositionj§ CalculateposterioroverEnglishpositions
§ (orjustusebestsinglealignment)§ Incrementcountofwordfj withwordei bytheseamounts§ Alsore-estimatedistortionprobabilitiesformodel2
§ Iterateuntilconvergence
Example
HMMModel:LocalMonotonicity
PhraseMovement
Des tremblements de terre ont à nouveau touché le Japon jeudi 4 novembre.
On Tuesday Nov. 4, earthquakes rocked Japan once again
A:
TheHMMModel
Thank you , I shall do so gladly .
1 3 7 6 9
1 2 3 4 5 76 8 9
Model ParametersTransitions: P( A2 = 3 | A1 = 1)Emissions: P( F1 = Gracias | EA1 = Thank )
Gracias , lo haré de muy buen grado .
8 8 88
E:
F:
TheHMMModel
§ Model2preferredglobalmonotonicity§ Wewantlocalmonotonicity:
§ Mostjumpsaresmall
§ HMMmodel(Vogel96)
§ Re-estimateusingtheforward-backwardalgorithm§ Handlingnullsrequiressomecare
§ Whatarewestillmissing?
-2 -1 0 1 2 3
HMMExamples
AERforHMMs
Model AERModel 1 INT 19.5HMM E®F 11.4HMM F®E 10.8HMM AND 7.1HMM INT 4.7GIZA M4 AND 6.9
Models3,4,and5:Fertility
IBMModels3/4/5
Mary did not slap the green witch
Mary not slap slap slap the green witch
Mary not slap slap slap NULL the green witch
n(3|slap)
Mary no daba una botefada a la verde bruja
Mary no daba una botefada a la bruja verde
P(NULL)
t(la|the)
d(j|i)
[from Al-Onaizan and Knight, 1998]
Examples:TranslationandFertility
Example:Idioms
il hoche la tête
he is nodding
Example:Morphology
SomeResults§ [OchandNey03]
PhraseMovement
TheHMMModel