neural machine transla#on

NeuralMachineTransla/on

ThangLuongLecture@CS224D

Spring2016

(SpecialthankstoChrisManningforfeedback!)

7billionpeople,7000languages

5/19/16 ThangLuong-NeuralMachineTranslaHon 2

Auniversaltranslator

(TheBabelFishfrom“theHitchhiker'sGuidetotheGalaxy”)

Machinevs.HumanTransla/on

“Nevertheless,withinthedisciplineofmachinelearning,aspecializa8oncalleddeeplearninghasbeendeveloped.Itsdis8nguishingfeatureisthatitisinspiredbyneurobiology.Deeplearningdealswithcomputa8onalelementswhichallowanetworkofar8ficialneuronstolearnamodelofthehumanbrain.”

Faithfultransla/on

•  GrammaHcallyincorrect.

Faithfultransla/on

•  Badwordchoices.

Faithfultransla/on

•  Badsentencestructures.

“However,inmachinelearningaspecializa8oncalleddeeplearninghasemerged.Itcanberecognizedbyitsdis8nc8veneurobiologicalinfluence.Deeplearningiscenteredaroundnetworksofar8ficialneuronswhichcanlearnmodelsofthehumanbrain.”

Fluenttransla/on

Abiggap!5/19/16 ThangLuong-NeuralMachineTranslaHon 7

HowhasMTevolved?

Phrase-basedMT

Sheloves

Elleaime

leschats

mignons(Brownetal.,1993;Koehnetal.,2003;Och&Ney,2004)

•  Breaksentencesintochunks.•  Transla8onmodel:lookupphrasetranslaHons.•  Languagemodel:Hephrasestogether.

Translatelocally LMusesonlytargetwords5/19/16 ThangLuong-NeuralMachineTranslaHon 9

JointNeuralLanguageModel

•  CondiHonedonsourcewords(Devlinetal.,2014)•  SHlltranslatelocally.

…alléspourunepromenadelelongdela

WewentforastrollalongtheSouthBank

walk along bank(river)

MTsystemsbecomemorecomplex!5/19/16 ThangLuong-NeuralMachineTranslaHon 10

NeuralMachineTransla/ontotherescue!

Let’sfindout!

•  Sequence-to-sequence:translateglobally.•  End-to-end:simple&generalizable.

(Sutskeveretal.,2014;Choetal.,2014)

Iamastudent Jesuisétudiant

Outline

•  BasicNMT– RNNRecap.– Encoder-Decoder.– Training.– TesHng.

•  AdvancedNMT

RecurrentNeuralNetworks(RNNs)

I am a studentinput:

(PictureadaptedfromAndrejKarparthy)

RecurrentNeuralNetworks(RNNs)

I am a studentinput:

RNNstorepresentsequences!

ht-1 ht

(PictureadaptedfromAndrejKarparthy)5/19/16 ThangLuong-NeuralMachineTranslaHon 14

NeuralMachineTransla/on(NMT)

am a student _ Je suis étudiant

Je suis étudiant _

•  RecurrentNeuralNetworks:– ModelP(target|source)directly.– Canbetrainedend-to-end.

Je suis étudiant _

Encoder

Je suis étudiant _

Encoder

Boundarymarker

Je suis étudiant _

Encoder

Je suis étudiant _

Encoder

Je suis étudiant _

Encoder Decoder

Boundarymarker

WordEmbeddings

Je suis étudiant _

•  Oneforeachlanguage:canlearnfromscratch.

Sourceembeddings

Targetembeddings

Je suis étudiant _

RecurrentConnec/onsIniHalstates

•  Olensetto0.5/19/16 ThangLuong-NeuralMachineTranslaHon 26

Je suis étudiant _

RecurrentConnec/ons

Encoder1stlayer

•  Different:{1stlayer,2ndlayer}x{encoder,decoder}.

Je suis étudiant _

RecurrentConnec/ons

Encoder2ndlayer

Je suis étudiant _

RecurrentConnec/ons

Decoder1stlayer

Je suis étudiant _

RecurrentConnec/ons

Decoder2ndlayer

RecurrentUnits

•  Vanilla:

•  LSTM:

C’mon,it’sbeenaroundfor20years!

Vanishinggradientproblem!

SoTmax:vectors↦categories

Je suis étudiant _

Solmaxparameters

Je suis étudiant _

Targethiddenstate

SoTmax:vectors↦categories

Je suis étudiant _

•  Hiddenstates↦scores↦probabiliHes.

Scores

P(Je|…)exp&normalize

TrainingLoss

•  MaximizeP(target|source)

am a student _ Je suis étudiantI

Je suis étudiant _

TrainingLoss

-log P(Je)

-log P(suis)

•  Sumofallindividuallosses5/19/16 ThangLuong-NeuralMachineTranslaHon 35

TrainingLoss

-log P(Je)

-log P(suis)

•  Sumofallindividuallosses5/19/16 ThangLuong-NeuralMachineTranslaHon 36

TrainingLoss

•  Sumofallindividuallosses

-log P(étudiant)

TrainingLoss

•  Sumofallindividuallosses

-log P(_)

Backpropaga/onThroughTime

-log P(_)

Initto0

-log P(étudiant)

-log P(suis)

RNNgradientsareaccumulated.5/19/16 ThangLuong-NeuralMachineTranslaHon 44

Trainingvs.Tes/ng

•  Training– CorrecttranslaHonsareavailable.

•  Tes8ng– Onlysourcesentencesaregiven.

Je suis étudiant _

I5/19/16 ThangLuong-NeuralMachineTranslaHon 45

Tes/ng

•  Feedthemostlikelyword5/19/16 ThangLuong-NeuralMachineTranslaHon 46

Tes/ng

•  Feedthemostlikelyword

Tes/ng

Simplebeam-searchdecoders!5/19/16 ThangLuong-NeuralMachineTranslaHon 50

SOTASMT(Durrani+,2014)

AvgSMT(Schwenk,2014)

NMT(Sutskever+,2014)

SMT+NMTrescore(Sutskever+,2014)

English-FrenchWMT’14results

SOTASMT(Durrani+,2014)

AvgSMT(Schwenk,2014)

NMT(Sutskever+,2014)

SMT+NMTrescore(Sutskever+,2014)

2decadesofresearch

1-2yearsofresearch

Encoder-decoderVariants

Encoder Decoder

(Sutskeveretal.,2014)MyNMTmodels DeepLSTM DeepLSTM

(Choetal.,2014)(Bahdanauetal.,2015)

(Jeanetal.,2015)

(BidirecHonal)GRU GRU

(Kalchbrenner&Blunsom,2013) CNN (InverseCNN)

Next,advancedNMT!5/19/16 ThangLuong-NeuralMachineTranslaHon 53

Deepfriedbaby Meatmusclestupidbeansprouts

Break/me:whenMTfails…

Saleofchickenmurder Gobacktowardyourbehind

Limita/ons

•  #1:thevocabularysizeproblem– Goal:extendthevocabularycoverage.

•  #2:thesentencelengthproblem– Goal:translatelongsentencesbever.

•  #3:thelanguagecomplexityproblem– Goal:handlemorelanguagevariaHons.

AdvancingNMT

•  #1:thevocabularysizeproblem– Sol:“copy”mechanism.

•  #2:thesentencelengthproblem– Sol:avenHonmechanism.

•  #3:thelanguagecomplexityproblem– Sol:character-leveltranslaHon.

The<unk>porHcoin<unk>

Le<unk><unk>de<unk>

ecotax Pont-de-Buis

por8que écotaxe Pont-de-Buis

Je suis étudiant _

CVvs.NLP

ComputerVision

1Kcategories 1Mcategories

Thecatsatona

#1TheVocabularySizeProblem

•  WordgeneraHonproblem– Vocabsaremodest:50K.– Simplesolmax:GPUfriendliness.

Je suis étudiant _

The<unk>porHcoin<unk>Le<unk><unk>de<unk>

TheecotaxporHcoinPont-de-BuisLeporHqueécotaxedePont-de-Buis

•  Propose“copy”mechanismsfor<unk>.

•  Simple&effecHve– TreatanyNMTasablackbox.– Annotatetrainingdata.– Post-processtranslaHons.

ThangLuong*,IlyaSutskever*,QuocLe*,OriolVinyals,andWojciechZaremba.AddressingtheRareWordProbleminNeuralMachineTransla>on.ACL2015.

SOTAforEnglish-FrenchtranslaHon.

Ourapproach–trainingannota>on

•  AddrelaHveposiHons.

“AvenHon”forrarewords

TheecotaxporHcoinPont-de-Buis

LeporHqueécotaxedePont-de-Buis

•  Learnalignments.

Leunk1unk-1deunk0

Ourapproach–post-process

Testsentence

Transla8on

LeporHqueunk-1deunk0

ecotax Pont-de-Buis

Testsentence

Transla8on

ecotax Pont-de-Buis

Post-editTransla8on

Dic/onarytranslaHon

Testsentence

Transla8on

ecotax Pont-de-Buis

Iden/tycopy

Post-editTransla8on

Orthogonaltolarge-vocabtechniques5/19/16 ThangLuong-NeuralMachineTranslaHon 63

EffectsofTransla/ngRareWords

Sentencesorderedbyaveragefrequencyrank

Durranietal.(37.0)

Sutskeveretal.(34.8)

Thiswork(37.5)

FirstSOTANMTsystem!

Sampletransla/ons

•  Predictwelllong-distancealignments.– Correct:cataractvs.cataracte.

source AnaddiHonal2600operaHonsincludingorthopedicandcataractsurgerywillhelpclearabacklog.

human 2600opéraHonssupplémentaires,notammentdansledomainedelachirurgieorthopédiqueetdelacataracte,aiderontàravraperleretard.

trans Enoutre,unk1opéraHonssupplémentaires,dontlachirurgieunk5etlaunk6,permevrontderésorberl'arriéré.

trans+unk

Enoutre,2600opéraHonssupplémentaires,dontlachirurgieorthopédiquesetlacataracte,permevrontderésorberl'arriéré.

Sampletransla/ons

•  Translatewelllongsentences.– Correct:JPMorganvs.JPMorgan.

sourceThistrader,RichardUsher,lelRBSin2010andisunderstandtohavebegivenleavefromhiscurrentposiHonasEuropeanheadofforexspottradingatJPMorgan.

humanCetrader,RichardUsher,aquivéRBSen2010etauraitétémissuspendudesonpostederesponsableeuropéendutradingaucomptantpourlesdeviseschezJPMorgan.

transCeunk0,Richardunk0,aquivéunk1en2010etacomprisqu'ilestautoriséàquiversonposteactuelentantqueleadereuropéendumarchédespointsdeventeauunk5.

trans+unk

Cenégociateur,RichardUsher,aquivéRBSen2010etacomprisqu'ilestautoriséàquiversonposteactuelentantqueleadereuropéendumarchédespointsdeventeauJPMorgan.

Sampletransla/ons

•  IncorrectalignmentpredicHon:was–étaitvs.abandonnait.

source ButconcernshavegrownalerMrMazangawasquotedassayingRenamowasabandoningthe1992peaceaccord.

human Maisl'inquiétudeagrandiaprèsqueM.MazangaadéclaréquelaRenamoabandonnaitl'accorddepaixde1992.

trans MaislesinquiétudessesontaccruesaprèsqueM.unkpos3adéclaréquelaunk3unk3l'accorddepaixde1992.

trans+unk

MaislesinquiétudessesontaccruesaprèsqueM.MazangaadéclaréquelaRenamoétaitl'accorddepaixde1992.

AdvancingNMT

Le<unk><unk>de<unk>

ecotax Pont-de-Buis

Je suis étudiant _

#2TheSentenceLengthProblem

Problem:sentencemeaningisrepresentedbyafixed-dimensionalvector.

Je suis étudiant _

•  TranslaHonqualitydegradeswithlongsentences.

Youcan'tcramthemeaningofawhole%&!$#sentenceintoasingle$&!#*vector!

(AdaptedfromKyungHuynCho’talk)

Amen/onMechanism

•  SoluHon:randomaccessmemory– Retrieveasneeded.

Je suis étudiant _

Poolofsourcestates

DzmitryBahdanau,KyungHuynCho,andYoshuaBengio.NeuralMachineTransla>onbyJointlyLearningtoTranslateandAlign.ICLR2015.

WithavenHonWithoutavenHon

Amen/onMechanism

am a student _ Je

Attention Layer

Context vector

Asimplifiedversionof(Bahdanauetal.,2015)5/19/16 ThangLuong-NeuralMachineTranslaHon 73

•  Comparetargetandsourcehiddenstates.

Amen/onMechanism–Scoring

am a student _ Je

Attention Layer

Context vector

am a student _ Je

Attention Layer

Context vector

am a student _ Je

Attention Layer

Context vector

am a student _ Je

Attention Layer

Context vector

13 5 1

•  Convertintoalignmentweights.

Amen/onMechanism–Normaliza>on

am a student _ Je

Attention Layer

Context vector

0.10.3 0.5 0.1

am a student _ Je

Context vector

•  Buildcontextvector:weightedaverage.

Amen/onMechanism–Contextvector

am a student _ Je

Context vector

•  Computethenexthiddenstate.

Amen/onMechanism–Hiddenstate

am a student _ Je

Context vector

•  Predictthenextword.

Amen/onMechanism–Predict

•  ExaminevariousavenHonmechanisms:

ThangLuong,HieuPham,andChrisManning.Effec>veApproachestoAGen>on-basedNeuralMachineTransla>on.EMNLP2015.

Global:allsourcestates. Local:subsetofsourcestates.

SOTAforEnglish-GermantranslaHon.

TranslateLongSentences

10 20 30 40 50 60 7010

Sent Lengths

��

ours, no attn (BLEU 13.9)ours, local−p attn (BLEU 20.9)ours, best system (BLEU 23.0)WMT’14 best (BLEU 20.7)Jeans et al., 2015 (BLEU 21.6)

NoAvenHon

AvenHon

SampleEnglish-Germantransla/ons

•  Translatenamescorrectly.

source OrlandoBloomandMirandaKerrsHllloveeachother

human OrlandoBloomundMirandaKerrliebensichnochimmer

bestOrlandoBloomundMirandaKerrliebeneinandernochimmer.

baseOrlandoBloomundLucasMirandaliebeneinandernochimmer.

•  Translateadoubly-negatedphrasecorrectly•  Failtotranslate“passengerexperience”.

sourceWeʹrepleasedtheFAArecognizesthatanenjoyablepassengerexperienceisnotincompa>blewithsafetyandsecurity,saidRogerDow,CEOoftheU.S.TravelAssociaHon.

humanWirfreuenuns,dassdieFAAerkennt,dasseinangenehmesPassagiererlebnisnichtimWider-spruchzurSicherheitsteht,sagteRogerDow,CEOderU.S.TravelAssociaHon.

best Wirfreuenuns,dassdieFAAanerkennt,dasseinangenehmesistnichtmitSicherheitundSicherheitunvereinbarist,sagteRogerDow,CEOderUS-die.

baseWirfreuenunsuberdie<unk>,dassein<unk><unk>mitSicherheitnichtvereinbaristmitSicherheitundSicherheit,sagteRogerCameron,CEOderUS-<unk>.

sourceWeʹrepleasedtheFAArecognizesthatanenjoyablepassengerexperienceisnotincompa>blewithsafetyandsecurity,saidRogerDow,CEOoftheU.S.TravelAssociaHon.

humanWirfreuenuns,dassdieFAAerkennt,dasseinangenehmesPassagiererlebnisnichtimWider-spruchzurSicherheitsteht,sagteRogerDow,CEOderU.S.TravelAssociaHon.

best Wirfreuenuns,dassdieFAAanerkennt,dasseinangenehmesistnichtmitSicherheitundSicherheitunvereinbarist,sagteRogerDow,CEOderUS-die.

baseWirfreuenunsuberdie<unk>,dassein<unk><unk>mitSicherheitnichtvereinbaristmitSicherheitundSicherheit,sagteRogerCameron,CEOderUS-<unk>.

•  Translateadoubly-negatedphrasecorrectly•  Failtotranslate“passengerexperience”.5/19/16 ThangLuong-NeuralMachineTranslaHon 86

TEDtalk,English-German

26.18 26.02 24.9622.51

21.84 22.67 23.42

Stanford EdinburghKarlsruheHeidelberg PJAIT

HUMANTER

ThangLuongandChrisManning.StanfordNeuralMachineTransla>onSystemsforSpokenLanguageDomain.IWSLT2015.

Winning

AdvancingNMT

Le<unk><unk>de<unk>

ecotax Pont-de-Buis

Je suis étudiant _

#3Therarewordproblem

•  “Copying”mechanismsarenotsufficient.– Differentalphabets:Christopher↦Kryštof– MulH-wordalignment:Solarsystem↦Sonnensystem

•  Needtohandlelarge,openvocabulary– Richmorphology:nejneobhospodařovávatelnějšímu

(“totheworstfarmableone”)–  Informalspelling:gooooooodmorning!!!!!

Beabletogenerateatthecharacterlevel.5/19/16 ThangLuong-NeuralMachineTranslaHon 89

Recentcharacter-levelNMT

•  UnsaHsfactoryperformance–  (WangLing,IsabelTrancoso,ChrisDyer,AlanBlack,arXiv2015)

•  IncompletesoluHon– Decoderonly(JunyoungChung,KyunghyunCho,YoshuaBengio.arXiv2016).

•  Abest-of-both-worldsarchitecture:– Translatemostlyatthewordlevel– Onlygothecharacterlevelwhenneeded.

•  AddiHonal+2.1↦+11.4BLEUimprovement.

ThangLuongandChrisManning.AchievingOpenVocabularyNeuralMachineTransla>onwithHybridWord-CharacterModels.Insubmission,ACL2016.

SOTAforEnglish-CzechtranslaHon.

HybridNMT

Word-level(4layers)

End-to-endtraining8-stackingLSTMlayers.

SourceRepresenta/ons

•  On-the-flyembeddings.– Zero-init,batchcomputaHon.

TargetGenera/on Initwithword

hiddenstates.

Purposely,use<unk>tomakedecodingeasier.

English-CzechWMT’15Results

Systems BLEUWinningentry(Bojar&Tamchyna,2015) 18.8

Exis8ngword-levelNMT(Jeanetal.,2015)

Singlemodel 15.7

Ensemble4models 18.3Largevocab+unkreplace

30xdata3systems

Singlemodel 15.7

Ensemble4models 18.3

Ourcharacter-basedNMT

Singlemodel(600-stepbackprop) 15.9

Largevocab+unkreplace

30xdata3systems

•  Purelycharacter-based:slowbutpromising!

Singlemodel 15.7

OurhybridNMT

Singlemodel 19.6

30xdata3systems

NewSOTA!

Singlemodel 15.7

OurhybridNMT

Singlemodel 19.6

30xdata3systems

BemerSOTA!

EffectsofVocabularySizes

02468101214161820

1K 10K 20K 50K

VocabularySize

Word Word+unkreplace Hybrid

02468101214161820

1K 10K 20K 50K

VocabularySize

AddiHonalgainsof+2.1↦+11.4BLEU

+4.5+3.5

02468101214161820

1K 10K 20K 50K

VocabularySize

Small-vocabhybrid=Large-vocabword5/19/16 ThangLuong-NeuralMachineTranslaHon 101

RareWordEmbeddings

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

acceptable

acknowledgement

admissionadmitadmittanceadmitting

advance

antagonistchoose

chooses

connect

decide

developdevelopments

evidentlyexplicit

founder

governance

immobileimmoveable

impossible

insensitiveinsufficiency

linkmanagement

necessary

nominated

noticeable

obvious

perceptible

possible

practice

satisfactory

sponsor

unacceptable

unaffected

uncomfortableunsatisfactory

unsuitable

antagonize

cofounderscompanionships

disrespectful

heartlesslyheartlessness

illiberal

impossibilitiesinabilities

loveless

narrowïmindednarrowïmindedness nonconscious

regretful

spiritless

unattainableness

unconcern

uncontroversial

unfeatheredunfledged

ungraceful

unrealizable

unsighted

untrustworthy

wholeheartedness

•  Word&character-basedembeddings.5/19/16 ThangLuong-NeuralMachineTranslaHon 102

RareWordEmbeddings

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

acceptable

acknowledgement

advance

antagonistchoose

chooses

connect

decide

developdevelopments

evidentlyexplicit

founder

governance

immobileimmoveable

impossible

linkmanagement

necessary

nominated

noticeable

obvious

perceptible

possible

practice

satisfactory

sponsor

unacceptable

unaffected

unsuitable

antagonize

disrespectful

illiberal

loveless

regretful

spiritless

unattainableness

unconcern

uncontroversial

ungraceful

unrealizable

unsighted

untrustworthy

wholeheartedness

RareWordEmbeddings

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

acceptable

acknowledgement

advance

antagonistchoose

chooses

connect

decide

developdevelopments

evidentlyexplicit

founder

governance

immobileimmoveable

impossible

linkmanagement

necessary

nominated

noticeable

obvious

perceptible

possible

practice

satisfactory

sponsor

unacceptable

unaffected

unsuitable

antagonize

disrespectful

illiberal

loveless

regretful

spiritless

unattainableness

unconcern

uncontroversial

ungraceful

unrealizable

unsighted

untrustworthy

wholeheartedness

SampleEnglish-Czechtransla/ons

source TheauthorStephenJayGoulddied20yearsalerdiagnosis.human AutorStephenJayGouldzemřel20letpodiagnóze.

char AutorStepherStepherzemřel20letpodiagnóze.

wordAutorStephenJay<unk>zemřel20letpo<unk>.

AutorStephenJayGouldzemřel20letpopo.

hybridAutorStephenJay<unk>zemřel20letpo<unk>.

AutorStephenJayGouldzemřel20letpodiagnóze.

PerfecttranslaHon!

•  Char-based:wrongnametranslaHon.•  Char-based:wrongnametranslaHon.

•  Word-based:incorrectalignment•  Char-based:wrongnametranslaHon.

•  Char-based&hybrid:correcttranslaHonofdiagnóze.•  Char-based:wrongnametranslaHon.

source AstheReverendMar>nLutherKingJr.saidfiTyyearsago:human Jakpředpadesá/letyřeklreverendMar/nLutherKingJr.:

char JakoreverendMar/nLutherkrálříkalpředpadesá>lety:

wordJakřeklreverendMarHn<unk>King<unk>předpadesáHlety:

JakřeklreverendMar/nLutherKingřeklpředpadesá>lety:

hybridJakřeklreverendMarHn<unk>King<unk>předpadesáHlety:

Jakpředpadesá/letyřeklreverendMar/nLutherKingJr.:

Correctreordering!

source AstheReverendMar>nLutherKingJr.saidfiTyyearsago:human Jakpředpadesá/letyřeklreverendMar/nLutherKingJr.:

char JakoreverendMar/nLutherkrálříkalpředpadesá>lety:

wordJakřeklreverendMarHn<unk>King<unk>předpadesáHlety:

JakřeklreverendMar/nLutherKingřeklpředpadesá>lety:

hybridJakřeklreverendMarHn<unk>King<unk>předpadesáHlety:

Jakpředpadesá/letyřeklreverendMar/nLutherKingJr.:

•  Char-based:“král”means“king”.•  Char-based:wrongnametranslaHon.5/19/16 ThangLuong-NeuralMachineTranslaHon 110

source Her11-year-olddaughter,ShaniBart,saiditfeltalivlebitweird

human Jejíjedenác/letádceraShaniBartováprozradila,žejetotrochuzvláštní

char Jejíjedenác/letádcera,ShaniBartová,říkala,žecí�trochudivně

wordJejí<unk>dcera<unk><unk>řekla,žejetotrochudivné

Její11-year-olddceraShani,řekla,žejetotrochudivné

hybridJejí<unk>dcera,<unk><unk>,řekla,žejeto<unk><unk>

Jejíjedenác/letádcera,GrahamBart,řekla,žecí�trochudivný

Generatecomplexwords!

•  Word-based:idenHtycopyfails.•  Char-based:wrongnametranslaHon.5/19/16 ThangLuong-NeuralMachineTranslaHon 112

•  Hybrid:translatenamesincorrectly.•  Char-based:wrongnametranslaHon.5/19/16 ThangLuong-NeuralMachineTranslaHon 113

WehaveadvancedNMT

•  #1:thevocabularysizeproblem–  Sol:“copy”mechanism.–  SOTAEnglish-French

•  #2:thesentencelengthproblem–  Sol:avenHonmechanism.–  SOTAEnglish-German

•  #3:thelanguagecomplexityproblem–  Sol:character-leveltranslaHon.–  SOTAEnglish-Czech

Le<unk><unk>de<unk>

ecotax Pont-de-Buis

Je suis étudiant _

NMT&beyond

•  UnsupervisedlearningforNMT– UHlizemonolingualdata.

•  Long-contextNMT–  TranslaHnganarHcle/abook.–  SmarteravenHon,longersequences.

•  MulH-modalLanguageUnderstandingSystem– MulH-lingualtranslaHon+speechrecogniHon+more– MulH-tasklearning

Thankyou!

I’mdone.Butifyouarecurious,readon!

•  CanweuHlizeallsequence-to-sequencedata?

•  CanwecompressNMTformobiledevices?

#4ForthefutureofNMT

MachinetranslaHon ConsHtuentparsing

Ourwork

•  MulH-tasklearning:– MachinetranslaHon –ConsHtuentparsing–  ImagecapHongeneraHon–Unsupervisedlearning

•  TranslaHonimprovement:upto+1.5BLEU.

•  State-of-the-artinconsHtuentparsing.

ThangLuong,QuocLe,IlyaSutskever,OriolVinyals,andLukaszKaiser.Mul>-tasksequencetosequencelearning.ICLR2016.

Many-to-one:shareddecoder

English (unsupervised)

Image (captioning) English

German (translation)

1616.517

17.518

18.519

German-EnglishTransla/on(BLEU)

Big(transla>on)+Medium(cap>on)

(Luongetal.,2015)

Single

+capHon(0.1x)

+capHon(0.05x)

+capHon(0.01x)

+0.7BLEU

1313.514

14.515

15.516

English-GermanTransla/on(BLEU)

Big(transla>on)+Small(PTBparsing)

(Luongetal.,2015)

Single

+parsing(1x)

+parsing(0.1x)

+parsing(0.01x)

+1.5BLEU

English (unsupervised)

German (translation)

Tags (parsing)English

One-to-many:sharedencoder

MixingraHo5/19/16 ThangLuong-NeuralMachineTranslaHon 120

Ourwork

AbigailSee*,ThangLuong*,andChrisManning.CompressionofNeuralMachineTransla>onModelsviaPruning.Insubmission.

•  CompressNMTviapruning&retraining:

0 10 20 30 40 50 60 70 80 90

percentage pruned

pruned

pruned and retrained

Figure 1: Performance of pruned models, immediately after pruning and after

retraining.

Originalmodel

Prunesmallestweights

Prune+retrain

Ourwork•  CompressNMTviapruning&retraining:

0 10 20 30 40 50 60 70 80 90

percentage pruned

pruned

pruned and retrained

Figure 1: Performance of pruned models, immediately after pruning and after

retraining.

Originalmodel

Prunesmallestweights

Prune+retrain

Prune80%withoutlossofperformance.5/19/16 ThangLuong-NeuralMachineTranslaHon 122

NMTRedundancy–Embeddings

•  Frequentwordshavelargerweights– white:large.– black:small.

target embedding weights

source embedding weights

least common wordmost common word

source layer 1 weights

recurrentfeed-forward

input gate

forget gate

output gate

source layer 2 weights source layer 3 weights source layer 4 weights

target layer 1 weights target layer 2 weights target layer 3 weights target layer 4 weights

input gate

forget gate

output gate

NMTRedundancy–LSTMLayer1 Layer2 Layer3 Layer4

Encoder

Decoder

Inputgate

Forgetgate

Outputgate

Inputsignal

Feed-forward Recurrent

input gate

forget gate

output gate

Encoder

Decoder

Inputgate

Forgetgate

Outputgate

Inputsignal

Inputsignalisimportant!

input gate

forget gate

output gate

Encoder

Decoder

Inputgate

Forgetgate

Outputgate

Inputsignal

Forgetgate:small–layer1large–layer4

FutureChallenges

Shesawanelephantinherdress.Theelephantmusthaveagood

senseoffashion!Needstounderstand

commonsense&largercontext.

Shesawanelephantinherdress.

Thankyou!

neural machine transla#on

Documents

cs536: machine learning artificial neural networks neural...

cs 391l: machine learning neural networks

an introduction to neural machine translation › wp-content...

making sense of neural machine...

machine learning: introduction to neural networks

fundamentals of machine learning for neural machine...

machine learning – neural network classification tutorial...

transla tia

neural networks and machine learning

mtma16-neural machine translation

leonid sigal - arxiv · 2017. 9. 1. · leonid sigal disney...

neural machine translation techniques used by google · a...

multimodal neural machine translation system

from treebanking to machine...

a arxiv:1409.0473v7 [cs.cl] 19 may 2016 · neural machine...

pre-translation for neural machine translation · neural...

microsoft · brain-machine interfaces can restore abilities...

neural machine transla/on · translation model + language...

06newexperiment6 transla

op inter flash transla 2