catching the falling knife of nmt - blogs at helsinkiuni · 5 -0.534 9 commercial 1 6 -0.950 10...

Catching theFalling Knife of NMT

Ondřej [email protected]

Institute of Formal and Applied LinguisticsFaculty of Mathematics and Physics

Charles University, Prague

November 1, 2017

1 / 67

Catching the Falling Knife

When markets fall they fall by gravity.There is no level one can calculate as bottom.

One needs to wait till the markets just fall and bottom out.http://www.niveza.in/stock-news/learn-investing/dont-catch-the-falling-knife

2 / 67

http://www.niveza.in/stock-news/learn-investing/dont-catch-the-falling-knife

Recent WMT History

English-Czech# score range system1 0.580 1-2 CU-BOJAR

0.578 1-2 CU-DEPFIX3 0.562 3 ONLINE-B4 0.525 4 UEDIN5 0.505 5-7 CU-ZEMAN

0.502 5-7 MES0.499 5-8 ONLINE-A0.484 7-9 CU-PHRASEFIX0.476 8-9 CU-TECTOMT

10 0.457 10-11 COMMERCIAL-10.450 10-11 COMMERCIAL-2

12 0.389 12 SHEF-WPROA

2013

3 / 67

Recent WMT History






2013

English–Czech# score range system1 0.371 1-3 CU-DEPFIX

0.356 1-3 UEDIN-UNCNSTR0.333 1-4 CU-BOJAR0.287 3-4 CU-FUNKY

2 0.169 5-6 ONLINE-B0.113 5-6 UEDIN-PHRASE

3 0.030 7 ONLINE-A4 -0.175 8 CU-TECTO5 -0.534 9 COMMERCIAL16 -0.950 10 COMMERCIAL2

2014

4 / 67

Recent WMT History






2013





2014English–Czech

# score range system1 0.686 1 CU-CHIMERA2 0.515 2-3 ONLINE-B

0.503 2-3 UEDIN-JHU3 0.467 4 MONTREAL4 0.426 5 ONLINE-A5 0.261 6 UEDIN-SYNTAX6 0.209 7 CU-TECTO7 0.114 8 COMMERCIAL18 -0.342 9-11 TT-DCU

-0.342 9-11 TT-AFRL-0.346 9-11 TT-BLEU-MIRA-D

9 -0.373 12 TT-USAAR-TUNA10 -0.406 13 TT-BLEU-MERT11 -0.563 14 TT-METEOR-CMU

2015

5 / 67

Recent WMT History






2013





2014English–Czech





2015

English–Czech# score range system1 0.59 1 UEDIN-NMT2 0.43 2 NYU-MONTREAL3 0.34 3 JHU-PBMT4 0.30 4-5 CU-CHIMERA

0.30 4-5 CU-TAMCHYNA5 0.22 6-7 UEDIN-CU-SYTX

0.19 6-7 ONLINE-B6 0.16 8-11 TT-BLEU-MIRA

0.15 8-12 TT-BEER-PRO0.15 8-13 TT-BLEU-MERT0.14 9-14 TT-AFRL20 14 9-14 TT-AFRL1

2016

6 / 67

CUNI Collective Efforts for WMT17▶ Neural Monkey (Helcl and Libovický, 2017).▶ NMT Training Task (Bojar et al., 2017).▶ BPE, Learning rate and other meta-parameters.▶ Batch sizing (smaller/larger/variable).▶ Additional training objective:

▶ Targetting GIZA++ alignments.▶ Scoring the set of produced words, disregarding position.

▶ Minibatch bucketing.▶ Curriculum learning. (Kocmi and Bojar, 2017)▶ Pre-trained embeddings.▶ Domain adaptation: Subsample for Testset / Each Doc.▶ Neural sys combination: Concatenative/Multi-encoder.

7 / 67

… If Gains, then Mediocre …▶ Neural Monkey (Helcl and Libovický, 2017).▶ NMT Training Task (Bojar et al., 2017).▶ BPE, Learning rate and other meta-parameters.▶ Batch sizing (smaller/larger/variable).▶ Additional training objective:



8 / 67

… But it Later Worked for Others!▶ Neural Monkey (Helcl and Libovický, 2017).▶ NMT Training Task (Bojar et al., 2017).▶ BPE, Learning rate and other meta-parameters.▶ Batch sizing (smaller/larger/variable).▶ Additional training objective:



9 / 67

Our WMT17 System

… so we sticked to phrase-based MT backbone:▶ Moses system with several phrase tables:

▶ Standard corpus-based one (synthetic mononews only!).▶ Output of TectoMT for the test set.▶ Output of Nematus 2016 and Neural Monkey 2017.

▶ Followed by Depfix (Rosa et al., 2012).▶ Fixing agreement.▶ Recovering lost negation.

All details in Sudarikov et al. (2017).

10 / 67

… And the Result:






2013





2014English–Czech





2015

English–Czech# score range system1 0.59 1 UEDIN-NMT2 0.43 2 NYU-MONTREAL3 0.34 3 JHU-PBMT4 0.30 4-5 CU-CHIMERA

0.30 4-5 CU-TAMCHYNA5 0.22 6-7 UEDIN-CU-SYTX

0.19 6-7 ONLINE-B6 0.16 8-11 TT-BLEU-MIRA

0.15 8-12 TT-BEER-PRO0.15 8-13 TT-BLEU-MERT0.14 9-14 TT-AFRL20 14 9-14 TT-AFRL1

2016English→ Czech

# Ave % Ave z system

1 62.0 0.308 uedin-nmt

2 59.7 0.240 online-B

3 55.9 0.111 limsi-factored-norm55.2 0.102 LIUM-FNMT55.2 0.090 LIUM-NMT54.1 0.050 CU-Chimera53.3 0.029 online-A

8 44.9 −0.236 TT-ufal-8GB

9 42.2 −0.315 TT-afrl-4GB41.9 −0.327 PJATK40.7 −0.373 TT-base-8GB40.5 −0.376 TT-afrl-8GB

2017

Fish by Frits Ahlefeldt

NMT

SMT

11 / 67

We Were Hoping to Be the Second!

Manual Automatic Scores# Ave % Ave z BLEU TER CharacTER BEER System1 62.0 0.308 22.8 0.667 0.588 0.540 uedin-nmt2 59.7 0.240 20.1 0.703 0.612 0.519 online-B3 55.9 0.111 20.2 0.696 0.607 0.524 limsi-factored

55.2 0.102 20.0 0.699 - - LIUM-FNMT55.2 0.090 20.2 0.701 0.605 0.522 LIUM-NMT54.1 0.050 20.5 0.696 0.624 0.523 CU-Chimera53.3 0.029 16.6 0.743 0.637 0.503 online-A

8 41.9 -0.327 16.2 0.757 0.697 0.485 PJATK

Automatic scores by http://matrix.statmt.org/.

12 / 67

http://matrix.statmt.org/

Outline

1. How good UEDIN’s WMT17 outputs are in fact.▶ Remaining errors.▶ News vs. doc-level phenomena.

2. An empirical comparison of toolkits.3. What NMT offers to computational linguistics.

13 / 67

Is UEDIN NMT That Much Better?

SRC 28-Year-Old Chef Found Dead at San Francisco Mall28letý šéfkuchař Found Dead v San Francisco MallOsmadvacetiletý šéfkuchař nalezen mrtev v obchoděv San Francisku

SRC A 28-year-old chef who had recently moved to San Franciscowas found dead in the stairwell of a local mall this week.Osmadvacetiletý kuchař, který se nedávno přestěhoval do SanFrancisca, byl tento týden nalezen mrtvý na schodišti místníhoobchodního centra.Osmadvacetiletý šéfkuchař, který se nedávno přistěhoval do SanFranciska, byl tento týden schodech místního obchodu.

14 / 67


SRC 28-Year-Old Chef Found Dead at San Francisco MallMT 28letý šéfkuchař Found Dead v San Francisco MallREF Osmadvacetiletý šéfkuchař nalezen mrtev v obchodě

v San Francisku

SRC A 28-year-old chef who had recently moved to San Franciscowas found dead in the stairwell of a local mall this week.Osmadvacetiletý kuchař, který se nedávno přestěhoval do SanFrancisca, byl tento týden nalezen mrtvý na schodišti místníhoobchodního centra.Osmadvacetiletý šéfkuchař, který se nedávno přistěhoval do SanFranciska, byl tento týden schodech místního obchodu.

14 / 67



v San Francisku

SRC A 28-year-old chef who had recently moved to San Franciscowas found dead in the stairwell of a local mall this week.Osmadvacetiletý kuchař, který se nedávno přestěhoval do SanFrancisca, byl tento týden nalezen mrtvý na schodišti místníhoobchodního centra.Osmadvacetiletý šéfkuchař, který se nedávno přistěhoval do SanFranciska, byl tento týden ∅ schodech místního obchodu.

14 / 67



v San Francisku

SRC A 28-year-old chef who had recently moved to San Franciscowas found dead in the stairwell of a local mall this week.

MT Osmadvacetiletý kuchař, který se nedávno přestěhoval do SanFrancisca, byl tento týden nalezen mrtvý na schodišti místníhoobchodního centra.

REF Osmadvacetiletý šéfkuchař, který se nedávno přistěhoval do SanFranciska, byl tento týden ∅ schodech místního obchodu.

14 / 67

Is UEDIN NMT That Much Better? (2/4)SRC A spokesperson for Sons & Daughters said they were

”shocked and devastated” by his death.Mluvčí společnosti Sons & Daughters uvedla, žejsou jeho smrtí ”šokováni a zdrceni”.Mluvčí restaurace Sons & Daughters řekl, že jsoujeho smrtí „šokováni a zničeni“.

SRC ”He found an apartment, he was dating a girl,” LouisGalicia told KGO.„Našel si byt, chodil s dívkou,“ řekl Louis Galicia proKGO.”Našel si byt, chodil s holkou,” řekl Louis GalicieKGO.

15 / 67


”shocked and devastated” by his death.MT Mluvčí společnosti Sons & Daughters uvedla, že

jsou jeho smrtí ”šokováni a zdrceni”.REF Mluvčí restaurace Sons & Daughters řekl, že jsou

jeho smrtí „šokováni a zničeni“.

SRC ”He found an apartment, he was dating a girl,” LouisGalicia told KGO.„Našel si byt, chodil s dívkou,“ řekl Louis Galicia proKGO.”Našel si byt, chodil s holkou,” řekl Louis GalicieKGO.

15 / 67


”shocked and devastated” by his death.MT Mluvčí společnosti Sons & Daughters uvedla, že

jsou jeho smrtí ”šokováni a zdrceni”.REF Mluvčí restaurace Sons & Daughters řekl, že jsou

jeho smrtí „šokováni a zničeni“.

SRC ”He found an apartment, he was dating a girl,” LouisGalicia told KGO.

REF „Našel si byt, chodil s dívkou,“ řekl Louis Galicia proKGO.

MT ”Našel si byt, chodil s holkou,” řekl Louis GalicieKGO.

15 / 67

Is UEDIN NMT That Much Better? (3/4)SRC The police arrested two men, who on Tuesday at-

tacked a thirty-five-year-old man with a knife and amachete.Policie obvinila dva útočníky, kteří v úterý v cen-tru Olomouce napadli nožem a mačetou pě-tatřicetiletého muže.Policie zatkla dva muže, kteří v úterý napadli pě-tatřicetiletého muže nožem a mačetou.

SRC There were creative differences on the set and a dis-agreement.Došlo ke vzniku kreativních rozdílů na scéně ak neshodám.Na place byly tvůrčí rozdíly a neshody.

16 / 67


tacked a thirty-five-year-old man with a knife and amachete.

REF Policie obvinila dva útočníky, kteří v úterý v cen-tru Olomouce napadli nožem a mačetou pě-tatřicetiletého muže.

MT Policie zatkla dva muže, kteří v úterý napadli pě-tatřicetiletého muže nožem a mačetou.

SRC There were creative differences on the set and a dis-agreement.Došlo ke vzniku kreativních rozdílů na scéně ak neshodám.Na place byly tvůrčí rozdíly a neshody.

16 / 67


tacked a thirty-five-year-old man with a knife and amachete.

REF Policie obvinila dva útočníky, kteří v úterý v cen-tru Olomouce napadli nožem a mačetou pě-tatřicetiletého muže.

MT Policie zatkla dva muže, kteří v úterý napadli pě-tatřicetiletého muže nožem a mačetou.

SRC There were creative differences on the set and a dis-agreement.

REF Došlo ke vzniku kreativních rozdílů na scéně ak neshodám.

MT Na place byly tvůrčí rozdíly a neshody.16 / 67

Is UEDIN NMT That Much Better? (4/4)

SRC Economy Secretary Keith Brown visited the sitetoday and was among the first to walk from the landon to the bridge.Ekonomický tajemník Keith Brown stavbu dnesnavštívil a byl mezi prvními, kteří přišli z pevniny namost.Ministr hospodářství Keith Brown dnes místonavštívil a byl mezi prvními, kteří vyšli ze země namost.

17 / 67

Is UEDIN NMT That Much Better? (4/4)

SRC Economy Secretary Keith Brown visited the sitetoday and was among the first to walk from the landon to the bridge.

REF Ekonomický tajemník Keith Brown stavbu dnesnavštívil a byl mezi prvními, kteří přišli z pevniny namost.

MT Ministr hospodářství Keith Brown dnes místonavštívil a byl mezi prvními, kteří vyšli ze země namost.

17 / 67

Luckily ;-), Catastropic Errors HappenAlso WMT17 UEDIN outputs (but not easy to spot):SRC ... said Frank initially stayed in hostels...MT ... řekl, že Frank původně zůstal v Budějovicích...↰Gloss ... said that Frank initically stayed in Budweis...SRC Most of the Clintons’ income...MT Většinu příjmů Kliniky...↰Gloss Most of the income of the Clinic...SRC The 63-year-old has now been made a special representative...MT 63letý mladík se nyní stal zvláštním zástupcem...↰Gloss The 63-year-old youngster has now become a special representative...

18 / 67

Catastropic Errors Happen (2/2)SRC Criminal Minds star Thomas Gibson sacked after hitting pro-

ducerREF Thomas Gibson, hvězda seriálu Myšlenky zločince, byl prop-

uštěn po té, co uhodil režiséraMT Kriminalisté Minsku hvězdu Thomase Gibsona vyhostili po

zásahu producenta↰Gloss Minsk criminal investigators have expelled the starThomas Gibson after striking the producer

SRC ...add to that its long-standing grudge...REF ...přidejte k tomu svou dlouholetou nenávist...MT ...přidejte k tomu svou dlouholetou záštitu...↰Gloss ...add to that its long-standing auspices...

(grudge = zášť → záštita = auspices)

19 / 67

UEDIN at WMT17▶ Our small annotation of up to 185 sentences.▶ Blind mix: reference or MT.Real MT was assumed to be:

OB DM DVMT 142 (76.8 %) 86 (77.5 %) 72 (87.8 %)didn’t know 34 (18.4 %) 9 (8.1 %) 6 (7.3 %)human 9 (4.9 %) 16 (14.4 %) 4 (4.9 %)Total 185 (100.0 %) 111 (100.0 %) 82 (100.0 %)

⇒ 10–20% of outputs indistinguishable from humans.

OB DM DValmost flawless 17 (9.19 %) 2 (1.80 %) 0 (0 %)flawless 82 (44.32 %) 37 (33.33 %) 27 (32.93 %)

⇒ 30–50% of outputs flawless or almost flawless.

20 / 67

UEDIN at WMT17▶ Our small annotation of up to 185 sentences.▶ Blind mix: reference or MT.Real MT was assumed to be:

OB DM DVMT 142 (76.8 %) 86 (77.5 %) 72 (87.8 %)didn’t know 34 (18.4 %) 9 (8.1 %) 6 (7.3 %)human 9 (4.9 %) 16 (14.4 %) 4 (4.9 %)Total 185 (100.0 %) 111 (100.0 %) 82 (100.0 %)

⇒ 10–20% of outputs indistinguishable from humans.OB DM DV

almost flawless 17 (9.19 %) 2 (1.80 %) 0 (0 %)flawless 82 (44.32 %) 37 (33.33 %) 27 (32.93 %)

⇒ 30–50% of outputs flawless or almost flawless.20 / 67

Errors Flagged# % Error Type35 26.1 lexical error↰13 carriages, 2 altercations, 2 decks, 2 plantings, …22 16.4 notNice10 7.5 namedEntity12 9.0 world knowledge needed or helpful7 5.2 terminology↰winter wheat, winter barley, emergency kill cord, …7 5.2 grammar6 4.5 extra5 3.7 minor5 3.7 anaphora3 2.2 valency2 1.5 global sentence structure2 1.5 units (CZK ̸= GBP)2 1.5 BPE2 1.5 SRL14 10.4 Other, 1 occurrence each

134 100.0 Total flags21 / 67

Negation in Nematus en→csManual analysis of HimL (medical) and news by Rudolf Rosa:

Negated Meaning Correct ErrorAnnotated No Yes Yes No in Negation Elsewhere

Czech sents 298 237 61 55 6 2 4% of Annotated 79.5% 20.5% 18.5% 2.0% 0.7% 1.3%% of Negated 100% 90% 10% 3.3% 6.6%

▶ Errors in negation very rare.▶ In many cases, hard negation phenomena correct.▶ No single error in Czech double negation.▶ Lexicalized negation also handled perfectly:

▶ was slurring = mluvila nesrozumitelně, recently = nedávno, unfortunately = bohužel,homeless = bezdomovce, failing to coordinate = nekoordinovala, rather than = nikoliv.

▶ In total, only 2 clear errors:▶ 1 missing negation,▶ 1 incorrect negation scope (due to subject-object marking error).

22 / 67

Doc-Level Effects in News?Does MT have to consider cross-sentence phenomena?Manual annotation of 40–92 “paragraphs” from WMT11:▶ 4 consecutive sentences per “paragraph”.▶ 4 manual versions of each sentence:

▶ Original Czech / translation from English▶ Translation from German (3 different translations).

▶ Some paragraphs “clean”, some “mixed” (each sentencecoming from a different source).

▶ Blind annotation to identify clean vs mixed.▶ 71–78% of “mixed” paragraphs marked as clean.▶ 17–21% of “clean” paragraphs marked as mixed.

⇒ in up to 80% sents, source probably captures everything.⇒ in up to 20%, humans seem to produce incoherent text.⇒ News domain exhibits too few cross-sentence links.

23 / 67

Call for WMT18 Test SuitesBurlot and Yvon (2017): test suite with automatic checks.1. Create contrastive source sentence pairs.2. Have everyone translate them.3. (Automatically) check if the desired phenomenon is

handled as expected.

My goal for WMT18:▶ Extend the standard 3k news test sentences

with your contributions:▶ Contrastive source sentence pairs.▶ Automatic checks of outputs.

▶ Participants will translate everything.▶ You will then evaluate your portion of the test set.⇒ Collectively, we will focus on many specific things.

24 / 67

Call for WMT18 Test SuitesBurlot and Yvon (2017): test suite with automatic checks.1. Create contrastive source sentence pairs.2. Have everyone translate them.3. (Automatically) check if the desired phenomenon is

handled as expected.My goal for WMT18:▶ Extend the standard 3k news test sentences

with your contributions:▶ Contrastive source sentence pairs.▶ Automatic checks of outputs.

▶ Participants will translate everything.▶ You will then evaluate your portion of the test set.⇒ Collectively, we will focus on many specific things.

24 / 67

Take-Home Message #1In large-data settings:▶ NMT has sufficiently resolved:

▶ morphology,▶ negation.

▶ Remaining errors concern primarily:▶ world knowledge,▶ terminology,▶ rare words (incl. named entities),▶ anaphora.

Dedicated test suites needed:▶ otherwise we’d be evaluating generally solid outputs.

Contact me to extend WMT18 test set with your data.

25 / 67

Open-Source ToolsToolkit / Language

DL4MT Theano / Python↰Nematus Theano / PythonMarian (incl. AmuNMT) C++seq2seq_attn Torch / Lua↰OpenNMT Torch / Lua+PythonLamtram DyNet / C++Neural Monkey Tensorflow / PythonGoogle seq2seq Tensorflow / PythonGoogle tensor2tensor (Transformer) Tensorflow / Python

▶ Hard to choose one, all have their goods and bads.▶ There is always a big cost of getting it running.A more complete list by Jon Dehdari: https://github.com/jonsafari/nmt-list

26 / 67

https://github.com/jonsafari/nmt-list

Nematus vs. Neural MonkeyGreedy Beam Training

System BLEU BLEU TimeNeural Monkey 21.13 22.74 4d07hNeural Monkey + ReLu 21.97 23.14 4d17hNeural Monkey + ReLu + Softmax Fix 21.73 22.99 4d18hNematus closest setup 22.77 24.32 10d

▶ Nematus better in BLEU but two times slower.▶ Neural Monkey got 1.5 BLEU improvement from:

initializer=tf.random_uniform_initializer(-0.5, 0.5)vs. initializer=tf.random_uniform_initializer(-0.5, 0-5)

▶ ∼2 BLEU point from TF change from 0.11 to 1.0▶ One reason is ReLU becoming the default activation function.

Common: Czech→English, CzEng 1.6 limited to 30M sent pairs. Fixed BPE 30k. (BLEU evaluated on the BPE). Encoder: emb 512;max length 50; RNN size 1000; GRU with no dropout. Decoder: emb 512; max length 50; RNN size 1000; conditional GRU with nodropout. Optimization: Adam with learning rate 10−4 optimized on the cross entropy. Batch size was 60. Beam search: maximumsteps 50; length normalization 0.6; beam size 20. NM run on GeForce 1080Ti, Nematus run on GeForce 1080.

27 / 67

A Little Messy Empirical Comparison▶ Czech→English, CzEng 1.6 limited to 30M sent. pairs.▶ Fixed BPE 30k. (BLEU evaluated on the BPE).System BLEU Steps Training TimeDeep Nematus 2017 28.96 30M (1 ep) 25d8hTransformer 201k*2k 27.84 201k (∼1 ep) 1d06hShallow Nematus (best known) 25.93 30M (1 ep) 10dMarian 1 GPU 24.65 29M (∼1 ep) 1d11hNeural Monkey (not best setup) 23.14 30M (1 ep) 5d17hOpenNMT default setting, 4 GPU 21.72 30M (1 ep) 15hTransformer 8*GPU 843k*1k + avg 32.68 ∼4.2 ep 7d00hTransformer 1820k*2k + avg 31.85 ∼9.0 ep 11d04hTransformer 1383k*2k + avg 31.72 ∼6.9 ep 8d12hTransformer 699k*2k + avg 30.86 ∼3.4 ep 4d08hTransformer 699k*2k 30.43 ∼3.4 ep 4d08hMarian 1 GPU 26.66 124M (∼4.1 ep) 4d16hOpenNMT default, 4 GPU 26.05 ∼17 ep

1 ep = 30 M sent pairs, 1 step = 1 sent. pair or 2048 BPE tokens28 / 67

Take-Home Message #2▶ Terrible amount of man/GPU time easily wasted

in chasing toolkits and baselines.⇒ Pick your favourite one and stick with it.⇒ Try to respect the choice of others when reviewing.

My personal picks: Marian, tensor2tensor, Neural Monkey

Now I understand the frustration of tryingto catch up with the WMT benchmark.

Anyone interested in a really constrained translation task?

29 / 67

30 / 67

Semiotic Triangle by Ogden and Richards

Thought or Reference

Symbol Referent

Cor

rect

sym

bol s

ymbo

lises

Adequate thought refers to

True symbol stands for

31 / 67



Symbol Referent

Cor

rect

sym

bol s

ymbo

lises



Dannyapproached

the chair witha yellow bag.

32 / 67



Symbol Referent

Cor

rect

sym

bol s

ymbo

lises



Dannyapproached


33 / 67



Symbol Referent

Cor

rect

sym

bol s

ymbo

lises



Dannyapproached


34 / 67



Symbol Referent

Cor

rect

sym

bol s

ymbo

lises



Dannyapproached


λp.λc.λb.person(p)∧chair(c)∧bag(b)

∧yellow(b)∧has(p,b)∧approach(p,c)

λp.λc.λb.person(p)∧chair(c)∧bag(b)

∧yellow(b)∧has(c,b)∧approach(p,c)

35 / 67

DiCarlo NIPS 2013 Tutorial on Vision

36 / 67

From Vision to LanguageDiCarlo (2013): Human object recognition explained by:▶ Recording apes’ neuronal activity

and attaching a single-layer NN to interpret it▶ Measuring human performance… on the same object recognition tasks.▶ and relating them.

Proposal: Instead of catching the falling knife:▶ Record NMT/NN behaviour (all parameters accessible)▶ and human behaviour, possibly recording:

▶ Objective: reading studies, eye-tracking, …▶ Subjective: introspection.

… on the same language processing tasks.▶ and relate them.

37 / 67

From Vision to LanguageDiCarlo (2013): Human object recognition explained by:▶ Recording apes’ neuronal activity

and attaching a single-layer NN to interpret it▶ Measuring human performance… on the same object recognition tasks.▶ and relating them.Proposal: Instead of catching the falling knife:▶ Record NMT/NN behaviour (all parameters accessible)▶ and human behaviour, possibly recording:

▶ Objective: reading studies, eye-tracking, …▶ Subjective: introspection.

… on the same language processing tasks.▶ and relate them.

37 / 67



Symbol Referent

Cor

rect

sym

bol s

ymbo

lises



Dannyapproached


38 / 67

Some Techniques of NN Inspection▶ MicroNNs, e.g. Shi et al. (2016) learning length.▶ Lobotomy (Li et al., 2016).▶ Observing activations, attentions…▶ Exploring representation space.

▶ t-SNE and PCA for sentence pairs▶ Translation by search = similarity in meaning reflected in space▶ Attaching an NN to see if it can infer:

▶ POS or morphology from NMT▶ Subject-Verb agreement (Linzen et al. TACL/EACL 2017)

▶ Linguistic exploration:▶ Various test suites (Burlot 2017, Burchhardt MQM, Lingeval97).▶ Stanford Natural Language Inference (SNLI)

https://nlp.stanford.edu/projects/snli/▶ Paraphrases (Dreyer and Marcu, 2012; Bojar et al., 2013).

▶ Comparing representations (Nili et al., 2014).39 / 67

https://nlp.stanford.edu/projects/snli/

Aspects of Meaning▶ Meaning is a coarsening:

▶ Pictures: Semantic segmentation (“reverse raytracing”)▶ Programs: The output they give (caveat: undecidable).▶ CL: Reference to real world? Speaker’s intention?

▶ Meaning can be shifted, modified.▶ Meanings can be compared.▶ Meaning is generally compositional.

▶ (along the linguistic structure).▶ Pragmatics: Named entities, numbers, anaphora…▶ Expressions are ambiguous.▶ Meanings are vague.▶ Are meanings stateful?▶ Are meanings continuous?

40 / 67

Meaning as a CoarseningSemantic Segmentation of Pictures

… and generating back with pix2pix:

Illustrations from http://www.cs.toronto.edu/~tingwuwang/semantic_segmentation.pdf andhttps://github.com/phillipi/pix2pix. 41 / 67

http://www.cs.toronto.edu/~tingwuwang/semantic_segmentation.pdf

https://github.com/phillipi/pix2pix

Meaning Statefulness

Stateful Meaning Representation:▶ “The state of mind after having read this and produced

this output so far.”▶ Corresponds to models with attention.▶ Btw needed to interpret humour (Gluscevskij, 2017).

Stateless Meaning Representation:▶ Points correspond to meanings.

▶ As in models without attention.

42 / 67

Continuous Spaces in NMT▶ … are plentiful:

▶ Word, sentence, document embeddings…▶ … (probably) crucially depend on the task:

▶ NMT vs. QA vs. summarization vs. sentiment …▶ … (probably) depend on the architecture.▶ … (obviously) depend on the point in the architecture.Desired properties of continuous sentence representations:(by Schwenk and Douze (2017)):

▶ semantic closeness,▶ multilingual closeness, incl. across many languages,▶ preservation of content (task-specific).… plus the properties I listed three slides back.

43 / 67

Is Sentence Meaning Continuous?10k–100k sentence paraphrases in English and Czech(Dreyer and Marcu, 2012; Bojar et al., 2013)

Premiere of Iraq Nuri al-Maliki was given an excuse by President Bush, whoexpressed his confidence in him, and he stated that the circumstances are com-plicated.

President Bush said that he trusts in Nouri Maliki, head of government of Iraq,and he stated that he finds an excuse for him ”because the situation is tricky”.Head of cabinet of Iraq Nuri al-Maliki was given an excuse by President Bush,who expressed his trust in him, and he indicated that the circumstances aredifficult.Iraq’s head of cabinet Nuri al-Maliki was given a reason by President Bush, whoexpressed his trust in him, and he indicated that the case is tricky.President Bush said that he has faith in Iraqi head of cabinet Nouri al-Maliki,and he stated that he finds an excuse for him ”for the case is complicated”.

Q: Are all these paraphrases close in sent embedding spaces?Q: How entagled are manifolds of different sents?… pushing Holger Schwenk to work on this with me.

44 / 67

Is Sentence Meaning Continuous?10k–100k sentence paraphrases in English and Czech(Dreyer and Marcu, 2012; Bojar et al., 2013)

Premiere of Iraq Nuri al-Maliki was given an excuse by President Bush, whoexpressed his confidence in him, and he stated that the circumstances are com-plicated.President Bush said that he trusts in Nouri Maliki, head of government of Iraq,

and he stated that he finds an excuse for him ”because the situation is tricky”.Head of cabinet of Iraq Nuri al-Maliki was given an excuse by President Bush,who expressed his trust in him, and he indicated that the circumstances aredifficult.Iraq’s head of cabinet Nuri al-Maliki was given a reason by President Bush, whoexpressed his trust in him, and he indicated that the case is tricky.President Bush said that he has faith in Iraqi head of cabinet Nouri al-Maliki,and he stated that he finds an excuse for him ”for the case is complicated”.

Q: Are all these paraphrases close in sent embedding spaces?Q: How entagled are manifolds of different sents?… pushing Holger Schwenk to work on this with me.

44 / 67

Examining Continuous Space of Sents.Stages of Space Mapping:1. Propose directions of exploration.2. Generate seed pairs of sentences for each of the directions.3. Collect specimens along the proposed directions:

▶ interpolation, a “sentence in between”,▶ extrapolation, “a sentence further in the hinted direction”.▶ Allow people to say “impossible”.

4. Validate the relations.5. Create the partially ordered set.6. Search for a manifold covering the ordered set.Work in progress with Chris Callison-Burch.

45 / 67

Directions of Exploration (1/2)

▶ Politeness.▶ Tense.▶ Verity: How much the speaker believes the message.▶ Modality: Willingness/Ability of the speaker to do it.▶ “Counting” / Generic Numerals, Scalar adjectives.

▶ I saw a handful of people there. / a big crowd / a massive crowd.▶ freezing / cold / chilly

▶ “Negation”, but not only reversing the main predicate.▶ Complexity / simplicity, Length.

46 / 67

Directions of Exploration (2/2)▶ Specificity / Generality, Vagueness.

▶ Geese fly / Geese migrate / Geese migrate south/ The Canadian geese flew over the pond at friendly Farms in theirsouthward migration.

▶ Hammer the hook into the wall. / Put the hook on the wall./ Do the thingy in there.

▶ Contextual boundness.▶ Give it to him. / Give the parcel to the man at the counter.

/ Give your parcel to the operator at the post office.▶ High/low style/English/class.

▶ Hey y’all it’s a nice day ain’t it?▶ Greetings! Lovely weather we are having.

Thanks to Sarka Zikanova for some of the ideas.Looking forward for any other ideas you can suggest.

47 / 67

First Results of Getting Pairs

Can you please give me a minute? Could you leave me alone?Close the door. Close the damn door man

Can you help me find something? I need you to help me get something.May I talk to Mary? Is Mary here?

I’m sorry-I don’t believe we have met. Who the hell are you?Can you move so I can see the screen? You aren’t made of glass, you know.

Will you kindly exit? I do not want you here!Would you please get the mail? Get the mail!

Can I help you? What do you want?Can you please help me with this? Get over here and help me!

Can you make me breakfast? Why are you not making me breakfast right now?I tried to call were you busy? You never answer your phone.

48 / 67

First Results of Midpointing (1/3)

Can you help me find something?Find this for me.Help me find something.Please help me find something.Will you help me?Would you help me look?Your assistance in finding something is required.I need you to help me get something.

49 / 67

First Results of Midpointing (2/3)Can you please give me a minute?Come back laterGive me a minute.Hey give me a minute.I’d like a minute alone.I need a minute to myself.I need more time.One minute.One moment.Please wait.Could you leave me alone?

50 / 67

First Results of Midpointing (3/3)

Can you move so I can see the screen?Blocking the view, friend.Can you move a bit?Can you please move?Could you move a little bit, you’re blocking the screen.Hey can you move.I can’t see, can you move a little?Move your blocking the screenPlease move.You aren’t made of glass, you know.

51 / 67

Collect All Variations

Can you hurry eating?

All done?

Are you almost done eating?

Are you done eating yet?

Are you finished with your food?

Are you finished with your food yet?

Done with the food?Finished yet?

When will you be done with your food?

You're still not done with your food?

52 / 67

Ask Crowd to Partially Sort Them


All done?









All done?








53 / 67

Find Methods for Manifold Learning


All done?









All done?








Manifold learning

54 / 67

Match Posets with Learned Manifolds


All done?









All done?








Manifold learning

Can you hurry eating?All done?

Are you almost done eating?Are you done eating yet?

Are you finished with your food?Are you finished with your food yet?




Can you hurry eating?All done?

Are you almost done eating?Are you done eating yet?

Are you finished with your food?Are you finished with your food yet?



You're still not done with your food?Manifold learning

semi-supervised.

55 / 67

Take-Home Message #3

Fish by Frits Ahlefeldt

DL

CLCL

56 / 67

Summary▶ Neural MT reaches and can surpass humans.

▶ Catastrophic errors still possible.▶ As a side-effect, continuous representations are learned.▶ Insight in vision thanks to relating

ape, computer and human vision.

▶ Computational linguistics has plenty of data.▶ Other data can be relatively easily obtained.⇒ Let’s train NMT/NLU systems and dissect them.

FinMT FiNMT MT. Fin? I’m not afraid.

57 / 67

ReferencesOndřej Bojar, Matouš Macháček, Aleš Tamchyna, and Daniel Zeman. 2013. Scratching the Surfaceof Possible Translations. In Proc. of TSD 2013, Lecture Notes in Artificial Intelligence, Berlin /Heidelberg. Západočeská univerzita v Plzni, Springer Verlag.Ondřej Bojar, Jindřich Helcl, Tom Kocmi, Jindřich Libovický, and Tomáš Musil. 2017. Results of theWMT17 Neural MT Training Task. In Proceedings of the Second Conference on MachineTranslation, Copenhagen, Denmark, September. Association for Computational Linguistics.Franck Burlot and François Yvon. 2017. Evaluating the morphological competence of machinetranslation systems. In Proceedings of the Second Conference on Machine Translation, Volume 1:Research Papers, pages 43–55, Copenhagen, Denmark, September. Association for ComputationalLinguistics.J. J. DiCarlo. 2013. Mechanisms underlying visual object recognition: Humans vs.neurons vs.machines. NIPS Tutorial.Markus Dreyer and Daniel Marcu. 2012. HyTER: Meaning-Equivalent Semantics for TranslationEvaluation. In Proceedings of the 2012 Conference of the North American Chapter of theAssociation for Computational Linguistics: Human Language Technologies, pages 162–171,Montréal, Canada, June. Association for Computational Linguistics.Dmitrij Gluscevskij. 2017. Methodological issues and prospects of semiotics of humour. SignSystems Studies, 45(1/2):137–151.Jindřich Helcl and Jindřich Libovický. 2017. Neural Monkey: An open-source tool for sequencelearning. The Prague Bulletin of Mathematical Linguistics, 107:5–17.Tom Kocmi and Ondřej Bojar. 2017. Curriculum Learning and Minibatch Bucketing in NeuralMachine Translation. In Proceedings of Recent Advances in NLP (RANLP 2017).Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. Understanding neural networks throughrepresentation erasure. CoRR, abs/1612.08220.Hamed Nili, Cai Wingfield, Alexander Walther, Li Su, William Marslen-Wilson, and NikolausKriegeskorte. 2014. A toolbox for representational similarity analysis. PLOS Computational Biology,10(4):1–11, 04.Rudolf Rosa, David Mareček, and Ondřej Dušek. 2012. DEPFIX: A System for Automatic Correctionof Czech MT Outputs. In Proceedings of the Seventh Workshop on Statistical Machine Translation,pages 362–368, Montréal, Canada, June. Association for Computational Linguistics.Holger Schwenk and Matthijs Douze. 2017. Learning joint multilingual sentence representations withneural machine translation. In Proceedings of the 2nd Workshop on Representation Learning forNLP, pages 157–167, Vancouver, Canada, August. Association for Computational Linguistics.Xing Shi, Kevin Knight, and Deniz Yuret. 2016. Why Neural Translations are the Right Length. InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages2278–2282, Austin, Texas, November. Association for Computational Linguistics.Roman Sudarikov, David Mareček, Tom Kocmi, Dušan Variš, and Ondřej Bojar. 2017. CUNISubmission in WMT17: Chimera Goes Neural. In Proceedings of the Second Conference on MachineTranslation, Copenhagen, Denmark, September. Association for Computational Linguistics.

58 / 67


59 / 67


60 / 67


61 / 67


62 / 67


63 / 67


64 / 67


65 / 67


66 / 67


67 / 67

catching the falling knife of nmt - blogs at helsinkiuni · 5 -0.534 9 commercial 1 6 -0.950 10...

Documents