overcoming the lack of parallel data in machine...

Overcoming the Lack of Parallel Data in Machine Translation

Kevin Knight & David Chiang USC/ISI

MURI Review

November 14, 2014

Two Talks: • Exploiting monolingual data • Exploiting deeper representations

Talk #1: Exploiting Monolingual Data cross-site collaboration (ISI/UT/CMU/MIT): using dependency parsers and word aligners to extract translation patterns from non-parallel text

Exploiting Monolingual Data

Malagasy text

Deciphering Engine

Malagasy/English translation dictionary, models for use in MT

Decipherment into English

treat Malagasy as a code for English... and decode

Step by Step

Letter Substitution Ciphers [Ravi/Knight 08, Ravi/Knight 09a]

Phoneme Substitution [Ravi/Knight 09b]

Word Substitution Ciphers [Ravi/Knight 11a, Dou/Knight 12]

Foreign Language as a Cipher for English [Ravi/Knight 11a, Dou/Knight 12, Dou/Knight 13, Dou/Vaswani/Knight 14]

Historical Ciphers [Snyder/Barzilay/Knight 10, Knight/Megyesi/Schaefer 11, Ravi/Knight 11b, Reddy/Knight 11]

2009 2010 2011 2012 2013 2008 2014

Letter Substitution Cipher

Letter Frequencies

2-grams: 3-grams: ? - 99 ? - ^ 47 C : 66 C : G 23 - ^ 49 Y ? - 22 : G 48 y ? - 18 z ) 44 H C | 17

Tendencies:

A, E, I, O, U followed by 3 and j A, E, I, O, U preceded by z and >

0

50

100

150

200

250

300

350

400

450

^ | z G- C Z j ! 3 Y ) U y + O F H = : I > b g RM E X c ? 6 K N n < / Q ~ A D p B P " S l Lkm1 & e 5f v h rJ 7 i T s o ] a t d u89[ 0w_ W 4 q @x2#, ` \*%

Letter Distributions

?]8R j 3 |^+C~DgBF/[4TM15-: 7Q >z6X9s qxJmvknwtrfhoai Lc bp uKei W”=Gd&<)OAZUEI y!Y PHN

unaccented Roman letters

circumflexed vowels

underlined letters

letters grouped if they have similar contexts (L/R neighbors)

thanks Jon Graehl

Statistical Modeling

P(c | p) P(p)

plaintext p ciphertext c

“key”

Statistical Modeling

ciphertext c P(c | p) P(p)

plaintext p

Find substitution-table values that maximize P(c) = Σp P(p, c) = Σp P(p) P(c | p)

best guess plaintext p

Find plaintext p that maximizes P(p | c) ∼ P(p) P(c | p)

EM

Viterbi

LM

plaintext samples, unrelated to ciphertext

ciphertext c

“key”

Letter Substitution: Results

[Ravi & Knight 08]

Plus other methods, such as based on integer linear programming.

Word Substitution

Each code number represents a plaintext word, not letter

Encipherment Key

Decipherment Key

Word Substitution Keys

Word Substitution

[Dou & Knight 2012]

Foreign Language as a Code for English

!l@!m !lywm !lth!ny& !l@!m !lm!Dy Sfr @!m th!ny& @!m 1992 @!m 1993 ywm !l!sbw@ !lm!Dy fy !ldqyq& !lsn& !lj!ry& !lsn& !lsh=hr !lm!Dy !lsh=hr !lj!ry snw!t sn& =hdh! !l@!m s!@& !l@Sr @!m 1991

!l@Swr =hdh! !lsh=hr fy ywm nys!n !sbw@ =hdh=h !l!'y!m qbl !'y!m fy !l@Sr mn !lsn& !lsnw!t b@d ywm !l!y!m 13 nys!n 1994 !lth!ny& @shr& thl!th& !y!m qbl !sbw@yn fy !lywm !lt!ly sh@b!n tmwz 3 dhw !lHj& 1414 fy shb!T !lm!Dy qbl ywmyn

@!m 1990 w!lth!ny& fy !lywm mn !lsh=hr !lj!ry !lqrn !'y!m @!m!aN !ls!@& 17 shb!T 1994 thl!th snw!t dqyq& =hdh=h !lsn& ywmyn mn !l@!m !lm!Dy !lsn& !lmqbl& fy !lsn& kl ywm fy !l@!m !lm!Dy

13 4 Hzyr!n 1967 12 fy 12 Hzyr!n 1993 7 5 Hzyr!n 1967 6 fy 30 Hzyr!n 1989 6 30 Hzyr!n 1989 4 fy 30 Hzyr!n 1994 4 fy 30 Hzyr!n 1993 3 fy 19 Hzyr!n 1967 2 ywm 30 Hzyr!n 1989 2 w 6 Hzyr!n 1994 2 qbl 5 Hzyr!n 1967 2 fy 9 Hzyr!n 1967 2 fy 7 Hzyr!n 1981 2 fy 6 Hzyr!n 1994 2 fy 5 Hzyr!n 1967

2 fy 30 Hzyr!n 1995 2 fy 18 Hzyr!n 1994 2 fy 14 Hzyr!n 1993 2 fy 14 Hzyr!n 1991 2 fy 12 Hzyr!n 1990 2 7 Hzyr!n 1994 2 6 Hzyr!n 1941 2 26 Hzyr!n 1994 2 21 Hzyr!n 1994 2 1 Hzyr!n 1994 2 19 Hzyr!n 1965 2 18 Hzyr!n 1994 2 18 Hzyr!n 1940 2 12 Hzyr!n 1993 2 11 Hzyr!n 1994

<n> Hzyr!n <n>



13 4 Hzyr!n 1967 12 fy 12 Hzyr!n 1993 7 5 Hzyr!n 1967 6 fy 30 Hzyr!n 1989 6 30 Hzyr!n 1989 4 fy 30 Hzyr!n 1994 4 fy 30 Hzyr!n 1993 3 fy 19 Hzyr!n 1967 2 ywm 30 Hzyr!n 1989 2 w 6 Hzyr!n 1994 2 qbl 5 Hzyr!n 1967 2 fy 9 Hzyr!n 1967 2 fy 7 Hzyr!n 1981 2 fy 6 Hzyr!n 1994 2 fy 5 Hzyr!n 1967

<n> Hzyr!n <n>

Search query Documents January 4, 1967 8040 February 4, 1967 9270 March 4, 1967 10700 April 4, 1967 21800 May 4, 1967 14000 June 4, 1967 39300 July 4, 1967 12600 August 4, 1967 7970 September 4, 1967 7390 October 4, 1967 8800 November 4, 1967 6560 December 4, 1967 9770

Parsing Helps Decipherment

How much foreign text (running words)

Accuracy, learned bilingual dictionary

Decipherment with parsing (Dou/Knight 2013)

Spanish/English

* of most freq 5000 word types, 1-best translation in parallel dict

*

Decipherment without parsing (Dou/Knight 2012)

Adjacent bigrams Naciones Unidas dogs run perros corren blue rock no necessito United Nations piedra azul need not

Dependency bigrams Naciones Unidas run dogs corren perros rock blue necessito no Nations United piedra azul need not

?

Exploit parsers in both languages

Accurate Parsing Is Important

0.5

1.5

2.5

3.5

4.5

5.5

6.5

7.5

8.5

100k 1m 10m

Accu

racy

Num of Tokens

Adjacent

Dep1

Dep2

Dep3

Malagasy/English

Malagasy parser #1

Malagasy parser #2

Malagasy parser #3

** of most freq 5000 word types, any of 5-best in parallel dict

**

Increased parsing data by manual projection through parallel data using online dictionary Improve POS tags with parallel data. UT tagger + CMU Turboparser. CMU Turbotagger and Turboparser

Malagasy English maro many monisipaly municipal ratsy bad midadasika large vavy female lalina fundamental manokana special taitra surprised

Combining Parallel and Non-Parallel Data

small word-aligned parallel corpus

decipherment of large monolingual corpus

improve decipherment by seeding with dictionary derived from parallel data

improved machine translation





improve alignment of parallel data using large monolingual resources






improve alignment of parallel data using large monolingual resources


Joint objective function: Πe,f P(f | e) α · Πf Σe P(e) P(f | e)

parallel data

non-parallel data

find bilingual dictionary that makes both model components happy


Small bilingual Malagasy/English text (need to align words [Brown et al 93])

Large Malagasy monolingual text (need to decipher [Dou & Knight 13])

Decipherment helps Word Alignment

Decipherment helps Machine Translation

joint

Bleu

Decipherment: Next Steps

Integrate decipherment and foreign language parsing – “project English syntax through non-parallel data” – learn better Malagasy parser automatically

Fully integrated processing: – decipherment + alignment + parsing

More accurate decipherment – word classes/embeddings

Open-source tool for decipherment

Need to keep going!

Work by Others on Decipherment • "Simple Effective Decipherment via Combinatorial Optimization," (T. Berg-Kirkpatrick and D.

Klein), Proc. EMNLP, 2011. • "Deciphering Foreign Language by Combining Language Models and Context Vectors," (M.

Nuhn, A. Mauser, and H. Ney), Proc. ACL, 2012. • "Decipherment Complexity in 1:1 Substitution Ciphers," (M. Nuhn and H. Ney), Proc. ACL,

2013. • "Beam Search for Solving Substitution Ciphers," (M. Nuhn, J. Schamper, and H. Ney), Proc. ACL,

2013. • “Unsupervised Consonant-Vowel Prediction over Hundreds of Languages,” (Y. Kim and B.

Snyder), Proc. ACL, 2013. • “EM Decipherment for Large Vocabularies,” (M. Nuhn and H. Ney), Proc. ACL, 2014. • “Scalable Decipherment for Machine Translation via Hash Sampling,” (S. Ravi), Proc. ACL, 2013. • “Decipherment with a Million Random Restarts,” (T. Berg-Kirkpatrick and D. Klein), Proc.

EMNLP, 2013. • “Combining Bilingual and Comparable Corpora for Low Resource Machine Translation,” (A.

Irvine and C. Callison-Burch), Proc. WMT, 2013. • “Hallucinating Phrase Translations for Low Resource MT” (A. Irvine and C. Callison-Burch), Proc.

CoNLL, 2014. • “Solving Substitution Ciphers with Combined Language Models” (B. Hauer, R. Hayward, and G.

Kondrak), Proc. COLING, 2014.

Talk #2: Exploiting Deeper Representations cross-site collaboration (ISI/CMU): mapping language onto meaning mapping meaning onto language combining these for meaning-based MT

Why Meaning-Based MT?

• That’s what translation is: – build grammatical target text… – that preserves the meaning of the source

Oh, we got the meaning wrong…

We got the right meaning, but rendered it disfluently…

- or -

Meaning-Based MT • What content goes into the meaning

representation? Abstract Meaning Representation (AMR)

• How are meaning representations probabilistically generated, transformed, scored, ranked? How to represent knowledge that drives these processes? Automata theory, efficient algorithms

• How can a full MT system be built? Engineering, modeling, features, training

MURI

Machine Translation Automata

Phrase-based MT

Syntax-based MT

source string

target string

source string

source tree

target tree

target string

Finite-State Transducer (FST)

k

n

i

g

h

t

q k q2 *e*

q2 n q N

q i q AY q g q3 *e*

q4 t qfinal T q3 h q4 *e*

Original input: Transformation: q k

n

i

g

h

t

FST

q q2

qfinal q3 q4

k : *e*

n : N

h : *e*

g : *e* t : T

i : AY

Finite-State (String) Transducer

q2 n

i

g

h

t

q k q2 *e*

q2 n q N

q i q AY q g q3 *e*


Original input: Transformation: k

n

i

g

h

t

FST

q q2

qfinal q3 q4

k : *e*

n : N

h : *e*

g : *e* t : T

i : AY


N

q i

g

h

t

q k q2 *e*

q2 n q N

q i q AY q g q3 *e*



n

i

g

h

t

FST

q q2

qfinal q3 q4

k : *e*

n : N

h : *e*

g : *e* t : T

i : AY


q g

h

t

q k q2 *e*

q2 n q N

q i q AY q g q3 *e*


AY

N


n

i

g

h

t

FST

q q2

qfinal q3 q4

k : *e*

n : N

h : *e*

g : *e* t : T

i : AY


q3 h

t

q k q2 *e*

q2 n q N

q i q AY q g q3 *e*


AY

N


n

i

g

h

t

FST

q q2

qfinal q3 q4

k : *e*

n : N

h : *e*

g : *e* t : T

i : AY


q4 t

q k q2 *e*

q2 n q N

q i q AY q g q3 *e*


AY

N


n

i

g

h

t

FST

q q2

qfinal q3 q4

k : *e*

n : N

h : *e*

g : *e* t : T

i : AY


q k q2 *e*

q2 n q N

q i q AY q g q3 *e*


T

qfinal

AY

N

k

n

i

g

h

t

Original input: Transformation:

FST

q q2

qfinal q3 q4

k : *e*

n : N

h : *e*

g : *e* t : T

i : AY

S

NP VP

PRO

he

VBZ

enjoys

NP

VBG

listening

VP

P

to

NP

SBAR

music


q S

NP VP

PRO

he

VBZ

enjoys

NP

VBG

listening

VP

P

to

NP

SBAR

music

Top-Down Tree Transducer (W. Rounds 1970; J. Thatcher 1970)

S

NP VP

PRO

he

VBZ

enjoys

NP

VBG

listening

VP

P

to

NP

SBAR

music


q S

NP VP

PRO

he

VBZ

enjoys

NP

VBG

listening

VP

P

to

NP

SBAR

music


q S

x0:NP VP

s x0, wa, r x2, ga, q x1

x1:VBZ x2:NP

0.2

S

NP VP

PRO

he

VBZ

enjoys

NP

VBG

listening

VP

P

to

NP

SBAR

music


s NP

PRO

he

q VBZ

enjoys

r NP

VBG

listening

VP

P

to

NP

SBAR

music

, ,


, wa , ga

S

NP VP

PRO

he

VBZ

enjoys

NP

VBG

listening

VP

P

to

NP

SBAR

music


s NP

PRO

he

q VBZ

enjoys

r NP

VBG

listening

VP

P

to

NP

SBAR

music

, ,


, wa , ga

s NP

PRO

kare

he

0.7

S

NP VP

PRO

he

VBZ

enjoys

NP

VBG

listening

VP

P

to

NP

SBAR

music


q VBZ

enjoys

r NP

VBG

listening

VP

P

to

NP

SBAR

music

, kare wa ,


, , ga

S

NP VP

PRO

he

VBZ

enjoys

NP

VBG

listening

VP

P

to

NP

SBAR

music

kare kiku ongaku o wa daisuki desu ga no

Original input: Final output:

, , , , , , , ,


General-Purpose Algorithms for Tree Automata String Automata

Algorithms Tree Automata

Algorithms N-best … … paths through an WFSA

(Viterbi, 1967; Eppstein, 1998) … trees in a weighted forest (Jiménez & Marzal, 2000; Huang & Chiang, 2005)

EM training Forward-backward EM (Baum/Welch, 1971; Eisner 2003)

Tree transducer EM training (Graehl & Knight, 2004)

Determinization … … of weighted string acceptors (Mohri, 1997)

… of weighted tree acceptors (Borchardt & Vogler, 2003; May & Knight, 2005)

Intersection WFSA intersection Tree acceptor intersection

Applying transducers string WFST WFSA tree TT weighted tree acceptor

Transducer composition WFST composition (Pereira & Riley, 1996)

Many tree transducers not closed under composition (Maletti et al 09)

General-purpose tools Carmel, OpenFST Tiburon (May & Knight 10)

Machine Translation

Phrase-based MT

Syntax-based MT

Meaning-based MT source string

meaning graphs

target string

source string

target string

source string

source tree

target tree

target string

source tree

target tree

Semantic Graphs “Pascale was charged with public intoxication and resisting arrest.”

15,000 sentences have been annotated with Abstract Meaning Representation (AMR) in [Banarescu et al 13].

Abstract Meaning Representation (AMR) Pascale was charged with public intoxication and resisting arrest. (c / charge-05 :ARG1 (p / person :name (n / name :op1 “Pascale”)) :ARG2 (a / and :op1 (i / intoxicate-01 :ARG1 p :location (p2 / public)) :op2 (r / resist-01 :ARG0 p :ARG1 (a / arrest-01 :ARG1 p))))

PropBank frames

Named entities of 80 types

Entities play multiple roles (coreference)

100 semantic roles

Implicit roles

Modality

Negation

Questions

Full exploitation of predicates

Bond investors might not react. (p / possible :domain (r / react-01 :polarity – :arg0 (p2 / person :arg0-of (i / invest-01 :arg1 (b / bond)))

Abstraction from POS Light Verbs Cause Sub-events etc.

Graph Automata for NLU and NLG String Automata

Algorithms Tree Automata

Algorithms Graph Automata

Algorithms N-best answer extraction

… paths through an WFSA (Viterbi, 1967; Eppstein, 1998)

… trees in a weighted forest (Jiménez & Marzal, 2000; Huang & Chiang, 2005)

Investigating: • Linguistically adequate representations • Efficient algorithms Using them in: • Text Meaning (NLU) • Meaning Text (NLG) • Meaning-based MT

Unsupervised EM training

Forward-backward EM (Baum/Welch, 1971; Eisner 2003)

Tree transducer EM training (Graehl & Knight, 2004)

Determinization, minimization

… of weighted string acceptors (Mohri, 1997)

… of weighted tree acceptors (Borchardt & Vogler, 2003; May & Knight, 2005)

Intersection WFSA intersection Tree acceptor intersection

Application of transducers

string WFST WFSA tree TT weighted tree acceptor

Composition of transducers

WFST composition (Pereira & Riley, 1996)

Many tree transducers not closed under composition (Maletti et al 09)

Software tools Carmel, OpenFST Tiburon (May & Knight 10) ISI jointly with CMU & ND

Mapping Between Meaning and Text

the boy wants to see WANT

BOY

SEE

instance

instance

instance agent

patient

agent

Umuhungu arashaka kubona.


the boy wants to be seen WANT

BOY

SEE

instance

instance

instance agent

patient

patient

Umuhungu arashaka kubonwa.


the boy wants to see the girl WANT

BOY

SEE

instance

instance

instance agent

patient

patient

GIRL

instance

agent

Umuhungu arashaka kubona umukobwa


the boy wants to see himself

WANT

BOY

SEE

instance

instance

instance agent

patient

patient agent

Umuhungu arashaka kwibona.

DAG-to-Tree Transducer [Kamimura & Slutski 82; Quernheim & Knight 2012ab]

• Bottom-up transformation of graph to tree

Hyperedge Replacement Grammar [Drewes et al 97; Chiang et al 2013]

probabilistic rules

initial graph

final graph

HRG Derivation

instance ARG0

WANT

B

ARG1

instance

ARG0

BELIEVE

ARG1

G

instance

WANT

ARG1

= boy wants girl to believe that he is wanted

LET’S DERIVE THIS:

HRG Derivation

instance ARG0

WANT

B

ARG1

instance

ARG0

BELIEVE

ARG1

G

instance

WANT

ARG1

instance

ARG0

WANT

B

X

“the boy wants something involving himself”

ARG1


HRG Derivation

instance ARG0

WANT

B

ARG1

instance

ARG0

BELIEVE

ARG1

G

instance

WANT

ARG1


instance

ARG0

WANT

B

X

ARG1


HRG Derivation

instance ARG0

WANT

B

ARG1

instance

ARG0

BELIEVE

ARG1

G

instance

WANT

ARG1

instance ARG0

WANT

B

X instance

ARG0

BELIEVE

G

“the boy wants the girl to believe something involving him”

ARG1


HRG Derivation

instance ARG0

WANT

B

ARG1

instance

ARG0

BELIEVE

ARG1

G

instance

WANT

ARG1

instance ARG0

WANT

B

X instance

ARG0

BELIEVE

G

“something involving B”

ARG1


HRG Derivation

instance ARG0

WANT

B

ARG1

instance

ARG0

BELIEVE

ARG1

G

instance

WANT

ARG1

instance ARG0

WANT

B

instance

ARG0

BELIEVE

G

instance

WANT

ARG1

ARG1

ARG1

FINISHED!


Synchronous Hyperedge Replacement Grammar (SHRG)

[Chiang et al 13]

• Each SHRG rule outputs a graph fragment and a tree fragment simultaneously

• Used for transducing meaning to language, and vice-versa

SHRG Derivation

S

wants B INF

instance ARG0

WANT

B

X


SHRG Derivation

S

wants B INF

to believe G S

instance ARG0

WANT

B

X

ARG1

instance

ARG0

BELIEVE

G

“something involving B”

SHRG Derivation

instance ARG0

WANT

B

ARG1

instance

ARG0

BELIEVE

G

instance

WANT

ARG1

S

wants B INF

to believe G S

is wanted he

FINISHED!

Formal Properties of HRG Acceptors

novel algorithm

very pleasant!

mildly unpleasant

d=input graph outdegree T=treewidth complexity

Formal Properties of SHRG transducers

New device

New Work

• DAG-to-tree transducer formalism – Prague JHU workshop, summer 2014

• David Chiang, Dan Gildea (US) • Frank Drewes, Giorgio Satta (Europe)

• Linguistic suitability of formalisms – Explain meaning/string corpora

Strings Graphs

FSA CFG DAG acceptor HRG

probabilistic yes yes yes yes

intersects with finite-

state yes yes yes yes

EM training yes yes yes yes

transduction O(n) O(n3) O(|Q|T+1n) O((3dn)T+1)

implemented yes yes yes yes

Results for DAG automata

d = graph degree for AMR, high in practice T = treewidth complexity for AMR, low in practice (2-3)

efficient algorithms for k-best, EM, etc + tools + high impact on practical machine translation

invented tree transducers

worked out basic theoretical properties of tree transducers

over 30 years

invented graph grammars & basic recognition invented synchronous graph grammars

for transduction. also: probabilities, training algorithms, theorems, toolkits.

improved algorithms (Prague 2014) summer workshop

aiming at high impact on MT

Linguistic Suitability of Formalisms

• Concisely capture all of the graph/string pairs in a corpus

• By manually building linguistic mapping knowledge

then, measure coverage & conciseness

“Stress Test” Data

• 10,000 smallest semantic graphs composed of: – Predicates BELIEVE and WANT – Entities BOY and GIRL

• Plus 10 English string realizations of each graph

He wants her to believe he wants her.

Solutions We Designed and Tested

All transducer cascades are bidirectional: we run forwards for NL generation task, and backwards for NL understanding task.

graph SHRG

string tree graph

string

graph DAG2Tree string tree xLNTs (take yield)

tree DAG2Tree (tree-ify)

xLNT (introduce

verbs)

xLNTs (take yield) tree

xLNT (introduce pronouns)

Empirical Results

Data is available -- amr.isi.edu/download/boygirl.tgz We hope that others will continue to design more elegant, efficient formalisms to capture the meaning/text relation!

Novel Contributions

• Hyperedge Replacement Grammar (HRG) – synchronous version for transduction [ACL 2013] – proof-of-concept MT system [COLING 2012]

• Novel algorithm for graph parsing [ACL 2013]

• Empirical fitness to linguistic data [LREC 2014]

• Map of theoretical and computational properties – closure, complexity [FSMNLP 2015, subm.]

Bolinas Graph Processing Toolkit

Graph Formalisms: Next Steps

• Graph transduction workshop – One week in Dagstuhl, Germany (March 2015) – 35 attendees from Theory and NLP – Organizers: F. Drewes, K. Knight, M. Kuhlman

• Automatic extraction/use of graph grammars

– Hook up with manually-created AMR bank of 15,000 sentences (fiction, news, blog)

• AMR to English generation • English to AMR parsing