semantic representation and formal transformation

Semantic Representation and Formal Transformation

kevin knightusc/isi

MURI meeting, CMU, Nov 3-4, 2011

Machine Translation

Phrase-based MT

Syntax-based MT

Meaning-based MTsourcestring

meaningrepresentation

targetstring

sourcestring

targetstring

sourcestring

sourcetree

targettree

targetstring

sourcetree

targettree

05

101520253035 NIST 2009 c2e

Meaning-based MT

• Too big for this MURI:– What content goes into the meaning

representation?• linguistics, annotation

– How are meaning representations probabilistically generated, transformed, scored, ranked?

• automata theory, efficient algorithms– How can a full MT system be built and tested?

• engineering, language modeling, features, training

Meaning-based MT

• Too big for this MURI:– What content goes into the meaning

representation?• linguistics, annotation

– How are meaning representations probabilistically generated, transformed, scored, ranked?

• automata theory, efficient algorithms– How can a full MT system be built and tested?

• engineering, language modeling, features, trainingLanguage-independent theory.But driven by practical desires.

Automata Frameworks

• How to represent and manipulate linguistic representations?

• Linguistics, NLP, and Automata Theory used to be together (1960s, 70s)– Context-free grammars were invented to model human

language– Tree transducers were invented to model transformational

grammar• They drifted apart• Renewed connections around MT (this century)• Role: greatly simplify systems!

Finite-State Transducer (FST)

k

n

i

g

h

t

q k q2 *e*

q2 n q N

q i q AYq g q3 *e*

q4 t qfinal Tq3 h q4 *e*

Original input: Transformation:q k

n

i

g

h

t

FST

qq2

qfinalq3 q4

k : *e*

n : N

h : *e*

g : *e*t : T

i : AY

Finite-State (String) Transducer

q2 n

i

g

h

t

q k q2 *e*

q2 n q N

q i q AYq g q3 *e*


Original input: Transformation:k

n

i

g

h

t

FST

qq2

qfinalq3 q4

k : *e*

n : N

h : *e*

g : *e*t : T

i : AY


N

q i

g

h

t

q k q2 *e*

q2 n q N

q i q AYq g q3 *e*



n

i

g

h

t

FST

qq2

qfinalq3 q4

k : *e*

n : N

h : *e*

g : *e*t : T

i : AY


q g

h

t

q k q2 *e*

q2 n q N

q i q AYq g q3 *e*


AY

N


n

i

g

h

t

FST

qq2

qfinalq3 q4

k : *e*

n : N

h : *e*

g : *e*t : T

i : AY


q3 h

t

q k q2 *e*

q2 n q N

q i q AYq g q3 *e*


AY

N


n

i

g

h

t

FST

qq2

qfinalq3 q4

k : *e*

n : N

h : *e*

g : *e*t : T

i : AY


q4 t

q k q2 *e*

q2 n q N

q i q AYq g q3 *e*


AY

N


n

i

g

h

t

FST

qq2

qfinalq3 q4

k : *e*

n : N

h : *e*

g : *e*t : T

i : AY


q k q2 *e*

q2 n q N

q i q AYq g q3 *e*


T

qfinal

AY

N

k

n

i

g

h

t

Original input: Transformation:

FST

qq2

qfinalq3 q4

k : *e*

n : N

h : *e*

g : *e*t : T

i : AY

Transliteration

Angela Knight

a n ji ra na i to

transliteration

Frequently occurring translation problem for languageswith different sound systems and character sets.(Japanese, Chinese, Arabic, Russian, English…)

Can’t be solved by dictionary lookup.

Transliteration

Angela KnightWFST

7 input symbols 13 output symbols

Transliteration

Angela Knight

WFST B

WFSA A

WFST D

AE N J EH L UH N AY T

WFST C a n j i r a n a i t o

WFST B

WFSA A

WFST D

WFST C a n j i r a n a i t o

AE N J IH R UH N AY TAH N J IH L UH N AY T OH

+ millions more

+ millions more

+ millions more

DECODE

General-Purpose Algorithms for String Automata

N-best … … paths through an WFSA (Viterbi, 1967; Eppstein, 1998)

EM training Forward-backward EM (Baum & Welch, 1971; Eisner 2001)

Determinization … … of weighted string acceptors (Mohri, 1997)

Intersection WFSA intersection

Application string WFST WFSA

Transducer composition WFST composition (Pereira & Riley, 1996)

General-purpose toolkit Carmel (Graehl & Knight 97), OpenFST (Google, via AT&T), ...

S

NP VP

PRO

he

VBZ

enjoys

NP

VBG

listening

VP

P

to

NP

SBAR

music


q S

NP VP

PRO

he

VBZ

enjoys

NP

VBG

listening

VP

P

to

NP

SBAR

music

Top-Down Tree Transducer(W. Rounds 1970; J. Thatcher 1970)

S

NP VP

PRO

he

VBZ

enjoys

NP

VBG

listening

VP

P

to

NP

SBAR

music


q S

NP VP

PRO

he

VBZ

enjoys

NP

VBG

listening

VP

P

to

NP

SBAR

music


q S

x0:NP VP

s x0, wa, r x2, ga, q x1

x1:VBZ x2:NP

0.2

S

NP VP

PRO

he

VBZ

enjoys

NP

VBG

listening

VP

P

to

NP

SBAR

music


s NP

PRO

he

q VBZ

enjoys

r NP

VBG

listening

VP

P

to

NP

SBAR

music

, ,


, wa ,ga

S

NP VP

PRO

he

VBZ

enjoys

NP

VBG

listening

VP

P

to

NP

SBAR

music


s NP

PRO

he

q VBZ

enjoys

r NP

VBG

listening

VP

P

to

NP

SBAR

music

, ,


, wa ,ga

s NP

PRO

kare

he

0.7

S

NP VP

PRO

he

VBZ

enjoys

NP

VBG

listening

VP

P

to

NP

SBAR

music


q VBZ

enjoys

r NP

VBG

listening

VP

P

to

NP

SBAR

music

,kare wa,


, ,ga

S

NP VP

PRO

he

VBZ

enjoys

NP

VBG

listening

VP

P

to

NP

SBAR

music

kare kikuongaku owa daisuki desugano

Original input: Final output:

, , , , , , ,,


Top-Down Tree Transducer

• Introduced by Rounds (1970) & Thatcher (1970)“Recent developments in the theory of automata have pointed

to an extension of the domain of definition of automata from strings to trees … parts of mathematical linguistics can be formalized easily in a tree-automaton setting …”

(Rounds 1970, “Mappings on Grammars and Trees”, Math. Systems Theory 4(3))

• Large theory literature– e.g., Gécseg & Steinby (1984), Comon et al (1997)

• Once again re-connecting with NLP practice– e.g., Knight & Graehl (2005), Galley et al (2004, 2006),

May & Knight (2006, 2010), Maletti et al (2009)

Tree Transducers Can be Extracted from Bilingual Data (Galley et al, 04)

i felt obliged to do my part

我有责任尽一份力

TREE TRANSDUCER RULES:

VBD(felt) 有VBN(obliged) 责任VB(do) 尽NN(part) 一份NN(part) 一份力VP-C(x0:VBN x1:SG-C) x0 x1VP(TO(to) x0:VP-C) x0 …S(x0:NP-C x1:VP) x0 x1

SNP-C VP VP-C VBD SG-C VP VBN TO VP-C VB NP-CNPB NPB

PRP PRP$ NN

这 7 人中包括来自法国和俄罗斯的宇航员 .

Syntax-Based Decoding


RULE 1:DT(these) 这

RULE 2:VBP(include) 中包括

RULE 6:NNP(Russia) 俄罗斯

RULE 4:NNP(France) 法国

RULE 8:NP(NNS(astronauts)) 宇航 , 员

RULE 5:CC(and) 和

RULE 9:PUNC(.) .

“these” “Russia” “astronauts” “.”“include” “France” “and”







RULE 5:CC(and) 和

RULE 9:PUNC(.) .


RULE 13:NP(x0:NNP, x1:CC, x2:NNP) x0 , x1 , x2

“France and Russia”

“include”“these” “France” “and” “Russia” “astronauts” “.”







RULE 5:CC(and) 和

RULE 9:PUNC(.) .



RULE 11:VP(VBG(coming), PP(IN(from), x0:NP)) 来自 , x0


“coming from France and Russia”

“these” “Russia” “astronauts” “.”“include” “France” “&”







RULE 5:CC(and) 和

RULE 9:PUNC(.) .




RULE 16:NP(x0:NP, x1:VP) x1 , 的 , x0

“astronauts coming fromFrance and Russia”










RULE 5:CC(and) 和

RULE 9:PUNC(.) .





RULE 14:VP(x0:VBP, x1:NP) x0 , x1

“include astronauts coming fromFrance and Russia”










RULE 5:CC(and) 和

RULE 9:PUNC(.) .


RULE 10:NP(x0:DT, CD(7), NNS(people) x0 , 7 人


RULE 15:S(x0:NP, x1:VP, x2:PUNC) x0 , x1 , x2



RULE 14:VP(x0:VBP, x1:NP) x0 , x1

“These 7 people include astronauts coming from France and Russia”




“these 7 people”

“include astronauts coming fromFrance and Russia”


These 7 people include astronauts coming from France and Russia .

DT CD VBP NNS IN NNP CC NNP PUNC

NPNP NP

VP

NP

VP

S

NNS VBG

PP

NPNP

DerivedEnglish Tree

General-Purpose Algorithms for Tree AutomataString Automata

AlgorithmsTree Automata

AlgorithmsN-best … … paths through an WFSA

(Viterbi, 1967; Eppstein, 1998)… trees in a weighted forest (Jiménez & Marzal, 2000; Huang & Chiang, 2005)

EM training Forward-backward EM (Baum/Welch, 1971; Eisner 2003)

Tree transducer EM training (Graehl & Knight, 2004)

Determinization … … of weighted string acceptors (Mohri, 1997)

… of weighted tree acceptors (Borchardt & Vogler, 2003; May & Knight, 2005)

Intersection WFSA intersection Tree acceptor intersection

Applying transducers string WFST WFSA tree TT weighted tree acceptor

Transducer composition WFST composition (Pereira & Riley, 1996)

Many tree transducers not closed under composition (Maletti et al 09)

General-purpose tools Carmel, OpenFST Tiburon (May & Knight 10)

Machine Translation

Phrase-based MT

Syntax-based MT

Meaning-based MTsourcestring

meaningrepresentation

targetstring

sourcestring

targetstring

sourcestring

sourcetree

targettree

targetstring

sourcetree

targettree

Five Equivalent Meaning Representation Formats

(w / WANT :agent (b / BOY) :patient (g / GO :agent b)))

w, b, g : instance(w, WANT) ^ instance(g, GO) ^ instance(b, BOY) ^ agent(w, b) ^ patient(w, g) ^ agent(g, b)

E

WANT

BOY

GO

instance

instance

instanceagent

patient

agent

((x0 instance) = WANT((x1 instance) = BOY((x2 instance) = GO((x0 agent) = x1((x0 patent) = x2((x2 agent) = x1

instance: WANTagent:patient: instance: GO

agent:

instance: BOY1

1

LOGICAL FORM PATH EQUATIONS

FEATURE STRUCTURE

DIRECTED ACYCLIC GRAPH

PENMAN

“The boy wantsto go.”

Example“Government forces closed on rebel outposts on Thursday, showering the western mountain city of Zintan with missiles and attacking insurgents holed up near the Tunisian border, according to rebel sources.”

(s / say :agent (s2 / source :mod (r / rebel)) :patient (a / and :op1 (c / close-on :agent (f / force :mod (g / government)) :patient (o / outpost :mod (r2 / rebel)) :temporal-locating (t / thursday)) :op2 (s / shower :agent f :patient (c / city :mod (m / mountain) :mod (w / west) :name "Zintan") :instrument (m2 / missile)) :op3 (a / attack :agent f :patient (i / insurgent :agent-of (h / hole-up :pp-near (b / border :gpi (c2 / country :name "Tunisia"]

Slogan: “more logical than a parse tree”

General-Purpose Algorithms for Feature Structures (Graphs)String Automata

AlgorithmsTree Automata

AlgorithmsGraph Automata

Algorithms?N-best … … paths through an WFSA

(Viterbi, 1967; Eppstein, 1998)… trees in a weighted forest (Jiménez & Marzal, 2000; Huang & Chiang, 2005)



Determinization… … of weighted string acceptors (Mohri, 1997)



Applying transducers

string WFST WFSA tree TT weighted tree acceptor

Transducer composition

WFST composition (Pereira & Riley, 1996)


General tools Carmel, OpenFST Tiburon (May & Knight 10)

Automata Frameworks• Hyperedge-replacement graph grammars

– (Drewes et al)• DAG acceptors

– (Hart 75)• DAG-to-tree transducers

– (Kamimura & Slutski 82)

Mapping Between Meaning and Text

the boy wants to seeWANT

BOY

SEE

instance

instance

instanceagent

patient

agent

foreigntext


the boy wants to be seenWANT

BOY

SEE

instance

instance

instanceagent

patient

patient

foreigntext


the boy wants the girlto be seen

WANT

BOY

SEE

instance

instance

instanceagent

patient

patient

GIRL

instance

foreigntext


the boy wants to see the girlWANT

BOY

SEE

instance

instance

instanceagent

patient

patient

GIRL

instance

agent

foreigntext


the boy wants to see himself

WANT

BOY

SEE

instance

instance

instanceagent

patient

patientagent

foreigntext

Bottom-Up DAG-to-Tree Transduction(Kamimura & Slutski 82)

PROMISE

BOY

GO

instance

instance

instanceagent

patient

recipient

GIRL

instance

agent


PROMISE

BOY

GO

instance

instance

instance

agent

patient

recipient

GIRL

instance


PROMISE

BOY

GO

instance

instance

instance

agent

patient

recipient

instance

q.NN | girl


PROMISE

GO

instance

instance

instance

agent

patient

recipient

instance

q.NN | girl

q.NN | boy


PROMISE

instance

instance

instance

agent

patient

recipient

instanceq.VBZ | goesq.NN

| girl

q.NN | boy


PROMISE

instance

instance

instance

agent

patient

recipient

instanceq.VB | goq.NN

| girl

q.NN | boy


instance

instance

instance

agent

patient

recipient

q.NN | girl

instanceq.VB | go

q.VBD | promised

q.NN | boy

General-Purpose Algorithms for Feature Structures

String Automata Algorithms

Tree Automata Algorithms

Graph Automata Algorithms?

N-best … … paths through an WFSA (Viterbi, 1967; Eppstein, 1998)

… trees in a weighted forest (Jiménez & Marzal, 2000; Huang & Chiang, 2005)



Determinization… … of weighted string acceptors (Mohri, 1997)



Applying transducers

string WFST WFSA tree TT weighted tree acceptor

Transducer composition

WFST composition (Pereira & Riley, 1996)


General tools Carmel, OpenFST Tiburon (May & Knight 10)

Automata for Statistical MTUsed in SMT

Automata Framework Devised1960 1970 1980 1990 2000

1990

2000

2010

2020

2010

FST

TDTT

DAG

semantic representation and formal transformation

Documents

q2 n q nq i q ayq g

q4 t qfinal tq3 h q4

q knightfstqq2qfinalq3q4k

meaning representation

aynoriginal input

automata theory

mt system

tqfinalaynknightoriginal