the nespole interchange format (if)

The NESPOLE Interchange Format (IF)

Lori Levin, Emanuele Pianta, Donna Gates, Kay Peterson, Dorcas Wallace, Herve Blanchon,

Roldano Cattoni, Jean-Philippe Gibaud, Chad Langley, Alon Lavie, Nadia Mana, Fabio Pianesi

Outline• Approaches to MT: Interlingua, Transfer,

Direct.• The NESPOLE! Interlingua.

– Overview and motivation – Linguistic coverage

• Tools and resources.• Evaluating an interlingua

– Coverage: how do we measure coverage of the domain

– Reliability: so that an analyzer written by one person in Italy can work with a generator written by someone he has never met in Korea.

– Scalability: move to broader semantic domains without a constant increase in the amount of work.

Outline

• Approaches to MT: Interlingua, Transfer, Direct.

• The NESPOLE! Interlingua.– Overview and motivation – Linguistic coverage

• Tools and resources.• Evaluating an interlingua

– Coverage– Reliability– Scalability

What is an interlingua?

• Representation of meaning or speaker intention.

• Sentences that are equivalent for the translation task have the same interlingua representation.

The room costs 100 Euros per night.

The room is 100 Euros per night.

The price of the room is 100 Euros per night.

Mi chiamo Alex Waibel My name is Alex Waibel.

Give-information+personal-data (name=alex_waibel)

[s [vp accusative_pronoun “chiamare” proper_name]]

[s [np [possessive_pronoun “name”] ] [vp “be” proper_name]]

Direct

Transfer

Interlingua

Vaquois MT Triangle

Other Approaches to Machine Translation

• Direct: – Very little analysis of the source language.

• Transfer: – Analysis of the source language.– The structure of the source language input may

not be the same as the structure of the target language sentence.

– Transfer rules relate source language structures to target language structures.

Note

• Some transfer systems may produce a more detailed meaning representation than some interlingua systems.

• The difference is whether translation equivalents in the source and target languages are related by a single canonical representation.

Multilingual Translation with an Interlingua

Japanese

Arabic

Chinese(input sentence)

San1 tian1 qian2, wo3 kai1 shi3 jue2 de2 tong4

English

French

German

Italian

Korean

Arabic

Chinese(paraphrase)wo3 yi3 jin1 tong4 le4 san1 tian1 English

(output sentence)The pain started three days ago.

FrenchGerman

ItalianJapanese

Korean

Analyzers

Generators

Spanish

Spanish

Catalan

Catalan

Interlinguagive-information+onset+body-state (body-state-spec=pain, time=(interval=3d, relative=before))

Multilingual translation with transfer

• Transfer-rules-1: Arabic-Catalan

• Transfer-rules-2: Catalan-Arabic

• Transfer-rules-3: Arabic-Chinese

• Transfer-rules-4: Chinese-Arabic

• Transfer-rules-5: Arabic-English

• Transfer-rules-6: English-Arabic

• Etc.

Advantages of Interlingua

• Add a new language easily – get all-ways translation to all previous

languages by adding one grammar for analysis and one grammar for generation

• Mono-lingual development teams.• Paraphrase

– Generate a new source language sentence from the interlingua so that the user can confirm the meaning

Disadvantages of Interlingua• “Meaning” is arbitrarily deep.

– What level of detail do you stop at?

• If it is too simple, meaning will be lost in translation.

• If it is too complex, analysis and generation will be too difficult.

• Should be applicable to all languages.

• Human development time.

Interlingual MT Systems

• University of Maryland – Lexical Conceptual Structure (Dorr)

• Carnegie Mellon– Kantoo (Mitamura and Nyberg)– Nespole/C-STAR (Waibel, Levin, Lavie)

• UNL (Universal Networking Language)• Microcosmos (Nirenburg)• Verbmobil – Domain actions (Block)

Outline

• Approaches to MT: Interlingua, Transfer, Direct.• The NESPOLE! Interlingua.

– Overview and motivation

– Linguistic coverage

• Tools and resources.• Evaluating an interlingua

– Reliability

– Coverage

A Travel DialogueTranslated from Italian

A: Albergo Gabbia D’Oro. Good evening.B: My name is Anna Maria DeGasperi. I’m calling from

Rome. I wish to book two single rooms.A: Yes.B: From Monday to Friday the 18th, I’m sorry, to Monday

the 21st.A: Friday the 18th of June.B: The 18th of July. I’m sorry.A: Friday the 18th of July to, you were saying, Sunday.B: No. Through Monday the 21st.

A Travel Dialogue(Continued)

B: So with departure on Tuesday the 22nd.A: Then leaving on the 22nd. Yes. We have two singles

certainly. B: Yes.A: Would you like breakfast?B: Is it possible to have all meals?A: No. We serve meals only in the evening.B: Ok. If you can do breakfast and dinner.A: Ok.B: Do you need a deposit?

A Travel Dialogue(Continued)

A: You can give me your credit card number.B: Ok. Just a moment. Ok. My name is Anna Maria

DeGaperi. The card is 005792005792.A: Good.B: Expiration 2002.A: 2002. Good. Thank you. We need a confirmation on the

18th of July before 6pm.B: Goodbye.A: Thanks. Goodbye.B: Thanks. Goodbye.

A Non-Task-Oriented Dialogue(We can’t translate this.)

A: Are you cooking?B: My father is cooking. I’m cleaning. I just finished

cleaning the bathroom.A: Look. What do you know about Monica?B: I don’t know anything. Look. I don’t know anything. A: You don’t know anything? I wrote her three weeks ago,

but if she hasn’t received the letter, they would have returned it. I hope she received it.

B: Because Celia told me that the address that Monica had given us was wrong. She said that if I was going to write to her, well, ….

From the Spanish CallHome corpus: unlimited conversation between family members.

The Ideal MT System…

• Fully automatic

• High quality

• Domain independent (any topic)

….isn’t within the current state-of-the-art.

Design Principles of the Interchange Format

• Suitable for task oriented dialogue• Based on speaker’s intent, not literal meaning

– Can you pass the salt is represented only as a request for the hearer to perform an action, not as a question about the hearer’s ability.

• Abstract away from the peculiarities of any particular language – resolve translation mismatches.

Translation Mismatches

• Sentences that are translation-equivalents in two languages do not have the same syntactic structure or predicate-argument structure. (Unitrans; Eurotra)– I like to swim.– I swam across the river.– Sue met with Sam/Sue met Sam.

Design Principles (continued)

• Domain independent framework with domain-specific parts

• Simple and reliable enough to use:

– at multiple research sites with high intercoder agreement.

– with widely varying type of parsers and generators.

• Allow robust language engines

– Underspeicification must be possible.

– Fragments must be represented.

Speech Acts:Speaker intention vs literal meaning

• Can you pass the salt?

• Literal meaning: The speaker asks for information about the hearer’s ability.

• Speaker intention: The speaker requests the hearer to perform an action.

Remember this term:

Domain Action

Domain Actions: Extended, Domain-Specific Speech Acts

give-information+existence+body-state

It hurts.

give-information+onset+body-object

The rash started three days ago.

request-information+availability+room

Are there any rooms available?

request-information+personal-data

What is your name?

Domain Actions:Extended, Domain-Specific Speech Acts

• In domain.– I sprained my ankle yesterday.– When did the headache start?

• Out of domain– Yesterday I slipped in the driveway on my way

to the garage.– The headache started after my boss noticed that

I deleted the file.

Formulaic Utterances

• Good night.

• tisbaH cala xEr

waking up on good

• Romanization of Arabic from CallHome Egypt

Same intention, different syntax

• rigly bitiwgacny my leg hurts• candy wagac fE rigly I have pain in my leg• rigly bitiClimny my leg hurts• fE wagac fE rigly there is pain in my leg• rigly bitinqaH calya my leg bothers on me

Romanization of Arabic from CallHome Egypt.

Language Neutrality

• Comes from representing speaker intention rather than literal meaning for formulaic and task-oriented sentences.

How about … suggestionWhy don’t you… suggestionCould you tell me… request info.

I was wondering… request info.

Domain Action Interlingua and Lexical Semantic Interlingua

and how will you be paying for this

Domain Action representation:Domain Action representation: a:request-information+payment (method=question)a:request-information+payment (method=question)

Lexical Semantic representation:Lexical Semantic representation: predicate: pay time: future agent: hearer product: distance: proximate, type: demonstrative manner: question

Complementary Approaches

• Domain actions – limited to task oriented sentences

• Lexical Semantics– less appropriate for formulaic speech acts that should not be translated literally

Components of the Interchange Format

speaker a: a: (agent)

speech act give-informationgive-information

concept* +availability+room+availability+room

argument* (room-type=(single & double), (room-type=(single & double), time=md12)time=md12)

Components of IFas of February 2002

• 61 speech acts give-information

– domain independent,

– 20 are dialog managing

• 108 concepts availability, accommodation

– mostly domain dependent

• 304 arguments room-type, time

– domain dependent and independent

• 7,652 values single, double, 12th

Examples no that’s not necessary c:negatec:negate

yes I am c:affirmc:affirm

my name is alex waibel c:give-information+personal-data (person-name=(given-name=alex, c:give-information+personal-data (person-name=(given-name=alex,

family-name=waibel))family-name=waibel))

and how will you be paying for this a:request-information+payment (method=question)a:request-information+payment (method=question)

I have a mastercard c:give-information+payment (method=mastercard)c:give-information+payment (method=mastercard)

Outline

• Approaches to MT: Interlingua, Transfer, Direct.• The NESPOLE! Interlingua.

– Overview and motivation – Linguistic coverage

• Tools and resources.• Evaluating an interlingua

– Reliability– Coverage– Scalability

Conventional Speec Acts

thank you. c:thank

can I help you ?a:offer+help (who=i, to-whom=you)

<uh> my name is Chadc:give-information+personal-data (person-name=(given-name=chad))

Fragments: ellipsis

<B> and <uh> <hm> in a restaurant. a:give-information+concept (conjunction=discourse, location=(restaurant, identifiability=no))

<uh> which town? c:request-information+concept (concept-spec=(town, identifiability=question))

Fragments: abandoned

• You should

a: suggest+concept (who=you)

• What should I

c: request-suggestion+concept (who=I)

Coordination of Sentences

I want to go to France and I would prefer to leave today.

c:give-information+disposition+trip (destination=(object-name=france), disposition=(who=i, desire))

c:give-information+disposition+departure (conjunction=discourse, time=(relative-time=today), disposition=(who=i, preference))

Coordination of sentences, reducedI want to leave Pittsburgh at 2 and return from Rome at 5.

c:give-information+disposition+departure (conjunction=discourse, origin=(object-name=pittsburgh), disposition=(who=i, desire), time=(clock=(hours=2)))

c:give-information+trip (conjunction=discourse, factuality=unspecified, trip-spec=return, origin=(object-name=rome), time=(clock=(hours=5)))

Conjunctive Set

I like festivals and plays.

c:give-information+disposition+event (... event-spec=(operator=conjunct, [(festival, quantity=plural), (play, quantity=plural)]))

Conjunction of modifiers

I prefer red and blue cars.c:give-information+disposition+vehicle (... vehicle-spec=(car, quantity=plural, color=(operator=conjunct, [red, blue])))

Disjunctive Sets

I prefer hotels or cabins.

c:give-information+disposition+accommodation (... accommodation-spec=(operator=disjunct, [(hotel, quantity=plural), (cabin, quantity=plural)]))

Contrastive Set

I like hotels but not cabins.

c:give-information+disposition+accommodation (... accommodation-spec=(operator=contrast, [(hotel, quantity=plural), (polarity=negative, cabin, quantity=plural)]))

Attitudes: often a source of mismatches

• Disposition• Eventuality• Evidentiality• Feasibility• Knowledge• Obligation• Main verbs in English that occur in other

languages as affixes, adverbs, or other construtions that are not clearly bi-clausal.

Disposition

<uhm> <P> and I would like to arrive <P> around September ninth.c:give-information+disposition+arrival (disposition=(who=i, desire), /* attitude */ conjunction=discourse, /* rhetorical information */ time=(exactness=approximate, month=9, md=9)) /* time */

Disposition

• I would like to stay in a hotel.– Disposition=desire

• I hate mushroom picking.– Disposition=dislike

• I am waiting to see the circle.– Disposition=expectation

• But wouldn’t matter.– Disposition=indifferent

• When do you plan on arriving in Pittsburgh?– Disposition=intention

Eventuality

• It is possible I may be arriving earlier.

give-information+eventuality+arrival

(eventuality=possible)

• I’m sure that they will arrive tomorrow.

• Maybe there is something beautiful to see.

• It is not impossible.

Evidentiality: Source of information

• Apparently there are many castles.

Give-information+evidentiality+attraction

• I heard there are many castles.

• I noticed there is a winter package available.

• I’ve been told I must leave before ten.

Feasibility

• You can rent skis at the resort.

Give-information+feasibility+rent+equipment

(feasibility=feasible….)

Knowledge

• I didn’t know that Trento has lakes.Give-information+negation+knowledge+contain+attraction

(knowledge=(who=I, polarity=negative), contain=(lake, quantity=plural), attraction-spec=name-trento)

• I know the location of the hotel.

Obligation

• You must make a reservation.

Give-information+obligation+reservation (obligation=required….)

• You may cancel at any time.

• We require that you cancel 24 hours in advance.

Negation: with limited facilities for representing scope

• Of conventional speech act:I didn’t hear.

Negate-dialogue-hear

• Of main predication:I did not make a reservation.Give-information+negation+reserve+accommodation

(polarity=negative…)

• Of attitude:I don’t know if it’s all right.

Give-information+negation+knowledge+feature+object

(knowledge=(who=I, polarity=negative), object-spec=pronoun, feature=(modifier=acceptable))

Negation

• I didn’t promise I would not come.– Negate-promise+negation+action

• Of a concept:There is no downhill skiiing?Request-information+existence+activity

(activity-spec=(polarity=negative, downhill_skiing)

Relative Clauses• Broken into two IF’s.

I want the hotel that you suggested.Give-information+disposition+accommodation(disposition=(desire, who=I), accommodation-spec=(hotel, identifiability=yes))Give-information+recommendation+object(object-spec=relative, who=you, e-time=previous)

• Sentence internal relative clauses (e.g., modifying the subject) are not handled very well.– The only hotel that I can show you is a four star hotel.

• No long-distance gaps.– They are rare anyway.

Some simple relative clauses aren’t broken

• The hotel that is in Cavalese

give-information+concept

(accommodation-spec=(hotel, identifiability=yes, location=name-cavalese))

Yes-No Questions• Conventional speech act:

Do you hear me?Dialog-request-hear

• Does the flight leave at 2:00? Tell me if the flight leaves at 2:00. request-information+departure (transportation-spec=(flight,

identifiability=yes), time=(clock=(hours=2)))

Wh-questions

• Who is traveling?request-information+trip (who=question)

• When are you traveling?• What date are you traveling?• How quiet is the hotel?• Where are you traveling to?• How are you traveling?• What are you doing?• No long distance gaps.

Rhetorical Relations

• Therefore I arrived late.

give-information+arrival

(cause=discourse, who=I…)

• I arrived late because of the snow

give-information+arrival

(who=I, cause=snow, e-time=previous, time=late)

Rhetorical Relations

• because I was tired.

give-information+feature+person

(rhetorical=cause, …)

• Other relations: after, before, besides, co-occurrence, concessive, condition, contrastive, dependency, purpose, related-to, restrictive-result, result, while

Outline

• Approaches to MT: Interlingua, Transfer, Direct.• The NESPOLE! Interlingua.

– Overview and motivation – Linguistic coverage

• Tools and resources.• Evaluating an interlingua

– Reliability– Coverage– Scalability

The Interchange Format Database

61.2.3 olang I lang I Prv IRST “telefono per prenotare delle stanze per quattro colleghi”

61.2.3 olang I lang E Prv IRST “I’m calling to book some rooms for four colleagues”

61.2.3 IF Prv IRST c:request-action+reservation

+features+room (for-whom=(associate, quantity=4))

61.2.3 comments: dial-oo5-spkB-roca0-02-3

d.u.sdu olang X lang Y Prv Z sdu in language Y on one lined.u.sdu olang X lang Z Prv Z sdu in language Z on one lined.u.sdu IF Prv Z IF on-one-lined.u. sdu comments: your commentsd.u. sdu comments: go here

NESPOLE! Database

• Annotated turns (end 2001):– English: 815 (235 distinct DAs)– German: 2,873 (367)– Italian: 1,286 (233)– French: 234 (94)

• Total distinct DAs: 610• Annotated turns (end 2002): 30/40 %

more

Tools and ResourcesIF specifications (available on the web)http://www.is.cs.cmu.edu/nespole/db/index.html

IF discussion boardhttp://peace.is.cs.cmu.edu/ISL/get/if.html

C-STAR and NESPOLE! Data Baseshttp://www.is.cs.cmu.edu/nespole/db/index.html

IF Checker (web interface)http://tcc.itc.it/projects/xig/xig-on-line.html

IF test suitehttp://tcc.itc.it/projects/xig/xig-ts.html

IF emacs mode

The C-STAR Interchange Format Database

English Dialogues

English Sentences

Korean Dialogues

Korean Sentences

Italian Dialogues

Italian Sentences

Japanese Dialogues

Japanese Utterances

Distinct Dialogue Acts

36

2466

70

1142

5

233

124

5887

554 (310 agent, 244 client)

Outline

• Approaches to MT: Interlingua, Transfer, Direct.• The NESPOLE! Interlingua.

– Overview and motivation – Linguistic coverage

• Tools and resources.• Evaluating an interlingua

– Reliability– Coverage– Scalability

Comparison of two interlinguasI would like to make a reservation for the fourth through the

seventh of July.IF-1 (C-STAR II, 1997-1999)

c:request-action+reservation+temporal+hotel(time=(start-time=md4, end-time=(md7,july)))

IF-2 (NESPOLE, 2000-2002)c:give-information+disposition+reservation +accommodation (disposition=(who=I, desire), reservation-spec=(reservation, identifiability=no), accommodation-spec=hotel, object-time=(start-time=(md=4), end-time=(md=7, month=7, incl-excl=inclusive)))

Comparison of four databases(travel domain, role playing, spontaneous speech)

• DB-1: C-STAR II English database tagged with IF-1– 2278 sentences

• DB-2: C-STAR II English database tagged with IF-2– 2564 sentences

• DB-3: NESPOLE English database tagged with IF-2– 1446 sentences– Only about 50% of the vocabulary overlaps with the C-STAR

database.

• DB-4: Combined database tagged with IF-2– 4010 sentences

Same data, different interlingua

Significantly larger domain

Outline

• Approaches to MT: Interlingua, Transfer, Direct.• The NESPOLE! Interlingua.

– Overview and motivation – Linguistic coverage

• Tools and resources.• Evaluating an interlingua

– Reliability– Coverage– Scalability

Measuring Coverage

• No-tag rate:– Can a human expert assign an interlingua

representation to each sentence?– C-STAR II no-tag rate: 7.3%– NESPOLE no-tag rate: 2.4%

• 300 more sentences were covered in the C-STAR English database

• End-to-end translation performance: Measures recognizer, analyzer, and generator performance in combination with interlingua coverage.

Outline

• Approaches to MT: Interlingua, Transfer, Direct.• The NESPOLE! Interlingua.

– Overview and motivation – Linguistic coverage

• Tools and resources.• Evaluating an interlingua

– Reliability– Coverage– Scalability

Example of failure of reliability

Input: 3:00, right?

Interlingua: verify (time=3:00)Poor choice of speech act name: does it mean that the

speaker is confirming the time or requesting verification from the user?

Output: 3:00 is right.

Measuring Reliability: Cross-site evaluations

• Compare performance of:– Analyzer interlingua generator– Where the analyzer and generator are built at the same

site (or by the same person)– Where the analyzer and generator are built at different

sites (or by different people who may not know each other)

• C-STAR II interlingua: comparable end-to-end performance within sites and across sites. – around 60% acceptable translations from speech

recognizer output.• NESPOLE interlingua: cross-site end-to-end

performance is lower (but not clearly because of the IF).

Intercoder agreement: average of percent agreeent pairwise

Speech act Domain Action Arguments

IF-1: Site 1 and Site2 (exp.)

82% 66% 86%

IF-2: Site 1 and Site 2(4 experts)

92% 75% 87 %

IF-2: Within Site 1(3 experts)

94% 88% 90%

IF-2: Site 1 vs Site 2(3 experts and 1 experts)

89% 62% 83%

IF-2: Site 1 and Site 2(experts and novices)

88% 63% 86%

IF-2: Within Site 2(expert and novices)

89 % 64 % 87%

IF-2: Within Site 2(novices)

91% 61% 83%

Workshop on InterlinguaReliabilitySIG-IL

• Association for Machine Translation in the Americas

• October 8, 2002

• Tiburon, California

• Intent to participate in coding experiment (dependency representation)

• (email to lsl@cs.cmu.edu)

Outline

• Approaches to MT: Interlingua, Transfer, Direct.• The NESPOLE! Interlingua.

– Overview and motivation – Linguistic coverage

• Tools and resources.• Evaluating an interlingua

– Reliability– Coverage– Scalability

Comparison of four databases(travel domain, role playing, spontaneous speech)

• DB-1: C-STAR II English database tagged with IF-1– 2278 sentences

• DB-2: C-STAR II English database tagged with IF-2– 2564 sentences

• DB-3: NESPOLE English database tagged with IF-2– 1446 sentences– Only about 50% of the vocabulary overlaps with the C-STAR

database.

• DB-4: Combined database tagged with IF-2– 4010 sentences

Same data, different interlingua

Significantly larger domain

Measuring Scalability: Coverage Rate

What percent of the database is covered by the top n most frequent domain actions?

Coverage of 50 most frequent domain actions

C-STAR client 66.7%

NESPOLE client 66.5%

Combined client 62.9%

C-STAR agent 67.3%

NESPOLE agent 71.4%

Combined agent 64.0%

Measuring Scalability: Number of domain actions as a function

of database size

• Sample size from 100 to 3000 sentences in increments of 25.

• Average number of unique domain actions over ten random samples for each sample size.

• Each sample includes a random selection of frequent and infrequent domain actions.

IF Coverage of Four Datasets

0100200300400500600700

100

700

1300

1900

2500

3100

number of SDUs in sample

aver

age

nu

mb

er o

f u

niq

ue

DA

s o

ver

10 r

and

om

sam

ple

s

Old CSTARNew CSTARNESPOLECombined

Comparison of four databases(travel domain, role playing, spontaneous speech)

• English database 1 tagged with interlingua 1: 2278 sentences

• English database 1 tagged with interlingua 2: 2564 sentences

• English database 2 tagged with interlingua 2: 1446 sentences– Only about 50% of the vocabulary overlaps with the English

database 1.

• Combined databases tagged with interlingua 2: 4010 sentences

Same data, different interlingua

Significantly larger domain

Conclusions

• An interlingua based on domain actions is suitable for task-oriented dialogue:– Reliable

– Good coverage

– Scalable without explosion of domain actions

• It is possible to evaluate an interlingua for– Realiability

– Expressivity

– Scalability

How to have success with an interlingua in a multi-site project

• Keep it simple.

• Periodically check for intercoder agreement.

• Good documentation

• Discussion board for developers

• Know your language typology.