tto3 semantic annotation analysis bonnie dorr september 9, 2008

TTO3 Semantic Annotation Analysis

Bonnie Dorr

September 9, 2008

Analysis of Annotation Study

Interannotator AgreementMulti-translation

EquivalenceMulti-language

Equivalence

Modality 95% 75% equivalence NA (no Arabic)

Temporal Type 95% 100% equivalence 62% equivalence

Inherent Time 96% 93% equivalence 85% equivalence

Temporal Parent 85% 100% equivalence 92% equivalence

Temporal Relations 97% 100% equivalence 75% equivalence

Belief NA (not doubly annotated) 100% equivalence 90.5% equivalence

Dialog Acts NA (not doubly annotated) NA (no multi-trans) NA (no parallel docs)

Event Coref NA (not doubly annotated) 100% equivalence NA (no parallel docs)

Annotation Agreement: UMBC Modality

*Loose = Within .2

Modality Item Strict Loose* Example

Belief (13) 96% 46% 100% Expect = .9

Epistemic (8) 94% 63% 88% May = .8

Epiteuctic (5) 91% 34% 95% Get = 1

Obligative (62) 99% 34% 98% Should = .7

Permissive (15) 91% 60% 100% Allow = .8

Potential (31) 98% 74% 87% Can = .8

Volitive (19) 95% 37% 95% Prefer = .7

Evaluative (11) 100% 100% 100% Best = 1

OVERALL (164) 97% 52% 95%

Labeling Equivalence: UMBC Modality

Modality Item Strict Loose* Example

Potential/Permissive (1) 100% 0% 0% Can/Allow = .8

Volitive (2) 100% 50% 100% Want = .7

Evaluative (1) 100% 100% 100% Best = 1

OVERALL (4) 100% 50% 75%

*Loose = Within .2

Modality Example 1Sheikh Mohamed, who is also the Defense Minister of the United Arab Emirates, announced at the inauguration ceremony that "we want to make Dubai a new trading center." INFORM-100(announce)

HUM-92(Shiekh Mohammed)CEREMONY-102(ceremony)MODALITY-103(want)[TYPE=VOLITIVE,VALUE=1]

CHANGE-3(make)GEO-POL-ENT-106(Dubai)LOCATION-OF-5()

COMMERCE-EVENT-108(trade)

Sheikh Mohammed, who is also the UAE Defence Minister, said during the launching ceremony "we wish to make Dubai a center of modern economy". ASSERTIVE-ACT-177(say)

HUM-172(Shiekh Mohamed)CEREMONY-179(ceremony)MODALITY-180(wish)[TYPE=VOLITIVE,VALUE=1]

EVENT-182(make)GEO-POL-ENT-184(Dubai)CENTER-POINT-186(center)

ECONOMY-187(economy)

Equivalence: Strict = 100%, Loose = 100%

Modality Example 2The minister, {who has his own website}, also said: "I want Dubai to be the best place in the world for state-of-the-art technology companies."

ASSERTIVE-ACT-70(say)MINISTER-67(minister)MODALITY-71(want)[TYPE=VOLITIVE,VALUE=.8]

MODALITY-6()[TYPE=EVALUATIVE,VALUE=1]GEO-POL-ENT-74(Dubai)FOR-PROF-CORP-79(company)

TECHNOLOGY-78(technology)

The minister {who has a personal website on the internet}, further said that he wanted Dubai to become the best place in the world for the advanced (hitech) technological companies.ASSERTIVE-ACT-941(say)

MINISTER-936(minister)DISCOURSE-940(further)MODALITY-942(want)[TYPE=VOLITIVE,VALUE=1]

CHANGE-EVENT-944(become)MODALITY-2()[TYPE=EVALUATIVE,VALUE=1]

GEO-POL-ENT-943(Dubai)FOR-PROF-CORP-949(company)

Equivalence: Strict = 0%, 100%, Loose = 100%, 100%

Note: {} units omitted for simplificity

Modality Example 3He said companies engaged in e-commerce would be able to set up offices, employ staff and own equipment in the open zone, including fully-owned foreign companies.ASSERTIVE-ACT-157(say)

HUMAN-156(he)MODALITY-170(would be able)[TYPE=PERMISSIVE,VALUE=.8]

SET-168() CAUSE-BY-169(set up)[FOR-PROFIT-158, OFFICE-161]HIRE-162(employ) [SET-163(staff)]OWN-164(own) [DEVICE-165(equipment) [ZONE-167(zone)]]

He said that it would he possible for the companies working in the electronic trade sector to establish offices here and provide employment and own machinery in the zone opened to companies fully owned by the foreigners.ASSERTIVE-ACT-598(say)

HUMAN-596(he)MODALITY-599(would be possible)[TYPE=POTENTIAL,VALUE=.6]

OWN-604(establish)[FOR-PROFIT-600, OFFICE-605]HIRE-606(employment)OWN-607(own) [SET-608(machinery) [PLACE-609(zone)]]

Equivalence: Strict = 0%, Loose = 0%

Annotation Agreement: BBN’s Temporal Annotations

Matches Clashes AgreementTemporal Type 104 6 94.5%Inherent 106 4 96.4%Parent Pointers 93 17 84.5%

(The following values are measured over agreed parent pointers.)

Exact Match Partial Match Clash Exact Agree Part AgreeTemp Relations 81 9 3 87.1% 96.8%

Labeling Equivalence: BBN’s Temporal Annotations

Temporal Type Matches Clashes EquivalenceMulti-Trans: Dubai (E1/E2) 14 0 100.0%Multilingual: Dubai (English/Arabic) 7 6 53.8%Total 21 6 77.8%

Inherent Matches Clashes EquivalenceMulti-Trans: Dubai (E1/E2) 13 1 92.9%Multilingual: Dubai (English/Arabic) 11 2 84.6%Total 24 3 88.9%

Parent Pointers Matches Clashes EquivalenceMulti-Trans: Dubai (E1/E2) 14 0 100.0%Multilingual: Dubai (English/Arabic) 10 3 76.9%Total 24 3 88.9%

(The following value is measured over agreed parent pointers.)

Temp Relations Exact Match Partial Mat Clash Exact Equiv Part EquivMulti-Trans: Dubai (E1/E2) 14 0 0 100.0% 100.0%Multilingual: Dubai (English/Arabic) 7 2 1 70.0% 90.0%Total 21 2 1 87.5% 95.8%

E1: Sheikh Mohammed, who is also the Defense Minister(وزير) announced(واعلن) at the inauguration ceremony(التدشين) that we want(نريد) to make(نجعل) Dubai a new trading center.

E2: Sheikh Mohamed, who is also the Defense Minister(وزير) said(واعلن) during the launching ceremony(التدشين) we wish(نريد) to make(نجعل) Dubai a new center of modern economy

TimeUnit Type Relation Parentminister(الوزير)* Present.State After/Before/Conc announcedannounced(واعلن) Past.Say* Before <writer>ceremony(التدشين) Unspec.Event After/Before/Conc announced*want(نريد) Present.State* After/Before/Conc announced

make(نجعل) Unspec.State* After* want

ANNOUNCE(واعلن) [Past.Say, Before(<writer>)] MINISTER(الوزير) [Present.State, After/Before/Conc(ANNOUNCED)] CEREMONY(التدشين) [Unspec.Event, After/Before/Conc(ANNOUNCED)] WANT(نريد) [Present.State, After/Before/Conc(ANNOUNCED)] MAKE(نجعل) [Unspec.State, After(WANT)]

Temporal Relation Example 1: Multi-translation and Multilingual

<writer>

announced

ceremony

minister

TIME

want

make

Equivalence Arabic-English English-English

Type Match 25% 100%

Parent Match 75% 100%

Relation Match 67% 100%

E1: The minister who has(يملك) his own website also said(واضاف) I want(اريد) Dubai to be the best place(تصبح) in the world for companies.

E2: The minister who has(يملك) a personal website further said(واضاف) he wanted(اريد) Dubai to become the best place(تصبح) in the world for companies.

TimeUnit Type Relation Parenthas(يملك) Present.State* After/Before/Concurrent saidsaid(واضاف) Past.Say* After <announced>(prev sent)*want(اريد) Present.State*+ After/Before/Concurrent saidplace(تصبح) Unspec.State* After want

SAID(واضاف) [Past.Say, After(<announced>)] HAS(يملك) [Present.State, After(SAID)] WANT(اريد) [Present.State, After/Before/Concurrent(SAID)] PLACE(تصبح) [Unspec.State, After(WANT)]



Type Match 0% 75%



<announced>

said

want

has

TIME

place

E1: He said(وقال) companies engaged in commerce would be able(سيكون) to set up(وتقيم) offices, employ(وتستخدم) staff and own(وتمتلك) equipment in the open zone.

E2: He said(وقال) that it would be possible(سيكون) for companies working in the electronic trade sector to establish(وتقيم) offices here and provide(وتستخدم) employment and own(وتمتلك) machinery in the zone.

TimeUnit Type Relation Parentsaid(وقال) Past.Say Before* <writer>*possible(سيكون) Future.State After saidestablish(وتقيم) Unspec.Event After possibleprovide(وتستخدم) Unspec.Event Concurrent establishown(وتمتلك) Unspec.Event Concurrent provide

SAID(وقال) [Past.Say, Before(<writer>)] POSSIBLE(سيكون) [Future.State, After(SAID)] ESTABLISH(وتقيم) [Unspec.Event, After(POSSIBLE)] PROVIDE(وتستخدم) [Unspec.Event, Concurrent(ESTABLISH)] OWN(وتمتلك)[Unspec.Event, Concurrent(PROVIDE)]


<writer>

said

establish

possible

TIME

provide

own


Type Match 100% 100%



Labeling Equivalence: CMU and Columbia’s Committed Belief

Belief Type Labeling Labeled Text 1 only Text 2 only Rec Prec F-ScoreMultilingual: Dubai (English/Arabic) 21 18 2 53.8% 91.3% 67.7%Multi-Trans: Dubai (E1/E2) 28 0 11 100.0% 71.8% 83.6%Total 49 18 13 73.1% 79.0% 76.0%

Belief Type Equivalence Assigned Match EquivalenceMultilingual: Dubai (English/Arabic) 21 19 90.5%Multi-Trans: Dubai (E1/E2) 28 28 100.0%Total 49 47 95.9%

Committed Belief Example 1E1: Sheikh Mohamed, who is also the Defense Minister of the United Arab Emirates, announced at the inauguration ceremony that "we want to make Dubai a new trading center."

E2: Sheikh Mohammed, who is also the UAE Defence Minister, said during the launching ceremony "we wish to make Dubai a center of modern economy".

[WILL INSERT EXAMPLE SOON]

Committed Belief Example 2E1: The minister, who has his own website, also said: "I want Dubai to be the best place in the world for state-of-the-art technology companies."

E2: The minister who has a personal website on the internet, further said that he wanted Dubai to become the best place in the world for the advanced (hitech) technological companies.


Committed Belief Example 3E1: He said companies engaged in e-commerce would be able to set up offices, employ staff and own equipment in the open zone, including fully-owned foreign companies.

E2: He said that it would he possible for the companies working in the electronic trade sector to establish offices here and provide employment and own machinery in the zone opened to companies fully owned by the foreigners.


Labeling Equivalence: CMU Event Coref

Event Coref Equivalence Assigned Strict Match Loose Match Equivalence EquivalenceMulti-Trans: Dubai (E1/E2) 10 7 10 70.0% 100.0%Total 10 7 10 70.0% 100.0%

*Loose = arguments unordered and/or extra unused arguments

Event Coref Labeling Labeled Text 1 only Text 2 only Rec Prec F-ScoreMulti-Trans: Dubai (E1/E2) 10 0 3 100.0% 76.9% 87.0%Total 10 0 3 100.0% 76.9% 87.0%

Event Coref Example 1E1: Sheikh Mohamed, who is also the Defense Minister of the United Arab Emirates, announced at the inauguration ceremony that "we want to make Dubai a new trading center."

E2: Sheikh Mohammed, who is also the UAE Defence Minister, said during the launching ceremony "we wish to make Dubai a center of modern economy".


Event Coref Example 2E1: The minister, who has his own website, also said: "I want Dubai to be the best place in the world for state-of-the-art technology companies."

E2: The minister who has a personal website on the internet, further said that he wanted Dubai to become the best place in the world for the advanced (hitech) technological companies.


Event Coref Example 3E1: He said companies engaged in e-commerce would be able to set up offices, employ staff and own equipment in the open zone, including fully-owned foreign companies.

E2: He said that it would he possible for the companies working in the electronic trade sector to establish offices here and provide employment and own machinery in the zone opened to companies fully owned by the foreigners.


Remaining Analyses

• Additional multilingual and multi-translation sets

• Non-parallel documents about the same event

• Additional genre: Informal texts

• Additional phenomena: Coreference

tto3 semantic annotation analysis bonnie dorr september 9, 2008

Documents