crowdsourcing for nlp ground truth data

Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo

gather annotation of types, events, relations, coref

Lora Aroyo and Chris Welty

From Crowd Knowledge to Machine Knowledge

Text

Text

1Wednesday, October 17, 12


Position

The human disagreement & vagueness are part of the

event & relations semantics

Flickr: elkabong2Wednesday, October 17, 12


PositionArtificially restricting humans does not help machines to learn.

Machines will learn from diversity

Flickr: elkabong3Wednesday, October 17, 12


Disagreement Framework

• ontology: disagreements on the basic status of events themselves as referents of linguistic utterances, e.g. are people events or do events exist at all.

• granularity: disagreements that result from issues of granularity, e.g. the location being a country, region, or city, the time being a day, week, month, etc.

• interpretation: disagreements that result from (non-granular) ambiguity, differences in perspective, or error in interpreting an expression, e.g. classifying a person as a terrorist/hero, ”October Revolution” took place in September.



Approach Principles1. tolerate, capture & exploit disagreement

2. understand the disagreement by creating a space of possibilities (frequencies & similarities)3. score the machine output based on where it falls in this space

4. adaptable to new annotation tasks

Flickr: auroille5Wednesday, October 17, 12

http://www.flickr.com/photos/30868936@N08/

http://www.flickr.com/photos/30868936@N08/

Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo Lora Aroyo

Event Extractioncrowdsourcing ground truth data



Event Participants Disagreement

Prime minister Benjamin

Netanyahu

Benjamin Netanyahu

Israeli Prime minister

Cabinet

Benjamin Netanyahu’s

Cabinet

Israeli Cabinet

his Cabinet

Israeli Government

{TOLD}

50%

35%

15%

10%

15%

5%

45%

15%



Temporal Disagreement

Spring 1998

March 1, 1998

March 1998

SundayPrime minister

Benjamin Netanyahu

Benjamin Netanyahu

Israeli Prime minister

{TOLD}

50%

35%

15%

25%

15%

50%

5%



Spatial Disagreement

Lebanon

IsraelSouthern Lebanon

Israel's Northern Frontier

Middle East

{WILLING TO WITHDRAW}

35%

45%

10%

30%

65%



it seems to refer to an inference or communicated feeling more than specific event.

does not refer to an event

a group of people did something specific at a specific point in 6me.

refers to an event

the actors in ques6on (top Israeli officials) performed an ac6on during a specified 6me (Sunday).

refers to an event

it refers to what the israelis did on sunday, a specific 6me.

Top Israeli officials SENT strong new SIGNALS Sunday that Israel wants to withdraw from southern Lebanon, ...



Because it is describing a historical issue concerning the resolu6on of 1978

refers to an event

That 1978 resolu6on calls for Israel's uncondi6onal WITHDRAWAL from the self-‐declared security zone it occupies in south Lebanon, ...

it is not a par6cular movement that has or is going on but a request that the country of Israel remove their forces from the zone they occupy.


the sentence is speaking of a demand for a withdrawal that had not yet occurred.




Relation Extractioncrowdsourcing ground truth data

Lora Aroyo



6 experiments• 2 professional

annotators per sentence @20sentences

• 10 CFworkers per sentence @20sentences



• 30 CFworkers per sentence @10sentences + relations definitions

• 10 CFworkers per sentence @20sentence explanation validation



The Task



The Steps: Example Sentence (1)



The Steps: Example Sentence (2)



OLANZAPINE is an atypical antipsychotic, approved by the U.S.Food and Drug Administration (FDA) for the treatment of SCHIZOPHRENIA and bipolar disorder.

?OLANZAPINEIs SCHIZOPHRENIA related to

treated_by treats may_treat

?RESPIRATORY ALKALOSIS

Is HYPERVENTILATION related to

cause_of cause symptom_ofmay_causediagnosed_by

RESPIRATORY ALKALOSIS is a medical condition in which increased respiration (HYPERVENTILATION) elevates the blood pH (a condition generally called alkalosis).



He was the first physician to identify the relationship between HEMOPHILIA and HEMOPHILIC ARTHROPATHY.

?HEMOPHILIC ARTHROPATHY

Is HEMOPHILIA related to

treated_by may_causeother

cause_of cause

has_manifestation

symptom_of

It just says there is a relation between the two but gives no specifics about what the relation is

There is a relationship between the two disorders but the sentence does not indicate what that relationship is.

identify the relationship between

relationship between HEMOPHILIA and HEMOPHILIC ARTHROPATHY



Distributions Compared

0"

20"

40"

60"

80"

100"

120"

140"

sTB" sT" sMT" sPB" sP" sMP" sDB" sD" sMD" sCO" sC" sMC" sLO" sHM" sDF" sSS" sOTH" sNO"

10"workers" 20"workers" 30"workers"



Sentence-Relation DistributionProfessional Annotators



Sentence-Relation Distribution10w x 20s




caused_bycaused_by

caused_by

is differentiate

d from

is_related

is_type_of,

has_type,

x

is_a

includes

is_related to

maybe related to




with relation explanations



The Task



The Steps: Example Sentence



The Dark Side of Crowdsourcing Disagreement

• disagreement is beautiful, except when it results from spamming• crowdsourcing has to account for people that want to get paid for

not doing any work• spammers generate disagreement for the wrong reasons• most spam detection requires gold standard



Phase I:A. Collect event annotations +

motivations

Extraction of Putative Events

input: putative events

Phase I:C. Filtering spam

event annotations

input: output of A

Phase II:A. Collect event types

+ motivations

input: list of events

Phase II:B. Filtering spam

event types

input: output of A

Manual selection of Gold Questions

input: output of A

Manual selection of

Gold Questions

input: output of A

Phase III:A. Collect event

modalities + motivations


Phase III:B. Filtering spam event modalities

input: output of A Manual

selection of Gold Questions

input: output of A

Phase IV:A. Collect event

role fillers + motivations

Phase IV:B. Filtering spam event role fillers

input: output of A Manual

selection of Gold Questions

input: output of A


• a new way of measuring ground truth

• a new set of semantic features for learning in event extraction



Questions?

@laroyohttp://lora-aroyo.org


http://lora-aroyo.org

http://lora-aroyo.org

crowdsourcing for nlp ground truth data

Documents