Event Extraction
“… Delta Flight 14
crashed in Mississippi
killing 40 …”
…
<Plane Crash>
<Flight Number = Flight 14>
<Operator = Delta>
<Fatalities = 40>
<Crash Site = Mississippi>
…
News Corpus Knowledge Base
Event Extraction1) Generate Candidates Flight 14 crashed in Mississippi.
2) Classify MentionsFeatures: (Unigram:Mississippi) (NEType:Location) (PrevWord:in) (ObjectOf:crashed)
Label: CrashSite
3) Aggregate Labels
<Mississippi=CrashSite><Mississippi=CrashSite><Mississippi=CrashSite><Mississippi=Operator>
Final Label: CrashSite
Run Named Entity Recognition on relevant docs
Training a Mention Classifier
Need Labeled Training Data
Problems:- Expensive- Does not scale
One year after [USAir]Operator [Flight 11]FlightNumber crashed in [Toronto]CrashSite, families of the [200]Fatalities victims attended a memorial service in [Vancouver]NIL.
Distant Supervision Solution: Use known events to automatically label training data.
Training Knowledge-Base
<Plane crash>
<Flight Number = Flight 11>
<Operator = USAir>
<Fatalities = 200>
<Crash Site = Toronto>
One year after [USAir]Operator [Flight 11]FlightNumber
crashed in [Toronto]CrashSite, families of the [200]Fatalities victims attended a memorial service in [Vancouver]NIL.
6
Distant Supervision (High Level)
• Begin with set of known facts.
• Use this set to automatically label training instances from corpus.
• Train and classify (handle noise)
7
Distant Supervision for Relation Extraction
• Slot filling for named entity relations. • Minz et al. 2009 (ACL); Surdeanu et al. 2011 (TAC-KBP).
• Example: • Company: <ceo of>, <founder of>, <founding date>,
<city of headquarters>, etc.• Known relations: founder_of(Steve Jobs, Apple)
• Noisy Labeling Rule: Slot value and entity name must be in same sentence.1. (+) Apple co-founder Steve Jobs passed away yesterday.2. (-) Steve Jobs delivered the Stanford commencement
address.3. (+) Steve Jobs was fired from Apple in 1985.
8
Distant Supervision for Event Extraction
• Sentence level labeling rule doesn’t work.
1. Events lack proper names.• “The crash of USAir Flight 11”
2. Slots values occur separate from names.• The plane went down in central Texas.• 10 died and 30 were injured in yesterday’s
tragic incident.
9
Automatic Labeling: Event Extraction
• Solution: Document Level Noisy Labeling Rule.
• Heuristic: Use Flight Number as proxy for event name.
• Labeling Rule: Slot value and Flight Number must appear in same document.
Training Fact: {<Flight Number = Flight 11>, <CrashSite= Toronto>}
…Flight 11 crash Sunday… …The plane went down in [Toronto]CrashSite…
Evaluation: 80 plane crashes from Wikipedia infoboxes. Training set: 32; Dev set: 8; Test set: 40
Corpus: Newswire data from 1989 – present.
Automatic Labeling
• 38,000 Training Instances.
• 39% Noise:• Examples: Good: At least 52 people survived the crash of the Boeing 737.
Bad: First envisioned in 1964, the Boeing 737 entered service in 1968.
Extraction Models• Local Model
• Train and classify each mention independently.
• Pipeline Model• Classify sequentially; use previous label as
feature.• Captures dependencies between labels.
• E.g., Passengers and Crew go together:“4 crew and 200 passengers were on board.”
• Joint Model• Searn Algorithm (Daumé III et al., 2009).• Jointly models all mentions in a sentence.
Results
Precision Recall F1 score
Baseline (Maj. Class)
0.026 0.237 0.047
Local Model 0.159 0.407 0.229
Pipeline Model 0.154 0.422 0.226
Joint Model 0.213 0.422 0.283
14
Label Aggregation
Exhaustive Aggregation
Four <NIL>Four <Crew>Four <Crew>Four <Fatalities>Four <NIL>Four <NIL>
<Crew> <Fatalities>
15
Label Aggregation: Noisy-OR
• Key idea: Classifier gives us distribution over labels:• Stockholm <NIL:0.8; Site: 0.1, Crew:0.01, etc.>• Stockholm <Site: 0.5; NIL: 0.3, Crew:0.1, etc.>
• Compute Noisy-OR for each label.
• If Noisy-OR > threshold, use label.
Results: Noisy-OR
Label Aggregation
Model Precision Recall F1
Exhaustive Local 0.16 0.41 0.23
Pipeline 0.15 0.42 0.23
Noisy-OR Local 0.19 0.39 0.25
Pipeline 0.19 0.39 0.25
Next Step
• Compare Distant Supervision with state of the art supervised approach (Huang & Rilloff, ACL-2011).
• MUC-4 Shared Task: Terrorist Attacks.• Slot Template: <Location>, <Perpetrator>, <Instrument>, <Target>, <Victim>
• Distant Supervision Source:• http://en.wikipedia.org/wiki/
List_of_terrorist_incidents• Short summaries of several hundred terrorist
attacks.