extreme extraction - machine reading in a week
DESCRIPTION
TRANSCRIPT
[Freedman+ EMNLP11] Extreme
Extraction – Machine Reading in a
Week
23 Dec 2011
Nakatani Shuyo @ Cybozu labs, Inc
twitter : @shuyo
Abstract
• Target:
– Rapid construction of concept and relation
extraction system
• Method:
– Extend an existing ACE system for new relation
– in short time with minimum training data
• in a Week (<50 person hours) with <20 example pairs
– Evaluate by question answering task
Phases
1. Ontology and resources
2. Extending system for new ontology
3. Extracting relations
4. Evaluation
1. Ontology and resources
• possibleTreatment( Substance, Condition )
– SSRIs(S) are effective treatments for depression(C)
• expectedDateOnMarket( Substance , Date )
– More drugs for type 2(S) expected on market soon(D)
• responsibleForTreatment( Substance, Agent )
– Officials(A) Responsible for Treatment of War Dead(S)
• studiesDisease( Agent , Condition )
– cancer(C) researcher Dr. Henri Joyeux(A)
• hasSideEffect( Substance, Condition )
not
sure
2. Extending system for new
ontology
• Add new relation/class detectors into “our”
extraction system for ACE task
– Details of the system are not clear...
• Class detectors with unsupervised word clustering
• Bootstrap relation learner with a template and seeds
• Pattern learning for relation extraction
• Annotate words for 4 classes
• Coreference
Bootstrap relation learner
• DAP(Double-Anchored Pattern) (Kozareva+ 08)
– Web search with a query based on “<CLASS>
such as <SEED> and *”
– Add words at the position “*” in snippet into the
class member as new seeds
– Repeat “the bootstraping loop” while seeds are
available
Relation detection with DAP
• CLASS = disease / SEED = cold
• Web search = “disease such as cold and”
Relation detection with DAP
• CLASS = disease / SEED = cold
• Web search = “disease such as cold and”
– disease such as cold and flu (9). ...
– disease such as cold and heat, external ...
– disease such as cold and pneumonia. ...
– disease (such as cold and hot diseases), ...
– disease such as cold and flu viruses. ...
– disease such as cold and food poisoning. ...
Four classes to annotate
• Substance-Name
– medicine name
• Substance-Description
– e.g. “new drags”
• Condition-Name
– name of disease
• Condition-Description
– e.g. “the illness”
Annotation
• Name tagging with active learning(Miller+ 04)
– Unsupervised word clustering on binary tree
(Brown+ 90)
– Tagging with clustering information
• Averaged Perceptron (Collins 02)
– Request annotation for selected sentence based on
“confidence score”
• score = (highest perceptron score) - (second one)
!?
Results of Class Detection
• substances & conditions
– -Name / -Description respectively
• without/with lists of known substances and conditions
from [Freedman+ 11]
What’s
GS(GoldStandard)?
Coreference
• It took the most time(20 of 43 hours)
• But its detail is not clear...
– domain independent heuristics
– appositive linking
3. Extracting relations
• Learned Patterns vs. Handwritten Patterns
from [Freedman+ 11]
from [Freedman+ 11]
4. Evaluation
• Question Answering with extracted
information
• Query examples
– Find possible treatments for diabetes
– What is expected date to market for Abilify?
Answer Example
• ACME produces a wide range of drugs
including treatments for malaria and
athletes foot
– responsibleForTreatment(drugs, ACME)
– possibleTreatment(drugs, malaria)
– possibleTreatment(drugs, athletes foot)
• useful = answering complex query from [Freedman+ 11]
When non-useful answers are removed
• annotator’s recall (A)
• using combining both (C)
• using only handwritten rules (H, HW)
• using only learned patterns (L)
from [Freedman+ 11]
from [Freedman+ 11]
Discussion
from [Freedman+ 11]
Conclusions
• The combination system can achieve
F1 of 0.51 in a new domain in a week.
• It requires so little training data.
• The effectiveness of learning algorithms is
still not competitive with handwritten
patterns.
References
• [Freedman+ 11] Extreme Extraction – Machine
Reading in a Week
• [Kozareva+ 08] Semantic Class Learning from the
Web with Hyponym Pattern Linkage
• [Miller+ 04] Name Tagging with Word Cluster and
Discriminative Training
– [Brown+ 90] Class-based n-gram models of natural
language
– [Collins 02] Discriminative Training Methods for Hidden
Markov Models: Theory and Experiments with Perceptron
Algorithm