some explorations in using world knowledge to improve language understanding peter clark network...
TRANSCRIPT
Some Explorations in Using World Knowledge to Improve Language
Understanding
Peter Clark
Network Systems Technology
The Boeing Company
Outline
Introduction Recognizing Textual Entailment (RTE)
Boeing’s RTE System How it fared with WordNet as world knowledge How it fared with machine-learned paraphrases Knowledge and Reasoning Limitations
From Entailment to Machine Reading “Reading” about who works for who “Reading” about Pearl Harbor
Conclusions
What is Language Understanding?
Not (just) parsing + word senses Construction of a coherent representation of the
scene the text describes Challenge: much of that representation is not in
the text
“A soldier was killed in a gun battle”
The soldier diedThe soldier was shotThere was a fight…
What is Language Understanding?
“A soldier was killed in a gun battle”
BecauseA battle involves a fight.Soldiers use guns.Guns shoot.Guns can kill.If you are killed, you are dead.….
How do we get this knowledge into the machine?
How do we exploit it?
The soldier diedThe soldier was shotThere was a fight…
What is Language Understanding?
“A soldier was killed in a gun battle”
BecauseA battle involves a fight.Soldiers use guns.Guns shoot.Guns can kill.If you are killed, you are dead.….
The soldier diedThe soldier was shotThere was a fight…
An entailment
What is Language Understanding?
“A soldier was killed in a gun battle”
BecauseA battle involves a fight.Soldiers use guns.Guns shoot.Guns can kill.If you are killed, you are dead.….
The soldier diedThe soldier was shotThere was a fight…
Another entailment
What is Language Understanding?
“A soldier was killed in a gun battle”
Soldiers use guns.Guns shoot.Guns can kill.If you are killed, you are dead.….
The soldier diedThe soldier was shotThere was a fight…
Another entailment
Entailment is part of language understanding
Outline
Introduction Recognizing Textual Entailment (RTE)
Boeing’s RTE System How it fared with WordNet as world knowledge How it fared with machine-learned paraphrases Knowledge and Reasoning Limitations
From Entailment to Machine Reading “Reading” about who works for who “Reading” about Pearl Harbor
Conclusions
Recognizing Textual Entailment (RTE)
Task: does H “reasonably” follow from T? Annual RTE competition for 4 years Is very difficult, and largely unsolved still
most problems require world knowledge typical scores ~50%-70% (baseline is 50%)
RTE4 (2008): Mean score was 57.5%
T: A soldier was killed in a gun battle.H: A soldier died.
!
RTE Examples
T: Loggers clear cut large tracts of rain forest to sell wood without replacing trees.H: Trees in the rain forest are cut without being replaced.
RTE3
T: Governments are looking nervously at rising food prices.H: Food prices are on the increase.
RTE4 #27
A few are easy(ish)….
but most are difficult…
Outline
Introduction Recognizing Textual Entailment (RTE)
Boeing’s RTE System How it fared with WordNet as world knowledge How it fared with machine-learned paraphrases Knowledge and Reasoning Limitations
From Entailment to Machine Reading “Reading” about who works for who “Reading” about Pearl Harbor
Conclusions
Boeing’s RTE System
1. Interpret T and H sentences individually BLUE (Boeing Language Understanding Engine)
2. See if: H subsumes (is implied by) T
H:“An animal eats a mouse” ← T:“A black cat eats a mouse”
H subsumes an elaboration of T H:“An animal digests a mouse” ← T:“A black cat eats a mouse” via IF X eats Y THEN X digests Y
Two sources of World Knowledge WordNet glosses converted to logical rules DIRT paraphrases
Blue’s Pipeline
(DECL ((VAR _X1 "a" “ball") (VAR _X2 "a" "cliff")) (S (PRESENT) NIL "throw" _X1 (PP "from" _X2))
“ball”(ball01),“cliff”(cliff01),“throw”(throw01),sobject(throw01,ball01),“from”(throw01,cliff01).
“A ball is thrown from a cliff”
Parse +Logical
form
InitialLogic
isa(ball01,ball_n1), isa(cliff01,cliff_n1), isa(throw01,throw_v1), object(throw01,ball01), origin(throw01,cliff01).
FinalLogic
Parsing
Uses “tuples” to guide it“The man ate spaghetti with a fork”
“(The man) ate (spaghetti with a fork)” ?
“(The man) ate (spaghetti) (with a fork)” ?
→ Prefer:
“(The man) ate (spaghetti) (with a fork)”
Tuples:“can eat with forks” (2)“Spaghetti can be with forks” (0)
Small improvement (~1%-2%)
“Lexico-semantic inference”
Subsumption
subject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)
“by”(eat01,animal01), object(eat01,mouse01)
T: A black cat ate a mouse
H: A mouse was eaten by an animal
predicates match if:samesubject(), by() matchof(), modifier() match anything
arguments match if same/more general word
With Inference…
T: A black cat ate a mouse
IF X isa cat_n1 THEN X has a tail_n1 IF X eats Y THEN X digests Y
T’: A black cat ate a mouse. The cat has a tail. The cat digests the mouse. The cat chewed the mouse. The cat is furry. ….
With Inference…
T: A black cat ate a mouse
IF X isa cat_n1 THEN X has a tail_n1 IF X eats Y THEN X digests Y
T’: A black cat ate a mouse. The cat has a tail. The cat digests the mouse. The cat chewed the mouse. The cat is furry. ….
H: An animal digested the mouse.
Subsumes
Outline
Introduction Recognizing Textual Entailment (RTE)
Boeing’s RTE System How it fared with WordNet as world knowledge How it fared with machine-learned paraphrases Knowledge and Reasoning Limitations
From Entailment to Machine Reading “Reading” about who works for who “Reading” about Pearl Harbor
Conclusions
Converting the Glosses to Logic
But often not. Primary problems:1. Errors in the language processing2. “flowery” language, many gaps, metonymy, ambiguity;
If logic closely follows syntax → “logico-babble”
→ Hammers hit things??
restrict#v2: place limits onisa(restrict01,restrict#v2), object(restrict01,X) → isa(place01,place#v3), object(place01,limit01), on(place01,X)
“hammer#n2: tool used to deliver an impulsive force by striking” isa(hammer01,hammer#n2) →
isa(hammer01,tool#n1), subject(use01,hammer01), to (use01,deliver01), sobject(deliver01,force01),
mod(force01,impulsive01), by(deliver01,strike01).
Sometimes we get good translations:
Successful Examples with the Glosses
T: Britain puts curbs on immigrant labor from Bulgaria and Romania.H: Britain restricted workers from Bulgaria.
14.H4
Good example
Successful Examples with the Glosses
Good example
T: Britain puts curbs on immigrant labor from Bulgaria and Romania.H: Britain restricted workers from Bulgaria.
WN: limit_v1:"restrict“: place limits on.
→ ENTAILED (correct)
14.H4
T: Britain puts curbs on immigrant labor from Bulgaria and Romania.
H: Britain placed limits on workers from Bulgaria.
T: The administration managed to track down the perpetrators.H: The perpetrators were being chased by the administration.
56.H3
Another (somewhat) good example
Successful Examples with the Glosses
T: The administration managed to track down the perpetrators.H: The perpetrators were being chased by the administration.
WN: hunt_v1 “hunt” “track down”: pursue for food or sport
→ ENTAILED (correct)
56.H3
T: The administration managed to pursue the perpetrators [for food or sport!].H: The perpetrators were being chased by the administration.
Another (somewhat) good example
Successful Examples with the Glosses
T: Foodstuffs are being blocked from entry into Iraq.H*: Food goes into Iraq. [NOT entailed]
29.H
Unsuccessful Examples with the Glosses
Bad example
T: Foodstuffs are being blocked from entry into Iraq.H*: Food goes into Iraq. [NOT entailed]
WN: go_v22:"go“: be contained in; How many times does 18 go into 54?
→ ENTAILED (incorrect)
29.H
T: Foodstuffs are being blocked from entry into Iraq.
H: Food is contained in Iraq.
Unsuccessful Examples with the Glosses
Bad example
Unsuccessful examples with the glosses
More common: Being “tantalizingly close”
T: Satomi Mitarai bled to death.H: His blood flowed out of his body.
16.H3
Unsuccessful examples with the glosses
More common: Being “tantalizingly close”
T: Satomi Mitarai bled to death.H: His blood flowed out of his body.
16.H3
bleed_v1: "shed blood", "bleed", "hemorrhage": lose blood from one's body
WordNet:
So close! (but no cigar…)
Need to also know: “lose liquid from container” → “liquid flows out of container”
usually
T: The National Philharmonic orchestra draws large crowds.H: Large crowds were drawn to listen to the orchestra.
20.H2
More common: Being “tantalizingly close”
Unsuccessful examples with the glosses
T: The National Philharmonic orchestra draws large crowds.H: Large crowds were drawn to listen to the orchestra.
20.H2
WN: orchestra = collection of musicians WN: musician: plays musical instrument WN: music = sound produced by musical instruments WN: listen = hear = perceive sound
WordNet:
So close!
More common: Being “tantalizingly close”
Unsuccessful examples with the glosses
Gloss-Based Axioms: Some Reflections
In practice, only a little leverage RTE4: 10 of 1000 entailments with WordNet glosses
8/10 correct, but: 6 were nonsensical reasoning, 1 half-ok, 1 ok
Very noisy short, simple glosses work best
In many cases is “tantalizingly close” 0.1M axioms is actually quite a small number (!)
WordNet: 15, 3, 10, 3, 41, 18, 7, 6, 24, 10, 13, 7, 15, 2DIRT: 0, 0, 1138, 0, 2550, 1896, 476, 72, 933, 394, 521, 195, 7096
RULEBASE # RULES FIRING ON A SENTENCE
Outline
Introduction Recognizing Textual Entailment (RTE)
Boeing’s RTE System How it fared with WordNet as world knowledge How it fared with machine-learned paraphrases Knowledge and Reasoning Limitations
From Entailment to Machine Reading “Reading” about who works for who “Reading” about Pearl Harbor
Conclusions
Paraphrases
Can say the same thing in multiple ways: “Joe works for DARPA” “DARPA hires Joe” “Joe goes to work for DARPA” Joe is a DARPA employee” …
Can we learn such equivalences? DIRT: The only (partially) successful attempt
12 million rules: IF X relation Y THEN X relation’ Y
Some selected paraphrases from DIRT
IF James Allen teaches a class THEN:
A class was taught by James Allen.James Allen grades a class.James Allen reads to a class.James Allen educates a class.James Allen is a teacher of a class.
A class was taught in James Allen.A class was enrolled in James Allen.A class trains a group of James Allens.A class demands a reinstatement of James Allen. James Allen humiliates in a class.
(Approximately) how DIRT learns rules
X loves Y
table
chair
bed
catd
og
Fred
Su
ep
erson
word
freqX
table
chair
bed
catd
og
Fred
Su
ep
erson
word
freq
Y
table
chair
bed
catd
og
Fred
Su
ep
erson
word
freqX
table
chair
bed
catd
og
Fred
Su
ep
erson
word
freq
Y
X falls to Y
?
(Approximately) how DIRT learns rules
X loves Y
table
chair
bed
catd
og
Fred
Su
ep
erson
word
freqX
table
chair
bed
catd
og
Fred
Su
ep
erson
word
freq
Y
table
chair
bed
catd
og
Fred
Su
ep
erson
word
freqX
table
chair
bed
catd
og
Fred
Su
ep
erson
word
freq
Y
X falls to Y
(Approximately) how DIRT learns rules
X loves Y
table
chair
bed
catd
og
Fred
Su
ep
erson
word
freqX
table
chair
bed
catd
og
Fred
Su
ep
erson
word
freq
Y
X likes Y
table
chair
bed
catd
og
Fred
Su
ep
erson
word
freqX
table
chair
bed
catd
og
Fred
Su
ep
erson
word
freq
Y
?
(Approximately) how DIRT learns rules
X loves Y
table
chair
bed
catd
og
Fred
Su
ep
erson
word
freqX
table
chair
bed
catd
og
Fred
Su
ep
erson
word
freq
Y
X likes Y
table
chair
bed
catd
og
Fred
Su
ep
erson
word
freqX
table
chair
bed
catd
og
Fred
Su
ep
erson
word
freq
YMI
MIMI
MI
(Approximately) how DIRT learns rules
X loves Y
table
chair
bed
catd
og
Fred
Su
ep
erson
word
freqX
table
chair
bed
catd
og
Fred
Su
ep
erson
word
freq
Y
X likes Y
table
chair
bed
catd
og
Fred
Su
ep
erson
word
freqX
table
chair
bed
catd
og
Fred
Su
ep
erson
word
freq
YMI
MIMI
MI
T: William Doyle works for an auction house in Manhattan.H: William Doyle goes to Manhattan.
24.H101
Successful Examples with DIRT
Good example
T: William Doyle works for an auction house in Manhattan.H: William Doyle goes to Manhattan.
24.H101
Yes! I have general knowledge that: IF Y works in X THEN Y goes to X Here: X = Manhattan, Y = DoyleThus, here: We are told in T: Doyle works in Manhattan Thus it follows that: Doyle goes to Manhattan
Successful Examples with DIRT
Good example
T: The president visited Iraq in September.H: The president traveled to Iraq.
54.H1
Successful Examples with DIRT
Good(ish) example
T: The president visited Iraq in September.H: The president traveled to Iraq.
54.H1
Yes! I have general knowledge that: IF Y is visited by X THEN X flocks to Y Here: X = the president, Y = IraqThus, here: We are told in T: Iraq is visited by the president Thus it follows that: the president flocks to Iraq
In addition, I know: "flock" is a type of "travel"Hence: The president traveled to Iraq.
Successful Examples with DIRT
Good(ish) example
T: The US troops stayed in Iraq although the war was over.H*: The US troops left Iraq when the war was over. [NOT entailed]
55.H100
Unsuccessful Examples with DIRT
Bad rule
T: The US troops stayed in Iraq although the war was over.H*: The US troops left Iraq when the war was over. [NOT entailed]
55.H100
Yes! I have general knowledge that: IF Y stays in X THEN Y leaves X Here: X = Iraq, Y = the troopThus, here: We are told in T: the troop stays in Iraq Thus it follows that: the troop leaves Iraq
Hence: The US troops left Iraq when the war was over.
Unsuccessful Examples with DIRT
Bad rule
(wrong)
T: In May, Denver underwent quadruple bypass surgery.H*: Denver died in May. [NOT entailed]
RTE4 797
Unsuccessful Examples with DIRT
Misapplied rule
T: In May, Denver underwent quadruple bypass surgery.H*: Denver died in May. [NOT entailed]
RTE4 797
Unsuccessful Examples with DIRT
Misapplied rule
Yes! I have general knowledge that: IF Y occurs in X THEN someone dies of Y in XHere: X = May, Y = DenverThus, here: I can see from T: Denver occurs in May (because "undergo" is a type of "occur") Thus it follows that: someone dies of Denver in May
Hence: Denver died in May. (wrong)
Results with DIRT
Mismatches allowed? RTE4 DIRT-based entailments
COVERED ACCURACY
0 (= deductive reasoning) 6.2% 67%
1 mismatch 18.3% 54%
2 mismatches 19% 49%
Using tuples for validating rule instantiations
“Joe shot Sue” → “Joe injured Sue”
IF X shoots Y THEN X injures Y
(plausible)
DIRT ISP: Y is animate#n1 or artifact#n1 Can also use “tuples” to validate instantiations:
(implausible)!
“Joe shot the gun” → “Joe injured the gun”
No injured guns
Using tuples for validating rule instantiations
Provides some benefit:
Ranking Method: Average Precision of rule-based
entailments for:
RTE3 RTE4
Random: 0.592 0.650
Tuple-derived confidence: 0.641 0.728
DIRT-supplied confidence: 0.634 0.712
Reflections on DIRT
Powerful, goes beyond just definitional knowledge
But: Noisy (but still useful) Only one rule type (can’t do “X buys Y → X pays money”)
fire on ~6% of the data (→ 250M needed?)
Many rules overly general, no word senses e.g., “run → “manage”
Y marries XX lives with YX kisses YX’s wife YX has a child with YX loves YX is murdered by Y (!)…
X marries Y →
Outline
Introduction Recognizing Textual Entailment (RTE)
Boeing’s RTE System How it fared with WordNet as world knowledge How it fared with machine-learned paraphrases Knowledge and Reasoning Limitations
From Entailment to Machine Reading “Reading” about who works for who “Reading” about Pearl Harbor
Conclusions
Knowledge Limitations
Huge amounts of knowledge still needed, e.g.,
bus collision → road accident WordNet: collision isa accident WordNet: “bus: a vehicle for transport” – but not road transport
X is a fatality → X was killed WordNet: “fatality: a death resulting from an accident” but “death” ↔ “killed” ?
T: A bus collision…resulted in…30 fatalities…H: 30 were killed in a road accident…
Reasoning Limitations
BLUE got this wrong, via: X is acquitted of Y → X is convicted for Y
Problem: BLUE ignored evidence against H: WordNet: “acquit” and “convict” are antonyms World: discredited witness → no conviction
Better (and towards Machine Reading):1. Look at multiple reasoning paths
2. Find “best”, consistent subset of facts
T: Kelly was acquitted of child pornography, after the star witness was discreditedH*: Kelly was convicted for child pornography. [NOT entailed]
Deductive reasoning is inappropriate
1. Interpret the text and explore Implications
Kelly was acquitted … after the star witness was discredited
Kelly was acquitted
Star witness was discredited
Kelly was convicted
Kelly was tried
Kelly was imprisoned
Kelly was freed
Witness was tarnished
“T” text:
2. Identify Conflicts
Kelly was acquitted … after the star witness was discredited
Kelly was acquitted
Star witness was discredited
Kelly was convicted
Kelly was tried
Kelly was imprisoned
Kelly was freed
Witness was tarnished
“T” text:
3. Identify Coherent Subset
Kelly was acquitted … after the star witness was discredited
Kelly was acquitted
Star witness was discredited
Kelly was convicted
Kelly was tried
Kelly was imprisoned
Kelly was freed
Witness was tarnished
“T” text:
Now, Entailment Reasoning…
Kelly was acquitted … after the star witness was discredited
Kelly was acquitted
Star witness was discredited
Kelly was convicted
Kelly was tried
Kelly was imprisoned
Kelly was freed
Witness was tarnished
“T” text:
“H” text:Kelly was convicted? Answer: No!
Now, Entailment Reasoning…
Kelly was acquitted … after the star witness was discredited
Kelly was acquitted
Star witness was discredited
Kelly was convicted
Kelly was tried
Kelly was imprisoned
Kelly was freed
Witness was tarnished
“T” text:
“H” text:Kelly was convicted? Answer: No!
What is the overall scene?Answer:
But also, we have a “model” of the scenario…
Kelly was acquitted … after the star witness was discredited
Kelly was acquitted
Star witness was discredited
Kelly was convicted
Kelly was tried
Kelly was imprisoned
Kelly was freed
Witness was tarnished
“T” text:
Getting towards text understanding!
Outline
Introduction Recognizing Textual Entailment (RTE)
Boeing’s RTE System How it fared with WordNet as world knowledge How it fared with machine-learned paraphrases Knowledge and Reasoning Limitations
From Entailment to Machine Reading “Reading” about who works for who “Reading” about Pearl Harbor
Conclusions
Former Boeing executive Alan Mulally has kept a relatively low profile since taking over as president and CEO of Ford Motor Co. last October…
Alan Mullaly asked Jim Farley to come on board as the new VP of Marketing at Ford Motors…
Reading about “who works for who”
Former Boeing executive Alan Mulally has kept a relatively low profile since taking over as president and CEO of Ford Motor Co. last October…
Alan Mullaly asked Jim Farley to come on board as the new VP of Marketing at Ford Motors…
1. Interpret the Text
Alan asks Jim
Jim is VP of Ford
Alan is president of Ford
Alan is Boeing executive
Alan takes over at Ford
Jim boards Ford
Former Boeing executive Alan Mulally has kept a relatively low profile since taking over as president and CEO of Ford Motor Co. last October…
Alan Mullaly asked Jim Farley to come on board as the new VP of Marketing at Ford Motors…
1. Interpret the Text and explore implications
Alan asks Jim
Jim is VP of Ford
Alan is president of Ford
Alan is Boeing executive
Alan takes over at Ford
Jim boards Ford
Alan works for Boeing
Alan works for Ford
Jim works for Alan
Jim flies on Ford
Alan works for Jim
Jim works for Ford
Former Boeing executive Alan Mulally has kept a relatively low profile since taking over as president and CEO of Ford Motor Co. last October…
Alan Mullaly asked Jim Farley to come on board as the new VP of Marketing at Ford Motors…
2. Identify Conflicts
Alan asks Jim
Jim is VP of Ford
Alan is president of Ford
Alan is Boeing executive
Alan takes over at Ford
Jim boards Ford
Alan works for Boeing
Alan works for Ford
Jim works for Alan
Jim flies on Ford
Alan works for Jim
Jim works for Ford
Former Boeing executive Alan Mulally has kept a relatively low profile since taking over as president and CEO of Ford Motor Co. last October…
Alan Mullaly asked Jim Farley to come on board as the new VP of Marketing at Ford Motors…
3. Identify Coherent Subset
Alan asks Jim
Jim is VP of Ford
Alan is president of Ford
Alan is Boeing executive
Alan takes over at Ford
Jim boards Ford
Alan works for Boeing
Alan works for Ford
Jim works for Alan
Jim flies on Ford
Alan works for Jim
Jim works for Ford
Coherent collection of facts
Outline
Introduction Recognizing Textual Entailment (RTE)
Boeing’s RTE System How it fared with WordNet as world knowledge How it fared with machine-learned paraphrases Knowledge and Reasoning Limitations
From Entailment to Machine Reading “Reading” about who works for who “Reading” about Pearl Harbor
Conclusions
Reading about Pearl Harbor
Will this scale to build a whole representation? Conjecture: Story has ~10 (say) events Each text presents a subset → can reconstruct story
Planes destroyed battleships
Planes took off from carrier
Planes flew to Pearl Harbor
Carrier sailed to Hawaii
Planes took off from carrier
Planes spotted on radar
Planes destroyed battleships
Carrier sailed to HawaiiPlanes took off from carrierPlanes flew to Pearl HarborPlanes spotted on radar Planes destroyed battleships
Integration
Model Integration
BUT: Isn’t that simple: Thousands of events in the scenario
Army radar operators at Kahuku Point saw a large formation of planes coming from the north.
Commander Fuchida, sent the coded messages "Tora Tora”
Model Integration
BUT: Isn’t that simple: Thousands of events in the scenario
Different levels of detail (cont). Keep most detailed? The planes destroyed the ships The planes dropped bombs, which destroyed the ships The bombs exploded, destroying the ships The Japanese destroyed the ships
Army radar operators at Kahuku Point saw a large formation of planes coming from the north.
Commander Fuchida, sent the coded messages "Tora Tora”
Model Integration
Entailment reasoning itself is error-prone
….Japanese submarines attacked Pearl Harbor…
….torpedoes attacked Pearl Harbor…
….bombers attacked Pearl Harbor…
sandwich#n2:“submarine” “hoagie” “torpedo” “sandwich” “poor boy”, “bomber”: a large sandwich made with meat and cheese
WordNet
Pearl Harbor is being attacked by
sandwiches (!)
Model Integration
Bottom Line: Integrating everything = bad idea Better:
Align fragments of texts where possible Extract a model from the fragments, e.g.,
Look for main protagonists Look for main events they are in Fill in other roles for those events
Outline
Introduction Recognizing Textual Entailment (RTE)
Boeing’s RTE System How it fared with WordNet as world knowledge How it fared with machine-learned paraphrases Knowledge and Reasoning Limitations
From Entailment to Machine Reading “Reading” about who works for who “Reading” about Pearl Harbor
Conclusions
Conclusions
Recognizing Textual Entailment A basic building block of machine reading Challenging, but inroads are being made Large amounts of knowledge needed
Boeing’s attempts WordNet glosses – limited leverage DIRT – better leverage deductive paradigm inappropriate
Towards Machine Reading elaboration, conflict detection, select consistent subset Machine Reading is not so far away?
Implications are huge!