![Page 1: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/1.jpg)
An Inference-Based Approach to Recognizing Entailment
Peter Clark and Phil HarrisonBoeing Research and Technology
Seattle, WA
![Page 2: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/2.jpg)
Outline
BLUE (our system)DescriptionGood and bad examples on RTE5Performance and ablations on RTE5
ReflectionsThe Knowledge ProblemThe Reasoning Problem
![Page 3: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/3.jpg)
BLUE (Boeing Language Understanding Engine)
Logic Representation
Bag-of-WordsRepresentationT,H
YES/NO YES
UNKNOWN UNKNOWN
WordNetDIRT
![Page 4: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/4.jpg)
BLUE (Boeing Language Understanding Engine)
Parse, generate logic for T and HSee if every clause in H subsumes part of TUse DIRT and WordNet
Logic Representation
Bag-of-WordsRepresentationT,H
YES/NO
UNKNOWN
YES
UNKNOWN
WordNetDIRT
![Page 5: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/5.jpg)
BLUE (Boeing Language Understanding Engine)
Logic Representation
Bag-of-WordsRepresentationT,H
YES/NO YES
UNKNOWN UNKNOWN
modifier(cat01,black01),subject(eat01,cat01),object(eat01,mouse01).
subject(eat01,animal01),object(eat01,mouse01).
subsumes?
T
H
REPRESENTATIONWordNet
DIRT
T: A black cat ate a mouse.H: A mouse was eaten by an animal.
![Page 6: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/6.jpg)
1. The Logic Module: Generating a Representation
(DECL ((VAR _X1 "a" "cat" (AN "black" "cat")) (VAR _X2 "a" "mouse"))
(S (PAST) _X1 "eat" _X2))
"cat"(cat01),"black"(black01),"eat"(eat01),"mouse"(mouse01),modifier(cat01,black01),subject(eat01,cat01),object(eat01,mouse01).
“A black cat ate a mouse.”
Parse +Logical form
InitialLogic
cat#n1(cat01),black#a1(black01),mouse#n1(mouse01),eat#v1(eat01),color(cat01,black01),agent(eat01,cat01),object(eat01,mouse01).
FinalLogic
![Page 7: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/7.jpg)
1. The Logic Module: Lexico-Semantic Inference
Computing subsumption (= entailment)
subject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)
“by”(eat01,animal01), object(eat01,mouse01)
T: A black cat ate a mouse
H: A mouse was eaten by an animal
![Page 8: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/8.jpg)
1. The Logic Module: Lexico-Semantic Inference
Subsumption
subject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)
“by”(eat01,animal01), object(eat01,mouse01)H: A mouse was eaten by an animal
T: A black cat ate a mouse
WordNet cat#n1 animal#n1hypernym
speedy#s2 fast#a1similar-to
rapidly#r1 quick#a1pertains-to
destroy#v1 destruction#n1derives
also…
![Page 9: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/9.jpg)
Inference with DIRT…
T: A black cat ate a mouse
IF X eats Y THEN X chews Y
T’: A black cat ate a mouse. The cat is black.The cat digests the mouse. The cat chewed the mouse. The cat swallows the mouse…
IF X eats Y THEN X digests Y
![Page 10: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/10.jpg)
With Inference…
T: A black cat ate a mouse
IF X eats Y THEN X digests Y
H: An animal digested the mouse.
Subsumes
IF X eats Y THEN X chews Y
T’: A black cat ate a mouse. The cat is black.The cat digests the mouse. The cat chewed the mouse. The cat swallows the mouse…
H entailed!
![Page 11: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/11.jpg)
BLUE (Boeing Language Understanding Engine)
WordNetDIRT
Logic Representation
Bag-of-WordsRepresentationT,H
YES/NO YES
UNKNOWN UNKNOWN
Ignore syntactic structure:Use bag of words for T and H
See if every word in H subsumes one in TUse DIRT and WordNet
![Page 12: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/12.jpg)
BLUE (Boeing Language Understanding Engine)
WordNetDIRT
Logic Representation
Bag-of-WordsRepresentationT,H
YES/NO YES
UNKNOWN UNKNOWN
REPRESENTATION
{ black cat eat mouse }
{ mouse digest animal }subsumes?
T: A black cat ate a mouse.H: A mouse was digested by an animal.
![Page 13: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/13.jpg)
Bag of Words Inference
{ black cat eat mouse }
{ mouse digest animal }
subsumes?
T: A black cat ate a mouse.H: A mouse was digested by an animal.
T
H
![Page 14: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/14.jpg)
Bag of Words Inference
{ black cat eat mouse }
{ mouse digest animal }
T: A black cat ate a mouse.H: A mouse was digested by an animal.
T
H
WordNet cat#n1 animal#n1hypernym
![Page 15: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/15.jpg)
Bag of Words Inference
{ black cat eat mouse }
{ mouse digest animal }
T: A black cat ate a mouse.H: A mouse was digested by an animal.
T
H
DIRT IF X eats Y THEN X digests Y
“eat” “digest”
![Page 16: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/16.jpg)
Bag of Words Inference
{ black cat eat mouse }
{ mouse digest animal }
T: A black cat ate a mouse.H: A mouse was digested by an animal.
T
H
H entailed!
![Page 17: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/17.jpg)
Outline
BLUE (our system)DescriptionGood and bad examples on RTE5Performance and ablations on RTE5
ReflectionsThe Knowledge ProblemThe Reasoning Problem
![Page 18: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/18.jpg)
The Good…
T: …Ernie Barnes…was an offensive linesman….H: Ernie Barnes was an athlete.
#191 (BLUE got this right)
via WordNet: linesman#n1 isa athlete#n1
T: …hijacking of a Norwegian tanker…by Somali piratesH: Somali pirates attacked a Norwegian tanker.
#333 (BLUE got this right)
via DIRT: IF X hijacks Y THEN Y is attacked by X.
T: …Charles divorced Diana…H: Prince Charles was married to Princess Diana.
Pilot H26 (BLUE got this right)
☺
☺
☺via DIRT: IF X divorces Y THEN X marries Y.
![Page 19: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/19.jpg)
The Good (Cont)…
HEADLINE: EU slams Nepalese king's dismissal…T: The EU…presidency called for …democracy.H: There has been a…call for ..democracy in Nepal
Pilot H142 (BLUE got this right)
via use of HEADLINE as context (and WordNet Nepalese/Nepal)
T: Crippa died..after he ate..deadly....wild mushroomsH: Crippa was killed by a wild mushroom.
#78 (BLUE got this right)
via DIRT: IF X dies of Y THEN Y is killed by X
☺
☺
![Page 20: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/20.jpg)
The Bad
T: Venus Williams triumphed over…Bartoli…H*: Venus Williams was defeated by…Bartoli…
#407 (BLUE got this wrong, predicting YES)
via (bad rule in) DIRT: IF Y wins over X THEN X defeats Y.
T: PepsiCo …acquired …Star Foods…H: PepsiCo holds Star Foods
#219 (BLUE got this right, but for nonsensical reasons)
via DIRT: IF X acquires Y THEN X sells Yand: IF Y sells X’s business THEN Y holds X’s tongue
and WordNet: “tongue” isa “food”
T: …even if Iceland offered Fischer citizenship….H*: Iceland granted Bobby Fischer citizenship.
Pilot H29 (BLUE got this wrong, predicting YES)
BLUE does not recognize the hypothetical blocks entailment
![Page 21: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/21.jpg)
The Bad (Cont)…
T: …Slumdog Millionaire director Danny Boyle….H: The movie “Slumdog Millionaire” has been directed by Danny Boyle.
#157 (BLUE got this wrong, predicting UNKNOWN)
(unable to conclude “movie” in H)
T: ..the oath taken by the 115 electors…H*: The cardinals electing the pope…
“115 is a cardinal” (!)
Pilot H75 (BLUE got this wrong, predicting YES)
![Page 22: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/22.jpg)
Outline
BLUE (our system)DescriptionGood and bad examples on RTE5Performance and ablations on RTE5
ReflectionsThe Knowledge ProblemThe Reasoning Problem
![Page 23: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/23.jpg)
Results: Main Task
Pipeline works best
Logic Bag
Logic
Bag
54.761.5Logic + bag-of-words
52.860.0Bag-of-words only
46.356.7Logic module only
3-way2-Way
![Page 24: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/24.jpg)
Results: Main Task
Pipeline works best
Logic Bag
Logic
Bag
54.761.5Logic + bag-of-words
52.860.0Bag-of-words only
46.356.7Logic module only
3-way2-Way
Logic alone is worse than bag aloneOnly decides 29% of cases, but does well (64%) on these
![Page 25: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/25.jpg)
Ablation Studies
Full BLUE- remove WordNet- remove DIRT- remove parsing
RTE5 Dev RTE5 Test
- 6.0 - 4.063.8 61.5
- 0.5 +1.2- 0.3 - 1.5
![Page 26: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/26.jpg)
Ablation Studies
Full BLUE- remove WordNet- remove DIRT- remove parsing
RTE5 Dev RTE5 Test
- 6.0 - 4.063.8 61.5
- 0.5 +1.2- 0.3 - 1.5
![Page 27: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/27.jpg)
Ablation Studies
WordNet is significantly helping
Full BLUE- remove WordNet- remove DIRT- remove parsing
RTE5 Dev RTE5 Test
- 6.0 - 4.063.8 61.5
- 0.5 +1.2- 0.3 - 1.5
![Page 28: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/28.jpg)
Ablation Studies
WordNet is significantly helpingDIRT is barely helping
Rules are noisy (≈ 50% are bad)Applicability is low (≈ 10%-15%) – most RTE problems
are outside DIRT’s scope
Full BLUE- remove WordNet- remove DIRT- remove parsing
RTE5 Dev RTE5 Test
- 6.0 - 4.063.8 61.5
- 0.5 +1.2- 0.3 - 1.5
![Page 29: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/29.jpg)
Ablation Studies
WordNet is significantly helpingDIRT is barely helpingParsing is barely helping
Extracting syntactic structure is very error-proneSemantic relationships usually persist from T to H and H*
Non-entailment caused by other factors
Full BLUE- remove WordNet- remove DIRT- remove parsing
RTE5 Dev RTE5 Test
- 6.0 - 4.063.8 61.5
- 0.5 +1.2- 0.3 - 1.5
![Page 30: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/30.jpg)
“Semantic Continuity”
How important is semantic (hence syntactic) structure?
T: Boyle directed Slumdog MillionaireH*: Slumdog Millionaire directed Boyle [NOT entailed]
IF T and H are “sensible”AND T and H are consistent with world knowledgeAND T and H are topically similarTHEN this heavily constrains the variability in
possible semantic (hence syntactic) relationships
→ reduced discriminatory power of semantic/syntactic analysis
“Semantic Continuity” Conjecture:
but this kind of example is unusual in RTE!
![Page 31: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/31.jpg)
Outline
BLUE (our system)DescriptionGood and bad examples on RTE5Performance and ablations on RTE5
ReflectionsThe Knowledge ProblemThe Reasoning Problem
![Page 32: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/32.jpg)
What are the Long Term Challenges?
The Knowledge ProblemStill missing a lot of world knowledge
T: ... volunteers...helped...create dikes to protect... against the Red River...flood.
H: Red River will overflow its banks.
#341
Need to know…flood: water...overflowing onto normally dry landbank: sloping land..beside..water
?
![Page 33: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/33.jpg)
What are the Long Term Challenges?
The Knowledge ProblemStill missing a lot of world knowledge
T: ... volunteers...helped...create dikes to protect... against the Red River...flood.
H: Red River will overflow its banks.
#341
Need to know…flood: water...overflowing onto normally dry landbank: sloping land..beside..water
includes
![Page 34: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/34.jpg)
What are the Long Term Challenges?
The Knowledge ProblemStill missing a lot of world knowledge
The Reasoning ProblemFinding some path from T to H is error-prone
![Page 35: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/35.jpg)
What are the Long Term Challenges?
The Knowledge ProblemStill missing a lot of world knowledge
The Reasoning ProblemFinding some path from T to H is error-prone
T: Venus Williams triumphed over Bartoli…to win…H*: Venus Williams was defeated by…Bartoli…
BUT: evidence against H:triumph=defeat, and defeat is antisymmetricWorld Knowledge: “win” implies defeat (not defeated by)
Better: look at multiple reasoning pathsfind the “best”, consistent subset of implications
IF Y triumphs over X THEN X defeats Y Wrong
#407
![Page 36: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/36.jpg)
Venus Williams triumphed over Bartoli to win…
Williams triumphed over Bartoli
Williams won
Williams was defeated by Bartoli
Williams triumphed
Williams lost to Bartoli
Williams defeated someone
Williams had a victory
“T” text:
![Page 37: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/37.jpg)
Williams triumphed over Bartoli
Venus Williams triumphed over Bartoli to win…
Williams won
Williams was defeated by Bartoli
Williams triumphed
Williams lost to Bartoli
Williams defeated someone
Williams had a victory
“T” text:
![Page 38: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/38.jpg)
Williams triumphed over Bartoli
Venus Williams triumphed over Bartoli to win…
Williams won
Williams was defeated by Bartoli
Williams triumphed
Williams lost to Bartoli
Williams defeated someone
Williams had a victory
“T” text:
“H” text:Was Williams defeated? Answer: No!
What is the overall scene?
![Page 39: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/39.jpg)
Williams triumphed over Bartoli
Venus Williams triumphed over Bartoli to win…
Williams won
Williams was defeated by Bartoli
Williams triumphed
Williams lost to Bartoli
Williams defeated someone
Williams had a victory
“T” text:
“H” text:Williams was defeated? Answer: No!
What is the overall scene? Answer:
![Page 40: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/40.jpg)
Williams triumphed over Bartoli
Venus Williams triumphed over Bartoli to win…
Williams won
Williams was defeated by Bartoli
Williams triumphed
Williams lost to Bartoli
Williams defeated someone
Williams had a victory
“T” text:
a step towards text “understanding”
![Page 41: An Inference-Based Approach to Recognizing Entailmenttac.nist.gov/publications/2009/presentations/TAC2009_RTE_Boeing.pdfsubject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01)](https://reader035.vdocuments.us/reader035/viewer/2022070908/5f888b20cff51534b8652d69/html5/thumbnails/41.jpg)
Summary
BLUE:Pipeline of logical representation + bag-of-wordsReasoning with WordNet and DIRTPerformance – ok (above the median)
Ablations:WordNet helps a lotDIRT and parsing barely helped
Two big challenges:Knowledge – need lots moreReasoning – need search for coherence, not a single path