from textual entailment to knowledgeable machines peter clark allen institute for artificial...
TRANSCRIPT
![Page 1: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/1.jpg)
From Textual Entailment to Knowledgeable Machines
Peter Clark
Allen Institute for Artificial Intelligence (AI2)
![Page 2: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/2.jpg)
Mission: achieve scientific breakthroughs by constructing AI systems with reasoning, learning, and reading capabilities.
2
![Page 3: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/3.jpg)
Overall Goals
Vision: The Digital Aristotle Large volumes of general and scientific
knowledge, stored in a "computable“ form that supports reasoning and explanation.
Intermediate Focus: Pass science exams as written Heavy emphasis on semi-automated knowledge acquisition Textual entailment at every step
≈ the “modus ponens” of reasoning
“Explainable Reasoning”
![Page 4: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/4.jpg)
![Page 5: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/5.jpg)
The Task
Current focus: 4th grade, multiple choice
science questions Wide variety of question
types Requires general, lexical,
and scientific knowledge
![Page 6: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/6.jpg)
The 4th Grade NY Regents’ Science Exam What types of questions are there? What would it take to answer them?
![Page 7: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/7.jpg)
The 4th Grade NY Regents’ Science Exam What types of questions are there? What would it take to answer them?
![Page 8: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/8.jpg)
Multiple Choice to Textual Entailment
QA: A potato is a fruit?QB: An onion is a fruit?QC: A carrot is a fruit?QD: A pumpkin is a fruit?
1. Convert to 4 true/false questions
2. Convert each true/false question to an entailment problem
T: potato H: fruit?
H: potato is a fruit?
entails?
T: entails?
OR: (for X isa Y questions, and questions with a setup)
![Page 9: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/9.jpg)
Multiple Choice to Textual Entailment
QA: A potato is a fruit?QB: An onion is a fruit?QC: A carrot is a fruit?QD: A pumpkin is a fruit?
Confidence?
0.040.120.210.64
1. Convert to 4 true/false questions
2. Convert each true/false question to an entailment problem
T: potato H: fruit?
H: potato is a fruit?
entails?
T: entails?
OR: (for X isa Y questions, and questions with a setup)
Answer is D
3. Pick highest confidence answer
![Page 10: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/10.jpg)
The 4th Grade NY Regents’ Science Exam What types of questions are there? What would it take to answer them? “Basic”
![Page 11: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/11.jpg)
1. Taxonomic
Simple lexical entailment e.g., T:“sleet” H:“precipitation”
Several good sources of simple “isa” knowledge WordNet, Cyc, Wikipedia
Is a basic operation for more complex entailment tasks
entails?
![Page 12: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/12.jpg)
2. Definitions
Search for best entailing definition
erosion: The process of being eroded by wind, water, or other natural agents.erosion: The wearing away of rocks and other deposits on the earth's surface …erosion: The gradual wearing away of land surface materials, especially rocks, …
Dictionary Resources
![Page 13: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/13.jpg)
2. Definitions
Search for best entailing definition
erosion: The process of being eroded by wind, water, or other natural agents.erosion: The wearing away of rocks and other deposits on the earth's surface …erosion: The gradual wearing away of land surface materials, especially rocks, …
Dictionary Resources
T: The gradual wearing away of land surface materials, especially rocks, sediments, and soils, by the action of water, wind, or a glacier
H: The movement of soil by wind or water
entails?
![Page 14: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/14.jpg)
T: The gradual wearing away of land surface materials, especially rocks, sediments, and soils, by the action of water, wind, or a glacier
H: The movement of soil by wind or water
entails?
# wordsin common
weighted∑ #words
# withhypernyms
∑ bi-grams
para-phrases … p(H|T)
3.0 4.31 3.45 2.0 1.20 ?
2.0 1.23 2.12 5.0 1.98
TRAINING DATAH TRUE
6.0 4.31 3.45 0.0 0.20 H FALSE1.0 5.43 1.11 1.0 0.24 H FALSE
3.0 1.12 3.45 2.0 1.76 H TRUE
p(H|T)
= 0.76
![Page 15: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/15.jpg)
2. Definitions
Search for best entailing definition
erosion: The process of being eroded by wind, water, or otheragents.erosion: The wearing away of rocks and other deposits on the earth …erosion: The gradual wearing away of land surface materials, …
friction: The rubbing of surfaces against each otherfriction: a resistance encountered when one body moves relative to
another body with which it is in contactfriction: surface resistance to relative motion, as of a body sliding
Dictionary Resources0.540.430.76
0.210.11
0.13
Answer is erosion (C)
![Page 16: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/16.jpg)
3. within-question entailments
T: A girl eating an apple
QB: A girl eating an apple is an example of an organism taking in nutrients?
H: an organism taking in nutrients
entails?
![Page 17: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/17.jpg)
3. within-question entailments
T: A girl eating an apple
QB: A girl eating an apple is an example of an organism taking in nutrients?
H: an organism taking in nutrients
p(H|T) = 0.91
![Page 18: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/18.jpg)
The 4th Grade NY Regents’ Science Exam What types of questions are there? What would it take to answer them?
“Entailmentfrom
Corpus”
![Page 19: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/19.jpg)
4. Entailment from a corpus
HA: The flower is the part of a plant that produces the seedsHB: The leaves are the part of a plant that produces the seedsHC: The stem is the part of a plant that produces the seedsHD: The roots are the part of a plant that produces the seeds
T: HA? HB? HC? HD?
![Page 20: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/20.jpg)
4. Entailment from a corpus
HA: The flower is the part of a plant that produces the seeds HB: The leaves are the part of a plant that produces the seedsHC: The stem is the part of a plant that produces the seedsHD: The roots are the part of a plant that produces the seeds
T:
…Plants can grow from a seed into a flower, tree, or bush.Plants reproduce by producing flowers and fruits that have seeds.The seeds then grow into plants.…
EntailmentConfidence?
0.840.120.210.04
![Page 21: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/21.jpg)
The 4th Grade NY Regents’ Science Exam What types of questions are there? What would it take to answer them?
“Models”
“Diagrams”
![Page 22: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/22.jpg)
5. Computational Models
Requires a specific computation over a representation
baby shake rattle rattle make noise
movement
mechanical energy
sound
sound energy
(C) Mechanical Energy
![Page 23: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/23.jpg)
Performance (excluding diagrams)
Works okay… ~55% score (vs. 25% random guessing)
![Page 24: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/24.jpg)
But…
If horses are kept inside in a barn, they require regular daily exercise for their physical health and mental well-being.
?
Carrots can also be used alone or with fruits in jam and preserves.
System answer: (B)
System answer: (C)
![Page 25: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/25.jpg)
System answer: (A)
Graders are commonly used in the construction and maintenance of dirt roads and gravel roads
In experiments in which statoliths were replaced with metal shavings, researchers "tricked" crayfish into swimming upside down by using magnets to pull the shavings to the upper end of the statocysts located at the base of their antennae.
?System answer: (D)
![Page 26: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/26.jpg)
What is going on?
Largely “smart guessing” based on word associations BUT:
doesn’t give us meaningful explanations doesn’t get us closer to machine reading performance max’es out at ~60%
What is missing? Richer representation of meaning
in the question in the corpus
![Page 27: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/27.jpg)
Entailment
? ?- has-part(ribosome,?x).
Text Logic
Query
logicalentailment
“textual”entailment
![Page 28: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/28.jpg)
Entailment
? ?- has-part(ribosome,?x).
Text Logic
Query
logicalentailment
“textual”entailment
![Page 29: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/29.jpg)
Textual Entailment
? ?- has-part(ribosome,?x).
Text Logic
Query
Semi-
Formal
logicalentailment
“textual”entailment
Bag of wordsN-gramsParse treesDependency trees…
?Textual Entailment =The Science of Semi-Formal
Representations
![Page 30: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/30.jpg)
Semi-formal representations
Lexical – good baseline Dependency trees – precise but complex to manipulate Full logic – very hard to translate into SVO units (“depth 1 parses”, “tuples”)?
top-level: syntactic structure lower level: phrasal
These simple propositions = a basic “unit of meaning” Inference is a mixture of structural and phrasal matching
The vibrations from sound move tiny bones in our ears.
(the vibrations from sound, move, tiny bones in our ears)
subject verb object [pps]
Text
Repn.
![Page 31: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/31.jpg)
Sentences may contain one or more related units
Some animals grow thick fur in winter to stay warm.
P1 P2(some animals, grow, thick fur, in winter) (some animals, stay, , warm)
Types of relations between units:
P1 AND P2P1 IMPLIES P2P1 EFFECT P2P1 CAUSES P2P1 PURPOSE P2
EFFECT
![Page 32: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/32.jpg)
Can define extraction patterns for these units
S V O “to help” V O P1 EFFECT P2
Some animals grow thick fur in winter to help maintain body heat
(some animals, grow, thick fur, in winter) EFFECT (some animals, maintain, body heat)
![Page 33: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/33.jpg)
Pattern-based Extraction
Can transduce text into this form
P1P2 AND P3P4P5 CAUSES P6P7 ENABLES P8 AND P9P10P11P12 IMPLIES P13P14 CAUSES P15P16
S V O “to help” S V OS V O “in order to” V OS V “cause” S V O… … …
![Page 34: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/34.jpg)
Pattern-based Extraction
S V O “to help” S V OS V O “in order to” V OS V “cause” S V O… … …
(gravity, pull) CAUSE (objects, fall, , towards Earth)(scientists, using, a model) EFFECT(scientists, understand, , better)(, dividing, a single cell) EFFECT(, form, two daughter cells)(Animals, use, saturated fatty acids) EFFECT(Animals, store, energy)(humans, get, regular rest) EFFECT (humans, be, healthy)(Fish, have, fins)EFFECT(fins, move, )(animals, move, to warmer climate) EFFECT (animals, avoid, change in seasons)(cactus, hold, water) EFFECT(cactus, survive, in the desert)
… … … …
Can transduce text into this form
![Page 35: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/35.jpg)
The child exhibits the facial features characteristic of this disorder.
HA A facial scar is a characteristic that a human offspring can inherit?
T
System Answer: (A)
![Page 36: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/36.jpg)
Some traits that can be inherited are color of hair, color of skin, color of eyes, and height.
HB Blue eyes is a characteristic that a human offspring can inherit?
T
System Answer: (B)
(human offspring, can inherit, the characteristic of blue eyes)
(, can inherit, color of hair)(, can inherit, color of skin)(, can inherit, color of eyes)(, can inherit, height)
![Page 37: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/37.jpg)
Where does this break down?
1. Quality of extractions is low
2. Sentence-level units are too small, context independent
(, create, a diagram) EFFECT (, show, the cells of multicellular organisms may be organized at different levels)
(, Say, thanks to the authors) EFFECT (, access, a customizable version of this book)(All cells, are, small, very) EFFECT (one or more cells, need, )
Need larger-sized structures spanning several sentences
![Page 38: From Textual Entailment to Knowledgeable Machines Peter Clark Allen Institute for Artificial Intelligence (AI2)](https://reader034.vdocuments.us/reader034/viewer/2022051517/56649cfa5503460f949cc23f/html5/thumbnails/38.jpg)
The Main Points Textual entailment is
less about “matching text” more about the science of
semi-formal representations Those representations
explicate the “world knowledge” the text encodes
can be sharable “knowledge resources” in their own right
take us a step closer to “knowledgeable machines”
?Query
Semi-
FormalText