what’s missing: a knowledge gap guided approach for multi...
TRANSCRIPT
![Page 1: What’s Missing: A Knowledge Gap Guided Approach for Multi ...cis700dr/Spring20/files/04-20-01.pdf · Tushar Khot, Ashish Sabharwal, Peter Clark. EMNLP 2019 2. QASC: A Dataset for](https://reader033.vdocuments.us/reader033/viewer/2022050110/5f47a8ad2c1a832f78192dd8/html5/thumbnails/1.jpg)
What’s Missing: A Knowledge Gap Guided Approach for Multi-hop Question Answering Tushar Khot and Ashish Sabharwal and Peter Clark EMNLP 2019
Krunal Shah ([email protected])
20th April 2020
![Page 2: What’s Missing: A Knowledge Gap Guided Approach for Multi ...cis700dr/Spring20/files/04-20-01.pdf · Tushar Khot, Ashish Sabharwal, Peter Clark. EMNLP 2019 2. QASC: A Dataset for](https://reader033.vdocuments.us/reader033/viewer/2022050110/5f47a8ad2c1a832f78192dd8/html5/thumbnails/2.jpg)
2
Example (from Open Book Question Answering)
Question: Which of these would let the most heat travel through?A) a new pair of jeans.B) a steel spoon in a cafeteria. C) a cotton candy at a store. D) a calvin klein cotton hat.
Core Fact: (annotated as part of dataset)
Metal lets heat travel through.
Knowledge Gap (similar gaps for other choices): steel spoon in a cafeteria _____ metal.
![Page 3: What’s Missing: A Knowledge Gap Guided Approach for Multi ...cis700dr/Spring20/files/04-20-01.pdf · Tushar Khot, Ashish Sabharwal, Peter Clark. EMNLP 2019 2. QASC: A Dataset for](https://reader033.vdocuments.us/reader033/viewer/2022050110/5f47a8ad2c1a832f78192dd8/html5/thumbnails/3.jpg)
3
Motivation
• This work concentrates on multi-hop QA in partial knowledge setting• Most existing QA datasets assume availability of all required knowledge• However, partial knowledge setting is quite intuitive, natural and
challenging
Example:Which of these would let the most heat travel through?a) Osmium
Hints?
![Page 4: What’s Missing: A Knowledge Gap Guided Approach for Multi ...cis700dr/Spring20/files/04-20-01.pdf · Tushar Khot, Ashish Sabharwal, Peter Clark. EMNLP 2019 2. QASC: A Dataset for](https://reader033.vdocuments.us/reader033/viewer/2022050110/5f47a8ad2c1a832f78192dd8/html5/thumbnails/4.jpg)
4
Knowledge Gaps
Broadly (inexhaustively) classified into three types:1. Question-to-fact Gap:
2. Fact-to-Answer Gap (this work)
3. Question-to-Answer (Fact) Gap
* image from the original paper
![Page 5: What’s Missing: A Knowledge Gap Guided Approach for Multi ...cis700dr/Spring20/files/04-20-01.pdf · Tushar Khot, Ashish Sabharwal, Peter Clark. EMNLP 2019 2. QASC: A Dataset for](https://reader033.vdocuments.us/reader033/viewer/2022050110/5f47a8ad2c1a832f78192dd8/html5/thumbnails/5.jpg)
5
Brief Approach
1. Identify “key span” in core fact
Question: Which of these would let the most heat travel through?A) a new pair of jeans.B) a steel spoon in a cafeteria. C) a cotton candy at a store.
D) a calvin klein cotton hat.
Core Fact: (annotated as part of dataset)
Metal lets heat travel through.
![Page 6: What’s Missing: A Knowledge Gap Guided Approach for Multi ...cis700dr/Spring20/files/04-20-01.pdf · Tushar Khot, Ashish Sabharwal, Peter Clark. EMNLP 2019 2. QASC: A Dataset for](https://reader033.vdocuments.us/reader033/viewer/2022050110/5f47a8ad2c1a832f78192dd8/html5/thumbnails/6.jpg)
6
Brief Approach
2. Retrieve some knowledge to help fill knowledge gap
Question: Which of these would let the most heat travel through?A) a new pair of jeans.B) a steel spoon in a cafeteria. C) a cotton candy at a store.
D) a calvin klein cotton hat.
Core Fact: (annotated as part of dataset)
Metal lets heat travel through.
![Page 7: What’s Missing: A Knowledge Gap Guided Approach for Multi ...cis700dr/Spring20/files/04-20-01.pdf · Tushar Khot, Ashish Sabharwal, Peter Clark. EMNLP 2019 2. QASC: A Dataset for](https://reader033.vdocuments.us/reader033/viewer/2022050110/5f47a8ad2c1a832f78192dd8/html5/thumbnails/7.jpg)
Question: Which of these would let the most heat travel through?A) a new pair of jeans.B) a steel spoon in a cafeteria. C) a cotton candy at a store.
D) a calvin klein cotton hat.
Core Fact: (annotated as part of dataset)
Metal lets heat travel through.
7
Brief Approach
(steel spoon, isA, metal)
3. Identify relations between key span and answer choices
(using retrieved knowledge)
![Page 8: What’s Missing: A Knowledge Gap Guided Approach for Multi ...cis700dr/Spring20/files/04-20-01.pdf · Tushar Khot, Ashish Sabharwal, Peter Clark. EMNLP 2019 2. QASC: A Dataset for](https://reader033.vdocuments.us/reader033/viewer/2022050110/5f47a8ad2c1a832f78192dd8/html5/thumbnails/8.jpg)
8
Open Book QA-Short
Narrow down questions as:1. Fact supports correct answer
- answer-fact gap2. Questions with small answer choices
- Long answer choices lead to noisy gaps
![Page 9: What’s Missing: A Knowledge Gap Guided Approach for Multi ...cis700dr/Spring20/files/04-20-01.pdf · Tushar Khot, Ashish Sabharwal, Peter Clark. EMNLP 2019 2. QASC: A Dataset for](https://reader033.vdocuments.us/reader033/viewer/2022050110/5f47a8ad2c1a832f78192dd8/html5/thumbnails/9.jpg)
9
Knowledge Gap Dataset (KGD)
Starting with questions included in Open Book QA-Short:
Annotate:1. Key span in core fact that could answer the question
- because answer-fact gap2. One or more relations that satisfy knowledge gap
Data point: {question, fact, spans, relations}Only include questions with agreement greater than 2/3.
* image from the original paper
![Page 10: What’s Missing: A Knowledge Gap Guided Approach for Multi ...cis700dr/Spring20/files/04-20-01.pdf · Tushar Khot, Ashish Sabharwal, Peter Clark. EMNLP 2019 2. QASC: A Dataset for](https://reader033.vdocuments.us/reader033/viewer/2022050110/5f47a8ad2c1a832f78192dd8/html5/thumbnails/10.jpg)
10
Proposed Model (GapQA)
Question: Which of these would let the most heat travel through?A) a new pair of jeans.B) a steel spoon in a cafeteria. C) a cotton candy at a store. D) a calvin klein cotton hat.
Core Fact:Metal lets heat travel through.
![Page 11: What’s Missing: A Knowledge Gap Guided Approach for Multi ...cis700dr/Spring20/files/04-20-01.pdf · Tushar Khot, Ashish Sabharwal, Peter Clark. EMNLP 2019 2. QASC: A Dataset for](https://reader033.vdocuments.us/reader033/viewer/2022050110/5f47a8ad2c1a832f78192dd8/html5/thumbnails/11.jpg)
11
Proposed Model (GapQA)
Repeat subsequent steps for all options
Question: Which of these would let the most heat travel through?A) a new pair of jeans.B) a steel spoon in a cafeteria. C) a cotton candy at a store.
D) a calvin klein cotton hat.
Core Fact: (annotated as part of dataset)
Metal lets heat travel through.
![Page 12: What’s Missing: A Knowledge Gap Guided Approach for Multi ...cis700dr/Spring20/files/04-20-01.pdf · Tushar Khot, Ashish Sabharwal, Peter Clark. EMNLP 2019 2. QASC: A Dataset for](https://reader033.vdocuments.us/reader033/viewer/2022050110/5f47a8ad2c1a832f78192dd8/html5/thumbnails/12.jpg)
12
Proposed Model (GapQA)
1. Run RC Model with (Question, Core Fact as context) to predict “key span”
Question: Which of these would let the most heat travel through?A) a new pair of jeans.
Core Fact: (annotated as part of dataset)
Metal lets heat travel through.
![Page 13: What’s Missing: A Knowledge Gap Guided Approach for Multi ...cis700dr/Spring20/files/04-20-01.pdf · Tushar Khot, Ashish Sabharwal, Peter Clark. EMNLP 2019 2. QASC: A Dataset for](https://reader033.vdocuments.us/reader033/viewer/2022050110/5f47a8ad2c1a832f78192dd8/html5/thumbnails/13.jpg)
13
Proposed Model (GapQA)
2. Knowledge Retrieval
Tuple Search:Subject matching sObject matching c
Text Search: Elastic Search using “s + c”
Convert everything to text.
Question: Which of these would let the most heat travel through?A) a new pair of jeans.
Core Fact: (annotated as part of dataset)
Metal lets heat travel through.
![Page 14: What’s Missing: A Knowledge Gap Guided Approach for Multi ...cis700dr/Spring20/files/04-20-01.pdf · Tushar Khot, Ashish Sabharwal, Peter Clark. EMNLP 2019 2. QASC: A Dataset for](https://reader033.vdocuments.us/reader033/viewer/2022050110/5f47a8ad2c1a832f78192dd8/html5/thumbnails/14.jpg)
14
Proposed Model (GapQA)
3. Scores the option choice- Fact relevance score:- Relation prediction score
Question: Which of these would let the most heat travel through?A) a new pair of jeans.
Core Fact: (annotated as part of dataset)
Metal lets heat travel through.
![Page 15: What’s Missing: A Knowledge Gap Guided Approach for Multi ...cis700dr/Spring20/files/04-20-01.pdf · Tushar Khot, Ashish Sabharwal, Peter Clark. EMNLP 2019 2. QASC: A Dataset for](https://reader033.vdocuments.us/reader033/viewer/2022050110/5f47a8ad2c1a832f78192dd8/html5/thumbnails/15.jpg)
15
B weighted representation of A (attB(A))
Ai . Bjmax
softmaxA
B
x + x + x + x =
A’s token representations
![Page 16: What’s Missing: A Knowledge Gap Guided Approach for Multi ...cis700dr/Spring20/files/04-20-01.pdf · Tushar Khot, Ashish Sabharwal, Peter Clark. EMNLP 2019 2. QASC: A Dataset for](https://reader033.vdocuments.us/reader033/viewer/2022050110/5f47a8ad2c1a832f78192dd8/html5/thumbnails/16.jpg)
16
Compare rep A and rep B (comp(A, B))
_
representation A
representation B
*
concatenation
![Page 17: What’s Missing: A Knowledge Gap Guided Approach for Multi ...cis700dr/Spring20/files/04-20-01.pdf · Tushar Khot, Ashish Sabharwal, Peter Clark. EMNLP 2019 2. QASC: A Dataset for](https://reader033.vdocuments.us/reader033/viewer/2022050110/5f47a8ad2c1a832f78192dd8/html5/thumbnails/17.jpg)
17
Fact Relevance
Question and choice attended fact representation:Squestion, choice(fact) = avg(attquestion(fact), attchoice(fact))
Aggregate fact representation:Rfact = avg(fact representations)
scorefact(choice) = FF( comp(Rfact,Squestion, choice(fact)) )
![Page 18: What’s Missing: A Knowledge Gap Guided Approach for Multi ...cis700dr/Spring20/files/04-20-01.pdf · Tushar Khot, Ashish Sabharwal, Peter Clark. EMNLP 2019 2. QASC: A Dataset for](https://reader033.vdocuments.us/reader033/viewer/2022050110/5f47a8ad2c1a832f78192dd8/html5/thumbnails/18.jpg)
18
Relation Prediction
Span attended knowledge sentence representation:Sspan (knowledge sent) = attspan(knowledge sent)
Choice attended knowledge sentence representation:Schoice (knowledge sent) = attchoice(knowledge sent)
Partial relation representation= FF( comp(Sspan (knowledge sent), Schoice (knowledge sent)) )
![Page 19: What’s Missing: A Knowledge Gap Guided Approach for Multi ...cis700dr/Spring20/files/04-20-01.pdf · Tushar Khot, Ashish Sabharwal, Peter Clark. EMNLP 2019 2. QASC: A Dataset for](https://reader033.vdocuments.us/reader033/viewer/2022050110/5f47a8ad2c1a832f78192dd8/html5/thumbnails/19.jpg)
19
Relation Prediction Score
Relation RepresentationRrelation = avgknowledge sentences(Partial Relation representation)
Question Fact Composed RepresentationRfact,question = comp(max pooled question rep., max pooled fact rep.)
Relation Prediction Score= FF( [Rfact,question; Rrelation])
![Page 20: What’s Missing: A Knowledge Gap Guided Approach for Multi ...cis700dr/Spring20/files/04-20-01.pdf · Tushar Khot, Ashish Sabharwal, Peter Clark. EMNLP 2019 2. QASC: A Dataset for](https://reader033.vdocuments.us/reader033/viewer/2022050110/5f47a8ad2c1a832f78192dd8/html5/thumbnails/20.jpg)
20
Training methodology
1. Use BiDAF trained on SQuAD + finetuned on Knowledge Gap Dataset to predict spans
2. Relation loss:- Project relation representation to multilabel relation classification- Binary cross-entropy loss
3. Train model on- Knowledge Gap Dataset- Open Book QA-Short using predicted spans and ignoring relation loss
![Page 21: What’s Missing: A Knowledge Gap Guided Approach for Multi ...cis700dr/Spring20/files/04-20-01.pdf · Tushar Khot, Ashish Sabharwal, Peter Clark. EMNLP 2019 2. QASC: A Dataset for](https://reader033.vdocuments.us/reader033/viewer/2022050110/5f47a8ad2c1a832f78192dd8/html5/thumbnails/21.jpg)
21
Results and analysis
Impressive results:~6.5% improvement in partial knowledge setting~3% improvement on complete knowledge setting
* denotes results are statistically significant
![Page 22: What’s Missing: A Knowledge Gap Guided Approach for Multi ...cis700dr/Spring20/files/04-20-01.pdf · Tushar Khot, Ashish Sabharwal, Peter Clark. EMNLP 2019 2. QASC: A Dataset for](https://reader033.vdocuments.us/reader033/viewer/2022050110/5f47a8ad2c1a832f78192dd8/html5/thumbnails/22.jpg)
22
Results and analysis
Ablation study results
![Page 23: What’s Missing: A Knowledge Gap Guided Approach for Multi ...cis700dr/Spring20/files/04-20-01.pdf · Tushar Khot, Ashish Sabharwal, Peter Clark. EMNLP 2019 2. QASC: A Dataset for](https://reader033.vdocuments.us/reader033/viewer/2022050110/5f47a8ad2c1a832f78192dd8/html5/thumbnails/23.jpg)
23
Personal thoughts
About the paper:+ Propose an excellent (realistic) task+ Thorough analysis of results- Only handle two hop questions
§ Jansen et al. show that even elementary science questions require 4 to 6 sentences to answer and explain on an average [3].
- Need a more general framework for dealing with gaps§ Handle more kinds of knowledge gap
• Results and limitations of the work suggest we are still quite far away from truly solving two hop questions
![Page 24: What’s Missing: A Knowledge Gap Guided Approach for Multi ...cis700dr/Spring20/files/04-20-01.pdf · Tushar Khot, Ashish Sabharwal, Peter Clark. EMNLP 2019 2. QASC: A Dataset for](https://reader033.vdocuments.us/reader033/viewer/2022050110/5f47a8ad2c1a832f78192dd8/html5/thumbnails/24.jpg)
24
Personal thoughts
• The ability to answer the following is integral to reasoning“What more do I need to know (to achieve something)?”
• An observation that can possibly help towards a more general framework
• Previously explored idea of multi-hop QA as path finding! [4][5]
• Recognize where a node is missing
![Page 25: What’s Missing: A Knowledge Gap Guided Approach for Multi ...cis700dr/Spring20/files/04-20-01.pdf · Tushar Khot, Ashish Sabharwal, Peter Clark. EMNLP 2019 2. QASC: A Dataset for](https://reader033.vdocuments.us/reader033/viewer/2022050110/5f47a8ad2c1a832f78192dd8/html5/thumbnails/25.jpg)
25
References
1. What's Missing: A Knowledge Gap Guided Approach for Multi-hop Question Answering. Tushar Khot, Ashish Sabharwal, Peter Clark. EMNLP 2019
2. QASC: A Dataset for Question Answering via Sentence Composition. Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, Ashish Sabharwal. AAAI 2020
3. What's in an Explanation? Characterizing Knowledge and Inference Requirements for Elementary Science Exams. Jansen, Peter and Balasubramanian, Niranjan and Surdeanu, Mihai and Clark, Peter. COLING 2016.
4. Question Answering via Integer Programming over Semi-Structured Knowledge. Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Peter Clark, Oren Etzioni, Dan Roth. IJCAI 2016.
5. Question Answering as Global Reasoning over Semantic Abstractions. Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Dan Roth. AAAI 2018.
![Page 26: What’s Missing: A Knowledge Gap Guided Approach for Multi ...cis700dr/Spring20/files/04-20-01.pdf · Tushar Khot, Ashish Sabharwal, Peter Clark. EMNLP 2019 2. QASC: A Dataset for](https://reader033.vdocuments.us/reader033/viewer/2022050110/5f47a8ad2c1a832f78192dd8/html5/thumbnails/26.jpg)
26
Results and analysis
Analyzing results using different knowledge sources
* denotes results are statistically significant
![Page 27: What’s Missing: A Knowledge Gap Guided Approach for Multi ...cis700dr/Spring20/files/04-20-01.pdf · Tushar Khot, Ashish Sabharwal, Peter Clark. EMNLP 2019 2. QASC: A Dataset for](https://reader033.vdocuments.us/reader033/viewer/2022050110/5f47a8ad2c1a832f78192dd8/html5/thumbnails/27.jpg)
27
Contributions
- Annotate a subset of Open Book QA for partial knowledge setting- Propose a novel approach of identify missing knowledge and filling it for
multi hop QA- Propose a model that learns to fill missing knowledge from external
knowledge and compose it with exisiting knowledge- State of the art results on QA with partial knowledge
![Page 28: What’s Missing: A Knowledge Gap Guided Approach for Multi ...cis700dr/Spring20/files/04-20-01.pdf · Tushar Khot, Ashish Sabharwal, Peter Clark. EMNLP 2019 2. QASC: A Dataset for](https://reader033.vdocuments.us/reader033/viewer/2022050110/5f47a8ad2c1a832f78192dd8/html5/thumbnails/28.jpg)
28
Another related dataset
QASC: Question Answering via Sentence Composition [2]
* image from [2]
4. Question
1. Seed fact2. Fact from large corpus
3. Composed fact
![Page 29: What’s Missing: A Knowledge Gap Guided Approach for Multi ...cis700dr/Spring20/files/04-20-01.pdf · Tushar Khot, Ashish Sabharwal, Peter Clark. EMNLP 2019 2. QASC: A Dataset for](https://reader033.vdocuments.us/reader033/viewer/2022050110/5f47a8ad2c1a832f78192dd8/html5/thumbnails/29.jpg)
30
Outline:
• Problem and Motivation• Brief overview and dataset• Proposed model (GapQA)• Experiments• Personal thoughts