improving qa accuracy by question inversion john prager, pablo duboue, jennifer chu-carroll...
DESCRIPTION
QA basic framework Question query processing Query Information retrieval Corpus Docs Answer processing AnswerTRANSCRIPT
![Page 1: Improving QA Accuracy by Question Inversion John Prager, Pablo Duboue, Jennifer Chu-Carroll Presentation by Sam Cunningham and Martin Wintz](https://reader035.vdocuments.us/reader035/viewer/2022062906/5a4d1b537f8b9ab0599a87a4/html5/thumbnails/1.jpg)
Improving QA Accuracy by Question Inversion
John Prager, Pablo Duboue, Jennifer Chu-Carroll
Presentation by Sam Cunningham and Martin Wintz
![Page 2: Improving QA Accuracy by Question Inversion John Prager, Pablo Duboue, Jennifer Chu-Carroll Presentation by Sam Cunningham and Martin Wintz](https://reader035.vdocuments.us/reader035/viewer/2022062906/5a4d1b537f8b9ab0599a87a4/html5/thumbnails/2.jpg)
Goal
• Re-rank answers based on additional “inverted questions” in order to improve accuracy.
• Also: Identify pressing issues related to question answering (QA)
![Page 3: Improving QA Accuracy by Question Inversion John Prager, Pablo Duboue, Jennifer Chu-Carroll Presentation by Sam Cunningham and Martin Wintz](https://reader035.vdocuments.us/reader035/viewer/2022062906/5a4d1b537f8b9ab0599a87a4/html5/thumbnails/3.jpg)
QA basic framework
Question
queryprocessing
Query
Informationretrieval
Corpus
Docs
Answerprocessing
Answer
![Page 4: Improving QA Accuracy by Question Inversion John Prager, Pablo Duboue, Jennifer Chu-Carroll Presentation by Sam Cunningham and Martin Wintz](https://reader035.vdocuments.us/reader035/viewer/2022062906/5a4d1b537f8b9ab0599a87a4/html5/thumbnails/4.jpg)
“Inverted Questions”
• “What is the capital of France?”• Becomes: “Of what country is Paris the
capital?”
![Page 5: Improving QA Accuracy by Question Inversion John Prager, Pablo Duboue, Jennifer Chu-Carroll Presentation by Sam Cunningham and Martin Wintz](https://reader035.vdocuments.us/reader035/viewer/2022062906/5a4d1b537f8b9ab0599a87a4/html5/thumbnails/5.jpg)
Related Work
• The LCC system (Moldovan & Rus, 2001)– “Logic Prover”
• Clarke et al., 2001; Prager et al. 2004b– Assign confidence boost based on redundancy of
answer
![Page 6: Improving QA Accuracy by Question Inversion John Prager, Pablo Duboue, Jennifer Chu-Carroll Presentation by Sam Cunningham and Martin Wintz](https://reader035.vdocuments.us/reader035/viewer/2022062906/5a4d1b537f8b9ab0599a87a4/html5/thumbnails/6.jpg)
System Overview
![Page 7: Improving QA Accuracy by Question Inversion John Prager, Pablo Duboue, Jennifer Chu-Carroll Presentation by Sam Cunningham and Martin Wintz](https://reader035.vdocuments.us/reader035/viewer/2022062906/5a4d1b537f8b9ab0599a87a4/html5/thumbnails/7.jpg)
Inverting the question
• Identify term with known type (the pivot term)– For example in: “What was the capital of Germany
in 1985” Germany is identified as a (COUNTRY).
• Then given a candidate answer <CandAns> formulate the inverted question as:– “Of what (COUNTRY) was <CandAns> the capital in
1985”
![Page 8: Improving QA Accuracy by Question Inversion John Prager, Pablo Duboue, Jennifer Chu-Carroll Presentation by Sam Cunningham and Martin Wintz](https://reader035.vdocuments.us/reader035/viewer/2022062906/5a4d1b537f8b9ab0599a87a4/html5/thumbnails/8.jpg)
Difficulties
• Estimated 79% of question in TREC can be inverted meaningfully and those questions are hard to identify.
• Need to have a comprehensive and accurate notion of types
• Some inverted questions have so many answers they’re not useful
![Page 9: Improving QA Accuracy by Question Inversion John Prager, Pablo Duboue, Jennifer Chu-Carroll Presentation by Sam Cunningham and Martin Wintz](https://reader035.vdocuments.us/reader035/viewer/2022062906/5a4d1b537f8b9ab0599a87a4/html5/thumbnails/9.jpg)
Inversion Algorithm
• Not actually formulating inversions in natural language.
• Qframe– Keywords– Answertype– Relationships
![Page 10: Improving QA Accuracy by Question Inversion John Prager, Pablo Duboue, Jennifer Chu-Carroll Presentation by Sam Cunningham and Martin Wintz](https://reader035.vdocuments.us/reader035/viewer/2022062906/5a4d1b537f8b9ab0599a87a4/html5/thumbnails/10.jpg)
![Page 11: Improving QA Accuracy by Question Inversion John Prager, Pablo Duboue, Jennifer Chu-Carroll Presentation by Sam Cunningham and Martin Wintz](https://reader035.vdocuments.us/reader035/viewer/2022062906/5a4d1b537f8b9ab0599a87a4/html5/thumbnails/11.jpg)
Using the inversion
• If the answer to an inverted question (validating answer) matches the original question, that question is validated
• Use this notion of validation, along with the scores of the validating answers, to re-rank candidate answers
![Page 12: Improving QA Accuracy by Question Inversion John Prager, Pablo Duboue, Jennifer Chu-Carroll Presentation by Sam Cunningham and Martin Wintz](https://reader035.vdocuments.us/reader035/viewer/2022062906/5a4d1b537f8b9ab0599a87a4/html5/thumbnails/12.jpg)
Using the inversion
• Only concerned with re-ranking top two results
• Learn a decision tree to decide whether to re-rank second result as first one
![Page 13: Improving QA Accuracy by Question Inversion John Prager, Pablo Duboue, Jennifer Chu-Carroll Presentation by Sam Cunningham and Martin Wintz](https://reader035.vdocuments.us/reader035/viewer/2022062906/5a4d1b537f8b9ab0599a87a4/html5/thumbnails/13.jpg)
Decision tree algorithm
VARIABLES
![Page 14: Improving QA Accuracy by Question Inversion John Prager, Pablo Duboue, Jennifer Chu-Carroll Presentation by Sam Cunningham and Martin Wintz](https://reader035.vdocuments.us/reader035/viewer/2022062906/5a4d1b537f8b9ab0599a87a4/html5/thumbnails/14.jpg)
Decision tree algorithm Don’t be scared by the
variables!
• ak: Learned parameters
• Ci: top two candidate answers• Si: Scores of candidate
answers• Vi: whether Ci is validated• Pi: rank of validating answer
• Ai: Score of validating answer
![Page 15: Improving QA Accuracy by Question Inversion John Prager, Pablo Duboue, Jennifer Chu-Carroll Presentation by Sam Cunningham and Martin Wintz](https://reader035.vdocuments.us/reader035/viewer/2022062906/5a4d1b537f8b9ab0599a87a4/html5/thumbnails/15.jpg)
Decision tree algorithmIf there is no first answer and the second answer has been validated return the second answer.
![Page 16: Improving QA Accuracy by Question Inversion John Prager, Pablo Duboue, Jennifer Chu-Carroll Presentation by Sam Cunningham and Martin Wintz](https://reader035.vdocuments.us/reader035/viewer/2022062906/5a4d1b537f8b9ab0599a87a4/html5/thumbnails/16.jpg)
Decision tree algorithmIf the first answer is validated with a score above a given threshold return the first answer.
![Page 17: Improving QA Accuracy by Question Inversion John Prager, Pablo Duboue, Jennifer Chu-Carroll Presentation by Sam Cunningham and Martin Wintz](https://reader035.vdocuments.us/reader035/viewer/2022062906/5a4d1b537f8b9ab0599a87a4/html5/thumbnails/17.jpg)
Decision tree algorithm
If neither answers have been validated, either reject both answers or possibly return the first one depending on the type of the pivot term.
![Page 18: Improving QA Accuracy by Question Inversion John Prager, Pablo Duboue, Jennifer Chu-Carroll Presentation by Sam Cunningham and Martin Wintz](https://reader035.vdocuments.us/reader035/viewer/2022062906/5a4d1b537f8b9ab0599a87a4/html5/thumbnails/18.jpg)
Decision tree algorithm
If only the second answer is validated then compare the score of both the answer and the validating answer.
![Page 19: Improving QA Accuracy by Question Inversion John Prager, Pablo Duboue, Jennifer Chu-Carroll Presentation by Sam Cunningham and Martin Wintz](https://reader035.vdocuments.us/reader035/viewer/2022062906/5a4d1b537f8b9ab0599a87a4/html5/thumbnails/19.jpg)
Decision tree algorithm
If both answers are validated compare the scores of both the candidate answers and the validating answers
![Page 20: Improving QA Accuracy by Question Inversion John Prager, Pablo Duboue, Jennifer Chu-Carroll Presentation by Sam Cunningham and Martin Wintz](https://reader035.vdocuments.us/reader035/viewer/2022062906/5a4d1b537f8b9ab0599a87a4/html5/thumbnails/20.jpg)
Decision tree algorithm
• Train on TREC11corpus of question-answer sets to learn threshold values (ak)
![Page 21: Improving QA Accuracy by Question Inversion John Prager, Pablo Duboue, Jennifer Chu-Carroll Presentation by Sam Cunningham and Martin Wintz](https://reader035.vdocuments.us/reader035/viewer/2022062906/5a4d1b537f8b9ab0599a87a4/html5/thumbnails/21.jpg)
Evaluation
• 50 hand-crafted questions of the form “What is the capital of X?”
• AQUAINT corpus– ~1 million news-wire documents.
• CNS corpus– 37,000 documents from the Center for
Nonproliferation Studies.
![Page 22: Improving QA Accuracy by Question Inversion John Prager, Pablo Duboue, Jennifer Chu-Carroll Presentation by Sam Cunningham and Martin Wintz](https://reader035.vdocuments.us/reader035/viewer/2022062906/5a4d1b537f8b9ab0599a87a4/html5/thumbnails/22.jpg)
![Page 23: Improving QA Accuracy by Question Inversion John Prager, Pablo Duboue, Jennifer Chu-Carroll Presentation by Sam Cunningham and Martin Wintz](https://reader035.vdocuments.us/reader035/viewer/2022062906/5a4d1b537f8b9ab0599a87a4/html5/thumbnails/23.jpg)
Evaluation II
• Processed 414 factoid questions from TREC12
![Page 24: Improving QA Accuracy by Question Inversion John Prager, Pablo Duboue, Jennifer Chu-Carroll Presentation by Sam Cunningham and Martin Wintz](https://reader035.vdocuments.us/reader035/viewer/2022062906/5a4d1b537f8b9ab0599a87a4/html5/thumbnails/24.jpg)
![Page 25: Improving QA Accuracy by Question Inversion John Prager, Pablo Duboue, Jennifer Chu-Carroll Presentation by Sam Cunningham and Martin Wintz](https://reader035.vdocuments.us/reader035/viewer/2022062906/5a4d1b537f8b9ab0599a87a4/html5/thumbnails/25.jpg)
Conclusion
• Slight improvement
• Computationally expensive
• Lacks robust notion of term equivalence
![Page 26: Improving QA Accuracy by Question Inversion John Prager, Pablo Duboue, Jennifer Chu-Carroll Presentation by Sam Cunningham and Martin Wintz](https://reader035.vdocuments.us/reader035/viewer/2022062906/5a4d1b537f8b9ab0599a87a4/html5/thumbnails/26.jpg)
Discussion
• Probability-based scores
• Better confidences
• Better NER
• Establishing term equivalence