answer extraction
DESCRIPTION
Answer Extraction. Ling573 NLP Systems and Applications May 17, 2011. Roadmap. Deliverable 3 Discussion What worked Deliverable 4 Answer extraction: Learning answer patterns Answer extraction: classification and ranking Noisy channel approaches. Reminder. Steve Sim - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/1.jpg)
Answer ExtractionLing573
NLP Systems and ApplicationsMay 17, 2011
![Page 2: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/2.jpg)
RoadmapDeliverable 3 Discussion
What workedDeliverable 4Answer extraction:
Learning answer patternsAnswer extraction: classification and rankingNoisy channel approaches
![Page 3: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/3.jpg)
ReminderSteve Sim
Career Exploration discussionAfter class today
![Page 4: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/4.jpg)
Deliverable #3Document & Passage RetrievalWhat was tried:
Query processing:
![Page 5: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/5.jpg)
Deliverable #3Document & Passage RetrievalWhat was tried:
Query processing:Stopwording: bigger/smaller lists
![Page 6: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/6.jpg)
Deliverable #3Document & Passage RetrievalWhat was tried:
Query processing:Stopwording: bigger/smaller listsStemming: Y/N: Krovetz/Porter/Snowball
![Page 7: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/7.jpg)
Deliverable #3Document & Passage RetrievalWhat was tried:
Query processing:Stopwording: bigger/smaller listsStemming: Y/N: Krovetz/Porter/SnowballTargets: Concatenation, pronoun substitution
![Page 8: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/8.jpg)
Deliverable #3Document & Passage RetrievalWhat was tried:
Query processing:Stopwording: bigger/smaller listsStemming: Y/N: Krovetz/Porter/SnowballTargets: Concatenation, pronoun substitutionReformulation
![Page 9: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/9.jpg)
Deliverable #3Document & Passage RetrievalWhat was tried:
Query processing:Stopwording: bigger/smaller listsStemming: Y/N: Krovetz/Porter/SnowballTargets: Concatenation, pronoun substitutionReformulationQuery expansion:
WordNet synonym expansion Pseudo-relevance feedback
Slight differences
![Page 10: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/10.jpg)
Deliverable #3What worked:
Query processing:Stopwording: bigger/smaller lists
Both - apparently
![Page 11: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/11.jpg)
Deliverable #3What worked:
Query processing:Stopwording: bigger/smaller lists
Both - apparentlyStemming: Y/N: Krovetz/Porter/Snowball
![Page 12: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/12.jpg)
Deliverable #3What worked:
Query processing:Stopwording: bigger/smaller lists
Both - apparentlyStemming: Y/N: Krovetz/Porter/SnowballTargets: Concatenation, pronoun substitution: Both
good
![Page 13: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/13.jpg)
Deliverable #3What worked:
Query processing:Stopwording: bigger/smaller lists
Both - apparentlyStemming: Y/N: Krovetz/Porter/SnowballTargets: Concatenation, pronoun substitution: Both
goodReformulation: little effect
![Page 14: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/14.jpg)
Deliverable #3What worked:
Query processing:Stopwording: bigger/smaller lists
Both - apparentlyStemming: Y/N: Krovetz/Porter/SnowballTargets: Concatenation, pronoun substitution: Both
goodReformulation: little effectQuery expansion:
Generally degraded results One group had some improvement in MRR
![Page 15: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/15.jpg)
Deliverable #3What was tried:
Indexing:Lucene & Indri
![Page 16: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/16.jpg)
Deliverable #3What was tried:
Indexing:Lucene & Indri
Passage retrieval:Indri passages: different window sizes/steps
![Page 17: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/17.jpg)
Deliverable #3What was tried:
Indexing:Lucene & Indri
Passage retrieval:Indri passages: different window sizes/stepsPassage reranking:
Multitext SiteQ ISI Classification
![Page 18: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/18.jpg)
Deliverable #3What worked:
Indexing:Lucene & Indri: Generally comparable: 0.30-0.35 MAP
![Page 19: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/19.jpg)
Deliverable #3What worked:
Indexing:Lucene & Indri: Generally comparable: 0.30-0.35 MAP
Passage retrieval:Indri passages: different window sizes/steps
Generally works well with moderate overlap: best results
![Page 20: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/20.jpg)
Deliverable #3What worked:
Indexing:Lucene & Indri: Generally comparable: 0.30-0.35 MAP
Passage retrieval:Indri passages: different window sizes/steps
Generally works well with moderate overlap: best resultsPassage reranking:
Multitext SiteQ ISI Classification: hard to beat Indri
![Page 21: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/21.jpg)
Deliverable #4End-to-end QA: Answer extraction/refinement
Jeopardy!-style answers: beyond scope Instead:
More fine-grained passages
![Page 22: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/22.jpg)
Deliverable #4End-to-end QA: Answer extraction/refinement
Jeopardy!-style answers: beyond scope Instead:
More fine-grained passages
Specifically, passages of:100-char,250-char, and1000-char
![Page 23: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/23.jpg)
Deliverable #4End-to-end QA: Answer extraction/refinement
Jeopardy!-style answers: beyond scope Instead:
More fine-grained passages
Specifically, passages of:100-char,250-char, and1000-char
Evaluated on QA 2004, 2005 (held out) data
![Page 24: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/24.jpg)
Deliverable #4Output Format:
Factoids only please
Top 20 results
Output lines:Qid run-tag DocID Answer_string
Answer string: different lengthsPlease no carriage returns….
![Page 25: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/25.jpg)
Answer ExtractionGoal:
Given a passage, find the specific answer in passageGo from ~1000 chars -> short answer span
![Page 26: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/26.jpg)
Answer ExtractionGoal:
Given a passage, find the specific answer in passageGo from ~1000 chars -> short answer span
Example:Q: What is the current population of the United
States?Pass: The United States enters 2011 with a
population of more than 310.5 million people, according to a U.S. Census Bureau estimate.
![Page 27: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/27.jpg)
Answer ExtractionGoal:
Given a passage, find the specific answer in passageGo from ~1000 chars -> short answer span
Example:Q: What is the current population of the United
States?Pass: The United States enters 2011 with a
population of more than 310.5 million people, according to a U.S. Census Bureau estimate.
Answer: 310.5 million
![Page 28: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/28.jpg)
ChallengesISI’s answer extraction experiment:
Given:Question: 413 TREC-2002 factoid questionsKnown answer typeAll correct answer passages
![Page 29: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/29.jpg)
ChallengesISI’s answer extraction experiment:
Given:Question: 413 TREC-2002 factoid questionsKnown answer typeAll correct answer passages
Task: Pin-point specific answer string
Accuracy: Systems: 68.2%, 63.4%, 56.7%
Still missing 30%+ answers
![Page 30: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/30.jpg)
ChallengesISI’s answer extraction experiment:
Given:Question: 413 TREC-2002 factoid questionsKnown answer typeAll correct answer passages
Task: Pin-point specific answer string
Accuracy: Systems: 68.2%, 63.4%, 56.7%
Still missing 30%+ answersOracle (any of 3 right): 78.9% (20% miss)
![Page 31: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/31.jpg)
Basic StrategiesAnswer-type matching:
Build patterns for answer locationsRestrict by answer type
![Page 32: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/32.jpg)
Basic StrategiesAnswer-type matching:
Build patterns for answer locationsRestrict by answer type
Information for pattern types:
![Page 33: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/33.jpg)
Basic StrategiesAnswer-type matching:
Build patterns for answer locationsRestrict by answer type
Information for pattern types:Lexical: word patternsSyntactic/structural:
Syntactic relations b/t question and answerSemantic:
Semantic/argument relations b/t question and answer
![Page 34: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/34.jpg)
Basic StrategiesAnswer-type matching:
Build patterns for answer locationsRestrict by answer type
Information for pattern types:Lexical: word patternsSyntactic/structural:
Syntactic relations b/t question and answerSemantic:
Semantic/argument relations b/t question and answerCombine with machine learning to select
![Page 35: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/35.jpg)
Pattern Matching ExampleAnswer type: Definition
![Page 36: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/36.jpg)
Pattern Matching ExampleAnswer type: Definition
![Page 37: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/37.jpg)
Pattern Matching ExampleAnswer type: Definition
Answer type: Birthdate
![Page 38: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/38.jpg)
Pattern Matching ExampleAnswer type: Definition
Answer type: BirthdateQuestion: When was Mozart born?Answer: Mozart was born on ….
![Page 39: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/39.jpg)
Pattern Matching ExampleAnswer type: Definition
Answer type: BirthdateQuestion: When was Mozart born?Answer: Mozart was born on ….Pattern: <QP> was born on <AP>Pattern: <QP> (<AP> - …..)
![Page 40: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/40.jpg)
Basic StrategiesN-gram tiling:
Typically as part of answer validation/verification
Integrated with web-based retrieval
Based on retrieval of search ‘snippets’
Identifies frequently occurring, overlapping n-gramsOf correct type
![Page 41: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/41.jpg)
41
N-gram Tiling
Dickens
Charles Dickens
Mr Charles
Scores
20
15
10
merged, discardold n-grams
Mr Charles DickensScore 45
N-Gramstile highest-scoring n-gram
N-Grams
Repeat, until no more overlap
![Page 42: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/42.jpg)
Automatic Pattern Learning
Ravichandran and Hovy 2002; Echihabi et al, 2005
Inspiration (Soubottin and Soubottin ’01)Best TREC 2001 system:
![Page 43: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/43.jpg)
Automatic Pattern Learning
Ravichandran and Hovy 2002; Echihabi et al, 2005
Inspiration (Soubottin and Soubottin ’01)Best TREC 2001 system:
Based on extensive list of surface patterns Mostly manually created
![Page 44: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/44.jpg)
Automatic Pattern Learning
Ravichandran and Hovy 2002; Echihabi et al, 2005
Inspiration (Soubottin and Soubottin ’01)Best TREC 2001 system:
Based on extensive list of surface patterns Mostly manually created
Many patterns strongly associated with answer typesE.g. <NAME> (<DATE>-<DATE>)
Person’s birth and death
![Page 45: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/45.jpg)
Pattern LearningS & S ‘01 worked well, but
![Page 46: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/46.jpg)
Pattern LearningS & S ‘01 worked well, but
Manual pattern creation is a hassle, impractical
![Page 47: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/47.jpg)
Pattern LearningS & S ‘01 worked well, but
Manual pattern creation is a hassle, impractical
Can we learn patterns?Supervised approaches:
![Page 48: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/48.jpg)
Pattern LearningS & S ‘01 worked well, but
Manual pattern creation is a hassle, impractical
Can we learn patterns?Supervised approaches:
Not much better, Have to tag training samples, need training samples
![Page 49: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/49.jpg)
Pattern LearningS & S ‘01 worked well, but
Manual pattern creation is a hassle, impractical
Can we learn patterns?Supervised approaches:
Not much better, Have to tag training samples, need training samples
Bootstrapping approaches:Promising:
![Page 50: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/50.jpg)
Pattern LearningS & S ‘01 worked well, but
Manual pattern creation is a hassle, impractical
Can we learn patterns?Supervised approaches:
Not much better, Have to tag training samples, need training samples
Bootstrapping approaches:Promising:
Guidance from small number of seed samples Can use answer data from web
![Page 51: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/51.jpg)
Finding Candidate PatternsFor a given question type
Identify an example with qterm and aterm
![Page 52: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/52.jpg)
Finding Candidate PatternsFor a given question type
Identify an example with qterm and atermSubmit to a search engineDownload top N web docs (N=1000)Select only sentences w/qterm and aterm
![Page 53: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/53.jpg)
Finding Candidate PatternsFor a given question type
Identify an example with qterm and atermSubmit to a search engineDownload top N web docs (N=1000)Select only sentences w/qterm and aterm Identify all substrings and their counts
Implemented using suffix trees for efficiencySelect only phrases with qterm AND atermReplace qterm and aterm instances w/generics
![Page 54: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/54.jpg)
ExampleQ: When was Mozart born?A: Mozart (1756-….
![Page 55: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/55.jpg)
ExampleQ: When was Mozart born?A: Mozart (1756 – Qterm: Mozart; Aterm: 1756
The great composer Mozart (1756–1791) achieved fame
Mozart (1756–1791) was a genius Indebted to the great music of Mozart (1756–1791)
![Page 56: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/56.jpg)
ExampleQ: When was Mozart born?A: Mozart (1756 – Qterm: Mozart; Aterm: 1756
The great composer Mozart (1756–1791) achieved fame
Mozart (1756–1791) was a genius Indebted to the great music of Mozart (1756–1791)
Phrase: Mozart (1756-1791); count =3
![Page 57: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/57.jpg)
ExampleQ: When was Mozart born?A: Mozart (1756 – Qterm: Mozart; Aterm: 1756
The great composer Mozart (1756–1791) achieved fame
Mozart (1756–1791) was a genius Indebted to the great music of Mozart (1756–1791)
Phrase: Mozart (1756-1791); count =3Convert to : <Name> (<ANSWER>
![Page 58: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/58.jpg)
PatternsTypically repeat with a few more examples
![Page 59: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/59.jpg)
PatternsTypically repeat with a few more examplesCollect more patterns:
E.g. for Birthdate a. born in <ANSWER> , <NAME>b. <NAME> was born on <ANSWER> ,c. <NAME> ( <ANSWER> -d. <NAME> ( <ANSWER> - )
Is this enough?
![Page 60: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/60.jpg)
PatternsTypically repeat with a few more examplesCollect more patterns:
E.g. for Birthdate a. born in <ANSWER> , <NAME>b. <NAME> was born on <ANSWER> ,c. <NAME> ( <ANSWER> -d. <NAME> ( <ANSWER> - )
Is this enough?No – some good patterns, but
Probably lots of junk, too; need to filter
![Page 61: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/61.jpg)
Computing Pattern Precision
For question type:Search only on qterm
![Page 62: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/62.jpg)
Computing Pattern Precision
For question type:Search only on qtermDownload top N web docs (N=1000)Select only sentences w/qterm
![Page 63: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/63.jpg)
Computing Pattern Precision
For question type:Search only on qtermDownload top N web docs (N=1000)Select only sentences w/qtermFor each pattern, check if
a) matches w/any aterm; Co
b)matches/w right aterm: Ca
![Page 64: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/64.jpg)
Computing Pattern Precision
For question type:Search only on qtermDownload top N web docs (N=1000)Select only sentences w/qtermFor each pattern, check if
a) matches w/any aterm; Co
b)matches/w right aterm: Ca
Compute precision P = Ca/Co
Retain if match > 5 examples
![Page 65: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/65.jpg)
Pattern Precision ExampleQterm: MozartPattern: <NAME> was born in <ANSWER>
![Page 66: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/66.jpg)
Pattern Precision ExampleQterm: MozartPattern: <NAME> was born in <ANSWER>Near-Miss: Mozart was born in SalzburgMatch: Mozart born in 1756.
![Page 67: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/67.jpg)
Pattern Precision ExampleQterm: MozartPattern: <NAME> was born in <ANSWER>Near-Miss: Mozart was born in SalzburgMatch: Mozart born in 1756.Precisions:
1.0 <NAME> (<ANSWER> - )0.6 <NAME> was born in <ANSWER>….
![Page 68: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/68.jpg)
NuancesAlternative forms:
Need to allow for alternate forms of question or answerE.g. dates in different formats, full names, etc
Use alternate forms in pattern search
![Page 69: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/69.jpg)
NuancesAlternative forms:
Need to allow for alternate forms of question or answerE.g. dates in different formats, full names, etc
Use alternate forms in pattern search
Precision assessment:Use other examples of same type to computeCross-checks patterns
![Page 70: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/70.jpg)
Answer Selection by Pattern
Identify question types and termsFilter retrieved passages, replace qterm by tagTry to match patterns and answer spansDiscard duplicates and sort by pattern precision
![Page 71: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/71.jpg)
Pattern Sets WHY-FAMOUS
1.0 <ANSWER> <NAME> called1.0 laureate <ANSWER> <NAME>1.0 by the <ANSWER> , <NAME> ,1.0 <NAME> - the <ANSWER> of1.0 <NAME> was the <ANSWER> of
BIRTHYEAR 1.0 <NAME> ( <ANSWER> - )0.85 <NAME> was born on <ANSWER> ,0.6 <NAME> was born in <ANSWER>0.59 <NAME> was born <ANSWER>0.53 <ANSWER> <NAME> was born
![Page 72: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/72.jpg)
ResultsImproves, though better with web data
![Page 73: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/73.jpg)
Limitations & ExtensionsWhere are the Rockies? ..with the Rockies in the background
![Page 74: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/74.jpg)
Limitations & ExtensionsWhere are the Rockies? ..with the Rockies in the background
Should restrict to semantic / NE type
![Page 75: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/75.jpg)
Limitations & ExtensionsWhere are the Rockies? ..with the Rockies in the background
Should restrict to semantic / NE typeLondon, which…., lies on the River Thames<QTERM> word* lies on <ANSWER>
Wildcards impractical
![Page 76: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/76.jpg)
Limitations & ExtensionsWhere are the Rockies? ..with the Rockies in the background
Should restrict to semantic / NE typeLondon, which…., lies on the River Thames<QTERM> word* lies on <ANSWER>
Wildcards impractical
Long-distance dependencies not practical
![Page 77: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/77.jpg)
Limitations & ExtensionsWhere are the Rockies? ..with the Rockies in the background
Should restrict to semantic / NE typeLondon, which…., lies on the River Thames<QTERM> word* lies on <ANSWER>
Wildcards impractical
Long-distance dependencies not practicalLess of an issue in Web search
Web highly redundant, many local dependenciesMany systems (LCC) use web to validate answers
![Page 78: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/78.jpg)
Limitations & ExtensionsWhen was LBJ born?Tower lost to Sen. LBJ, who ran for both the…
![Page 79: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/79.jpg)
Limitations & ExtensionsWhen was LBJ born?Tower lost to Sen. LBJ, who ran for both the…
Requires information about:Answer length, type; logical distance (1-2 chunks)
![Page 80: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/80.jpg)
Limitations & ExtensionsWhen was LBJ born?Tower lost to Sen. LBJ, who ran for both the…
Requires information about:Answer length, type; logical distance (1-2 chunks)
Also, Can only handle single continuous qterms Ignores caseNeeds handle canonicalization, e.g of names/dates
![Page 81: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/81.jpg)
Integrating Patterns IIFundamental problem:
![Page 82: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/82.jpg)
Integrating Patterns IIFundamental problem:
What is there’s no pattern??
![Page 83: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/83.jpg)
Integrating Patterns IIFundamental problem:
What is there’s no pattern??No pattern -> No answer!!!
More robust solution:Not JUST patterns
![Page 84: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/84.jpg)
Integrating Patterns IIFundamental problem:
What is there’s no pattern??No pattern -> No answer!!!
More robust solution:Not JUST patterns Integrate with machine learning
MAXENT!!!Re-ranking approach
![Page 85: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/85.jpg)
Answering w/Maxent
![Page 86: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/86.jpg)
Feature FunctionsPattern fired:
Binary feature
![Page 87: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/87.jpg)
Feature FunctionsPattern fired:
Binary featureAnswer frequency/Redundancy factor:
# times answer appears in retrieval results
![Page 88: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/88.jpg)
Feature FunctionsPattern fired:
Binary featureAnswer frequency/Redundancy factor:
# times answer appears in retrieval resultsAnswer type match (binary)
![Page 89: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/89.jpg)
Feature FunctionsPattern fired:
Binary featureAnswer frequency/Redundancy factor:
# times answer appears in retrieval resultsAnswer type match (binary)Question word absent (binary):
No question words in answer span
![Page 90: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/90.jpg)
Feature FunctionsPattern fired:
Binary featureAnswer frequency/Redundancy factor:
# times answer appears in retrieval resultsAnswer type match (binary)Question word absent (binary):
No question words in answer spanWord match:
Sum of ITF of words matching b/t questions & sent
![Page 91: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/91.jpg)
Training & TestingTrained on NIST QA questions
Train: TREC 8,9; Cross-validation: TREC-10
5000 candidate answers/questionPositive examples:
NIST pattern matchesNegative examples:
NIST pattern doesn’t matchTest: TREC-2003: MRR: 28.6%; 35.6% exact top 5
![Page 92: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/92.jpg)
Noisy Channel QAEmployed for speech, POS tagging, MT, summ,
etcIntuition:
Question is a noisy representation of the answer
![Page 93: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/93.jpg)
Noisy Channel QAEmployed for speech, POS tagging, MT, summ,
etcIntuition:
Question is a noisy representation of the answerBasic approach:
Given a corpus of (Q,SA) pairsTrain P(Q|SA)Find sentence with answer as
Si,Aij that maximize P(Q|Si,Aij)
![Page 94: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/94.jpg)
QA Noisy ChannelA: Presley died of heart disease at Graceland in 1977,
and..Q: When did Elvis Presley die?
![Page 95: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/95.jpg)
QA Noisy ChannelA: Presley died of heart disease at Graceland in 1977,
and..Q: When did Elvis Presley die?
Goal:Align parts of Ans parse tree to question
Mark candidate answersFind highest probability answer
![Page 96: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/96.jpg)
ApproachAlignment issue:
![Page 97: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/97.jpg)
ApproachAlignment issue:
Answer sentences longer than questionsMinimize length gap
Represent answer as mix of words/syn/sem/NE units
![Page 98: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/98.jpg)
ApproachAlignment issue:
Answer sentences longer than questionsMinimize length gap
Represent answer as mix of words/syn/sem/NE unitsCreate ‘cut’ through parse tree
Every word –or an ancestor – in cutOnly one element on path from root to word
![Page 99: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/99.jpg)
ApproachAlignment issue:
Answer sentences longer than questionsMinimize length gap
Represent answer as mix of words/syn/sem/NE unitsCreate ‘cut’ through parse tree
Every word –or an ancestor – in cutOnly one element on path from root to word
Presley died of heart disease at Graceland in 1977, and..Presley died PP PP in DATE, and..When did Elvis Presley die?
![Page 100: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/100.jpg)
Approach (Cont’d)Assign one element in cut to be ‘Answer’Issue: Cut STILL may not be same length as Q
![Page 101: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/101.jpg)
Approach (Cont’d)Assign one element in cut to be ‘Answer’Issue: Cut STILL may not be same length as QSolution: (typical MT)
Assign each element a fertility 0 – delete the word; > 1: repeat word that many
times
![Page 102: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/102.jpg)
Approach (Cont’d)Assign one element in cut to be ‘Answer’Issue: Cut STILL may not be same length as QSolution: (typical MT)
Assign each element a fertility 0 – delete the word; > 1: repeat word that many times
Replace A words with Q words based on alignmentPermute result to match original QuestionEverything except cut computed with OTS MT code
![Page 103: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/103.jpg)
SchematicAssume cut, answer guess all equally likely
![Page 104: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/104.jpg)
Training Sample Generation
Given question and answer sentencesParse answer sentenceCreate cut s.t.:
Words in both Q & A are preservedAnswer reduced to ‘A_’ syn/sem class labelNodes with no surface children reduced to syn
classKeep surface form of all other nodes
20K TREC QA pairs; 6.5K web question pairs
![Page 105: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/105.jpg)
Selecting AnswersFor any candidate answer sentence:
Do same cut process
![Page 106: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/106.jpg)
Selecting AnswersFor any candidate answer sentence:
Do same cut processGenerate all candidate answer nodes:
Syntactic/Semantic nodes in tree
![Page 107: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/107.jpg)
Selecting AnswersFor any candidate answer sentence:
Do same cut processGenerate all candidate answer nodes:
Syntactic/Semantic nodes in treeWhat’s a bad candidate answer?
![Page 108: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/108.jpg)
Selecting AnswersFor any candidate answer sentence:
Do same cut processGenerate all candidate answer nodes:
Syntactic/Semantic nodes in treeWhat’s a bad candidate answer?
StopwordsQuestion words!
Create cuts with each answer candidate annotatedSelect one with highest probability by model
![Page 109: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/109.jpg)
Example Answer CutsQ: When did Elvis Presley die?SA1: Presley died A_PP PP PP, and …
SA2: Presley died PP A_PP PP, and ….
SA3: Presley died PP PP in A_DATE, and …
Results: MRR: 24.8%; 31.2% in top 5
![Page 110: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/110.jpg)
Error AnalysisComponent specific errors:
Patterns: Some question types work better with patternsTypically specific NE categories (NAM, LOC, ORG..)Bad if ‘vague’
Stats based:No restrictions on answer type – frequently ‘it’
Patterns and stats:‘Blatant’ errors:
Select ‘bad’ strings (esp. pronouns) if fit position/pattern
![Page 111: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/111.jpg)
Error AnalysisComponent specific errors:
Patterns: Some question types work better with patternsTypically specific NE categories (NAM, LOC, ORG..)Bad if ‘vague’
![Page 112: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/112.jpg)
Error AnalysisComponent specific errors:
Patterns: Some question types work better with patternsTypically specific NE categories (NAM, LOC, ORG..)Bad if ‘vague’
Stats based:No restrictions on answer type – frequently ‘it’
![Page 113: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/113.jpg)
Error AnalysisComponent specific errors:
Patterns: Some question types work better with patternsTypically specific NE categories (NAM, LOC, ORG..)Bad if ‘vague’
Stats based:No restrictions on answer type – frequently ‘it’
Patterns and stats:‘Blatant’ errors:
Select ‘bad’ strings (esp. pronouns) if fit position/pattern
![Page 114: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/114.jpg)
Combining UnitsLinear sum of weights?
![Page 115: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/115.jpg)
Combining UnitsLinear sum of weights?
Problematic:Misses different strengths/weaknesses
![Page 116: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/116.jpg)
Combining UnitsLinear sum of weights?
Problematic:Misses different strengths/weaknesses
Learning! (of course)Maxent re-ranking
Linear
![Page 117: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/117.jpg)
Feature Functions48 in totalComponent-specific:
Scores, ranks from different modulesPatterns. Stats, IR, even QA word overlap
![Page 118: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/118.jpg)
Feature Functions48 in totalComponent-specific:
Scores, ranks from different modulesPatterns. Stats, IR, even QA word overlap
Redundancy-specific:# times candidate answer appears (log, sqrt)
![Page 119: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/119.jpg)
Feature Functions48 in totalComponent-specific:
Scores, ranks from different modulesPatterns. Stats, IR, even QA word overlap
Redundancy-specific:# times candidate answer appears (log, sqrt)
Qtype-specific:Some components better for certain types:
type+mod
![Page 120: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/120.jpg)
Feature Functions48 in totalComponent-specific:
Scores, ranks from different modulesPatterns. Stats, IR, even QA word overlap
Redundancy-specific:# times candidate answer appears (log, sqrt)
Qtype-specific:Some components better for certain types: type+mod
Blatant ‘errors’: no pronouns, when NOT DoW
![Page 121: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/121.jpg)
ExperimentsPer-module reranking:
Use redundancy, qtype, blatant, and feature from mod
![Page 122: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/122.jpg)
ExperimentsPer-module reranking:
Use redundancy, qtype, blatant, and feature from mod
Combined reranking:All features (after feature selection to 31)
![Page 123: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/123.jpg)
ExperimentsPer-module reranking:
Use redundancy, qtype, blatant, and feature from mod
Combined reranking:All features (after feature selection to 31)
Patterns: Exact in top 5: 35.6% -> 43.1%Stats: Exact in top 5: 31.2% -> 41%Manual/knowledge based: 57%
![Page 124: Answer Extraction](https://reader030.vdocuments.us/reader030/viewer/2022021100/5681629e550346895dd31502/html5/thumbnails/124.jpg)