Group Members:Satadru Biswas (05005021)

Tanmay Khirwadkar (05005016)Arun Karthikeyan Karra (05d05020)

CS 626-460 Course SeminarGroup-2

Question Answering

OutlineIntroduction

Why Question Answering ?

AskMSR

FALCON

Conclusion

IntroductionQuestion Answering (QA) is the task of

automatically answering a question posed in natural language.

To find the answer to a question, a QA computer program may use either a pre-structured database or a collection of natural language documents (a text corpus such as the World Wide Web or some local collection).

A few sample questionsQ: Who shot President Abraham Lincoln?A: John Wilkes Booth

Q: How many lives were lost in the Pan Am crash in Lockerbie?

A: 270

Q: How long does it take to travel from London to Paris through the Channel?

A: three hours 45 minutes

Q: Which Atlantic hurricane had the highest recorded wind speed?

A: Gilbert (200 mph)

Why Question Answering ?Google – Query driven search

Answers to a query are documentsQuestion Answering – Answer driven search

Answers to a query are phrases

ApproachesQuestion classificationFinding entailed answer typeUse of WordNetHigh-quality document search

Question Classes Class 1

Answer: single datum or list of items C: who, when, where, how (old, much, large) Example: Who shot President Abraham Lincoln? Answer: John Wilkes Booth

Class 2 A: multi-sentence C: extract from multiple sentences Example: Who was Picasso? Answer: Picasso was great Spanish painter

Class 3 A: across several texts C: comparative/contrastive Example: What are the Valdez Principles?

Question Classes (contd…)

Class 4 A: an analysis of retrieved information C: synthesized coherently from several retrieved fragments Example: Which Atlantic hurricane had the highest recorded

wind speed? Answer: Gilbert (200 mph)

Class 5 A: result of reasoning C: word/domain knowledge and common sense reasoning Example: What did Richard Feynman say upon hearing he

would receive the Nobel Prize in Physics?

Types of QAClosed-domain question answering deals with

questions under a specific domain, and can be seen as an easier task because NLP systems can exploit domain-specific knowledge frequently formalized in ontologies.

Open-domain question answering deals with questions about nearly everything, and can only rely on general ontologies and world knowledge. On the other hand, these systems usually have much more data available from which to extract the answer.

QA - ConceptsQuestion Classes: Different types of questions

require the use of different strategies to find the answer.

Question Processing: A semantic model of question understanding and processing is needed, one that would recognize equivalent questions, regardless of the speech act or of the words, syntactic inter-relations or idiomatic forms.

Context and QA: Questions are usually asked within a context and answers are provided within that specific context.

Data sources for QA: Before a question can be answered, it must be known what knowledge sources are available.

Cont...Answer Extraction: Answer extraction depends on the

complexity of the question, on the answer type provided by question processing, on the actual data where the answer is searched, on the search method and on the question focus and context.

Answer Formulation: The result of a QA system should be presented in a way as natural as possible.

Real time question answering: There is need for developing Q&A systems that are capable of extracting answers from large data sets in several seconds, regardless of the complexity of the question, the size and multitude of the data sources or the ambiguity of the question.

Multi-lingual QA: The ability to answer a question posed in one language using an answer corpus in another language (or even several).

Cont...Interactive QA: Often the questioner might

want not only to reformulate the question, but (s)he might want to have a dialogue with the system.

Advanced reasoning for QA: More sophisticated questioners expect answers which are outside the scope of written texts or structured databases.

User profiling for QA: The user profile captures data about the questioner, comprising context data, domain of interest, reasoning schemes frequently used by the questioner, common ground established within different dialogues between the system and the user etc.

Tanmay Khirwadkar

Question Answering with the Help of WEB

Issues with traditional QA SystemsRetrieval is performed against small set of

documentsExtensive use of linguistic resources

POS tagging, Named Entity Tagging, WordNet etc.

Difficult to recognize answers that do not match question syntaxE.g. Q: Who shot President Abraham Lincoln?

A: John Wilkes Booth is perhaps America’s most infamous assassin having fired the bullet that killed Abraham Lincoln.

The Web can help !Web – A gigantic data repository with

extensive data redundancyFactoids likely to be expressed in hundreds

of different waysAt-least a few will match the way the

question was askedE.g. Q: Who shot President Abraham Lincoln?

A: John Wilkes Booth shot President Abraham Lincoln.

AskMSRBased on Data-Redundancy of the WebProcess the question

Form a web-search engine queryRecognize the answer-type

Rank answers on basis of frequencyProject the answers on TREC-corpus

1. Query ReformulationQuestion is often syntactically close to

answerE.g. Where is the Louvre Museum located?

The Louvre Museum is located in ParisWho created the character of Scrooge?

Charles Dickens created the character of Scrooge.

1. Query ReformulationClassify the query into 7 categories

Who, When, Where …Hand-crafted category-specific rewrite rules

[String, L/R/-, Weight]Weight – preference for a query

“Abraham Lincoln born on” preferred to “Abraham” “Lincoln” “born”

String – Simple String Manipulations

1. Query ReformulationE.g. For ‘where’ questions move ‘is’ to all

possible locations –Q: What is relative humidity?

[“is relative humidity”, LEFT, 5][”relative is humidity”, RIGHT, 5][”relative humidity is”, RIGHT, 5][”relative humidity”, NULL, 2][”relative” AND “humidity”, NULL, 1]

Some rewrites may be non-sensical

2. Query Search EngineSend all rewrites to a Web search engineRetrieve top N answers (100-200)For speed, rely just on search engine’s

“snippets”, not the full text of the actual document

3. N-gram HarvestingProcess the snippet to retrieve string to

left/right of queryEnumerate all n-grams (1, 2 and 3)Score of n-gram -

Occurrence frequency weighted by ‘weight’ of rewrite rule that fetched the summary

Formula:

summariesgramn

weightrewritegramnweight )(

4. Filtering AnswersApply filters based on question-types of

queriesRegular ExpressionsNatural Language Analysis

E.g. “Genghis Khan”, “Benedict XVI”

Boost score of answer when it matches expected answer-type

Remove answers from candidate listWhen set of answers is a closed set

“Which country …” , “How many times …”

5. Answer TilingShorter N-grams have higher weights

Solution: Perform tilingCombine overlapping shorter n-grams into

longer n-gramsScore = maximum(constituent n-grams)E.g.

Pierre Baron (5)

Baron de Coubertin (20) de Coubertin (10)

Pierre Baron de Coubertin (20)

6. Answer ProjectionRetrieve support ing documents from

document collection for each answerUse a standard IR system

IR Query : Web-Query + Candidate Answer

Results

System Strict Lenient

AskMSR

MRR 0.347 0.434

No Answers 49.2 40.0

AskMSR2

MRR 0.347 0.437

No Answers 49.6 39.6

Qq q

Q rankMRR )

1(||

1

-Arun Karthikeyan Karra(05d05020)

FALCON(Boosting Knowledge for QA systems)

FALCON IntroductionAnother QA system

Integrates syntactic, semantic and pragmatic knowledge for achieving better performance

Handles question reformulations, incorporates Wordnet semantic net, performs unifications on semantic forms to extract answers

Architecture of FALCON

Source: FALCON: Boosting Knowledge for Answer Engines, Harabagiu et. al.

Working of FALCON: A gist

Source: FALCON: Boosting Knowledge for Answer Engines, Harabagiu et. al.

Question Reformulations (1)

Source: FALCON: Boosting Knowledge for Answer Engines, Harabagiu et. al.

Question Reformulations (2)

Source: FALCON: Boosting Knowledge for Answer Engines, Harabagiu et. al.

Expected Answer Type (1)

Expected Answer Type (2)

Source: FALCON: Boosting Knowledge for Answer Engines, Harabagiu et. al.

Semantic Knowledge

Source: FALCON: Boosting Knowledge for Answer Engines, Harabagiu et. al.

Key words and AlternationsMorphological Alternations

Lexical AlternationsWho killed Martin Luther King?How far is the moon?

Semantic Alternations

Results Reported

692 QuestionsKey word alternations used for 89

questionsTREC-9 (Text Retrieval Conference)

Source: FALCON: Boosting Knowledge for Answer Engines, Harabagiu et. al.

ConclusionQuestion Answering requires more complex

NLP techniques compared to other forms of Information Retrieval

Two main approaches;Data Redundancy – AskMSRBoosting Knowledge Base – FALCON

Ultimate Goal : System which we can ‘talk’ to

There is a long way to go ... And a lot more money to come

ReferencesData Intensive Question Answering, Eric Brill

et.al., TREC-10, 2001An Analysis of the AskMSR Question-Answering

System, Eric Brill et. al., Proceedings of the Conference on Empirical Methods in Natural Association for Computational Linguistics. Language Processing (EMNLP), Philadelphia, July 2002, pp. 257-264.

FALCON: Boosting Knowledge for Answer Engines, Sanda Harabagiu, Dan Moldovan et. al., Southern Methodist University, TREC-9, 2000.

Wikipedia

EXTRA SLIDES

Abductive Knowledge