generation of referring expressions: modeling partner effects surabhi gupta advisor: amanda stent...
TRANSCRIPT
Generation of Generation of Referring Referring
Expressions: Expressions: Modeling Partner Modeling Partner
EffectsEffectsSurabhi GuptaSurabhi Gupta
Advisor: Amanda StentAdvisor: Amanda Stent
Department of Computer Department of Computer ScienceScience
OutlineOutline
IntroductionIntroduction DataData Previous workPrevious work Modeling partner effectsModeling partner effects Generating NP postmodifiersGenerating NP postmodifiers A little statistical experimentA little statistical experiment Discussion and Future WorkDiscussion and Future Work
Referring ExpressionsReferring Expressions A referring expression denotes (or points A referring expression denotes (or points
to) an object in the world of a discourse.to) an object in the world of a discourse. Examples of referring expressions include Examples of referring expressions include the the
red chairred chair, , the 400 dollar red chairthe 400 dollar red chair and and 5 red 5 red chairschairs. .
Referring expressions are usually noun phrasesReferring expressions are usually noun phrases Improper construction of a referring Improper construction of a referring
expression can result in expression can result in referring expressions that are ambiguous (e.g. referring expressions that are ambiguous (e.g.
the bookthe book when there are two books). when there are two books). referring expressions that are too descriptive referring expressions that are too descriptive
and lead to false implicatures (e.g. and lead to false implicatures (e.g. the 400 the 400 dollar chairdollar chair when there is only one chair) when there is only one chair)
Structure of a Noun Structure of a Noun PhrasePhrase
A definite/indefinite noun phrase is A definite/indefinite noun phrase is constructed of:constructed of: An (optional) determiner or quantifier e.g. An (optional) determiner or quantifier e.g. aa, ,
threethree A number of premodifiers (adjectives, adverbs, A number of premodifiers (adjectives, adverbs,
noun modifiers) e.g. noun modifiers) e.g. redred A number of postmodifiers (prepositional A number of postmodifiers (prepositional
phrases, relative clauses) e.g. phrases, relative clauses) e.g. worth 400 worth 400 dollarsdollars, , that is redthat is red
Other noun phrases include pronouns, Other noun phrases include pronouns, proper nouns, deicticsproper nouns, deictics
Adaptation in Adaptation in ConversationConversation
When people talk with each other, they When people talk with each other, they adapt to the other’s choice of referring adapt to the other’s choice of referring expression (Clark 1996, Levinson 1983, expression (Clark 1996, Levinson 1983, Brennan 1987).Brennan 1987).
Example:Example: (A): Lets buy (A): Lets buy the 400 dollar red chairthe 400 dollar red chair (B): That’s a good idea. The (B): That’s a good idea. The chairchair matches matches
with the red table.with the red table. (A): The (A): The chair chair it is then.it is then.
Generation of Referring Generation of Referring Expressions in DialogExpressions in Dialog
When a computer constructs human When a computer constructs human language, it is called language, it is called generationgeneration NewsBlaster summaries, or Google NewsBlaster summaries, or Google
translationtranslation Generation for dialog must involve Generation for dialog must involve
consideration of the dialog partner consideration of the dialog partner (the human)(the human)
Good Generation of Good Generation of Referring ExpressionsReferring Expressions
The algorithm should generate a referring The algorithm should generate a referring expression for which the human reader can expression for which the human reader can identify the referent.identify the referent.
The algorithm should generate referring The algorithm should generate referring expressions that do not lead the human reader to expressions that do not lead the human reader to make false implicatures (Grice 1968).make false implicatures (Grice 1968).
The algorithm should model how conversational The algorithm should model how conversational partners adapt to each other.partners adapt to each other.
The algorithm should be able to generate the The algorithm should be able to generate the whole range of referring expressions observed in whole range of referring expressions observed in discourse.discourse.
The algorithm should be computationally feasible.The algorithm should be computationally feasible.
Our ObjectiveOur Objective We are building a model of referring We are building a model of referring
expression generation that captures expression generation that captures adaptation to partners in adaptation to partners in conversation.conversation.
Related work in this field does not Related work in this field does not include partner adaptation for dialog include partner adaptation for dialog (Dale and Reiter 1995, Siddharthan (Dale and Reiter 1995, Siddharthan and Copestake 2004).and Copestake 2004).
OutlineOutline
IntroductionIntroduction DataData Previous workPrevious work Modeling partner effectsModeling partner effects Generating NP postmodifiersGenerating NP postmodifiers A little statistical experimentA little statistical experiment Discussion and Future WorkDiscussion and Future Work
DataData Two corpora of spoken dialog rich in noun phrases:Two corpora of spoken dialog rich in noun phrases:
Maptask - Speaker A giving instructions to Maptask - Speaker A giving instructions to Speaker B about following directions in a mapSpeaker B about following directions in a map
Coconut - Two participants trying to buy furniture Coconut - Two participants trying to buy furniture by using both of their inventories and money.by using both of their inventories and money.
For each corpus, we:For each corpus, we: Automatically extracted the noun phrasesAutomatically extracted the noun phrases Annotated the noun phrases by hand for referent Annotated the noun phrases by hand for referent
(in a knowledge representation we built), type (in a knowledge representation we built), type (noun phrase or pronoun), and to indicate (noun phrase or pronoun), and to indicate whether the noun phrase was embedded in whether the noun phrase was embedded in another noun phrase. another noun phrase.
CoconutCoconut MaptaskMaptask
DefDef 116116 21182118
IndefIndef 967967 14111411
1st person pronoun1st person pronoun 440440 563563
2nd person pronoun2nd person pronoun 165165 12751275
3rd person pronoun3rd person pronoun 7979 614614
DeicticsDeictics 00 00
Proper NounsProper Nouns 00 00
Quantity NounsQuantity Nouns 291291 160160
Mass NounsMass Nouns 00 00
No ModifiersNo Modifiers 1313 113113
Not EmbeddedNot Embedded 229229 16331633
EmbeddedEmbedded 242242 2626
Set ConstructionsSet Constructions 00 00
Not in KRNot in KR 612612 18751875
NPs UsedNPs Used 471471 12941294
TotalTotal 17671767 59865986
OutlineOutline
IntroductionIntroduction DataData Previous workPrevious work Modeling partner effectsModeling partner effects Generating NP postmodifiersGenerating NP postmodifiers A little statistical experimentA little statistical experiment Discussion and Future WorkDiscussion and Future Work
Algorithms ComparedAlgorithms Compared Rule BasedRule Based
Dale and Reiter 1995Dale and Reiter 1995 With partner effects (x 2)With partner effects (x 2) With postmodifier ordering (x 4)With postmodifier ordering (x 4)
Siddharthan and Copestake 2004Siddharthan and Copestake 2004 With partner effects (x 2)With partner effects (x 2) With postmodifier ordering (x 4)With postmodifier ordering (x 4)
StatisticalStatistical Support Vector MachinesSupport Vector Machines
Rule-Based AlgorithmsRule-Based Algorithms
Terms used:Terms used: Contrast Set:Contrast Set: contains information of all contains information of all
the objects in the world.the objects in the world. Preferred list of attributes:Preferred list of attributes: the attributes the attributes
that are known for the objects. that are known for the objects. For Coconut: type, quantity, cost, color, stateFor Coconut: type, quantity, cost, color, state
E.g. three green high tables worth $400E.g. three green high tables worth $400 Intended Referent:Intended Referent: The object from the The object from the
world, which we are trying to describe.world, which we are trying to describe.
Dale and ReiterDale and Reiter
Basic idea:Basic idea: Specify the preference list by handSpecify the preference list by hand Repeat until all members of the Repeat until all members of the
contrast set are gone:contrast set are gone: Add the value for the next attribute from Add the value for the next attribute from
the preference list for the intended the preference list for the intended referent to the noun phrase to be referent to the noun phrase to be generatedgenerated
Example:Example:Preference list: {Type, Color, Cost, Quantity, Preference list: {Type, Color, Cost, Quantity, State}State}Contrast set: {Contrast set: {300 dollar red couch, 200 300 dollar red couch, 200 dollar green couch, 250 dollar brown tabledollar green couch, 250 dollar brown table } }Intended referent: Intended referent: 200 dollar green couch200 dollar green couchGenerated NP: Generated NP: green couchgreen couch
300 dollar red couch
200 dollar green couch
250 dollar brown table
Siddharthan and Siddharthan and CopestakeCopestake
Basic idea: See Dale and ReiterBasic idea: See Dale and Reiter Preference list is reordered by using Preference list is reordered by using
synonyms and antonyms of words in synonyms and antonyms of words in each attribute each attribute
Benefits to Rule Based Benefits to Rule Based AlgorithmsAlgorithms
They consider the way humans actually converse They consider the way humans actually converse ie. humans use unnecessary attributes, they also ie. humans use unnecessary attributes, they also begin mentioning a referring expression without begin mentioning a referring expression without scanning the entire list of distractors.scanning the entire list of distractors.
They do not attempt to look for the optimal They do not attempt to look for the optimal number of attributes. They just go through the number of attributes. They just go through the list of preferred attributes and iteratively list of preferred attributes and iteratively includes those attributes that rule out at least includes those attributes that rule out at least one distractor from the contrast set.one distractor from the contrast set.
There is no backtracking and the head noun is There is no backtracking and the head noun is always included.always included.
Disadvantages to Rule Disadvantages to Rule Based AlgorithmsBased Algorithms
They don’t generate the whole range of They don’t generate the whole range of referring expressionsreferring expressions Ones with postmodifiersOnes with postmodifiers PronounsPronouns DeicticsDeictics
They don’t model adaptation to They don’t model adaptation to partnerspartners..
OutlineOutline
IntroductionIntroduction DataData Previous workPrevious work Modeling partner effectsModeling partner effects Generating NP postmodifiersGenerating NP postmodifiers A little statistical experimentA little statistical experiment Discussion and Future WorkDiscussion and Future Work
Adding Partner EffectsAdding Partner Effects
A rule based algorithmA rule based algorithm Basic idea: See Dale and Reiter, Basic idea: See Dale and Reiter,
Siddharthan and CopestakeSiddharthan and Copestake Preference list is reordered to match Preference list is reordered to match
selection of attributes in previous selection of attributes in previous mentions of the intended referent.mentions of the intended referent.
Variant to this where those attributes Variant to this where those attributes mentioned previously are definitely mentioned previously are definitely included even if all the competitors have included even if all the competitors have been eliminated.been eliminated.
EvaluationEvaluation
Metric: Correct / Correct + Inserted Metric: Correct / Correct + Inserted + Deleted + Moved+ Deleted + Moved
Example:Example: Human: the big fat green catHuman: the big fat green cat Computer: the green happy catComputer: the green happy cat
Correct: the, catCorrect: the, cat Inserted: happyInserted: happy Deleted: big, fatDeleted: big, fat Moved: greenMoved: green Score: 2 / 6Score: 2 / 6
ResultsResults
The variant to our partner effects The variant to our partner effects algorithms performs significantly algorithms performs significantly better that our Baseline, Dale and better that our Baseline, Dale and Reiter and Siddharthan and Reiter and Siddharthan and Copestake for both the cropora Copestake for both the cropora used.used.
OutlineOutline
IntroductionIntroduction DataData Previous workPrevious work Modeling partner effectsModeling partner effects Generating NP postmodifiersGenerating NP postmodifiers A little statistical experimentA little statistical experiment Discussion and Future WorkDiscussion and Future Work
Discussion and Discussion and ConclusionsConclusions
The corpus you choose makes a The corpus you choose makes a differencedifference Maptask: Few distractors, no Maptask: Few distractors, no
significant different between Baseline, significant different between Baseline, Dale and Reiter and Siddharthan and Dale and Reiter and Siddharthan and CopestakeCopestake
Do partner effects make a Do partner effects make a difference?difference?
ReferencesReferences Advaith Siddharthan and Ann Copestake. 2004. Advaith Siddharthan and Ann Copestake. 2004.
Generating Referring Expressions in Open Domains. Generating Referring Expressions in Open Domains. In In Proceedings of the 42th Meeting of the Association Proceedings of the 42th Meeting of the Association for Computational Linguistics Annual Conference for Computational Linguistics Annual Conference (ACL 2004),(ACL 2004), Barcelona, Spain. Barcelona, Spain.
Grice, H P (1975). Logic and conversation. In P. Cole Grice, H P (1975). Logic and conversation. In P. Cole and J. Morgan, editors, and J. Morgan, editors, Syntax and SemanticsSyntax and Semantics: Vol 3, : Vol 3, Speech ActsSpeech Acts, pages 43-58. New York: Academic Press., pages 43-58. New York: Academic Press.
Grosz, B and Sidner, C (1986). Attention, intention, Grosz, B and Sidner, C (1986). Attention, intention, and the structure of discourse. and the structure of discourse. Computational Computational LinguisticsLinguistics, 12: 175-206., 12: 175-206.
Robert Dale and Ehud Reiter. 1995. Computational Robert Dale and Ehud Reiter. 1995. Computational interpretations of the Gricean maxims in the interpretations of the Gricean maxims in the generation of referring expressions. Cognitive generation of referring expressions. Cognitive Science, 19:233–263.Science, 19:233–263.
AcknowledgementsAcknowledgements
Dr. Amanda Stent, for all her time and Dr. Amanda Stent, for all her time and efforts during the last three years.efforts during the last three years.
The Natural Language Processing Lab The Natural Language Processing Lab in Computer Science.in Computer Science.
The Honors College for giving me the The Honors College for giving me the chance of working on this year long chance of working on this year long project.project.
NSFNSF
OutlineOutline
IntroductionIntroduction DataData Previous workPrevious work Modeling partner effectsModeling partner effects Generating NP postmodifiersGenerating NP postmodifiers A little statistical experimentA little statistical experiment Discussion and Future WorkDiscussion and Future Work
Generating with Generating with PostmodifiersPostmodifiers
Why? -- because previous algorithms don’t Why? -- because previous algorithms don’t but it’s a big part of the corpus we have but it’s a big part of the corpus we have used.used.
Random - randomly decide whether the Random - randomly decide whether the attribute selected should be a post attribute selected should be a post modifier or premodifiermodifier or premodifier
Unigrams - see where the attribute is in Unigrams - see where the attribute is in relation to the ‘type’.relation to the ‘type’.
Bigrams - statistics of pairs of attributes. Bigrams - statistics of pairs of attributes. E.g probability of finding an attribute E.g probability of finding an attribute given another.given another.
OutlineOutline
IntroductionIntroduction DataData Previous workPrevious work Modeling partner effectsModeling partner effects Generating NP postmodifiersGenerating NP postmodifiers A little statistical experimentA little statistical experiment Discussion and Future WorkDiscussion and Future Work
Support Vector MachinesSupport Vector Machines SVMs are a set of machine learning SVMs are a set of machine learning
algorithms for binary classification that algorithms for binary classification that have been applied to NLP.have been applied to NLP.
We used a set of SVMs, one per attribute, We used a set of SVMs, one per attribute, that voted “yes” or “no” to use this that voted “yes” or “no” to use this attribute at this point in the noun phrase.attribute at this point in the noun phrase. Maptask: 6 attributes, Coconut: 5 attributesMaptask: 6 attributes, Coconut: 5 attributes
We evaluated using:We evaluated using: 10-fold cross-validation for Maptask.10-fold cross-validation for Maptask. 4-fold cross-validation for Coconut.4-fold cross-validation for Coconut.