argument generation with retrieval, planning, and...
TRANSCRIPT
![Page 1: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/1.jpg)
July 30, 2019ACL
Florence, Italy
Argument Generation with Retrieval, Planning, and Realization
1
Xinyu Hua, Zhe Hu, and Lu Wang
Project page: https://xinyuhua.github.io/Resources/acl19
![Page 2: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/2.jpg)
Motivation
•One of the most useful function of language is persuasion.
2
![Page 3: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/3.jpg)
Motivation
•One of the most useful function of language is persuasion.• Argumentation, in its various forms, affects critical
decisions we make about our lives.
3
Political debate Decision making Legal debate
![Page 4: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/4.jpg)
Motivation
• Constructing high quality arguments is difficult.- Abundant world knowledge is necessary.- Argumentative structure requires careful planning.
4
![Page 5: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/5.jpg)
Motivation
• Constructing high quality arguments is difficult.- Abundant world knowledge is necessary.- Argumentative structure requires careful planning.
• Recent deep neural generation models have shown impressive ability in creating long fluent text.
5
![Page 6: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/6.jpg)
GPT-2’s take on argumentative text
6
I believe death penalty should be abolished.
Credit: OpenAI GPT-2 https://talktotransformer.com/
![Page 7: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/7.jpg)
GPT-2’s take on argumentative text
7
I believe death penalty should be abolished. But the only place we can go is to the bar."
He said he supports the right to bear arms, and called on people to exercise their constitutionally guaranteed right to vote.
"Some of the people protesting in our city were actually protesting for their mother's rights, her rights to be able to walk the streets and not be threatened with a death sentence," Mr. O'Brien said.
Credit: OpenAI GPT-2 https://talktotransformer.com/
![Page 8: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/8.jpg)
Motivation
•Human-level argumentation is still hard to generate with current conditional LM based approaches.
8
![Page 9: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/9.jpg)
Motivation
•Human-level argumentation is still hard to generate with current conditional LM based approaches.
- World knowledge has to be acquired only from training data.
9
![Page 10: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/10.jpg)
Motivation
•Human-level argumentation is still hard to generate with current conditional LM based approaches.
- World knowledge has to be acquired only from training data. - Token level sequence models do not produce explicit text plans.
10
![Page 11: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/11.jpg)
Proposal
11
I believe death penalty should be abolished.
![Page 12: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/12.jpg)
Proposal
12
I believe death penalty should be abolished.Retrieval
The Grim Facts About Lethal Injection
Our justice system is a joke and we are asking other people…
The problem of innocence in death penalty cases
The evidence in death penalty cases is not always very strong.
List of exonerated death row inmates.
There had been 156 exonerations of prisoners on death row
![Page 13: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/13.jpg)
Proposal
13
I believe death penalty should be abolished.
Text planning q imperfect justice systemq unreliable evidenceq wrongful convictionq execution of innocent peopleq little effect on crime rateq<STOP>
![Page 14: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/14.jpg)
Proposal
14
I believe death penalty should be abolished.
Text planning q imperfect justice systemq unreliable evidenceq wrongful convictionq execution of innocent peopleq little effect on crime rateq<STOP>
Sentence 1: [unreliable evidence; wrongful conviction];
![Page 15: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/15.jpg)
Proposal
15
I believe death penalty should be abolished.
Text planning q imperfect justice systemq unreliable evidenceq wrongful convictionq execution of innocent peopleq little effect on crime rateq<STOP>
Sentence 1: [unreliable evidence; wrongful conviction];
Sentence 2: [imperfect justice system; execution of innocent people];
![Page 16: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/16.jpg)
Proposal
16
I believe death penalty should be abolished.
Text planning q imperfect justice systemq unreliable evidenceq wrongful convictionq execution of innocent peopleq little effect on crime rateq<STOP>
Sentence 1: [unreliable evidence; wrongful conviction];
Sentence 2: [imperfect justice system; execution of innocent people];
Sentence 3: [little effect on crime rate]
![Page 17: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/17.jpg)
Proposal
17
I believe death penalty should be abolished.
Text planning q imperfect justice systemq unreliable evidenceq wrongful convictionq execution of innocent peopleq little effect on crime rateq<STOP>
Sentence 1: [unreliable evidence; wrongful conviction];
Sentence 2: [imperfect justice system; execution of innocent people];
Sentence 3: [little effect on crime rate]
![Page 18: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/18.jpg)
Proposal
18
I believe death penalty should be abolished. Sentence 1: [unreliable evidence; wrongful conviction];
Sentence 2: [imperfect justice system; execution of innocent people];
Sentence 3: [little effect on crime rate]
Content realization
![Page 19: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/19.jpg)
Proposal
19
I believe death penalty should be abolished. Unreliable evidence might be used when there is no live witness, which results in wrongful conviction.
Sentence 1: [unreliable evidence; wrongful conviction];
Sentence 2: [imperfect justice system; execution of innocent people];
Sentence 3: [little effect on crime rate]
Content realization
![Page 20: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/20.jpg)
Proposal
20
I believe death penalty should be abolished.Unreliable evidence might be used when there is no live witness, which results in wrongful conviction. Our justice system is not perfect, court could order the execution of innocent people.
Sentence 1: [unreliable evidence; wrongful conviction];
Sentence 2: [imperfect justice system; execution of innocent people];
Sentence 3: [little effect on crime rate]
Content realization
![Page 21: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/21.jpg)
Proposal
21
I believe death penalty should be abolished.Unreliable evidence might be used when there is no live witness, which results in wrongful conviction. Our justice system is not perfect, court could order the execution of innocent people. Lastly, study has shown that death penalty does not reduce crime rate.
Sentence 1: [unreliable evidence; wrongful conviction];
Sentence 2: [imperfect justice system; execution of innocent people];
Sentence 3: [little effect on crime rate]
Content realization
![Page 22: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/22.jpg)
Proposal
22
I believe death penalty should be abolished.Unreliable evidence might be used when there is no live witness, which results in wrongful conviction. Our justice system is not perfect, court could order the execution of innocent people. Lastly, study has shown that death penalty does not reduce crime rate.
Sentence 1: [unreliable evidence; wrongful conviction];
Sentence 2: [imperfect justice system; execution of innocent people];
Sentence 3: [little effect on crime rate]
Content realization
CANDELA: Counter-Arguments with two step Neural Decoders and ExternaL knowledge Augmentation
![Page 23: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/23.jpg)
Roadmap• Prior Work• Argument Retrieval• Argument Generation Model• Experiments• Conclusion
23
![Page 24: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/24.jpg)
Prior Work
• Rule-based models [Reed, Long, and Fox, 1996; Careniniand Moore, 2000]• Retrieval-based systems [Sato et al., 2015; Reisert et al.,
2015; Yanase et al., 2015]• Concept-to-text generation [Moryossef, Goldberg, and
Dagan, 2019; Koncel-Kedziorski et al., 2019; Le et al., 2018; Wiseman, Shieber, and Rush, 2017]
24
![Page 25: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/25.jpg)
Roadmap• Prior Work• Argument Retrieval• Argument Generation Model• Experiments• Conclusion
25
![Page 26: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/26.jpg)
Argument Retrieval
•Goal: to collect both subjective and factual external resources that can form the argument talking points
26
![Page 27: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/27.jpg)
Argument Retrieval
• Goal: to collect both subjective and factual external resources that can form the argument talking points
• Indexed data:
27
Source # documentsWikipedia 5,743,901Washington Post 1,109,672The New York Times 1,952,446Reuters 1,052,592Wall Street Journal 2,059,128Total 11,917,739
![Page 28: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/28.jpg)
Argument Retrieval
• Goal: to collect both subjective and factual external resources that can form the argument talking points
• Indexed data:
28
Objective, fact-basedSource # documents
Wikipedia 5,743,901Washington Post 1,109,672The New York Times 1,952,446Reuters 1,052,592Wall Street Journal 2,059,128Total 11,917,739
![Page 29: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/29.jpg)
Argument Retrieval
• Goal: to collect both subjective and factual external resources that can form the argument talking points
• Indexed data:
29
Objective, fact-based
Left
Right
By https: //www.adfontesmedia.com/
Source # documentsWikipedia 5,743,901Washington Post 1,109,672The New York Times 1,952,446Reuters 1,052,592Wall Street Journal 2,059,128Total 11,917,739
![Page 30: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/30.jpg)
Ranking and Filtering• Step 1: Documents are breakdown into passages (of 3 sentences).
30
The American death penalty has a big innocence problem, and it is not going away. The events of last week show why.On Wednesday, Missouri planned to execute Marcellus Williams. The problem was that he may be innocent. Governor Eric Greitens wisely put that execution on hold while a panel investigates further. On Thursday, Florida did execute Mark Asay. We may never fully know whether he actually deserved the death penalty…
![Page 31: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/31.jpg)
Ranking and Filtering• Step 1: Documents are breakdown into passages (of 3 sentences).
31
The American death penalty has a big innocence problem, and it is not going away. The events of last week show why.On Wednesday, Missouri planned to execute Marcellus Williams. The problem was that he may be innocent. Governor Eric Greitens wisely put that execution on hold while a panel investigates further. On Thursday, Florida did execute Mark Asay. We may never fully know whether he actually deserved the death penalty…
![Page 32: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/32.jpg)
Ranking and Filtering• Step 1: Documents are breakdown into passages (of 3 sentences).
32
The American death penalty has a big innocence problem, and it is not going away. The events of last week show why.On Wednesday, Missouri planned to execute Marcellus Williams. The problem was that he may be innocent. Governor Eric Greitens wisely put that execution on hold while a panel investigates further. On Thursday, Florida did execute Mark Asay. We may never fully know whether he actually deserved the death penalty…
![Page 33: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/33.jpg)
Ranking and Filtering• Step 1: Documents are breakdown into passages (of 3 sentences).
33
The American death penalty has a big innocence problem, and it is
not going away. The events of last week show why.
On Wednesday, Missouri planned to execute Marcellus Williams. The
problem was that he may be innocent. Governor Eric Greitens wisely
put that execution on hold while a panel investigates further. On
Thursday, Florida did execute Mark Asay. We may never fully know
whether he actually deserved the death penalty…
![Page 34: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/34.jpg)
Ranking and Filtering• Step 1: Documents are breakdown into passages (of 3 sentences).• Step 2: Passages are retrieved and ranked based on input queries.
34
Death penalty is more rational than life in prison.
death penalty rational life in prison
QUERY
-----------------------------------------------
-----------------------------------------------
-----------------------------------------------
-----------------------------------------------
PASSAGESINPUT SENT
BM25
![Page 35: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/35.jpg)
Ranking and Filtering• Step 1: Documents are breakdown into passages (of 3 sentences).• Step 2: Passages are retrieved and ranked based on input queries.• Step 3: Passages with wrong stance are discarded.
35
Death penalty is more rational than life in prison.
PASSAGESINPUT SENT
Death penalty is the most effective way to deter severe crime such as murders and rapes…
![Page 36: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/36.jpg)
Ranking and Filtering• Step 1: Documents are breakdown into passages (of 3 sentences).• Step 2: Passages are retrieved and ranked based on input queries.• Step 3: Passages with wrong stance are discarded.
36
Death penalty is more rational than life in prison.
PASSAGESINPUT SENT
Death penalty is the most effective way to deter severe crime such as murders and rapes…
![Page 37: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/37.jpg)
Ranking and Filtering• Step 1: Documents are breakdown into passages (of 3 sentences).• Step 2: Passages are retrieved and ranked based on input queries.• Step 3: Passages with wrong stance are discarded.
37
Death penalty is more rational than life in prison.
PASSAGESINPUT SENT
Death penalty is the most effective way to deter severe crime such as murders and rapes…
![Page 38: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/38.jpg)
Roadmap• Prior Work• Argument Retrieval• Argument Generation Model• Experiments• Conclusion
38
![Page 39: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/39.jpg)
CANDELA Model• Sequence-to-sequence framework:
39
death penalty is more rational …
…
Encoding input with BiLSTM
![Page 40: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/40.jpg)
CANDELA Model• Sequence-to-sequence framework:
40
death penalty is more rational …
…
![Page 41: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/41.jpg)
CANDELA Model• Sequence-to-sequence framework:
41
death penalty is more rational …
…
S1
Text planning decoding
![Page 42: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/42.jpg)
CANDELA Model• Sequence-to-sequence framework:
42
death penalty is more rational …
…
S1
Text planning decoding
![Page 43: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/43.jpg)
CANDELA Model• Sequence-to-sequence framework:
43
death penalty is more rational …
…
S1
1
0
0
Argumentative function type (binary)
Text planning decoding
![Page 44: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/44.jpg)
Argumentative Function Type
•Goal: to inform the realization decoder of the sentence style• Argumentative content sentence
- to deliver crucial ideas and supply evidence, e.g. “unreliable evidence is used when there is no witness.”.
• Argumentative filler sentence- generic statement, e.g. “in reality I agree, but in practice this is
problematic.”
44
![Page 45: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/45.jpg)
sigmoid(!"#$%&'())
CANDELA Model• Sequence-to-sequence framework:
45
death penalty is more rational …
…
S1
Content selection for next sentence
![Page 46: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/46.jpg)
sigmoid(!"#$%&'())
CANDELA Model• Sequence-to-sequence framework:
46
death penalty is more rational …
…
S1
Content selection for next sentence
![Page 47: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/47.jpg)
CANDELA Model• Sequence-to-sequence framework:
47
death penalty is more rational …
…
S1
Content selection for next sentence
sigmoid(!"#$%&'())
![Page 48: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/48.jpg)
CANDELA Model• Sequence-to-sequence framework:
48
death penalty is more rational …
…
S1 S2
Content selection for next sentence
![Page 49: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/49.jpg)
CANDELA Model• Sequence-to-sequence framework:
49
death penalty is more rational …
…
S1 S2
Recurrently learn sentence representation
![Page 50: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/50.jpg)
CANDELA Model• Sequence-to-sequence framework:
50
death penalty is more rational …
…
S1 S2
Recurrently learn sentence representation
S3 S4
![Page 51: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/51.jpg)
CANDELA Model
51
S1 S2
Content realization decoding
S3 S4
<start>
![Page 52: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/52.jpg)
CANDELA Model
52
S1 S2
Content realization decoding
S3 S4
<start>
death
death
penalty
penalty
in
in
reality
…
![Page 53: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/53.jpg)
CANDELA Model
53
S1 S2
Content realization decoding
S3 S4
<start>
death
death
penalty
penalty
in
in
reality
…
Sentence hidden state is accessible to the word-level decoder
![Page 54: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/54.jpg)
CANDELA Model
54
S1 S2 S3 S4
<start>
death
death
penalty
penalty
in
in
reality
…
Content realization decoder(Word level decoder)
![Page 55: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/55.jpg)
Learning Objective
55
![Page 56: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/56.jpg)
Learning Objective
56
Cross-entropy on content realization decoding
![Page 57: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/57.jpg)
Learning Objective
57
Cross-entropy on content realization decoding
Cross-entropy on argumentative function type
![Page 58: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/58.jpg)
Learning Objective
58
Cross-entropy on content realization decoding
Cross-entropy on argumentative function type
Binary cross-entropy on content selection
![Page 59: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/59.jpg)
Roadmap• Prior Work• Argument Retrieval• Argument Generation Model• Experiments• Conclusion
59
![Page 60: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/60.jpg)
Experiments - Data
•Dataset: statement-argument pairs from /r/ChangeMyView community • 217K pairs for train, 33K and 36K for dev and test• LM pre-training: an extended set of replies (353K)
60
![Page 61: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/61.jpg)
Experiments - Data
•Dataset: statement-argument pairs from /r/ChangeMyView community • 217K pairs for train, 33K and 36K for dev and test• LM pre-training: an extended set of replies (353K)• Topics: politics and policy making related• Keyphrase memory: noun phrases/verb phrases that
contains a Wikipedia title OR a topic signature word [Lin and Hovy, 2000]
61
![Page 62: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/62.jpg)
Experiments - Data
62
Average # words per statement 383.7
Average # words per argument 66.0
Average # passage 4.3
Average # keyphrase 57.1
Input
Output
![Page 63: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/63.jpg)
Experiments - Models
• Comparisons:- Retrieval: directly return the top ranked passage as output- Seq2seq: encode input and generate counter-argument- Seq2seqAug: encode input + retrieved passages + keyphrases- our prior work (HW2018): with only Wikipedia passages and an
auxiliary keyphrase generation task
63
![Page 64: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/64.jpg)
Experiments - Results
• Evaluation setups:- System (realistic): passages are retrieved by using the input
statement only- Oracle (upper bound w.r.t retrieval): passages are retrieved by
using the gold-standard arguments
64
![Page 65: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/65.jpg)
Experiments - Results
• Automatic evaluation results (System)
65
BLEU-2 BLEU-4 ROUGE-2 METEOR #WordSystem
![Page 66: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/66.jpg)
Experiments - Results
• Automatic evaluation results (System)
66
BLEU-2 BLEU-4 ROUGE-2 METEOR #Word7.55 1.11 8.64 14.38 1236.92 2.13 13.02 15.08 68
SystemRetrievalSeq2seq
![Page 67: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/67.jpg)
Experiments - Results
• Automatic evaluation results (System)
67
BLEU-2 BLEU-4 ROUGE-2 METEOR #Word7.55 1.11 8.64 14.38 1236.92 2.13 13.02 15.08 688.26 2.24 13.79 15.75 783.64 0.92 8.83 11.78 51
SystemRetrievalSeq2seqSeq2seqAugHW2018
![Page 68: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/68.jpg)
Experiments - Results
• Automatic evaluation results (System)
68
BLEU-2 BLEU-4 ROUGE-2 METEOR #Word7.55 1.11 8.64 14.38 1236.92 2.13 13.02 15.08 688.26 2.24 13.79 15.75 783.64 0.92 8.83 11.78 5112.02 2.99 14.93 16.92 11912.33 2.86 14.53 16.60 123
SystemRetrievalSeq2seqSeq2seqAugHW2018CANDELACANDELAw/o psg
CANDELA model outperforms all comparisons.
![Page 69: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/69.jpg)
Experiments - Results
• Automatic evaluation results (Oracle)
69
BLEU-2 BLEU-4 ROUGE-2 METEOR #Word10.97 3.05 23.49 20.08 140
6.92 2.13 13.02 15.08 68
10.98 4.41 22.97 19.62 71
8.51 2.86 18.89 17.18 58
15.80 5.00 23.75 20.18 116
16.33 4.98 23.65 19.94 123
SystemRetrieval
Seq2seq
Seq2seqAug
HW2018
CANDELACANDELA w/o psg
3-4 BLEU/METEOR points to be expected from Oracle retrieval
![Page 70: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/70.jpg)
Human Evaluation
70
Grammaticality Appropriateness Content richness
![Page 71: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/71.jpg)
Human Evaluation
71
Grammaticality1 (ungrammatical): “to sought sentencing is is numerous at to to”
5 (fluent): “in theory i agree, but in reality the justice system is not perfect. ”
Appropriateness Content richness
![Page 72: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/72.jpg)
Human Evaluation
72
Grammaticality1 (ungrammatical): “to sought sentencing is is numerous at to to”
5 (fluent): “in theory i agree, but in reality the justice system is not perfect. ”
Appropriateness1 (off-topic): “the gap between rich and poor is the major issue.”
5 (on-topic, correct stance): “the problem with death penalty is that wrongful conviction exists, and it is irreversible in such cases.”
Content richness
![Page 73: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/73.jpg)
Human Evaluation
73
Grammaticality1 (ungrammatical): “to sought sentencing is is numerous at to to”
5 (fluent): “in theory i agree, but in reality the justice system is not perfect. ”
Appropriateness1 (off-topic): “the gap between rich and poor is the major issue.”
5 (on-topic, correct stance): “the problem with death penalty is that wrongful conviction exists, and it is irreversible in such cases.”
Content richness1 (generic response): “i don’t care.”
5 (sufficient supports): “a 2015 study showed that death penalty cases cost an average of one million more to prosecute, it has cost california more than four billion since 1978”
![Page 74: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/74.jpg)
Human Evaluation - Results
74
0
1
2
3
4
5
6
Grammaticality Appropriateness Content richness
Human Retrieval Seq2seqAug CANDELA
![Page 75: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/75.jpg)
Human Evaluation - Results
75
0
1
2
3
4
5
6
Grammaticality Appropriateness Content richness
Human Retrieval Seq2seqAug CANDELA
![Page 76: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/76.jpg)
Roadmap• Prior Work• Argument Retrieval• Argument Generation Model• Experiments• Conclusion
76
![Page 77: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/77.jpg)
Conclusion•We study the challenging argument generation task with diverse
external knowledge.
• Our proposed CANDELA system is able to conduct text planning and content realization, unified through an end-to-end trained model.
• This task still remains far from being solved. Future directions include adding common-sense knowledge, and multi-turn argumentation.
77
![Page 78: Argument Generation with Retrieval, Planning, and Realizationxinyuhua.github.io/resources/acl2019/slides_final.pdf · •Goal: to inform the realization decoder of the sentence style](https://reader035.vdocuments.us/reader035/viewer/2022071004/5fc19fef68d53e549c0d1c5f/html5/thumbnails/78.jpg)
Thank you!
•Online demo: https://xinyuhua.github.io/candela
• Project page: https://xinyuhua.github.io/Resources/acl19/
• Contact: [email protected]
78