persuasiveness and audience reactions in political speeches
Post on 14-Dec-2014
1.004 Views
Preview:
DESCRIPTION
TRANSCRIPT
Persuasiveness and Audience Reactions in Political Speeches
Marco Guerini
Marco Guerini Carlo Strapparava Oliviero Stock Danilo Giampiccolo Rachele Sprugnoli Giovanni Moretti
Contributors
INTRODUCTION
Persuasive NLP
• Persuasion is becoming a hot topic in Natural Language Processing.
• Automatic analysis and recognition of the persuasive impact of communication.
• Address the various effects which persuasive communication can have in different contexts on different audiences.
Approaches
• Knowledge-based: Starting from theory. • Data-driven: Starting from linguistic data.
Linguistic data should be possibly augmented with annotation of various audience reactions.
Resources Examples
• Long texts: Political Speeches
• Short texts: Posts on Social networks
• Short sentences: Advertising Slogans
• Single words: Evocative Brand Names
CORPUS DESCRIPTION
Characteristics & Collection
• CORPS: CORpus of tagged Political Speeches • Hypothesis: tags about audience reaction,
such as APPLAUSE, are indicators of hot-spots, where persuasion attempts succeeded
• Collection: Annotated speeches from various web sources
• Normalization: Metadata insertion (speaker, date, title, etc.) and Semi-automatic conversion of tags names to make them homogeneous
CORPS - Main Statistics
Total number of speeches: ~ 3,600
Total number of speakers: ~ 200
Total number of words: ~ 8 M
Total number of tags: ~ 66,000
Temporal range: from 1917 to 2010
Characteristics & Collection
CorpsFormatConverter
• Four annotators have been trained.
• Annotation supported by an ad-hoc standalone application.
• The tool facilitates the extraction of the speech text and metadata from the Web sources.
• The tool automatically converts the most frequent tags.
Example of Tags and Conversion
Document Structure ex. - JFK
{title} Ich bin ein Berliner {/title} {event} ----- {/event} {speaker} John F. Kennedy {/speaker} {date} 26 June 1963 {/date} {source} americanspeech.com {/source} {description} ----- {/description} {speech} … {/speech}
Speech Fragment ex. - JFK
Freedom has many difficulties and democracy is not perfect. But we have never had to put a wall up to keep our people in, to prevent them from leaving us. {APPLAUSE ; CHEERS} I want to say on behalf of my countrymen who live many miles away on the other side of the Atlantic, who are far distant from you, that they take the greatest pride, that they have been able to share with you, even from a distance, the story of the last 18 years. I know of no town, no city, that has been besieged for 18 years that still lives with the vitality and the force, and the hope, and the determination of the city of West Berlin. {APPLAUSE ; CHEERS}
Speeches Distribution
Number of speeches per Speaker.
Speeches Distribution
Temporal distribution of the Speeches
SINGLE TAGS {APPLAUSE} 46310 {LAUGHTER} 14055 {AUDIENCE} 1803 {BOOING} 756
{SPONTANEOUS-DEMONSTRATION} 313
{CHEERS} 234 {SUSTAINED APPLAUSE} 97 {STANDING-OVATION} 51
MULTIPLE TAGS
{LAUGHTER ; APPLAUSE} 1579
{CHEERS ; APPLAUSE} 837 OTHERS 47
SPECIAL TAGS {AUDIENCE-MEMBER} 999 {COMMENT} 787 {OTHER-SPEAK} 404
Tags Count
Tag Count
{AUDIENCE} Yes! {/AUDIENCE} 482 {AUDIENCE} No! {/AUDIENCE} 390
{AUDIENCE} Four more years! Four more years! {/AUDIENCE} 346
{AUDIENCE} Yes, sir {/AUDIENCE} 87
{AUDIENCE} U.S.A.! U.S.A.! U.S.A.! {/AUDIENCE} 41
{AUDIENCE} All right {/AUDIENCE} 39
{AUDIENCE} Flip-flop! Flip-flop! Flip-flop! {/AUDIENCE} 39
{AUDIENCE} Hooah. {/AUDIENCE} 38
{AUDIENCE} Reagan! Reagan! Reagan! {/AUDIENCE} 37
… … {AUDIENCE} Hooah! {/AUDIENCE} 24 {AUDIENCE} Tell it {/AUDIENCE} 23 … …
Audience Tags - Count
Tag Frequencies
{COMMENT="Inaudible"} 257 {COMMENT="A toast is offered"} 30 {COMMENT="The bill is signed"} 30
{COMMENT="The medal was presented"} 26
{COMMENT="The medal was awarded"} 24
{COMMENT="Recording interrupted"} 18
{COMMENT="The citation is read"} 18 {COMMENT="The citation was read"} 16 {COMMENT="Interruption"} 9
{COMMENT="A moment of silence was observed"} 8
… …
Comment Tags - Count
Audience Reactions Typologies
• Positive-Focus: a persuasive attempt that sets a positive focus in the audience. Tags considered:
{APPLAUSE} , {STANDING-OVATION} , {SUSTAINED-APPLAUSE} , {CHEERING} , etc.
• Negative-Focus: a persuasive attempt that sets a negative focus in the audience. Negative focus set towards the object of the speech not on the speaker.
{BOOING} , {AUDIENCE} No! {/AUDIENCE}
• Ironical: Indicate the use of ironical devices in persuasion. Tags considered:
{LAUGHTER} and multiple tags containing laughter.
Audience Reactions Typologies
• These 3 groups represents different effects which political communication can have in different contexts on different audiences.
Reaction Typology Count Percentage
POSITIVE-FOCUS TAGS 49275 0.74
IRONICAL TAGS 15660 0.24
NEGATIVE-FOCUS TAGS 1147 0.02
MACRO ANALYSIS
Tag Density
• How much “persuasive” is, on average, a speech or group of speeches?
• Compute how many audience reaction tags are present in a speech (normalize according to speech length).
Tag Density
• Given a set of speeches - e.g. Democrats’ speeches -, tag density can be computed in two different ways:
– Micro-averaged tag density (µ) - counting all tag occurrences in the set and dividing the result for the total number of words. – Macro-averaged tag density (M) - computing the tag density for each category (e.g. each Democrat speaker) and then averaging over the results of each speaker.
• µ gives the “real” tag density of the dataset, while M avoids over-representation of unbalanced classes (e.g. a vast majority of Bill Clinton’s speeches).
Tag Density
A set of n speeches, |ti| represents the number of tags in a given speech/category |wi| represents the number of words in the speech/category |C| represent the number of categories (speakers) in the set of speeches.
the set and then dividing the result for the total number of words contained inthose speeches (µ), or by computing the tag density for each category (Democratspeakers in our example) and then averaging over the results of each categoryin the set (M).
More formally, given a set of n speeches S, where a single speech is repre-sented with si (i.e. si ∈ S), |ti| represents the number of tags in a given speechsi and |wi| represents the number of words in the same speech; we can define µas:
µ =
�ni=1 |ti|�ni=1 |wi|
(1)
In a similar way M can be defined as:
M =
�|C|i=1
|ti||wi|
|C| (2)
where |C| represent the number of categories (speakers) in the set of speeches,and |ti| and |wi| represent the total number of tags and words for the category.
In the rest of the paper we will mainly provide micro-averaged tag den-sity, since it represents the more general density within the corpus, but macro-averaged values will be provided as well when necessary to further analysis.
In Table 7 statistics about main speakers are provided. We will not discussit in details, since it is out of the scope of the present paper to analyze thecharacteristic of each speaker; still we will introduce some interesting insightsafter aggregating the speakers in the subsequent tables.
Table 7. Main speakers statistics - Micro-averaged densities (µ)
Speaker Total Speeches Tag-Density PF-density I-density NF-density
Bill Clinton 889 0.007 0.005 0.002 0.00001
George W. Bush 427 0.015 0.012 0.002 0.00005
Ronald Reagan 388 0.004 0.001 0.003 0.00044
Dick Cheney 356 0.011 0.008 0.002 0.00061
Barack Obama 347 0.010 0.008 0.003 0.00007
John F. Kennedy 316 0.009 0.008 0.001 0.00000
Michelle Obama 107 0.009 0.005 0.003 0.00001
Margaret Thatcher 102 0.005 0.004 0.001 0.00001
Laura Bush 93 0.015 0.014 0.001 0.00000
Richard M. Nixon 61 0.006 0.005 0.000 0.00008
Al Gore 53 0.007 0.005 0.002 0.00004
Alan Keyes 51 0.004 0.003 0.001 0.00007
In Table 8 some statistics about tag-densities are provided according to twomain categorizations: Democrats/Conservatives and Male/Female speakers. Forthe first categorization we used a subset of the most prominent speaker (i.e. 12
the set and then dividing the result for the total number of words contained inthose speeches (µ), or by computing the tag density for each category (Democratspeakers in our example) and then averaging over the results of each categoryin the set (M).
More formally, given a set of n speeches S, where a single speech is repre-sented with si (i.e. si ∈ S), |ti| represents the number of tags in a given speechsi and |wi| represents the number of words in the same speech; we can define µas:
µ =
�ni=1 |ti|�ni=1 |wi|
(1)
In a similar way M can be defined as:
M =
�|C|i=1
|ti||wi|
|C| (2)
where |C| represent the number of categories (speakers) in the set of speeches,and |ti| and |wi| represent the total number of tags and words for the category.
In the rest of the paper we will mainly provide micro-averaged tag den-sity, since it represents the more general density within the corpus, but macro-averaged values will be provided as well when necessary to further analysis.
In Table 7 statistics about main speakers are provided. We will not discussit in details, since it is out of the scope of the present paper to analyze thecharacteristic of each speaker; still we will introduce some interesting insightsafter aggregating the speakers in the subsequent tables.
Table 7. Main speakers statistics - Micro-averaged densities (µ)
Speaker Total Speeches Tag-Density PF-density I-density NF-density
Bill Clinton 889 0.007 0.005 0.002 0.00001
George W. Bush 427 0.015 0.012 0.002 0.00005
Ronald Reagan 388 0.004 0.001 0.003 0.00044
Dick Cheney 356 0.011 0.008 0.002 0.00061
Barack Obama 347 0.010 0.008 0.003 0.00007
John F. Kennedy 316 0.009 0.008 0.001 0.00000
Michelle Obama 107 0.009 0.005 0.003 0.00001
Margaret Thatcher 102 0.005 0.004 0.001 0.00001
Laura Bush 93 0.015 0.014 0.001 0.00000
Richard M. Nixon 61 0.006 0.005 0.000 0.00008
Al Gore 53 0.007 0.005 0.002 0.00004
Alan Keyes 51 0.004 0.003 0.001 0.00007
In Table 8 some statistics about tag-densities are provided according to twomain categorizations: Democrats/Conservatives and Male/Female speakers. Forthe first categorization we used a subset of the most prominent speaker (i.e. 12
Overall Tag density (μ): 0.0084
PF-density (μ): 0.0062
I-density (μ): 0.0020
NF-density (μ): 0.0002
Tags Density - Corpus
Speaker Speeches Tag-Density PF-density I-density NF-density
Bill Clinton 889 0.007 0.005 0.002 0.00001
George W. Bush 427 0.015 0.012 0.002 0.00005
Ronald Reagan 388 0.004 0.001 0.003 0.00044
Dick Cheney 356 0.011 0.008 0.002 0.00061
Barack Obama 347 0.01 0.008 0.003 0.00007
John F. Kennedy 316 0.009 0.008 0.001 0
Michelle Obama 107 0.009 0.005 0.003 0.00001
Margaret Thatcher 102 0.005 0.004 0.001 0.00001
Laura Bush 93 0.015 0.014 0.001 0
Richard M. Nixon 61 0.006 0.005 0 0.00008
Al Gore 53 0.007 0.005 0.002 0.00004
Alan Keyes 51 0.004 0.003 0.001 0.00007
Tags Density – Main Speakers
Tags Density – Party and Gender
Party Corpus-Cover. Tag-Density PF-density I-density NF-density
Democrats 0.45 0.0075 0,0055 0,0019 0,000027
Conservatives 0.55 0.0097 0,0072 0,0022 0,000309
Gender Corpus-Cover. Tag-Density PF-density I-density NF-density
Females 0.11 0.0085 0.0067 0.0018 0.000007
Males 0.89 0.0083 0.0062 0.0020 0.000158
Micro-averaged densities (μ)
Party Corpus-Cover. Tag-Density PF-density I-density NF-density
Democrats 0.45 0.0076 0.0056 0.0019 0.000036
Conservatives 0.55 0.0094 0.0076 0.0017 0.000199
Gender Corpus-Cover. Tag-Density PF-density I-density NF-density
Females 0.11 0.0068 0.0055 0.0013 0.0000007
Males 0.89 0.0070 0.0052 0.0017 0.0000444
Macro-averaged densities (M)
Tags Density – Party and Gender
Tags Density – Party and Gender
• While the Democrats/Conservatives partition is well balanced (0.45 vs. 0.55), the Males/Females partition is unbalanced (0.89 vs. 0.11).
• Tag density is slightly higher for Conservative speakers (the same holds for positive-focus tags), while the ironical-focus tags have almost the same density in both groups.
• Analysis ex. Negative-focus tags (representing a more “aggressive” kind of rhetoric): density in the Conservative group is 11 times higher than the in Democrats. A similar consideration for the male/female distinction: while other tag densities are almost the same, for the negative-focus tags we have a density 60 times higher for male speakers.
Tag Density - Temporal Distribution
Language and Micro Analysis
Language Persuasiveness
• Are there words, linguistic expressions that are more “persuasive” than others?
• In a speech not all text fragments have the same importance. Consider audience reaction tags.
Possible Uses
• Persuasive expression mining. recognition and classification of phenomena such as audience reactions, speaker vocal effort can improve information retrieval (Bertoldi et al. 2002; Hu et al., 2008). New approaches for extracting relevant linguistic material, e.g. words persuasive impact (pi), see (Guerini et al., 2008).
• Automatic analysis of political communication. Computational linguistics to automatize analysis on politicians’ rhetoric. Considering audience’s reactions new rhetorical phenomena discovered (vs. traditional approaches based on words counting).
• Prediction of text impact. Machine learning for predicting the persuasive impact of novel speeches (Strapparava et al., 2008).
• Persuasive natural language generation. Eg. lexical choice: on the basis of lemma impact rather than lemma use.
Approach
• In analyzing CORPS, we focused on the lexical level.
• We considered: – Windows of different width wn of terms
preceding audience reactions tags. – The typology of audience reaction.
Approach ex. Fragment from JFK
Freedom has many difficulties and democracy is not perfect. But we have never had to put a wall up to keep our people in, to prevent them from leaving us. {APPLAUSE ; CHEERS} I want to say on behalf of my countrymen who live many miles away on the other side of the Atlantic, who are far distant from you, that they take the greatest pride, that they have been able to share with you, even from a distance, the story of the last 18 years. I know of no town, no city, that has been besieged for 18 years that still lives with the vitality and the force, and the hope, and the determination of the city of West Berlin. {APPLAUSE ; CHEERS}
wn = 15
positive-focus
Valence and Persuasion
The phase that leads to audience reaction, if it presents valence dynamics, is characterized by a valence crescendo
-
Words persuasive impact
• Basic idea: a word is more persuasive if at the same time its occurrences appear close to audience reactions tags and they do not appear far from them.
• We extracted “persuasive words” by using a coefficient of persuasive impact (pi) based on a weighted tf-idf (pi = tf × idf).
Words persuasive impact (cont’d)
• We created a “virtual document” by collecting terms inside windows preceding audience reactions tags (wn = 15).
• |D| = number of speeches in the corpus (included the virtual document)
• ni = number of times the term (word) ti appears in the virtual document
• Σni si = sum of word scores (closer to the tag, higher score) • Σknk = number of occurrences of all words in the virtual
document = wn × |tags number| • |{d : d ∋ ti}| = number of documents where the term ti
appears (we made a hypothesis of equidistribution)
Corpus Pre-processing
• POS-tagged all the speeches to reduce data sparseness, e.g.
– win, won, wins win#v – war, wars war#n
Topmost Persuasive Words
Advantages
For persuasive political communication the approach using the persuasive impact (pi) of words is much more effective than simple word count.
Examples of Use - Reagan
Many qualitative researches on Reagan’s (aka “the great communicator”) rhetorics: conversational style, irony, etc. • Great Communicator? 32 Reagan’s speeches, mean tag
density 1/2 of the whole corpus (t-test; α < 0.001). Being a “great communicator” not bound to “firing up” rate.
• Reagan’s style: “simple and conversational”. Hp: conversational style more polysemic than a “cultured” style (richer in technical, less polysemic, terms). No statistical diff. between mean polysemy of Reagan’s words and whole corpus. But mean polysemy of Reagan persuasive words is double of the whole corpus (t-test; α < 0.001).
• Use of irony: Density of ironical tags in Reagan’s speeches almost double as compared to the whole corpus (t-test; α < 0.001). In Reagan’s speeches the mean ironical-tags ratio (mtri) is about 7.5 times greater than the mtri of the whole corpus (t-test; α < 0.001).
Examples of Use – Bush and 9/11
• How do political speeches change after key historical events? Bush’s speeches before and after 9/11 (70 + 70 speeches)
– While words positive valence remains unvaried, the negative increases by 15% (t-test; α < 0.001).
– Words counts only partially reflects word impact…
Lemma pi before pi after Count before Count after
win#v 112 7 27 52
justice#n x 9 15 111
military#n 197 36 23 29
defeat#v x 16 1 44
right#r x 25 94 55
victory#n 826 65 9 26
evil#a - 129 0 44
death#n 4 450 65 32
war#n 36 x 80 258
soldier#n 70 296 20 47
tax#n x 93 702 81
drug-free#a 87 x 9 3
leadership#n 81 261 40 75
future#n 83 394 54 51
dream#n 99 321 77 30
Notes. In the second and third column, the number represents the rank in the list of persuasive words; an “x” indicates a pi = 0; an “–” indicates the word is not present in the corpus at all. In the fourth and fifth columns the total number of occurrences.
Bush and 9/11- Analysis Example
• For every word, we can record an increase or decrease of use (word count) compared with an increase or decrease of persuasiveness (pi).
• Let us consider the words military#n or treat#v. Both words are used almost the same number of times before and after 9/11. So their informativeness, based on number of occurrences, is null. But considering the persuasiveness score, we see that their impact varies (respectively from 197 to 36 and from 54 to 473).
• Let us also consider the word war#n; if we consider only the number of occurrences, we could infer that after 9/11 this topic was much more “felt” (mentioned three times more after 9/11), but if we look at persuasiveness we see that before 9/11 the word war#n was very “popular” (position 36) while after it never got audiences’ reactions.
PREDICTION OF PERSUASIVE EFFECTS
Experiments
• Using machine learning for predicting the persuasive impact of novel discourses. – Distinguishing Democrats from Republicans – Predicting the passages that trigger a positive
audience reaction – Cross classification (training made on adverse
party speeches, and test on the others) – Experimenting the classifiers on plain and
typical non-persuasive texts taken from British National Corpus and on speeches from the Obama-McCain political campaign.
Framework and Dataset
• We used the Support Vector Machines (SVM) framework. • Dataset preprocessing: to reduce sparseness, used lemma#pos
instead of tokens. • We did not make any frequency cutoff or feature selection. • All the speeches divided into fragments of about four sentences
(if a tag is present in the fragment the chunk ends at that point). • Obtained chunks are then labeled as Neutral (i.e., no tag), and
Positive-ironical (i.e., all positive-focus and ironical tags). We did not consider the negative-focus tags, since they are only a few.
• A total of ~38000 four-sentence chunks, roughly equally partitioned into the two considered labels.
• This accounts for a baseline of 0.5 in distinguishing between Neutral and Positive-ironical chunks. In all the experiments we randomly split the corpus in 80% training and 20% test.
Democrats vs. Republican
Precision Recall F1
Democrats 0.842 0.756 0.797
Republicans 0.773 0.854 0.811
Average (μ) 0.804 0.804 0.804
Positive vs. Neutral
• Whole Corpus
Precision Recall F1
Positive-Ironical 0.646 0.683 0.664
Neutral 0.676 0.641 0.658
Average (μ) 0.660 0.660 0.660
• Republican only
• Democrat Only
Positive vs. Neutral
Precision Recall F1
Positive-Ironical 0.660 0.766 0.709
Neutral 0.663 0.549 0.601
Average (μ) 0.661 0.661 0.661
Precision Recall F1
Positive-Ironical 0.666 0.674 0.670
Neutral 0.686 0.680 0.683
Average (μ) 0.676 0.676 0.676
Cross Classification
• Training on Democrats, Test on Republicans
• Training on Republicans, Test on Democrats
Precision Recall F1
Positive-Ironical 0.642 0.632 0.637
Neutral 0.579 0.599 0.589
Average (μ) 0.612 0.612 0.612
Precision Recall F1
Positive-Ironical 0.625 0.660 0.642
Neutral 0.658 0.626 0.641
Average (μ) 0.641 0.641 0.641
Untagged texts Classification
• Typical non-persuasive texts from BNC (A00 to A0H) Supposing all chunks are neutral
• Typical persuasive texts from the last Obama-McCain presidential campaign
Obama McCain
Positive-Ironical 2372 2360
Neutral 68 80
Total chunks 2440 2440
Total chunks 7243
Positive-Ironical 784
Neutral 6459
Prec/Rec/F1 0.892
Conclusions
• We have presented a resource and some approaches for persuasive NLP: – a Corpus of tagged Political Speeches (CORPS)
and a method for extracting persuasive words.
– a measure of persuasive impacts of words
Future Work
• Consider also persuasive rhetorical pattern extraction from CORPS.
• Consider windows width (wn) based on sentences rather than tokens.
• …
Some References
• Marco Guerini, Danilo Giampiccolo, Rachele Sprugnoli, Giovanni Moretti and Carlo Strapparava. The New Release of CORPS: Tagged Political Speeches for Persuasive Communication Processing, to appear.
• Marco Guerini, Carlo Strapparava and Oliviero Stock. CORPS: A corpus of tagged political speeches for persuasive communication processing. Journal of Information Technology & Politics 5 (1), 19-32, 2008.
• Marco Guerini, Carlo Strapparava and Oliviero Stock. Audience Reactions for information extraction about persuasive language in political communication. In M. Maybury (ed.) Multimodal Information Extraction, to appear.
• Carlo Strapparava, Marco Guerini and Oliviero Stock. Predicting Persuasiveness in Political Discourses. In Proceedings of LREC2010.
top related