persuasiveness and audience reactions in political speeches

Persuasiveness and Audience Reactions in Political Speeches

Marco Guerini

Marco Guerini Carlo Strapparava Oliviero Stock Danilo Giampiccolo Rachele Sprugnoli Giovanni Moretti

Contributors

INTRODUCTION

Persuasive NLP

•  Persuasion is becoming a hot topic in Natural Language Processing.

•  Automatic analysis and recognition of the persuasive impact of communication.

•  Address the various effects which persuasive communication can have in different contexts on different audiences.

Approaches

•  Knowledge-based: Starting from theory. •  Data-driven: Starting from linguistic data.

Linguistic data should be possibly augmented with annotation of various audience reactions.

Resources Examples

•  Long texts: Political Speeches

•  Short texts: Posts on Social networks

•  Short sentences: Advertising Slogans

•  Single words: Evocative Brand Names

CORPUS DESCRIPTION

Characteristics & Collection

•  CORPS: CORpus of tagged Political Speeches •  Hypothesis: tags about audience reaction,

such as APPLAUSE, are indicators of hot-spots, where persuasion attempts succeeded

•  Collection: Annotated speeches from various web sources

•  Normalization: Metadata insertion (speaker, date, title, etc.) and Semi-automatic conversion of tags names to make them homogeneous

CORPS - Main Statistics

Total number of speeches: ~ 3,600

Total number of speakers: ~ 200

Total number of words: ~ 8 M

Total number of tags: ~ 66,000

Temporal range: from 1917 to 2010

Characteristics & Collection

CorpsFormatConverter

•  Four annotators have been trained.

•  Annotation supported by an ad-hoc standalone application.

•  The tool facilitates the extraction of the speech text and metadata from the Web sources.

•  The tool automatically converts the most frequent tags.

Example of Tags and Conversion

Document Structure ex. - JFK

{title} Ich bin ein Berliner {/title} {event} ----- {/event} {speaker} John F. Kennedy {/speaker} {date} 26 June 1963 {/date} {source} americanspeech.com {/source} {description} ----- {/description} {speech} … {/speech}

Speech Fragment ex. - JFK

Freedom has many difficulties and democracy is not perfect. But we have never had to put a wall up to keep our people in, to prevent them from leaving us. {APPLAUSE ; CHEERS} I want to say on behalf of my countrymen who live many miles away on the other side of the Atlantic, who are far distant from you, that they take the greatest pride, that they have been able to share with you, even from a distance, the story of the last 18 years. I know of no town, no city, that has been besieged for 18 years that still lives with the vitality and the force, and the hope, and the determination of the city of West Berlin. {APPLAUSE ; CHEERS}

Speeches Distribution

Number of speeches per Speaker.

Speeches Distribution

Temporal distribution of the Speeches

SINGLE TAGS {APPLAUSE} 46310 {LAUGHTER} 14055 {AUDIENCE} 1803 {BOOING} 756

{SPONTANEOUS-DEMONSTRATION} 313

{CHEERS} 234 {SUSTAINED APPLAUSE} 97 {STANDING-OVATION} 51

MULTIPLE TAGS

{LAUGHTER ; APPLAUSE} 1579

{CHEERS ; APPLAUSE} 837 OTHERS 47

SPECIAL TAGS {AUDIENCE-MEMBER} 999 {COMMENT} 787 {OTHER-SPEAK} 404

Tags Count

Tag Count

{AUDIENCE} Yes! {/AUDIENCE} 482 {AUDIENCE} No! {/AUDIENCE} 390

{AUDIENCE} Four more years! Four more years! {/AUDIENCE} 346

{AUDIENCE} Yes, sir {/AUDIENCE} 87

{AUDIENCE} U.S.A.! U.S.A.! U.S.A.! {/AUDIENCE} 41

{AUDIENCE} All right {/AUDIENCE} 39

{AUDIENCE} Flip-flop! Flip-flop! Flip-flop! {/AUDIENCE} 39

{AUDIENCE} Hooah. {/AUDIENCE} 38

{AUDIENCE} Reagan! Reagan! Reagan! {/AUDIENCE} 37

… … {AUDIENCE} Hooah! {/AUDIENCE} 24 {AUDIENCE} Tell it {/AUDIENCE} 23 … …

Audience Tags - Count

Tag Frequencies

{COMMENT="Inaudible"} 257 {COMMENT="A toast is offered"} 30 {COMMENT="The bill is signed"} 30

{COMMENT="The medal was presented"} 26

{COMMENT="The medal was awarded"} 24

{COMMENT="Recording interrupted"} 18

{COMMENT="The citation is read"} 18 {COMMENT="The citation was read"} 16 {COMMENT="Interruption"} 9

{COMMENT="A moment of silence was observed"} 8

… …

Comment Tags - Count

Audience Reactions Typologies

•  Positive-Focus: a persuasive attempt that sets a positive focus in the audience. Tags considered:

{APPLAUSE} , {STANDING-OVATION} , {SUSTAINED-APPLAUSE} , {CHEERING} , etc.

•  Negative-Focus: a persuasive attempt that sets a negative focus in the audience. Negative focus set towards the object of the speech not on the speaker.

{BOOING} , {AUDIENCE} No! {/AUDIENCE}

•  Ironical: Indicate the use of ironical devices in persuasion. Tags considered:

{LAUGHTER} and multiple tags containing laughter.

Audience Reactions Typologies

•  These 3 groups represents different effects which political communication can have in different contexts on different audiences.

Reaction Typology Count Percentage

POSITIVE-FOCUS TAGS 49275 0.74

IRONICAL TAGS 15660 0.24

NEGATIVE-FOCUS TAGS 1147 0.02

MACRO ANALYSIS

Tag Density

•  How much “persuasive” is, on average, a speech or group of speeches?

•  Compute how many audience reaction tags are present in a speech (normalize according to speech length).

Tag Density

•  Given a set of speeches - e.g. Democrats’ speeches -, tag density can be computed in two different ways:

–  Micro-averaged tag density (µ) - counting all tag occurrences in the set and dividing the result for the total number of words. –  Macro-averaged tag density (M) - computing the tag density for each category (e.g. each Democrat speaker) and then averaging over the results of each speaker.

•  µ gives the “real” tag density of the dataset, while M avoids over-representation of unbalanced classes (e.g. a vast majority of Bill Clinton’s speeches).

Tag Density

A set of n speeches, |ti| represents the number of tags in a given speech/category |wi| represents the number of words in the speech/category |C| represent the number of categories (speakers) in the set of speeches.

the set and then dividing the result for the total number of words contained inthose speeches (µ), or by computing the tag density for each category (Democratspeakers in our example) and then averaging over the results of each categoryin the set (M).

More formally, given a set of n speeches S, where a single speech is repre-sented with si (i.e. si ∈ S), |ti| represents the number of tags in a given speechsi and |wi| represents the number of words in the same speech; we can define µas:

µ =

�ni=1 |ti|�ni=1 |wi|

(1)

In a similar way M can be defined as:

M =

�|C|i=1

|ti||wi|

|C| (2)

where |C| represent the number of categories (speakers) in the set of speeches,and |ti| and |wi| represent the total number of tags and words for the category.

In the rest of the paper we will mainly provide micro-averaged tag den-sity, since it represents the more general density within the corpus, but macro-averaged values will be provided as well when necessary to further analysis.

In Table 7 statistics about main speakers are provided. We will not discussit in details, since it is out of the scope of the present paper to analyze thecharacteristic of each speaker; still we will introduce some interesting insightsafter aggregating the speakers in the subsequent tables.

Table 7. Main speakers statistics - Micro-averaged densities (µ)

Speaker Total Speeches Tag-Density PF-density I-density NF-density

Bill Clinton 889 0.007 0.005 0.002 0.00001

George W. Bush 427 0.015 0.012 0.002 0.00005

Ronald Reagan 388 0.004 0.001 0.003 0.00044

Dick Cheney 356 0.011 0.008 0.002 0.00061

Barack Obama 347 0.010 0.008 0.003 0.00007

John F. Kennedy 316 0.009 0.008 0.001 0.00000

Michelle Obama 107 0.009 0.005 0.003 0.00001

Margaret Thatcher 102 0.005 0.004 0.001 0.00001

Laura Bush 93 0.015 0.014 0.001 0.00000

Richard M. Nixon 61 0.006 0.005 0.000 0.00008

Al Gore 53 0.007 0.005 0.002 0.00004

Alan Keyes 51 0.004 0.003 0.001 0.00007

In Table 8 some statistics about tag-densities are provided according to twomain categorizations: Democrats/Conservatives and Male/Female speakers. Forthe first categorization we used a subset of the most prominent speaker (i.e. 12

the set and then dividing the result for the total number of words contained inthose speeches (µ), or by computing the tag density for each category (Democratspeakers in our example) and then averaging over the results of each categoryin the set (M).

More formally, given a set of n speeches S, where a single speech is repre-sented with si (i.e. si ∈ S), |ti| represents the number of tags in a given speechsi and |wi| represents the number of words in the same speech; we can define µas:

µ =

�ni=1 |ti|�ni=1 |wi|

(1)

In a similar way M can be defined as:

M =

�|C|i=1

|ti||wi|

|C| (2)

where |C| represent the number of categories (speakers) in the set of speeches,and |ti| and |wi| represent the total number of tags and words for the category.

In the rest of the paper we will mainly provide micro-averaged tag den-sity, since it represents the more general density within the corpus, but macro-averaged values will be provided as well when necessary to further analysis.

In Table 7 statistics about main speakers are provided. We will not discussit in details, since it is out of the scope of the present paper to analyze thecharacteristic of each speaker; still we will introduce some interesting insightsafter aggregating the speakers in the subsequent tables.

Table 7. Main speakers statistics - Micro-averaged densities (µ)

Speaker Total Speeches Tag-Density PF-density I-density NF-density

Bill Clinton 889 0.007 0.005 0.002 0.00001

George W. Bush 427 0.015 0.012 0.002 0.00005

Ronald Reagan 388 0.004 0.001 0.003 0.00044

Dick Cheney 356 0.011 0.008 0.002 0.00061

Barack Obama 347 0.010 0.008 0.003 0.00007

John F. Kennedy 316 0.009 0.008 0.001 0.00000

Michelle Obama 107 0.009 0.005 0.003 0.00001


Laura Bush 93 0.015 0.014 0.001 0.00000

Richard M. Nixon 61 0.006 0.005 0.000 0.00008

Al Gore 53 0.007 0.005 0.002 0.00004

Alan Keyes 51 0.004 0.003 0.001 0.00007

In Table 8 some statistics about tag-densities are provided according to twomain categorizations: Democrats/Conservatives and Male/Female speakers. Forthe first categorization we used a subset of the most prominent speaker (i.e. 12

Overall Tag density (μ): 0.0084

PF-density (μ): 0.0062

I-density (μ): 0.0020

NF-density (μ): 0.0002

Tags Density - Corpus

Speaker Speeches Tag-Density PF-density I-density NF-density

Bill Clinton 889 0.007 0.005 0.002 0.00001

George W. Bush 427 0.015 0.012 0.002 0.00005

Ronald Reagan 388 0.004 0.001 0.003 0.00044

Dick Cheney 356 0.011 0.008 0.002 0.00061

Barack Obama 347 0.01 0.008 0.003 0.00007

John F. Kennedy 316 0.009 0.008 0.001 0

Michelle Obama 107 0.009 0.005 0.003 0.00001


Laura Bush 93 0.015 0.014 0.001 0

Richard M. Nixon 61 0.006 0.005 0 0.00008

Al Gore 53 0.007 0.005 0.002 0.00004

Alan Keyes 51 0.004 0.003 0.001 0.00007

Tags Density – Main Speakers

Tags Density – Party and Gender

Party Corpus-Cover. Tag-Density PF-density I-density NF-density

Democrats 0.45 0.0075 0,0055 0,0019 0,000027

Conservatives 0.55 0.0097 0,0072 0,0022 0,000309

Gender Corpus-Cover. Tag-Density PF-density I-density NF-density

Females 0.11 0.0085 0.0067 0.0018 0.000007

Males 0.89 0.0083 0.0062 0.0020 0.000158

Micro-averaged densities (μ)

Party Corpus-Cover. Tag-Density PF-density I-density NF-density

Democrats 0.45 0.0076 0.0056 0.0019 0.000036

Conservatives 0.55 0.0094 0.0076 0.0017 0.000199

Gender Corpus-Cover. Tag-Density PF-density I-density NF-density

Females 0.11 0.0068 0.0055 0.0013 0.0000007

Males 0.89 0.0070 0.0052 0.0017 0.0000444

Macro-averaged densities (M)



•  While the Democrats/Conservatives partition is well balanced (0.45 vs. 0.55), the Males/Females partition is unbalanced (0.89 vs. 0.11).

•  Tag density is slightly higher for Conservative speakers (the same holds for positive-focus tags), while the ironical-focus tags have almost the same density in both groups.

•  Analysis ex. Negative-focus tags (representing a more “aggressive” kind of rhetoric): density in the Conservative group is 11 times higher than the in Democrats. A similar consideration for the male/female distinction: while other tag densities are almost the same, for the negative-focus tags we have a density 60 times higher for male speakers.

Tag Density - Temporal Distribution

Language and Micro Analysis

Language Persuasiveness

•  Are there words, linguistic expressions that are more “persuasive” than others?

•  In a speech not all text fragments have the same importance. Consider audience reaction tags.

Possible Uses

•  Persuasive expression mining. recognition and classification of phenomena such as audience reactions, speaker vocal effort can improve information retrieval (Bertoldi et al. 2002; Hu et al., 2008). New approaches for extracting relevant linguistic material, e.g. words persuasive impact (pi), see (Guerini et al., 2008).

•  Automatic analysis of political communication. Computational linguistics to automatize analysis on politicians’ rhetoric. Considering audience’s reactions new rhetorical phenomena discovered (vs. traditional approaches based on words counting).

•  Prediction of text impact. Machine learning for predicting the persuasive impact of novel speeches (Strapparava et al., 2008).

•  Persuasive natural language generation. Eg. lexical choice: on the basis of lemma impact rather than lemma use.

Approach

•  In analyzing CORPS, we focused on the lexical level.

•  We considered: – Windows of different width wn of terms

preceding audience reactions tags. – The typology of audience reaction.

Approach ex. Fragment from JFK

Freedom has many difficulties and democracy is not perfect. But we have never had to put a wall up to keep our people in, to prevent them from leaving us. {APPLAUSE ; CHEERS} I want to say on behalf of my countrymen who live many miles away on the other side of the Atlantic, who are far distant from you, that they take the greatest pride, that they have been able to share with you, even from a distance, the story of the last 18 years. I know of no town, no city, that has been besieged for 18 years that still lives with the vitality and the force, and the hope, and the determination of the city of West Berlin. {APPLAUSE ; CHEERS}

wn = 15

positive-focus

Valence and Persuasion

The phase that leads to audience reaction, if it presents valence dynamics, is characterized by a valence crescendo

-

Words persuasive impact

•  Basic idea: a word is more persuasive if at the same time its occurrences appear close to audience reactions tags and they do not appear far from them.

•  We extracted “persuasive words” by using a coefficient of persuasive impact (pi) based on a weighted tf-idf (pi = tf × idf).

Words persuasive impact (cont’d)

•  We created a “virtual document” by collecting terms inside windows preceding audience reactions tags (wn = 15).

•  |D| = number of speeches in the corpus (included the virtual document)

•  ni = number of times the term (word) ti appears in the virtual document

•  Σni si = sum of word scores (closer to the tag, higher score) •  Σknk = number of occurrences of all words in the virtual

document = wn × |tags number| •  |{d : d ∋ ti}| = number of documents where the term ti

appears (we made a hypothesis of equidistribution)

Corpus Pre-processing

•  POS-tagged all the speeches to reduce data sparseness, e.g.

–  win, won, wins win#v –  war, wars war#n

Topmost Persuasive Words

Advantages

For persuasive political communication the approach using the persuasive impact (pi) of words is much more effective than simple word count.

Examples of Use - Reagan

Many qualitative researches on Reagan’s (aka “the great communicator”) rhetorics: conversational style, irony, etc. •  Great Communicator? 32 Reagan’s speeches, mean tag

density 1/2 of the whole corpus (t-test; α < 0.001). Being a “great communicator” not bound to “firing up” rate.

•  Reagan’s style: “simple and conversational”. Hp: conversational style more polysemic than a “cultured” style (richer in technical, less polysemic, terms). No statistical diff. between mean polysemy of Reagan’s words and whole corpus. But mean polysemy of Reagan persuasive words is double of the whole corpus (t-test; α < 0.001).

•  Use of irony: Density of ironical tags in Reagan’s speeches almost double as compared to the whole corpus (t-test; α < 0.001). In Reagan’s speeches the mean ironical-tags ratio (mtri) is about 7.5 times greater than the mtri of the whole corpus (t-test; α < 0.001).

Examples of Use – Bush and 9/11

•  How do political speeches change after key historical events? Bush’s speeches before and after 9/11 (70 + 70 speeches)

–  While words positive valence remains unvaried, the negative increases by 15% (t-test; α < 0.001).

–  Words counts only partially reflects word impact…

Lemma pi before pi after Count before Count after

win#v 112 7 27 52

justice#n x 9 15 111

military#n 197 36 23 29

defeat#v x 16 1 44

right#r x 25 94 55

victory#n 826 65 9 26

evil#a - 129 0 44

death#n 4 450 65 32

war#n 36 x 80 258

soldier#n 70 296 20 47

tax#n x 93 702 81

drug-free#a 87 x 9 3

leadership#n 81 261 40 75

future#n 83 394 54 51

dream#n 99 321 77 30

Notes. In the second and third column, the number represents the rank in the list of persuasive words; an “x” indicates a pi = 0; an “–” indicates the word is not present in the corpus at all. In the fourth and fifth columns the total number of occurrences.

Bush and 9/11- Analysis Example

•  For every word, we can record an increase or decrease of use (word count) compared with an increase or decrease of persuasiveness (pi).

•  Let us consider the words military#n or treat#v. Both words are used almost the same number of times before and after 9/11. So their informativeness, based on number of occurrences, is null. But considering the persuasiveness score, we see that their impact varies (respectively from 197 to 36 and from 54 to 473).

•  Let us also consider the word war#n; if we consider only the number of occurrences, we could infer that after 9/11 this topic was much more “felt” (mentioned three times more after 9/11), but if we look at persuasiveness we see that before 9/11 the word war#n was very “popular” (position 36) while after it never got audiences’ reactions.

PREDICTION OF PERSUASIVE EFFECTS

Experiments

•  Using machine learning for predicting the persuasive impact of novel discourses. – Distinguishing Democrats from Republicans – Predicting the passages that trigger a positive

audience reaction – Cross classification (training made on adverse

party speeches, and test on the others) – Experimenting the classifiers on plain and

typical non-persuasive texts taken from British National Corpus and on speeches from the Obama-McCain political campaign.

Framework and Dataset

•  We used the Support Vector Machines (SVM) framework. •  Dataset preprocessing: to reduce sparseness, used lemma#pos

instead of tokens. •  We did not make any frequency cutoff or feature selection. •  All the speeches divided into fragments of about four sentences

(if a tag is present in the fragment the chunk ends at that point). •  Obtained chunks are then labeled as Neutral (i.e., no tag), and

Positive-ironical (i.e., all positive-focus and ironical tags). We did not consider the negative-focus tags, since they are only a few.

•  A total of ~38000 four-sentence chunks, roughly equally partitioned into the two considered labels.

•  This accounts for a baseline of 0.5 in distinguishing between Neutral and Positive-ironical chunks. In all the experiments we randomly split the corpus in 80% training and 20% test.

Democrats vs. Republican

Precision Recall F1

Democrats 0.842 0.756 0.797

Republicans 0.773 0.854 0.811

Average (μ) 0.804 0.804 0.804

Positive vs. Neutral

•  Whole Corpus

Precision Recall F1

Positive-Ironical 0.646 0.683 0.664

Neutral 0.676 0.641 0.658

Average (μ) 0.660 0.660 0.660

•  Republican only

•  Democrat Only

Positive vs. Neutral

Precision Recall F1


Neutral 0.663 0.549 0.601

Average (μ) 0.661 0.661 0.661

Precision Recall F1


Neutral 0.686 0.680 0.683

Average (μ) 0.676 0.676 0.676

Cross Classification

•  Training on Democrats, Test on Republicans

•  Training on Republicans, Test on Democrats

Precision Recall F1


Neutral 0.579 0.599 0.589

Average (μ) 0.612 0.612 0.612

Precision Recall F1


Neutral 0.658 0.626 0.641

Average (μ) 0.641 0.641 0.641

Untagged texts Classification

•  Typical non-persuasive texts from BNC (A00 to A0H) Supposing all chunks are neutral

•  Typical persuasive texts from the last Obama-McCain presidential campaign

Obama McCain

Positive-Ironical 2372 2360

Neutral 68 80

Total chunks 2440 2440

Total chunks 7243

Positive-Ironical 784

Neutral 6459

Prec/Rec/F1 0.892

Conclusions

•  We have presented a resource and some approaches for persuasive NLP: –  a Corpus of tagged Political Speeches (CORPS)

and a method for extracting persuasive words.

–  a measure of persuasive impacts of words

Future Work

•  Consider also persuasive rhetorical pattern extraction from CORPS.

•  Consider windows width (wn) based on sentences rather than tokens.

•  …

Some References

•  Marco Guerini, Danilo Giampiccolo, Rachele Sprugnoli, Giovanni Moretti and Carlo Strapparava. The New Release of CORPS: Tagged Political Speeches for Persuasive Communication Processing, to appear.

•  Marco Guerini, Carlo Strapparava and Oliviero Stock. CORPS: A corpus of tagged political speeches for persuasive communication processing. Journal of Information Technology & Politics 5 (1), 19-32, 2008.

•  Marco Guerini, Carlo Strapparava and Oliviero Stock. Audience Reactions for information extraction about persuasive language in political communication. In M. Maybury (ed.) Multimodal Information Extraction, to appear.

•  Carlo Strapparava, Marco Guerini and Oliviero Stock. Predicting Persuasiveness in Political Discourses. In Proceedings of LREC2010.