Persuasive Message Evaluation with Google AdWords

1

Marco Guerini Carlo Strapparava Oliviero Stock

Introduction

2

3

Problems •  Evaluating systems and theories about persuasion is a

bottleneck: expensive and time consuming experiments.

•  In lab settings results are biased by subjects awareness

of being tested.

designing the experiment, inventing cover stories, recruiting a sufficient number of subjects, making them take part to the experiment, dispense questionnaire (pre-post), gather data, analyze them.

Crowdsourcing - Mechanical Turk? Mechanical Turk used for behavioral studies (Mason and Suri, 2010) to overcome part of these limitations - like subject recruitment.

4

We believe that it is not the right instrument to assess behavioral changes, and, more generally, persuasive effects…

Requirements

1.  Studies as ecological as possible (real, even if controlled, scenarios).

2.  Subjects must not be aware of being under observation.

3.  Subjects must not be biased by external rewards.

5

6

Proposed Approach

Guerini M., Strapparava C. & Stock O. “Evaluation Metrics for Persuasive NLP with Google AdWords”. In Proceedings of LREC2010, May 2010.

7

8

Approach •  Goal: develop “cheap and fast” methodologies to

measure the persuasiveness of communication. •  Solution: tweaking and using existing commercial tools

for advertising on the web, such as Google AdWords.

Conducting field experiments with thousands of subjects for a few dollars in a few hours

!

9

Adwords Description •  Google AdWords is Google advertising

program. It lets advertisers display their ads only to relevant audiences by means of keyword-based contextualization on the Google network, divided into:

•  Search network: Google search pages, search sites etc.

•  Content network: news pages, topic specific websites, blogs and other

10

Put your test messages on the Web.

11

AdWords Metrics Identifying the performance (persuasiveness) of each ad from various perspectives:

•  CTR: the number of clicks divided by the number of times an ad has been displayed. Indicates the initial impact.

•  Web Analytics: insights on landing site: number of visited pages, time spent on the site, etc. Indicates interest/attitude generated.

•  Conversion Rate: how many user turned into conversions (e.g. buying a product, download a brochure). Indicates message complete success.

•  ROI: The more relevant (from a research pov) the action the user performs, the higher the value.

AdWords Targeting Targeting options for different evaluation scenarios:

•  Language message impact in different languages.

•  Location message impact in different cultures sharing the same language.

•  Keyword matching message impact with users having different interests.

•  Placements message impact on people with different values –democrats/republican sites.

12

AdWords Tuning To facilitate statistically meaningful exp. (subj randomness, etc.) some tuning needed:

•  Ad serving (optimize vs rotate) •  Ad delivery (standard vs. accelerated) •  …

No longer necessary.

Since our first experiments some proposal have been implemented in AdWords, under the tool AdWords Campaign Experiments (ACE)

13

Experiments

14

15

Experimental Setup •  Pilot experiments to test irony, using a real

promotion campaign. •  Experiments based on a between-subject

design with two conditions, one ad for each condition:

•  Control condition, with a neutral message. •  Experimental condition with ironical message. •  χ2 test used on CTR.

16

Experimental Design

Two experiments with img based ads on the content network (promoting the book series).

Two experiments with text based ads on the search network (promoting a book).

Stimuli Material •  “Diversamente Occupati” (tr. “differently

employed”) mocks the Italian “diversamente abile” (tr. “differently able”), a politically correct word for handicapped persons.

•  “Diversamente Occupati” ironically refers to temporary employed (i.e. “differently employed”) people, generally considered a less than ideal state of employment.

17

18

Experiments with text based ads

Context: Keyword “Precario”

Dura%on   Cost   Subjects  

24  h   1,2  €  -­‐  CPC  0,04  €      ~  3.000  

Dura%on   Cost   Subjects  

24  h   1,5  €  -­‐  CPC  0,04  €      ~  4.000  

19

Results 1st Experiment - CTR (1A) = 1,7% CTR (1B) = 0,5% •  Test significant condition 1A “Sei Precario?” (tr. “Are you

temporarily employed?”) performs three times better than 1B “Sei Diversamenteoccupato?”.

•  Possible bias: given keyword matching the word “Precario” in condition 1A is automatically highlighted by Google.

2nd Experiment - CTR (2A) = 1,2% CTR (2B) = 0,8% •  Test not significant first experiment biased by highlighting. •  2B “Sei DiversamenteOccupato?” outperformed other conditions (1A,

1B, 2A) w.r.t. permanence of users on landing site (~3 min vs ~1 min). •  Capitalizing letter “O” in “DiversamenteOccupati” helped users spelling

the word and get the irony - that induced more interest in the subjects.

20

Dura%on   Cost   Subjects  

12  h   7  €  -­‐  CPC  0,60  €      ~  4.200  

Dura%on   Cost   Subjects  

1  week   2  €  -­‐  CPC  0,20  €      ~  12.000  

Experiments with image based ads

Context 1: a site of job offers Context 2: the whole network

21

Results 3rd Experiment - CTR (C1) = 0,5% CTR (C2) = 0,1% •  Test significant, condition C1 performs five times better than C2. •  Should we conclude that the img has a negative impact on the

message?

4th Experiment - CTR (C1) = 0,16% CTR (C2) = 0,08% •  Test not significant In the previous experiment the img did not

have a negative impact by itself, rather it had a “positive” impact in cutting uninterested users, by helping them in “disambiguating” the message (users were looking for a job, not for amusement).

Conclusion •  AdWords is an appropriate context for

evaluating persuasive messages.

•  AdWords proved to be very accurate, flexible and fast, far exceeding our expectations.

•  The ACE tool bring us a step further.

22

Further details in: Guerini M., Strapparava C. & Stock O. “Evaluation Metrics for Persuasive NLP with Google AdWords”. In Proceedings of LREC2010.

23

24