Einstein on Evaluation: Spencer on Stats2nd November 2011

“Not everything that can be counted counts, and not everything

that counts can be counted”

“Not everything that can be counted counts ...”

What can be counted?

Clip/story counts

Cost per impression

Audience impressions

Weighted Media Costs

(WMC)

Advertising Value

Equivalent

(AVE)

Tone / sentiment

MessagePhoto

QuotePlacement

What can be counted?

Who thinks what?

How many think x?

How many do x?How many

have seen x?

Opinions?

6

So, what counts?

What counts is success

What is success?

SuccessPronunciation: / sək’sɛs /noun

[mass noun]

1 the accomplishment of an aim or purpose: the mission had some success in restoring confidence in the country

2  archaic the good or bad outcome of an undertaking: the good or ill success of their peacekeeping mission

Origin:

mid 16th century: from Latin successus, from the verb succedere 'come close after' (see SUCCEED)

By using smart objectives

S pecific

M easurable

A chievable

R ealistic

T ime

And how can we count success?

The ladder of success

Did the message resonate and stick?

Did the target audience have the opportunity to see it?

Did the message appear?

Did an attitude or behaviour change?

Did the audience actually see it?

Was the message placed in the right medium?

Was the messages changed or distorted?

Was the message compelling to the medium or intermediary?

The laddering approach to measurement

Measures of effectiveness

(MoE)

Measures of performance

(MoP)

Measures, metrics & methodsMetrics MethodsMeasures

Attitudes & behaviours

Message pick-up/take out

Out

com

e

Audience penetration (direct)

Reach of audience (indirect)

Message placement

Media uptake

Pre/post surveys

Pre-testing/ evaluation

Events/ ‘contacts’

Media analysis (readership/listenership/

viewership)

Media evaluation (traditional & new social

media)Media

coverage/ interviews

Qual. & Quant. research

Out

-ta

keO

utpu

t

Methods – Primary Research

Quantitative approaches

• Face to Face research

• Telephone research

• Online research

• Postal research (rare)

Data collection methods:

Face to Face research

• Perfect for:-• Subject sensitive studies• Where other methods (phone, emails)

impractical• Pen & paper or CAPI* where available• Street (max 5 mins.) or in home (20 to

40 mins.)

Benefits: Very thorough, can interview people in situ and comfort of their own home, rich informationDrawbacks: Expensive fieldwork, time consuming, some areas are “no go” areas

*Note -Computer Aided Personal Interviewing

Telephone research

• Ideal for geographically dispersed populations

• Pre-provided telephone numbers or “direct dial”

• Trained interviewers calling from a telephone facility

• 10-30 mins is the norm

Benefits: Potential to reach broad populationDrawbacks: Expensive NB mobile phone interviewing.

Online research• Fastest growing data collection tool• But only suitable where internet penetration

high• DIY option

• Using survey software• Outsource data processing to DP House

• Write own questionnaire, script and web-host• Online Panel option

Benefits: inexpensive, quick & easy & good for showing stimulus materialDrawbacks: not suitable for hard to reach or low literacy populations

Designing sample

Samples

• Consider the population and the different subgroups you will need to analyse• dictates overall sample size and design.

• Random sampling techniques produce representative results

• Census data is a challenge

Why is it important to consider base sizes?• The larger the base size the more

confidence we have in the results being a true reflection of the study results

• We cannot act on the results if the sample size is not robust and the results therefore reliable

Sample SizeAnswers accurate + or - %*

(using a statistical measure of 95% confidence limit)

400 4.9%500 4.6%600 4.0%800 3.5%1000 3.1%1600 2.3%

* For example if your base size is 1600, and the results show that 50% of the sample have voted in the

recent election, you can be confident that 47.7%-52.3% of the total population have voted.

Principles of good research practice (applies to all methods)

• Pilot where you can – test out the survey if at all possible with the appropriate population

• Check the flow, the language, the layout, do not ask for feats of memory from the respondent

• Make the questionnaire as inviting to complete and as interesting as possible

• But do not make it so complex it confuses them• Include some open ended questions if possible so people can

express their own views in their own way• Give time for analysis at the end – it is the output that is important

not the research process in itself

Recommendations

• Face-to-Face methodologies should be considered first • Do not rely upon government statistics; NGOs often have good

reports on social movements and situations • Contact NGOs who have completed research in the area. Find out

about the work they have done and with who • Be aware of social mores when assigning interviewers• Timelines must be flexible, and projects on a best efforts basis• Monitor political and social situation closely.

Qualitative approachesData collection methods:

• Focus Groups o Mini groups of 4 to 5 (or full

groups of 6 to 9 attending)

• In-depth interviewso Face to face or telephone

Qualitative Approaches – some rules of thumbIn depth interviews work when:• People cannot travel easily and it is

easier for you to go to them (disability, ill health, lack of mobility, suspicious, too busy..)

• People you want to speak to are geographically dispersed

• The subject is highly sensitive and people would not want to discuss openly with others there

• There is time for the executives to do individual phoning or to travel to interviews

Advantages: Much richness in the encounters; puts most onus on the researcher; reaches “difficult to reach” groups

Disadvantages: Time consuming, costly, more difficult to cover a wider sample

Focus Groups work when:• People can travel to the groups and are

prepared to do so• The subject is one that people are

happy to discuss in the open and share their thoughts with others

• There is enough sample to cluster people so that you can run a group (ie not too geographically dispersed)

Advantages: Richness in the exchange and thrust and parry of the interchange; takes people out of their usual environment; more effective in terms of researcher time, more cost effective to reach a wider sample (than depths)

Disadvantages: Logistics (viewing facilities etc) can be costly; people do not always turn up; analysis more intense & what people says more difficult to disaggregate

Methods – Secondary Research

Media evaluation

What counts? The principles of effective communication count...

Reach of audience: did you reach them sufficient times for message to stick?

Delivery of message: did the message even appear, and how did it appear (credibly, negatively?)?

AVE/WMC/OTS/GIs – what purpose do they serve?

Considerations:-• It provides a deep dive into online discourse that is taking place but is not intended

to provide a representative study of people living in a certain community• Supplements, rather than replaces, other sources of information that are available

to the organisation, whether gathered via primary research studies or informally from stakeholders and ‘allies’

• It is a channel for communications and like any comms. needs to be measured!

Monitoring the online environmentUses & Benefits:-• Proxy for public opinions• Will tell you how people feel about issues and

how online discussion is enhancing or undermining a position

• Identify how current events shape content and discourse about organisations or social issues in online spaces.

What counts is the messageAround half of the content analysed had a key strategic message embedded in it – primarily support for pro-democracy initiatives

158

98

151

205

0

50

100

150

200

250

Anti-westernrhetoric

Anti-extremist

Anti-violence/ pro-peace

Support for Initiatives

Main Messages Featured in Content820

356

Volu

me

of C

onte

nt

Total Volume of Content to Feature a

Pro-democracy Message

Base: 820 items of Online Content (Jan 2008 to Oct 2009)

• Content analysed sourced from:• Websites and online medias• Blogs and microblogging sites• Forums/ Discussion groups• Social Network sites• File sharing sites (e.g. YouTube and Flickr)• Wikis/ Answer sites/ Podcasts

• Software tools used to uncover content posted:• Alterian SM2• Free search tools e.g. Google

Methodology

Deciding what counts

Rear view Forward looking

How did the campaign perform in terms of:-- Media- Return

How effective was the campaign in terms of:-- Messaging- Attitude/behavioural shift

Stop doing

Start doing

Keep doing

Decisions that count

Shoot the messenger?

Shoot the message?

“Not everything that counts is counted...”

Accounting for flat-lining of measures

And accounting for casuality issues

How they already act & feel about voting

15%

Almost always vote

42%

38%

36%

35%

60%

34%

Out of Habit

3%

29%

29%

14%

4%

25%

29%

Vote at some elections

24%

69%

63%

41%

36%

88%

65%

Never voted

25%

8%

9%

8%

8%

5%

8%

Committed to vote

33%

87%

86%

65%

61%

100%

84%

key relationship & response measures

Right as a citizen to vote

rela

tio

nsh

ipre

spo

nse

Important aspect of democr.

Sets an example for younger people

Shows pride in nation

Would vote

Would urge others to vote

Beh

avio

ur

Att

itu

de

Combined effect of communication - vote at some elections

70%

65%

60%

55%

50%

Pre Total

% stating that it is very important or important to vote to set an example for younger people

56%

Post Total

56%

Print only

OOH* /PR only

50% 49% 51%

TV only

Saw nothing

51%

TV, Print, online

TV, OOH/PR

TV, Prt,online, OOH/PR

59%61%

65%

Online only

51%

*OOH= Out of House Media Q14: Please indicate how you feel about voting in terms of the following aspects..

Setting an example for younger people (very important, important, not very important, not at all important)

Quantifying campaign effect

Out of Habit

0

Never voted0

Almost always vote 1,800

11,900Vote at some elections

Committed to vote 32,300

+ 46,000more citizensaged 18+ who now score high on this statement because of the recent communication

% stating that it is very important or important to vote to set an example for younger people

Calculating cost benefit

Right as a citizen to vote

To set example for younger people

Way to express views

Way to shape the community

Peaceful way to change power

To make democracy stronger

Shows pride in nation and government

Helps to reinforce regional government

Not voting might end democracy

Important aspect of democracy

Voting urges to get involved

Tells candidates to be accountable

Democr. works only if citizens are active

Sets a precedence for younger people 21,600

27,300

29,200

31,800

33,700

33,800

34,600

35,400

40,000

40,100

45,000

45,500

46,000

measure incremental populationReality£478,100

£10

£11

£11

£11

£12

£12

£14

£14

£14

£14

£15

£17

£18

£22

It cost you £__ in campaign spend to get 1 more person to feel positively on this attribute

48,000

When you can’t count, important to triangulate the data

Pri

mar

y su

rvey

data

Ethnographics

Secondary (media) data

Thank you and questions Claire Spencer FCIPR, i to i research limited

DD: +44 (0) 203 178 2162

Mob: +44 (0) 7786 543 506

email: claire.spencer@itoiresearch.com@claireyscherubs