SPECTRUM

V o l u m e 1 I I s s u e 2 R O I a n d M a r k e t R e s e a r c h

2 0 1 3

P u t t i n g R e s e a r c h i n C o n t e x t

ROI and Market Research

1

Stephen Phillips - Chief

Happiness Officer and

Founder, Spring Research

Stephen Phillips - Chief Happiness Officer and Founder, Spring Research

Steve@springresearch.com

020 7428 7379

In our second issue of SPECTRUM - our quarterly research project on the client

context - Spring founder Steve Phillips explores ROI and Market Research

Welcome back to the second issue of SPECTRUM magazine! For those of you who weren’t with us for

issue 1, Spectrum is a series of quarterly research projects that we’ve begun in order to understand

the client context. Our aim is to provide you with some insight on both what your peers in other

categories are experiencing, and what’s working well for them.

Issue 2 concerns ROI; a key aspiration for most of us, it seems, but not one which readily lends itself to

the service research provides to the business. The challenges faced by research departments across

the range of clients we spoke to seemed fairly consistent, but actual solutions were far scarcer.

However, as a result of this study, we’ve put together what we think constitutes a good base for

measuring the success of research. Read on to see what this entails...

As always, your comments and opinions stimulate some of the best insights, so we’d love to hear from

you. In fact, if you’d like to be interviewed for our next project, please get in touch. We’d love you to

help us continue to make SPECTRUM a success!

W h e r e t o f i n d t h i n g s :

□ I n t r o d u c t i o n a n d C o n t a c t D e t a i l s p 1

□ T h e H o l y G r a i l o f I n s i g h t D e p a r t m e n t s p 3

□ R O I i n t h e B u s i n e s s p 3

□ T h e C r i t i c a l Q u e s t i o n p 4

□ R e s e a r c h a s R O I I n p u t p 5

□ P i c k i n g a N u m b e r p 5

□ E n t e r t h e S u c c e s s M e t r i c p 6

□ S o , T h a t ’ l l H a v e t o D o ? p 7

□ F o c u s i n g o u r G a z e p 9

□ R o u t e s f o r B e s t P r a c t i s e p 1 0

□ A M e a n s a n d a M o t i v a t i o n p 1 2

2

The initial enthusiasm from clients to take

part in this issue of Spectrum was

overwhelming.

“Great”, we thought, “We’ll have

umpteen research ROI tools lined up

before we know it”.

After the first few interviews, however,

the real reason for our generous

respondents’ eagerness became obvious.

Most people just wanted to know how

everyone else did it. How were other

research teams managing to draw the

myriad outcomes of a research

department into a clear measurement

that offered some support in the regular

need to justify budgets

and build reputation in the

corporate environment?

It was clear that within the wider

business, ROI was being discussed and

indeed focused on. But precise

calculation seemed to be fairly rare

phenomena, even for marketing

in general.

Building a new factory naturally lends

itself to a workable definition of ROI – ‘it

costs x, will increase capacity by y, and

we estimate a sales uplift of z’. Like any

modelling, it’s full of assumptions, but

there’s enough clarity in the spend to

keep the Finance department happy and

confident. The question of ‘how far we’re

prepared to put our finger in the air for a

number’ lies at the heart of ROI

calculation, and the evidence suggests

that other departments are far more

confident about it than Research or

even Marketing.

Incidentally, our favourite quote from the

interviews? “Who measures the ROI of

the Finance department?!”.

Which leads us to...

H o l y G r a i l o f I n s i g h t D e p a r t m e n t s

3

T h e

R O I i n t h e B u s i n e s s

T h e C r i t i c a l Q u e s t i o n

4

How much should the research budget

be? Does the return on investment for

our labours justify a large increase in the

research budget? Many of us researchers

would believe so.

At a recent ISBA conference, we

presented the genesis of this article to a

room of market research procurement

experts. The questions raised by the

room in the subsequent discussion

probably show where the industry

currently stands in terms of how big the

market research budget should be:

Delegate 1: I’ve always used 5% of the

marketing budget as a rule of thumb

Delegate 2: Really?! I thought it was

supposed to be 1%?

Delegate 3: As low as 5%?

And so on, and so on...

Even the fiscal hawks of the procurement

department don’t have a reliable guide to

how big the research budget should be.

And the magnitude of the uncertainty

was clearly not about fine-tuning

budgets, not the difference between

£100 and £105, but about whether it

should be £100 or £200. The result was a

consensus that the size of the research

budget was, in most cases, a fairly

arbitrary decision.

Actually, arbitrary is the wrong word, as

what both procurement and our

respondents concluded was that the size

of the research budget was a political

decision. The most cited cause of a

significant change in research spend was

the arrival of a new CEO and the different

priorities – not to mention experience of

(or even, whisper it, respect for) research

- that this so often brought to a business.

This introduces our first requirement of

our research ROI measure:

O b j e c t i v e # 1

We need something based on provable success to use in discussions

with Finance

R e s e a r c h a s R O I I n p u t

5

It’s not as though research and ROI are

never mentioned in the same breath.

Indeed, research often provides key

variables for calculating the return on

marketing activity – for example brand

awareness, or favourability. But before

we think about research’s contribution,

we have to consider several reasons why

measuring the ROI of marketing like this

can run into significant difficulties.

To begin with, there’s a huge discrepancy

in the basic size of elements of the

marketing budget; it’s dominated by TV.

This means that during econometric

modelling most other fluctuations in

spend don’t register enough of an affect

to assess their impact. So marketers,

even those lucky enough to have great

sales data, typically can’t assess the ROI

of much of their marketing spend with

any accuracy.

Often, instead of these complex

econometrics which look backwards (and

from the sounds of it, mostly at TV),

marketing departments rely on more

simple research measures to determine

effectiveness.

P i c k i n g a N u m b e r

Isolating the value of advertising testing can be complicated.

When we think about a creative test, for example, consider the value of these 3 research outcomes:

1. The advert is great, don’t change a thing

2. The advert is good, make some changes

3. The advert is awful, don’t run it

In outcome 1, the campaign is a success but research has had no impact, so you could say there is no ROI. In option 2 you could argue that research has improved the effectiveness of the campaign and so claim perhaps 15% of the campaign’s positive brand impact – a reasonable potential ROI measure. But in option 3, you could say the research has saved the company the entire media spend, which is a lot of money and so a great ROI. One test, the same research, but 3 very different ROI’s.

Of course, there are a number of caveats – not least the degree to which developers and creatives actually embrace the research

This highlights one of Spring’s mottos: value is always co-created, it’s never just supplied.

t h e S U C C E S S M E T R I C Enter...

6

Input is easy; it’s about procurement.

‘Have we saved money compared to the

last time we conducted a study like this?’,

‘Have I driven the cost down through a

tender process?’, or ‘Have I come in

under budget?’.

Output is harder, but people were still

attempting to judge projects a success or

failure based on ‘softer’ questions like ‘Is

the marketing person happy with the

results?’, ‘Did I get more insight than I

expected?’, or ‘Did it feel like a high-

quality project?’.

That’s not to say that measures don’t

exist for the success of research projects.

Two types of success metrics emerged,

one with a quantitative feel, the other

somewhat qualitative. Simply put, you

either focus on inputs or outputs.

• Cost vs. competitors

• Cost vs. last time

• Cost vs. budget

• Perceived quality

• Stakeholder satisfaction

• Insight vs. expectations

hard soft

focus INPUTS OUTPUTS

“We go out to pitch” “Is the marketing

person happy?”

feel

measures

mindset

S o , t h a t ’ l l

h a v e t o d o ?

7

Perhaps it’s not perfect, then, but the impact

of marketing can be measured in some way .

What’s more, the contribution of research to

that success can clearly be felt. With

experience, marketers can even begin to

estimate what research is contributing to the

success – maybe a 15% premium on the

advertising spend, or 5% of the value of a

product’s eventual success. That, in turn, is a

starting point for discussing the

research budget.

These figures, however, remain anecdotal -

speculative even - and limited to research

which tests something with an identifiable

short-term impact. They favour the testing of

executions or prototypes.

That introduces the wider problem for

measuring the contribution of research

- it’s not just about testing.

What about more strategic, development

research? Admirably, Procter & Gamble CEO

Robert McDonald regularly visits consumers’

homes in emerging markets to get closer to

his audience, but how would you even begin

to attach a value and analyse the benefit of

this type of research?

And that’s for one of the purest forms of

research – ethnographically connecting

stakeholders to their consumers. Ever since

Theodore Levitt coined the phrase ‘marketing

myopia’, the industry has understood that

there’s huge value in getting mindsets out of

the office and into the real world, but

calculating ROI on it throws up a million and

one problems – and usually no answers.

So where does that leave us? Well, at this

point, it seems like ROI of testing research is

flawed, but possible, and ROI of development

research is nigh on impossible.

8

What this model does clarify, though, is a

second requirement of our research

ROI measure:

• ‘Buy’ focus

• Try to reduce budget

• High-level sponsor

• Less input scrutiny –

more output focus

testing development

focus INPUTS OUTPUTS

type

implications

O b j e c t i v e # 2

It has to work for Development research as well as Testing research

Fortunately, the input/output model

does identify a space for

development research.

Testing research has a very natural fit

with inputs, and the buying decision

usually comes down to choosing how

much to spend. It also provides a fairly

clear opportunity for measuring the

success of a project – once the ‘hygiene

factors’ of data delivery and clarity are

met, the focus is on cost reduction.

Development research, however, is much

harder. These projects tend to be high-

profile, often with a senior internal

stakeholder, which inevitably moves the

focus from costs to impact.

Success metrics are clearly not limited

to a focus on projects; departments and

people can also be subject to

evaluation.

In fact, the internal reputation was

commonly cited as the best indication of

the success of research. This was subject

to both hard and soft measurement,

from recording ‘hits’ on research

documents to ‘getting in the right

rooms’ as an internal consultant.

The success of people, of course, is one

of the most routine processes of

measurement in any organisation.

Personal reviews for some researchers

include a score for ‘impact on decisions’

– although, again, this is subjectively

calculated. To complicate matters, it can

actually be a better reflection of the

strength of someone’s ability to get their

point across than (necessarily) the

quality of the insight they’ve discovered.

So far, the ROI of research seems distant

from the technical performance of the

department, or the research projects

themselves.

Altogether, this gives us our third and

final requirement of a research ROI

measure:

F o c u s i n g o u r G a z e :

D e p a r t m e n t s , P e o p l e , o r P r o j e c t s ?

9

O b j e c t i v e # 3

It must work as evidence of the success at a project, person, and

departmental level

Before we consider some solutions for

some kind of ROI metric for research

outcomes, let’s recap what we need

from it:

A veneer of economic rigour should

improve a research departments’ ability

to make a financially viable argument in

budget discussions. That means that

whatever solution we adopt, they need at

least a quantitative element or

component.

We don’t expect to see this next idea

happen in the near future, but at a macro

level we’d love ESOMAR or the MRS to

commit to a review of key clients across

the industry. With a methodical

assessment of sales growth by research

spend (with all of the complexity of time-

lag, defining growth etc. which that

entails), the industry would at last have a

normative base to begin conversations

about the optimum size of the research

department. What’s more, it would give

an evidence-based rule of thumb for

research spend by marketing budget –

whether that’s 1%, 5%, or perhaps even

higher...

On a project-by-project basis, we propose

a mixed-methodology.

Firstly, there should always be a

discussion on the value key stakeholders

got from the project. This lets research

teams learn from not just the research

data, but from the entire process of

internal data delivery. Just as importantly,

since this in itself raises the profile of the

research team, it shows a department

keen to enhance and develop the value it

delivers. This builds reputation within the

business.

R o u t e s f o r B e s t P r a c t i c e

10

M e a s u r i n g t h e S u c c e s s o f D e v e l o p m e n t R e s e a r c h

1. Something concrete to take to the Finance department

2. Something that works for development research as well as testing research

3. Something that reflects on the person, the project, and the department

This can be done with some standardised

metrics such as:

• How much the research added to their understanding of the consumer (from high to low)

• How much the research gave them confidence to take the make the next decision

• How valuable the research was compared to the cost

Our second suggestion is a little bolder which

is to capture a quantitative valuation from

key marketing stakeholders. This needs to

record what these stakeholders believe the

insight impact was on the project. This can be

assessed in relation to the overall project

itself and measured as an improvement in

the project:

ROI can then be assessed by calculating the

role of research against the size of the

project:

Of course you may not know the value of the

project up front (it may not be a success of

course) but as a proxy you can use the cost of

the project which probably is known

This obviously gives us a financial metric to

show during budget discussions but also

allows us to compare across research

projects conducted and hence better allocate

budgets.

These metrics let us measure the outcomes

of the research, however, one party has so

far escaped scrutiny: the research supplier.

Since responsibility for the technical

expertise of the research lies with the

agency, we believe both clients and suppliers

would (and typically do) benefit from looking

at reviewing the two most important aspects:

timeliness of delivery, and quality of data

delivered.

11

I m p a c t E v a l u a t i o n

Q . T o w h a t e x t e n t d i d t h e r e s e a r c h i m p a c t o n t h e p r o j e c t

a ) M a d e n o i m p a c t

b ) I m p r o v e d t h e p r o j e c t o u t c o m e b y a p p r o x . 5 %

c ) I m p r o v e d t h e p r o j e c t o u t c o m e b y a p p r o x . 1 0 %

d ) I m p r o v e d t h e p r o j e c t o u t c o m e b y a p p r o x . 2 0 %

value of project

ROI =

% impact on project x

cost of research

We believe that this does provide a

methodical way to prove the value of

research to the business – and that means

all research, not just research with an

obvious short-term impact.

At a practical level, it provides a measure

of impact on stakeholders outside of the

research department, which lends it

credibility for interrogation by Finance,

and it also adds a measurement of output

(impact, insight, or assisting decision-

making) to research, relieving it of the

purely procurement-focused ‘input’

measurements like cost or savings.

Finally, through the focus on the impact of

our research and the process of collection

of the data from stakeholders, it’s

evidence of importance research places

on being involved in and impacting on

decision-making.

This, of course, means these success

metrics perform a dual function - they

contribute to both a researcher’s career,

and the standing of research within the

business and the wider corporate world.

Does it justify the investment? We think

it’s a good start.

Va l u i n g R e s e a r c h :

A M e a n s a n d a M o t i v a t i o n

12

www.springresearch.com

S P R I N G R E S E A R C H B E D F O R D H O U S E 125-133 CAMDEN HIGH STREET

LO N D O N N W 1 7 J R U N I T E D K I N G D O M