examples of cost calculation for various mixed mode ... · fixing the distribution of web and cati...

1

ESRA Conference 2013

Examples of cost calculation for various mixed mode approaches: Cost drivers and their effectiveness Stefan Klug and Birgit Schrödle

DemoSCOPE Research & Marketing AG, Switzerland

2


Mixed mode in (commercial) practise Mixing modes is a rare case where something cheaper can be better than

something more expensive. It is not a quality against price trade off.

But, cost reduction is the side-effect and not the primary purpose for doing mixed mode. The primary purpose is to achieve a higher response rate by leaving the choice of interview mode to the respondent.

Therefore, the trade off is between a (higher) mode error against a (lower ) non-response error.

We find that in practice “mixed mode surveying” was abused as a sales argument to shift interview mode from CATI to CAWI instead of selling the advantage of combined samples.

I believe that by introducing additional mode effect variance the total estimate is closer to reality than the single mode estimate.

If this does not hold true, it is still true that mode effects can at least be described and understood (instead of being ignored).

3


Mixed mode in (commercial) practise

Mixed mode designs to study mode effect: An interpretation of mixed mode? Fixing the distribution of Web and CATI e.g. to 2’000 CATI and 2’000 CAWI.

The purpose for this is to achieve equal sample sizes to facilitate comparison and to decrease costs on a fixed base (by shifting the commercial risk to the institute).

• An even more technical variant is to randomly assign and enforce the

interviewing mode instead of leaving the choice to the interviewee. Purpose is the elimination of self-selection bias from the mode bias.

Mixed mode w ithout free choice of mode is an interpretation of

mixed mode. Such designs are applicable if the understanding of the mode effect is more important than the reduction of the non-response bias (w ith free choice of mode).

4


Definition: Mode vs. Device

Mode Device

CATI

Landline Phone

Mobile Phone

CAPI

Portable PC (Laptop / Netbook)

Tablet Computer

Mobile (Smart) Phone

CAWI

Desktop PC or portable PC

Tablet Computer

Mobile (Smart) Phone

PAPI Paper

Interviewer guided control of order and pace

Self administered control of order and pace

Presence of interviewer

Visual Presentation

Presentation of stimuli

Acoustic Presentation / higher cognitive burden / help from interviewer available etc.

Drivers of mode effects (examples)

5


«Views on the MR world»

There is no «CAMI»

(Computer Aided Mobile Interviewing). CATI Landline

+ CATI Mobile = Dual Frame CATI

There is only one internet:

«CAWI Desktop» +

«CAWI Mobile» = (still) CAWI

Mixed Device (in CAWI) is a must, not an

option

Resarch in Social

Networks is not a new mode but another sample

recruitment source

6


Traditional Single Mode (zero node)

Telefone

Face to face

Internet

Letter of Invitation

No switching node. One contact mode assigned to each answer mode

CATI

CAPI

CAWI

PAPI

CATI

CAPI

CAWI

PAPI

Zero node Contact Mode

Answer Mode Start

Answer Mode End

7


Single Node Mixed Mode

CATI

CAPI

CAWI

PAPI

Telefone

Face to face

Internet


Mode switching at a single defined time in contact-interview flow

CATI

CAPI

CAWI

PAPI

Contact Mode

Answer Mode Start

Answer Mode End

Single node

Single node can be placed anywhere but it has a fixed position in interview.

8


Multi Node Mixed Mode

CATI

CAPI

CAWI

PAPI

Telefone

Face to face

Internet


Single node

Mode switching at multiple times in interview flow

CATI

CAPI

CAWI

PAPI

Contact Mode

Answer Mode Start

Answer Mode End

Multi node

9


Classification of Research Methodes

• Costs for survey

preparation

increase

• Participation rate

increases

• Completion rate

increases

Mode Nodes Device

Single Mode Zero Node Single

Multiple

Mixed Mode Single Node Single

Multiple

Multi Node

Single

Multiple

Enforced mode vs. free choice as an additional classification dimension

10


Is it (financially) worth mixing modes?

• Yes, when the extra costs for mixed mode can be compensated through mixing more expensive with cheaper interview modes

• If the number of interviews from break-even analysis is achievable,

than mixed mode has a positive cost effect

• If the extra costs for mixed mode setup are 0 than mixed mode offerings will always be cheaper than doing the more expensive mode

• For consequent mixed mode offerings a software platform is needed that reduces extra costs.

• But, mixed mode offering is not about cost saving it is about decreasing

the non-response error (at the cost of introducing mode effects).

11


Mixed mode break even calculation

2 estimates + 1 simple calculation

How much higher are the (fixed) costs if doing mixed mode compared

to single mode?

Value 1: Additional (fixed) costs for mixed mode = $20’000

What is the price difference (variable costs) per interview between the most

expensive mode and the cheapest mode?

Value 2: (variable) costs per interview expensive ($40) - (variable) costs per interview cheap ($10) = $30

How many cheap interviews do I need to

compensate for additional fixed costs through mixed mode?

Calculation: Value 1 $20’000 / Value 2 $30 = # Interviews 667

12


Example of mixed mode calculation

COSTS Best case Worst Case

h $ h $ h $ h $ $ $Setup 2'500$ 2'500$ 2'500$ 2'500$ 2'500$ 10'000$ Labor Costs

Programming 10 1'000$ 10 1'000$ 10 1'000$ 10 1'000$ 1'000$ 4'000$

Data base design/Sampling etc. 10 1'000$ 10 1'000$ 10 1'000$ 10 1'000$ 1'000$ 4'000$

Question wording 5 500$ 5 500$ 5 500$ 5 500$ 500$ 2'000$

Contact 10'000$ -$ 10'000$ 10'000$ 10'000$ 10'000$ Labor costs 200 10'000$

Adress Costs 10'000$ 10'000$

Interview 40'000$ 70'000$ 5'000$ 100$ 20'050$ 20'050$ Labor costs 667 33'333$ 1000 50'000$

Other costsTelephone expenses 6'667$ Travel expenses 20'000$ Incentives 5'000$ 100$

Analysis 2'000$ 2'000$ 2'000$ 2'000$ 2'000$ 8'000$ Labor Costs

Reporting 20 2'000$ 20 2'000$ 20 2'000$ 20 2'000$ 2'000$ 8'000$

Total Costs 54'500$ 74'500$ 19'500$ 14'600$ 34'550$ 48'050$ Fixed Costs 4'500$ 4'500$ 4'500$ 4'500$ 4'500$ 18'000$ Variable Costs 50'000$ 70'000$ 15'000$ 10'100$ 30'050$ 30'050$

Single Mode Mixed Mode

CATI CAPI PAPI CAWI

13


$0

$10

$20

$30

$40

$50

$60

$70

$80

$90

$100

$110

$120

$130

Mul

tiple

con

tact

, CA

PI >

=75

%

Mul

tiple

con

tact

, CA

PI +

CAT

I >

=75

%

Sing

le o

r m

ultip

le c

onta

ct,

CAPI

= 5

0%

Sing

le c

onta

ct,

CAPI

>=

75%

Mul

tiple

con

tact

, CA

TI >

=75

%

Mul

tiple

con

tact

, CA

TI >

=50

%

Mul

tiple

con

tact

, CA

PI a

nd P

API

50%

Sing

le c

onta

ct,

CATI

und

CAP

I =

100%

Mul

tiple

con

tact

, CA

PI

50%

and

PA

PI 2

5%

Mul

tiple

con

tact

, CA

PI

50%

and

CAW

I >

= 2

5%

Mul

tiple

con

tact

, CA

TI u

nd P

API

>=

75%

Mul

tiple

con

tact

, CA

TI 7

5% u

nd C

AWI

25%

Sing

le o

r m

ultip

le c

onta

ct,

CAPI

<=

25%

Sing

le c

onta

ct,

CAPI

=50

%,

CATI

= 0

%

Mul

tiple

con

tact

PAP

I an

d CA

WI

>=

50%

Siin

gle

cont

act,

CAP

I <

=25

%

Sing

le a

nd m

ultip

le c

onta

ct P

API

>=

75%

Sing

le a

nd m

ultip

le c

onta

ct,

PAPI

>=

75%

Sing

le a

nd m

ultip

le c

onta

ct C

AWI

>=

75%

Sing

le C

onta

ct P

API

>=

75%

Sing

le c

onta

ct,

CAW

I >

=50

%

Total Costs per Interview Fixed Costs per Interview Variable Costs per Interview Contact costs per Interview Interview costs per Interview

Costs and mode combinations

CAPI only

Equal mix

CATI only CATI 75%, CAWI 25%

CAWI only

PAPI only

14


Cost calculation scheme

$-

$20

$40

$60

$80

$100

$120

CAPI CATI PAPI CAWI

0%

25%

50%

75%

100%

Total Mode A Mode B Mode C Mode D

Total CATI CAPI PAPI CAWI

Fixed costs 18'000.0$ 4'500.0$ 4'500.0$ 4'500.0$ 4'500.0$ Fixed costs (basic) 4'500.0$ 4'500.0$ Fixed costs mixed mode 13'500.0$ 4'500.0$ 4'500.0$ 4'500.0$

Variable costs 48'775.0$ 12'500.0$ 17'500.0$ 3'750.0$ 2'525.0$ Variable costs contact (per interview) 20'000.0$ 10.0$ -$ 10.0$ 10.0$ Variable costs interview (per interview) 28'775.0$ 40.0$ 70.0$ 5.0$ 0.1$

Number of interviewsShare of contacts 200% 100% 0% 100% 0%Share of interviews 100% 25% 25% 25% 25%Number of interviews 1'000 250 250 250 250

Total costs 66'775.0$ 17'000.0$ 22'000.0$ 8'250.0$ 7'025.0$ Total costs per interviews 66.8$

15


Inbound CATI in mixed mode surveys

• Inbound CATI is a logical extension of the traditional CATI mode to decrease non-response leaving the free choice to call the institute at the best time.

• It requires:

• Invitation letter showing a (unique and toll-free) number

• Full daytime coverage in the call center

• Messaging and IVR routings

• Identification of caller in the questionnaire (e.g. printed ID on invitation letter)

16


SMS Invitation

• SMS invitation can be used to inform users of RDD mobile numbers about a study. Similar to an invitation letter sent to a landline phone related address.

• It requires:

• Short message text. (SMS costs are little but with high quantities they become expensive).

• Full multi-language text on a webpage.

• Response channels to “opt out”: e-mail, SMS, telephone

We found no better response rates in a test on 2’000 RDD interviews.

0%

10%

20%

30%

40%

50%

60%

No

conn

ectio

n

No

abili

ty

Not

in t

arge

tgr

oup

No

will

ingn

ess

No

avai

labi

lity

Appo

intm

ent

Succ

ess

SMS send No SMS send

17


Conclusion

• Cost saving is not the primary argument to do mixed mode (if you want to save costs: do CAWI)

• Mixing mode design should be extended to the “full mixed mode” approach leaving the choice of the mode to the interviewee and allowing for multiple switching nodes

• Mixed devices is a Must more than an option due to the interviewee’s usage of these devices

• CAPI and PAPI should not be (generally) excluded from mixed mode designs

• It is recommended to use multiple contact modes

• Contact modes can be extended by Inbound CATI.

• SMS invitation can be useful (but not proven yet)

examples of cost calculation for various mixed mode ... · fixing the distribution of web and cati...

Documents