bad boy matrix question - whatcha gonna do when they come for you?

BAD BOY MATRIX QUESTION Whatcha gonna do when they come for you?

Florian Tress

ODC Services GmbH, Germany

GENERAL ONLINE RESEARCH 2012

5 – 7 March 2012 at the DHBW Mannheim

2 THE VENN DIAGRAM OF FIELDWORK

PEOPLE RESEARCHERS

Fun & entertainment

Incentives

Disport

Curiosity

Express an opinion

Solve problems

Collect (a lot of) valid data

Gain a lot of knowledge

Implement special

(and possibly boring)

methods

3 BAD BOY MATRIX QUESTION

PEOPLE

Monotonous, boring

Overwhelming variety of options: “decision paralysis”

Nondifferentiation, satisficing behavior

Another question format would be more appropriate: Inferior data

RESEARCHERS

Standardized data format, comparability of responses (with common orientation)

Maximize amount of data, minimize length of interview

Facilitate statistical procedures, e.g. factorial analysis, indices, etc.

Availability of validated test instruments , e.g. Big Five

4 STANDARD ALTERNATIVES

MULTIPLE CHOICE

Matrix with a two point scale

Additional option “None of these” (mandatory question)

RANKING

Bring statements in an order

Perfectly differentiated data

5 SPECIAL ALTERNATIVE: THE CAROUSEL

Only one statement presented at once

New statements slide in from the left

Response options remain in the same place

6 SPECIAL ALTERNATIVE: DRAG ME

All statements presented at once

Center stands for respondent

Measures distance from center

Arrangement of statements unimportant

7 COMPARISON OF THESE ALTERNATIVES

Monadic Questionnaire : Random assignment to one of these alternatives

n = 1080; 216 interviews per alternative; good spread over age and education

Indicators: Comparability, Trustworthiness, Data Quality, Satisfaction, Technical Requirements

8 COMPARABILITY: BRAND LIKEABILITY

39

39

43

47

52

53

54

62

66

73

McDonalds

Siemens

BMW

Coca Cola

Nutella

Volkswagen

IKEA

Google

Nivea

Amazon

Multiple Choice

3,2

3,7

3,8

3,9

4,2

4,2

4,6

5,0

5,8

6,6

McDonalds

Siemens

IKEA

Nutella

Coca Cola

BMW

Volkswagen

Google

Nivea

Amazon

Ranking

3,1

3,5

3,5

3,5

3,5

3,6

3,7

3,9

3,9

4,0

McDonalds

IKEA

BMW

Siemens

Coca Cola

Nutella

Volkswagen

Google

Nivea

Amazon

Matrix

44

48

49

51

52

52

57

61

66

68

McDonalds

BMW

Nutella

Volkswagen

Siemens

Ikea

Coca Cola

Nivea

Google

Amazon

Drag Me

3,2

3,4

3,4

3,5

3,5

3,5

3,7

3,9

4,1

4,1

McDonalds

Nutella

Coca Cola

BMW

Siemens

Ikea

Volkswagen

Google

Nivea

Amazon

Carousel

3 F

acto

rs (

Cum

.%:

49)

| C

ronbach„s

α:

n.a

. 3 F

acto

rs (

Cum

.%:

56)

| C

ronbach„s

α:

0,7

7.

3 F

acto

rs (

Cum

.%:

58)

| C

ronbach„s

α:

0,7

8.

3 F

acto

rs (

Cum

.%:

57)

| C

ronbach„s

α:

0,7

3.

3 Factors (Cum.%: 60) | Cronbach„s α: 0,80.

9 COMPARABILITY: ATTITUDES

11

14

18

24

28

28

30

38

44

62

I

H

J

G

B

D

F

E

C

A

Multiple Choice

3,0

3,4

3,6

3,8

4,0

4,6

4,9

4,9

6,4

6,4

G

H

I

J

B

F

E

D

A

C

Ranking

2,8

2,8

3,0

3,1

3,2

3,2

3,5

3,5

3,6

3,8

J

I

H

G

F

E

D

C

B

A

Matrix

32

39

41

44

49

50

52

55

63

65

H

I

J

B

D

G

F

E

C

A

Drag Me

2,9

3,0

3,1

3,1

3,4

3,4

3,5

3,5

3,7

3,9

I

J

G

H

F

E

D

B

C

A

Carousel

(A) Wenn ich gute Erfahrungen mit einer Marke mache, empfehle ich sie aktiv weiter. (B) Werbung sollte mich stärker über Marken informieren, die ich noch nicht kenne. (C) Ich

habe feste Marken, die ich bevorzugt einkaufe. (D) Neuartige und innovative Produkte passen gut zu meinem Lebensstil. (E) Ich probiere häufig neue Marken aus, die ich noch

nicht kenne. (F) Ich bevorzuge Marken, die auf eine lange Tradition zurückblicken. (G) Ich bin bereit, für Markenprodukte mehr Geld auszugeben. (H) Werbung sollte mich stärker

über Marken informieren, die ich bereits gut kenne. (I) Je bekannter eine Marke ist, desto leichter kann man ihr vertrauen. (J) Die Bekanntheit einer Marke sagt etwas ihre Qualität aus.

3 F

acto

rs (

Cum

.%:

51)

| C

ronbach„s

α:

n.a

. 3 F

acto

rs (

Cum

.%:

50)

| C

ronbach„s

α:

0,5

6.

3 F

acto

rs (

Cum

.%:

61)

| C

ronbach„s

α:

0,8

0.

3 F

acto

rs (

Cum

.%:

63)

| C

ronbach„s

α:

0,8

2.

3 Factors (Cum.%: 63) | Cronbach„s α: 0,82.

10 RESULT: COMPARABILITY

WINNER

Results correspond (roughly) for the most / least likeable brands (agreeable statements)

But: Drag Me seems to measure something different / to be biased by third variables

LOSER

11 TRUSTWORTHINESS: BRAND LIKEABILITY

MULTIPLE CHOICE

Ø Words pos.: 5,5

Ø Words neg.: 5,4

Non-Response: 3%

MATRIX

Ø Words pos.: 6,8

Ø Words neg.: 7,3

Non-Response: 2%

RANKING

Ø Words pos.: 6,3

Ø Words neg.: 6,7

Non-Response: 3%

CAROUSEL

Ø Words pos.: 7,7

Ø Words neg.: 7,0

Non-Response: 3%

DRAG ME

Ø Words pos.: 7,2

Ø Words neg.: 7,2

Non-Response: 2%

Follow-Up-Exploration: Why do you think, this is the most / least likeable brand?

12 TRUSTWORTHINESS: ATTITUDES

MULTIPLE CHOICE

Ø Words pos.: 7,4

Ø Words neg.: 8,6

Non-Response: 11%

MATRIX

Ø Words pos.: 7,9

Ø Words neg.: 9,6

Non-Response: 9%

RANKING

Ø Words pos.: 8,3

Ø Words neg.: 8,1

Non-Response: 7%

CAROUSEL

Ø Words pos.: 9,3

Ø Words neg.: 10,0

Non-Response: 8%

DRAG ME

Ø Words pos.: 8,4

Ø Words neg.: 8,6

Non-Response: 7%

Follow-Up-Exploration: Why did you agree / disagree with this statement?

13

WINNER

RESULT: TRUSTWORTHINESS

LOSER

nonspecific / general

„because I like it“

specific / rich in detail

„good quality, fair prices“

14 DATA QUALITY: NONDIFFERENTIATION

13%

0%

6%

4%

0%

8%

0%

8%

5%

0%

Multiple Choice Ranking Matrix Carousel Drag Me

Brand Likeability Attitudes

Low sample size!

15 DATA QUALITY: NONDIFFERENTIATION

2% 5%

3%

3%

Brand Likeability Attitudes

2% 2%

2% 2%

Brand Likeability Attitudes

6% 3%

7%

4%

Brand Likeability Attitudes

6%

2%

7%

5%

Brand Likeability Attitudes

3% 5%

2%

3%

Brand Likeability Attitudes

2% 3%

2% 1%

Brand Likeability Attitudes

MULTIPLE CHOICE MATRIX CAROUSEL

Educati

on

Age

younger

40

old

er

low

er

A-L

. h

igher

Low sample size!

16

WINNER

RESULT: DATA QUALITY

LOSER

The data is perfectly differentiated with the Ranking and Drag Me Question.

Among the other three alternatives, the Carousel performs best.

17 SATISFACTION

66

70

71

73

75

28

23

18

20

19

3

6

8

6

3

Multiple Choice

Matrix

DragMe

Ranking

Carousel

Usability

63

65

66

67

71

31

25

23

22

20

4

9

10

11

5

Multiple Choice

Matrix

DragMe

Ranking

Carousel

Layout

60

61

64

66

69

22

21

19

18

15

15

15

11

11

13

Matrix

Multiple Choice

DragMe

Ranking

Carousel

Length

54

54

55

57

64

12

15

16

14

15

14

11

11

11

8

Matrix

Multiple Choice

DragMe

Ranking

Carousel

Topic

53

54

54

57

62

14

16

13

13

14

12

11

12

10

10

Multiple Choice

DragMe

Matrix

Ranking

Carousel

Fun

67

70

70

71

74

17

16

16

15

15

4

5

4

4

2

Multiple Choice

DragMe

Matrix

Ranking

Carousel

Comprehensibility

18 SATISFACTION

45

46

57

58

59

35

34

25

25

24

17

15

15

16

14

Multiple Choice

Matrix

DragMe

Ranking

Carousel

Overall

WINNER LOSER

19 SUMMARY

MULTIPLE

CHOICE RANKING MATRIX CAROUSEL DRAG ME

Comparability

Trustworthiness

Data Quality

Satisfaction

Technical

Requirements none JScript none JScript, Flash JScript, Flash

20 RECOMMENDATIONS

CHECK, IF YOUR STUDY PERMITS THE USAGE OF JSCRIPT AND FLASH!

(in most cases, it will)

SELECT THE QUESTIONTYPES CAREFULLY!

(there might be better alternatives)

LAYOUT AND USABILITY MATTER!

(the longer the interview, the more they matter)

IF YOU HAVE DOUBTS ASK YOUR FIELDWORK PROVIDER!

(they should have enough experience)

www.odc-services.com

f.tress@odc-services.com

@FTress

FLORIAN TRESS

THANK YOU

VERY MUCH!