what your customers really think about you (parts 1 & 2)

53
#RelateLive What Your Customers Really Think About You Part 1: Do’s and Don'ts of Survey Design

Upload: relate

Post on 11-Apr-2017

171 views

Category:

Business


3 download

TRANSCRIPT

Page 1: What your customers really think about you (parts 1 & 2)

#RelateLive

What Your Customers Really Think About You

Part 1: Do’s and Don'ts of Survey Design

Page 2: What your customers really think about you (parts 1 & 2)

Lori Gauthier, Ph.D.ZendeskDirector of Marketing Research

@datadocgauthier

Page 3: What your customers really think about you (parts 1 & 2)
Page 4: What your customers really think about you (parts 1 & 2)

Know What You Need from Your Data

DestinationInformation Construct Question

Page 5: What your customers really think about you (parts 1 & 2)

What Are You Measuring?Are You Sure?

Page 6: What your customers really think about you (parts 1 & 2)

“I know you think you understand what you thought I said but I'm not sure you realize that what you heard is not what I meant”

- Unknown

Page 7: What your customers really think about you (parts 1 & 2)

Define What You Need to Measure

Words Mean ThingsSearch definitions, synonyms, antonyms.

Page 8: What your customers really think about you (parts 1 & 2)

Source: snappywords.com

Page 9: What your customers really think about you (parts 1 & 2)

Define What You Need to Measure

Words Mean ThingsSearch definitions, synonyms, antonyms.

Use the language and tone appropriate for your population.

Result: Respondents answer the question you think you’re asking.

Page 10: What your customers really think about you (parts 1 & 2)

What Questions Should You Ask? What Response Options Should You

Provide?Understanding Construct Polarity and Scale Sensitivity

Page 11: What your customers really think about you (parts 1 & 2)

Which Way Do We Go?Construct polarity

Unipolar Construct Bipolar ConstructVery common; typically specific; often descriptive Very rare; typically global; occasionally comparative

Measures absence to maximum: not at all likely to extremely likely

Measures maximum negative to maximum positive:disapprove a great deal to approve a great deal

Midpoint represents half of construct Midpoint represents ambiguity or neutrality

5-point scale is ideal 7- or 9-point scale is ideal

How likely are you to vote in a primary this year? Do you approve or disapprove of negative campaigning?

Examples: likelihood, frequency, duration, intensity Examples: bad/good, dis/satisfied, dis/like, worse/better

common labels: not at all, slightly, moderately, very, extremely

none, a little, a moderate amount, a lot, a great deal

common labels (mirrored sides):extremely, very, moderately, slightly, neither/nor …a great deal, a lot, a moderate amount, a little, neither/nor …

zero????

Page 12: What your customers really think about you (parts 1 & 2)

Ideal scale sensitivity (example 1)How Many Scale Points Should You Use?

unipolar

not a

t all

extre

mely

moder

ately

sligh

tly very

1000 5025 75

bipolar

neith

er/no

r

extre

mely

moder

ately

sligh

tly very

1000 5025 75

sligh

tlyvery

extre

mely

moder

ately

-25-75-100 -50

Page 13: What your customers really think about you (parts 1 & 2)

Ideal scale sensitivity (example 2)How Many Scale Points Should You Use?

unipolar

not a

t all

a gre

at de

al

a mod

erate

amou

nt

a litt

lea l

ot1000 5025 75

bipolar

neith

er/no

r

a gre

at de

al

a mod

erate

amou

nt

a litt

lea l

ot1000 5025 75

a litt

lea l

ot

a gre

at de

al

a mod

erate

amou

nt-25-75-100 -50

Page 14: What your customers really think about you (parts 1 & 2)

How Many Scale Points Should You Use?Sensitivity reduced as scale points removed

unipolar

not at alllikely

extremelylikely

moderatelylikely

slightlylikely

verylikel

y

1000 5025 75

????not likely likely

Page 15: What your customers really think about you (parts 1 & 2)

How Many Scale Points Should You Use?Sensitivity reduced as scale points removed

bipolar

1000 5025 75-25-75-100 -50

neitherlike nordislike

like a great

deal

like a moderate

amount

likea little

like a lot

dislikea little

dislikea lot

dislikea great

deal

dislike a moderate

amount

Page 16: What your customers really think about you (parts 1 & 2)

How Many Scale Points Should You Use?Sensitivity reduced as scale points removed

neitherlike nordislike

like a great

deal

like a moderate

amount

likea little

dislikea little

dislikea great

deal

dislike a moderate

amount

1000 33 67-33-67-100

bipolar

1000 5025 75-25-75-100 -50

neitherlike nordislike

like a great

deal

like a moderate

amount

likea little

like a lot

dislikea little

dislikea lot

dislikea great

deal

dislike a moderate

amount

Page 17: What your customers really think about you (parts 1 & 2)

How Many Scale Points Should You Use?Sensitivity reduced as scale points removed

bipolar

1000 5025 75-25-75-100 -50

neitherlike nordislike

like a great

deal

like a moderate

amount

likea little

like a lot

dislikea little

dislikea lot

dislikea great

deal

dislike a moderate

amount

1000 50-100 -50

neitherlike nordislike

like a great

deal

like a moderate

amount

dislikea great

deal

dislike a moderate

amount

Page 18: What your customers really think about you (parts 1 & 2)

How Many Scale Points Should You Use?Sensitivity reduced as scale points removed

bipolar

1000 5025 75-25-75-100 -50

neitherlike nordislike

like a great

deal

like a moderate

amount

likea little

like a lot

dislikea little

dislikea lot

dislikea great

deal

dislike a moderate

amount

1000-100

neitherlike nordislike

like a great

deal

dislikea great

deal

Page 19: What your customers really think about you (parts 1 & 2)

A step-by-step approach to designing sound surveysWhat Have We Learned So Far?

start at your destination

define your construct

scale your construct

draft your question

Page 20: What your customers really think about you (parts 1 & 2)

Is Measurement Error Destroying Your Data?

Done with the Do’s. Let’s get to the Don’ts.

Page 21: What your customers really think about you (parts 1 & 2)

Stewie DataLook at him go!

Page 22: What your customers really think about you (parts 1 & 2)

Stewie DataLook at him go!

Random ErrorBad survey design can introduce data-destroying random error, making your data — and decisions — bounce all over the place.

Page 23: What your customers really think about you (parts 1 & 2)

Rooting Out Random ErrorSo long, Stewie!

no!nooo!

noo!

double barreled questionunexpected scale direction

insensitive scaleoverly sensitive scale

scale without midpointscale without verbal labels

overlapping scale labelsnon construct-specific scale

confusing question or scale

true|false, yes|no, agree|disagree scale

Page 24: What your customers really think about you (parts 1 & 2)

Tower of Pisa DataOne way or another, it’s gonna getcha!

Systematic ErrorBad survey design can introduce data-destroying systematic error, leading you to make biased decisions.

Page 25: What your customers really think about you (parts 1 & 2)

Banishing BiasArrivederci, Pisa!

worstever!!

!thingunbalanced scale

leading question

true|false, yes|no, agree|disagree scale

missing extreme endpoints bipolar scale without midpoint

order effectscontext effects

unbalanced question

question formatted as statement

Page 26: What your customers really think about you (parts 1 & 2)

That’s A Lot of Stuff to Remember. Let’s Recap.Phew!

Page 27: What your customers really think about you (parts 1 & 2)

A step-by-step approach to designing sound surveysWhat Have We Learned So Far?

start at your destination

define your construct

scale your construct

check for random error

check for systematic error

collect good data

bing!bing

!bing

!draft your question

Page 28: What your customers really think about you (parts 1 & 2)

Q&A

Page 29: What your customers really think about you (parts 1 & 2)

Let Me Know What YOU Think!

Share your thoughts about Part 1 of today’s workshop.

Two minutes, a few taps in your Relate Live app, and I’ll know what you think.

Thank you!

Your finger here!

Page 30: What your customers really think about you (parts 1 & 2)

#RelateLive

Page 31: What your customers really think about you (parts 1 & 2)

#RelateLive

What Your Customers Really Think About You

Part 2: Critique and Create Survey Questions

Page 32: What your customers really think about you (parts 1 & 2)

Problem

leading/unbalanced question

unbalanced scale

no construct-specific verbal labels

missing low-end scale point

scale missing midpoint

RE/SE

Response Effect

How satisfied are you with Acme’s customer support?

1 3 42

What’s wrong with this question?Measuring Customer Satisfaction

Page 33: What your customers really think about you (parts 1 & 2)

Problem

leading/unbalanced question

unbalanced scale

no construct-specific verbal labels

missing low-end scale point

scale missing midpoint

RE/SE

RE

Response Effect

semantic confusion ups volatility

How satisfied are you with Acme’s customer support?

1 3 42

What’s wrong with this question?Measuring Customer Satisfaction

Page 34: What your customers really think about you (parts 1 & 2)

Problem RE/SE

Response Effect

What’s wrong with this question?Measuring Customer Effort

To what extent do you agree or disagree with the following statement? The company made it easy for me to handle my issue.

Strongly disagree

Strongly agree

Neither agree nor disagree

Disagree AgreeSomewhat disagree

Somewhat agree

Question source: The Effortless Experience

Page 35: What your customers really think about you (parts 1 & 2)

Problem

statement as question

RE/SE

SE

Response Effect

acquiescence bias inflates ratings

What’s wrong with this question?Measuring Customer Effort

To what extent do you agree or disagree with the following statement? The company made it easy for me to handle my issue.

Strongly disagree

Strongly agree

Neither agree nor disagree

Disagree AgreeSomewhat disagree

Somewhat agree

Question source: The Effortless Experience

Page 36: What your customers really think about you (parts 1 & 2)

Critique Two Questions in EIGHT MinutesGroup Work

Page 37: What your customers really think about you (parts 1 & 2)
Page 38: What your customers really think about you (parts 1 & 2)

Review Question CritiquesGroup Work

Page 39: What your customers really think about you (parts 1 & 2)

Problem

leading/unbalanced question

unbalanced scale

no construct-specific verbal labels

missing low-end scale point

scale missing midpoint

RE/SE

SE

SE

RE

SE

RE

Response Effect

STM bias inflates ratings

DS/NN Rs pick 1, inflating ratings

semantic confusion ups volatility

zero sat Rs pick 1, inflating ratings

midpoint Rs pick?, upping volatility

How satisfied are you with Acme’s customer support?

1 3 42

What’s wrong with this question?Measuring Customer Satisfaction

Page 40: What your customers really think about you (parts 1 & 2)

Problem

incorrectly defined construct

leading/unbalanced question

confusing scale

scale missing N/N midpoint

missing scale extremes

RE/SE

SE

RE

RE

SE

Response Effect

won’t measure CSAT

STM bias inflates ratings

misinterpretations up volatility

ambig Rs pick?, upping volatility

“all the time” Rs pushed inward

What’s wrong with this question?Measuring Customer Satisfaction

What do you think about Acme’s customer support? Are you happy with it?

no, most of the time

no, some of the time

yes, some of the time

yes, most of the time

no yes

Page 41: What your customers really think about you (parts 1 & 2)

Problem

incorrectly defined construct

awkward question

confusing scale

missing low-end scale point

scale missing actual midpoint

RE/SE

RE

RE

SE

RE

Response Effect

won’t measure org-created effort

misinterpretations up volatility

“neutral” misinterps up volatility

zero Rs pick low, inflating ratings

mod Rs pick?, upping volatility

What’s wrong with this question?Measuring Customer Effort

How much effort did you personally have to put forth to get your issue resolved?

Very low effort Very high effortNeutral High effortLow effort

Question source: The Effortless Experience

Page 42: What your customers really think about you (parts 1 & 2)

Problem

statement as question

A/DA scale

non construct-specific scale

A/DA scale

confusing scale

RE/SE

SE

SE

RE

RE

RE

Response Effect

acquiescence bias inflates ratings

acquiescence bias inflates ratings

mismapping ups volatility

misinterpretations up volatility

moderately A/DA Rs pick?

What’s wrong with this question?Measuring Customer Effort

To what extent do you agree or disagree with the following statement? The company made it easy for me to handle my issue.

Strongly disagree

Strongly agree

Neither agree nor disagree

Disagree AgreeSomewhat disagree

Somewhat agree

Question source: The Effortless Experience

Page 43: What your customers really think about you (parts 1 & 2)

Create One New Question in FOUR MinutesGroup Work

Page 44: What your customers really think about you (parts 1 & 2)
Page 45: What your customers really think about you (parts 1 & 2)

Review New QuestionsGroup Work

Page 46: What your customers really think about you (parts 1 & 2)

A methodologically sound questionMeasuring Customer Satisfaction

Overall, how satisfied or dissatisfied are you with Acme’s customer support?

moderatelydissatisfied

slightlydissatisfied

neithersatisfied nor dissatisfied

slightlysatisfied

moderatelysatisfied

extremelydissatisfied

extremelysatisfied

7-point, fully labeled, construct-specific,

bipolar scale

measures what we want to measure: satisfaction with customer support

“overall” appropriate for global-level measure

balanced question

ambivalent midpoint

Page 47: What your customers really think about you (parts 1 & 2)

Measuring Customer EffortA methodologically sound question

How easy was it to get the help you needed from us today?

not at alleasy

extremely easy

moderatelyeasy

veryeasy

slightly easy

measures what we want to measure: effort needed to get company’s help “today” appropriate for

transaction-level measure

5-point, fully labeled, construct-specific,

unipolar scale

Page 48: What your customers really think about you (parts 1 & 2)

Measuring Customer EffortWhat is driving customer effort?

Content source for drivers of effort: The Effortless Experience

How did we make it difficult? (Check all that apply)

You didn’t solve the problem I had to contact the company multiple timesI felt like I was talking to a robotI had to repeat myselfI had to use a channel I don’t like (phone, web form, chat, email, FAQ)I was transferred from person to personSome other reason (Please specify)

don’t assume resolution

pick list Q measures freq of known responses

open-ended option capturesunknown responses

limit list to 7-9 options

random rotate pick list

Page 49: What your customers really think about you (parts 1 & 2)

Workshop Recap

What Your Customers Really Think About You

Page 50: What your customers really think about you (parts 1 & 2)

start at your destination

define your construct

scale your construct

check for random error

check for systematic error

collect good data

bing!bing

!bing

!draft your question

Remember!Use this step-by-step approach for designing sound surveys

Page 51: What your customers really think about you (parts 1 & 2)

Thank You!Questions? Contact me at [email protected] or

@datadocgauthier.

Page 52: What your customers really think about you (parts 1 & 2)

Let Me Know What YOU Think!

Your finger here!

Share your thoughts about Parts 1 + 2 of today’s

workshop.

Two minutes, a few taps in your Relate Live app, and I’ll

know what you think.

Thank you!

Page 53: What your customers really think about you (parts 1 & 2)

#RelateLive