what your customers really think about you (parts 1 & 2)
TRANSCRIPT
#RelateLive
What Your Customers Really Think About You
Part 1: Do’s and Don'ts of Survey Design
Lori Gauthier, Ph.D.ZendeskDirector of Marketing Research
@datadocgauthier
Know What You Need from Your Data
DestinationInformation Construct Question
What Are You Measuring?Are You Sure?
“I know you think you understand what you thought I said but I'm not sure you realize that what you heard is not what I meant”
- Unknown
Define What You Need to Measure
Words Mean ThingsSearch definitions, synonyms, antonyms.
Source: snappywords.com
Define What You Need to Measure
Words Mean ThingsSearch definitions, synonyms, antonyms.
Use the language and tone appropriate for your population.
Result: Respondents answer the question you think you’re asking.
What Questions Should You Ask? What Response Options Should You
Provide?Understanding Construct Polarity and Scale Sensitivity
Which Way Do We Go?Construct polarity
Unipolar Construct Bipolar ConstructVery common; typically specific; often descriptive Very rare; typically global; occasionally comparative
Measures absence to maximum: not at all likely to extremely likely
Measures maximum negative to maximum positive:disapprove a great deal to approve a great deal
Midpoint represents half of construct Midpoint represents ambiguity or neutrality
5-point scale is ideal 7- or 9-point scale is ideal
How likely are you to vote in a primary this year? Do you approve or disapprove of negative campaigning?
Examples: likelihood, frequency, duration, intensity Examples: bad/good, dis/satisfied, dis/like, worse/better
common labels: not at all, slightly, moderately, very, extremely
none, a little, a moderate amount, a lot, a great deal
common labels (mirrored sides):extremely, very, moderately, slightly, neither/nor …a great deal, a lot, a moderate amount, a little, neither/nor …
zero????
Ideal scale sensitivity (example 1)How Many Scale Points Should You Use?
unipolar
not a
t all
extre
mely
moder
ately
sligh
tly very
1000 5025 75
bipolar
neith
er/no
r
extre
mely
moder
ately
sligh
tly very
1000 5025 75
sligh
tlyvery
extre
mely
moder
ately
-25-75-100 -50
Ideal scale sensitivity (example 2)How Many Scale Points Should You Use?
unipolar
not a
t all
a gre
at de
al
a mod
erate
amou
nt
a litt
lea l
ot1000 5025 75
bipolar
neith
er/no
r
a gre
at de
al
a mod
erate
amou
nt
a litt
lea l
ot1000 5025 75
a litt
lea l
ot
a gre
at de
al
a mod
erate
amou
nt-25-75-100 -50
How Many Scale Points Should You Use?Sensitivity reduced as scale points removed
unipolar
not at alllikely
extremelylikely
moderatelylikely
slightlylikely
verylikel
y
1000 5025 75
????not likely likely
How Many Scale Points Should You Use?Sensitivity reduced as scale points removed
bipolar
1000 5025 75-25-75-100 -50
neitherlike nordislike
like a great
deal
like a moderate
amount
likea little
like a lot
dislikea little
dislikea lot
dislikea great
deal
dislike a moderate
amount
How Many Scale Points Should You Use?Sensitivity reduced as scale points removed
neitherlike nordislike
like a great
deal
like a moderate
amount
likea little
dislikea little
dislikea great
deal
dislike a moderate
amount
1000 33 67-33-67-100
bipolar
1000 5025 75-25-75-100 -50
neitherlike nordislike
like a great
deal
like a moderate
amount
likea little
like a lot
dislikea little
dislikea lot
dislikea great
deal
dislike a moderate
amount
How Many Scale Points Should You Use?Sensitivity reduced as scale points removed
bipolar
1000 5025 75-25-75-100 -50
neitherlike nordislike
like a great
deal
like a moderate
amount
likea little
like a lot
dislikea little
dislikea lot
dislikea great
deal
dislike a moderate
amount
1000 50-100 -50
neitherlike nordislike
like a great
deal
like a moderate
amount
dislikea great
deal
dislike a moderate
amount
How Many Scale Points Should You Use?Sensitivity reduced as scale points removed
bipolar
1000 5025 75-25-75-100 -50
neitherlike nordislike
like a great
deal
like a moderate
amount
likea little
like a lot
dislikea little
dislikea lot
dislikea great
deal
dislike a moderate
amount
1000-100
neitherlike nordislike
like a great
deal
dislikea great
deal
A step-by-step approach to designing sound surveysWhat Have We Learned So Far?
start at your destination
define your construct
scale your construct
draft your question
Is Measurement Error Destroying Your Data?
Done with the Do’s. Let’s get to the Don’ts.
Stewie DataLook at him go!
Stewie DataLook at him go!
Random ErrorBad survey design can introduce data-destroying random error, making your data — and decisions — bounce all over the place.
Rooting Out Random ErrorSo long, Stewie!
no!nooo!
noo!
double barreled questionunexpected scale direction
insensitive scaleoverly sensitive scale
scale without midpointscale without verbal labels
overlapping scale labelsnon construct-specific scale
confusing question or scale
true|false, yes|no, agree|disagree scale
Tower of Pisa DataOne way or another, it’s gonna getcha!
Systematic ErrorBad survey design can introduce data-destroying systematic error, leading you to make biased decisions.
Banishing BiasArrivederci, Pisa!
worstever!!
!thingunbalanced scale
leading question
true|false, yes|no, agree|disagree scale
missing extreme endpoints bipolar scale without midpoint
order effectscontext effects
unbalanced question
question formatted as statement
That’s A Lot of Stuff to Remember. Let’s Recap.Phew!
A step-by-step approach to designing sound surveysWhat Have We Learned So Far?
start at your destination
define your construct
scale your construct
check for random error
check for systematic error
collect good data
bing!bing
!bing
!draft your question
Q&A
Let Me Know What YOU Think!
Share your thoughts about Part 1 of today’s workshop.
Two minutes, a few taps in your Relate Live app, and I’ll know what you think.
Thank you!
Your finger here!
#RelateLive
#RelateLive
What Your Customers Really Think About You
Part 2: Critique and Create Survey Questions
Problem
leading/unbalanced question
unbalanced scale
no construct-specific verbal labels
missing low-end scale point
scale missing midpoint
RE/SE
Response Effect
How satisfied are you with Acme’s customer support?
1 3 42
What’s wrong with this question?Measuring Customer Satisfaction
Problem
leading/unbalanced question
unbalanced scale
no construct-specific verbal labels
missing low-end scale point
scale missing midpoint
RE/SE
RE
Response Effect
semantic confusion ups volatility
How satisfied are you with Acme’s customer support?
1 3 42
What’s wrong with this question?Measuring Customer Satisfaction
Problem RE/SE
Response Effect
What’s wrong with this question?Measuring Customer Effort
To what extent do you agree or disagree with the following statement? The company made it easy for me to handle my issue.
Strongly disagree
Strongly agree
Neither agree nor disagree
Disagree AgreeSomewhat disagree
Somewhat agree
Question source: The Effortless Experience
Problem
statement as question
RE/SE
SE
Response Effect
acquiescence bias inflates ratings
What’s wrong with this question?Measuring Customer Effort
To what extent do you agree or disagree with the following statement? The company made it easy for me to handle my issue.
Strongly disagree
Strongly agree
Neither agree nor disagree
Disagree AgreeSomewhat disagree
Somewhat agree
Question source: The Effortless Experience
Critique Two Questions in EIGHT MinutesGroup Work
Review Question CritiquesGroup Work
Problem
leading/unbalanced question
unbalanced scale
no construct-specific verbal labels
missing low-end scale point
scale missing midpoint
RE/SE
SE
SE
RE
SE
RE
Response Effect
STM bias inflates ratings
DS/NN Rs pick 1, inflating ratings
semantic confusion ups volatility
zero sat Rs pick 1, inflating ratings
midpoint Rs pick?, upping volatility
How satisfied are you with Acme’s customer support?
1 3 42
What’s wrong with this question?Measuring Customer Satisfaction
Problem
incorrectly defined construct
leading/unbalanced question
confusing scale
scale missing N/N midpoint
missing scale extremes
RE/SE
—
SE
RE
RE
SE
Response Effect
won’t measure CSAT
STM bias inflates ratings
misinterpretations up volatility
ambig Rs pick?, upping volatility
“all the time” Rs pushed inward
What’s wrong with this question?Measuring Customer Satisfaction
What do you think about Acme’s customer support? Are you happy with it?
no, most of the time
no, some of the time
yes, some of the time
yes, most of the time
no yes
Problem
incorrectly defined construct
awkward question
confusing scale
missing low-end scale point
scale missing actual midpoint
RE/SE
—
RE
RE
SE
RE
Response Effect
won’t measure org-created effort
misinterpretations up volatility
“neutral” misinterps up volatility
zero Rs pick low, inflating ratings
mod Rs pick?, upping volatility
What’s wrong with this question?Measuring Customer Effort
How much effort did you personally have to put forth to get your issue resolved?
Very low effort Very high effortNeutral High effortLow effort
Question source: The Effortless Experience
Problem
statement as question
A/DA scale
non construct-specific scale
A/DA scale
confusing scale
RE/SE
SE
SE
RE
RE
RE
Response Effect
acquiescence bias inflates ratings
acquiescence bias inflates ratings
mismapping ups volatility
misinterpretations up volatility
moderately A/DA Rs pick?
What’s wrong with this question?Measuring Customer Effort
To what extent do you agree or disagree with the following statement? The company made it easy for me to handle my issue.
Strongly disagree
Strongly agree
Neither agree nor disagree
Disagree AgreeSomewhat disagree
Somewhat agree
Question source: The Effortless Experience
Create One New Question in FOUR MinutesGroup Work
Review New QuestionsGroup Work
A methodologically sound questionMeasuring Customer Satisfaction
Overall, how satisfied or dissatisfied are you with Acme’s customer support?
moderatelydissatisfied
slightlydissatisfied
neithersatisfied nor dissatisfied
slightlysatisfied
moderatelysatisfied
extremelydissatisfied
extremelysatisfied
7-point, fully labeled, construct-specific,
bipolar scale
measures what we want to measure: satisfaction with customer support
“overall” appropriate for global-level measure
balanced question
ambivalent midpoint
Measuring Customer EffortA methodologically sound question
How easy was it to get the help you needed from us today?
not at alleasy
extremely easy
moderatelyeasy
veryeasy
slightly easy
measures what we want to measure: effort needed to get company’s help “today” appropriate for
transaction-level measure
5-point, fully labeled, construct-specific,
unipolar scale
Measuring Customer EffortWhat is driving customer effort?
Content source for drivers of effort: The Effortless Experience
How did we make it difficult? (Check all that apply)
You didn’t solve the problem I had to contact the company multiple timesI felt like I was talking to a robotI had to repeat myselfI had to use a channel I don’t like (phone, web form, chat, email, FAQ)I was transferred from person to personSome other reason (Please specify)
don’t assume resolution
pick list Q measures freq of known responses
open-ended option capturesunknown responses
limit list to 7-9 options
random rotate pick list
Workshop Recap
What Your Customers Really Think About You
start at your destination
define your construct
scale your construct
check for random error
check for systematic error
collect good data
bing!bing
!bing
!draft your question
Remember!Use this step-by-step approach for designing sound surveys
Thank You!Questions? Contact me at [email protected] or
@datadocgauthier.
Let Me Know What YOU Think!
Your finger here!
Share your thoughts about Parts 1 + 2 of today’s
workshop.
Two minutes, a few taps in your Relate Live app, and I’ll
know what you think.
Thank you!
#RelateLive