make better surveys
TRANSCRIPT
@jeremy74
About me
UX researcher & designer for 7 years
Agency and client side
BSc. in Psychology & Cognitive Neuroscience
MSc. in Human Computer Interaction (HCI)
Published psychology study in peer reviewed journal
Run many surveys!
@jeremy74
Surveys are hard
Face to face conversations are rich interactions
We seek feedback and modulate our responses
We can know when there is shared understanding
Surveys are rigid and prone to misunderstanding
@jeremy74
Is a survey the best methodology?
Avoid doing it instead of direct user contact e.g. interviews
Are there better ways to answer your question?
@jeremy74
Be clear about the purpose
Be clear about what you want to find out
What will you do with the information
Hypotheses
@jeremy74
A bad survey doesn’t smell
Often don’t know if the questions were bad/missed asking the right things
@jeremy74
Be ruthless with question choice
People will only answer so many questions
Justify every question
Respect the users’ time
Branch it
@jeremy74
Don’t assume people can/want to answer
Adds noise to the data or increases drop-out
Allow skipping
Include active skipping e.g. n/a
Include ‘other’ option
@jeremy74
Put boring questions at the end
Improves completion rates
Completion tendency/sunk cost
Still get partial info for drop outs
Roberson, M.T. & Sundstrom, E. (1990). Questionnaire design, return rates, and response favorableness in an employee attitude questionnaire. Journal of Applied Psychology, 75, 354-357.
Martin, J.D. & McConnell, J.P. (1970). Mail questionnaire response induction: the effect of four variables on the response of a random sample to a difficult questionnaire. Social Science Quarterly, 51, 409-414.
@jeremy74
Be clear/specific
“How often do you cook?”
“How often do you prepare a meal?”
“How often do you prepare a meal, excluding ready-meals”
@jeremy74
Be clear/specific
“How often do you cook?”
“How often do you prepare a meal?”
“How often do you prepare a meal, excluding ready-meals”
“How often do you prepare an evening meal, excluding ready meals?”
@jeremy74
Be clear/specific
“How many times did you prepare an evening meal, excluding ready-meals in the last 7 days?”
@jeremy74
Be clear/specific
“How many times did you prepare an evening meal, excluding ready-meals in the last 7 days?”
“Did you prepare any evening meals, excluding ready-meals, in the last 7 days?”
then, if so,
“How many did you prepare?”
@jeremy74
Subjective questions are unreliable
Like usability testing, behaviour is more reliable than attitudes/preferences
Bertrand, M. & Mullainathan, S. (2001). Do people mean what they say? Implications for subjective survey data. American Economic Review, 91, 67-72.
@jeremy74
Anticipate biases
“Is sustainable seafood important to you”
versus
“Are you interested in any of the following?”
@jeremy74
Free text responses
Be judicious
Do you really have no idea of range of responses?
Free text as an answer option
@jeremy74
Other things to get right
Avoid leading questions
Avoid double barrelled questions
Avoid absolutes
Target the appropriate audience
Reduce reliance on memory
Consider cost/benefit of open ended questions
Map it out if it’s complicated
@jeremy74
Other things to get right
Consider a pilot study
Randomise question order, and answer options, where possible
Make sure it works on mobile
Map it out if it’s complicated
@jeremy74
Other things to get right
Consider a pilot study
Randomise question order, and answer options, where possible
Make sure it works on mobile
Map it out if it’s complicated