survey research in design
DESCRIPTION
Presentation to the Master's of Design students at Ontario College of Art and Design, Oct. 14, 2010.TRANSCRIPT
![Page 1: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/1.jpg)
Using Surveys in Design
Sam Ladner, PhDOctober 14, 2010
![Page 2: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/2.jpg)
Key Themes
• When to use surveys in design• Qual vs quant• Sampling• From interviews to instruments• Survey basics• Should we do a survey?• Asking good questions• The limits of surveys
![Page 3: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/3.jpg)
The Four Cs of ResearchDesign
• Creativity• Complexity• Compromise• Choice
Borja De Mozota, Brigitte. 2003. Design Management: Using Design to Build Value. New York: All Worth Press .
![Page 4: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/4.jpg)
Make your choice:
![Page 5: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/5.jpg)
What research design would you choose?
![Page 6: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/6.jpg)
Quantitative Research in Design
![Page 7: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/7.jpg)
I want to know what causes something else.
What do quant researchers worry about?
I really spend a lot of time wondering how to measure things.
I wonder how small patterns generalize to big patterns.
I want to make sure others can repeat my findings.
![Page 8: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/8.jpg)
I want to see the world through the eyes of my respondents.
What do qual researchers worry about?
I want to describe the context in a lot of detail.
I want to show how social change occurs. I’m interested in how things come to be.
I really want my research approach to be flexible and able to change.
![Page 9: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/9.jpg)
Qualitative vs. Quantitative
Quantitative Qualitative
Role of theory in research
Deductive, testing of theory
Inductive, generating theory
Ontological orientation
Objectivism Constructionism
Epistemological orientation
Natural science model
Interpretivism
![Page 10: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/10.jpg)
What is your orientation?
![Page 11: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/11.jpg)
A famous sampling mistake
![Page 12: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/12.jpg)
Even with proper sampling…beware!
“…predicting behavior on the basis of knowledge of attitude is a very hazardous
venture.”
![Page 13: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/13.jpg)
What exactly IS a “sample”?
![Page 14: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/14.jpg)
What exactly IS a “sample”?
A subset of the population, selected by either “probability” or “non-probability”
methods. If you have a “probability sample” you simply know the likelihood of any member of the population being
included (not necessarily that it is “random”).
![Page 15: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/15.jpg)
Assumptions of quantitative sampling
We want to generalize to the population.
Random events are predictable.
Therefore…We can compare random events to our results.
Probability sampling is the best approach.
![Page 16: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/16.jpg)
Assumptions of qualitative sampling
Social actors are not predictable like objects.
Randomized events are irrelevant to social life.
Probability sampling is expensive and inefficient.
Therefore…
Non-probability sampling is the best approach.
![Page 17: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/17.jpg)
What kind of sample would you use for a design project?
![Page 18: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/18.jpg)
From interviews to instruments…
“concepts are…categories for the organization of ideas and observations” (Bulmer, 1984: 43)
![Page 19: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/19.jpg)
A “variable” varies…
![Page 20: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/20.jpg)
What is varying?
![Page 21: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/21.jpg)
Concepts and variables
![Page 22: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/22.jpg)
What is a measure?
Something that can be counted.
Age
Height
Number of books
![Page 23: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/23.jpg)
What is an indicator?
Something that “indicates” the presence of something else.
GDP
IQ
Number of hospital visits
![Page 24: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/24.jpg)
What is survey research?
Print or Online surveys
Telephone surveysStructured face-to-face interviews
Asking structured questions
![Page 25: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/25.jpg)
Surveys versus other qualitative methodsWhen a you want to know… …you need
What are users’ opinions of the design?
What is the demographic profile of our users?
What kinds of beliefs or attitudes do our users have about our…
-brand-customer service
How can we be sure that design changes changed users opinions of the product or experience?
A survey
How usable is our Web site?
What do our users generally dislike about our product?
What kinds of features should we include in our new product?
Qualitative
research
method
![Page 26: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/26.jpg)
Is a survey the right research method?
• I know a lot about the topic• I need to summarize the
findings in numerical format• I don’t need a lot of detail and
nuance• I need to generalize the results
to the population• I need to predict the likelihood
of a certain thing happening• I want to measure incremental
change
• I don’t really know much about the topic
• I don’t need to summarize the findings in numerical format
• I need to communicate a lot of detail and nuance
• I’m interested in case study information
• I’m not planning on predicting the likelihood of anything
• I don’t need to measure any change
Yes No
![Page 27: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/27.jpg)
Is it time to do a survey?
Yes
No
Discovered questions and
answers throughQual research?
Clear idea of all possible questions
and answers?
YesNo
Formulated all questions and
possible answers?Do a survey
Do more Qual research
Yes
Pilot test the questions
No
![Page 28: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/28.jpg)
Asking good questions
• Mutually exclusive and exhaustive– No overlap between answers– All possible answers
• No double-barreled questions– Wrong: “What do you think about our response times and customer
service?”– Right: “What is your opinion of our response time?”
• Focus on opinions or beliefs of participants– Wrong: “How usable is our Web site?”– Right: “What are users’ opinions of the site’s look and feel?”
• Close-ended is best– Cuts down on analysis time
• Demographic questions at the end
![Page 29: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/29.jpg)
The lay of the land: descriptive statistics• Summarizes the responses• What is the most “typical” response?
– Mean– Mode– Median
• How much variation is there between responses?– Standard deviation– Inter-quartile range
![Page 30: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/30.jpg)
Predicting behavior: inferential statistics• Infers or predicts future behavior• Which respondents are more likely to…
– Cross-tabulation
• Which factors most influence– Behavior– Satisfaction– etc
![Page 31: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/31.jpg)
The tools you need
Online Survey Tool
Statistical Analysis Software
![Page 32: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/32.jpg)
The skills you need
• Knowledge of the topic• Ability to write good questions• Familiarity with the survey tool• Some basic knowledge of statistics• A framework for writing the report
– Important to write the outline of the final report before you start!
![Page 33: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/33.jpg)
The payoff you’ll get• “More men are dissatisfied with the
experience.” • “Younger users are much more unhappy with
customer service that older users.”• “Higher-income individuals are the most
unlikely to recommend this product to friends and family.”
• “Men are more likely to prefer a two-step process than women.”
![Page 34: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/34.jpg)
They always seem to measure artificial things
and say they’re really precise.
Yeah and they treat people like they’re test tubes or
something. It’s the way they do things…it makes it hard for people to see their research as relevant to
them.
Well their research is just so static. Real life actually
changes.
What’s wrong with quant research?
![Page 35: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/35.jpg)
What won’t you learn from your survey?
![Page 36: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/36.jpg)
What you won’t learn…
• Why aren’t people using our product?• How can we innovate on our current
design?• Why don’t people finish the checkout
process?• What are the main characteristics of the
people who don’t use our product?
![Page 37: Survey Research in Design](https://reader033.vdocuments.us/reader033/viewer/2022051515/54c788984a7959f6208b4741/html5/thumbnails/37.jpg)
Copernicus ConsultingDesign research and strategy
http://www.copernicusconsulting.net