diy: research on a shoestring budget
DESCRIPTION
Presentation given at the CASE Communications, Marketing & Technology Conference in Boston on April 15, 2009.Learn the tools of the trade for do-it-yourself research for little or no money. This session will teach you how to conduct focus groups, surveys, usability tests and more.TRANSCRIPT
DIY: Research On a
Shoestring Budget
J. Todd BennettManaging Partner,
decimal152
Why do research?
Justify
Improve
Monitor
Inform
This is not academic research
Quantitative vs. Qualitative
• QualitativeBest in earlier phases of project. Not sure
what you’re looking for.
• Quantitative
Best in later phases. Test hypotheses.
Some qualitative techniques
• Focus Groups• Interviews• Observations • Subjective analysis• Environmental Scans
Some quantitative techniques
• Surveys and questionnaires
• Data mining/ modeling
Mixed?
• Web analytics
• Usability testing
The qualitative helps give meaning and context to the
quantitative
Steps in the process
1. Define the problem2. Design the study 3. Collect data 4. Analyze data 5. Write report/ presentation
Lay the foundation
The problem“My boss wants me to do a survey”
The research questionsWhat do you want to know?
The methodWhich method will help you get there?
Be careful
How to…
Focus Groups
group interview with 10-12 members of a target audience (students,
faculty, alumni, etc.), conducted by a moderator
– relatively low cost– probe deeper as issues arise– results are almost immediate– should not be generalized to the greater
population
Recruiting participants
Need 8-12 participants, so recruit 20-25.
They should: – represent your target audience and be as
homogenous as possible – not have expertise in market research or marketing
Provide an incentive. Use existing groups only if necessary.
Focus group location
• Don’t need a fancy room• Free from distractions– private
– No public spaces
• Not too large or noisy• Seated in a circle or around a table• Pay attention to lighting and
temperature
Choosing a moderator
• Professionally trained, ideally
• Excellent speaking & listening skills
• Not too close to the project at hand
• Objective
Role of the moderator
• promote open discussion and to draw out people’s differences
• probe for details or move things forward- keep focused
• ensure everyone participates and gets a chance to speak
• LISTEN- do not control conversation or answer questions
Record the session
• Digital audio recording best
• Transcribe: http://www.verbalink.com/
• Have a notetaker in the room– Note the non-verbal– Back-up for audio
Lay the ground rules
• Introduction• Why are they here?• Encourage participation- no right or
wrong answers• Respectful of disagreement• Ensure confidentiality• Permission to record
Sample Introduction Script
Welcome to our session on [PROJECT]. Thank you for taking the time to meet with us and to discuss [PROJECT]. My name is [NAME] . Assisting me is [NAME].
You were selected because you [AUDIENCE]. We are particularly interested in your views because [REASON].
I’ll start by asking some questions and would like you to respond to them. There are no right or wrong answers. Please feel free to share your point of view even if it differs from what others have said. We do ask that you are all respectful of others’ opinions and give everyone a chance to share their views. We also ask that only one person speak at a time. Because we have a number of issues to discuss, I may have to cut you short and ask to move on. Please don’t be offended. We just need to keep things moving so we can finish on-time.
We want you to be as open and honest as possible, so we guarantee that everything you say will be kept confidential in the sense that your name will never be linked to your comments shared with [CLIENT]. However, we would like to record our conversation because it is difficult to write and follow the discussion at the same time and it’s important we don’t miss any of your comments. Do I have everyone’s consent to record the session? If so, let’s begin with the first question.
The questions
• 8-12 broad questions prepared– Several probing questions (prompts) for each
• Start with the broadest questions first
• Several seconds of silence is ok– Avoid the temptation to give examples
• Unexpected paths are ok
The results
• Notes/ Transcripts
• Content analysis
• Identify hot-button issues or areas for additional research
A few applications
Image studies
Design evaluations
Website audience needs assessment
Surveys
Survey Options
• Mail• Telephone• Online
– Solicited– Site intercept
• In-person– “mall intercept”– Group setting
Steps in conducting surveys
1. Decide the purpose2. Develop the research questions3. Define the target population4. Create the sample5. Write & design the survey instrument6. Pilot test the questionnaire7. Prepare letter/email/intro to distribute8. Follow-up9. Analyze and report
Purpose
• What are you trying to do/learn?– May be more than one objective
Example:
The purpose is to compare the content preferences of our various website audiences.
Research questions
The guide to what you will ask in the survey
Survey questions should always be
tied back to your research questions.
Purpose: to compare the content preferences of our various website
audiences.
Research questions:
• What content is most important to prospective students, faculty and alumni?
• What tasks are most important to prospective students, faculty and alumni?
• Do differences exist in preferences of prospective students, faculty and alumni?
Target Population
Who do you send the survey to?
– Determine your target population
– Is the entire population known or available?
– Where does that list come from?• HR, Admissions, Alumni database, SIS, purchased
lists (chamber of commerce, etc)
Sampling
How many people do you need to distribute it to?
– Experiment with online sample size calculator, confidence interval .05
– Expect 10-20% response from prospective students (or others who don’t know you). 60%+ from internal audiences
– Use your gut– how many responses would make you feel comfortable with results?
Sampling
Writing the survey
• NEVER ask a question unless you intend to do something with the info– Nice to know isn’t reason enough
• Tie each question to your research questions• Careful with language- avoid jargon, use
simple words, watch for bias and unequal comparisons
• Avoid “why” questions• Don’t ask “when” questions• No hypotheticals
Writing the survey
• Objectionable questions– Does it need to be asked?– For income or demographics, give ranges
• Behavior– Within 6 months– Ask “typically” or “usually”
• Attitude questions– Avoid middle alternative– Or follow with an intensity question: “How strongly do
you feel about…”
Attitudes, opinions, values
Use Likert or Semantic Differential scales
• Likert• Strongly agree, agree, disagree, strongly disagree
• Very happy, happy, unhappy, very unhappy
• Semantic Differential (bipolar adjectives)
• Efficient 7 6 5 4 3 2 1 Inefficient
• Fair +3, 2, 1, 0, -1, -2, -3 Unfair
Tips for using scales
• Scales usually indicate intensity of feeling or amount of something
• Do not include “don’t know” or “undecided” on Likert
• Use an even number of points, equally distant
• Avoid absolutes like “always” or “never” as end points
• Ask one question to determine if they are qualified to answer the following one– “Have you ever… if yes, …”
Building the survey
• Surveymonkey– Free for 100 responses. $19.95/ 1000
responses, $200 annually unlimited
• Does your institution have a tool?
The invite
• The purpose of the survey
• How will their responses help?
• What will they get in return?
• Ensure confidentiality
• Provide a deadline
• How to get results
• Thanks
Pilot test
• Feedback on the invite
• Test the flow (does the order make sense, is it monotonous?)
• Time to complete
• Do the response patterns make sense? Are they consistent?
• Do any questions cause confusion?
Distribution of surveys
– First follow-up in 5-7 days– Final reminder 1-2 days before close of
survey
You will get exponentially fewer responses to follow-ups after the first. Don’t harass your sample!
Analysis and reporting
• Basic frequencies usually sufficient• Cross-tabs by
audience/group/variable optional • Surveymonkey produces a report of
frequencies in nice charts• Excel also works fine. No real need
for SPSS.
Competitive analysis
WebsitesPublications
NewsPrograms
Competitive analysis
• Heuristics (rules of thumb)
– Navigation– Aesthetic, readability & design– Content– Multimedia and features– External Web presence
“Why aren’t we doing that?”
It’s easy to judge without context.
Institutions’ goals and strategies may differ.
While one might perceive a site/publication/strategy to be “better”
than another, each may be equally successful at fulfilling its intended
purpose.
Usability testing
Talk aloud method
• Develop scenarios – Task and information based
• Complete the tasks, talking aloud
• Observe paths, clicks, verbal impressions, body language– Note areas of difficulty
Data collection
Don’t need fancy software, but it helps
– Morae- fairly expensive– Silverback- $49, for Mac only– Cool service- www.userfly.com
Prototyping
• Paper or HTML
• Useful before development
• Quick, iterative– Mock-up, test, revise, test again
Prototyping
flickr.com/photos/bensauerflickr.com/photos/cesarastudillo
Card sorting• Useful for site architecture
(navigation, organization, labels)
• Looking for patterns (mental models) of how users group and organize content
• Open vs. closed card sorting
• Focuses on content, but not tasks
Card Sorting
.flickr.com/photos/yandle
5-second testswww.fivesecondtest.com
• Classic– list the things they can recall after viewing your site for 5 sec. – good for discovering the most prominent calls to action on pages
• Compare– choose a preferred interface after viewing each for 2.5 sec. – good for assessing design variations
• Sentiment– choose most and least favorite elements after viewing your site
for 5 sec. – good for identifying strengths and weaknesses in the design
Web analytics
Useful for measurement, but also for research
Do people do what they say they do?
• Look for ways the behavioral data supports/refutes what you learned in other
research