how to design effective web surveys kevin tharp, lilian yahng, and ashley bowers 2012-2013 workshop...
TRANSCRIPT
How to Design Effective Web Surveys
Kevin Tharp, Lilian Yahng, and Ashley Bowers
2012-2013 Workshop in MethodsOctober 19, 2012
Outline
• Overview of Exciting Opportunities and Challenges of Web Surveys
• Effective Layout and Design of Web Surveys• Implementation of Web Surveys to Maximize Data
Quality• Considerations in Selecting Web Survey Tool
Opportunities of Web Surveys
• Improve survey measurement and reduce error• Eliminates interviewer effects• Features of computer-assisted instrument
(customized wording, automated skips/routing, edit checks, randomization)
• Can engage respondent • Can present rich multi-media information
• Anyone can build even a fairly sophisticated survey quickly and easily
Very
ImportantModerately Important
Somewhat Important
Not Important
At All
Did Not Use
Not Available
a. General SRO Orientation session
b. General Interviewing Techniques (GIT) training
c. Technical Tools training
d. Initial orientation (formal or informal) provided by functional team supervisor
e. Initial orientation (formal or informal) provided by administrative staff in your unit (e.g., Ann, Liza, Nancy B)
f. Ongoing assistance from the buddy formally assigned to me
g. Ongoing informal help from peers/co-workers on my functional team (e.g., TSG, PDMG, DCO, etc.)
B6. How important have each of the following sources been in helping you learn how to perform your job? If you were offered or made aware of a source but did not use it, please select “Did not use”. If you were not offered or made aware of a source, please indicate “Not available”.
B6a. [Programmer Note: Please only ask this question if more than three sources in Question 6 are marked as “Very Important”] You listed the following sources as being very important to helping you learn how to perform your job well:
ANSWER 1: FILL a-g
ANSWER 2: FILL a-g
ANSWER 3: FILL a-g
ANSWER 4: FILL a-g
ANSWER 5: FILL a-g
…
Of these, which three were the most important in helping you learn how to perform your job well?
1. [DROP-DOWN MENU WITH ANSWERS 1-5…]
2. [DROP-DOWN MENU WITH ANSWERS 1-5…]
3. [DROP-DOWN MENU WITH ANSWERS 1-5…]
Complicated Skips and Customized Wording
Randomization
Randomization
Ease of Adding Access to Other Information
(THANK YOU SCREEN)Thank you for your participation in this survey!
We would like to provide you with some resources that would be valuable to refer to while you are working at SRO. •For a list of SRO acronyms: please go to https://webtrak.isr.umich.edu/sro/•For general understanding of survey research: Survey Methodology (2nd edition), by: Robert M. Groves, et al. and the Encyclopedia of Survey Research Methods, edited by: Paul Lavrakas.
Completeness Check
http://www.270towin.com/
http://media3.surveycenter.com/FlashDevs/NPD_products/demo/black/MediaPlayers/analyzer/player.asp
http://media4.surveycenter.com/FlashDevs/WebSite/EmotionPicker/index.html
Opportunities of Web Surveys (2)
• Lowers cost• Greater speed• Environmentally friendly• Reduces respondent burden - complete survey at
convenience and over multiple time periods• Wealth of survey process data for analysis
Challenges of Web Surveys
• Reduce quality of survey measurement• Lack of interviewer presence• Constrained format (versus paper)• Look and feel not under direct control and may affect
measurement• Browser, connection type and speed, font size• Type of device used (smartphone, iPad, laptop,
desktop)• Anyone can easily and quickly put together a survey
http://www.quizrocket.com/pirate-vs-ninja-quiz
Optimized for Device
• Please text TOLUNA8 to the following #: 91318
http://www.polleverywhere
Challenges of Web Surveys (2)
• Poor reporting behavior (person respond multiple times, straightlining)
• Complexity – more time and money• Lower response rates• No comprehensive list of email addresses for US
population
Strategies for
maximizing response and minimizing error
Web Survey Layout and Design
Case study in web survey design: NSSE (National Survey of Student Engagement)
• Annual survey of college students• 600+ institutions w/ customization• Large sample (1 mil+)• Administered since 2000
– Mostly paper originally– Change in web design every few years
Scales
• 2008 • 2010
Scales (2012)
Birth year
• 2003 • 2008
Birth year (2012)
Major (2000)
Major (2003)
Major (2008)
Major (2012)
Pagination
• nsse 2000• nsse 2003• nsse 2008• nsse 2010• nsse 2012 (2.0 pilot)
2013?
• Progress indicator• Submit on click• More paths• Mobile challenges
Strategies for
maximizing response and minimizing error
Web Survey Implementation
“Implementation” – data collection protocols
Implementation Aims:•Maximize unit response (response rate)•Minimize “error” (deviations in survey processes that lead to inaccuracy or bias)
• With regard to implementation: nonresponse error
But this is not the only source of error!
High response rates do not guarantee the absence of response bias. Neither do low response rates necessitate response bias.
October 19, 2012Center for Survey Research
Groves R M , Lyberg L Public Opin Q 2010;74:849-879
© The Author 2011. Published by Oxford University Press on behalf of the American Association for Public Opinion Research. All rights reserved. For permissions, please e-mail: [email protected]
TOTAL SURVEY ERROR
High response rates reduce the risk of response bias.
So, how to turn sample members into respondents?
(without introducing more potential for error)
October 19, 2012Center for Survey Research
• Think strategically about your target population
• What would induce them in particular to take the survey?
• What would facilitate their giving accurate responses?
• Monitor your data during data collection for responsive measures
• Look at the survey data: response distributions by subsample, variable crosstabs, open-response items, etc.
• Look at process data: track individuals, break-offs, times of completion
• Construct responsive (possibly corrective) strategies
• Try experimenting, particularly if multiple administrations or years
October 19, 2012Center for Survey Research
Implementation Planning Checklist
Sample preparation Data storage and archiving Survey distribution, timing, field period Follow-up and nonresponse protocols Email message text and signature Incentives Leave time for testing And… do a pilot, if possible!
October 19, 2012Center for Survey Research
Sample Preparation Sample preparation
• How is the sample to be acquired? How current is the contact information? How reliable are the email addresses?
• Need to assign case id numbers? • Append other (e.g. demographic or subsample) information? • Check the list for blank fields, duplicate emails, conflict of
interest, ineligibles, etc.
Data management: storage and archiving plan• What is the level of security needed for the project/grant?• IU Libraries Data Management Resources
October 19, 2012Center for Survey Research
Email MessagesWrite to your audience (target population):
Content/composition: Survey duration, sponsor, purpose, study contact info. First person singular/plural? Length of message? Deadline?Branding your survey: Logo? Survey title?
Signature: Who should sign it? An individual vs. an entity?
Survey link: Placement? Masked link? Subject and From lines Tech considerations: html vs. plain text, images,
emphasis tags, email clients, etc.
October 19, 2012Center for Survey Research
Survey Distribution• Method(s) of contact? Via individual email addresses,
list-serv, on a website, postal mail, phone call, text message, something else?
• Make it convenient for your target population: Timing: Month, day, time? Holidays or special schedules? Follow-up reminders: How many, time between them?
Different contact method? Field length
October 19, 2012Center for Survey Research
Incentives
• Do they work? How much of a boost can be expected?
• What kinds of incentives are most effective?
• Differential incentives?• See university or grant policies on prizes
and lotteries
October 19, 2012Center for Survey Research
Nonresponse
What kind of nonresponse was it?• Delivery failure (noncontact) • Refusal to participate (busy, not interested)• Inability to participate (language, visual
impairment)• Mistaken perceptions of ineligibility
Compare to other data?
Post-survey interviews?October 19, 2012Center for Survey Research
Some Considerations and Available Sets of Tools
Selecting a Web Survey Tool
What Are Top 2-3 Features You Would Need or Want to Have in Web Survey Tool?
Disclaimer: I do not endorse any web survey tool. This presentation is only one person’s view. Only a few tools are mentioned. This is not intended to be a comprehensive review.
Some Considerations in Selecting Web Survey Tool
• Survey Complexity
• Layout Customization
• Multilanguage Capabilities
• Ease of Development/Programming
• Able to Integrate with Other Modes of Data Collection
• Data Output and Analysis
• Security
• Number of Users/Scalability
• Integrated Email and Sample Management Capabilities
• Cost
• Help and Support
Set #1
• Examples: SurveyMonkey, SurveyExpression, SurveyGizmo, QuestionPro, eSurveysPro, FluidSurveys, PollDaddy…………………………many more
• Generally less expensive or even free
• Fair amount of features for designing survey but may be limited somewhat in customization (customized wording)
• Easy to program/develop
• Less sophisticated in data output and analysis
• More limited in scalability/number of users
• Data stored in company servers with differing security protocols
Set #2
• Examples: Qualtrics, LimeSurvey, DatStat Illume, Key Survey• Generally more expensive than Set #1• More options for survey design although still not as flexible and
customizable as in-house developed product• Fairly easy to program/develop• More sophisticated in data output and analysis• Data stored in company servers with often stringent security
protocols to meet government or other standards• Additional option to consider: Email management tools with
survey add-on (e.g., Constant Contact)
Set #3
• Examples: In-House Systems, Blaise IS, CASES Web
• More expensive
• Highly scalable
• Security developed by organization so can meet stringent standards
• High degree of customization and flexibility
• Highly complex instruments in multiple languages possible
• Possibility of integrating with other modes of data collection
Sources for Presentation/Resources
• Couper, Designing Effective Web Surveys• Dillman, Internet, Mail and Mixed Mode Surveys:
The Tailored Design Method • http://www.websm.org/• Survey Geek Blog:
http://regbaker.typepad.com/regs_blog/• Comparisons of web survey tools, e.g.:
http://www.idealware.org/articles/fgt_online_surveys.php
Thank you!
Kevin [email protected]
October 19, 2012Center for Survey Research
Indiana UniversityCenter for Survey Research
For more information:
Lilian [email protected]
Ashley [email protected]