the effect of incentives on internet surveys john m. kennedy judith a. ouimet indiana university...
TRANSCRIPT
The Effect of Incentives on Internet Surveys
John M. KennedyJudith A. Ouimet
Indiana University Bloomington
Why Provide Incentives
• To improve survey data quality– Higher response rates– Fewer breakoffs– Fewer missing items– More precise responses
• To improve the respondent experience– Respect for their time– Reward for cooperation
Improving Data Quality
• Most incentive research focuses on improvements in response rates
• Improved response rates do not necessarily indicate better quality survey data
• Better quality data is difficult to determine– Rarely are external data available to evaluate
responses– Differences may be due to sampling
Incentives May Not Be the Solution
• Incentives may not be cost effective– Resources used for incentives might be used for
other procedures that improve survey data quality • Incentives may reduce survey data quality– Encourage respondents to complete to receive
incentive– Reduced attention to survey questions– Trigger spam filters
Improving Respondent Experience
• Declining response rates may be due to badly designed surveys which demonstrate indifference to the respondent experience
• Improving respondent experience requires demonstration of respect for respondents’ time and effort
• Not all incentives show respect for all participants• The research on incentives does not indicate which
incentives work best under which conditions
Internet Survey Incentives
• More difficult to manage than postal, in-person, or telephone surveys– Often less contact information is available for
Internet users– Distribution may require postal mailings– Additional costs and modes may take resources
from the survey administration
Types of Incentives:Internet Only
• Non-contingent– Advanced cash or gift– All in sample entered into a lottery– Resources provided to those who do not
participate– Generally requires postal mailing or requires
additional effort of respondent to pick-up incentive
Contingent Incentives
• Usually easier to administer– Contact information can be gathered
• Types– Lotteries, cash, gift cards
• Provided to those who participate in the survey― Reduced costs
Internet Survey Incentives Research
• Research conducted in US and Europe• Mixed results– Not all incentives improve survey data quality
• Evaluation standards differ across studies– Some evaluate response rates, item nonresponse,
breakoffs.– Many examine multiple indicators of survey data
quality
Internet Survey Incentive Research
• 30 articles and chapters– Four meta-analyses or literature reviews
• Internet surveys only or main focus• Either experiments or comparisons with
similar surveys• 15 conducted in the US and Canada; 11 in
Europe• Students, physicians, scientists, panels, etc.
More Incentives Research
• Non-contingent and Contingent• Types– Cash – Gift cards– Lotteries
• Focus– Response rates– Item non-response– breakoffs
Response Rates Summary
• Overall – incentives appear to improve response rates
• Lotteries improve response rates often but not always
• Non-contingent almost always improve response rates
• Improvements are usually small
Other Outcomes
• Incentives generally reduce item non-response and breakoffs
• Cash is more effective than lotteries• Incentives appear to improve response rates
for the first wave of a panel only• Incentives can have differential effects on
different groups• Incentives can affect survey responses
Summary
• Results are mixed• Differences by groups, contingency, stage of
survey process, etc.• Cost effectiveness is not easily determined• Incentives are determined by the researchers,
not the participants– Not necessarily respect the participants’ choice– May reduce effectiveness with the wrong
incentives
Student Survey Example
• National Survey of Student Engagement• Survey of undergraduates in US and Canada• Started in 2000• Generally lower response rates than desired• Incentives offered differentially– Universities offer incentives– Different incentives over time
NSSE and IUB
• Response rates declining over time• Higher response rates desired• Incentives introduced in 2010• Follow-up survey to determine reasons for
participation or non-participation • Asked about incentives that would make
response more likely
NSSE 2001-2009 Response rates & Trend line
'01 '02 '03 '04 '05 '06 '07 '08 '0915%
20%
25%
30%
35%
40%
45%
38%36%
29%
24%
31%
27%
32%
26%24%
Incentives Offered 2010
• Every responder received a free soda at the IMU
• Prizes:– 5 iPods– 2 lunch for 4 at the Tudor Room– 50 semester lockers & towel service– 25 CycleFit 3 session punchcards
Invitation Follow-up Reminder 1 Reminder 2 Final Reminder0%
5%
10%
15%
20%
25%
30%
35%
40%
7%
4%5% 4%6% 5%4%
12%
9%
6%3%
17%
22%
26%
11%16%
20%
24%
21%
27%
32%
35%
NSSE Response Rates by Year & ReminderCompletes & Partials
As of 5/7/10
2008 2009 2010Overall 2008 Overall 2009 Overall 2010
Incentive Survey Process• Surveyed students after the NSSE survey ended• Created three questionnaires with focused response options
depending on whether students participated in the survey or did not.– Respondents– Non-respondents
• First year students• Seniors
• Most questions focused on reasons for participation or non-participation
• Asked students what kinds of incentives would encourage them to participate in NSSE
What contributed to your willingness to respond?
Other
Complain about IU
Chance to win locker & towel service
Chance to win the free CycleFit class
Free fountain drink coupon
Chance to win the free lunch for four
Wanted my opinion heard
Chance to win the IPOD Touch
Help IU
Usually complete surveys when asked
9%
8%
9%
10%
17%
17%
26%
35%
44%
51%
RESPONDER STUDENT-SUGGESTED INCENTIVES T-SUGGESTED INCENTIVES
NON-RESPONDER STUDENT- SUGGESTED INCENTIVES
Closing the Loop – Incentives in 2011
• Increased the number of incentives• Increased the odds of winning• Drawing only
Incentives Offered in 2011• Apple iPad 32 GB (value $599—1 winner)• Apple IPod Touch (value $199—3 winners)• Crimson Hooded sweatshirt with white IU Trident (value $40—48 winners) • Lunch for two at the Tudor Room (value $26—8 winners)• Crimson T-shirt with white IU Trident (value $18—145 winners)• Mother Bear’s Pizza large, one-topping pizza (value $14—250 winners)• Crimson foldable 42 inch umbrella with white IU Trident (value $13—50 winners)• Cyclefit 3 session punch card (value $12—5 winners)• Circuit Training session punch card (value $12—5 winners)• TIS gift card (value $10—250 winners)• Scholar’s Inn Bakehouse gift card (value $10—125 winners)• SRSC T-shirt (value $10—10 winners)• Gift card—Target, Amazon, or Starbucks (value $10—100 winners)• Bloomington Bagel Company 5 Wooden Nickels (value $5 total—257 winners)
Invitation Follow-up Reminder 1 Reminder 2 Final Reminder0%
5%
10%
15%
20%
25%
30%
35%
40%
7%
4%5%
4%6% 5%
12%9%
6%
3%
8%
4% 4% 3%
17%
22%
26%
11%
16%
20%
24%
21%
27%
32%
35%
13%
22%
26%
30%32%
NSSE Response Rates by Year & Reminder Completes & Partials
2008 2009 2010 2011 Overall 2008Overall 2009 Overall 2010 Overall 2011
Follow-Up Survey 2011
• Conducted after NSSE IUB field period ended• Similar to 2010 Follow-up survey– Reasons for participation or non-participation– Desired incentives
Other
None of the above
IU Umbrella
CycleFit or Circuit Strength Punch Card
$5 Bloomington Bagel Company coins
$10 Scholar's Inn Bakehouse gift card
IU T-shirt
Tudor Room
$10 TIS gift card
Mother Bear's Large one-topping pizza
IU Sweatshirt
iPod Touch
Apple iPad2
6%
19%
20%
21%
26%
29%
32%
37%
38%
42%
43%
60%
72%
WHICH AFFECTED YOUR DECISION TO COMPLETE THE SURVEY?
Incentives in 2012
• Two stage incentives– All completers• Scoop of ice cream (coupon sent electronically)• Free bagel coupon (picked-up at library)
– All completers• Entered into a weekly drawing to win one of 115 prizes• Earlier students complete survey, more opportunities to
win
Drawing Incentives in 2012Prizes Total Number Prizes Total Number
iPad 32 GB 1$10 Cycle Fit 3 session punch cards 5
$10 Target Gift Card 6Circuit Strength 3 session package 5
Crimson Hooded sweatshirt w/ white IU Trident 13 $10 gift card at BBC 2Mother Bear's large one-topping pizza 150 $25 gift card TIS 4
Lunch for Two at the Tudor Room 12Jackets from Underground Printing 6
Crimson foldable umbrella w/White IU Trident 16 Underground T-shirts 300
TIS $10 gift card 47 $50 Visa gift card 1Scholar's Inn Bakehouse ($10 gift card) 55 IU Flag (5x8) 1Crimson T-shirt w/ white IU Trident 67 IU Soccer Scarf 1
Track & Field Baseball Cap 1 IU Athletic Team T-shirts 4
SRSC T-shirts 10 TOTAL PRIZES 707
2012 Response Rate
• Changes– Response rate decreased to 30%– Bus with NSSE facts– List of past winners on website (views 1489 time)– List of prizes embedded in invitation and follow-up
emails (viewed 1495 times)• Speculation on why– Later start date– Competing surveys
Lessons Learned
• Sample members may not accurately predict the utility of incentives
• Managing a complex set of incentives requires resources
• Respondents may attribute more of their decision to participate to the incentives when asked later
Conclusions
• Incentives can improve response rates for Internet surveys
• The effect of incentives on data quality and differences across groups needs much more research
• More research is needed on why and how incentives affect the decision to participate
