the effect of incentives on internet surveys john m. kennedy judith a. ouimet indiana university...

The Effect of Incentives on Internet Surveys

John M. KennedyJudith A. Ouimet

Indiana University Bloomington

Why Provide Incentives

• To improve survey data quality– Higher response rates– Fewer breakoffs– Fewer missing items– More precise responses

• To improve the respondent experience– Respect for their time– Reward for cooperation

Improving Data Quality

• Most incentive research focuses on improvements in response rates

• Improved response rates do not necessarily indicate better quality survey data

• Better quality data is difficult to determine– Rarely are external data available to evaluate

responses– Differences may be due to sampling

Incentives May Not Be the Solution

• Incentives may not be cost effective– Resources used for incentives might be used for

other procedures that improve survey data quality • Incentives may reduce survey data quality– Encourage respondents to complete to receive

incentive– Reduced attention to survey questions– Trigger spam filters

Improving Respondent Experience

• Declining response rates may be due to badly designed surveys which demonstrate indifference to the respondent experience

• Improving respondent experience requires demonstration of respect for respondents’ time and effort

• Not all incentives show respect for all participants• The research on incentives does not indicate which

incentives work best under which conditions

Internet Survey Incentives

• More difficult to manage than postal, in-person, or telephone surveys– Often less contact information is available for

Internet users– Distribution may require postal mailings– Additional costs and modes may take resources

from the survey administration

Types of Incentives:Internet Only

• Non-contingent– Advanced cash or gift– All in sample entered into a lottery– Resources provided to those who do not

participate– Generally requires postal mailing or requires

additional effort of respondent to pick-up incentive

Contingent Incentives

• Usually easier to administer– Contact information can be gathered

• Types– Lotteries, cash, gift cards

• Provided to those who participate in the survey― Reduced costs

Internet Survey Incentives Research

• Research conducted in US and Europe• Mixed results– Not all incentives improve survey data quality

• Evaluation standards differ across studies– Some evaluate response rates, item nonresponse,

breakoffs.– Many examine multiple indicators of survey data

quality

Internet Survey Incentive Research

• 30 articles and chapters– Four meta-analyses or literature reviews

• Internet surveys only or main focus• Either experiments or comparisons with

similar surveys• 15 conducted in the US and Canada; 11 in

Europe• Students, physicians, scientists, panels, etc.

More Incentives Research

• Non-contingent and Contingent• Types– Cash – Gift cards– Lotteries

• Focus– Response rates– Item non-response– breakoffs

Response Rates Summary

• Overall – incentives appear to improve response rates

• Lotteries improve response rates often but not always

• Non-contingent almost always improve response rates

• Improvements are usually small

Other Outcomes

• Incentives generally reduce item non-response and breakoffs

• Cash is more effective than lotteries• Incentives appear to improve response rates

for the first wave of a panel only• Incentives can have differential effects on

different groups• Incentives can affect survey responses

Summary

• Results are mixed• Differences by groups, contingency, stage of

survey process, etc.• Cost effectiveness is not easily determined• Incentives are determined by the researchers,

not the participants– Not necessarily respect the participants’ choice– May reduce effectiveness with the wrong

incentives

Student Survey Example

• National Survey of Student Engagement• Survey of undergraduates in US and Canada• Started in 2000• Generally lower response rates than desired• Incentives offered differentially– Universities offer incentives– Different incentives over time

NSSE and IUB

• Response rates declining over time• Higher response rates desired• Incentives introduced in 2010• Follow-up survey to determine reasons for

participation or non-participation • Asked about incentives that would make

response more likely

NSSE 2001-2009 Response rates & Trend line

'01 '02 '03 '04 '05 '06 '07 '08 '0915%

20%

25%

30%

35%

40%

45%

38%36%

29%

24%

31%

27%

32%

26%24%

Incentives Offered 2010

• Every responder received a free soda at the IMU

• Prizes:– 5 iPods– 2 lunch for 4 at the Tudor Room– 50 semester lockers & towel service– 25 CycleFit 3 session punchcards

Invitation Follow-up Reminder 1 Reminder 2 Final Reminder0%

5%

10%

15%

20%

25%

30%

35%

40%

7%

4%5% 4%6% 5%4%

12%

9%

6%3%

17%

22%

26%

11%16%

20%

24%

21%

27%

32%

35%

NSSE Response Rates by Year & ReminderCompletes & Partials

As of 5/7/10

2008 2009 2010Overall 2008 Overall 2009 Overall 2010

Incentive Survey Process• Surveyed students after the NSSE survey ended• Created three questionnaires with focused response options

depending on whether students participated in the survey or did not.– Respondents– Non-respondents

• First year students• Seniors

• Most questions focused on reasons for participation or non-participation

• Asked students what kinds of incentives would encourage them to participate in NSSE

What contributed to your willingness to respond?

Other

Complain about IU

Chance to win locker & towel service

Chance to win the free CycleFit class

Free fountain drink coupon

Chance to win the free lunch for four

Wanted my opinion heard

Chance to win the IPOD Touch

Help IU

Usually complete surveys when asked

9%

8%

9%

10%

17%

17%

26%

35%

44%

51%

RESPONDER STUDENT-SUGGESTED INCENTIVES T-SUGGESTED INCENTIVES

NON-RESPONDER STUDENT- SUGGESTED INCENTIVES

Closing the Loop – Incentives in 2011

• Increased the number of incentives• Increased the odds of winning• Drawing only

Incentives Offered in 2011• Apple iPad 32 GB (value $599—1 winner)• Apple IPod Touch (value $199—3 winners)• Crimson Hooded sweatshirt with white IU Trident (value $40—48 winners) • Lunch for two at the Tudor Room (value $26—8 winners)• Crimson T-shirt with white IU Trident (value $18—145 winners)• Mother Bear’s Pizza large, one-topping pizza (value $14—250 winners)• Crimson foldable 42 inch umbrella with white IU Trident (value $13—50 winners)• Cyclefit 3 session punch card (value $12—5 winners)• Circuit Training session punch card (value $12—5 winners)• TIS gift card (value $10—250 winners)• Scholar’s Inn Bakehouse gift card (value $10—125 winners)• SRSC T-shirt (value $10—10 winners)• Gift card—Target, Amazon, or Starbucks (value $10—100 winners)• Bloomington Bagel Company 5 Wooden Nickels (value $5 total—257 winners)

Invitation Follow-up Reminder 1 Reminder 2 Final Reminder0%

5%

10%

15%

20%

25%

30%

35%

40%

7%

4%5%

4%6% 5%

12%9%

6%

3%

8%

4% 4% 3%

17%

22%

26%

11%

16%

20%

24%

21%

27%

32%

35%

13%

22%

26%

30%32%

NSSE Response Rates by Year & Reminder Completes & Partials

2008 2009 2010 2011 Overall 2008Overall 2009 Overall 2010 Overall 2011

Follow-Up Survey 2011

• Conducted after NSSE IUB field period ended• Similar to 2010 Follow-up survey– Reasons for participation or non-participation– Desired incentives

Other

None of the above

IU Umbrella

CycleFit or Circuit Strength Punch Card

$5 Bloomington Bagel Company coins

$10 Scholar's Inn Bakehouse gift card

IU T-shirt

Tudor Room

$10 TIS gift card

Mother Bear's Large one-topping pizza

IU Sweatshirt

iPod Touch

Apple iPad2

6%

19%

20%

21%

26%

29%

32%

37%

38%

42%

43%

60%

72%

WHICH AFFECTED YOUR DECISION TO COMPLETE THE SURVEY?

Incentives in 2012

• Two stage incentives– All completers• Scoop of ice cream (coupon sent electronically)• Free bagel coupon (picked-up at library)

– All completers• Entered into a weekly drawing to win one of 115 prizes• Earlier students complete survey, more opportunities to

win

Drawing Incentives in 2012Prizes Total Number Prizes Total Number

iPad 32 GB 1$10 Cycle Fit 3 session punch cards 5

$10 Target Gift Card 6Circuit Strength 3 session package 5

Crimson Hooded sweatshirt w/ white IU Trident 13 $10 gift card at BBC 2Mother Bear's large one-topping pizza 150 $25 gift card TIS 4

Lunch for Two at the Tudor Room 12Jackets from Underground Printing 6

Crimson foldable umbrella w/White IU Trident 16 Underground T-shirts 300

TIS $10 gift card 47 $50 Visa gift card 1Scholar's Inn Bakehouse ($10 gift card) 55 IU Flag (5x8) 1Crimson T-shirt w/ white IU Trident 67 IU Soccer Scarf 1

Track & Field Baseball Cap 1 IU Athletic Team T-shirts 4

SRSC T-shirts 10 TOTAL PRIZES 707

2012 Response Rate

• Changes– Response rate decreased to 30%– Bus with NSSE facts– List of past winners on website (views 1489 time)– List of prizes embedded in invitation and follow-up

emails (viewed 1495 times)• Speculation on why– Later start date– Competing surveys

Lessons Learned

• Sample members may not accurately predict the utility of incentives

• Managing a complex set of incentives requires resources

• Respondents may attribute more of their decision to participate to the incentives when asked later

Conclusions

• Incentives can improve response rates for Internet surveys

• The effect of incentives on data quality and differences across groups needs much more research

• More research is needed on why and how incentives affect the decision to participate

Contact Information

John Kennedy kennedyj@indiana.edu

Judy Ouimet ouimet@indiana.edu