increasing survey cooperation: motivating chronic late responders to an annual survey national...
TRANSCRIPT
![Page 1: Increasing Survey Cooperation: Motivating Chronic Late Responders to an Annual Survey National Science Foundation Division of Science Resources Statistics](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514bcb955034693478b4584/html5/thumbnails/1.jpg)
Increasing Survey Cooperation: Motivating Chronic Late Responders to
an Annual Survey
National Science Foundation
Division of Science Resources Statistics
Ronda Britt and Fran Featherston
ICES IIIJune 19, 2007
National Science FoundationDivision of Science Resources Statistics
www.nsf.gov/statistics
![Page 2: Increasing Survey Cooperation: Motivating Chronic Late Responders to an Annual Survey National Science Foundation Division of Science Resources Statistics](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514bcb955034693478b4584/html5/thumbnails/2.jpg)
2
NSF Academic R&D Survey
National Science FoundationDivision of Science Resources Statistics
• Voluntary survey of 700 research & development - performing universities and colleges
• Conducted annually since 1972
• Web only since 2001
• Requests expenditures for R&D performed during previous year
• Survey data based on respondent’s fiscal year
![Page 3: Increasing Survey Cooperation: Motivating Chronic Late Responders to an Annual Survey National Science Foundation Division of Science Resources Statistics](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514bcb955034693478b4584/html5/thumbnails/3.jpg)
3
Historical Response Pattern
National Science FoundationDivision of Science Resources Statistics
•Early November email launches survey & sets January 31 deadline
•50-60% respond by deadline, despite multiple reminders
•Several months after deadline to reach response rate of ~95%
•Respondents were allowed to extend deadline, creating cycle of late response
![Page 4: Increasing Survey Cooperation: Motivating Chronic Late Responders to an Annual Survey National Science Foundation Division of Science Resources Statistics](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514bcb955034693478b4584/html5/thumbnails/4.jpg)
4
Goals of Experiments FY 2004 - FY 2005
National Science FoundationDivision of Science Resources Statistics
•Increase response rate at survey deadline
-Speed up first interaction with web survey
-Increase number of reminders
-Vary mode of reminders
•Decrease number of weeks to reach 95% response rate
![Page 5: Increasing Survey Cooperation: Motivating Chronic Late Responders to an Annual Survey National Science Foundation Division of Science Resources Statistics](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514bcb955034693478b4584/html5/thumbnails/5.jpg)
5
Experiment 1- FY 2004Increase Response Rate at Survey Deadline, Part 1
National Science FoundationDivision of Science Resources Statistics
• Hypothesis: Respondents who visit website soon after launch (acknowledge survey) are more likely to respond by the survey deadline
•3 study groups received varying modes of reminders:
Group 1 received one e-mail reminder in December (control group)
Group 2 received two e-mail reminders in December
Group 3 received 2 phone calls in December
•All groups received final e-mail in January requesting acknowledgement
![Page 6: Increasing Survey Cooperation: Motivating Chronic Late Responders to an Annual Survey National Science Foundation Division of Science Resources Statistics](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514bcb955034693478b4584/html5/thumbnails/6.jpg)
6
Experiment 1 Design
National Science FoundationDivision of Science Resources Statistics
Key Dates
Study Groups
Group 1(control)
Group 2 Group 3
November 23 Survey Launch
December 14Acknowledgement
e-mailAcknowledgement
phone calls
December 30Acknowledgement
e-mailAcknowledgement
e-mailAcknowledgement
phone calls
January 11 Acknowledgement e-mail
January 31 Survey Due Date
![Page 7: Increasing Survey Cooperation: Motivating Chronic Late Responders to an Annual Survey National Science Foundation Division of Science Resources Statistics](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514bcb955034693478b4584/html5/thumbnails/7.jpg)
7
Experiment 1 Results
National Science FoundationDivision of Science Resources Statistics
•You can lead a respondent to the website, but you can’t encourage an earlier survey completion
Multiple reminders to acknowledge succeeded in decreasing elapsed days until acknowledgement
Multiple reminders to acknowledge had no effect on decreasing elapsed days to survey completion
![Page 8: Increasing Survey Cooperation: Motivating Chronic Late Responders to an Annual Survey National Science Foundation Division of Science Resources Statistics](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514bcb955034693478b4584/html5/thumbnails/8.jpg)
8
Experiment 1 Results
National Science FoundationDivision of Science Resources Statistics
ControlGroup 1
E-mail reminders
Group 2
Phone Reminders
Group 3All
Groups
Mean days from survey launch to acknowledgement
37 days
(n=177)
29 days
(n=179)
30 days
(n=179)
32 days
(n=535)
Response rate at survey deadline
46.3% 49.7% 49.2% 48.4%
![Page 9: Increasing Survey Cooperation: Motivating Chronic Late Responders to an Annual Survey National Science Foundation Division of Science Resources Statistics](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514bcb955034693478b4584/html5/thumbnails/9.jpg)
9
Experiment 2 – FY 2004 Increase Response Rate at Survey Deadline, Part 2
National Science FoundationDivision of Science Resources Statistics
• Hypotheses:
Respondents who receive more reminders are more likely to respond by the survey deadline
Respondents who receive reminders that vary in mode are more likely to respond by the survey deadline
•During month prior to survey deadline:
Group 1: One e-mail reminder (control group)
Group 2: One e-mail, one mail reminder
Group 3: Two e-mail reminders
![Page 10: Increasing Survey Cooperation: Motivating Chronic Late Responders to an Annual Survey National Science Foundation Division of Science Resources Statistics](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514bcb955034693478b4584/html5/thumbnails/10.jpg)
10
Experiment 2 Design
National Science FoundationDivision of Science Resources Statistics
Key DatesStudy Groups
Group 1 Group 2 Group 3
January 18-20(expecteddelivery)
ReminderMailing
January 19Reminder E-mail 1
ReminderE-mail 1
January 26 Reminder E-mail 2
January 31 Survey Due Date
![Page 11: Increasing Survey Cooperation: Motivating Chronic Late Responders to an Annual Survey National Science Foundation Division of Science Resources Statistics](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514bcb955034693478b4584/html5/thumbnails/11.jpg)
11
Experiment 2 Results
National Science FoundationDivision of Science Resources Statistics
•Variety and intensity will spice up the response rate.
Group who received both mail and e-mail responded at a higher rate by deadline than group receiving only e-mail
Experiment 4 (FY 2005) validated the effect of two contacts vs. one, but no significant difference between two e-mails vs. e-mail and mailing
![Page 12: Increasing Survey Cooperation: Motivating Chronic Late Responders to an Annual Survey National Science Foundation Division of Science Resources Statistics](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514bcb955034693478b4584/html5/thumbnails/12.jpg)
12
Experiment 2 Results – Single mode, 1 contact vs. mixed mode, 2 contacts
National Science FoundationDivision of Science Resources Statistics
Group 11 e-mail
Group 21 e-mail
and 1 mailing
Responded by deadline 35.4% 45.9%
51 62
Total 100% 100%
144 135
![Page 13: Increasing Survey Cooperation: Motivating Chronic Late Responders to an Annual Survey National Science Foundation Division of Science Resources Statistics](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514bcb955034693478b4584/html5/thumbnails/13.jpg)
13
Experiment 3 – FY 2004Increase Timeliness after Survey Deadline
National Science FoundationDivision of Science Resources Statistics
•Hypothesis: Respondents will respond sooner after the deadline if follow-up messages are individualized and less frequent than weekly
•During 24 weeks after survey deadline:
Group 1: Low tailoring – standard weekly messages (control group)
Group 2: Medium tailoring – less frequent messages with different content each time
Group 3: High tailoring –adjusted to individual respondent history (more frequent and tailored contacts for previous non-responders)
![Page 14: Increasing Survey Cooperation: Motivating Chronic Late Responders to an Annual Survey National Science Foundation Division of Science Resources Statistics](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514bcb955034693478b4584/html5/thumbnails/14.jpg)
14
Experiment 3 Design
National Science FoundationDivision of Science Resources Statistics
Follow-up Period
Group 1 Low Tailoring
Group 2Medium Tailoring
Group 3High Tailoring
Likely non-responders
All others
0-11 weeks
10 e-mail or phone contacts
Same messages
3 e-mails
Varied messages
Dependent on past behavior
Tailored messages
2 e-mails
Same messages
12-24 weeks
12 contacts; alternating e-mail or phone weekly
4 contacts spaced 3 to 4 weeks; e-mail, phone, letter, and final phone call
6 contacts; alternating e-mail and phone every two or three weeks
Same messages Varied messages and senders
Same messages
![Page 15: Increasing Survey Cooperation: Motivating Chronic Late Responders to an Annual Survey National Science Foundation Division of Science Resources Statistics](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514bcb955034693478b4584/html5/thumbnails/15.jpg)
15
Experiment 3 Results
National Science FoundationDivision of Science Resources Statistics
•It is more important to stay in contact than it is to use a particular contacting strategy
Little variation of final response rate across 3 groups
Weekly reminders were not found to perform better than tailored reminders sent every 3 to 4 weeks
![Page 16: Increasing Survey Cooperation: Motivating Chronic Late Responders to an Annual Survey National Science Foundation Division of Science Resources Statistics](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514bcb955034693478b4584/html5/thumbnails/16.jpg)
16
Experiment 3 Results
National Science FoundationDivision of Science Resources Statistics
Group 1Low
Tailoring
Group 2Medium Tailoring
Group 3High
TailoringAll
Groups
Number of respondents
105 95 102 302
Response rate on January 31
43.9% 49.2% 45.5% 46.2%
Final response rate 90.5% 94.7% 90.2% 91.7%
Response added between January 31 and August 10
50.8% 48.1% 49.2% 49.3%
![Page 17: Increasing Survey Cooperation: Motivating Chronic Late Responders to an Annual Survey National Science Foundation Division of Science Resources Statistics](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514bcb955034693478b4584/html5/thumbnails/17.jpg)
17
Changing Procedures
National Science FoundationDivision of Science Resources Statistics
•Used experimental results as starting point for making changes
•Consulted with Dr. Don Dillman on message crafting and changing survey procedures
•Determined four areas for change:
Acknowledgement process
Mode/frequency of contacts
Extension policy
Survey close-out policy (letters to institution presidents)
![Page 18: Increasing Survey Cooperation: Motivating Chronic Late Responders to an Annual Survey National Science Foundation Division of Science Resources Statistics](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514bcb955034693478b4584/html5/thumbnails/18.jpg)
18
Acknowledgement Process
National Science FoundationDivision of Science Resources Statistics
•Simplified acknowledgement process in FY 2006 — less burdensome for respondents
Prior to FY 2006, asked respondents to log-on to survey website to acknowledge continuation as survey respondent
Now, respondent replies to e-mail
Prior to FY 2006, 23% acknowledged in first week of survey, 72.5% month before due date
Now, increased to 63% and 98.7%, respectively
Results: Earlier identification of changes in respondents
![Page 19: Increasing Survey Cooperation: Motivating Chronic Late Responders to an Annual Survey National Science Foundation Division of Science Resources Statistics](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514bcb955034693478b4584/html5/thumbnails/19.jpg)
19
Mode/frequency of Respondent contacts
National Science FoundationDivision of Science Resources Statistics
•Changed mode and frequency, varied content in FY 2006
Pre-due date reminders increased in frequency: Early December – additional e-mail reminder January - 2 reminders (1 mail and 1 e-mail)
Post-due date follow-ups decreased in frequency: First 6 weeks - 3 e-mails (every 2 weeks) 7th week - phone call 8th week - letter to president’s office 9th week – phone call 11th week - final e-mail announcing survey closing date
Results: Increased response rate at deadline from 56% in FY 2005 to 68% in FY 2006, shortened post-deadline follow-up period
![Page 20: Increasing Survey Cooperation: Motivating Chronic Late Responders to an Annual Survey National Science Foundation Division of Science Resources Statistics](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514bcb955034693478b4584/html5/thumbnails/20.jpg)
20
Extension Policy
National Science FoundationDivision of Science Resources Statistics
•Eliminated unlimited extensions
Prior to FY 2005: Extension requests up to 36 weeks after survey due date
FY 2005 and FY 2006:
Requests for extensions restricted to shorter time frame (only granted for dates before President’s letter)
After president’s letter mail-out, respondents were told survey closes when target response rate reached and 2 week final warning would be given
Results: Shortened post-deadline follow-up period
![Page 21: Increasing Survey Cooperation: Motivating Chronic Late Responders to an Annual Survey National Science Foundation Division of Science Resources Statistics](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514bcb955034693478b4584/html5/thumbnails/21.jpg)
21
Survey Close-out Policy
National Science FoundationDivision of Science Resources Statistics
•President’s letter was added in FY 2004 as final call for response
Goal set to mail President’s letter to nonrespondents 1 month earlier each year
FY 2004 - May 25 (87% response rate at mailing)
FY 2005 - April 21 (88% response rate at mailing)
FY 2006 - March 23 (92% response rate at mailing)
Results: Earlier survey closeout on May 21, 2007 v. August 10, 2005
![Page 22: Increasing Survey Cooperation: Motivating Chronic Late Responders to an Annual Survey National Science Foundation Division of Science Resources Statistics](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514bcb955034693478b4584/html5/thumbnails/22.jpg)
22
Response Rate History, FY 1997- FY 2006
National Science FoundationDivision of Science Resources Statistics
0
5
10
15
20
25
30
35
40
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
Fiscal Year
Nu
mb
er
of
po
st-
du
e
da
te w
ee
ks
Weeks to reach final response rate
Weeks to reach 80%
Weeks to reach 88%
![Page 23: Increasing Survey Cooperation: Motivating Chronic Late Responders to an Annual Survey National Science Foundation Division of Science Resources Statistics](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514bcb955034693478b4584/html5/thumbnails/23.jpg)
23
Final Response Rates, FY 1997- FY 2006
National Science FoundationDivision of Science Resources Statistics
97.9 98.6 98.5 97.3 95.2 96.0 94.1 94.0 93.397.2
0
20
40
60
80
100
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
![Page 24: Increasing Survey Cooperation: Motivating Chronic Late Responders to an Annual Survey National Science Foundation Division of Science Resources Statistics](https://reader036.vdocuments.us/reader036/viewer/2022062511/5514bcb955034693478b4584/html5/thumbnails/24.jpg)
24
Questions?
For more information contact:Ronda Britt, Academic R&D Survey Manager
National Science FoundationDivision of Science Resources Statistics