the effects of pushing web in a mixed-mode establishment ... › ... › 2013 ›...
TRANSCRIPT
RTI International
RTI International is a trade name of Research Triangle Institute. www.rti.org
The Effects of Pushing Web in a Mixed-modeEstablishment Data Collection
American Association for Public Opinion Research68th Annual Conference
May 16, 2013
Chris Ellis*, Kim Aspinwall, Todd Heinrich, Scott Ginder, Hope Smiley McDonaldRTI International
Margaret NoonanBureau of Justice Statistics
1
RTI International
Overview of Presentation
Acknowledgements Overview of Deaths in Custody Reporting Program
(DCRP) Genesis of DCRP Web Push Experiment The Problem Research Questions Experiment Methodology Results Limitations Conclusions Areas for Further Research
2
RTI International
Acknowledgements
Bureau of Justice Statistics (BJS)– Daniela Golinelli, Chief Corrections Statistics Program– Margaret Noonan, Program Manager for DCRP
RTI International– Kim Aspinwall, DCRP Data Collection Task Leader– Todd Heinrich, Systems and Programming Task Leader– Scott Ginder, DCRP Analysis and Reporting Task Leader– Hope Smiley McDonald, DCRP Jail Universe Task Leader
The views expressed in this presentation do not necessarily reflect the official policies of the U.S.Department of Justice or the Bureau of Justice Statistics; nor does mention of trade names,commercial practices, or organizations imply endorsement by the U.S. Government.
This presentation is sponsored by RTI International’s Survey Research Division.
3
RTI International
Deaths in Custody Reporting Program Primary source of mortality statistics within the American
correctional system– 50 state departments of correction (DOCs)– Approximately 3,000 local jail jurisdictions
Multimode data collection– Respondents are typically prison and jail administrators– Self-reporting– Two forms: individual death reports and post-hoc annual
summary– Web, paper, fax, e-mail, bulk data file, and (during NRFU)
telephone
4
RTI International
Deaths in Custody Reporting Program (cont.)
“Dillman-esque” data collection protocol implementedthroughout post-reference year reporting period– Initial lead mailing (including reporting/Web login instructions)– Thank you/reminder e-mail or postcard– Replacement forms mailing– Nonresponse telephone prompts– Data quality follow-up calls for critical missing/conflicting items
2011 response rates (AAPOR RR2):– 100% for 50 state DOCs– 96.7% for “Top 150” jail jurisdictions– 96.8% for remaining jail jurisdictions
[RR2 % = (I+P) / (I+P) + (R+NC+O) + (UH+UO)]
5
RTI International
Genesis of DCRP Web Push Experiment
Study-specific insights– Using concurrent multimode approach, DCRP respondents used
the Web for their primary response (Heinrich et al., 2012)
68.8% in 2009 69.4% in 2010
– An even greater number of respondents indicated willingness torespond via Web
– Over 90% of agencies had one or more e-mail addresses on file
Additional factors– Data quality objectives would be aided by increased Web uptake– The reduction of paper form submissions would likely decrease
data collection costs
6
RTI International
The Problem
Study-specific concerns– DCRP study population has known paper-only submitters– Further, some remote jails are known to not have computers/IT– Historically, DCRP has always offered paper forms
Other concerns from the literature– Giving respondents “buffet-like” choice of mode can lead to
(Medway & Fulton, 2012) Perceived increased complexity, leading to Lower response rates
– “Pushing” one mode at the onset of datacollection may (Mooney et al., 2012) Increase respondents’ selection of mode, but Lower overall response rates
7
RTI International
Research Questions BJS and RTI embedded an experiment in the 2011 data
collection cycle (which occurred in 2012 following the CY)– Treatment: withhold paper forms from the initial survey request
(i.e., “push” the Web)– Control: continue to offer paper forms concurrent with other mode
invitations (e.g., Web, bulk data file)
The research questions:1. Do overall response rates vary when the Web mode is pushed?2. Does time-to-response (TTR) vary when the Web mode is
pushed?3. What are the cost implications when the Web-push method
(and its outcomes) are applied to the entire sample?4. Does pushing the Web have an impact on mode self-selection?
8
RTI International
Experiment Methodology Treatment affected contents of initial January mailing
Cohort 2010 Mode Treatment or Control Sample Size
1 Paper Control (Paper Forms) 337
2 Paper Treatment (No Paper) 338
3 Web Control (Paper Forms) 887
4 Web Treatment (No Paper) 888
5 Mixed Control (Paper Forms) 118
6 Mixed Treatment (No Paper) 119
Total 2,687• Randomization controlled for 2010 response mode and speed of response• DOCs, 2010 nonresponders, and special situations were excluded
9
RTI International
Experiment Methodology (cont.) The treatment cohort received
– A DOJ-signed cover letter, with Web logincredentials
– DCRP informational handout– 2011 reporting instructions– 2012 reporting instructions– Enclosed in a 10 x 13 outer mailing envelope
The control cohort received– All of the above, but– A 2011-specific inner envelope with those instructions,
a business reply envelope, and 2011 paper forms– A 2012-specific inner envelope with those instructions,
a business reply envelope, and a 2012 paper form
All mailings were simultaneous to measure TTR10
RTI International
Results* – Response Rates
* Analysis included 2,653 of the original 2,687 cases
78% 73%
10%14%
10%
98%
11%
98%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Paper Included(PR group)
Paper Excluded(WP group)
Res
pons
e R
ate
Cohort
Response Rates by Treatment Cohort
Gain BetweenNonresponse Calling andEnd of Data Collection
Gain BetweenReplacement FormsMailing and NonresponseFollow-up
Gain Between Initial andReplacement FormsMailing
11
RTI International
Results* – Time-to-Response (TTR)
The treatment (Web push) cohort responded 1.6days faster than the control cohort on average
That said, the difference was only marginallystatistically significant (p = 0.07)
Withholding paper certainly did not negatively impactTTR
* Analysis included 2,653 of the original 2,687 cases
2011 TTR (Days) 2010 TTR (Days)Cohort Mean Median Mean MedianPaper
Included19.0 10.0 36.5 20.5
PaperExcluded
17.4 8.0 35.4 19.0
12
RTI International
Results* – Mode Selection
224 (22%)
20 (2%)
777 (75%)
921 (95%)
31 (3%) 32 (3%)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Forms included Forms excluded
Perc
ent
Treatment Cohort
Response Mode, by Treatment Cohort
2011 mode duringexperiment period Other(mixed, phone, etc.)
2011 mode duringexperiment period Web
2011 mode duringexperiment period Paper
* Analysis included 2,653 of the original 2,687 cases13
RTI International
Results* – Mode Selection (cont.)
Not surprisingly, clear correlation between treatmentcohort and respondent mode selection– Inclusion of paper forms increased the incidence of paper mode
response by 20 percentage points– Withholding paper resulted in 95% of respondents choosing Web– An equal amount responded using mixed or multiple modes (3%)
Paper inclusion led to higher selection of Web responseacross both cohorts; upon receipt of the replacementforms mailing– 30.8% of Web push (treatment) members responded via paper– 33.3% of Paper (control) members responded via paper
The propensity for Web response changed* Analysis included 2,653 of the original 2,687 cases
14
RTI International
Results* – Costs of Data Collection
To compute the impact on costs:– The performance of each cohort was measured according to key
data collection components,– Noting how many agencies required which types of follow-up,
plus– How many required paper forms processing
* Analysis included 2,653 of the original 2,687 cases
CohortInitial
MailoutReplacement
Forms Mailout
NonresponseTelephone
ContactRespondents
via Paper
PaperIncluded
2,942 642 351 633
PaperExcluded
2,942 795 385 212
15
RTI International
Results* – Costs of Data Collection (cont.)
$10,768
$19,182
$4,023
$3,249
$6,988
$6,371
$1,105
$3,298
$22,883
$32,099
$0 $10,000 $20,000 $30,000
PaperExcluded (WP
Group)
Paper Included(PR Group)
Estimated Cost
Trea
tmen
t
Estimated Component and Total Costs, by TreatmentCohort
Initial MailingCosts
ReplacementForms MailingCosts
NonresponsePhone ContactCosts
PaperResponse ModeCosts
16 * Analysis included 2,653 of the original 2,687 cases
RTI International
Results* – Costs of Data Collection (cont.)
The Web push (treatment) cohort incurred– Fewer initial mailing costs (no printing, less postage and
handling)– Slightly more replacement forms mailing costs– Almost identical nonresponse telephone contacting costs, and– Fewer paper processing costs (receipt and processing, data
entry)
Costs of the two cohorts were computed, as if applied tothe entire study
The Web push (treatment) approach wouldeffect a savings of $9,200 (~$3.47 per case)
* Analysis included 2,653 of the original 2,687 cases
17
RTI International
Limitations Some limitations associated with experiment; the DCRP
study– Is a time series collection that is well known to the field– Was a mandatory collection until 2006– Typically enjoys high response rates– Employs a robust nonresponse prompting protocol– Allows for a multi-month response period
18
RTI International
Conclusions1. Web can be pushed at the onset without jeopardizing
overall response rates2. Implementing this change does not adversely affect
TTR3. Withholding paper forms can lead to cost savings,
despite some interim cohort increases4. Pushing the Web mode clearly influences respondents’
selection5. Including paper as an initial option led to higher rates of
paper being used to respond
19
RTI International
Areas for Further Research What are the effects of withholding paper from later-
stage promptings, too? What are the effects of not explicitly offering paper, but
instead providing it only upon request? Can the results of this experiment be replicated across
other establishment surveys?
20
RTI International
Referencesde Leeuw, E. D. (2005). To Mix or Not to Mix Data Collection Modes in Surveys. The
Journal of Official Statistics, 21(2), 233-255.
Dillman, D. A. (2007 updated). Mail and Internet surveys: The tailored design method(2nd ed.), Hoboken, NJ: John Wiley Co.
Heinrich, T. D., Ellis, C. S., Ham, M. W., Ginder, S. A., Smiley McDonald, H. M.,Aspinwall, K. R., & Noonan, M. (2012, June). Have It Your Way: Managing Dataand Business Preferences in a Multimode Collection. Presented at FourthInternational Conference on Establishment Surveys, Montreal, Quebec, Canada.
Medway, R. L., & Fulton, J. (2012). When more gets you less: a meta-analysis of theeffect of concurrent web options on mail survey response rates. Public OpinionQuarterly, 76(4), 733-746.
Messer, B. L. (2012). Mixed-Mode & Internet Surveys: Lessons from AAPOR 2012.Presented at the Pacific Association for Public Opinion Research Mini-Conference, San Francisco, CA June 22, 2012.
21
RTI International
References (cont.)Minton, T.D. (2011). Jail Inmates at Midyear 2010 – Statistical Tables. Washington,
DC: U.S. Department of Justice, Office of Justice Programs, Bureau of JusticeStatistics, Washington DC. Available athttp://www.bjs.gov/index.cfm?ty=pbdetail&iid=2375.
Mooney, G., Lan, F., Lin, X., & Hurwitz, A. (2012). Influencing Mode Choice in aMixed Mode Survey. Presented at the American Association for Public OpinionResearch 67th Annual Conference. Orlando, FL. May 2012.
Noonan, M. E. (2012). Prison and jail deaths in custody, 2000-2010 – Statisticaltables. Bureau of Justice Statistics, Washington DC. Available athttp://bjs.gov/index.cfm?ty=pbdetail&iid=4558.
Schaefer, D. R., & Dillman, D. A. (1998). Development of a standard e-mailmethodology: Results of an experiment. Public Opinion Quarterly, 62(3), 378-397.
22
RTI International
More Information
Chris EllisSenior Survey DirectorRTI [email protected]
Presentation available at:www.rti.org/aapor
23
SurveyPostblogs.rti.org/surveypost
@SurveyPost