the effects of pushing web in a mixed-mode establishment ... › ... › 2013 ›...

23
RTI International RTI International is a trade name of Research Triangle Institute. www.rti.org The Effects of Pushing Web in a Mixed-mode Establishment Data Collection American Association for Public Opinion Research 68th Annual Conference May 16, 2013 Chris Ellis*, Kim Aspinwall, Todd Heinrich, Scott Ginder, Hope Smiley McDonald RTI International Margaret Noonan Bureau of Justice Statistics 1

Upload: others

Post on 06-Jul-2020

0 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: The Effects of Pushing Web in a Mixed-mode Establishment ... › ... › 2013 › Session_A-2-1-Ellis.pdf · Paper inclusion led to higher selection of Web response across both cohorts;

RTI International

RTI International is a trade name of Research Triangle Institute. www.rti.org

The Effects of Pushing Web in a Mixed-modeEstablishment Data Collection

American Association for Public Opinion Research68th Annual Conference

May 16, 2013

Chris Ellis*, Kim Aspinwall, Todd Heinrich, Scott Ginder, Hope Smiley McDonaldRTI International

Margaret NoonanBureau of Justice Statistics

1

Page 2: The Effects of Pushing Web in a Mixed-mode Establishment ... › ... › 2013 › Session_A-2-1-Ellis.pdf · Paper inclusion led to higher selection of Web response across both cohorts;

RTI International

Overview of Presentation

Acknowledgements Overview of Deaths in Custody Reporting Program

(DCRP) Genesis of DCRP Web Push Experiment The Problem Research Questions Experiment Methodology Results Limitations Conclusions Areas for Further Research

2

Page 3: The Effects of Pushing Web in a Mixed-mode Establishment ... › ... › 2013 › Session_A-2-1-Ellis.pdf · Paper inclusion led to higher selection of Web response across both cohorts;

RTI International

Acknowledgements

Bureau of Justice Statistics (BJS)– Daniela Golinelli, Chief Corrections Statistics Program– Margaret Noonan, Program Manager for DCRP

RTI International– Kim Aspinwall, DCRP Data Collection Task Leader– Todd Heinrich, Systems and Programming Task Leader– Scott Ginder, DCRP Analysis and Reporting Task Leader– Hope Smiley McDonald, DCRP Jail Universe Task Leader

The views expressed in this presentation do not necessarily reflect the official policies of the U.S.Department of Justice or the Bureau of Justice Statistics; nor does mention of trade names,commercial practices, or organizations imply endorsement by the U.S. Government.

This presentation is sponsored by RTI International’s Survey Research Division.

3

Page 4: The Effects of Pushing Web in a Mixed-mode Establishment ... › ... › 2013 › Session_A-2-1-Ellis.pdf · Paper inclusion led to higher selection of Web response across both cohorts;

RTI International

Deaths in Custody Reporting Program Primary source of mortality statistics within the American

correctional system– 50 state departments of correction (DOCs)– Approximately 3,000 local jail jurisdictions

Multimode data collection– Respondents are typically prison and jail administrators– Self-reporting– Two forms: individual death reports and post-hoc annual

summary– Web, paper, fax, e-mail, bulk data file, and (during NRFU)

telephone

4

Page 5: The Effects of Pushing Web in a Mixed-mode Establishment ... › ... › 2013 › Session_A-2-1-Ellis.pdf · Paper inclusion led to higher selection of Web response across both cohorts;

RTI International

Deaths in Custody Reporting Program (cont.)

“Dillman-esque” data collection protocol implementedthroughout post-reference year reporting period– Initial lead mailing (including reporting/Web login instructions)– Thank you/reminder e-mail or postcard– Replacement forms mailing– Nonresponse telephone prompts– Data quality follow-up calls for critical missing/conflicting items

2011 response rates (AAPOR RR2):– 100% for 50 state DOCs– 96.7% for “Top 150” jail jurisdictions– 96.8% for remaining jail jurisdictions

[RR2 % = (I+P) / (I+P) + (R+NC+O) + (UH+UO)]

5

Page 6: The Effects of Pushing Web in a Mixed-mode Establishment ... › ... › 2013 › Session_A-2-1-Ellis.pdf · Paper inclusion led to higher selection of Web response across both cohorts;

RTI International

Genesis of DCRP Web Push Experiment

Study-specific insights– Using concurrent multimode approach, DCRP respondents used

the Web for their primary response (Heinrich et al., 2012)

68.8% in 2009 69.4% in 2010

– An even greater number of respondents indicated willingness torespond via Web

– Over 90% of agencies had one or more e-mail addresses on file

Additional factors– Data quality objectives would be aided by increased Web uptake– The reduction of paper form submissions would likely decrease

data collection costs

6

Page 7: The Effects of Pushing Web in a Mixed-mode Establishment ... › ... › 2013 › Session_A-2-1-Ellis.pdf · Paper inclusion led to higher selection of Web response across both cohorts;

RTI International

The Problem

Study-specific concerns– DCRP study population has known paper-only submitters– Further, some remote jails are known to not have computers/IT– Historically, DCRP has always offered paper forms

Other concerns from the literature– Giving respondents “buffet-like” choice of mode can lead to

(Medway & Fulton, 2012) Perceived increased complexity, leading to Lower response rates

– “Pushing” one mode at the onset of datacollection may (Mooney et al., 2012) Increase respondents’ selection of mode, but Lower overall response rates

7

Page 8: The Effects of Pushing Web in a Mixed-mode Establishment ... › ... › 2013 › Session_A-2-1-Ellis.pdf · Paper inclusion led to higher selection of Web response across both cohorts;

RTI International

Research Questions BJS and RTI embedded an experiment in the 2011 data

collection cycle (which occurred in 2012 following the CY)– Treatment: withhold paper forms from the initial survey request

(i.e., “push” the Web)– Control: continue to offer paper forms concurrent with other mode

invitations (e.g., Web, bulk data file)

The research questions:1. Do overall response rates vary when the Web mode is pushed?2. Does time-to-response (TTR) vary when the Web mode is

pushed?3. What are the cost implications when the Web-push method

(and its outcomes) are applied to the entire sample?4. Does pushing the Web have an impact on mode self-selection?

8

Page 9: The Effects of Pushing Web in a Mixed-mode Establishment ... › ... › 2013 › Session_A-2-1-Ellis.pdf · Paper inclusion led to higher selection of Web response across both cohorts;

RTI International

Experiment Methodology Treatment affected contents of initial January mailing

Cohort 2010 Mode Treatment or Control Sample Size

1 Paper Control (Paper Forms) 337

2 Paper Treatment (No Paper) 338

3 Web Control (Paper Forms) 887

4 Web Treatment (No Paper) 888

5 Mixed Control (Paper Forms) 118

6 Mixed Treatment (No Paper) 119

Total 2,687• Randomization controlled for 2010 response mode and speed of response• DOCs, 2010 nonresponders, and special situations were excluded

9

Page 10: The Effects of Pushing Web in a Mixed-mode Establishment ... › ... › 2013 › Session_A-2-1-Ellis.pdf · Paper inclusion led to higher selection of Web response across both cohorts;

RTI International

Experiment Methodology (cont.) The treatment cohort received

– A DOJ-signed cover letter, with Web logincredentials

– DCRP informational handout– 2011 reporting instructions– 2012 reporting instructions– Enclosed in a 10 x 13 outer mailing envelope

The control cohort received– All of the above, but– A 2011-specific inner envelope with those instructions,

a business reply envelope, and 2011 paper forms– A 2012-specific inner envelope with those instructions,

a business reply envelope, and a 2012 paper form

All mailings were simultaneous to measure TTR10

Page 11: The Effects of Pushing Web in a Mixed-mode Establishment ... › ... › 2013 › Session_A-2-1-Ellis.pdf · Paper inclusion led to higher selection of Web response across both cohorts;

RTI International

Results* – Response Rates

* Analysis included 2,653 of the original 2,687 cases

78% 73%

10%14%

10%

98%

11%

98%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Paper Included(PR group)

Paper Excluded(WP group)

Res

pons

e R

ate

Cohort

Response Rates by Treatment Cohort

Gain BetweenNonresponse Calling andEnd of Data Collection

Gain BetweenReplacement FormsMailing and NonresponseFollow-up

Gain Between Initial andReplacement FormsMailing

11

Page 12: The Effects of Pushing Web in a Mixed-mode Establishment ... › ... › 2013 › Session_A-2-1-Ellis.pdf · Paper inclusion led to higher selection of Web response across both cohorts;

RTI International

Results* – Time-to-Response (TTR)

The treatment (Web push) cohort responded 1.6days faster than the control cohort on average

That said, the difference was only marginallystatistically significant (p = 0.07)

Withholding paper certainly did not negatively impactTTR

* Analysis included 2,653 of the original 2,687 cases

2011 TTR (Days) 2010 TTR (Days)Cohort Mean Median Mean MedianPaper

Included19.0 10.0 36.5 20.5

PaperExcluded

17.4 8.0 35.4 19.0

12

Page 13: The Effects of Pushing Web in a Mixed-mode Establishment ... › ... › 2013 › Session_A-2-1-Ellis.pdf · Paper inclusion led to higher selection of Web response across both cohorts;

RTI International

Results* – Mode Selection

224 (22%)

20 (2%)

777 (75%)

921 (95%)

31 (3%) 32 (3%)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Forms included Forms excluded

Perc

ent

Treatment Cohort

Response Mode, by Treatment Cohort

2011 mode duringexperiment period Other(mixed, phone, etc.)

2011 mode duringexperiment period Web

2011 mode duringexperiment period Paper

* Analysis included 2,653 of the original 2,687 cases13

Page 14: The Effects of Pushing Web in a Mixed-mode Establishment ... › ... › 2013 › Session_A-2-1-Ellis.pdf · Paper inclusion led to higher selection of Web response across both cohorts;

RTI International

Results* – Mode Selection (cont.)

Not surprisingly, clear correlation between treatmentcohort and respondent mode selection– Inclusion of paper forms increased the incidence of paper mode

response by 20 percentage points– Withholding paper resulted in 95% of respondents choosing Web– An equal amount responded using mixed or multiple modes (3%)

Paper inclusion led to higher selection of Web responseacross both cohorts; upon receipt of the replacementforms mailing– 30.8% of Web push (treatment) members responded via paper– 33.3% of Paper (control) members responded via paper

The propensity for Web response changed* Analysis included 2,653 of the original 2,687 cases

14

Page 15: The Effects of Pushing Web in a Mixed-mode Establishment ... › ... › 2013 › Session_A-2-1-Ellis.pdf · Paper inclusion led to higher selection of Web response across both cohorts;

RTI International

Results* – Costs of Data Collection

To compute the impact on costs:– The performance of each cohort was measured according to key

data collection components,– Noting how many agencies required which types of follow-up,

plus– How many required paper forms processing

* Analysis included 2,653 of the original 2,687 cases

CohortInitial

MailoutReplacement

Forms Mailout

NonresponseTelephone

ContactRespondents

via Paper

PaperIncluded

2,942 642 351 633

PaperExcluded

2,942 795 385 212

15

Page 16: The Effects of Pushing Web in a Mixed-mode Establishment ... › ... › 2013 › Session_A-2-1-Ellis.pdf · Paper inclusion led to higher selection of Web response across both cohorts;

RTI International

Results* – Costs of Data Collection (cont.)

$10,768

$19,182

$4,023

$3,249

$6,988

$6,371

$1,105

$3,298

$22,883

$32,099

$0 $10,000 $20,000 $30,000

PaperExcluded (WP

Group)

Paper Included(PR Group)

Estimated Cost

Trea

tmen

t

Estimated Component and Total Costs, by TreatmentCohort

Initial MailingCosts

ReplacementForms MailingCosts

NonresponsePhone ContactCosts

PaperResponse ModeCosts

16 * Analysis included 2,653 of the original 2,687 cases

Page 17: The Effects of Pushing Web in a Mixed-mode Establishment ... › ... › 2013 › Session_A-2-1-Ellis.pdf · Paper inclusion led to higher selection of Web response across both cohorts;

RTI International

Results* – Costs of Data Collection (cont.)

The Web push (treatment) cohort incurred– Fewer initial mailing costs (no printing, less postage and

handling)– Slightly more replacement forms mailing costs– Almost identical nonresponse telephone contacting costs, and– Fewer paper processing costs (receipt and processing, data

entry)

Costs of the two cohorts were computed, as if applied tothe entire study

The Web push (treatment) approach wouldeffect a savings of $9,200 (~$3.47 per case)

* Analysis included 2,653 of the original 2,687 cases

17

Page 18: The Effects of Pushing Web in a Mixed-mode Establishment ... › ... › 2013 › Session_A-2-1-Ellis.pdf · Paper inclusion led to higher selection of Web response across both cohorts;

RTI International

Limitations Some limitations associated with experiment; the DCRP

study– Is a time series collection that is well known to the field– Was a mandatory collection until 2006– Typically enjoys high response rates– Employs a robust nonresponse prompting protocol– Allows for a multi-month response period

18

Page 19: The Effects of Pushing Web in a Mixed-mode Establishment ... › ... › 2013 › Session_A-2-1-Ellis.pdf · Paper inclusion led to higher selection of Web response across both cohorts;

RTI International

Conclusions1. Web can be pushed at the onset without jeopardizing

overall response rates2. Implementing this change does not adversely affect

TTR3. Withholding paper forms can lead to cost savings,

despite some interim cohort increases4. Pushing the Web mode clearly influences respondents’

selection5. Including paper as an initial option led to higher rates of

paper being used to respond

19

Page 20: The Effects of Pushing Web in a Mixed-mode Establishment ... › ... › 2013 › Session_A-2-1-Ellis.pdf · Paper inclusion led to higher selection of Web response across both cohorts;

RTI International

Areas for Further Research What are the effects of withholding paper from later-

stage promptings, too? What are the effects of not explicitly offering paper, but

instead providing it only upon request? Can the results of this experiment be replicated across

other establishment surveys?

20

Page 21: The Effects of Pushing Web in a Mixed-mode Establishment ... › ... › 2013 › Session_A-2-1-Ellis.pdf · Paper inclusion led to higher selection of Web response across both cohorts;

RTI International

Referencesde Leeuw, E. D. (2005). To Mix or Not to Mix Data Collection Modes in Surveys. The

Journal of Official Statistics, 21(2), 233-255.

Dillman, D. A. (2007 updated). Mail and Internet surveys: The tailored design method(2nd ed.), Hoboken, NJ: John Wiley Co.

Heinrich, T. D., Ellis, C. S., Ham, M. W., Ginder, S. A., Smiley McDonald, H. M.,Aspinwall, K. R., & Noonan, M. (2012, June). Have It Your Way: Managing Dataand Business Preferences in a Multimode Collection. Presented at FourthInternational Conference on Establishment Surveys, Montreal, Quebec, Canada.

Medway, R. L., & Fulton, J. (2012). When more gets you less: a meta-analysis of theeffect of concurrent web options on mail survey response rates. Public OpinionQuarterly, 76(4), 733-746.

Messer, B. L. (2012). Mixed-Mode & Internet Surveys: Lessons from AAPOR 2012.Presented at the Pacific Association for Public Opinion Research Mini-Conference, San Francisco, CA June 22, 2012.

21

Page 22: The Effects of Pushing Web in a Mixed-mode Establishment ... › ... › 2013 › Session_A-2-1-Ellis.pdf · Paper inclusion led to higher selection of Web response across both cohorts;

RTI International

References (cont.)Minton, T.D. (2011). Jail Inmates at Midyear 2010 – Statistical Tables. Washington,

DC: U.S. Department of Justice, Office of Justice Programs, Bureau of JusticeStatistics, Washington DC. Available athttp://www.bjs.gov/index.cfm?ty=pbdetail&iid=2375.

Mooney, G., Lan, F., Lin, X., & Hurwitz, A. (2012). Influencing Mode Choice in aMixed Mode Survey. Presented at the American Association for Public OpinionResearch 67th Annual Conference. Orlando, FL. May 2012.

Noonan, M. E. (2012). Prison and jail deaths in custody, 2000-2010 – Statisticaltables. Bureau of Justice Statistics, Washington DC. Available athttp://bjs.gov/index.cfm?ty=pbdetail&iid=4558.

Schaefer, D. R., & Dillman, D. A. (1998). Development of a standard e-mailmethodology: Results of an experiment. Public Opinion Quarterly, 62(3), 378-397.

22

Page 23: The Effects of Pushing Web in a Mixed-mode Establishment ... › ... › 2013 › Session_A-2-1-Ellis.pdf · Paper inclusion led to higher selection of Web response across both cohorts;

RTI International

More Information

Chris EllisSenior Survey DirectorRTI [email protected]

Presentation available at:www.rti.org/aapor

23

SurveyPostblogs.rti.org/surveypost

@SurveyPost