the effects of time and study conduction on response ratesin ......burgard, t., bosnjak, m. &...

Burgard, Wedderhoff, Bosnjak| Response rates in psychological online surveys | GOR, 10.09.2020

The effects of time and studyconduction on response rates in

psychological online surveysA living meta-analysis

General Online Research, online| 10.09.2020

Tanja Burgard, Nadine Wedderhoff, Michael Bosnjak | ZPID – Leibniz Institute for Psychology Information

1


Nonresponse in psychological online surveys• Declining response rates in surveys in social and political sciences (Brick & Williams,

2013), as well as in psychology (Van Horn, Green, & Martinussen, 2009)

• Possible reasons: Oversurveying (Weiner & Dalessio, 2006), scarcity of attention• Growing popularity of online surveys in psychology yet, they suffer even more from nonresponse and representativeness issues(Daikeler, Bosnjak, & Lozar-Manfreda, 2019)

• Research questions: • Does the trend of declining response rates hold for psychological online

surveys? • What effects do study design characteristics have on response rates?

2


Extension of previous meta-analysis• Publication:

Burgard, T., Bosnjak, M. & Wedderhoff, N. (2020). Response rates in online surveys with affective disorder participants. A meta-analysis of study design and time effects between 2008 and 2019. Zeitschrift für Psychologie, 228(1), 14-24.

• Results: Mean RR 43 %, RR lower in more recent years and in case of longer questionnaires

• Limitations: Target population was defined rather narrow, which led to a lack of informationand thus, potentially relevant moderator variables (e.g. incentives) could not be examined

• Aims of the current study: • Build a broader database on response rates in psychological online surveys Greater generalizability, opportunity to test more moderator effects and conductsubgroup analyses

• Publication on the platform PsychOpen CAMA ( See poster presentation!)• Idea CAMA: Collaborative curation and augmentation of meta-analyses

3


Hypotheses

• H1: Response rates in psychological online surveys have decreased over time.Study conduction: Costs and benefits• H2: The use of incentives increases response rates in online surveys• H3: The more burdensome the survey is, the lower the response rate in

psychological online surveysStudy conduction: Contact protocols• H4: Personal, phone or mail contact for the invitation yields higher response

rates than e-Mail invitations• H5: The use of prenotification increases response rates in online surveys

4


Eligibility criteria

5

Inclusion ExclusionPopulation Potential participants in online surveys Organisations

Intervention Initial and explicit invitation to an online survey

Later waves of panel studies

Outcomes Participant flow or information needed to calculate a response rate

Insufficient information (e.g. number ofinvitations is not given)

Study type Studies reporting results of online surveys

Studies reporting on any survey type other than online surveysStudies reporting on mixed survey types that do not explicitly report on an online survey subgroup


Literature search and selectionSearch terms:((Online Survey or Web survey or

Internet survey or email survey or

electronic survey) and

(response rate or nonresponse rate))

Databases: • PsycInfo

• PubPsych

6

PRISMA Flow Chart


• Main outcome: Initial response rates (AAPOR, 2016): Number of usable questionnaires / number of potential respondents contacted

• Relevant moderators:

Coding and data extraction

Report Treatment Sample

• First author• Sponsorship of the

study• Publication type• Publication year

• Country of conduction• Year of data collection• Incentives (yes, no)• Pre-notification (yes, no)• Contact mode survey invitation• Number of reminders• Survey burden (minutes / items)

• Type of recruitment (e.g. list, register, conference, panel)

• Target population (e.g. students, health professionals)

• Mean age of the sample• Proportion of females

7

Burgard, Wedderhoff, Bosnjak| Response rates in psychological online surveys | GOR, 10.09.2020 8

Response rates and study characteristics


Overall effect and variance distribution

3-Level-RE-model• Mean RR: 40.5 % [CI: 38.0; 43.0]• Variance distribution:

58.8 % between samples (k=368), 41.2 % between reports (n=281)

• Test for heterogeneity:Q(df = 367) = 700713, p-val < .0001

9

Egger‘s test: z=4.1379Possible explanation: Higher RR in case of smaller samples


Publication year and study size

10


Multilevel mixed-effects models

11

Moderator Costs and benefits Contact protocols Full model

Intercept 0.386*** [0.348; 0.424];p<.001

0.450*** [0.393; 0.506]; p<.001

0.413*** [0.332; 0.492]; p<.001

H1: Publication year -0.015 [-0.051; 0.024]; p=0.490

H2: Incentives(Yes vs. No)

0.098** [0.025; 0.172];p=0.008

0.088* [0.005; 0.158]; p=0.036

H3: Number of items(Log)

-0.031 . [-0.064; -0.002];p=0.064

-0.034* [-0.067; -0.000]; p=0.048

H4: Invitation mode(E-Mail vs. Other)

-0.058* [-0.115; -0.000]; p=0.05

-0.033 [-0.109; 0.048]; p=0.448

H5: Prenotification(Yes vs. No)

0.009 [-0.053; 0.071];p=0.782

0.023 [-0.065; 0.100]; p=0.673

Pseudo R^2 (k) 0.062 (k=213) 0.061 (k=361) 0.101 (k=212)

?

11


Conclusions and outlook• Overall response rate for online surveys in psychology about 40 %• RR higher if incentives are given• RR lower for surveys with more items• It seems that the kind of invitation plays a role:

• More personal forms of contact are better

• Personal address of the respondent in the invitation is recommended

• Next steps: Finish coding, rerun analyses, publish data in PsychOpen CAMA continuous updating

12


Publication in PsychOpen CAMA

13

• PsychOpen CAMA will be a platform hosted by ZPID where …• … meta-analytic data will be stored and continuously updated and• … meta-analyses are reproducible and can be modified (e.g. variation of moderators)

• Different datasets are interoperable to make the same analyses run Template with pre-defined data structure and variable naming conventions Benefit: Use of information from previous similar meta-analyses

• Example: • Meta-analysis of Daikeler et al. (2019): Comparison of response rates of web versus other survey modes• Codings for the web mode available with similar variables (incentives, reminders, contactmode,…) Studies screened for eligibility in this meta-analysis and merged with new data

• Goal: Augment the database in PsychOpen CAMA with this meta-analysis


Literature• American Association for Public Opinion Research. (2016). Standard definitions: Final dispositions of case codes and outcome rates for surveys. 9th edition. AAPOR. Retrieved from: https://www.aapor.org/AAPOR_Main/media/publications/Standard-Definitions20169theditionfinal.pdf• Brick; Williams (2013): Explaining rising nonresponse rates in cross-sectional surveys. Annals of the American Academy ofpolitical and social science, 645.• Burgard, T., Wedderhoff, N., & Bosnjak, M. (2020). Response Rates in Online Surveys with Affective Disorder Participants. A Meta-Analysis of Study Design and Time Effects between 2008 and 2019. Hotspots in Psychology – 2020 Edition. Zeitschrift für Psychologie• Daikeler, J., Bosnjak, M., & Lozar Manfreda, K. (2019): Web Versus Other Survey Modes: An Updated and Extended Meta-Analysis Comparing Response Rates. Journal of Survey Statistics and Methodology, smz008. http://dx.doi.org/10.1093/jssam/smz008• Harrer, M. & Ebert, D. D. (2018). Doing Meta-Analysis in R: A practical Guide. PROTECT Lab Friedrich-Alexander University Erlangen-Nuremberg. https://bookdown.org/MathiasHarrer/Doing_Meta_Analysis_in_R/ • Van Horn; Green; Martinussen (2009): Survey Response Rates and Survey Administration in Counseling and Clinical Psychology. Educational and psychological measurement, 69.•Viechtbauer, W. (August 01, 2010). Conducting meta-analyses in R with the metafor. Journal of Statistical Software, 36, 3, 1-48. •Weiner, S. P., & Dalessio, A. T. (2006). Oversurveying: Causes, consequences, and cures. In A. I. Kraut (Ed.), Getting action from organizational surveys: New concepts, technologies, and applications (pp. 294-311). San Francisco: Jossey-Bass. Chapter 12.

14

https://www.aapor.org/AAPOR_Main/media/publications/Standard-Definitions20169theditionfinal.pdf

http://dx.doi.org/10.1093/jssam/smz008

https://bookdown.org/MathiasHarrer/Doing_Meta_Analysis_in_R/


Analysis methods

15

Level Unit Variance

3 Reports Betweenreports

2 Samples Within reports

1 Participants Sampling

Quelle: Harrer, M. & Ebert, D. D. (2018). Doing Meta-Analysis in R: A practical Guide. PROTECT Lab Friedrich-Alexander University Erlangen-Nuremberg. https://bookdown.org/MathiasHarrer/Doing_Meta_Analysis_in_R/

All analyses are conducted with the metafor package in R (Viechtbauer, 2010)

https://bookdown.org/MathiasHarrer/Doing_Meta_Analysis_in_R/

the effects of time and study conduction on response ratesin ......burgard, t., bosnjak, m. &...

Documents