study eligibility criteria prepared for: the agency for healthcare research and quality (ahrq)...
TRANSCRIPT
![Page 1: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/1.jpg)
Study Eligibility CriteriaPrepared for:
The Agency for Healthcare Research and Quality (AHRQ)
Training Modules for Systematic Reviews Methods Guide
www.ahrq.gov
![Page 2: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/2.jpg)
Systematic Review Process Overview
![Page 3: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/3.jpg)
To understand the role of study eligibility criteria (inclusion/exclusion) in framing a systematic review
To know when and how to set study eligibility criteria
To understand the effect of study eligibility criteria on interpretation of a review
Learning Objectives
![Page 4: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/4.jpg)
Determine which studies will be included in the analysis
Function the same in systematic reviews as in primary research
Should reflect the analytic framework and key questions
Are powerful tools for widening or narrowing the scope of a review
Provide information to determine whether reviews can be compared or combined
Study Eligibility Criteria
![Page 5: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/5.jpg)
Population type (adult, community-dwelling females)
Intervention (screening tool to identify depression risk)
Country (United States only) Setting (hospital-based studies only) Study population size (N > 200) Study design (randomized controlled trials)
Some Sample Criteria
![Page 6: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/6.jpg)
Can be as broad as “all related studies” Can be helpful for exploring “what is known” May result in too much literature to feasibly
review Could pull in disparate literature that cannot
be compared
Using Broad Criteria
![Page 7: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/7.jpg)
Can be helpful in culling homogenous literature
Can reduce size of the literature to a manageable scope
Could be appropriate for a narrow research question
May reduce applicability of the review and sometimes risks introducing bias
Using Narrow Criteria
![Page 8: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/8.jpg)
Choosing how broad or how narrow to set criteria requires a balance of obtaining adequate information to answer a given question without obscuring the results with irrelevant literature.
Review teams should work together to find this balance.
The overarching goal is to minimize bias related to which studies are selected.
Refining Criteria
![Page 9: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/9.jpg)
Distortion of the estimate of effect that comes from how studies are selected for inclusion
Affects the applicability, or “external validity,” of the review itself
Bias in This Context
![Page 10: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/10.jpg)
Inappropriate eligibilty criteria may limit applicability of the review. Example: the use of studies of twin pregnancies in a review
of preterm labor management for low-risk women
Eligibility criteria may result in the inclusion of more of a certain study type that either overestimates or underestimates effectiveness.
Examples of Bias in This Context
![Page 11: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/11.jpg)
Review study goals Assess analytic framework and key
questions Set criteria before beginning abstract review
Selecting Criteria
![Page 12: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/12.jpg)
Population — condition, disease severity and stage, comorbidities, patient demographics
Intervention — dosage, frequency, method of administration
Comparator — placebo, usual care, or active control
Outcome — health outcomes, morbidity, mortality, quality of life
Timing — Duration of followup Setting — Primary, specialty, inpatient,
cointerventions
Tie Criteria to PICOTS
![Page 13: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/13.jpg)
What study designs should be included? Include studies in foreign languages? Include studies conducted in other countries
but published in English? Include “grey” or “fugitive” literature? Include year of publication?
Other Considerations for Setting Criteria
![Page 14: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/14.jpg)
Limit to randomized controlled trials? Include observational studies?
If so, what specific types?
What is the value of a case series? How is the definition of case series
operationalized?
Types of Studies To Be Included
![Page 15: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/15.jpg)
There are four main types of observational studies: Cohorts (with comparisons) Case controls Case series Registries/databases
Observational Studies (I)
![Page 16: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/16.jpg)
Well-conducted observational studies can address issues of applicability and the need for longer-term outcomes if they: include more representative patient populations, have relevant comparators, and report more meaningful clinical outcomes over longer time
frames.
Observational studies may be a better source of information about harms.
Observational Studies (II)
![Page 17: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/17.jpg)
Positive findings may be more likely to be published in high-profile English-language journals.
Therefore, to include only English-language journals may overestimate the positive effect of an intervention.
Empirically, the bias associated with limiting a review to English-language reports has been shown to be small.
Non-English–Language Study Reports
Gregoire G, et al. J Clin Epidemiol 1995;48:159-63; Moher D, et al. J Clin Epidemiol 2000;53:964-72.
![Page 18: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/18.jpg)
Material that may be difficult to identify and retrieve: Government reports and regulatory sites (Center for Drug Evaluation and
Research/U.S. Food and Drug Administration)
Pharmacoepidemiologic databases, including postmarketing surveillance
Book chapters
Conference proceedings
Published dissertations
Scientific information packets (Agency for Healthcare Research and Quality)
Rationale for a search: Because journals may publish positive or statistically significant results,
finding grey literature of unpublished nonsignificant or null results may indicate the presence of publication bias.
Grey or “Fugitive “Literature
![Page 19: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/19.jpg)
Literature searches typically have date parameters (i.e., earliest or latest date of publication).
Date parameters are particularly important when there has been a change in policy, practice, or formulation that makes older studies less applicable.
Year of Publication
![Page 20: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/20.jpg)
The rest of this module contains a number of questions and some sample criteria.
Using your newly acquired knowledge of selection criteria and the guidelines described in the previous slides, work through the thought exercises and examples that follow.
Using the Criteria
![Page 21: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/21.jpg)
What would you do if you were asked to review the literature on transition support for adolescents with autism spectrum disorder who are entering adulthood? Before seeing the key questions, consider the categories of criteria that you will want to apply.
Exercise 1
![Page 22: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/22.jpg)
What is the relevant population? What is the intervention of interest? To what exposure is the intervention being
compared? What outcomes are relevant? What short-term or long-term outcomes
should be considered? In what setting would the results be
applicable?
Exercise 1: Basic PICOTS Questions
PICOTS = population, intervention, comparators, outcomes, timing, and setting
![Page 23: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/23.jpg)
Population Population What constitutes an adolescent? What constitutes a diagnosis of autism spectrum disorder (ASD)?
InterventiInterventionon
How is transition support defined?
ComparatoComparator r
Do we compare to no transition support or directly compare types of support?
OutcomeOutcomeWhat are the goals for adolescents with ASD as they transition to adulthood?
Timing Timing How quickly should the outcomes be apparent?
SettingSettingIs transition support provided in multiple settings, such as schools, clinics, and the community?
Exercise 1: PICOTS
![Page 24: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/24.jpg)
A study that included “individuals over 10 years of age”?
A paper about an intervention for individuals with a range of developmental disabilities, not only autism spectrum disorder (ASD)?
A study of children with Asperger’s syndrome but not other forms of ASD?
Exercise 1: What Would You Do With . . .
![Page 25: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/25.jpg)
There are no “right” answers to these questions.
The team should consider how selection criteria may bias the review or affect applicability.
The team should clearly define how eligibility criteria are operationalized and record these decisions throughout the review.
Exercise 1: No “Right” Answers
![Page 26: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/26.jpg)
What is the efficacy of home uterine activity monitoring for preventing preterm birth among women at low risk of a preterm birth?
Exercise 2: Selecting Criteria for a Narrow Research Question
![Page 27: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/27.jpg)
Efficacy: randomized controlled trials only Low risk: no previous preterm birth Yield: 11 studies
Exercise 2: Implications of a Question That Requires Narrow Criteria
![Page 28: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/28.jpg)
What is the best management strategy for overactive bladder (OAB) among women? consideration:
OAB is a fairly difficult condition to define. The study of OAB is often combined with other
types of urological conditions, such as stress incontinence or prostate issues, when studied.
OAB is treated pharmacologically, behaviorally, and surgically and with complementary and alternative interventions — each area includes very different types of studies.
Exercise 3:Selecting Criteria for a Broad Research Question
![Page 29: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/29.jpg)
The research team required a minimum sample size N ≥ 50 in each study at study start.
Implications: Total excluded for N < 50: 79
Excluded for N < 20: 36 Excluded for N 20 to 29: 23 Excluded for N 30 to 39: 8 Excluded for N 40 to 49: 12
Exercise 3: Impact of Study-Size Criteria on Excluded Studies
![Page 30: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/30.jpg)
Studies had to include at least 75 percent women. This decision was based on expert opinion and the size and
scope of the literature.
Based on this criterion, 40 studies were excluded.
Another 27 studies would have been excluded had the review been limited to studies of only women.
Exercise 3:Impact of Gender on Study Selection Criteria
![Page 31: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/31.jpg)
Case series with N ≥ 2 were included. Only 3 of 169 studies were randomized
controlled trials; 122 were case series. Because of the relative newness of this area
of research, it was important to capture data even from studies without comparison groups.
Example 1: A Maternal-Fetal Surgery Technical Brief
![Page 32: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/32.jpg)
A systematic review on outcomes of cesarean delivery on maternal request.
Conducted for the National Institutes of Health-Office of Medical Applications of Research (NIH-OMAR) State-of-the-Science Conference
Example 2:Systematic Review of Cesarean Delivery
![Page 33: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/33.jpg)
No evidence on outcomes of cesarean delivery on maternal request (CDMR) versus other modes of delivery
Urgent need for actionable evidence Need to recognize and account for
confounders
Example 2: The Challenge in the Systematic Review on Cesarean Delivery
![Page 34: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/34.jpg)
Expand search to include proxies Weight rungs of evidence to account for
confounding Highest rung: trials of breech delivery, but only for maternal
outcomes Lower rung: planned cesarean versus planned vaginal
delivery Lowest rung: comparisons of maternal and neonatal
outcomes from actual modes of delivery
Example 2: The Solution in the Systematic Review on Cesarean Delivery
![Page 35: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/35.jpg)
Eligibility criteria: Specify the scope of the literature to be included Should be tied to the analytic framework, key questions,
and PICOTS Can reduce bias and support applicability, but run the risk
of introducing bias if not chosen appropriately. Must be applied consistently among individuals on the
review team Need to be recorded throughout the project and in the
report for transparency and replicability
Key Messages
PICOTS = population, intervention, comparator, outcome, timing, and setting
![Page 36: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/36.jpg)
Dickersin K, Scherer R, Lefebvre C. Identifying relevant studies for systematic reviews. BMJ 1994;309:1286-91.
Gregoire G, Derderain F, Le L. Selecting the language of the publication included in a meta-analysis: is there a Tower of Babel bias? J Clin Epidemiol 1995;48:159-63.
McAuley L, Pham B, Tugwell P, et al. Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? Lancet 2000;356:1228-31.
McPheeters ML, Briss P, Teutsch SJ, et al. Systematic reviews in public health. In: Brownson RC and Petitti DB, eds. Applied epidemiology. New York: Oxford University Press; 2006. p. 99-124.
Moher D, Pham B, Klassen TP, et al. What contribution do languages other than English make on the results of meta-analyses? J Clin Epidemiol 2000; 53:964-72.
References
![Page 37: Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide](https://reader036.vdocuments.us/reader036/viewer/2022062620/551a8c67550346b52d8b5bcc/html5/thumbnails/37.jpg)
This presentation was prepared by Melissa L. McPheeters, Ph.D., M.P.H., a member of the Vanderbilt University Evidence-based Practice Center.
The information in this module is currently not included in Version 1.0 of the Methods Guide for Comparative Effectiveness Reviews (available at: http://www.effectivehealthcare. ahrq.gov/ehc/products/60/294/2009_0805_principles1.pdf).
Author