assessing quality of individual studies
TRANSCRIPT
![Page 1: Assessing Quality of Individual Studies](https://reader034.vdocuments.us/reader034/viewer/2022042503/5597e2a51a28abea0e8b45bd/html5/thumbnails/1.jpg)
Assessing the Quality of Individual Studies
Prepared for:
The Agency for Healthcare Research and Quality (AHRQ)
Training Modules for Systematic Reviews Methods Guide
www.ahrq.gov
![Page 2: Assessing Quality of Individual Studies](https://reader034.vdocuments.us/reader034/viewer/2022042503/5597e2a51a28abea0e8b45bd/html5/thumbnails/2.jpg)
Systematic Review Process Overview
![Page 3: Assessing Quality of Individual Studies](https://reader034.vdocuments.us/reader034/viewer/2022042503/5597e2a51a28abea0e8b45bd/html5/thumbnails/3.jpg)
To describe the concept of quality assessment
To identify reasons for quality assessment
To list the steps in quality assessment
To describe and report the methods for quality assessment
Learning Objectives
![Page 4: Assessing Quality of Individual Studies](https://reader034.vdocuments.us/reader034/viewer/2022042503/5597e2a51a28abea0e8b45bd/html5/thumbnails/4.jpg)
Definition of quality:“[T]he extent to which all aspects of a study’s design and conduct
can be shown to protect against systematic bias, nonsystematic bias, and inferential error.” (Lohr & Carey, 1999)
Considered to be synonymous with internal validity
Relevant for individual studies
Distinct from assessment of risk of bias for a body of evidence
What Is Quality Assessment?
Lohr KN, Cary TS. Jt Comm J Qual Improv 1999;25:470-9.
![Page 5: Assessing Quality of Individual Studies](https://reader034.vdocuments.us/reader034/viewer/2022042503/5597e2a51a28abea0e8b45bd/html5/thumbnails/5.jpg)
Systematic Errors: Include selection bias and confounding, in which values tend to be
inaccurate in a particular direction
Nonsystematic Errors:Are attributable to chance
Inferential Errors:Result from problems in data analysis and interpretation, such as
choice of the wrong statistical measure or wrongly rejecting the null hypothesis
What Are the Components of Quality Assessment?
Lohr KN, Cary TS. Jt Comm J Qual Improv 1999;25:470-9.
![Page 6: Assessing Quality of Individual Studies](https://reader034.vdocuments.us/reader034/viewer/2022042503/5597e2a51a28abea0e8b45bd/html5/thumbnails/6.jpg)
Consider the Contribution of an Individual Study to the Body of Evidence
Quality (systematic error and inferential error, type of study)
Size of study (nonsystematic or random error)
Direction and magnitude of results
Relevance of results to key questions
Risk of Bias
Precision
Directness
Consistency
Applicability
Owens DK, et al. In: Methods guide for comparative effectiveness reviews. Available at: http://effectivehealthcare.ahrq.gov/ehc/products/60/318/2009_0805_grading.pdf.
Direct vs. indirect comparisons, health outcomes vs. surrogate outcomes
![Page 7: Assessing Quality of Individual Studies](https://reader034.vdocuments.us/reader034/viewer/2022042503/5597e2a51a28abea0e8b45bd/html5/thumbnails/7.jpg)
Quality assessment is required for:Interpreting results
Grading the strength of the body of evidence
Quality assessment may also be used for:Selecting studies for the review (based on a priori assessment of
evidence gaps)
Selecting studies for qualitative synthesis
Selecting studies for quantitative synthesis
Interpreting heterogeneous findings
Reasons for Quality Assessment
![Page 8: Assessing Quality of Individual Studies](https://reader034.vdocuments.us/reader034/viewer/2022042503/5597e2a51a28abea0e8b45bd/html5/thumbnails/8.jpg)
1. Classify the study design
2. Apply predefined criteria for quality assessment of each outcome based on:Study design — sources of bias may vary by design
Study conduct — poor study conduct and discrepancy between design and conduct may increase risk of bias
Reporting — quality assessment may be influenced by adequacy of reporting when information on study design and conduct are missing
What Are the Steps in Quality Assessmentof Each Individual Study (I)?
Helfand M, Balshem H. In: Helfand M, Balshem H. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ehc/products/60/294/2009_0805_principles1.pdf.
.
![Page 9: Assessing Quality of Individual Studies](https://reader034.vdocuments.us/reader034/viewer/2022042503/5597e2a51a28abea0e8b45bd/html5/thumbnails/9.jpg)
3. Arrive at a summary judgment of the study’s quality to rate studies good, fair, or poorRequires resolution of conflicts when two reviewers
independently evaluate quality
What Are the Steps in Quality Assessment of Each Individual Study (II)?
Helfand M, Balshem H. In: Helfand M, Balshem H. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ehc/products/60/294/2009_0805_principles1.pdf.
![Page 10: Assessing Quality of Individual Studies](https://reader034.vdocuments.us/reader034/viewer/2022042503/5597e2a51a28abea0e8b45bd/html5/thumbnails/10.jpg)
Did the study have more than one group or arm? If so, was a control group present?
Did investigators have control over allocation and timing of the intervention?
Did investigators randomly allocate subjects to interventions?
Did investigators concurrently measure intervention and exposure status for intervention and comparison groups?
Did investigators concurrently measure outcomes for intervention and comparison groups?
Questions To ConsiderWhen Classifying Study Design
![Page 11: Assessing Quality of Individual Studies](https://reader034.vdocuments.us/reader034/viewer/2022042503/5597e2a51a28abea0e8b45bd/html5/thumbnails/11.jpg)
Select a tool that is based on coverage of important criteria
Based on the topic, select and apply one of several available tools that consider and explain how to evaluate:Similarity of groups at baseline in terms of baseline characteristics
and prognostic factors
Validity of primary outcomes
Blinded measurement of outcomes
Apply Predefined Criteria for All Study Types (I)
Helfand M, Balshem H. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ehc/products/60/294/2009_0805_principles1.pdf.
![Page 12: Assessing Quality of Individual Studies](https://reader034.vdocuments.us/reader034/viewer/2022042503/5597e2a51a28abea0e8b45bd/html5/thumbnails/12.jpg)
Apply one of several available tools that considerIntention-to-treat analysis
Differential loss to followup between the compared groups or overall high loss to followup
Conflict of interest
Apply Predefined Criteria for All Study Types (II)
![Page 13: Assessing Quality of Individual Studies](https://reader034.vdocuments.us/reader034/viewer/2022042503/5597e2a51a28abea0e8b45bd/html5/thumbnails/13.jpg)
Methods used for randomization
Allocation concealment
Blinding of subjects and providers
Additional Criteria for Trials
Helfand M, Balshem H. In: Helfand M, Balshem H. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ehc/products/60/294/2009_0805_principles1.pdf.
![Page 14: Assessing Quality of Individual Studies](https://reader034.vdocuments.us/reader034/viewer/2022042503/5597e2a51a28abea0e8b45bd/html5/thumbnails/14.jpg)
Sample size, width of confidence intervals, or power
Methods for selecting participantsInception cohort, methods to adjust for or avoid selection bias
Methods for measuring exposure variables
Additional Criteria for Observational Studies (I)
Helfand M, Balshem H. In: Helfand M, Balshem H. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ehc/products/60/294/2009_0805_principles1.pdf.
![Page 15: Assessing Quality of Individual Studies](https://reader034.vdocuments.us/reader034/viewer/2022042503/5597e2a51a28abea0e8b45bd/html5/thumbnails/15.jpg)
Methods for dealing with any design-specific issues such as recall bias and interviewer bias
Analytical methods to control confoundingMatching, stratification, multivariate analysis, or other statistical
adjustment
Additional Criteria for Observational Studies (II)
Helfand M, Balshem H. In: Helfand M, Balshem H. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ehc/products/60/294/2009_0805_principles1.pdf.
![Page 16: Assessing Quality of Individual Studies](https://reader034.vdocuments.us/reader034/viewer/2022042503/5597e2a51a28abea0e8b45bd/html5/thumbnails/16.jpg)
After assessment of individual criteria, assign ratings of “good,” “fair,” or “poor” (attributes described in later slides)
Assess quality for each outcome of interest
Base ratings on the evaluation of likely effect of design or execution flaws on internal validity, rather than a nominal failure to meet every quality criterion
Adjudicate differences between raters in a transparent manner when two raters independently assess overall quality
Arrive at a Comprehensive Judgment of Quality
Helfand M, Balshem H. In: Helfand M, Balshem H. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ehc/products/60/294/2009_0805_principles1.pdf.
![Page 17: Assessing Quality of Individual Studies](https://reader034.vdocuments.us/reader034/viewer/2022042503/5597e2a51a28abea0e8b45bd/html5/thumbnails/17.jpg)
Design and conduct of study address risk of bias
Appropriate measurement of outcomes
Appropriate statistical and analytical methods
Low drop-out rates
Adequate reporting of statistical and analytical methods, drop-out rates and reasons, and outcomes (no reporting errors)
Attributes of Good-Quality Studies
Helfand M, Balshem H. In: Helfand M, Balshem H. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ehc/products/60/294/2009_0805_principles1.pdf.
![Page 18: Assessing Quality of Individual Studies](https://reader034.vdocuments.us/reader034/viewer/2022042503/5597e2a51a28abea0e8b45bd/html5/thumbnails/18.jpg)
Do not meet all the criteria required for a rating of good quality
No flaw is likely to cause major bias
Missing information often drives rating
Attributes of Fair-Quality Studies
Helfand M, Balshem H. In: Helfand M, Balshem H. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ehc/products/60/294/2009_0805_principles1.pdf.
![Page 19: Assessing Quality of Individual Studies](https://reader034.vdocuments.us/reader034/viewer/2022042503/5597e2a51a28abea0e8b45bd/html5/thumbnails/19.jpg)
Significant biasesInappropriate design, conduct, analysis, or reporting
Large amounts of missing information
Discrepancies in reporting
Attributes of Poor-Quality Studies
Helfand M, Balshem H. In: Helfand M, Balshem H. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ehc/products/60/294/2009_0805_principles1.pdf.
![Page 20: Assessing Quality of Individual Studies](https://reader034.vdocuments.us/reader034/viewer/2022042503/5597e2a51a28abea0e8b45bd/html5/thumbnails/20.jpg)
Poor-quality studies may be excluded or included
Base decisions on gaps in current evidence and availability of good-quality or fair-quality studies
Justify selective inclusion of poor-quality studies for subgroups or subquestions
Treatment of Poor-Quality Studies in the Review
![Page 21: Assessing Quality of Individual Studies](https://reader034.vdocuments.us/reader034/viewer/2022042503/5597e2a51a28abea0e8b45bd/html5/thumbnails/21.jpg)
Accompany overall quality rating for individual studies with a statement of:Flaws in the design or execution of a study
The potential consequences of that flaw
Report the criteria and the process used to arrive at a quality rating
Reporting Quality Ratings
![Page 22: Assessing Quality of Individual Studies](https://reader034.vdocuments.us/reader034/viewer/2022042503/5597e2a51a28abea0e8b45bd/html5/thumbnails/22.jpg)
Quality assessment:Is synonymous with internal validity
Refers to individual studies
Contributes to, but is separate from, the evaluation of the risk of systematic bias for the body of evidence
Key Messages (I):Definition of Quality Assessment
![Page 23: Assessing Quality of Individual Studies](https://reader034.vdocuments.us/reader034/viewer/2022042503/5597e2a51a28abea0e8b45bd/html5/thumbnails/23.jpg)
Results of quality assessment are used in multiple steps in the systematic review process, from final inclusion of studies to interpretation of evidence
Steps in quality assessmentStudy design classification
Assessment of individual quality criteria
Summary judgment of the study quality
Key Messages (II): Rationale of and Steps in Quality Assessment
![Page 24: Assessing Quality of Individual Studies](https://reader034.vdocuments.us/reader034/viewer/2022042503/5597e2a51a28abea0e8b45bd/html5/thumbnails/24.jpg)
Transparency of processFull reporting of all elements of quality for each individual study
Explicit description (and examples) of how each criterion was operationalized
Clear reporting of how team members scored quality
Description of how conflicts between raters were resolved
Transparency of judgmentExplanation of final rating
Key Messages (III): Reporting
![Page 25: Assessing Quality of Individual Studies](https://reader034.vdocuments.us/reader034/viewer/2022042503/5597e2a51a28abea0e8b45bd/html5/thumbnails/25.jpg)
Deeks JJ, Dinnes J, D’Amico R, et al, for the International Stroke Trial Collaborative Group and the European Carotid Surgery Trial Collaborative Group. Evaluating non-randomised intervention studies. Health Technol Assess 2003;7(27):iii-x,1-173.
Hartling L, Bond K, Harvey K, et al. Developing and testing a tool for the classification of study designs in systematic reviews of interventions and exposures. (Prepared by the University of Alberta Evidence-based Practice Center under contract no. 290-02-0023.) In press.
Helfand M, Balshem H. Principles in developing and applying guidance. In: Methods guide for comparative effectiveness reviews. Rockville, MD: Agency for Healthcare Research and Quality, Posted August 2009. Available at: http://www.effectivehealthcare.ahrq. gov/ehc/products/ 60/294/2009_0805_principles1.pdf.
References (I)
![Page 26: Assessing Quality of Individual Studies](https://reader034.vdocuments.us/reader034/viewer/2022042503/5597e2a51a28abea0e8b45bd/html5/thumbnails/26.jpg)
Lohr KN, Carey TS. Assessing “best evidence”: issues in grading the quality of studies for systematic reviews. Jt Comm J Qual Improv 1999;25:470-9.
Owens DK, Lohr KN, Atkins D, et al. Grading the strength of a body of evidence when comparing medical interventions. In: Methods guide for comparative effectiveness reviews. Rockville, MD: Agency for Healthcare Research and Quality, Posted July 2009. Available at: http://effectivehealthcare.ahrq.gov/ehc/products/60/318/2009_0805_grading.pdf.
Sanderson S, Tatt ID, Higgins JP. Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol 2007;36:666-76.
References (II)
![Page 27: Assessing Quality of Individual Studies](https://reader034.vdocuments.us/reader034/viewer/2022042503/5597e2a51a28abea0e8b45bd/html5/thumbnails/27.jpg)
West S, King V, Carey TS, et al. Systems to Rate the Strength of Scientific Evidence, Evidence Report/Technology Assessment No. 47 (Prepared by the Research Triangle Institute–University of North Carolina Evidence-based Practice Center under Contract No. 290-97-0011. Rockville, MD: Agency for Healthcare Research and Quality, March 2002. AHRQ Publication No. 02-E015. Available at: http://www.ahrq.gov/clinic/epcsums/strengthsum.pdf.
Whiting P, Rutjes AWS, Dinnes J, et al. Development and validation of methods for assessing the quality of diagnostic accuracy studies. Health Technol Assess 2004;8(25):iii, 1-234.
References (III)
![Page 28: Assessing Quality of Individual Studies](https://reader034.vdocuments.us/reader034/viewer/2022042503/5597e2a51a28abea0e8b45bd/html5/thumbnails/28.jpg)
This presentation was prepared by Meera Viswanathan, Ph.D., a member of the Research Triangle Institute–University of North Carolina Evidence-based Practice Center.
The presentation is based on an update of chapter 6 in version 1.0 of the Methods Guide for Comparative Effectiveness Reviews (update available at: http://www.effectivehealthcare. ahrq.gov/ehc/products/60/294/2009_0805_principles1.pdf).
Author