Reporting the Review Prepared for:

The Agency for Healthcare Research and Quality (AHRQ)

Training Modules for Systematic Reviews Methods Guide

www.ahrq.gov

To describe the various elements that need to be reported upon completion of a systematic review

To distinguish examples of reporting that are adequate from those that are inadequate

Learning Objectives

Systematic Review Process Overview

Follow a standard template for the overall report: Abstract and Executive Summary Chapter 1. Introduction Chapter 2. Methods Chapter 3. Results Chapter 4. Discussion

Ordering of subsections may vary but: Should adhere to principles of clarity Should be consistent with key questions May be guided by PICOTS

Writing the Report

PICOT(S) = population, intervention, comparator, outcome, time frame, and study design or setting

Abstract and Executive Summary Chapter 1: Introduction

The purpose of this chapter is to define the project, the purpose and scope of the review, the key research questions, the analytic framework, et cetera.

Chapter 2: Methods The purpose of this chapter is to explain the methods used in

the review, including the experts involved, the literature search strategy used, the inclusion and exclusion criteria applied, how the evidence tables were developed, the approach used to assess the quality of studies, and data abstraction and data synthesis methods.

This chapter should not present any “results” (e.g., tables), but rather should serve as a guide for how the study information was collected and the evidence tables were created so that the research can be replicated.

Systematic Review Report Structure (I)

Chapter 3: Results The purpose of this chapter is to report the results of the

data analyses, which should be broken down according to the key research questions.

Subsections should be used when applicable.

Chapter 4: Discussion The purpose of this chapter is to discuss the strength of the

literature and evidence, the principal findings (broken down by key questions) of the review, areas of future research, and any conclusions that can be drawn.

Systematic Review Report Structure (II)

The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) Statement is a guideline that was developed to help improve the quality of review reports. The guideline consists of a 27-item checklist and a flow

diagram. Investigators can access the guideline online

(http://www.prisma-statement.org/index.htm).

Using Formal Guidelines To Improvethe Reporting of Systematic Reviews

Identify the report as a systematic review, meta-analysis, or comparative effectiveness review.

Use the PICOTS framework to guide construction of the title.

Make the title as succinct as possible yet keep it informative. Example of a short title:

Comparative Effectiveness of Lipid-Modifying Agents Example of a longer but more informative title:

Mortality in Randomized Trials of Antioxidant Supplements for Primary And Secondary Prevention: Systematic Review and Meta-analysis

Title of the Review Report

PICOT(S) = population, intervention, comparator, outcome, time frame, and study design or setting

Should be structured as follows: BackgroundObjectivesKey questionsMethods

Data sources Eligibility criteria Study appraisal Data synthesis

ResultsLimitationsConclusions

Implications of key findings

Executive Summary (I)

Should be a distillation of the entire report Should exclude study-by-study results Should describe the evidence that supports

all summary statements Should describe the strength of the

evidence as categorized in the evidence review

Executive Summary (II)

Should state the purpose and scope of the review: Identify the clinical decisional dilemma. Identify the current literature and state of practice. Give readers the context in which the review was conducted.

Should include these important components: Objective(s) and key question(s)

Provide an explicit statement of the research questions being addressed with reference to the PICOTS framework.

Analytic framework Use such a framework to model existing evidence (refer to

the Analytic Framework module for details).

Introduction

PICOTS = population, intervention, comparator, outcome, time frame, and study design or setting

Should provide the following information in a clear and transparent manner: Literature Search Strategy and Data Sources Eligibility Criteria

Specific to study characteristics (e.g., PICOTS, length of followup)

Specific to report characteristics (e.g., years considered, language, publication status)

Data Extraction and Data Items (e.g., variables for which data were sought, assumptions and simplifications )

Quality Assessment Synthesis of Results Grading Strength of Evidence Additional Analyses

Methods Overview

Present the complete electronic search strategy — including any limits used — in the Appendix of the report. Brief example:

We used the following search terms to search all trials registers and databases: immunoglobulin; IVIG [intravenous immunoglobulin]; sepsis; septic shock; septicaemia; and septicemia.

The purpose of including the entire search strategy is to ensure transparency and to permit replication of the review.

Methods: Literature Search Strategy

Alejandria MM, et al. Cochrane Database Syst Rev 2002;(1):CD001090.

Describe all information sources used in the literature search: Databases with dates of coverage Contacts with study authors to identify additional

studies Date of the last search Example:

We searched the following databases for primary studies for the periods in parentheses: MEDLINE® (1966 to January 2006), EMBASE® (1974 to January 2006), and the Cochrane Central Register of Controlled Trials (1966 to January 2006). We also searched for systematic reviews until November 2005.

Methods: Data Sources

Bolen S, et al. AHRQ Comparative Effectiveness Review No. 5. Available at: http://www.effectivehealthcare.ahrq.gov/ehc/products/6/39/OralFullReport.pdf.

State the processes used to select studies for review: Screening Eligibility assessment Inclusion/exclusion criteria applied for the systematic review and, if

applicable, the meta-analysis Example:

We included trials if the randomization scheme included groups that assigned patients to treatment guided by the PAC [pulmonary artery catheter] or treatment without the PAC. We only included trials if they reported death and number of days hospitalized or the number of days in the ICU as outcome measures. Studies were excluded if the randomization scheme did not specify groups as PAC or no PAC, if patients were not randomized to a conventional PAC, if investigators combined randomized and nonrandomized groups when reporting outcomes, or if there were no outcome data on death or hospitalizations.

Methods: Eligibility Criteria (l)

Shah MR, et al. JAMA 2005;294:1664-70.

Type of studies: Example:

Randomised clinical trials studying the administration of hepatitis B vaccine to CRF [chronic renal failure] patients, with or without dialysis. No language, publication date, or publication status restrictions were imposed.

Types of participants: Example:

Participants of any age with CRF or receiving dialysis (haemodialysis or peritoneal dialysis) were considered.…Renal transplant patients were excluded from this review.

Methods: Eligibility Criteria (II)

Schroth RJ, et al. Cochrane Database Syst Rev 2004;(3):CD003775.

Types of interventions:

Example:

Trials comparing the beneficial and harmful effects of hepatitis B vaccines with adjuvant or cytokine co-interventions [and] trials comparing the beneficial and harmful effects of immunoglobulin prophylaxis.…Hepatitis B vaccines (plasma or recombinant [yeast] derived) of all types, dose, and regimens versus placebo, control vaccine, or no vaccine.

Types of outcomes:

Example:

Primary outcome measures: Seroconversion, [that is], proportion of patients with adequate anti-HBs response (10 IU/L or Sample Ratio Units).…Secondary outcome measures: Adverse events of hepatitis B vaccinations…[and] mortality.

Methods: Eligibility Criteria (III)

Schroth RJ, et al. Cochrane Database Syst Rev 2004;(3):CD003775.

Describe the method of data extraction.Example:

We developed a data extraction sheet[,]…pilot-tested it on ten randomly-selected included studies, and refined it accordingly. One review author extracted the…data…and the second author checked the extracted data. …Disagreements were resolved by discussion between the two review authors.

Describe any processes used to obtain and confirm data from other investigators.

Methods: Data Extraction

Mistiaen P, Poot E. Cochrane Database Syst Rev 2006;(4):CD004510.

List and define all variables for which data were sought, using the PICOTS framework as a guide.

List any assumptions and simplifications that were made in defining the variables.Example:

Information was extracted from each included trial on: (1) characteristics of trial participants…and the trial’s inclusion and exclusion criteria; (2) type of intervention…(versus placebo or versus the type, dose, duration and frequency of another NSAID [nonsteroidal antiinflammatory drug]; or versus another pain management drug; or versus no treatment); (3) type of outcome measure.

Methods: Data Items

Allen C, et al. Cochrane Database Syst Rev 2005;(4):CD004753.

PICOT(S) = population, intervention, comparator, outcome, timing, and study design or setting

Describe the methods and criteria used to assess the quality (risk of bias) of individual studies.Specify whether or not the assessment was carried out at the

study or outcome level, or both.Describe how this information is to be used in any data

synthesis.Examples from two separate studies:

Pairs of reviewers…determined the adequacy of randomization and concealment of allocation, blinding of patients, health care providers, data collectors, and outcome assessors; and extent of loss to follow-up.

To explore variability in study results (heterogeneity) we specified the following hypotheses…

Methods: Quality Assessment

Tracz MJ, et al. J Clin Endocrinol Metab 2006;91:2011-6; Bucher HC, et al. BMJ 2000;321:73-7.

For each meta-analysis:Describe the methods used to handle the data and to

combine the results of studies.Describe measures of consistency (e.g., I-squared).Examples from two separate studies:

In very few instances, estimates of baseline mean or mean QOL [quality of life] responses were obtained without corresponding estimates of variance (standard deviation [SD] or standard error). In these instances, an SD was imputed from the mean of the known SDs. In a number of cases, the response data available were the mean and variance in a prestudy condition and after therapy.…

We tested for heterogeneity with the Breslow-Day test, and used the method proposed by Higgins et al. to measure inconsistency.

Methods: Synthesis of Results

Jones M, et al. Cancer 2004;101:1720-32; Briel M, et al. Am J Med 2004;117:596-606.

Detail any assessment of risk of bias that may affect the cumulative evidence.Publication biasSelective reporting within studiesExample:

For each trial we plotted the effect by the inverse of its standard error. The symmetry of such “funnel plots” was assessed both visually, and formally with Egger’s test, to see if the effect decreased with increasing sample size.

Methods: Grading Strength of Evidence

Hróbjartsson A, Gøtzsche PC. Cochrane Database Syst Rev 2004;(1):CD003974.

Describe the methods of additional analyses (e.g., sensitivity or subgroup analyses, meta-regression).

Indicate which of these analyses were prespecified. Example:

Sensitivity analyses were pre-specified. The treatment effects were examined according to quality components (concealed treatment allocation, blinding of patients and caregivers, blinded outcome assessment), time to initiation of statins, and the type of statin. One post-hoc sensitivity analysis was conducted including unpublished data from a trial using cerivastatin.

Methods: Additional Analyses

Briel M, et al. JAMA 2006;295:2046-56.

The results section of the review should contain the following subsections: Study Selection Study Characteristics Quality Assessment Individual Studies Synthesis of Results Grading Strength of Evidence Additional Analyses

Sensitivity Subgroup

Results Overview

Place at the beginning of the Results section, not in the Methods section.

Give numbers of studies screened, assessed for eligibility, and included in the review.

Give the reasons for exclusions at each stage of the assessment, ideally illustrated with a flow diagram.Example:

The search of Medline, PsycInfo and Cinahl [sic] databases provided a total of 584 citations. After adjusting for duplicates 509 remained. Of these, 479 studies were discarded because…. Three additional studies…were discarded because…. The full text of the remaining 27 citations was examined in more detail. It appeared that 22 studies did not meet the inclusion criteria as described. Five studies…met the inclusion criteria and were included in the systematic review.

Results: Study Selection

Uitterhoeve RJ, et al. Br J Cancer. 2004;91:1050-62.

Examples of Flow Diagrams

Fuccio L, et al. Ann Intern Med 2007;147:53-62. Reprinted with permission from the American College of Physicians.

Sharma M, et al. Ann Intern Med 2009:151:622-30. Reprinted with permission from the American College of Physicians.

Study characteristics can be presented: Within text In summary tables and graphs Example:

All four studies finally selected for the review were randomised controlled trials published in English. The duration of the intervention was 24 months for the RIO-North America and 12 months for the RIO-Diabetes RIO-Lipids and RIO-Europe study. Although the last two described a period of 24 months during which they were conducted, only the first 12-months results are provided. All trials had a run-in, as a single blind period before the randomisation.

Results: Study Characteristics

Curioni C, André C. Cochrane Database Syst Rev 2006;(4):CD006162.

Present data on risk of bias for each study analyzed.

Present the results of outcome-level assessments, if available. Example:

Results: Quality Assessment

Devereaux PJ, et al. BMJ 2005;331:313-21, as adapted in Liberati A, et al. Ann Intern Med 2009;151:W65-94.

For all outcomes considered (benefits and harms), present the following for each study: simple summary data for each intervention group, and effect estimates and confidence intervals (ideally with a forest

plot).

Present the results of individual studies in evidence tables and summary tables and not in the text.

Refer to the Presentation of Findings module to see examples of the tables and graphs used to present summaries of individual studies.

Results: Individual Studies*

* This may appear in the appendix for Evidence-based Practice Center reports.

Present synthesized results in text, summary tables, or evidence maps.Text example:

Mortality data were available for all six trials, randomizing 311 patients and reporting data for 305 patients. There were no deaths reported in the three respiratory syncytial virus/severe bronchiolitis trials; thus our estimate is based on three trials randomizing 232 patients, 64 of whom died. In the pooled analysis, surfactant was associated with significantly lower mortality (relative risk = 0.7, 95% confidence interval = 0.4–0.97, P = 0.04). There was no evidence of heterogeneity (I2 = 0%).

Summary tables: refer to the Presentation of Findings module Evidence maps: refer to the Presentation of Findings module

Results: Synthesis of Results

Duffett M, et al. Crit Care 2007;11:R66.

Include risk of bias, directness, consistency, and precision in reporting how evidence was graded. Example:

There is a low level of evidence…that RFA [radiofrequency catheter ablation] improves quality of life more than medical treatment. Three RCTs [randomized controlled trials] and one observational study reported more improvement in the general or physical functioning…in patients who underwent RFA.…However, these studies assessed the results at nonuniform time points and therefore the findings may be difficult to interpret.

Results: Grading Strength of Evidence

Ip S, et al. AHRQ Comparative Effectiveness Review No. 15. Available at: http://www.effectivehealthcare.ahrq.gov/ehc/products/51/114/2009_0623RadiofrequencyFinal.pdf.

Give the results of additional analyses, such as sensitivity or subgroup analyses and meta-regressions.

Include the results of additional analyses to facilitate a better understanding of heterogeneity. Example 1:

[B]enefits of chondroitin were smaller in trials with adequate concealment of allocation compared with trials with unclear concealment (P for interaction = 0.050), in trials with an intention-to-treat analysis compared with those that had excluded patients from the analysis (P for interaction = 0.017), and in large compared with small trials (P for interaction = 0.022).

Results: Additional Sensitivity Analyses

Reichenbach S, et al. Ann Intern Med 2007;146:580-90.

Example 2:Subgroup analyses according to antibody status, antiviral medications, organ transplanted, treatment duration, use of antilymphocyte therapy, time to outcome assessment, study quality and other aspects of study design did not demonstrate any differences in treatment effects. Multivariate meta-regression showed no significant difference in CMV [cytomegalovirus] disease after allowing for potential confounding or effect-modification by prophylactic drug used, organ transplanted or recipient serostatus in CMV positive recipients and CMV negative recipients of CMV positive donors.

Results: Additional Subgroup Analyses

Hodson EM, et al. Cochrane Database Syst Rev 2008;(2):CD003774.

Summarize the main findings, including the strength of evidence for each main outcome.

Consider the applicability of findings to key groups (e.g., health care providers, users, and policymakers).

Refer to the Assessing Applicability and Grading Strength of Evidence modules for additional guidance.

Discussion: Summary of Evidence

Example:

Compared with men who used watchful waiting , men with clinically localized prostate cancer detected by methods other than PSA [prostate-specific antigen] testing and treated with radical prostatectomy experienced fewer deaths from prostate cancer, marginally fewer deaths from any cause, and fewer distant metastases. The greater benefit of RP on cancer-specific and overall mortality appears to be limited to men under 65 years of age but is not dependent on baseline PSA level or histologic grade.

Discussion: Summary of Evidence

Wilt TJ, et al. AHRQ Comparative Effectiveness Review No. 13. Available at: . http://www.effectivehealthcare.ahrq.gov/ehc/products/9/80/2008_0204ProstateCancerFinal.pdf.

Discuss the limitations of the review at different levels: Study level (e.g., risk of bias) Outcome level (e.g., benefits or harms) Review level (e.g., incomplete retrieval of identified

research; reporting bias) Example:

The meta-analysis reported here combines data across studies in order to estimate treatment effects with more precision than is possible in a single study. The main limitation of this meta-analysis, as with any overview, is that the patient population, the antibiotic regimen and the outcome definitions are not the same across studies.

Discussion: Limitations

Liberati A, et al. Cochrane Database Syst Rev 2004;(1):CD000022.

Provide a general interpretation of the results in the context of other evidence and the implications for future research. Example:

[T]he available clinical trial evidence supporting the use of combination therapies over higher dose statin therapy is insufficient to guide clinical decisions. The long term clinical benefits and risks of combination therapies have yet to be demonstrated. There are some instances, such as failure to reach targets in spite of maximal statin therapy, and populations with elevated triglycerides who need to achieve secondary goals, in which clinicians may choose combinations pending definitive evidence.

Discussion: Conclusions

Sharma M, et al. AHRQ Comparative Effectiveness Review No. 16. Available at: http://www.effectivehealthcare.ahrq.gov/ehc/products/11/171/reptbodyfin-typofixed4-12-2010.pdf.

Reporting a systematic review is the final step of the review process.

The report should convey in a transparent manner the methods and results to readers, including consumers, clinicians, and policymakers.

Inadequate reporting makes it more difficult to judge the validity of the methods and results.

Key Messages

Alejandria MM, Lansang MA, Dans LF, et al. Intravenous immunoglobulin for treating sepsis and septic shock. Cochrane Database Syst Rev 2002;(1):CD001090.

Allen C, et al. Non-steroidal anti-inflammatory drugs for pain in women with endometriosis. Cochrane Database Syst Rev 2005;(4):CD004753.

Bolen S, Wilson L, Vassy J, et al. Comparative Effectiveness and Safety of Oral Diabetes Medications for Adults With Type 2 Diabetes. Comparative Effectiveness Review No. 8 (Prepared by The Johns Hopkins University Evidence-based Practice Center under Contract No. 290-02-0018). Rockville, MD: Agency for Health Care Research and Quality, July 2007. AHRQ Publication No. 07-EHC010-EF.

Briel M, Studer M, Glass T, et al. Effects of statins on stroke prevention in patients with and without coronary heart disease: a meta-analysis of randomized controlled trials. Am J Med 2004;117:596-606.

References (I)

Briel M, Schwartz GG, Thompson PL, et al. Effects of early treatment with statins on short-term clinical outcomes in acute coronary syndromes: a meta-analysis of randomized controlled trials. JAMA 2006;295:2046-56.

Bucher HC, Hengstler P, Schindler C, et al. Percutaneous transluminal coronary angioplasty versus medical treatment for non-acute coronary heart disease: meta-analysis of randomised controlled trials. BMJ 2000;321:73-7.

Curioni C, André C. Rimonabant for overweight or obesity. Cochrane Database Syst Rev 2006;(4):CD006162.

Devereaux PJ, Beattie WS, Choi PT, et al. How strong is the evidence for the use of perioperative beta blockers in non-cardiac surgery? Systematic review and meta-analysis of randomised controlled trials. BMJ 2005;331:313-21.

References (II)

Duffett M, Choong K, Ng V, Randolph A, et al. Surfactant therapy for acute respiratory failure in children: a systematic review and meta-analysis. Crit Care 2007;11:R66.

Fuccio L, Minardi ME, Zagari RM, et al. Meta-analysis: duration of first-line proton-pump inhibitor based triple therapy for Helicobacter pylori eradication. Ann Intern Med 2007;147:553-62.

Hodson EM, Craig JC, Strippoli GF, et al. Antiviral medications for preventing cytomegalovirus disease in solid organ transplant recipients. Cochrane Database Syst Rev 2008;(2):CD003774.

Hróbjartsson A, Gøtzsche PC. Placebo interventions for all clinical conditions. Cochrane Database Syst Rev 2004;(1):CD003974.

References (III)

Ip S, Terasawa T, Balk EM, et al. Comparative Effectiveness of Radiofrequency Catheter Ablation for Atrial Fibrillation. Comparative Effectiveness Review No. 15 (Prepared by Tufts–Mew England Medical Center Evidence-based Practice Center under Contract No. 290-02-0022). Rockville, MD: Agency for Healthcare Research and Quality, July 2009. AHRQ Publication No. 09-EHC015-EF.

Jones M, Schenkel B, Just J, et al. Epoetin alfa improves quality of life in patients with cancer: results of metaanalysis. Cancer 2004;101:1720-32.

Lakhdar R, Al-Mallah MH, Lanfear DE. Safety and tolerability of angiotensin-converting enzyme inhibitor versus the combination of angiotensin-converting enzyme inhibitor and angiotensin receptor blocker in patients with left ventricular dysfunction: a systematic review and meta-analysis of randomized controlled trials. J Card Fail 2008;14:181-8.

Liberati A, Altman DF, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Ann Intern Med 2009;151:W65-94.

References (IV)

Liberati A, D'Amico R, Pifferi, et al. Antibiotic prophylaxis to reduce respiratory tract infections and mortality in adults receiving intensive care. Cochrane Database Syst Rev 2004;(1):CD000022.

Mistiaen P, Poot E. Telephone follow-up, initiated by a hospital-based health professional, for postdischarge problems in patients discharged from hospital to home. Cochrane Database Syst Rev 2006;(4):CD004510.

PRISMA Statement Web site. Home page. Available at: http://www.prisma-statement.org/index.htm. Reichenbach S, Sterchi R, Scherer M, et al. Meta-analysis: chondroitin for osteoarthritis of the knee or hip. Ann Intern Med 2007;146:580-90.

Schroth RJ, Hitchon CA, Uhanova J, et al. Hepatitis B vaccination for patients with chromic renal failure. Cochrane Database Syst Rev 2004;(3):CD003775.

References (V)

Shah MR, Hasselvlad V, Stevenson LW, et al. Impact of the pulmonary artery catheter in critically ill patients: meta-analysis of randomzied clinical trials. JAMA 2005;294:1664-70.

Sharma M, Ansari MT, Soares-Weiser K, et al. Comparative Effectiveness of Lipid-Modifying Agents. Comparative Effectiveness Review No. 16 (Prepared by the University of Ottawa Evidence-based Practice Center under contract No. 290-02-0021). Rockville, MD: Agency for Healthcare Research and Quality, September 2009. AHRQ Publication No. 09-EHC024-EF.

Sharma M, Ansari MT, Abou-Setta AM, et al. Systematic review: comparative effectiveness and harms of combination therapy and monotherapy for dyslipidemia. Ann Intern Med 2009;151:622-30.

References (VI)

Tracz MJ, Sideras K, Boloña ER, et al. Testosterone use in men and its effects on bone health. A systematic review and meta-analysis of randomized placebo-controlled trials. J Clin Endocrinol Metab 2006;91:2011-6.

Uitterhoeve RJ, Vernooy M, Litjens M, et al. Psychosocial interventions for patients with advanced cancer — a systematic review of the literature. Br J Cancer 2004;91:1050-62.

Wilt TJ, Shamliyan T, Taylor B, et al. Comparative Effectiveness of Therapies for Clinically Localized Prostate Cancer. Comparative Effectiveness Review No. 13 (Prepared by Minnesota Evidence-based Practice Center under Contract No. 290-02-0009). Rockville, MD: Agency for Healthcare Research and Quality, February 2008. AHRQ Publication No. 08-EHC010-EF.

References (VII)

This presentation was prepared by David Moher, Ph.D., director, University of Ottawa Evidence-based Practice Center.

Many of the examples in the presentations are taken from the PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions (Liberati A, et al. PLoS Med 2009;6(7): e1000100).

Author