updated - what makes systematic review systematic - anna sidorchuk
DESCRIPTION
Updated file! Author: Anna Sidurchuk Presentation held at a Karolinska Institutet Epidemiology Program seminar on April 29th 2013.TRANSCRIPT
What makes a systematic review systematic? Epidemiologic perspectives
ANNA SIDORCHUK, MD, PHD, RESEARCH COORDINATOR
KAROLINSKA INSTITUTET DEPARTMENT OF PUBLIC HEALTH SCIENCES
THE EPIDEMIOLOGY PHD PROGRAM SEMINAR
APRIL 29, 2013
Outline of the speech
Non-systematic vs. systematic review
Why do systematic reviews? What is the added value?
PRISMA/MOOSE/AMSTAR guidelines and checklists
Randomized vs. observational data
Process (criteria, optimize search, determine data to be
abstracted, quality score)
Anna Sidorchuk 2
30 april 2013 Anna Sidorchuk
Primary data versus secondary data
Traditionally, researchers collect and analyze their own data
(referred to as primary data)
Secondary data analysis is based on data collected by someone
else (or reanalysis of your own published data)
Systematic review is one of the ways to analyze secondary data
Systematic review - systematic, qualitative review of published research in a particular field
Adapted from: ESRC Workshop, Researcher Development Initiative, Department of Education, University of Oxford
30 april 2013 Anna Sidorchuk
What The Lancet thinks about systematic reviews
In 2005, the editors wrote:
“. . . we will require authors of clinical trials submitted to The
Lancet to include a clear summary of previous research findings,
and to explain how their trial’s findings affect this summary. The
relation between existing and new evidence should be illustrated
by direct reference to an existing systematic review and meta-
analysis…”
Stages of waste in the production and reporting of research evidence relevant to clinicians and patients by Chalmers and Glasziou (Lancet, vol. 374, 2009)
Anna Sidorchuk 5
30 april 2013 Anna Sidorchuk
Why do we do systematic reviews?
To help busy clinicians to summarize current knowledge in
relation to the area of interest
To provide high-quality research evidence to guide clinical
practice
To support clinical decision-making
To support research proposals
30 april 2013 Anna Sidorchuk
How systematic review can help busy clinicians?
By systematically:
identifying,
appraising,
synthesizing,
and, if appropriate, statistically combining studies on a
specific topic
30 april 2013 Anna Sidorchuk
In classifications of levels of evidence,
systematic reviews are included in the highest level of
evidence
Hierarchy of evidence: a framework for ranking evidence evaluating healthcare interventions
Journal of Clinical Nursing Volume 12, Issue 1, pages 77-84, 20 DEC 2002 DOI: 10.1046/j.1365-2702.2003.00662.x http://onlinelibrary.wiley.com/doi/10.1046/j.1365-2702.2003.00662.x/full#f1
30 april 2013 Anna Sidorchuk
Meta-analysis
A statistical method to combine findings across studies
Should be considered within the framework of systematic reviews
review needs to use a systematic approach to minimize bias, address
the issues of the completeness of the evidence, quality of studies
and combinability of studies
Meta-analysis - the statistical pooling of the results of
studies that are part of a systematic review
When systematic review is required?
Before undertaking a systematic review it is necessary to
check whether there are already existing or ongoing reviews,
and whether a new review is justified
Search:
the Database of Abstracts of Reviews of Effects (DARE)
the Cochrane Database of Systematic Reviews (CDSR)
30 april 2013 Anna Sidorchuk
Adapted from: Systematic Reviews. CRD’s guidance for undertaking reviews in health care (2009). Centre for Reviews and Dissemination, University of York. ISBN 978-1-900640-47-3.
When is meta-analysis appropriate?
There exists a critical mass of comparable studies designed to
address a common research question
Data are presented in a form that allows the meta-analyst to
compute an effect size for each study
Characteristics of each study are described in sufficient detail
to allow meta-analysts to compare characteristics of different
studies and to judge the quality of each study 30 april 2013 Anna Sidorchuk
Systematic reviews vs. non-systematic
Compared to traditional literature reviews:
there is a definite methodology employed in the research analysis
(more like that used in primary research); and
the results of the included studies are quantified to a standard metric
thus allowing for statistical techniques for further analysis.
Therefore process of reviewing research literature is more objective, transparent, and replicable; less biased and idiosyncratic to the whims of a particular researcher
30 april 2013 Anna Sidorchuk
Systematic reviews vs. non-systematic Feature Systematic review Narrative review
Question Often a focused one question Often broad in scope
Sources and search
Comprehensive sources and
explicit search strategy
Not usually specified,
potentially biased
Selection
Criterion-based selection,
uniformly applied
Not usually specified,
potentially biased
Appraisal Rigorous critical appraisal Variable
Synthesis
Qualitative summary that
includes statistical synthesis
(meta-analysis)
Often a qualitative summary
Inferences Usually evidence-based Sometimes evidence-based
Anna Sidorchuk 14
Adapted from: Y Yuan and R H Hunt, Am J Gastroenterol 2009; 104:1086–1092; doi:10.1038/ajg.2009.118
18 September, 2009 Anna Sidorchuk 15
Increase in the number of published meta-analyses (PubMed references)
Publication year
• The essence of good science is replicable and generalisable results
Do we get the same answer to research questions when we run the study again?
• The primary aims of meta-analysis is to test the generalisability of results
across a set of studies designed to answer the same research question
Are the results consistent? If not, what are the differences in the studies that explain
the lack of consistency?
Why are systematic review and meta-analysis important?
30 april 2013 Anna Sidorchuk
When there is systematic variation in outcomes from different studies, meta-analysis tries to explain these differences in terms of study characteristics:
measures used
study design
participant characteristics
controls for potential bias
etc.
If the results of individual studies are inconclusive or controversial…
30 april 2013 Anna Sidorchuk
“… doing a meta-analysis is easy,
doing one well is hard.”
Ingram Olkin, Stanford University
Benefits of systematic reviews, incl. meta-analysis
Increased power: by combining information from many
individual studies, the meta-analyst is able to detect systematic
trends not obvious in the individual studies
Conclusions based on the set of studies are likely to be more
accurate than any one study
30 april 2013 Anna Sidorchuk
Benefits of systematic reviews, incl. meta-analysis
Improved precision: based on information from many studies,
the meta-analyst can provide a more precise estimate of the
population effect size (and a confidence interval)
Provides potential corrections for potential biases,
measurement error and other possible artefacts
Identifies directions for further primary studies to address unresolved issues
30 april 2013 Anna Sidorchuk
Benefits of systematic reviews, incl. meta-analysis
Typically there is study-to-study variation in results. When
this is the case, the meta-analyst can explore what
characteristics of the studies explain these differences (e.g.,
study design) in ways not easy to do in individual studies
Easy to interpret summary statistics (useful if communicating
findings to a non-academic audience)
30 april 2013 Anna Sidorchuk
Limitations of systematic reviews, incl. meta-analysis
Comparing apples and oranges
Quality of the studies included in the meta-analysis
What to do when studies don’t report sufficient information
(e.g., “non-significant” findings)?
Including multiple outcomes in the analysis (e.g., different
achievement scores)
Publication bias
22
Limitations of systematic reviews, incl. meta-analysis
Meta-analysis conclusions may still differ if different studies are sampled or excluded for different reasons
Need to be explicit
Quality standards
Inappropriate handling of data can lead to wrong conclusions
Sample size consideration
23
Publication bias
Studies that are published are more likely to report
statistically significant findings. This is a source of potential
bias
30 april 2013 Anna Sidorchuk
Publication bias The debate about using only published studies:
peer-reviewed studies are presumably of a higher quality
VERSUS
significant findings are more likely to be published than non-
significant findings
There is no agreed upon solution. However, one should retrieve all studies
that meet the eligibility criteria, and be explicit with how they dealt with
publication bias. Some methods for dealing with publication bias have
been developed (e.g., Fail-safe N, Trim and Fill method)
30 april 2013 Anna Sidorchuk
RCT vs. observational studies
Both are fine as long as all the methodological issues are
addressed
If your focus is effect of therapy or prevention – use RCTs
If you are interested in prognosis, etiology, adverse
effects, association between risk factors and outcomes –
use observational studies
Fine to use both types of studies in the same review
Anna Sidorchuk 26
Steps in a meta-analysis Establish research question
Define relevant studies
Develop code
materials
Locate and collate studies
Pilot coding; coding
Data entry and effect
size calculation
Main analyses
Supplementary analyses
30 april 2013 Anna Sidorchuk Adapted from: ESRC Workshop, Researcher Development Initiative, Department of Education, University of Oxford
Practical tips
Define all key elements (PICO model) a priory
Formulate the research question as clear as possible
Make an agreement with all your co-authors on the key elements
If needed ask the experts in the area of outcome or/and exposure for
advice
Keep everything in the protocol
When writing an article include the elements in the aim
30 april 2013 Anna Sidorchuk
Practical tips
Define all inclusion/exclusion criteria and components of search a priory
Make criteria and the components as clear as possible, but feel free to
make some changes during the actual literature search (usually due to
availability of data, e.g. if only retrospective studies are available on the
issue, or no info on ethnicity, etc)
Make an agreement with all co-authors on abovementioned
Again, keep everything in the review protocol
30 april 2013 Anna Sidorchuk
Practical tips
Define coding and form for data extraction a priory
Make an agreement with all co-authors on abovementioned
If use Excel for data extraction and, therefore, for coding, keep the legend
for each coding element within the same file on the separate sheet
As an alternative – use Access
Keep everything in the review protocol!
30 april 2013 Anna Sidorchuk
Practical tips
Define as detailed as possible by reading other meta-analysis and
systematic reviews on the similar outcome(s) and exposure(s):
All key words
MESH terms
Define the electronic sources to be used for your search
Ask librarians for advice on search strategy and technic
Define if grey literature will be included
Keep everything in the protocol!!!
30 april 2013 Anna Sidorchuk
Critical issue
Quality, quality, quality!
30 april 2013 Anna Sidorchuk
Quality Standards in Systematic reviews and Meta-Analysis Several organizations work on this
Cochrane collaboration
http://www.cochrane.org
Campbell collaboration
http://www.campbellcollaboration.org
Consort: Consolidated Standards of Reporting Trials, includes Quorum: Quality of Reporting of Meta-analyses group
http://www.consort-statement.org/mod_product/uploads/QUOROM Statement 1999.pdf
PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses), AMSTAR (A measurement tool to assess systematic reviews)
30 april 2013 Anna Sidorchuk
30 april 2013 Anna Sidorchuk
PRISMA: Checklist of items to include when reporting a systematic review or meta-analysis.
Section/Topic # Checklist Item Reported
on Page #
TITLE
Title 1 Identify the report as a systematic review, meta-analysis, or both.
ABSTRACT
Structured
summary 2
Provide a structured summary including, as applicable:
background; objectives; data sources; study eligibility criteria,
participants, and interventions; study appraisal and synthesis
methods; results; limitations; conclusions and implications of key
findings; systematic review registration number.
INTRODUCTION
Rationale 3 Describe the rationale for the review in the context of what is
already known.
Objectives 4
Provide an explicit statement of questions being addressed with
reference to participants, interventions, comparisons, outcomes,
and study design (PICOS).
30 april 2013 Anna Sidorchuk
PRISMA: Checklist of items to include when reporting a systematic review or meta-analysis.
Section/Topic # Checklist Item Reported
on Page #
METHODS
Protocol and
registration 5
Indicate if a review protocol exists, if and where it can be
accessed (e.g., Web address), and, if available, provide
registration information including registration number.
Eligibility
criteria 6
Specify study characteristics (e.g., PICOS, length of follow-
up) and report characteristics (e.g., years considered,
language, publication status) used as criteria for eligibility,
giving rationale.
Information
sources 7
Describe all information sources (e.g., databases with dates of
coverage, contact with study authors to identify additional
studies) in the search and date last searched.
Search 8
Present full electronic search strategy for at least one
database, including any limits used, such that it could be
repeated.
Study selection 9
State the process for selecting studies (i.e., screening,
eligibility, included in systematic review, and, if applicable,
included in the meta-analysis).
Data collection
process 10
Describe method of data extraction from reports (e.g., piloted
forms, independently, in duplicate) and any processes for
obtaining and confirming data from investigators.
Data items 11
List and define all variables for which data were sought (e.g.,
PICOS, funding sources) and any assumptions and
simplifications made.
30 april 2013 Anna Sidorchuk
PRISMA: Checklist of items to include when reporting a systematic review or meta-analysis.
Section/Topic # Checklist Item Reported
on Page #
METHODS
Risk of bias in
individual studies 12
Describe methods used for assessing risk of bias of
individual studies (including specification of whether this
was done at the study or outcome level), and how this
information is to be used in any data synthesis.
Summary
measures 13
State the principal summary measures (e.g., risk ratio,
difference in means).
Synthesis of
results 14
Describe the methods of handling data and combining
results of studies, if done, including measures of consistency
(e.g., I2) for each meta-analysis.
Risk of bias
across studies 15
Specify any assessment of risk of bias that may affect the
cumulative evidence (e.g., publication bias, selective
reporting within studies).
Additional
analyses 16
Describe methods of additional analyses (e.g., sensitivity or
subgroup analyses, meta-regression), if done, indicating
which were pre-specified.
30 april 2013 Anna Sidorchuk
PRISMA: Checklist of items to include when reporting a systematic review or meta-analysis.
Section/Topic # Checklist Item Reported
on Page #
RESULTS
Study selection 17
Give numbers of studies screened, assessed for eligibility,
and included in the review, with reasons for exclusions at
each stage, ideally with a flow diagram.
Study
characteristics 18
For each study, present characteristics for which data were
extracted (e.g., study size, PICOS, follow-up period) and
provide the citations.
Risk of bias
within studies 19
Present data on risk of bias of each study and, if available,
any outcome-level assessment (see Item 12).
Results of
individual studies 20
For all outcomes considered (benefits or harms), present, for
each study: (a) simple summary data for each intervention
group and (b) effect estimates and confidence intervals,
ideally with a forest plot.
Synthesis of
results 21
Present results of each meta-analysis done, including
confidence intervals and measures of consistency.
Risk of bias
across studies 22
Present results of any assessment of risk of bias across
studies (see Item 15).
Additional
analysis 23
Give results of additional analyses, if done (e.g., sensitivity
or subgroup analyses, meta-regression [see Item 16]).
30 april 2013 Anna Sidorchuk
PRISMA: Checklist of items to include when reporting a systematic review or meta-analysis.
Section/Topic # Checklist Item Reported
on Page #
DISCUSSION
Summary of
evidence 24
Summarize the main findings including the strength of
evidence for each main outcome; consider their relevance to
key groups (e.g., health care providers, users, and policy
makers).
Limitations 25
Discuss limitations at study and outcome level (e.g., risk of
bias), and at review level (e.g., incomplete retrieval of
identified research, reporting bias).
Conclusions 26 Provide a general interpretation of the results in the context of
other evidence, and implications for future research.
FUNDING
Funding 27
Describe sources of funding for the systematic review and
other support (e.g., supply of data); role of funders for the
systematic review.
Quality Assessment of Primary Studies
A common practice is to assign a quality score to primary studies
Preferably based on an explicit checklist usually
The judgment to include or exclude a study must be stated
and justified explicitly
This judgment is mostly based on quantitative criteria, but
involves some subjectivity
30 april 2013 Anna Sidorchuk
NEWCASTLE - OTTAWA QUALITY ASSESSMENT SCALE CASE CONTROL STUDIES
Note: A study can be awarded a maximum of one star for each numbered item within the Selection and Exposure categories. A maximum of two stars can be given for Comparability.
Selection
1) Is the case definition adequate? a) yes, with independent validation ¯ b) yes, eg record linkage or based on self reports
c) no description
2) Representativeness of the cases a) consecutive or obviously representative series of cases ¯
b) potential for selection biases or not stated
3) Selection of Controls a) community controls ¯ b) hospital controls
c) no description
4) Definition of Controls
a) no history of disease (endpoint)
b) no description of source
30 april 2013 Anna Sidorchuk
NEWCASTLE - OTTAWA QUALITY ASSESSMENT SCALE CASE CONTROL STUDIES
Comparability
1) Comparability of cases and controls on the basis of the design or analysis
a) study controls for _______________ (Select the most important factor.) ¯
b) study controls for any additional factor ¯ (This criteria could be modified to indicate specific control for a second important factor.)
Exposure
1) Ascertainment of exposure
a) secure record (eg surgical records) ¯
b) structured interview where blind to case/control status ¯
c) interview not blinded to case/control status
d) written self report or medical record only
e) no description
2) Same method of ascertainment for cases and controls
a) yes ¯
b) no
3) Non-Response rate
a) same rate for both groups ¯
b) non respondents described
c) rate different and no designation
30 april 2013 Anna Sidorchuk
“Ugly face” of systematic reviews and meta-analyses
Can be misleading
Can be misused
30 april 2013 Anna Sidorchuk