1
Disc laimer: The contents of this presentation are the views of the author and do not necessarily represent an offic ial position of the European C ommission. © European Union, 2013
GRADE workshop:
Meta-analysis - the basics
Juan A Blasco Amaro, MD MPH
Public Health Policy Support Unit
Institute for Health and Consumer Protection
(JRC-IHCP)
Joint Research Centre
The European Commission’s
in-house science service
2 GRADE Workshop – Ispra– 11-12 December 2013
GRADE Workshop: Agenda
9:00 - 10:15 Introduction. Guideline development process and the GRADE approach
10:15-11:00 Types of questions. Framing a question: PICO question. Exercise
11:00-11:15 Coffee
11:15-12:00 Choosing outcomes. Relative importance of outcomes. Exercise
12:00-12:45 Study designs. Exercise
12:45-13:30 Lunch
13:30-14:30 Search of the literature. Exercise
14:30-15:00 Determinants of quality of evidence: What can lower the evidence? (I)
15:00-15:15 Coffee
15:15-17:00 Determinants of quality of evidence: What can lower the evidence? (II) Exercise
9:00-11:15 Determinants of QoE: What can lower the evidence? (III). ExerciseDeterminants of quality of evidence: What can upgrade evidence?
11:15-11:30 Coffee
11:30-13:00 Going from the evidence to the recommendation. Exercise
13:00-13:45 Lunch
13:45-16:00 Using the Guideline Development Tool (GDT) software. ExerciseFeedback and conclusions
3 GRADE Workshop – Ispra– 11-12 December 2013
a. Introduction
b. Representation
c. Imprecision
d. Inconsistency
e. Publication Bias
f. Exercise
Outline
4 GRADE Workshop – Ispra– 11-12 December 2013
¿What is a meta-analysis?
a) A combination of studies
b) Estimation of a common value
c) A graphical plot
d) A type of Systematic Review
e) Combined analyses of randomized
clinical trials
f) Research project
g) Observational study
h) Statistical method
5 GRADE Workshop – Ispra– 11-12 December 2013
Meta-analysis History
� -Background
� -Karl Pearson, 1904
� -First “meta-analisys” published assessing
intervention, what intervention?
- it was effective in 35%!
• -Term is introduced by Glass in 1976 and
became popular in Social Sciences
• - In 1992, a network of epidemiologists and
healthcare professionals -> Cochrane
Collaboration
6 GRADE Workshop – Ispra– 11-12 December 2013
Evolution - meta published in medline
20042005
20062007
20082009
20102011
20122013
0
1000
2000
3000
4000
5000
6000
7000
8000
7000-8000
6000-7000
5000-6000
4000-5000
3000-4000
2000-3000
1000-2000
0-1000
2
7 GRADE Workshop – Ispra– 11-12 December 2013
Meta-analysis
Define Question
Search forStudies
CollectData
HeterogeneityAnalysis
Combining effects
Interpret resultsAnd
Draw conclusions
Apply elegibility criteria
Assess studies for Risk of Bias
Sensitivity analysis Publication bias
Graphical plot
Heterogeneity measures
8 GRADE Workshop – Ispra– 11-12 December 2013
Review level
↓
Effect measure
Study A
Effect measure
Outcome data
Effect measure
Outcome data
Study B
Effect measure
Outcome data
Study C
Effect measure
Outcome data
Study D
Study level
↓
Source: Jo McKenzie & Miranda
Cumpston
9 GRADE Workshop – Ispra– 11-12 December 2013
What is a meta-analysis?
• combines the results from two or more studies
• estimates an ‘average’ or ‘common’ effect
• optional part of a systematic review
Systematic reviews
Meta-analyses
10 GRADE Workshop – Ispra– 11-12 December 2013
Source: Julian Higgins
Why perform a meta-analysis?
� quantify treatment effects and their uncertainty
� increase power
� increase precision
� explore differences between studies
� settle controversies from conflicting studies
� generate new hypotheses
11 GRADE Workshop – Ispra– 11-12 December 2013
When not to do a meta-analysis
• mixing apples with oranges• each included study must address same question
consider comparison and outcomes
requires your subjective judgement
• garbage in – garbage out• a meta-analysis is only as good as the studies in it
• if included studies are biased:
meta-analysis result will also be incorrect
will give more credibility and narrower confidence interval
• if serious reporting biases present:
unrepresentative set of studies may give misleading result
Source: Julian Higgins
12 GRADE Workshop – Ispra– 11-12 December 2013
When can you do a meta-analysis?
• more than one study has measured an effect
• the studies are sufficiently similar to produce a meaningful and
useful result
• the outcome has been measured in similar ways data are
available in a format we can use
3
13 GRADE Workshop – Ispra– 11-12 December 2013
Software (I)
• MetaXL software page
• ClinTools (commercial)
• Comprehensive Meta-Analysis (commercial)
• MIX 2.0 Professional Excel addin with Ribbon interface for meta-
analysis (free and commercial versions).
• Meta-analysis features for Stata? (free add-ons to commercial package)
• The Meta-Analysis Calculator free on-line tool for conducting a
meta-analysis
• Metastat (Free)
• Meta-Analyst Free Windows-based tool
• ProMeta Professional software for meta-analysis Java (commercial)
• Revman A free software for meta-analysis
• Metafor-project A free software package for meta-analyses in R
• Macros in SPSS Free Macros to conduct meta-analyses in SPSS
• MAd GUI User friendly graphical user interface package for R (Free)
14 GRADE Workshop – Ispra– 11-12 December 2013
Software (IV)Software (III)
15 GRADE Workshop – Ispra– 11-12 December 2013
¿What’s behind the
“software”?
16 GRADE Workshop – Ispra– 11-12 December 2013
Statistical analysis
17 GRADE Workshop – Ispra– 11-12 December 2013
Meta-analysis is typically a two-stage process
• a summary statistic is calculated for each study, to describe the observed intervention effect.
• summary (pooled) intervention effect estimate is calculated as a weighted average of the intervention effects estimated in the individual studies
18 GRADE Workshop – Ispra– 11-12 December 2013
Steps in a meta-analysis
• identify comparisons to be made
• identify outcomes to be reported and statistics to be used
• collect data from each relevant study
• combine the results to obtain the summary of effect
• explore differences between the studies
• interpret the results
4
19 GRADE Workshop – Ispra– 11-12 December 2013
Calculating the summary result
• collect a summary statistic from each contributing study
• how do we bring them together?
• treat as one big study – add intervention & control data?
breaks randomisation, will give the wrong answer
• simple average?
weights all studies equally – some studies closer to the truth
• weighted average
20 GRADE Workshop – Ispra– 11-12 December 2013
Headache Caffeine Decaf Weight
Amore-Coffea 2000
2/31 10/34
Deliciozza 2004 10/40 9/40
Mama-Kaffa 1999 12/53 9/61
Morrocona 1998 3/15 1/17
Norscafe 1998 19/68 9/64
Oohlahlazza 1998 4/35 2/37
Piazza-Allerta 2003
8/35 6/37
For example
21 GRADE Workshop – Ispra– 11-12 December 2013
Headache Caffeine Decaf Weight
Amore-Coffea 2000
2/31 10/34 6.6%
Deliciozza 2004 10/40 9/40 21.9%
Mama-Kaffa 1999 12/53 9/61 22.2%
Morrocona 1998 3/15 1/17 2.9%
Norscafe 1998 19/68 9/64 26.4%
Oohlahlazza 1998 4/35 2/37 5.1%
Piazza-Allerta 2003
8/35 6/37 14.9%
For example
22 GRADE Workshop – Ispra– 11-12 December 2013
Overview
23 GRADE Workshop – Ispra– 11-12 December 2013 24 GRADE Workshop – Ispra– 11-12 December 2013
5
25 GRADE Workshop – Ispra– 11-12 December 2013
• for dichotomous or continuous data
• inverse-variancestraightforward, general method
• for dichotomous data only
• Mantel-Haenszel (default)good with few events – common in Cochrane reviews
weighting system depends on effect measure
• Petofor odds ratios only
good with few events and small effect sizes (OR close to 1)
Meta-analysis options
26 GRADE Workshop – Ispra– 11-12 December 2013
a. Introduction
b. Representation
c. Imprecision
c. Inconsistency
d. Publication Bias
e. Exercise
Outline
27 GRADE Workshop – Ispra– 11-12 December 2013
A forest of lines
28 GRADE Workshop – Ispra– 11-12 December 2013
Forest plots
Headache at 24 hours
• headings explain the comparison
29 GRADE Workshop – Ispra– 11-12 December 2013
Forest plots
Headache at 24 hours
• list of included studies
30 GRADE Workshop – Ispra– 11-12 December 2013
Headache at 24 hours
• effect estimate for each study, with CI
Forest plots
6
31 GRADE Workshop – Ispra– 11-12 December 2013
Headache at 24 hours
• scale and direction of benefit
Forest plots
32 GRADE Workshop – Ispra– 11-12 December 2013
Headache at 24 hours
• pooled effect estimate for all studies, with CI
Forest plots
33 GRADE Workshop – Ispra– 11-12 December 2013
a. Introduction
b. Representation
c. Imprecision
d. Inconsistency
e. Publication Bias
f. Exercise
Outline
34 GRADE Workshop – Ispra– 11-12 December 2013
Interpreting confidence intervals
• always present estimate with a confidence interval
• precision• point estimate is the best guess of the effect
• CI expresses uncertainty – range of values we can be reasonably
sure includes the true effect
• significance• if the CI includes the null value
rarely means evidence of no effect
effect cannot be confirmed or refuted by the available evidence
• consider what level of change is clinically important
35 GRADE Workshop – Ispra– 11-12 December 2013
When are results imprecise?
• For dichotomous outcomes
• Sample size, optimal information size
• Number of events
• For continuous outcomes
• minimal important difference
Assessing precision of the Results
36 GRADE Workshop – Ispra– 11-12 December 2013
a. Introduction
b. Representation
c. Imprecision
d. Inconsistency
e. Publication Bias
f. Exercise
Outline
7
37 GRADE Workshop – Ispra– 11-12 December 2013
Heterogeneity
• clinical diversity (sometimes called clinical heterogeneity)
• methodological diversity (sometimes called methodological
heterogeneity)
• statistical heterogeneity simply as heterogeneity
37 38 GRADE Workshop – Ispra– 11-12 December 2013
Clinical diversity
• participants
• e.g. condition, age, gender, location, study eligibility criteria
• interventions
• intensity/dose, duration, delivery, additional components, experience of practitioners, control (placebo, none, standard care)
• outcomes
• follow-up duration, ways of measuring, definition of an event, cut-off points
39 GRADE Workshop – Ispra– 11-12 December 2013
Methodological diversity
• design
• e.g. randomised vs non-randomised, crossover vs parallel,
individual vs cluster randomised
• conduct
• e.g. risk of bias (allocation concealment, blinding, etc.), approach to analysis
40 GRADE Workshop – Ispra– 11-12 December 2013
Statistical heterogeneity
• there will always be some random (sampling) variation between the results of different studies
• heterogeneity is variation between the effects being evaluated in the different studies
• caused by clinical and methodological diversity
• alternative to homogeneity (identical true effects underlying every study)
• study results will be more different from each other than if random variation is the only reason for the differences between the estimated intervention effects
41 GRADE Workshop – Ispra– 11-12 December 2013
Identifying heterogeneity
• Visual inspection of the forest plots
• Chi-squared (χχχχ2) test (Q test)
• I2 statistic to quantify heterogeneity
42 GRADE Workshop – Ispra– 11-12 December 2013
Visual inspection
Forest plot A Forest plot B
8
43 GRADE Workshop – Ispra– 11-12 December 2013
Cochran’s Q and his test
• Cochran's Q statistic, (χχχχ2) :
• Follow a distribution c2 with k-1 degrees of freedom
• (k = number of studies).
• Low power for few studies (low k).
• Statistic I2 :
• I2 = 100% x (Q – [k-1])/Q
• Percentage of variation related to heterogeinity and not to random.
44 GRADE Workshop – Ispra– 11-12 December 2013
Thresholds for the interpretation of I2 can be misleading
• 25% low - might not be important;
• 50% moderate - may represent moderate heterogeneity;
• 75% high - may represent substantial heterogeneity;
• 75% to 100% considerable heterogeneity.
44
45 GRADE Workshop – Ispra– 11-12 December 2013
Example
45 46 GRADE Workshop – Ispra– 11-12 December 2013
The I2 statistic
47 GRADE Workshop – Ispra– 11-12 December 2013
Fixed-effect vs random-effects
• Two models for meta-analysis available in RevMan
• Make different assumptions about heterogeneity
48 GRADE Workshop – Ispra– 11-12 December 2013
Fixed-effect model
• assumes all studies are measuring the same treatment effect
• estimates that one effect
• if not for random (sampling) error, all results would be identical
Common
Random (sampling)error
true effect
Studyresult
Source: Julian Higgins
9
49 GRADE Workshop – Ispra– 11-12 December 2013
Random-effects model
Randomerror
Study-specificeffect
Mean of trueeffects
• assumes the treatment effect varies
between studies
• estimates the mean of the
distribution of effects
• weighted for both
within-study and between-study
variation (tau2, τ2)
Source: Julian Higgins
50 GRADE Workshop – Ispra– 11-12 December 2013
No heterogeneity
Adapted from Ohlsson A, Aher SM. Early erythropoietin for preventing red blood cell transfusion in
preterm and/or low birth weight infants. Cochrane Database of Systematic Reviews 2006, Issue 3.
Fixed Random
51 GRADE Workshop – Ispra– 11-12 December 2013
Some heterogeneity
Fixed Random
Adapted from Adams CE, Awad G, Rathbone J, Thornley B. Chlorpromazine versus
placebo for schizophrenia. Cochrane Database of Systematic Reviews 2007, Issue 2.
52 GRADE Workshop – Ispra– 11-12 December 2013
Which to choose?
• Do you expect your results to be very diverse?
• Consider the underlying assumptions of the model
• fixed-effect
may be unrealistic – ignores heterogeneity
• random-effects
allows for heterogeneity
estimate of distribution of studies may not be accurate if biases are present, few studies or few events
53 GRADE Workshop – Ispra– 11-12 December 2013
What to do about heterogeneity
• Check that the data are correct
• Especially if the direction of effect varies if heterogeneity is very high
• Interpret fixed-effect results with cautionconsider sensitivity analysis – would random-effects have made an important
difference?
• May choose not to meta-analyseaverage result may be meaningless in practice
consider clinical & methodological comparability of studies
• Avoidchanging your effect measure or analysis model
excluding outlying studies
• explore heterogeneity
54 GRADE Workshop – Ispra– 11-12 December 2013
Exploring your results
• what is heterogeneity?
• assumptions about heterogeneity
• identifying heterogeneity
• exploring your results
10
55 GRADE Workshop – Ispra– 11-12 December 2013
Two methods available
• subgroup analysis
• Group studies by pre-specified factors
• look for differences in results and heterogeneity
• meta-regression
• examine interaction with categorical and continuous variables
• not available in RevMan
56 GRADE Workshop – Ispra– 11-12 December 2013
Participant subgroups
Based on Stead LF, Perera R, Bullen C, Mant D, Lancaster T. Nicotine replacement therapy for smoking cessation. Cochrane Database of Systematic Reviews 2008, Issue 1. Art. No.: CD000146. DOI: 10.1002/14651858.CD000146.pub3.
57 GRADE Workshop – Ispra– 11-12 December 2013
Intervention subgroups
Based o
n L
inde K
, B
ern
er
MM
, K
risto
n L
. St
John's
wort
for
majo
r depre
ssio
n. Cochra
ne D
ata
base o
f Syste
matic R
evie
ws
2008,
Issue 4
. Art
. N
o.:
CD
000448. D
OI:
10.1
002/1
4651858.C
D000448.p
ub3.
58 GRADE Workshop – Ispra– 11-12 December 2013
Sensitivity analysis
• not the same as subgroup analysis
• testing the impact of decisions made during the review
• inclusion of studies in the review
• definition of low risk of bias
• choice of effect measure
• assumptions about missing data
• cut-off points for dichotomised ordinal scales
• correlation coefficients
• repeat analysis using an alternative method or assumption
• don’t present multiple forest plots – just report the results
• if difference is minimal, can be more confident of conclusions
• if difference is large, interpret results with caution
59 GRADE Workshop – Ispra– 11-12 December 2013
Adapted from Li J, Zhang Q, Zhang M, Egger M. Intravenous magnesium for acute myocardial infarction. Cochrane Database of
Systematic Reviews 2007, Issue 2.
Sensitivity analysis
60 GRADE Workshop – Ispra– 11-12 December 2013
Differences in underlying treatment effect
When heterogeneity exists, but investigators fail to identify a plausible
explanation, the quality of evidence should be downgraded by one or two
levels, depending on the magnitude of the inconsistency in the results
Inconsistency may arise from differences in:
• populations (e.g. drugs may have larger relative effects in sicker
populations) • interventions (e.g. larger effects with higher drug doses)
• outcomes (e.g. diminishing treatment effect with time)
*Proposed by GRADE Working Group
Assessing inconsistency *
11
61 GRADE Workshop – Ispra– 11-12 December 2013
a. Introduction
b. Representation
c. Imprecision
d. Inconsistency
e. Publication Bias
f. Exercise
Outline
62 GRADE Workshop – Ispra– 11-12 December 2013
Funnel plots
• plot effect size against study size
• study size usually indicated by a measure like standard error
• studies will be scattered around the effect estimate
• larger studies at the top, smaller studies further down
• small studies expected to scatter more widely
• a symmetrical plot will look like an inverted funnel or triangle
• RevMan can generate funnel plots
• only appropriate with ≥ 10 studies of varying size
63 GRADE Workshop – Ispra– 11-12 December 2013
Symmetrical funnel plot
Sta
nd
ard
Erro
r
EffectSource: Matthias Egger & Jonathan Sterne
0.1 0.33 1 33
2
1
0
100.6
64 GRADE Workshop – Ispra– 11-12 December 2013
Asymmetrical funnel plot
0.1 0.33 1 33
2
1
0
100.6
Effect
Sta
nd
ard
Erro
r
Source: Matthias Egger & Jonathan Sterne
Unpublished studies
65 GRADE Workshop – Ispra– 11-12 December 2013
Colloids vs crystalloids for fluid resuscitation
Adapted from Perel P, Roberts I. Colloids versus crystalloids for fluid resuscitation in critically ill patients. Cochrane Database of
Systematic Reviews 2011, Issue 3.
Death
66 GRADE Workshop – Ispra– 11-12 December 2013
Magnesium for myocardial infarction
Adapted from Li J, Zhang Q, Zhang M, Egger M. Intravenous magnesium for acute myocardial infarction. Cochrane Database of
Systematic Reviews 2007, Issue 2.
12
67 GRADE Workshop – Ispra– 11-12 December 2013
Reasons for funnel plot asymmetry
1. chance
• artefact
• some statistics are correlated to SE, e.g. OR
• clinical diversity
• different populations different in small studies
• different implementation different in small studies
• methodological diversity
• greater risk of bias in small studies
• reporting biases (publication bias)
Source: Egger M et al. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997; 315: 629
68 GRADE Workshop – Ispra– 11-12 December 2013
Outline
• understanding reporting biases
69 GRADE Workshop – Ispra– 11-12 December 2013
unavailable(unpublished)
available in principle(thesis,
conference, small journal)
easily available(Medline-indexed)
activelydisseminated(news, drug company)
Source: Matthias Egger
The dissemination of evidence
70 GRADE Workshop – Ispra– 11-12 December 2013
Reporting biases
• dissemination of research findings is influenced by the nature and direction of results
• statistically significant, ‘positive’ results more likely to be published…
• …therefore more likely to be included
• leads to exaggerated effects
• large studies likely to be published anyway, so small studies most likely to be affected
• non-significant results are as important to your review as significant results
71 GRADE Workshop – Ispra– 11-12 December 2013
Evidence for reporting bias
Source: Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research
projects BMJ 1997;315:640-645.
Proportion of
studies not
published
Years since conducted
SignificantNon-significant trendNull
72 GRADE Workshop – Ispra– 11-12 December 2013
Conceived
Performed
Submitted
Cited
Published
Positive studies are more likely to be
• submitted for publication...
• …and accepted (publication bias)
• …quickly (time lag bias)
• …as more than one paper(multiple publication bias)
• …in English (language bias)
• …in high-impact, indexed journals (location bias)
• … including positive outcomes (selective outcome reporting)
• …and cited by others (citation bias)
Source: Julian Higgins
13
73 GRADE Workshop – Ispra– 11-12 December 2013
Publication bias is a systematic underestimate or an overestimate of the underlying beneficial or harmful effect
• Investigators fail to report studies the have undertaken
• If meta-analysis is influenced, downgrade the quality of evidence
*Proposed by GRADE Working Group
Assessing publication bias *
JRC xxxxx – © European Union, 2013Disc laimer: The contents of this presentation are the views of the author and do not necessarily represent an offic ial position of the European C ommission.
Determinants of quality of evidence: What can upgrade evidence?