introduction to meta-analysis - newcastle university · when to conduct meta-analysis •a...
TRANSCRIPT
![Page 1: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/1.jpg)
Introduction to Meta-analysis
Stephen RiceIHS Evidence Synthesis Group
1
![Page 2: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/2.jpg)
What is meta-analysis?
“The combination of data or results from a collection of studies which address a common scientific question”
Glass 1977
‘Meta’ means ‘after’ or ‘beyond’
2
![Page 3: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/3.jpg)
A Quantitative Systematic Review Question
Systematic reviews can designed to address a variety of questions:
- What is the treatment effectiveness of medication X
- QUANTITATIVE
- What are the barriers and facilitators to the implementation of a training programme for midwives to implement X
- QUALITATIVE
(A typology of reviews: an analysis of 14 review types and associated methodologies. Grant M, Booth A. Health Information and Libraries Journal, 26, pp.91–108)
3
![Page 4: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/4.jpg)
What are you estimating?
4
You are trying to estimate a statistic for a specific population, such as a treatment effect, risk of a complication, the sensitivity and specificity of a diagnostic test, a correlation between two continuous variables.
![Page 5: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/5.jpg)
The Sample
5
The sample in a randomised controlled trial consists of patients.
The sample in a meta-analysis consists of studies.
There is a population of studies that have been conducted, with varying populations, interventions, outcomes and biases. Some are published in journals, others languish in PhD theses or as conference abstracts.
The sample of studies should be as representative of the available evidence as possible. The best available evidence should be sought. This requires a systematic approach to identify the evidence, and consequently meta-analyses are often conducted as part of systematic reviews of the literature.
![Page 6: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/6.jpg)
Similarity of study characteristics?
6
Same population: if there are several published studies investigating the same population, then incorporating all studies will - increase the sample size, the amount of evidence- have the advantage of including studies demonstrating the
reproducibility of the experiment
Different population: clinical studies may use specific populations in order to maximise the outcome prevalence and therefore allow a lower sample size, or because it is a convenient population for the clinical researchers. Including studies of slightly different, but related populations may allow generalisation to a slightly broader population, and with more evidence.
![Page 7: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/7.jpg)
Benefits of meta-analysis?
• Increase in precision: less uncertainty around the estimate of the outcome
• Extra power: more likely to reject a hypothesis if it is in fact false
• Estimate heterogeneity: variation in true effects
Greater power
Greater precision
Sample size
Generalise to broader
population
Describe & explore
heterogeneity
Heterogeneity
7
![Page 8: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/8.jpg)
Meta-analysis: a statistical method
8
Meta-analysis is a statistical method that synthesises the results from individual studies to estimate the statistic of interest.
What are the data that may be used in a meta-analysis?
![Page 9: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/9.jpg)
Mean, Standard deviation and Standard errorThe mean (average) is the sum of the sample values divided by the
sample size
A standard deviation describes the variation in a population
The effect estimate (mean difference) is the difference in the mean values of the intervention and control arms of a study
A standard error describes the uncertainty in the estimate of the mean or in the mean difference
9
![Page 10: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/10.jpg)
Deriving study summary statistics: continuous data
Study 1 Study 2
Morphine Paracetamol Morphine Paracetamol
2.8 4 3 3.4
3 3.6 3.3 4.2
2 4.5 2.5 3.8
4 4.1 2.4 4
3.5 3.8 3.8 4.6
2.4 3.4 3.1 3.8
3.2 5 3.6 3.6
2.6 3.2 3.2 4.2
N 8 8 N 8 8
Mean 2.9375 3.95 Mean 3.1125 3.95
SD 0.6346 0.5904 SD 0.4853 0.3817
10
![Page 11: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/11.jpg)
Deriving summary study summary statistics• Study 1
• 𝑀𝑒𝑎𝑛 𝑑𝑖𝑓𝑓𝑒𝑟𝑒𝑛𝑐𝑒 = 3.95 − 2.94 = 1.01
• 𝑆𝐸(𝑑𝑖𝑓𝑓𝑒𝑟𝑒𝑛𝑐𝑒) =𝑆𝐷1
2
𝑁1+
𝑆𝐷22
𝑁2=0.31
• 95% CI: (1.01-0.31*1.96) to (1.01+0.31*1.96) =0.41 to 1.61
• Study 2
• 𝑀𝑒𝑎𝑛 𝑑𝑖𝑓𝑓𝑒𝑟𝑒𝑛𝑐𝑒 = 3.95 − 3.11 = 0.84
• 𝑆𝐸(𝑑𝑖𝑓𝑓𝑒𝑟𝑒𝑛𝑐𝑒) =𝑆𝐷1
2
𝑁1+
𝑆𝐷22
𝑁2=0.22
• 95% CI: (1.01-0.22*1.96) to (1.01+0.22*1.96) =0.41 to 1.2711
![Page 12: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/12.jpg)
What is meta-analysis?
Study NT MeanT SDT NC MeanC SDC
1 100 4.87 0.97 75 5.03 0.86
2 155 6.85 0.97 135 7.12 0.74
3 100 8.04 0.97 90 8.08 0.75
4 50 7.47 1.15 40 7.54 0.97
Mean difference (95% CI)
-0.16 (-0.43 to 0.11)
-0.27 (-0.47 to -0.07)
-0.04 (-0.29 to 0.21)
-0.07 (-0.51 to 0.37)
Summary
-0.17 (-0.29 to -0.04)12
![Page 13: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/13.jpg)
Line of no effect
Point estimate
Favours controlFavours intervention
Pooled result
Forest plots
Study N
A 175
B 290
C 190
D 90
95% Confidence Interval
Weight of trial
Effect estimate
95% Confidence Interval
Meta-analysis estimate
13
![Page 14: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/14.jpg)
Greater precision
Study
A
B
C
D
Meta-analysis
The confidence interval of the pooled estimate is narrower than the confidence intervals of the individual studies
14
![Page 15: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/15.jpg)
Combining studies: inverse variance
Study NT MeanT SDT NC MeanC SDC
1 100 4.77 0.84 75 4.96 0.7
2 155 6.75 0.99 135 7.06 0.85
3 100 7.72 0.92 90 8.01 0.83
4 50 7.36 0.91 40 7.47 0.65
Mean SE
-0.19 0.12
-0.31 0.11
-0.29 0.13
-0.11 0.16
Possible study weights Summary effect estimate
Equal −0.225
Trial sample size −0.253
Inverse variance −0.242
Inverse variance =1
𝑣𝑎𝑟=
1
𝑆𝐸𝑀𝐷2
MD: mean difference; difference in mean effects between trial arms
15
![Page 16: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/16.jpg)
Study weights: inverse variance
Inverse variance =1
𝑣𝑎𝑟=
1
𝑆𝐸𝐼𝐸2
In an inverse variance meta-analysis, the weight given to each study is the inverse of the variance of the intervention effect in each study
var: varianceSE: standard errorIE: intervention effect, e.g. difference in mean pain scores between trial arms
16
![Page 17: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/17.jpg)
Fixed-effect meta-analysis
Study weights
17
![Page 18: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/18.jpg)
Variables and Outcomes
Variable type Example variable Outcome
Continuous Pain Mean difference
Dichotomous (Binary) Heart attack Risk ratio/odds ratio
Time-to-event Time to death Hazard ratio
18
![Page 19: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/19.jpg)
Dichotomous (binary) outcomes: 2-by-2 table
Event No event Total
Treatment a=10 b=40 NT=50
Control c=20 d=30 NC=50
Total N=100
The risk of an event in the treatment group is a/(a+b) = 10/50 = 0.2
The risk of an event in the control group is c/(c+d) = 20/50 = 0.4
19
![Page 20: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/20.jpg)
Equal effects = 1 for a ratio
• When the risk of the outcome is the same in both treatment arms
𝑅𝑅 = 1
• When the odds of the outcome is the same in both treatment arms
𝑂𝑅 = 1
20
![Page 21: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/21.jpg)
Further reading- binary outcomes• Hancock M, Kent P. Interpretation of dichotomous outcomes: risk, odds, risk
ratios, odds ratios and number needed to treat. Journal of Physisotherapy. Vol. 62, Issue 3, 2016, P172-174.
• Sackett DL, Deeks, JJ, Altman DJ. Down with odds ratios! BMJ September/October 1996.
http://ebm.bmj.com/content/ebmed/1/6/164.full.pdf
• Borenstein M et al. Introduction to Meta-analysis. Chapter 5.
• Grant RL. Converting an odds ratio to a range of plausible relative risks for better communication of research findings. BMJ. 2014;348:f7450.
http://www.bmj.com/content/348/bmj.f7450.full.print21
![Page 22: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/22.jpg)
Meta-analysis process for binary data
Study eT NT eC NC
1 24 240 40 240
2 32 224 36 240
3 28 280 35 264
4 20 224 28 208
LN(RR) SE
-0.51 0.24
-0.05 0.22
-0.28 0.24
-0.41 0.28
Ln(RR) (95% CI): -0.30 (-0.53 to -0.06)RR (95% CI): 0.74 (0.59 to 0.94)NNT??
22
![Page 23: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/23.jpg)
Fixed effect meta-analysis on RR scaleAlthough meta-analysis is conducted on the log scale, all the data and results are presented on a scale such as risk ratio or odds ratio.
23
![Page 24: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/24.jpg)
Zero events: inverse-variance and Mantel-Haenszel method• Add ½ to every cell of a 2x2 table (Revman automatically does this)
24
![Page 25: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/25.jpg)
Clinical heterogeneity (diversity)
Study Mean Age % Female Follow up time point
Mean difference in pain score
1 55 45 4 weeks 0.75
2 50 70 4 weeks 1
3 56 65 1 week 0.7
4 40 60 3 weeks 1.2
Clinical heterogeneity is variation in patient, intervention and outcome characteristics
25
![Page 26: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/26.jpg)
Statistical heterogeneity
Low statistical heterogeneity High statistical heterogeneity
26
![Page 27: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/27.jpg)
Clinical & Statistical heterogeneity
Clinical heterogeneity (variation, diversity) is variation between studies in patient, intervention and outcome characteristics
Statistical heterogeneity is greater variation in study effects than would be expected due to chance (sampling error)
Clinical heterogeneity may result in statistical heterogeneity where clinical variation affects the intervention effect
Sometimes, clinical heterogeneity is referred to as clinical variation and statistical heterogeneity as simply heterogeneity
27
![Page 28: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/28.jpg)
Fixed and random effects meta-analysis
• A fixed effect meta-analysis assumes that there is one true intervention effect. Variation between studies is purely due to chance.
• A random-effects meta-analysis does not assume that there is one true effect.
- Because of clinical heterogeneity, it assumes that the true intervention effect varies from one study to the next, and these effects are random samples from a normal distribution.
- Hence, the pooled estimate is the average effect from the PICO characteristics of the included studies
28
![Page 29: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/29.jpg)
Random-effects meta-analysisVariation in intervention effects greater than chanceSome confidence intervals don’t even overlap
29
![Page 30: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/30.jpg)
When to conduct meta-analysis
• A meta-analysis is a statistical technique used to answer a question specified in the protocol of a systematic review
• If the question is relevant, the inclusion criteria are relevant, and a meta-analysis is planned to estimate a statistic then conduct the meta-analysis
• Any meta-analyses of subgroups should be stated in the protocol
• The presence of statistical heterogeneity is not a reason to abandon a meta-analysis; it affects how you interpret the results and the conclusions you draw
30
![Page 31: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/31.jpg)
Limitations of meta-analysis
• If the included studies are poor quality, the summary result will also be poor. The uncertainty associated with the potential bias in the included studies will not be captured in the confidence interval of the summary estimate.
• The value of a meta-analysis depends on the identification of the relevant studies from the systematic review of the literature. A biased search or the omission of the most important studies will mean that the result is biased. If there is significant publication bias in a field then even a systematic review may not identify all relevant data.
31
![Page 32: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/32.jpg)
Useful resources• Cochrane Handbook, Chapter 9: Analysing data and undertaking meta-analyses (free):
http://handbook.cochrane.org/chapter_9/9_analysing_data_and_undertaking_meta_analyses.htm
• Revman 5 software (free):
http://tech.cochrane.org/revman/download
• Introduction to meta-analysis. Michael. Borenstein, Larry V Hedges; Julian P. T Higgins; Hannah Rothstein; Ebooks Corporation. 2009
• Meta PDF document for r (free):
https://cran.r-project.org/web/packages/meta/meta.pdf
32
![Page 33: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/33.jpg)
Recommended reading:
• A good introduction to meta-analysis can be found in a series of six short education articles published in the BMJ during 1997-98, which are as relevant today as they were in 1997 and the Cochrane Handbook for systematic reviews of interventions which was updated much more recently (2011). These BMJ articles form much of the book: "Systematic reviews in health care: meta-analysis in context, 2nd Edition" by Egger M, Davey Smith G, and Altman DG (BMJ Publishing Group 2001). The Cochrane Handbook and the first two BMJ articles provide a general overview:
• Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from www.cochrane-handbook.org
• Egger M, Davey Smith G. Meta-analysis: Potentials and promise. BMJ. 1997;315(7119):1371-4.
• Egger M, Davey Smith G, Philips AN. Meta-analysis: Principles and procedures. BMJ. 1997;315(7121):1533-7
33
![Page 34: Introduction to meta-analysis - Newcastle University · When to conduct meta-analysis •A meta-analysis is a statistical technique used to answer a question specified in the protocol](https://reader035.vdocuments.us/reader035/viewer/2022062402/5f03a2147e708231d40a035b/html5/thumbnails/34.jpg)
Additional reading:
• Davey Smith G, Egger M, Philips AN. Meta-analysis: Beyond the grand mean. BMJ. 1997;315(7122):1610-4
• Egger M, Davey Smith G. Meta-analysis: Bias in location and selection of studies. BMJ. 1998;316(7123):61-6
• Egger M, Schneider M, Davey Smith G. Meta-analysis: Spurious precision? Meta-analysis of observational studies. BMJ. 1998;316(7124):140-4
• Davey Smith G, Egger M. Meta analysis: Unresolved issues and future developments. BMJ. 316(7125):221-5
34