meta-analysis in risk-benefit evaluation india.pdfapplicable, included in the meta-analysis). data...
TRANSCRIPT
Meta-analysis in Risk-Benefit Evaluation
Demissie Alemayehu, Ph.D.April 2012
2
Outline• Part I: Standard Meta-
Analysis
– What is Meta-
Analysis?
– Benefits of Meta-
Analysis
– Procedures for
Meta-Analysis
– Criteria for
Causality
– Case Study
• Part II: Network Meta-
analysis
• Introduction
• Indirect and Mixed
Treatment Comparisons
• Exchangeability
– Issues and
Approaches
• Heterogeneity
—Heterogeneity in CER
—Issues and
Approaches
• Recent Developments
• Concluding Remarks
3
Workshop GoalsNon-practitioners of Meta-analysis
– Understand meaning of meta-analysis
– Understand limitations and strengths of MA
– Critically review MA literature
Practitioners of Meta-analysis
• Review standard approaches
• Review recent evelopments
• Explore further opportunities for research
“Many of the groups…are far too small to allow of any definite opinion being formed at all, having regard to the size of the probable error involved.”
Karl Pearson, 1904
5
6
Definition: Meta-Analysis
• Meta-Analysis is a statistical approach for
combining information from independent
studies to address a pre-specified
hypothesis of interest.
– Based on systematic literature review
– Provides precise estimate of treatment effect,
giving due weight to studies included
77
Benefits of Meta-analysis
• Precision / Power
– Individual clinical trials may lack power to provide conclusive results
– Particularly useful for rare events
• To assess consistency (generalizability) of results
– To settle controversies arising from conflicting studies
• Answer questions not posed by the individual studies
– Generate new hypotheses
– Subgroup risk/benefit
Unsystematic clinical observations
Physiologic Studies
Single Observational Study
Systematic Review of Observational Studies
Single Randomized Trial (RCT)
Systematic review of RCT
Harbour R, Miller J. A new system for grading recommendations in
evidence based guidelines. BMJ 2001;323:334–6.
Hierarchy of Evidence
When to Do Meta-Analysis
• When more than one study has estimated an effect
• When there are no differences in the study characteristics that are likely to substantially affect outcome
• When the outcome has been measured in similar ways
• When all the data are available
9
1010
When not to do a Meta-Analysis
• „Garbage in - garbage out‟
– A meta-analysis is only as good as the studies in it
• Beware of „mixing apples with oranges‟
– Compare like with like
• Beware of reporting biases
11
Questions to Ask
1. Detailed written protocol developed?
2. Comprehensive / systematic search for eligible studies performed?
3. Presentation of results appropriate?
4. Sensitivity analysis / test for heterogeneity performed?
5. Limitations of analysis discussed?
6. Results reported in light of available body of knowledge?
12
• Formulation of problem to be addressed
• Eligibility criteria for studies to be included
defined a priori
• Statistical methods for combining the data
Protocol Development
13
Selection of Studies
• Minimize Sources of Bias
– Publication Bias
• Positive studies more likely to be published than
negative studies
– Reviewer Bias
• Tendency to include only studies that favor the
hypothesis of interest
• Define universe of studies to be considered
• Define explicit & objective criteria for study
inclusion/ rejection based on quality grounds
14
Selection of Studies, cont’d
• Popular Databases
– PubMed / MedLine
– Cochrane Review
– Trial result registries
• Other Sources
– Hand Search: FDA, library
– Personal references, emails
– Web: Google, etc.
15
Funnel Plot ▪ Precision: increasing function of study size
▪ In absence of bias, results from small studies scatter widely at bottom of graph
▪ Publication bias: asymmetrical funnel plots
Publication Bias
16
• Effect Measures– Mean Difference
– Relative Risk
– Odds Ratio
– Risk Difference
• Peto OR:
Approximation to the
Odds Ratio.
– Used in rare events
Statistical Methods
• Choice depends
– Ease of
communication
of effect
– Study design
– Analytical
convenience
Total Exposed
EventObserved
Risk of Event
Active 1000 10 10/1000=0.01
Control 1000 1 1/1000 =0.001
Experiment 1
Risk Difference: 0.01-0.001 = 0.009
Relative Risk: 0.01/0.001 = 10
Odds Ratio:
Odds for Active: 0.01/0.99 = 0.010101
Odds for Control: 0.001/0.999 =0.001001
OR : 0.010101/0.001001 =10.09
Total Exposed
EventObserved
Risk of Event
Active 1000 410 410/1000=0.410
Control 1000 401 401/1000 =0.401
Experiment 2
Risk Difference: 0.410-0.401 = 0.009
Relative Risk: 0.410/0.401 = 1.02
Odds Ratio:
Odds for Active: 0.410/(1-0.410) = 0.70
Odds for Control: 0.401/(1-0.401) = 0.67
OR : 0.70/0.67 = 1.05
Statistical Methods (cont.)
18
Inference: • Different approaches exist,
but there is no single
"correct" method
• Two general approaches
–Fixed Effect Models
–Random Effects Models
•Bayesian procedure
–Increasingly used,
especially in network
meta-analysis
Statistical Methods (cont.)
19
• Fixed Effect Models
− If it is reasonable to assume underlying effect is the SAME for all studies
− Only one source of sampling error: within study
− Methods: inverse variance, Mantel-Haenszel
Statistical Methods, cont’d
20
• Random Effects Models
− True effect could vary from study to study
− E.g., effect size higher in older subjects
− Sources of sampling error: within study & between studies
− Common method: DerSimonian-Laird
Statistical Methods, cont’d
21
• Bayesian Methods– Use prior information in
statistical inference
Statistical Methods, cont’d
Thomas Bayes
(1702-1761)
21
22
• Bayesian Formulations
Statistical Methods, cont’d
Thomas Byes
(1702-1761)
23
• Large-sample procedures may not be
appropriate for pooling rare events
− Comparison of Procedures
• Bradburn et al., Stat Med 2007;26:53-77
• Event Rates < 1%
– Peto one-step odds ratio
• Reasonable bias, power
• Bias substantial, unbalanced case
Meta-Analytical Procedures:
Rare Events
24
• Event Rates 2%-5%
– Logistic Regression, Exact Method, MH OR better performance (in terms of bias) than Peto
• Risk Difference
– Conservative confidence intervals
– No need to exclude studies with 0 events in both groups
• For sparse data, incorporation of heterogeneity into estimation of treatment effects has minimal impact on performance
Meta-Analytical Procedures:
Rare Events, cont’d
25
Graphical display of results from individual
studies on a common scale: Blobbograms
http://www.childrens-mercy.org/stats/model/metaanalysis.asp
Data Presentation
26
• Heterogeneity is variation between the studies‟ results
• Causes:
– Differences in patient groups studied
– Differences in interventions studied
– Differences in primary outcome studies
– Studies carried out in distinct settings
e.g., different countries
Heterogeneity
27
• Cochran‟s Q Test– Low power, esp.
when # of studies is small
– Liberal if number of studies is large
Heterogeneity (cont.)
Under random effects el
Q w
pooled relative risk fixed effects el
study specific estimated relative risk
w Var
Q under H
s ss
S
s
s s
S B
mod ,
( ) ,
log , mod
log
[ ( )]
~ :
2
1
1
1
2
0
2 0
28
• I2
– proportion of variation that is due to
heterogeneity rather than chance
– Large values of I2 suggest heterogeneity
– Roughly, I2 values of 25%, 50%, and 75% could be
interpreted as indicating low, moderate, and high
heterogeneity
– Higgins JPT et al. Measuring inconsistency in meta-
analyses. BMJ 2003;327:557-60.
Heterogeneity (cont.)
29
Handling Heterogeneity
HETEROGENEOUS
TREATMENT EFFECTS
IGNORE ACCOUNT
FOR
EXPLAIN
FIXED
EFFECTS
MODEL
RANDOM
EFFECTS
MODEL
SUBGROUP
ANALYSES
META-
REGRESSION
(control rate,
covariates)
Post-hoc natureMultiplicities
Apples and oranges phenom
Interpretation?
Meta-Regression
Assess heterogeneity of effects by covariates that are constant within
study
• E.g., gender, smoking status
Model:
β s = β 0 + β1 GENDERs + β2 CURRENTs
+ β2 PASTs + bs + εs
GENDERs = 1 if studys is male; 0 if female
CURRENTs = 1 if studys has current smokers only, 0 otherwise
PASTs = 1 if studys has past smokers only, 0 otherwise
H0: β1 = 0 no effect-modification by gender
Mixed effects models can be used to test hypotheses and estimate
parameters (Stram, Biometrics, 1996)
• Limitations– Ecological fallacy
• Association present at patient level may not be necessarily true at the study level
– A model that includes a covariate that is an aggregate of a person-level characteristic rather than a study characteristic can produce biased results.
– Post-hoc specification of prognostic factors may lead to spurious results
– Cannot handle factors that vary by patient within study
Meta-Regression (cont.)
32
• Examine Robustness of Findings
– Examine effects in certain studies, certain
groups of patients, or certain interventions
– Robustness to departures from model
assumptions
– Sensitivity to departure from study selection
criteria
Sensitivity Analysis
33
• When interpreting results, don‟t just emphasise the
positive results.
• Discuss Limitations of Analysis
– Bias: Publication
– Use of aggregate data: Confounding factors
– Model assumptions
– Heterogeneity
• Summarized in light of available body of knowledge
Reporting of Results
Alemayehu D. (2011) Perspectives on Pooled Data Analysis: the Case for an Integrated
Approach. Journal of Data Science, 9(3): 399-426
34
"Correlation does not imply
causation"
35
• Strength of Association
– Validity of measure: OR, RR, etc.
– Clinical / statistical significance
• Biological Plausibility
– Known potential biologic basis to suggest a
causal link
– Reference pharmacogenomic evidence
Assessment of Causality
• Assessment of Consistency
– Review similar reports for drug
– Review similar reports for class
36
• Epidemiology
– Consider the expected rate in the general
population for comparable demographic groups
• False Positive Findings
– Address issue of multiplicity
– FDR
• Imbalance (wrt Relevant Confounding Factors)
– Identify relevant risk factors
• Pharmacovigilance Databases
– Recent advances in analysis of such data
– Measures of disproportionality
Assessment of Causality, cont’d
37
• Specificity
– Cause and effect relationship already established for a
similar exposure-disease combination
• Temporality
– Cause precedes effect problem
• Dose-Response
– Larger exposures associated with higher rates of
disease
Assessment of Causality, cont’d
• Reversibility
– Reversing the exposure is associated with lower
rates of disease
38
“Talking about unsettled science is a muchmore complicated communications challengethan simply disseminating hard conclusions.”
“The goal should be to effectively communicatefindings to patients, healthcare providers, andregulatory agencies to enable informeddecision-making.”
Gottlieb, 2006
Communications of Findings
• PRISMA: – Preferred Reporting Items for Systematic Reviews and Meta-
Analyses.
– An evidence-based minimum set of items for reporting in systematic reviews and meta-analyses.
– A 27-item checklist, and a four-phase flow diagram.
– An update and expansion of the now-out dated QUOROM Statement
• Aim : – Help authors improve the reporting of systematic reviews and meta-
analyses.
– Focus on randomized trials, can also be used as a basis for reporting systematic reviews of other types of research
Source: http://www.prisma-statement.org/statement.htm
PRISMA 2009 Flow Diagram
Records identified through database searching
(n = )
Scre
en
ing
Incl
ud
ed
Elig
ibili
tyId
en
tifi
cati
on
Additional records identified through other sources
(n = )
Records after duplicates removed(n = )
Records screened(n = )
Records excluded(n = )
Full-text articles assessed for eligibility
(n = )
Full-text articles excluded, with reasons
(n = )
Studies included in qualitative synthesis
(n = )
Studies included in quantitative synthesis
(meta-analysis)(n = )
Source: http://www.prisma-
statement.org/statement.htm
Section/topic # Checklist item Reported
on page #
TITLE
Title 1 Identify the report as a systematic review, meta-analysis, or both.
ABSTRACT
Structured summary 2 Provide a structured summary including, as applicable: background; objectives; data sources; study
eligibility criteria, participants, and interventions; study appraisal and synthesis methods; results;
limitations; conclusions and implications of key findings; systematic review registration number.
INTRODUCTION
Rationale 3 Describe the rationale for the review in the context of what is already known.
Objectives 4 Provide an explicit statement of questions being addressed with reference to participants,
interventions, comparisons, outcomes, and study design (PICOS).
METHODS
Protocol and registration 5 Indicate if a review protocol exists, if and where it can be accessed (e.g., Web address), and, if
available, provide registration information including registration number.
Eligibility criteria 6 Specify study characteristics (e.g., PICOS, length of follow-up) and report characteristics (e.g., years
considered, language, publication status) used as criteria for eligibility, giving rationale.
Information sources 7 Describe all information sources (e.g., databases with dates of coverage, contact with study authors to
identify additional studies) in the search and date last searched.
Search 8 Present full electronic search strategy for at least one database, including any limits used, such that it
could be repeated.
Study selection 9 State the process for selecting studies (i.e., screening, eligibility, included in systematic review, and, if
applicable, included in the meta-analysis).
Data collection process 10 Describe method of data extraction from reports (e.g., piloted forms, independently, in duplicate) and
any processes for obtaining and confirming data from investigators.
Data items 11 List and define all variables for which data were sought (e.g., PICOS, funding sources) and any
assumptions and simplifications made.
Risk of bias in individual
studies
12 Describe methods used for assessing risk of bias of individual studies (including specification of
whether this was done at the study or outcome level), and how this information is to be used in any
data synthesis.
Summary measures 13 State the principal summary measures (e.g., risk ratio, difference in means).
Synthesis of results 14 Describe the methods of handling data and combining results of studies, if done, including measures of
consistency (e.g., I2) for each meta-analysis.
Section/topic # Checklist item Reported
on page #
Risk of bias across studies
15 Specify any assessment of risk of bias that may affect the cumulative evidence (e.g., publication bias, selective reporting within studies).
Additional analyses 16 Describe methods of additional analyses (e.g., sensitivity or subgroup analyses, meta-regression), if done, indicating which were pre-specified.
RESULTS
Study selection 17 Give numbers of studies screened, assessed for eligibility, and included in the review, with
reasons for exclusions at each stage, ideally with a flow diagram.
Study characteristics 18 For each study, present characteristics for which data were extracted (e.g., study size, PICOS, follow-up period) and provide the citations.
Risk of bias within studies
19 Present data on risk of bias of each study and, if available, any outcome level assessment (see item 12).
Results of individual studies
20 For all outcomes considered (benefits or harms), present, for each study: (a) simple summary data for each intervention group (b) effect estimates and confidence intervals, ideally with a forest plot.
Synthesis of results 21 Present results of each meta-analysis done, including confidence intervals and measures of consistency.
Risk of bias across studies
22 Present results of any assessment of risk of bias across studies (see Item 15).
Additional analysis 23 Give results of additional analyses, if done (e.g., sensitivity or subgroup analyses, meta-regression [see Item 16]).
DISCUSSION
Summary of evidence 24 Summarize the main findings including the strength of evidence for each main outcome; consider their relevance to key groups (e.g., healthcare providers, users, and policy makers).
Limitations 25 Discuss limitations at study and outcome level (e.g., risk of bias), and at review-level (e.g., incomplete retrieval of identified research, reporting bias).
Conclusions 26 Provide a general interpretation of the results in the context of other evidence, and implications for future research.
FUNDING
Funding 27 Describe sources of funding for the systematic review and other support (e.g., supply of data); role of funders for the systematic review.
QUOROM vs. PRISMA checklists
45
45
Cumulative Meta-Analysis
• Commonly executed with chronologically ordered RCTs
– Perform a new statistical pooling every time a new RCT becomes available
– Impact of each study on the pooled estimate may be assessed
– Reveals (temporal) trend towards superiority of the treatment or the control, or towards indifference
– Performed retrospectively, the year when a treatment could have been found to be effective could be identified
– Performed prospectively, an effective treatment may be identified at the earliest possible moment
46
Example of Cumulative Meta-Analysis
Lau et al (1995) J Clin Epi (48) 45-57
47
4747
Basic Method of Cumulative Meta-Analysis
Studies ordered
chronologically ofby covariates Study 1
Study 2
Study 3
Study 4
Cumulative M-A 1
Cumulative M-A 2
Cumulative M-A 3
Pool Studies 1 to 2
Pool Studies 1 to 4
Pool Studies 1 to 3
Study n-1 Cumulative M-A n-2
Study n Cumulative M-A n-1Pool Studies 1 to n
Pool Studies 1 to n-1
• Arrangement of trials– Reporting time
– Effect size
– Study size
– Study quality
– Event rate of control
– Other covariates of interest
• Benefits– Identify signal early
– Design of future trials
– Need for further trials
– Identify subgroups of interest
Cumulative Meta-Analysis (cont.)
• Issues– Multiplicity: Use of p-
values without correction– Use of Bayesian approaches
– Contributions of large studies minimized
– Experience shows that large studies reflect meta-analysis results of small studies
– Evolution of treatment effect
over time
– Changes in patient
demographics
– Changes in healthcare delivery
– Evolving medical practices
Software
• “rmeta” package in R for statistical computing
http://cran.r-project.org/web/packages/HSAUR/vignettes/Ch_meta_analysis.pdf
• EpiMeta: from Epi Info
• Revman: from Cochrane Collaboration
• meta module in STATA
• MIX for Excel 2000 or 2003
http://www.mix-for-meta-analysis.info
Index:
meta.DSL Random effects (DerSimonian-Laird) meta-analysismeta.MH Fixed effects (Mantel-Haenszel) meta-analysis
meta.summaries Meta-analysis based on effect estimatesmetaplot Confidence interval (forest) plotfunnelplot Funnel plot for publication bias
meta.colors Control colours in meta-analysis plot
cummeta Cumulative meta-analysiscummeta.summaries Cumulative meta-analysis
Package: rmetaAuthor: Thomas Lumley <[email protected]>
ntrt Number of subjects in treated/exposed group
nctrl Number of subjects in control group
ptrt Number of events in treated/exposed group
pctrl Number of events in control group
conf.level Coverage for confidence intervals
names names or labels for studies
statistic "OR" for odds ratio, "RR" for relative risk
x,object a meta.DSL object
summary Plot the summary odds ratio?
meta.DSL(ntrt, nctrl, ptrt, pctrl, conf.level=0.95,. . . ,statistic="OR")
summary(object, conf.level=NULL, ...)
plot(x, conf.level=NULL, colors=meta.colors(), xlab=NULL,...)
Arguments:
x Treatment difference
se Standard error of x
size Variable for the vertical axis
summ summary treatment difference
funnelplot(x, se, size=1/se,
summ=NULL, xlab="Effect",
ylab="Size", colors=meta.colors(),
conf.level=0.95, plot.conf=FALSE, zero=NULL, mirror=FALSE, ...)
Arguments
Graphics:
forestplot ()
metaplot
plot.meta.DSL,
plot.meta.MH,
plot.meta.summaries
Part II: Network Meta-Analysis
• Introduction
• Indirect and Mixed Treatment Comparisons
• Exchangeability
– Issues and Approaches
• Heterogeneity
—Heterogeneity in CER
—Issues and Approaches
• Recent Developments
• Concluding Remarks
Introduction
• Comparative Effectiveness Research
– Institute of Medicine, IOM:• Generation and synthesis of evidence that compares the benefits
and harms of alternative methods to prevent, diagnose, treat, and monitor a clinical condition or to improve the delivery of care
– The purpose is to assist consumers, clinicians, purchasers, and policy makers to make the informed decisions that will improve health care at both the individual and population levels. http://books.nap.edu/openbook.php?record_id=12648&page=29
– Congressional Budget Office: • A rigorous evaluation of the impact of different options that are
available for treating a given medical condition for a particular set of patients.
– Such a study may compare similar treatments, such as competing drugs, or it may analyze very different approaches
Introduction (cont.)RCTs vis-à-vis CER• Randomized controlled trials (RCTs) designed to provide
data on a treatment efficacy on average for a given population.
• Applicability of results of RCTs to real-world situations limited by factors inherent to the design– Stringent entry criteria exclude subgroups of interest– For operational reasons, certain subgroups may not be
adequately represented in study– Other protocol requirements impose constraints that are
inconsistent with real-world practice and adherence
• CER: Emphasis on generalizability of RCT results to different subgroups or settings― Knowledge of heterogeneity of treatment effects is critical in
decision making about treatment options for individual patients and subpopulations
• Limitations of traditional meta-analysis also apply to CER
– Quality • Publication bias
– Heterogeneity• “Apples and oranges”
phenomenon
– Methodological issues• Lack of uniform
approaches
• Handling heterogeneity
• Assessing bias
• Covariate issues
Issues specific to CER
• Lack of head-to-head RCTs for comprehensive assessment
− Need to use non-standard procedures
− Stringent model assumptions!
• Assessment of heterogeneity more complex
Traditional Meta-Analysis vs. CER Meta-Analysis
– Several glaucoma drugs
– Rank available glaucoma drugs according to their intraocular pressure (IOP)-reducing effect.
– Several RCTs available, but not all treatments have been compared directly.
Source: R. van der Valk et al. / Journal of Clinical Epidemiology 62 (2009) 1279-1283
NB: n: Number
of direct
comparisons
between two
drugs.
Example
Standard Meta-Analytic Framework
Goal:
Synthesize information and obtain a pooled estimate of treatment
effect.
Standard Meta-analytic Framework (cont.)
Key assumption:
Homogeneity across trials
Standard Meta-analytic Framework (cont.)
Key Issue:
Problem of interpretation, when trials are heterogeneous
Standard Meta-analytic Framework (cont.)
Key Issue:
Choice of priors
Indirect Comparison
A
BA vs. C?
C
Scenario:
i) Direct head-to-head
comparative evidence available
for A vs. B, and for B vs. C
• Could be single RTCs or
pooled data in meta-
analyses
ii) Interested in A vs. C
iii) Limitations of using direct effect
measures for A and B
• Preservation of
randomization (at least
partially
Indirect Comparison
Ref: Bucher HC, et al. Journal of Clinical Epidemiology 1997;50:683-691.
Song F, et al;. BMJ 2003;326:472-476.
Partial preservation of randomization
Indirect Comparison (cont.)
• Cannot handle multi-arm trials or situations with direct and indirect evidence.
• Optimality of procedure
– Generally conservative
Indirect Treatment Comparisons: 2-step Approach
Jansen JP, et al. (2008)
Network Meta-analysisA
C
DB
Lumley (2002). Statist. Med; 21:2313-23A measure of inconsistency
Limitations :
• Indirect comparison follows through a closed loop design: Necessary for inference
• Cannot handle correlations in multi-arm studies
• Measure of inconsistency requires a sizeable number of studies
Mixed Treatment Comparisons
• Allow combining
evidence from direct and
indirect estimates
– Assumption of
consistency: Lu and
Ades (2006, JASA)
and Salanati et al.
(2008)
• MTC also permits
handling of multi-arm
studies: Lu and Ades
(2004, Stat Med
• Most approaches based
on Bayesian techniques
A vs C comparison via direct and
indirect means
Li et al. BMC Medicine 2011 9:79 doi:10.1186/1741-7015-9-79
Mixed Treatment Comparisons (cont.)
Non-informative priors
a) Normal for “treatment effect”
b) inverse gamma or uniform prior for dispersion.
WinBUGS Codes: www.bris.ac.uk/cobm/docs/intro%20to%20mtc.doc
Statistical Issues
Exchangeability Assumption:
• Relative efficacy of a treatment is the same in all trials included in the indirect comparison
• Same synthesized comparative treatment effect would result if direct comparisons were performed with trials mimicking the other (indirect) experimental conditions:
– Comparability of distribution of effect modifiers (treatment x covariate interactions) among trials
– Methodological similarity (e.g., quality, definition of outcomes) also required
Statistical Issues (cont.)
• Reliable statistical procedures unavailable to validate assumptions– Exchangeability/consistency
• Available procedures do not readily adjust for study heterogeneity– Adjusted indirect comparison, misleading term
– Meta-regression in the context of MTC not well studies
• Rigorous study of the operating characteristics of commonly used techniques not available– Low power, as a result of use of two separate variances
• Impacts reliability of non-inferiority conclusions
• Often leads to indeterminate results, due to inflated Type II error
• Most methods do not handle correlated data
Approaches for Assessing Exchangeability Assumptions
In general, exchangeability assumption difficult to verify
Approaches for Assessing Exchangeability Assumptions (cont.)
Qualitative approaches
• Differences in quality and design
– Blinding
– Duration of follow-up
– Outcome measures
– Treatment: Dose, timing
• External factors
– Healthcare systems
– Geography
– Setting: Hospital vs. ambulatory care
– Temporal
• Patient characteristics
− Demography
− Disease severity, etc.
Quantitative approaches
• Consistency: Constant event rates of reference group across trials
– Inference to rule out play of chance
• Absence of heterogeneity when meta-analysis is involved
– Usual issues with tests of homogeneity
• Formal inference in a (closed) network meta analysis
Approaches for Assessing Exchangeability Assumptions (cont.)
Approaches for Assessing Exchangeability Assumptions (cont.)
Approaches for Assessing Exchangeability Assumptions (cont.)
• Mahalanobis distance matching:
– ML estimators of location and covariance
• Modified propensity score analysis
– Perform usual case, with study assignment as dependent variable, rather than treatment
– Use propensity scores to identify subgroups of patients with a high probability of being in a given study
Alemayehu D. (2011) Assessing exchangeability in indirect and mixed treatment comparisons.
Comparative Effectiveness Research, 1: 51 - 55
76
Suppose we have treatment pairs AB, AC, BC with direct evidence
Indirect estimate:ˆ ˆ ˆindirect direct direct
BC AC ABd d d
Measure of
inconsistency: ˆ ˆˆ indirect direct
BC BC BCd d
Approximate test ˆ
ˆ
BCBC
BC
zV
ˆ direct direct direct
BC BC AC ABV V d V d V d
Measuring Consistency
Heterogeneity
• Heterogeneity and exchangeability different, but related concepts
• Heterogeneity
– Consistency of overall results across subgroups
• CER guidelines based on average results may not be optimal, and may even be substantially harmful
– Consistency of results across studies
• Heterogeneity issues associated with traditional meta-analysis apply to CER
• Objectives of heterogeneity analysis:
– Identify sources of heterogeneity
• Tailor treatments accordingly
– Adjust for heterogeneity
• Traditional regression approach for measured confounders
• Instrumental variables (Basu et al http://www.ssc.wisc.edu/~snavarro/Research/Limitations_of_IV_v28.pdf)
• Need to distinguish between clinical vs. statistical heterogeneity
Clinical vs. Statistical Heterogeneity
• Clinical heterogeneity (CH):
– Variation in study population characteristics, coexisting conditions, cointerventions, and outcomes evaluated across studies included in an SR or CER that may influence or modify the magnitude of the intervention measure of effect
• Statistical heterogeneity (SH):
– Variability in the observed treatment effects beyond what would be expected by play of chance
• CH may lead to SH. However SH may result from:
– CH, methodological heterogeneity or play of chance
http://www.effectivehealthcare.ahrq.gov/ehc/products/93/533/Clinical_Heterogeneity_Revised_Report_
FINAL%2010-5-10.pdf
Statistical Issues with Heterogeneity Analyses
• Post hoc vs. Pre-specified Analyses– Bias introduced by timing of specification of hypotheses
being tested relative to examination of data
• Multiplicity
– With multiple subgroup analyses, probability of a false positive finding substantial.
• Power– Chance of missing significant effect a function of size
• Inappropriate Heterogeneity Assessment– Claiming heterogeneity on basis of separate tests of
treatment effects within each subgroup– Claiming heterogeneity on basis of observed treatment-
effect sizes within each subgroup
Best Practices: HeterogeneityPre-specification • Clear Definition of Clinical
Heterogeneity– Demographic Characteristics– Clinical variables: Risk factors for
the principal condition under study– Expert consensus vs. Statistical
criteria
• Pre-Specification of factors in Protocol– Essential for confirmatory evidence
of efficacy in a subgroup
• Sample Size– A subgroup should be large enough
to ensure reliability of inferences
• Handling of Multiplicity– Specify adjustment strategy for
multiplicity
• Reporting of Results– Transparency/balance
Analytical Strategy
• No universally accepted method even in standard meta-analysis:
– Low power for small number of studies or rare events
• Meta-regression
– Limitations: Ecological fallacy
• Sensitivity Analysis
– Sensitivity to departures from definition of clinical variables
– Sensitivity to inclusion/exclusion of studies
– Sensitivity to departures from model assumptions
Alemayehu D. (2011) Evaluation of Heterogeneity of Treatment Effects in Comparative Effectiveness Research.
J Biomet Biostat 2:125. doi:10.4172/2155- 6180.1000125
Recent Developments
• AHRQ– Comparative Effectiveness Review Methods: Clinical Heterogeneity
– Best practices for addressing clinical heterogeneity in systematic reviews and CER
http://www.effectivehealthcare.ahrq.gov/ehc/products/93/533/Clinical_Heterogeneity_Revised_Report.pdf
• FDA CER Initiative– Framework for analyzing heterogeneity of treatment effects in CER
– Emphasis on value of pre-specification, exploratory vs. confirmatory subgroup analyses, testing for interactions, displaying graphically results, validating subgroup results.
http://www.fda.gov/downloads/AdvisoryCommittees/CommitteesMeetingMaterials/ScienceBoardtotheFoodandDrugAdministration
Recent Developments (cont.)
• Role of simulation in MTC
– Recent progress in simulated treatment comparisons (Caro & Itshak, 2010)
– Simulated experiments can serve several purposes:
• Perform head-to-head treatment comparisons, in the absence of such data
• Assess differences among trials
• Assess heterogeneity and exchangeability assumptions
Concluding Remarks
• Meta-analysis and Indirect comparison an integral tool of medical research– Considerable practical and methodological challenges
• Opportunity to engage in methodological work for statisticians– Assess exchangeability
– Assess/Adjust for heterogeneity, confounders
• Effective assessment of heterogeneity important to diverse stakeholders: – Clinicians, patients, policymakers and other healthcare providers
• Without sound methodological foundation, potential for adverse impacts on public health.
84
84
Selected References• Bucher HC, Guyatt GH, Griffith LE, Walter SD. The results of direct and indirect treatment comparisons in meta-
analysis of randomized controlled trials. J Clin Epidemiol 1997;50(6):683-91.
• Eddy DM, Hasselblad V, Shachter R. An introduction to a Bayesian method for meta-analysis: The confidence profile method. Med Decis Making. 1990 Jan-Mar;10(1):15-23.
• Song F, Altman DG, Glenny AM, Deeks JJ. Validity of indirect comparison for estimating efficacy of competing interventions: empirical evidence from published meta-analyses. BMJ. 2003;326:472.
• Lu G, Ades AE. Combination of direct and indirect evidence in mixed treatment comparisons. Stat Med 2004;23:3105–24.
• Caldwell DM, Ades AE, Higgins JP. Simultaneous comparison of multiple treatments: combining direct and indirect evidence. BMJ 2005;331:897–900.
• Jansen JP, Crawford B, Bergman G, Stam W. Bayesian meta-analysis of multiple treatment comparisons: an introduction to mixed treatment comparisons. Value Health. 2008; 11:956-64.
• Cooper NJ, Sutton AJ, Morris D, Ades AE, Welton NJ. Addressing between-study heterogeneity and inconsistency in mixed treatment comparisons: Application to stroke prevention treatments in individuals with non-rheumatic atrial fibrillation. Stat Med. 2009;28:1861-81.
• Song F, Loke YK, Walsh T, Glenny AM, Eastwood AJ, Altman DG. Methodological problems in the use of indirect comparisons for evaluating healthcare interventions: survey of published systematic reviews. BMJ. 2009 Apr 3;338
• Lumley T. Network meta-analysis for indirect treatment comparisons. Stat Med 2002;21(16):2313-24.
• Ades AE. A chain of evidence with mixed comparisons: models for multi-parameter synthesis and consistency of evidence. Stat Med 2003;22(19):2995-3016.
• Caldwell DM, Ades AE, Higgins JP. Simultaneous comparison and indirect evidence. BMJ 2005;331(7521):897-900.
• Vandermeer BW, Buscemi N, Liang Y, Witmans M. Comparison of meta-analytic results of indirect, direct, and combined comparisons of drugs for chronic insomnia in adults: a case study. Med Care 2007;45(10 Supl 2):S166-S172.
• Song F, Harvey I, Lilford R. Adjusted indirect comparison may be less biased than direct comparison for evaluating new pharmaceutical interventions. J Clin Epidemiol 2008;61(5):455-63. 40.
• Glenny AM, Altman DG, Song F, Sakarovitch C, Deeks JJ, D'Amico R, et al. Indirect comparisons ofcompeting interventions. Health Technol Assess 2005;9(26):1-134.