systematic review and meta-analysis: applications · pdf file1 systematic review and...
TRANSCRIPT
Systematic Review and MetaSystematic Review and Meta--Analysis: Analysis: ApplicationsApplications
Susan D. Ross MD, FRCPCBoston MA
1
Systematic Review and Meta-Analysis: Applications
ISPOR Distance Learning Module 2
Susan D. Ross MD, FRCPC
Boston MA
2
Learning Objectives
To appreciate the pros and cons of using systematic reviews and meta-analyses to inform healthcare policy and practice decisions re: the efficacy, effectiveness, and safety of healthcare interventions.
To learn how systematic reviews and meta-analyses are and will be used for drug development and commercialization activities by industry.
3
Who uses systematic reviews?
Policy makers:
CMS (Centers for Medicare & Medicaid Services) US federal agency which administers Medicare, Medicaid, and the State Children's Health Insurance Program
NICE (National Institute for Clinical Excellence ) UK organization responsible for providing national guidance on the promotion of good health and the prevention and treatment of ill health.
4
CMS Follows General Principles of EBMAssess the Evidence re: “reasonable and necessary” care =
Published and non-published studies, expert opinion, technology assessments, professional societies, recommendations from the Medicare Coverage Advisory Committee (MCAC)
Key areas of focus include:
Methodological considerations, including study design, implementation, and analysis
Relevance of chosen outcomes; preference for patient reported outcomes
Generalizable to the Medicare population
Qualitative assessment of net risks and benefits based on individual studies
5
CMS Example: Bariatric surgery
August 2004 – November 2004: Coverage Analysis Group prepared an evidence review and summary of evidence for presentation to MCAC
May 2005: CMS opened a national coverage determination (NCD) at the request of the American Society for Bariatric Surgery, the American Obesity Association, Ethicon Endo-Surgery Inc. (a Johnson & Johnson company), INAMED Corp., U.S. Surgical, a division of Tyco Healthcare Group LP, and Transneuronix Corp.
February 2006: CMS has determined that the evidence is adequate to conclude that certain open and laparoscopic procedures are reasonable and necessary for Medicare beneficiaries who have a body-mass index (BMI) > 35, have at least one co-morbidity related to obesity, and have been previously unsuccessful with medical treatment for obesity.
6
But..Frequently Unanswered Questions in CMS Decision-making
- Cost-effectiveness?
- Adverse events?
- Off-label uses or new combinations of approved uses?
- Risks and benefits in subgroups?
- Outcomes for patient and provider types excluded from trials?
- Comparative effectiveness?
- Outcomes not measured in trials?
- Clinical utility of diagnostic tests?
- Surgery, devices and other technologies with limited regulatory review?
7
NICE
New medicines are licensed by the Medicines and Healthcare products Regulatory Agency/European Medicines Evaluation Agency.
NICE commissions independent academics to review (and Bayesian meta-analyze) published evidence on selected new & existing drugs & interventions.
NICE issues guidance on the clinical and cost effectiveness of Rx for the NHS in England and Wales.
Primary Care Trusts (local health bodies) then implement the guidance by funding Rx to patients when clinicians recommend it.
8
NICE Example: Herceptin
NICE system has led to rationing of (delayed access to) new anticancer drugs.
Herceptin (£20,000/year) was approved 1998 in the U.S. for advanced breast cancer.
EMEA approval 2000.
But not until March 2002 was NICE guidance released, for HER2+advanced breast cancer. And even then, patient access was spotty, due to variable funding decisions from UK PCTs.
11
Oregon DERP
Compares efficacy of drugs within each of the 25 most expensiveclasses (the top 25 classes accounting for >50% of all pharmaexpenditures by states).
Originally a partnership between state of Oregon and Oregon EPC….now by Center for Evidence-based Policy at the OHSU…several EPCs continue to develop and update the 25 Systematic Reviews.
Project will “make available to governments, businesses, non-profit organizations, and citizens the best existing synthesis of global research on the relative effectiveness of similar medications. This research will allow these entities to determine for themselves how best to obtain the most value for their pharmaceutical expenditures”.
12
Examples from Oregon DERP
2002: Statins
all 5 statins were similar wrt Cholesterol lowering; only some had clinical outcomes; only 1 was generic – that was chosen for preferred drug lists.
2005: Triptans: Oregon and Washington used the same review of Triptans to reach different conclusions
Oregon accorded priority to the outcome of patients being pain free at 2 hours, its policymakers chose rizatriptan
Washington chose both rizatriptan and sumatriptan after considering a broader array of clinical end-points.
2007: 17 states and 2 non-profits using DERP results for drug coverage policy for their Medicaid programs
13
BC/BS uses (and produces) systematic reviews
Technology Evaluation Center (AHRQ EPC, with Kaiser Permanente collaboration)
5 Criteria to assess whether a technology improves health outcomes (e.g. length of life, quality of life, functional ability):
1. The technology must have final approval from the appropriate governmental regulatory bodies.
2. The scientific evidence must permit conclusions concerning the effect of the technology on health outcomes.
3. The technology must improve the net health outcome
4. The technology must be as beneficial as any established alternatives.
5. The improvement must be attainable outside the investigational settings.
14
Kaiser Permanente uses systematic reviews…
Kaiser Permanente is the nation's largest HMO
Nonprofit, founded in 1945
9 million members in 9 states & D.C.
Develop interregional guidelines, using evidence-based methods and teams who are expert in searching, summarizing, and critically appraising the medical literature. Kaiser Permanente invests > $80m of its $17b annual budget in research, including $12.3m for its care management institute, to "synthesize knowledge about the best clinical approaches and create, implement, and evaluate effective and efficient health care programs."
Stiefel M, et al. Kaiser Permanente's national integrated diabetes care management program. In Fox DM, Oxman AD, eds, Informing judgment: case studies of health policy and research in six countries. New York: Milbank Memorial Fund, 2001.
15
AMCP uses systematic reviews…
Academy of Managed Care Pharmacy –
In 2000, AMCP created the Format for Formulary Submissions (“AMCP” dossiers), requiring drug manufacturers to provide:
detailed product descriptions
discussion of potential problems
actual or anticipated off-label uses
data comparing the drug to other medications
reviews & summaries of key published and unpublished clinical studies
costs
spreadsheet model for predicting plan-specific outcome
16
AMCP uses systematic reviews…
By 2008:
health systems that cover approximately 150 million Americans have adopted the AMCP Format or a similar decision-making process, including national and regional managed health care systems, PBMs, the DoD, hospitals and state Medicaid agencies.
AMCP dossier Format is now viewed as an industry standard.
18
Providers use systematic reviews…
Witness:
PDA’s ubiquitous
EMR’s upsurge
Practice guidelines
Cochrane Collaboration (www.cochrane.org) - Library of systematic reviews
See: Sackett DL, Richardson WS, Rosenberg WMC, Haynes RB. Evidence-based Medicine: How to Practice and Teach EBM. London: Churchill Livingstone; 1997.
19
Providers use systematic reviews…
But not as much as they should…..
Witness:
Geographic variations in care, costs, and outcomes
Low level of compliance with evidence-based guidelines
Translating evidence to practice is now the biggest challenge of EBM.
20
What about Pharma use of systematic reviews?
Clinical R&D
1. To assess trial feasibility 2. To replace a 2nd RCT for sNDA
Risk Management
3. Cumulative meta-analyses for safety monitoring
Commercialization
4. To compare directly or indirectly drug efficacy or safety
Life-Cycle management
5.Efficient portfolio management (cumulative meta-analyses)
21
Case 1. Trial Feasibility:
Asthma
Question:
Will a planned multicenter phase 4 trial ($10 million estimate) succeed in proving that drug P is better than drug Z? (i.e., is efficacyof drug P superior to efficacy of drug Z?)
EBM tool:
Meta-analysis of all sponsor’s trials of drug P vs. SBA drug Z.
Answer:
NO!
22
Drug P or Z vs. Placebo: All Outcomes
Meta-Analyses of Effect Size and 95% C.I.
2 4 60Favors Treatment
-6 -4 -2Favors Placebo
DerSimonian & Laird Random Effects Model
#Study/Outcome Treatment Arms #PtsFEV 1 Z1 3 1042FEV 1 P1 4 B05
AM PEFR Z 3 1042AM PEFR P1 4 B05
PM PEFR Z1 3 1042PM PEFR P1 4 2B05
β agonist Z 3 1042
β agonist P1 4 B05
23
Drug P or Z vs. Placebo: All Outcomes, Adults
FEV1 - All 5REV1 - Adults only 3
A.M. PEFR - All 5A.M. PEFR - Adults only 3
P.M. PEFR - All 4P.M. PEFR - Adults only 2
β agonist - All 4
β agonist - Adults only 22 4 60
Favors Treatment-6 -4 -2
Favors PlaceboDerSimonian & Laird Random Effects Model
Sensitivity Analysis Adult Patients OnlyOutcome #Study/Arms
24
Drug P or Z vs. Placebo for FEV-1 By dose
P1 282P1 507P1 871P1 701P1 all dose 2361
2 40Favors Treatment
-4 --6Favors Placebo
DerSimonian & Laird Random Effects Model
Meta - Analyses of Effect Size and 95% C.I.Treatment # of Patients
Z1 88Z1 138Z1 234Z1 1182Z1 1274Z1 87Z1 81Z1 all dose 3084
25
Case 2. Replace a 2nd RCT for sNDA:
PPI in gastric ulcer
Question:
Can we avoid a clinical trial by using existing data in a sNDA?
EBM tool:
Meta-analysis of sponsor’s unpublished RCTs
Answer:
Efficacy proven in meta-analyses. Trial avoided. FDA success.
Tunis S, Sheinhait IA, Schmid C, Bishop DJ, Ross SD. The efficacy of lansoprazolecompared to histamine-2 receptor antagonists in healing gastric ulcers: a meta-analysis. Clinical Therapeutics 1997; 19: 743-757.
26
PPI in gastric ulcer
*H2RA = histamine-2 receptor antagonist
**REM = random effects model
***FEM = fixed effects model
Bayes REM** FEM*** Bayes REM** FEM*** Bayes REM** FEM***
Estimate 1.33 1.32 1.36 0.17 0.17 0.17 2.00 2.00 2.00
(95% CI) (1.19-1.49) (1.21-1.45) (0.24-1.49) (0.11-0.22) (0.12-0.22) (0.12-0.22) (1.59-2.52) (1.63-2.45) (1.63-2.45)
Estimate 1.12 1.12 1.14 0.10 0.09 0.09 4.67 4.47 5.20
(95% CI) (1.06-1.19) (1.07-1.17) (1.08-1.20) (0.05-0.14) (0.05-0.14) (0.06-0.13) (1.49-2.63) (1.54-2.57) (1.53-2.56)
PPI vs.H2RA Healing in Randomized Patients
Risk Ratio Risk Difference Odds Ratio
4 W
eeks
8 W
eeks
27
PPI vs. H2RA Healing at 4 Weeks
Intention to Treat
01020304050607080
1 2 3 4 5 6 7 8 9 10 11 12
Study ID
Hea
ling
Rat
es (%
)
PPI
H2RA
28
Risk Ratios: PPI versus H2RA Healing at 4 Weeks
Study Year #Pts1 Aoyama 1995 1072 Della Fave 1992 683 Fludas 1995 695 Takemoto 1992 1456 Okai 1995 247 Grymbowski 1990 1068 Peters 1990 4429 Facklam 1992 174
10 Sakaguchi 1995 1811 Takemoto 1990 33612 Colin 1990 15013 Tsuji 1995 16
OVERALL 1655
0.1 0.2 0.5 1 2 5 10
Z=5.95 2P<0.00001
Favors H2RAs Favors Lansoprazole
Z=5.95 2P<0.00001
29
Case 3. Cumulative meta-analyses for safety monitoring
COX-II and myocardial infarction
Question:
Could safety problems with COX-II’s have been detected earlierthan they were?
EBM tool:
Cumulative meta-analysis (real-time) of all published and unpublished RCTs
Answer:
YES (probably)!
30
Cumulative Meta-analysis (post hoc): Vioxx & RR of MI [“our findings indicate that rofecoxib should have been withdrawn several years earlier”]
Juni, et al. Lancet 2004;364:2021-29
.
31
Cumulative Meta-analysis (chronologically ordered RCTs)
• Perform a new statistical pooling every time a new
randomized controlled trial (RCT) becomes available. • Impact of each study on the pooled estimate is assessed. • Reveals (temporal) trend towards superiority of the treatment
or the control, or indifference. • Performed retrospectively, the year when a treatment could
have been found to be effective, at a given significant level (typically P < 0.05), could be identified.
• Performed prospectively, effective treatment may be
identified at the earliest possible moment.
32
Study 1
Study 2
Study 3
Study 4
Cumulative M-A 1
Cumulative M-A 2
Cumulative M-A 3
Pool Studies 1 to 2
Basic Method of Cumulative Meta-AnalysisStudies ordered
chronologically orby covariates
Pool Studies 1 to 4
Pool Studies 1 to 3
Study n-1 Cumulative M-A n-2
Study n Cumulative M-A n-1 Pool Studies 1 to n
Pool Studies 1 to n-1
33Favors Treatment Favors Control
Study Year Pts 1 Flectcher
Odds Ratio 95% CI
Intravenous Streptokinase Therapy for Acute Myocardial Infarction
0.5 1 2
P < 0.00001Favors Treatment Favors Control
0.01 1 1000.1Odds Ratio 95% CI
Fixed Effects Model (Mantel-Haenszel)
10
2 Dewar 3 European 1 4 European 2 5 Heikinheimo 6 Italian 7 Australian 8 Franfurt 9 NHLBI SMIT10 Frank11 Valerie12 Klein13 UK-Collab14 Austrain15 Australian 216 Lasierra17 N Ger Collab18 Witchitz19 European 320 ISAM21 GISSI-122 Olson23 Baroffio24 Schreiber25 Cribier26 Sainsous27 Durand28 White29 Bassand30 Vlay31 Kennedy32 ISIS-2
2365
232962
33 Wisenberg
OVERALL 36974
1388170922262432253926472738276133564084431443384821487951946935
18647186991875818796188401893819002192211932819353197213690836974
Pts1959 231963 421969 1671971 7301971 4261971 3211973 5171973 206
1974 107 1975 1081975 911976 231976 5951977 7281977 2301977 241977 4831977 581979 315 1986 17411986 117121986 52
1986 59 1986 381986 44
1986 981987 64 1987 219 1987 1071988 251988 3681988 17187
1988 66
34Favors Treatment Favors Control
Textbook / ReviewRecommendations
CumulativeYear RCTs Pts
1960
1965
1970
1975
1980
1985
1990
1 29
2 65
3 149
4 316
7 1793
10 2544 11 2651 15 3311
21
5
1 10
1 2
2 8
7
8
1 12
1 8 4
1 7 3
5 2 2 1
15 8 1
6 1
Rou
tine
Spe
cific
Rar
e / N
ever
Exp
erim
enta
l
Not
Men
tione
d
Odds Ratio (Log Scale)
17 3929 22 5452
23 5767
Thrombolytic Therapy for Acute Myocardial Infarction
33 6571 30 6346 27 6125
43 21059 54 22051 65 47185 67 47531 70 48154
0.5 1 2
P < 0.01
P < 0.001
P < 0.00001
35
Case 4: Indirect Comparisons
.Questions: Can I compare treatments (A vs. B) when no direct comparisons exist?
A ← ← ← ? → → → B
CEBM tool: meta-analysis
Answer: Yes!
36
Indirect comparisons
Example:
Systematic review & meta-analysis* of published RCTs of GPIIb/IIIainhibitors in PCI patients
Meta-analysis of OR for 30 day death or non-fatal MI for each GPIIb/IIIa inhibitor
Ross S, Klawansky S, Allen IE. Indirect comparisons of drugs using meta-analysis: validation of results. Value in Health 2001; 4: 55.
37
Results: 30 day Death/MI = OR 1.4 (advantage abciximab)(normalize OR’s to permit a simple hypothesis test on significance of Δ)
ODDS RATIO
CO
MPA
RIS
ON
GR
OU
P
38
Indirect comparison validation
TARGET*
head-to-head comparison of tirofiban vs. abciximab in PCI
OR for 30 day D/MI = 1.3 (p=0.04) (advantage abciximab)
* Topol E. TARGET (Do Tirofiban and ReoPro Give Similar Efficacy Outcomes Trial) American Heart Association 2000: 401.
39
Beyond pair-wise indirect comparisons…
Question: Can I compare treatments A, B, C…. to each other, all at once?
EBM tool: Network meta-analysis (aka, Mixed Treatment Comparisons, “umbrella” reviews – Cochrane)
Answer: Yes!
40Copyright restrictions may apply.
Psaty, B. M. et al. JAMA 2003;289:2534-2544.
Network Meta-analysis: First-Line Antihypertensive Drug Rx and Health Outcomes
41
Case 5. Efficient portfolio management
Question: Can I monitor “real-time” results of all relevant studies (RCTsand non-RCTs) as soon as they are available? (and can I make course corrections if needed?)
EBM tool: Prospective cumulative meta-analysis
Answer: Yes, you should be.
42
Planning a Prospective Cumulative Meta-Analysis
Specific hypothesis objectives
Eligibility criteria of trials and of patients within trials
Outcome definitions
Subgroups
Analysis plan
Prospective registration of trials
Collection and analysis of IPD
Publication of main results
Permit additional questions (or trials) only if data are still blinded
Simes RJ. Am J Cardiology; 1995; 76: 899-905.
43
Cumulative Meta-analyses
Examples:
Cholesterol reduction (Prospective Pravastatin Pooling (PPP) Project , Cholesterol Treatment Trialists' )
FICSIT (Frailty and Injuries: Cooperative Studies of Intervention Techniques intervention efficacy in reducing falls and frailty in elderly patients)
Atrial fibrillation antithrombotic prophylaxis (SPORTIF III and V)
44
Summary: Applications of Systematic reviews in decision-making
Pros
Really can provide best available evidence
Transparent, comprehensive, & current
Improves process of decision-making
Improves decisions
Improves outcomes (?)
45
Summary: Applications of Systematic reviews in decision-making
Cons:
Application & interpretation of results may not be consistent (or transparent) across stakeholders (since local resources, values, & preferences also differ)
Potential for misapplication of results to support cost-containment strategies
“Easy” to mislead non-sophisticated audiences (hide behind EBM shield)
Only as good as the ‘raw material’ : Most medical practice is not supported by published literature; Safety info from literature is poor; Off-label use info is variable; studies of comparative effectiveness are infrequent) etc.
Labor-intensive to do & to update
Frequently only good for “first-order” effects (i.e., not subgroups) & no costs or utilization info
46
Systematic review & Meta-analysis: Applications
Thank you!
Send Questions or Comments to: [email protected]
47
Resources
Systematic reviews and evidence synthesis
1. Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med 1997;126:376-380
2. Users guide to the medical literature. Series in JAMA. http://pubs.ama-assn.org/misc/usersguides
3. http://www.cebm.net (Centre for Evidence-Based Medicine, Oxford, UK )
4. http://www.cochrane.org/resources/handbook/
48
Resources
Indirect treatment comparisons
1. Bucher HC, et al. The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. J Clin Epidemiol. 1997; 50:683-691.
2. McAlister FA, et al. JAMA 1999; 282: 1371-1377.
3. Caldwell DM et al. Simultaneous comparison of multiple treatments: combining direct and indirect evidence. BMJ 2005; 331: 897-900.
4. Eliot W & Mayer P. Incidence of diabetes in clinical trials of antihypertensive drugs: a network meta-analysis. Lancet 2007; 369: 207-.
5. Lu G & Ades AE. Combination of direct and indirect evidence in mixed treatment comparisons. Stats Med 2004; 23: 3105-3114.
6. Lumley T. Network meta-analysis for indirect treatment comparisons. Stats Med 2002; 21: 2313-2324.
49
Resources
Bayes:
1. Berger, JO (2000) “Bayesian Analysis: A Look at Today and Thoughts of Tomorrow,”Journal of the American Statistical Association. 95(452) 1269-1276.
2. Berry, D (2006) “Bayesian Statistics,” editorial in Medical Decision Making, Sept-Oct.
3. Raudenbush, SW & Bryk, AS (1985) “Empirical Bayes Meta-Analysis,” Journal of Educational Statistics. 10(2) 78-98.
4. Spiegelhalter, DJ, Myles, JP, Jones, DR, & Abrams, KR (1999) “An introduction to Bayesian methods in health technology assessment,” British Medical Journal. 319: 508-512.
5. Schmid CH. Using Bayesian Inference to Perform Meta-analysis. Evaluation & the Health Professions, Vol. 24, No. 2, 165-189 (2001)
50
Resources Individual Patient Data Meta-analysis
1. Stewart LA, Parmar MB. Meta-analysis of the literature or of individual patient data: Is there a difference? Lancet 1993; 341:418-422.
2. Steinberg K, Smith S, Stroup D, Olkin I, Lee N, Williamson G, and Thacker S. Comparison of Effect Estimates from a Meta-Analysis of Summary Data from Published Studies and from a Meta-Analysis Using Individual Patient Data for Ovarian Cancer Studies. Am J Epidemiol 1997; 145: 917–25.
Cumulative Meta-Analysis
1. Lau J, Schmid CH, Chalmers TC. Cumulative meta-analysis of clinical trials builds evidence for exemplary medical care. J Clin Epidemiol. 1995 Jan;48(1):45-57.
2. Berlin JA. Colditz GA.The Role of Meta-analysis in the Regulatory Process for Foods, Drugs, and Devices. JAMA. 1999;281:830-834.
3. Hu M, Cappelleri JC, KKG Lan. Applying the law of iterated logarithm to control type 1 error in cumulative meta-analysis of binary outcomes. Clinical Trials. 2007; 4:329-340.
4. Lan KKG, Hu M, Cappelleri JC. Applying the law of iterated logarithm to cumulative meta-analysis of a continuous endpoint. Statistica Sinica. 2003; 13:1135-1145.