The Incorporation of Meta-Analysis Results into Evidence-Based

Decision Modelling

Nicola Cooper, Alex Sutton,

Keith Abrams, Paul Lambert, David JonesDepartment of Epidemiology & Public Health,

University of Leicester.

CHEBS, Multi-Parameter Evidence Synthesis Workshop, Sheffield, March 2002

Where we fit in with Tony’s intro

• Process Model relationship between evidence & parameters

– Consistency check

• Uncertainty Panacea Statistical error

– ½ Evidence relates to parameters indirectly

– Systematic errors

– Data quality, publication bias, etc

1) Pooled estimates

Odds - log scale.1 .25 1 5

Combined

Bonneterre

Sjostrom

Nabholtz

Chan

                                     METHODOLOGIC PRINCIPLE

1) Pooled estimates

Odds - log scale.1 .25 1 5

Combined

Bonneterre

Sjostrom

Nabholtz

Chan

mu.rsprtD sample: 12001

-5.0 0.0 5.0

0.0 0.5 1.0 1.5 2.0

2) Distribution

                                     METHODOLOGIC PRINCIPLE

1) Pooled estimates

Odds - log scale.1 .25 1 5

Combined

Bonneterre

Sjostrom

Nabholtz

Chan

mu.rsprtD sample: 12001

-5.0 0.0 5.0

0.0 0.5 1.0 1.5 2.0

3) Transformation of distribution to transition probability (if required)

2) Distribution

(i) time variables:

(ii) prob. variables:

j

ttP j)],(1ln[exp1

0

jjo ttP /1)],(1[1

                                     METHODOLOGIC PRINCIPLE

1) Pooled estimates

Odds - log scale.1 .25 1 5

Combined

Bonneterre

Sjostrom

Nabholtz

Chan

mu.rsprtD sample: 12001

-5.0 0.0 5.0

0.0 0.5 1.0 1.5 2.0

3) Transformation of distribution to transition probability (if required)

2) Distribution

4) Application to model

(i) time variables:

(ii) prob. variables:

j

ttP j)],(1ln[exp1

0

jjo ttP /1)],(1[1

Respond

Stable

Progressive

Death

                                     METHODOLOGIC PRINCIPLE

                                     

1) Net Clinical Benefit Approach

• Warfarin use for atrial fibrillation

2) Simple Economic Decision Model

• Prophylactic antibiotic use in caesarean section

3) Markov Economic Decision Model

• Taxane use in advanced breast cancer

1) Net Clinical Benefit Approach

• Warfarin use for atrial fibrillation

2) Simple Economic Decision Model

• Prophylactic antibiotic use in caesarean section

3) Markov Economic Decision Model

• Taxane use in advanced breast cancer

EXAMPLES

• Bayesian methods implemented using Markov Chain Monte Carlo simulation within WinBUGS software

• Random effect meta-analysis models used throughout• All prior distributions intended to be ‘vague’ unless

otherwise indicated• Where uncertainty exists in the value of parameters

(i.e. most of them!) they are treated as random variables

• All analyses (decision model and subsidiary analyses) implemented in one cohesive program

                                     MODELLING ISSUES COMMON TO

ALL EXAMPLES

                                     EXAMPLE 1: NET (CLINICAL) BENEFIT

Net Benefit = (Risk level x Risk reduction) – Harm

• Glasziou, P. P. and Irwig, L. M. An evidence based approach to individualizing treatment. Br.Med.J. 1995; 311:1356-1359.

Risk

Red

uctio

n in

abs

olut

e ris

k (b

enef

it)

Exc

ess

abso

lute

ris

k (h

arm

)

Harm

Benefit

Threshold

• Evidence that post MI, the risk of a stroke is reduced in patients with atrial fibrillation by taking warfarin

• However, there is a risk of a fatal hemorrhage as a result of taking warfarin

• For whom do the benefits outweigh the risks?

                                     RE-ANALYSIS OF WARFARIN FOR NON-

RHEUMATIC ATRIAL FIBRILLATION

1) Perform a meta-analysis of the RCTs to estimate the relative risk for benefit of the intervention

2) Use this to check the assumption that RR does not vary with patient risk

3) Check harm (adverse events) is constant across levels of risk (use RCTs and/or data from other sources) & estimate this risk

4) Place benefit & harm on same scale (assessment of QoL following different events)

5) Apply model - need to predict patients risk (identify risk factors and construct multivariate risk prediction equations)

                                     METHOD OUTLINE

                                     SOURCES OF EVIDENCE

Net Benefit

= (risk of stroke x relative reduction in risk of stroke)

- (risk of fatal bleed x outcome ratio)

Multivariate riskequations M-A of RCTs

M-A of RCTs/obsstudies

QoL study

Risk of embolic stroke in control arm (%/year)

Re

du

ctio

n in

ab

solu

te r

isk

of

em

bo

lic s

tro

ke (

%/y

ea

r)

4 6 8 10 12

02

46

81

0

Embolic stroke

Intracranial haemorrhage

02

46

81

0E

xce

ss a

bso

lute

ris

k o

f in

tra

cra

nia

l ha

em

orr

ha

ge

(%

/ye

ar)

Singer,D.E. Overview of the randomized trials to prevent stroke in atrial fibrillation. Ann Epidemiol 1993;3:567-7.

                                     EVALUATING THE TRADE-OFF BETWEEN STROKE

AND HEMORRHAGE EVENTS IN TERMS OF QOL

• QoL following a fatal bleed = 0• Data available on QoL of patients following

stroke

– Glasziou, P. P., Bromwich, S., and Simes, R. J. Quality of life six months after myocardial infarction treated with thrombolytic therapy. The Medical Journal of Australia. 1994; 161532-536

Proportion with index greater than horizontal axis value

Time trade-off index

EVALUATION OF NET BENEFIT

(risk of stroke relative reduction in risk of stroke)

-

(risk of fatal bleed outcome ratio)

=

Net Benefit

Multivariate riskequations

Meta-analysesof RCTs

Meta-Analysis of RCTs /obs studies QoL study

0.002 0.004 0.006 0.008 0.010 0.012 0.014

050

100

150

200

250

300

risk of bleed per year

-2.95 -2.90 -2.85 -2.80 -2.75 -2.70 -2.65

02

46

810

-1.5 -1.0 -0.5 0.0 0.5 1.0

02

46

reduction in relative risk

0 20 40 60 80 100

0.0

0.1

0.2

0.3

0.4

Outcome ratio

Multivariate Risk Equation Data Net Benefit (measured in stroke equivalents) No.

Clinical risk

factors

No. of

patients

% of

cohort

Thrombo -embolism rate (% per year

(95% CI))

Mean (s.e.)

Median (95% CrI)

Probability of Benefit > 0

Simulated PDF

2 or 3

68

12

17.6 (10.5 to 29.9)

- 0.0004 (0.15)

0.06 (- 0.29 to 0.20)

54.2 %

-0.8 -0.6 -0.4 -0.2 0.0 0.2 0.4

01

23

45

6

2 or 3 Clinical factors

Relative risk reduction for strokes takingwarfarin (1-RR): 0.23 (0.13 to 0.41)

Outcome ratio (1/QoLreduction) Median 3.75 (1.07 to 50), Mean 26.14,indicating the number of strokes that are equivalent to one death

Risk of stroke per year e.g. for 1 or 2 clinical risk factors: 6.0% (4.1 to 8.8)

Risk of fatal bleed per year takingwarfarin : 0.52% (0.27 to 0.84)

Risk of Stroke (Rate % per year)

Net

Ben

efit

(Str

oke

Equ

ival

ents

)

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

-1.0

-0.8

-0.6

-0.4

-0.2

0.0

0.2

0.4

0.6

0.8

1.0

Probability of benefit > 0.95

0 1 or 2 >=3

0 1 2 or 3 No. of clinical risk factors

No. of combined risk factors

MeanMedian

                                     “TAKE-HOME” POINTS 1

Net-benefit provides a transparent quantitative framework to weigh up benefits and harms of an intervention

Utilises results from two meta-analyses and allows for correlation induced where studies included in both benefit and harm meta-analyses

Credible interval for net benefit can be constructed allowing for uncertainty in all model parameters

Use of prophylactic antibiotics to prevent wound infection following caesarean section

                                     EXAMPLE 2: SIMPLE DECISION TREE

No infection (1-p2) Cost with antibiotics

Yes

Infection (p2) Cost with antibiotics + Cost of treatment

Prophylactic antibiotics?

No infection (1-p1) Cost with no antibiotics

No

Infection (p1) Cost of treatment

1) Cochrane review of 61 RCTs evaluating prophylactic antibiotics use for caesarean section

2) Event data rare: use “Exact” model for RR 3) Meta-regression: Does treatment effect vary with patients’

underlying risk (pc)?

ln(RRadjusted ) = ln(RRaverage)+ [ln(pc) - mean(ln(pc))]4) Risk of infection without treatment from ‘local’ hospital

data (p1)5) Derive relative risk of treatment effect for ‘local’ hospital

(using regression equation with pc=p1)6) Derive risk of infection if antibiotics introduced to ‘local’

hospital (p2)

p2 = p1 * RRadjusted

                                     METHOD OUTLINE

                                     UNDERLYING BASELINE RISK

ln(R

ela

tive

Ris

k)

ln(control group risk) centred on mean)

ln(relative risk) fit

-2.5 -2 -1.5 -1 -.5 0 .5 1 1.5

-3

-2.5

-2

-1.5

-1

-.5

0

.5

1

1.5

2 =0.24 (-0.28 to 0.81)

Local hospital event rate

No treatment effect

                                     

rr[1] sample: 20000

0.1 0.2 0.3 0.4 0.5

0.0 2.5 5.0 7.5 10.0

p1 sample: 20000

0.025 0.075 0.1 0.125

0.0

20.0

40.0

60.0

p2[1] sample: 20000

0.0 0.02 0.04

0.0 25.0 50.0 75.0 100.0

  Mean (95% Credible Interval)

Posterior distribution

Relative Risk, RRadjusted 0.30 (0.21 to 0.40)

 

Prob(wound infection/placebo), p1

0.08 (0.06 to 0.10)

 

Prob(wound infection/antibiotics), p2

0.02 (0.015 to 0.034)

 

RESULTS

nwdreduct[1] sample: 20000

20.0 40.0 60.0 80.0

0.0

0.02

0.04

0.06

RESULTS

diff[1] sample: 20000

-150.0 -100.0 -50.0

0.0 0.01 0.02 0.03 0.04

tau.squared[1] sample: 20000

0.0 0.5 1.0 1.5

0.0

1.0

2.0

3.0

  Mean (95% Credible Interval)

Posterior distribution

Reduction in cost using antibiotics

-£49.53 (-£77.09 to

-£26.79)

 

Number of wound infections avoided using antibiotics per 1,000

53.09(42.12 to

73.37)

 

Between study variance (random effect in M-A), 2

0.30 (0.05 to 0.74)

 

                                     RESULTS (cont.)

                                     COST-EFFECTIVENESS PLANE

Control dominates

Treatment less effective & less costly

Treatmentdominates

Treatment more effective & more costly

-140

-120

-100

-80

-60

-40

-20

0

20

-20 0 20 40 60 80 100

Number of wound infections avoided per 1,000 caesarean sectionsC

ost

diff

ere

nce

                                     SENSITIVTY OF PRIORS

[1] Gamma(0.001,0.001) on 2

[2] Normal(0,1.0-6) truncated at zero on

[3] Uniform(0,20) on

[1]

[2]

[3]

caterpillar plot

Cost difference -80.0 -60.0 -40.0 -20.0

“TAKE-HOME” POINTS 2

Incorporates M-A into a decision model adjusting for a differential treatment effect with changes in baseline risk

Meta-regression model takes into account the fact that covariate is part of the definition of outcome

Rare event data modelled ‘exactly’ (i.e. removes the need for continuity corrections) & asymmetry in posterior distribution propogated

Sensitivity of overall results to prior distribution placed on the random effect term in a M-A

                                     EXAMPLE 3: USE OF TAXANES FOR 2ND LINE

TREATMENT OF BREAST CANCER

Stages 1 & 2(cycles 1 to 3)

Stage 3(cycles 4 to 7)

Stage 4(cycles 8 to 35)

In 2nd line treatment

Respond Stable Progressive Dead

Respond Stable Progressive Dead

Respond Stable Progressive Dead

Treatment cycles

Post -Treatment

cycles

1) Define structure of Markov model2) Identify evidence used to inform each model

parameter using meta-analysis where multiple sources available

3) Transform meta-analysis results, where necessary, into format required for model (e.g. rates into transition probabilities)

4) Informative prior distributions derived from elicited prior beliefs from clinicians

5) Evaluate Markov model

                                     METHOD OUTLINE

                                     META-ANALYSES No. of

studies Time in weeks

(95% Credible Interval) Progression-free time 3 25 (15 to 24)

Time to response from stable 1 12 (6 to 18) Time to progressive from response 1 35 (29 to 41)

Overall survival time 3 53 (35 to 74) Probabilities

Response rate 4 0.43 (0.29 to 0.58) % moving directly to progressive at stage 2. 1 0.13 (0.08 to 0.18)

% with infections / febrile neutropenia 3 0.18 (0.04 to 0.56) % hospitalised with infection / febrile neutropenia 1 0.08 (0.05 to 0.11)

% dying from infections / febrile neutropenia 1 0.01 (0.00 to 0.02) % discontinue treatment due to adverse event 3 0.16 (0.03 to 0.49)

% with Neutropenia grades 3 & 4 2 0.94 (0.82 to 0.98) % with Anaemia grades 3 & 4 2 0.03 (0.00 to 0.28) % with Diarrhoea grades 3 & 4 3 0.09 (0.06 to 0.14) % with Stomatis grades 3 & 4 3 0.08 (0.04 to 0.14) % with vomiting grades 3 & 4 2 0.03 (0.00 to 0.12)

% with fluid retention grades 3 & 4 3 0.05 (0.02 to 0.12) % with cardiac toxicity grades 3 & 4 1 0.00 (0.00 to 0.02)

                                     TRANSITION PROBABILITIES

Transition Probabilities (95% Credible Interval)

Infection/FN 0.09 (0.02 to 0.32)

Hospitalised due to infection/FN 0.04 (0.03 to 0.05)

Dying from infection/FN after hospitalisation 0.00 (0.00 to 0.01)

Discontinuation due to major adverse events 0.04 (0.04 to 0.16)

Adverse events – Neutropenia 0.50 (0.34 to 0.63)

Adverse events – Anaemia 0.01 (0.00 to 0.07)

Adverse events – Diarrhoea 0.02 (0.01 to 0.37)

Adverse events – Stomatis 0.02 (0.01 to 0.04)

Adverse events – Vomiting 0.01 (0.00 to 0.03)

Adverse events – Fluid retention 0.01 (0.00 to 0.03)

Adverse events – Cardiac toxicity 0.00 (0.00 to 0.01)

Transition directly to ‘progressive’ state 0.12 (0.08 to 0.18)

Transition ‘stable’ to ‘stable’ 0.65 (0.44 to 0.75)

Transition ‘stable’ to ‘response’ 0.16 (0.11 to 0.28)

Transition ‘stable’ to ‘progressive’ 0.18 (0.11 to 0.37)

Transition ‘response’ to ‘response’ 0.94 (0.93 to 0.95)

Transition ‘response’ to ‘progressive’ 0.06 (0.05 to 0.07)

Transition ‘progressive’ to ‘progressive’ 0.93 (0.79 to 0.96)

Transition ‘progressive’ to ‘death’ 0.07 (0.04 to 0.21)

1) Pooled estimates

Odds - log scale.1 .25 1 5

Combined

Bonneterre

Sjostrom

Nabholtz

Chan

mu.rsprtD sample: 12001

-5.0 0.0 5.0

0.0 0.5 1.0 1.5 2.0

3) Transformation of distribution to transition probability (if required)

2) Distribution

4) Application to model

(i) time variables:

(ii) prob. variables:

j

ttP j)],(1ln[exp1

0

jjo ttP /1)],(1[1

Respond

Stable

Progressive

Death

                                     METHODOLOGIC PRINCIPLE

                                     ELICITATION OF PRIORS

e.g. Response RateTaxane

x

x

x x x

x x x x x

x x x x x

x x

x x

x

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Standard

x

x

x x x x

x x x x x

x x x x x

x x

x x

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

                                     RESPONSE RATE

Response rate (docetaxel)

9585756555453525155

50

40

30

20

10

0

Std. Dev = 17.28

Mean = 38

N = 280.00

logit (Response rate for docetaxel)

3.5

3.0

2.5

2.0

1.5

1.0.5-.0

-.5

-1.0

-1.5

-2.0

-2.5

-3.0

-3.5

80

60

40

20

0

Std. Dev = .87

Mean = -.6

N = 280.00

Response rate for doxorubicin

85756555453525155

50

40

30

20

10

0

Std. Dev = 14.75

Mean = 31

N = 300.00

logit (Response rate for doxorubicin)

2.5

1.5.5-.5

-1.5

-2.5

-3.5

-4.5

-5.5

-6.5

-7.5

100

80

60

40

20

0

Std. Dev = .92

Mean = -1.0

N = 300.00

Bayesian (MCMC) Simulations

-£4,000

-£2,000

£0

£2,000

£4,000

£6,000

£8,000

£10,000

-0.50 -0.40 -0.30 -0.20 -0.10 0.00 0.10 0.20 0.30 0.40 0.50

Incremental utility

Incr

eme

nta

l co

st

Doxorubicin dominates

Docetaxel more effective but more costly

Docetaxel less costly but less

effective

Docetaxel dominates

                                     COST-EFFECTIVENESS PLANE

“TAKE-HOME” POINTS 3

Synthesis of evidence, transformation of variables & evaluation of a complex Markov model carried out in a unified framework (facilitating sensitivity analysis)

Provides a framework to incorporate prior beliefs of experts

                                     FURTHER ISSUES

• Handling indirect comparisons correctly•E.g. Want to compare A v C but evidence only available on A v B & B v C etc.•Avoid breaking randomisation

• Necessary complexity of model?•When to use approaches 1,2,3 above?

• Use of predictive distributions•Necessary when inferences made at ‘unit’ level (e.g. hospital in 2nd example) rather than ‘population’ level?

• Incorporation of EVI

• Handling indirect comparisons correctly•E.g. Want to compare A v C but evidence only available on A v B & B v C etc.•Avoid breaking randomisation

• Necessary complexity of model?•When to use approaches 1,2,3 above?

• Use of predictive distributions•Necessary when inferences made at ‘unit’ level (e.g. hospital in 2nd example) rather than ‘population’ level?

• Incorporation of EVI

                                     MODEL SPECIFICATION

)(exp ),(~

))log(,min()log( )log(

1,.....,61 ),(~ ),(~

2 ΔRRΔNormal

ppp

ipnBinomialrpnBinomialr

santibiotici

ciii

ti

cii

ti

ti

ti

ci

ci

ci

Warn et al 2002 Stats in Med (in press)

Bayesian random effects M-A model specification: ln(RR)

)001.0,001.0(~ (0,0.1)~

)100,1(~ )100,1(~ ),(~

2 maInverseGamτNormalΔ

UniformUniformBetapci

Prior distributions: