uncertainty analysis - mahidol university · uncertainty analysis 5 sources of uncertainty are...

UNCERTAINTY ANALYSIS

Usa Chaikledkaew, Ph.D.Division of Social and Administrative Pharmacy, Department of Pharmacy,

Faculty of Pharmacy

Mahidol University, Thailand

4 MAY 2017

Outline2

Type of uncertaintyType of parameter uncertaintyDeterministic approachUnivariate or One-way sensitivity analysis Multivariate sensitivity analysis

Probabilistic approachProbabilistic sensitivity analysis (PSA)

ขั้นตอนการวิเคราะห ์CEA และ CUA3

กาํหนดปัญหา

กาํหนดทางเลือกที่ใช้ในการเปรียบเทียบ

กาํหนดต้นทนุและประสิทธิผลที่จะวดั

ประเมินค่าของต้นทนุและประสิทธิผล

แปลผลและรายงานผล

ประเมินค่าของตน้ทุนและประสิทธิผล

4

Model basedDecision tree modelMarkov model

DiscountingUncertainty analysis

4

Uncertainty analysis5

Sources of uncertainty are efficacy rates, adverse reaction rates, and cost estimates.

Because many of estimates used in the CEA are uncertain, we need to test the sensitivity of the results.

A mathematical and analytical technique designed to improve the quality and usefulness of analyses

6

Uses of Uncertainty

To test hypotheses about the sign and magnitude of costs, effectiveness, and C/E ratio

Decision makers may want to know how much confidence they should place on the results of an analysis

Help guide decisions about further research

7

Type of Uncertainty

8

Type of Parameter Uncertainty Analysis

Deterministic approachUnivariate or One-way sensitivity analysis Tornado diagramThreshold sensitivity analysis

Multivariate sensitivity analysisProbabilistic approachProbabilistic sensitivity analysis (PSA)Cost-effectiveness planeCost-Effectiveness Acceptability Curve 2

9

Weakness in Sensitivity Analysis

Analyst must choose which variables to vary and which to treat as know or fixed.

Analyst must choose the amount of variation around the base case value of the parameter that is considered clinically meaningful.

Analyst must determine how much of a change in the base result is acceptable or constitutes a robust finding.

10

Univariate Sensitivity Analysis

To examine one variable at a time by using High and low values Best and worst values One standard deviation 95% confidence intervals

Difficult to decide what a plausible range will be or to develop reasonable worst case extremes

11

Univariate Sensitivity Analysis

The results from a one-at-a-time sensitivity analysis will understate the overall uncertainty in C/E ratio because incremental cost and effectiveness depend on multiple parameters not just one.

However, it should be done because Logical and easy to understand the structure of a

particular CEA

12

One-way Sensitivity AnalysisICER

VariablesValue

ChangeCost/QALY

gained % Change

M in 0 . 4 5 1 4 , 9 5 3 3 . 8 0 %

Price of NSAIDs Base case 2 . 4 0 14,405M a x 5 .5 6 1 3 , 5 1 7 - 6 . 1 6 %

( + 1 0 % ) 5 4 . 6 7 1 5 , 9 2 9 1 0 . 5 8 %

Price of COX-2 Inhibitors Base case 4 9 . 7 0 14,405( - 1 0 % ) 4 4 . 7 3 1 2 , 8 8 0 - 1 0 . 5 8 %

M in 4 . 1 0 1 4 , 4 7 3 0 . 4 7 %

Probability of GI side effect due to NSAIDs Base case 6 . 6 5 14,405M a x 9 .2 0 1 4 , 3 3 7 - 0 . 4 7 %

M in 1 . 9 6 1 4 , 5 1 4 0 . 7 6 %

Probability of GI side effect due to COX-2 Inhibitors Base case 2 . 8 0 14,405M a x 3 .6 4 1 4 , 2 9 6 - 0 . 7 6 %

( - 1 0 % ) - 3 . 7 8 2 4 , 2 0 5 6 8 . 0 3 %

QALY lost due to NSAIDs Base case - 4 . 2 0 14,405( + 1 0 % ) - 4 . 6 1 1 0 , 2 5 4 - 2 8 . 8 2 %

( - 1 0 % ) - 2 . 8 5 1 0 , 9 7 8 - 2 3 . 7 9 %

QALY lost due to COX-2 Inhibitors Base case - 3 . 1 7 14,405( + 1 0 % ) - 3 . 4 9 2 0 , 9 4 3 4 5 . 3 9 %

One-way sensitivity analysisTornado diagram13

Threshold Sensitivity Analysis สาํหรบัการเจรจาตอ่รองราคายา

14

WTP or CE threshold = 100,000 –300,000 THB per QALY gained

ผลจากการเจรจาตอ่รองราคายา

Source: Teerawattananon et al, Z. Evid. Fortbild. Qual. Gesundh. wesen (ZEFQ) (2014) 108, 397—404

15

ราคายา Oxaliplatin ลดลง 68.75% สามารถประหยดังบประมาณได้ 152 ล้านบาท

16

The Analysis of Pricing Threshold of the HPV Vaccine Against the WTP Threshold

0

3 , 0 0 0

6 , 0 0 0

9 , 0 0 0

1 2 , 0 0 0

1 5 , 0 0 0

1 8 , 0 0 0

Vaccination at theage of 15 years




HPV price threshold at 1X GDPHPV price threshold at 3X GDP

48%55%

61%

26%

86%

74%

37%

97%

Source: HITAP 2008

Vaccine price in Thai Baht

Source: Praditsittikorn et al. HITAP 2009 14

17

Multiple Sensitivity Analysis

To vary multiple parameters at a timeTo find the set of extreme circumstances

across parameters (a combination of parameter values that yield the worst and the best CER)

18

Multiple Sensitivity Analysis

To select the combination of parameter values that would provide a worst-case analysis rather than looking for all extreme values

To use probabilistic (Monte Carlo) methods to simulate the model with assumptions about the variability in each of the parameters

Two-way sensitivity analysis19

Kiatpongsan et al. Plos One 2014. https://doi.org/10.1371/journal.pone.0106836

20

Probabilistic Sensitivity Analysis

21

Advantages of PSA

22

Steps of PSA

23

Choices of Distributions

Normal distributionBeta distributionGamma distribution

24

Beta, Gamma, Normal Distribution

10

Gamma

Beta

Normal

Variables e.g., price of drugs, probability of

death, etc.

Distributions of Variables e.g., normal, gamma, and

beta

Monte Carlo Simulation

“Simulated Trial”

Probability of ICER at Different Level

Probabilistic sensitivity analysis (PSA)

25

26

What Is Simulation?

A technique that imitates the operation of a real life system as it changes over time

Normally done by developing a simulation model

Modeling an artificial representation of a real life system

Today modeling is done on computers.

27

Simulation Model

Simulation model takes the form of a set of assumptions about the operation of the system, expressed as mathematical or logical relations between the objects of interest in the system

Input Parameters

(random)

Model for the system

Relationships developed to

represent the actual operation

of the real life system

Output Parameters

(random)

28

What Is Monte Carlo Simulation?

Invented by Metropolis and von Neumann A more scientific approach to manage

uncertainty in the model Analytical tool used to assure the random

selection of the input parameters according to their respective probability distributions for each evaluation

Several predictions of the behavior are obtained.

29

Principle of Monte Carlo Simulation

To develop an analytical model (computer based) that predicts the behavior of a system

Model is evaluated and behavior is predicted several times.

Each evaluation or simulation cycle is based on some randomly selected conditions for the input parameters for the system

30

Monte Carlo Simulation

Based on the distribution of the parameter estimates, we can simulate the cost-effectiveness ratio by repeatedly taking draws from the multivariate distribution of the estimates and the doing the cost-effectiveness analysis for each of those draws.

Software: @Risk, Crystal Ball

31

Distribution Attached Input variables

Results

Randomly sampled

Output

Collected

Model

Run many times

Monte Carlo Simulation Model

32

Probabilistic Sensitivity Analysis:

Cost-effectiveness plane

32

Median ICER = $268,000 per QALY

If a third party payer was willing to pay $50,000 per QALY gained for COX-2 therapy, only 0% of the patents in this simulation would fall within the budget.

Source: Spiegel et al. Ann Intern Med. 2003; 138: 795-806.

33

Probabilistic Sensitivity Analysis (PSA)

Median ICER = $268,000 per QALY

If a third party payer was willing to pay $50,000 per QALY gained for COX-2 therapy, only 0% of the patents in this simulation would fall within the budget.

Spiegel et al. Ann Intern Med. 2003; 138: 795-806.

34

B

A

D

C

increase in health eff

more costly

less costly

Intervention is less effective and more costly

Intervention is more effective and more costly

Intervention is less effective and less costly

Intervention is more effective and less costly

Cost‐effectiveness plane

decrease in health effects

Cost-effectiveness Plane ของ Current Practice และการใหย้าในกลุม่ statin

เปรยีบเทียบกบัการไม่ใหย้า statin ในเพศหญิงที่มีความเสี่ยงรอ้ยละ 15-20 ใน

10 ปี35

simvastatin(generic)_15_20

Pravastatin_15_20

Atorvastatin_15_20

- 6 0 ,0 0 0

- 4 0 ,0 0 0

- 2 0 ,0 0 0

-

2 0 ,0 0 0

4 0 ,0 0 0

6 0 ,0 0 0

8 0 ,0 0 0

1 0 0 ,0 0 0

1 2 0 ,0 0 0

1 4 0 ,0 0 0

1 6 0 ,0 0 0

1 8 0 ,0 0 0

2 0 0 ,0 0 0

2 2 0 ,0 0 0

- 1 5 0 ,0 0 0 - 1 0 0 ,0 0 0 - 5 0 ,0 0 0 0 5 0 ,0 0 0 1 0 0 ,0 0 0 1 5 0 ,0 0 0 2 0 0 ,0 0 0 2 5 0 ,0 0 0

(ล้าน

บาท

)

อรรถประโยชนท์ีไ่ด ้(Healthy Life Year saved)

ตน้ทนุทีใ่ช้

แพงกว่าและแย่กว่า

แพงกว่าแต่ดีกว่า

ถกูกว่าแต่แย่กว่า ถกูกว่าและดีกว่า

current practice_15_20

Simvastatin (original)_15_20

การแปลผลและรายงานผล36

Incremental cost-effectiveness ratio (ICER) is equal to

The cost that on average needs to be sustained to obtain “an

additional success”

(cost of treatment A) – (cost of treatment B)(clinical success treatment A) – (clinical success treatment B)

Or (cost of treatment A) – (cost of treatment B)(LYG A – LYG B)

Or (cost of treatment A) – (cost of treatment B)(QALY A – QALY B)

37

†ICERs are rounded up to nearest 1000 THB.

‡Negative ICER due to lower effectiveness and higher costs of the first-line 5-FU/LV and the second-line XELOX compared with

the first-line 5-FU/LV and the second-line capecitabine.

5-FU/LV: 5-Fluorouracil and leucovorin; ICER: Incremental cost–effectiveness ratio; FOLFOX: Fluorouracil/leucovorin/oxaliplatin;

THB: Thai baht; XELOX: Capecitabine/oxaliplatin;

FOLFIRI; Irinotecan plus 5-FU/LV.

การแปลผลและรายงานผล

38

เกณฑก์ารประเมินความคุม้ค่าฯ

UK: < £30,000 per QALY gained1

USA: < $50,000 per QALY gained2

Countries in the World: < 3 x GDP per capita per DALY averted

คณะอนุกรรมการพฒันาบญัชียาหลกัแห่งชาต3ิ

< 1 GDP per capita: คุม้ค่ามาก

1 – 3 GDP: อาจจะคุม้ค่า

> 3 GDP: อาจจะไม่คุม้ค่า

ผลการศึกษาบรรทดัฐานการประเมินคณุค่าของสงัคมตอ่การลงทุนทางการแพทยแ์ละ

สาธารณสุขในประเทศไทย

1 ปีสุขภาวะ (QALY) = 105,669 THB (about 1 GDP per capita in 2008)4

1 ปีสุขภาวะสาํหรบัมาตรการที่ใชใ้นการรกัษามีค่าประมาณ 160,000 บาท5

Source: 1) Towse, A., Devlin, N., Pritchard, C (eds) (2002) Cost effectiveness thresholds: economic and ethical issues. London: Office for Health Economics/King's Fund; 2) Devlin, N. and Parkin, D. Health Economics, 2004; 13: 437-452.; 3) คณะอนุกรรมการพฒันาบญัชียาหลกัแห่งชาติ. Threshold ที่เหมาะสมสาํหรับประเทศไทย. การประชุมคณะอนุกรรมการพฒันาบญัชี

ยาหลกัแห่งชาติ ครั้งที่ 9/2550, 20 ธนัวาคม 2550; หอ้งประชุมชยันาถนเรนทร สาํนกังานคณะกรรมการอาหารและยา; 4) Thavorncharoensap et al, 2009; 4) Thavorncharoensap et al, 2013

Cost‐effectiveness acceptability curve

Presents the relationship between X-axis: the value of ceiling ratios (Rc)

or willingness to pay for a unit of outcomesY-axis: the probability of favoring

each treatment strategies

1639

41

Cost‐effectiveness acceptability curve

WTP or CE threshold = 100,000 –300,000 THB per QALY gained

42

Recommendations

Trials that provide data on both costs and health effects should be large enough to provide adequate power for both costs and effectiveness.

Very little is known about the contribution of variations in the modeling process itself to the overall uncertainty in CEA results and this could be a priority area for research in the near future.

43

Thai HTA Guideline Recommendations

Probabilistic modelling and univariatesensitivity analysis should be used.

The results of each alternative analysis should be presented separately and in a probabilistic format.

Clear reporting of input parameters, model assumptions, and methods used in the modelling exercise is particularly important.

Thank you very much, Any question?

[email protected]

uncertainty analysis - mahidol university · uncertainty analysis 5 sources of uncertainty are...

Documents