evaluating sampling uncertainty in cost-effectiveness analysis daniel polsky university of...
TRANSCRIPT
![Page 1: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/1.jpg)
EVALUATING SAMPLINGUNCERTAINTY IN
COST-EFFECTIVENESS ANALYSIS
Daniel Polsky
University of Pennsylvania
http://www.uphs.upenn.edu/dgimhsr/
![Page 2: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/2.jpg)
Sampling Uncertainty
• A clinical trial is conducted on a sample from a population– Sampling uncertainty is the degree to which different samples
from the same population would produce different estimates– Sampling uncertainty is used when deciding whether to adopt a
new therapy based on efficacy results in a clinical trial • One must demonstrate that it is unlikely that a favorable estimate
from a clinical trial can be attributed to the luck of the draw
• This type of uncertainty is embedded in cost-effectiveness estimates that use clinical trial data– Sampling uncertainty for decisions based on cost-effectiveness
is equally important
![Page 3: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/3.jpg)
Outline
• Provide detailed steps on how to use a sample estimate of a cost effectiveness ratio in the presence of sampling uncertainty from a decision making perspective– Point estimates– Decision threshold– Confidence intervals and p‑values
• The objective is to improve the understanding of the various methods of expressing sampling uncertainty for the purposes of decision making– I will not focus on the technical aspects of estimation.
Computer code is available on my website.
![Page 4: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/4.jpg)
STEPS WHEN USING ESTIMATES FROM A SAMPLE FOR
DECISION MAKING
1. Determine point estimate in the sample
2. Compare against decision threshold
3. Quantify statistical uncertainty relative to decision threshold
‑ Confidence intervals
‑ P‑values
![Page 5: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/5.jpg)
Common Example• A drug company claims that their new drug (drug A) reduces blood
pressure more than placebo (drug B). They randomized 100 subjects to group A and 100 subjects to group B. They measured systolic blood pressure after 6 months.
1. What is the point estimate?
2. What is the decision threshold?
3. How would we express the uncertainty?
![Page 6: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/6.jpg)
COMMON EXAMPLE: ANSWERS
1. What is the point estimate? SBPb ‑ SBPa = 135.2 ‑ 125.8 = 9.4
2. What is the decision threshold? 0
3. How would we express the uncertainty?
Confidence interval: 9.4 + - (ttail* s.e.)95% confidence interval = (3.16, 15.64)
P-value: =.0035
![Page 7: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/7.jpg)
INTERPRETING 95% CI
Less
Greater
2) Confident A greater than B (A - B)
1) Not confident A differs from B (A - B)
3) Confident A less than B (A - B)
LL UL
LL UL
LL UL
Greater
Greater
Less
Less
Decision
threshold
^
Decision
threshold
^
Decision
threshold
^
•In our example, we are in category 2.
![Page 8: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/8.jpg)
COST EFFECTIVENESS EXAMPLE
• A pharmaceutical company has conducted an economic evaluation of two drugs. Their new drug (drug A) is cost effective relative to placebo (drug B).
• (The correlation between costs and effects=-.7102 and n=250 in each treatment group)
• They claim that their new drug is cost effective relative to drug B.
STEPS:1. What is the point estimate?2. What is the decision threshold?3. How would we express the uncertainty?
Drug A Drug B A-B s.e. of diff
Mean Cost 2000 1000 1000 3254
Mean QALY 0.86 0.85 .01 0.001929
![Page 9: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/9.jpg)
1. POINT ESTIMATE OF COST EFFECTIVENESS RATIO (CER)
NW
Treatment Dominated
SW
Cost-Effectiveness Ratio
SE
Treatment Dominates
(-) Difference in Effects (+)
(-)
Diff
ere
nce
in
Co
sts
(+
) NE
Cost-Efectiveness Ratio
•
CER=$100,000 / QALY
2000 1000 1000
.86 .85 .01CER
A B
A A
C C CCER
E E E
![Page 10: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/10.jpg)
2. DECISION THRESHOLD FOR THE COST EFFECTIVENESS RATIO
• The decision threshold for a CER is the maximum willingness to pay (WTP) for the marginal effects of the therapy– SE quadrant: less costly and more effective:
• Therapy would meet any WTP
– NW quadrant: more costly and less effective: • Therapy would not meet any WTP
– NE quadrant: More costly and more effective:• A tradeoff between costs and effects• Whether the effects are worth the cost depends on whether
the CER is less than the maximum WTP for those effects
![Page 11: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/11.jpg)
λ: WILLINGNESS TO PAY (WTP)
• Is there a value that could be applied across an entire population?– If so, what is it?– Is it $50,000 per QALY?
• There is no single value of λ that applies to all decision makers– When estimating effect differences, there is consensus that an
effect greater than zero is a reasonable decision threshold. No comparable certainty will ever exist for a decision threshold for cost effectiveness
• For cost-effectiveness analysis to be relevant for decision making, results must have meaning for decision makers with different decision thresholds (i.e., different WTP)
![Page 12: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/12.jpg)
DECISION RULE
• NEW THERAPY COST EFFECTIVE IF:
CER < λ
• NEW THERAPY NOT COST EFFECTIVE IF:
CER > λ
![Page 13: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/13.jpg)
NET MONETARY BENEFITS• An alternative method for expressing the results of a cost-
effectiveness analysis
• NMB is a rearrangement of the cost-effectiveness decision rule:CER=(ΔC) / (ΔE)<λ NMB=λ × (ΔE) - (ΔC)>0
• The decision threshold for NMB: Always 0• Decision rule: Interventions are cost-effective if NMB > 0
CAN ONLY NUMERICALLY EXPRESS NMB FOR A SINGLE λ • Given that λ might vary, this expression is limited in its usefulness to
decision makers
A GRAPH CAN EXPRESS NMB ACROSS A RANGE OF λ
![Page 14: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/14.jpg)
NMB GRAPH
0 100000 200000 300000 400000
Willingness to Pay
-3000
-2000
-1000
0
1000
2000
3000N
et
Mo
ne
tary
Be
ne
fits
Experiment 1
![Page 15: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/15.jpg)
3. EXPRESSING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS
1. 95% Confidence Intervals
- CER
- NMB
2. P-values
- Acceptability Curves
![Page 16: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/16.jpg)
CONFIDENCE INTERVALS FOR COST EFFECTIVENESS RATIOS
(-) Difference in Effects (+)
(-)
Diff
ere
nce
in C
ost
s (
+)
•Lower
Limit
Upper
Limit
Lines through the origin that each exclude α/2 of the distribution of the difference in costs and effects
![Page 17: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/17.jpg)
CONFIDENCE INTERVAL FOR EXAMPLE CER
-0.045 0.000 0.045 0.090 0.135
Difference in QALYs
-3000
-2000
-1000
0
1000
2000
3000
Diff
ere
nce
in
Co
sts
LL: 28,300LL: 245,200
Recall data from example:ΔC = 1000; SEC = 324; ΔQ = 0.01; SEQ = 0.001929; ρ = -.7102; DOF = 498
![Page 18: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/18.jpg)
CONFIDENCE INTERVAL OF CER RELATIVE TO DECISION THRESHOLD
2) Confident A cost-effective compared to B
1) Not confident A differs from B
3) Confident B cost-effective compared to A
Maximum
WTP
^
Maximum
WTP
^
Maximum
WTP
^
28,300 245,200
28,300 245,200
28,300 245,200
Confidence regarding whether the estimated cost-effectiveness ratio is good value for the money depends on the decision threshold (i.e., the maximum willingness to pay (WTP) for a unit out effectiveness).
![Page 19: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/19.jpg)
DISCONTINUITIES IN THE DISTRIBUTION OF THE COST-
EFFECTIVENESS RATIO • The distribution of a CER is discontinuous
on the real line– Because the denominator of a ratio can equal
0 (i.e., the ratio can be undefined)
• The standard techniques for estimating uncertainty can be problematic
![Page 20: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/20.jpg)
CONFIDENCE INTERVAL OF NMB
Must be estimated at a particular λ.
Using the example, the estimate the confidence interval of NMB at a λ of $50,000/ QALY is:
NMB=-$500CI of NMB= (-$1283, $283)
![Page 21: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/21.jpg)
CONFIDENCE INTERVAL OF NMB RELATIVE TO ITS DECISION THRESHOLD OF 0
Less
1) 8 = 50,000: Not confident A differs from B (A - B)
Greater
0^
0^
0^
2) 8 = 250,000: Confident A net beneficial compared to B
3) 8 = 10,000: Confident B net beneficial compared to A-236-1564
283
30 2970
-1283
One must calculate separate CI for each policy-relevant λ (i.e., at the decision threshold for CER)
The confidence interval derived for a single λ may have little in common with the confidence interval - and resulting policy inference - derived for other λ.
![Page 22: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/22.jpg)
CONFIDENCE FRONTIER OF NMB
0 100000 200000 300000 400000
Willingness to Pay
-3000
-2000
-1000
0
1000
2000
3000N
et
Mo
ne
tary
Be
ne
fits
Experiment 1
28,300 245,200
What conclusions would you draw about one’s confidence in the differences between drugs A and B given that CI for NMB includes its decision threshold of 0?
CI for NMB
![Page 23: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/23.jpg)
QUANTIFYING UNCERTAINTY USING P-VALUES /
ACCEPTABILITY CURVES
![Page 24: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/24.jpg)
PROBABILITY OF ACCEPTABILITY
• The probability that the estimated ratio falls below a specified ceiling ratio (i.e., a particular WTP or λ)
• This is analogous to (1-pvalue) for a CER at a decision threshold of that particular λ.
• Given that there is no agreed upon ceiling ratio, we routinely estimate the probability of acceptability over the range of possible positive values
NW
Treatment Dominated
NE
Cost Effectiveness Ratio
(-) Difference in Effects (+)
Acceptability Region
SW
Cost Effectiveness Ratio
SE
Cost Effectiveness Ratio
(-)
Diff
eren
ce in
Cos
ts (
+)
WSW
SSW
NNE
ENE
Ceiling R
atio
![Page 25: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/25.jpg)
ALTERNATIVE ACCEPTABILITY CRITERIA
• In this example, the ratio of $245,200 per QALY saved has the equivalent of a one-tailed p-value less than 0.025- i.e., 2.5% of the distribution falls above the $245,200 line)
-0.005 0.000 0.005 0.010 0.015 0.020
Difference in QALYs
-500
0
500
1000
1500
2000
2500D
iffe
ren
ce in
co
sts
4000 Replicates100 = 2.5%
3900: 245,200
3600: 179,600
2800: 127,700
1200: 76,800
100:28,300
400:49,100
![Page 26: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/26.jpg)
ACCEPTABILITY CURVES
• The acceptability curve is the plot of the probability of acceptability across values of WTP
• The curve’s value on the y-axis is the probability that therapy A is cost-effective. -This is analogous to 1 - P-value (one sided)
0 100000 200000 300000 400000
Willingness to Pay
0.00
0.25
0.50
0.75
1.00
Pro
po
rtio
n
Experiment 1
28,300 245,200
![Page 27: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/27.jpg)
Confidence interval for CERCER CI: (28,300 to 245,200)
Confidence Frontier of NMBnotice same CER CI: (28,300 to 245,200)
Acceptability Curvenotice same CER CI: (28,300 to 245,200)
-0.045 0.000 0.045 0.090 0.135
Difference in QALYs
-3000
-2000
-1000
0
1000
2000
3000
Diffe
ren
ce
in
Co
sts
LL: 28,300LL: 245,200
0 100000 200000 300000 400000
Willingness to Pay
-3000
-2000
-1000
0
1000
2000
3000
Ne
t M
on
eta
ry B
en
efits
Experiment 1
28,300 245,200
0 100000 200000 300000 400000
Willingness to Pay
0.00
0.25
0.50
0.75
1.00
Pro
po
rtio
n
Experiment 1
28,300 245,200
REVIEW OF THREE METHODSExample 1
![Page 28: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/28.jpg)
![Page 29: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/29.jpg)
BOTTOM LINE
If the goal is to provide information for decision makers, ultimately all three methods provide the same information. The same decision should be made under each method!
The difference is in the way the information is expressed
The choice of method should be based on the most effective way to express the point estimate and the uncertainty to decision makers rather than basing the choice on the statistical properties of each estimator
![Page 30: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/30.jpg)
Pattern #1: Where Expression Of Confidence Is Straight Forward
• Findings like those in Example 1 are referred to as pattern #1 findings
• Observed when:– Confidence interval for CI excludes the Y axis
• LL < point estimate < UL– Each NMB confidence limit crosses the decision threshold (0)
once– Acceptability curve crosses both α/2 and 1-α/2
PATTERN #1
One cannot be
confident the two
therapies differ
from one another
One can be
confident the
more effective
therapy is not
good value
One can be
confident the
more effective
therapy is
good value
Ceiling Ratiooo- oo
![Page 31: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/31.jpg)
There are situations whenConfidence Intervals for CERs do not follow
the standard pattern on a real line
-0.150 0.000 0.150 0.300
Difference in QALYs
-1500
-500
500
1500
2500
Diff
ere
nce
in C
ost
s
Experiment 3
Pattern #2: Pattern #3:
-0.045 0.000 0.045 0.090 0.135
Difference in QALYs
-3000
-2000
-1000
0
1000
2000
3000
Diff
ere
nce
in C
ost
s
UL: 28,100
LL: 245,800Experiment 2
The upper and lower limits are both greater than the point estimate
There is no line that can be drawn through the origin that excludes α/2 of the distribution of the difference in costs and effects
![Page 32: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/32.jpg)
Implications of odd patterns
• CIs for CERs can have seemingly odd properties– upper and lower limits that are both greater than or both less
than the point estimate (Pattern #2) – intervals that are undefined (Pattern #3)
• NMB and acceptability curve have been proposed as alternatives to overcome these problems
• Yet, all three methods yield identical policy inferences in for every joint distribution of ΔC and ΔE
• Choice of method to express sample uncertainty should be based on goals of communication of uncertainty rather than a statistical rationale
![Page 33: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/33.jpg)
Recommendations:Each method has advantages and
disadvantages • Graph expresses joint distribution of ΔC and
ΔE on CE plane • For a CER (one dimension), the CI for CER
provides the most information for decision making with the fewest numbers– The NMB is only relevant for a particular λ
• The acceptability curve reports boundaries for different levels of confidence
• NMB can be estimated on a per-patient basis which makes calculations of uncertainty trivial
![Page 34: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/34.jpg)
ESTIMATING UNCERTAINTYIN COST-EFFECTIVENESS
ANALYSIS
1. Confidence intervals for CERs (Fieller’s theorem)
2. Confidence intervals for NMB
3. Acceptability curves
See programs onlinehttp://www.uphs.upenn.edu/dgimhsr/stat%20cicer.htm
![Page 35: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/35.jpg)
Under which scenarios should costs and effects be expressed jointly?
• (1) Tradeoff between costs and effects – Joint comparison necessary
• (2) One therapy is unambiguously dominant over its alternative (i.e. significantly more effective and significantly less costly)– Joint comparison may not be necessary
![Page 36: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/36.jpg)
• (3) No significant difference in effect – Mistaken interpretation: A cost-minimization approach
is sufficient and there is no need to perform a joint comparison of costs and effects
• But if effects have a great deal of sampling uncertainty, a joint comparison may also provide a highly uncertain estimate even if costs have little uncertainty
• Need for a joint comparison still remains under most circumstances
Implications For Joint Comparison Of Costs And Effects
![Page 37: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/37.jpg)
• (4) No significant difference in costs or effects– Because it is possible to have more confidence in the
combined outcome of differences in costs and effects than in either outcome alone, observing no significant difference in costs and effects need not rule out that one can be confident that one of the two therapies is good value.
– One should jointly compare costs and effects, and one should report on their sampling uncertainty.
Implications For Joint Comparison Of Costs And Effects
![Page 38: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/38.jpg)
APPENDIX 1
• Detail for patterns 2 and 3
![Page 39: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/39.jpg)
CASE #2: WHERE EXPRESSION OF CONFIDENCE IS
NOT STRAIGHT FORWARD
• What happens when there is a possibility that the cost effectiveness ratio of a sample from a population may fall into three of the quadrants on the cost effectiveness plane?
![Page 40: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/40.jpg)
Case #2 Example:Confidence intervals for CER
-1 0 1 2
Difference in QALYS
-10000
-5000
0
5000
10000
Diffe
ren
ce
in
Co
sts
19,740
5,045NW
SW
NE
SE
![Page 41: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/41.jpg)
Case #2 Example:Confidence Frontier of NMB
0 10000 20000 30000 40000 50000
Acceptability Criterion (Ceiling Ratio)
-5000
-2500
0
2500
5000
7500
Ne
t M
on
eta
ry B
en
efits
![Page 42: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/42.jpg)
Case #2 Example:Acceptability Curve
0 10000 20000 30000 40000 50000
Acceptability Criterion (Ceiling Ratio)
0.00
0.25
0.50
0.75
1.00
Pro
po
rtio
n
![Page 43: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/43.jpg)
Case #2: Example
Confidence interval for CERCER CI: (19,740 [SW] to 5,045 [NE])
Confidence Frontier of NMBnotice same CER CI: (19,740 [SW] to 5,045 [NE])
Acceptability Curvenotice same CER CI: (19,740 [SW] to 5,045 [NE])
-1 0 1 2
Difference in QALYS
-10000
-5000
0
5000
10000
Diffe
ren
ce in
Co
sts
19,740
5,045NW
SW
NE
SE
0 10000 20000 30000 40000 50000
Acceptability Criterion (Ceiling Ratio)
-5000
-2500
0
2500
5000
7500
Ne
t M
on
eta
ry B
en
efits
0 10000 20000 30000 40000 50000
Acceptability Criterion (Ceiling Ratio)
0.00
0.25
0.50
0.75
1.00
Pro
po
rtio
n
![Page 44: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/44.jpg)
THIS EXAMPLE HAS A NON STANDARD
CONFIDENCE LIMIT PATTERN
PATTERN #2
Ceiling Ratiooo- oo
One cannot be
confident the two
therapies differ
from one another
One can be
confident that
one of the
therapies is
good value
One cannot be
confident the two
therapies differ
from one another
• Observed when: – Confidence interval for CI includes the Y axis (ΔE = 0)– One NMB confidence limit crosses the decision threshold twice; the
other limit never crosses the decision threshold– The acceptability curve crosses α/2 or 1-α/2 twice
![Page 45: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/45.jpg)
CASE #2BOTTOM LINE
If the goal is to provide information about whether there is an acceptable level of confidence relative to the decision threshold, all three methods still provide the same information
![Page 46: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/46.jpg)
CASE #3: No information
- The CI for the CER is undefined- The NMB CI contain the decision threshold for all ceiling ratios- The acceptability curve is always above α/2 and below 1-α/2
PATTERN #3
Ceiling Ratiooo- oo
One cannot be confident the two therapies
differ from one another
•One cannot be confident that one therapy differs from another
![Page 47: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/47.jpg)
APPENDIX 2:ESTIMATING UNCERTAINTYIN COST-EFFECTIVENESS
ANALYSIS
1. Confidence intervals for CERs (Fieller’s theorem)
2. Confidence intervals for NMB
3. Acceptability curves
See programs onlinehttp://www.uphs.upenn.edu/dgimhsr/stat%20cicer.htm
![Page 48: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/48.jpg)
Estimation of Confidence intervals for CER: Fieller’s theorem
• Fieller’s theorem method: A parametric method based on the assumption that the differences in costs and the differences in effects follow a bivariate normal distribution– i.e., the expression RΔE - ΔC is normally distributed
with mean zero (where R equals ΔC/ΔE and ΔE and ΔC denote the mean difference in effects and costs, respectively)
– Standardizing this statistic by its standard error and setting it equal to the critical value from a normal distribution generates a quadratic equation in R
• The roots of the quadratic equation give the confidence limits
![Page 49: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/49.jpg)
Fieller’s Theorem FormulaLower limit (LL): (M - [M2 - NO]½) / N
Upper limit (UL): (M + [M2 - NO]½) / N
Where:
• M = ΔEΔC – (tα/22) ρ sΔE sΔC
• N = ΔE2 – (tα/22)(s2
ΔE)
• O = ΔC2 – (tα/22)(s2
ΔC)
– ΔE and ΔC denote mean difference in effect and cost; – sΔE and sΔC denote estimated standard errors for the diff in costs and effects; – ρ equals the estimated Pearson correlation coefficient between the difference in
costs and effects; – tα/2 is the critical value from the t -distribution
![Page 50: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/50.jpg)
Data Required for Fieller’s Theorem
• The data needed to calculate these limits can be obtained from most statistical packages by:
• Testing the difference in costs (and obtaining ΔC and sΔC)
• Testing the difference in effects (and obtaining ΔE and sΔE)
• Estimating the correlation between the difference in costs and effects
![Page 51: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/51.jpg)
STATA code: Fiellerhttp://www.uphs.upenn.edu/dgimhsr/stat%20cicer.htm
Variable | Obs Mean Std. Dev. Min Max-------------+-------------------------------------------------------- cost | 500 25000 3655.213 15127.93 36227.73 qaly | 500 1 .0221151 .9266726 1.069604 treat | 500 .5 .5005008 0 1
. ipinputs cost qaly treat
Difference in cost: 999.99 SE, difference in cost: 324.18 Difference in effect: .01 SE, difference in effect: .0019 Correlation of differences: -.71 Degrees of freedom: 498
. fielleri 999.99 324.18 .01 .0019 -.71 498 0.95
![Page 52: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/52.jpg)
STATA results: Fieller
• Point Estimate (pe): 100000• Quadrant: NE
• Fieller 95 % Confidence Interval
• Lower limit : 28295
• Upper limit: 245263
![Page 53: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/53.jpg)
Estimation of NMB Confidence Intervals
• Use formula for a difference in two normally distributed continuous variables
NMB CI = NMB +- tα/2 SENMB
• Standard error for NMB equals:
2 2 2NMB C c E c C CSE s R s 2R s s
• where sΔE and sΔC denote estimated standard errors for the difference in costs and effects; ρ equals the estimated Pearson correlation coefficient between the difference in costs and effects; tα/2 is the critical value from the normal distribution
![Page 54: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/54.jpg)
STATA code: NMB confidence intervals
. ipinputs cost qaly treat
Difference in cost: 999.99 SE, difference in cost: 324.18 Difference in effect: .01 SE, difference in effect: .0019 Correlation of differences: -.71 Degrees of freedom: 498
. nmbi 999.99 324.18 .01 .0019 -.71 498 .95
![Page 55: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/55.jpg)
STATA Results: NMB
95 % 95 % Lower Upper Rc NMB limit limit P-value
-41604 -1416 -1953 -879 0.0000 -28322 -1283 -1849 -717 0.0000…
19240 -808 -1498 -117 0.0220 26226 -738 -1449 -27 0.0420 30977 -690 -1415 35 0.0620…
221172 1212 -154 2578 0.0820 234664 1347 -68 2761 0.0620 254013 1540 56 3024 0.0420 287956 1880 272 3487 0.0220
… 1196574 10966 5959 15972 0.0000 1465543 13655 7633 19678 0.0000
![Page 56: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/56.jpg)
Estimation of Acceptability Curves
• One means of deriving parametric acceptability curves is by estimating the 1-tailed probability that the net monetary benefits, calculated by use of the ceiling ratios defined on the X-axis, are greater than 0
![Page 57: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/57.jpg)
STATA code: Acceptability Curves
. ipinputs cost qaly treat
Difference in cost: 999.99 SE, difference in cost: 324.18 Difference in effect: .01 SE, difference in effect: .0019 Correlation of differences: -.71 Degrees of freedom: 498
. accepti 999.99 324.18 .01 .0019 -.71 498
![Page 58: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/58.jpg)
STATA Results: Acceptability Curves
Rc % Accept P-value
-7655559 0.03780 0.0756 -6886919 0.03785 0.0757 -3456387 0.03835 0.0767 …
18289 0.01000 0.0200 19240 0.01100 0.0220 26226 0.02100 0.0420 30977 0.03100 0.0620 …
221172 0.95900 0.0820 234664 0.96900 0.0620 254013 0.97900 0.0420 287956 0.98900 0.0220 … 2259921 0.96415 0.0717 3403528 0.96365 0.0727 6834078 0.96315 0.0737
![Page 59: EVALUATING SAMPLING UNCERTAINTY IN COST-EFFECTIVENESS ANALYSIS Daniel Polsky University of Pennsylvania](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649ddc5503460f94ad3d12/html5/thumbnails/59.jpg)
Summary
http://www.uphs.upenn.edu/dgimhsr/stat%20cicer.htm