Technology Assessment, Outcomes Research and

Economic Analyses

Deborah A. Baumgarten, M.D., M.P.H.Emory University HospitalDepartment of Radiology

Rationale• Cost of medical care increasing faster than

cost of living• 5.6% of GDP in 1965; 17.6% 2012; 19.8% 2020• $27.5 B 1960; $2.71 T 2011; $4.64 T 2020 • $8649 per person in 2011; $13,709 2020

• US spends more than 33 others in Organization for Economic Co-operation and Development• But fewer MDs, hospital beds per capita

OECD spending on health care

OECD Health Data 2012

Health care expenditure as % GDP

OECD Health Data 2012

1980

1981

1982

1983

1984

1985

1986

1987

1988

1989

1990

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

2011

2012

2013

0

2

4

6

8

10

12

14

16

18 US (17.1%)FR (11.6%)SWE (11.5%)GER (11.2%)NETH (11.1%)SWIZ (11.1%)DEN (11.1%)NZ (11.0%)CAN (10.7%)JAP (10.2%)NOR (9.4%)AUS (9.4%)*UK (8.8%)

Health Care Spending as Percentage of GDP, 1980–2013

Percent

OECD Health Data 2015

Rationale• Workers now paying at least $1,094 more in

premiums annually for coverage than in 2000• Still waiting to see what ACA does…

• Despite expenditures, can we document significant improvement in health status?• Life expectancy• Perinatal mortality

Life expectancy

Infant mortality rate

Our poor health• Life expectancy 78.8 vs. 81.2 (2013)• Infant mortality 6.1/1000 vs. 3.5 (2011)• 68% adults over 65 have at least 2 chronic

conditions (33% UK, 56% Canada; 2014)• Obesity rate 35.3% (3.7% Japan, 30.6% NZ; 2014)• Smokers 13.7 (12.8 Australia, 24.1 France; 2013)

US does more of…United States Rank compared

with OECD countries

OECD median

MRI units 35.5 per 1,000,000 2nd (Japan 1st) 11.4 per 1,000,000

MRI exams 106.9 per 1000 50.6 per 1000

CT units 43.5 per 1,000,000 3rd (Japan 1st, Australia 2nd)

17.6 per 1,000,000

CT exams 240 per 1000 136 per 1000

Tonsillectomy 254.4 per 100,000 1st 130 per 100,000

Coronary bypass 79.0 per 100,000 3rd 47.3 per 100,000

Knee replacement 226 per 100,000 1st 121.6 per 100,000

Caesarian section 32.9 per 1000 live births

6th 26.1 per 1000 live births

OECD Health Data 2012 and 2015

Why?• Current payment structure means MDs get

paid more to do more interventions• ACA has not had effect yet…

• Fear of litigation• Patients ask for more—and usually do not

pay out of pocket (insurance covers)

Why?• MDs in US paid more but seen less

• 4 visits/year (OECD average 6.5 visits/year)• US uses more pharmaceuticals

• 2.2 drugs per adult (1.2 Netherlands)• Cost more (US=100; Australia=49; Germany=95)

• Procedures cost more • Bypass $75,345 vs. 42,130 in Australia• Appendectomy $13,910 vs 4,995 Netherlands

Technology Assessment

Bit of background• Technology assessment hierarchy• Based on efficacy (benefit under ideal

circumstances)• Differs from effectiveness (benefit in real

world)• Differs from efficiency (sufficient value?

adds economic component)

Fryback D, Thornbury J. The efficacy of diagnostic imaging. Med Dec Making 1991;11:88-94

Technology assessment hierarchy• Technical efficacy• Diagnostic accuracy• Diagnostic impact• Therapeutic impact• Patient outcomes• Societal outcomes

Fryback D, Thornbury J. The efficacy of diagnostic imaging. Med Dec Making 1991;11:88-94

If technology fails at lower level,

no point going further

Technical efficacy• Ability of technology to produce image• Measured by image characteristics

• Signal to noise, resolution of line pairs, etc.• Developmental: prototype and phantom,

safety• Exploratory: possible use, wide range of

patients, situations, conditions

Technical efficacy• Ideas of what falls here?

Diagnostic accuracy• Distinguish between disease, non-disease,

or normal, abnormal• Establish diagnosis if symptoms• Screen in asymptomatic patient

• Quantify severity or extent of disease• Prognostic information (staging)• Monitor therapy

• Need gold standard; measure of truth

Diagnostic accuracy• Sensitivity, specificity, positive and negative

predictive values • 2 x 2 table to display comparison to gold

standard or reference• Area under ROC curve• Most radiology research falls into this

category

Diagnostic accuracy• Ideas?

Diagnostic impact

• Which imaging tests have greatest impact on clinical decisions?

• Before test need to know• Differential diagnosis• Degree of certainty of diagnoses• Probability of each consideration

• Reassess after test performed• Best done prospectively

Diagnostic impact• Ideas?

Therapeutic impact• Which imaging tests have greatest impact on

therapeutic decisions?• Percent procedures avoided due to imaging• Percent imaging changed plan

• Retrospectively inferred from records• Prospectively assessed using clinicians’

stated plans

Therapeutic impact• Ideas?

Patient outcomes• Traditionally, morbidity or mortality• Change in quality adjusted life years (QALY)• Can be difficult to prove link as imaging

study several steps removed from patient outcome

Societal outcomes• Cost analyses from societal perspective• Usually reported as cost per QALY• Must be below some threshold to be

accepted (? $50,000/QALY)• Example, head CT to screen for brain mets

in lung cancer patients or only if symptomatic?• $70,000/QALY deemed too high to adopt

Outcomes research

Outcomes research—goals• Use imaging wisely

• Avoid over-diagnosis and over-treatment• Risks of treatment in early or mild disease

may outweigh benefits• Data that will permit maintenance of quality

at reduced cost• Increasingly important• Radiology needs to prove worth

Randomized controlled trial• Gold standard of outcomes research• Patients assigned to different groups based

on imaging (or no imaging) to be performed • Measure appropriate outcomes• Confounders and biases accounted for in

design

RCT—limitations• Cost• Size of population

• Power calculation• Estimate expected differences

• Time to complete• Obsolete by time outcomes assessed?• Technology as moving target

• Results may only apply to narrow spectrum of patients; generalizeability

RCT—unnecessary when?• Anecdotal evidence of benefit dramatic;

observational studies suffice• Example: head CT in trauma

• New technology:• Has same or fewer side effects• Has same or better accuracy• Is less expensive

RCT—necessary when?• Really don’t know answer• New technology:

• More expensive• May result in more morbidity• Very controversial

ACRIN—NLST• National Lung cancer Screening Trial• 25,000 patients randomized; low dose CT vs. CXR• Baseline, years 1 and 2• Q6 month health status and medical intervention

questionnaire• Very detailed description of what constitutes

positive screen and what to do• Outcomes

• Primary: Lung cancer specific mortality• Secondary: huge list

ACRIN—NLST• Low dose CT more sensitive than

radiography• Two annual incidence screenings with low-

dose CT resulted in • Decrease number of advanced-stage

cancers• Increase number of early-stage lung

cancers

NEJM 2013;369:920-921

Outcomes research—getting started• Choose technology to be evaluated• Define specific clinical indication• Define target population• Choose alternative techniques for

comparison (may include no imaging)• Define outcomes affected by technology

Example to work through• CT for appendicitis: improve outcomes?• Technology:

Example to work through• CT for appendicitis: improve outcomes?• Technology: MDCT

Example to work through• CT for appendicitis: improve outcomes?• Technology: MDCT (?oral, ?IV, ?rectal)• Indication:

Example to work through• CT for appendicitis: improve outcomes?• Technology: MDCT (?oral, ?IV, ?rectal)• Indication: RLQ pain, fever, etc.• Population:

Example to work through• CT for appendicitis: improve outcomes?• Technology: MDCT (?oral, ?IV, ?rectal)• Indication: RLQ pain, fever, etc.• Population: patients presenting to ER• Alternative:

Example to work through• CT for appendicitis: improve outcomes?• Technology: MDCT (?oral, ?IV, ?rectal)• Indication: RLQ pain, fever, etc.• Population: patients presenting to ER• Alternative: best clinical judgment plus lab• Outcomes:

Example to work through• CT for appendicitis: improve outcomes?• Technology: MDCT (?oral, ?IV, ?rectal)• Indication: RLQ pain, fever, etc.• Population: patients presenting to ER• Alternative: best clinical judgment plus lab• Outcomes: morbidity, mortality, perforation rate,

length of hospital stay, surgery rate, % normal appendices removed, cost to hospital, etc.

Example to work through• CT for appendicitis: improve outcomes?• Technology: MDCT (?oral, ?IV, ?rectal)• Indication: RLQ pain, fever, etc.• Population: patients presenting to ER• Alternative: best clinical judgment plus lab• Outcomes: morbidity, mortality, perforation rate,

length of hospital stay, surgery rate, % normal appendices removed, cost to hospital, etc.

• Is this feasible?

What if RCT not feasible?• Modeling: Use of decision analysis software• Need information from other studies

• Accuracy, sensitivity, specificity• Course of disease, natural history• Effectiveness of therapy• Patient utilities and costs

• Meta-analyses for combining results of studies

Modeling• Assumptions about test parameters

(sensitivity, specificity) based on literature • Then vary assumptions

• Assumptions about frequency of outcomes based on personal experience or literature (mortality and morbidity rates, etc.)• Allows variation of these

Modeling—literature exampleCervical spine screening with CT vs. plain films

Blackmore CC et al. Radiology 1999;212:117-125

CT in appendicitis example• Retrospective study of 651 patients treated before

use of CT for appendicitis (1992-1995)• 52% went to surgery; 24% for observation then

surgery; 24% observation to discharge• Initial surgical group, 81% appendicitis• Observation to surgery group, 80% appendicitis• Observation to D/C group, assumed no

appendicitis (no returns to their hospital)

Rhea JT et al. AJR 1997;169:113-118

CT in appendicitis example• Apply sensitivity and specificity of CT from

literature to 100 theoretical patients• Use morbidity, mortality, perforation, correct

diagnosis rates from study of 651 patients• Model impact of CT on outcomes

• Decrease in nontherapeutic surgery (13 patients), observation days (1 day less on average), perforation rate, etc.

• Alternative diagnoses also confirmed

Rhea JT et al. AJR 1997;169:113-118

Intermediate outcomes

• Choose levels lower in hierarchy• Diagnostic impact

• Change in diagnostic possibilities and confidence

• Before and after test ask referring MD:• Differential diagnosis• % certainty for each diagnosis• Degree of satisfaction with test

Intermediate outcomes• Seattle Low Back Pain Imaging Project (SLIP)• RCT of plain films vs. MR as initial imaging• Diagnostic impact

• 15% new diagnosis with plain film• 30% for MRI (spinal stenosis, HNP, etc)

• Change in probability for most likely diagnosis for HNP (p<.002)

Jarvik JG et al. Radiology 1997;204:447-454

Intermediate outcomes• Therapeutic impact

• Before test, ask MD for treatment plan• Observe if plan changes after test

• If temporally related, infer causal relationship between test and plan change

• SLIP: 38% avoided additional test if plain film first, 64% if MRI first (p<.001)

Intermediate outcomes• Advantages

• Cheaper and easier than full RCT (RCT may show intermediate outcomes if assessed as in SLIP)

• No one denied potentially good test• Works well for “add-on tests”

• Tests that would not necessarily replace existing work-up

Intermediate outcomes• Disadvantages

• Stated plans do not necessarily equal clinical action• If treatment invasive, difficult, expensive

or risky, easier to hypothetically choose• RCT still better for “replacement test”

Patient outcomes• Reduced mortality• Reduced morbidity

• Decrease complications from other tests• Avoid invasive procedures• Faster return to normal (or pretest) state

Patient outcomes• Quality of life: ability to undertake

rewarding and enjoyable activities• How person feels, functions

• Quality adjusted life years (QALYs)• Patient satisfaction, reassurance• Physiologic marker improvement• Improved function

Functional status• General and specific measures • SF-36 (Short Form 36)

• 36 questions, 8 domains• General health, pain, physical functioning,

social functioning, mental health, etc.• Roland score (back pain specific index)• Sciatic frequency index

SLIP follow-up• RCT ultimately enrolled 380 patients• Roland score at 12 months did not differ between 2

groups (8.75 plain film vs. 9.34 MRI, p = .53)• p >.75 for all domains of SF-36• Patient reassurance measure

• 1 month p=.09• 3 month p=.08• 12 month p=.002 (58% plain film vs. 74% MRI)• Increased reassurance with MRI over time

• 10 in MRI vs. 4 in plain film had surgery (p =.09)

Jarvik JG et al. JAMA 2003;289:2810-2818

Outcomes research—limitations• Tenuous link between test and patient outcome

• Test usually steps removed from outcome• Best situation: proven link between therapy

and outcome link diagnosis to therapy to outcome

• Example: detection of cerebral aneurysms with MRA treatment decreases morbidity and mortality link MRA to reduced morbidity/mortality

Outcomes research—limitations• If no proven impact on outcome, may be:

• Diagnostic test (inaccurate, insensitive)• Referring clinician (inappropriate use of

info; info does not reach)• Therapy (ineffective or unavailable)• Patient (lack of compliance)• System (lack of patient access)

Alternative outcome• Therapeutic value of diagnostic test• Diagnostic test can have direct impact on

quality of life• Reassurance important to patients and

referring clinicians• If consider only positive tests fail to consider

that negative test has value

Alternative outcome• Example: nonspecific chest pain• 176 patients randomized to receive no

studies or EKG and CPK level• Test group had less short term disability

(20% versus 46%; P=.001)• Use of diagnostic test independent predictor

of recovery in logistic regression• Test group felt care “better than usual”

Sox HC, et al. Ann Intern Med 1981;95:680-5

Economic Analyses

Economic evaluations• Must establish viewpoint of analysis• Society, insurer, provider, patient• All have different inputs and outcomes;

different costs to consider• Example: patient may value increased

quality of life at any cost while HMO may value decreasing overhead costs

Costs

• Value of resources • Charges as estimate• Medicare/Medicaid reimbursement as

estimate• Micro-costing techniques—all resources

identified and quantified• Consider cost of missed abnormalities and

superfluous work-up of false positives

Costs—examples• Medical

• Diagnostic test and interpretation• Nursing care

• Nonmedical• Time (travel, tests and treatment)• Travel• Support with daily activities• Lost productivity

Types of economic analyses• Cost identification or cost minimization

• Compares alternative health care strategies assumed equally effective

Cost identification—example• CT vs. US plus plain film after ESWL• 25 patients underwent all 3• Assumed equal accuracy• Exams timed; direct technical costs calculated

using procedural-based accounting system• CT 15.3 minutes; US + plain film 37.2 minutes• CT $36.86; US + plain film $57.60• Sensitivity analysis on time; within reasonable

range CT always cost less

Remer EM et al. Radiology 1997;204:33-37

Types of economic analyses• Cost benefit

• Costs and benefits assigned dollar values• Comparison of different technologies• Limited use in medicine due to difficulty

assigning monetary values to health states• Cost effectiveness leaves outcomes in

natural, objective units• # lives saved, # cancers detected, etc.

CEA—example• What gain from sixth stool guaiac?• Progression from fifth to sixth stool guaiac

costs $47 million/cancer detected• Can use intermediate outcomes (length of

stay, readmission rates, number of unnecessary surgeries prevented)

Neuhauser D, Lewicki AM. N Engl J Med 1975; 293

Cost utility analysis• Similar to CEA but uses subjective outcomes that

explicitly incorporate patient preferences• QALY: basic outcome measure• Year of life with significant morbidity less desirable

than year with excellent health• Full health equals 1• Death equals 0• Morbidity somewhere between 0 and 1

• Preferred method for economic evaluations

Utility values• Side effects from interferon—.93• Mild angina—.90• Prophylactic mastectomy, oophorectomy—.86• Moderate angina—.70• Hospitalization for tuberculosis—.60• Severe angina—.50• Moderate to severe stroke—.39• Recurrent stroke—.12

Components of economic analysis• Most with modeling software• Reference case (defines typical patient)• Strategies (set of alternative care pathways)• Time horizon (period to consider risks and

benefits)• Perspective (viewpoint of decision maker)• Effectiveness measure (outcome)• Decision tree allows varying components

Decision tree

Singer ME and Applegate KE. Radiology 2001;219:611-620

Components of economic analysis• Probabilities (all events in model have one)

• Estimated from historical data; literature, pilot study or expert opinion

• Utilities (assign utility to every health condition in model)

• Costs (identify all relevant)• Assumptions (due to limitations imposed by

available data; must explicitly state)

Economic analysis—example• Whole body CT screening• Base-case: cohort of 500,000 50 year old men• Sex, age varied in analysis• Benefits of screening assumed due to earlier

detection of disease, improvement in survival• 8 conditions included in model: ovarian, pancreatic,

lung, liver, kidney and colon cancer; aortic aneurysm, coronary artery disease

• Costs of screening, follow-up tests, patient care

Beinfeld, M. T. et al. Radiology 2005;234:415-422

Beinfeld, M. T. et al. Radiology 2005;234:415-422

Schematic of decision-analytic model with life expectancy and cost outcomes used in the cost effectiveness analysis

Economic analysis—example• Compared to routine care

• 6 days of life gained at $2513/patient• Incremental cost: $151,000/life-year gained• 90.8% had finding; only 2% had disease• 32.3% of total cost = work-up of FPs

• With favorable assumptions, one time screening not cost effective; would add burden to healthcare system

Beinfeld, M. T. et al. Radiology 2005;234:415-422

Economic analysis—limitations• Not all agree with strategies chosen

• Was comparison to standard of care?• Were all reasonable options considered?• Was reference case typical of patient population?

• Fears of bias if study funded by drug company or equipment manufacturer

• If too early, data scant; if late, decisions made and model obsolete• Noncontrast CT for renal colic: accepted standard

without study of cost

Conclusions

• Need to be familiar with concepts• Way to inform health care policy makers, guide

research• Goal to improve quality in rational manner, decrease

waste and unnecessary expenditure• Enhance Radiology community’s ability to participate

in health care decision-making locally and nationally• Preserve our specialty!

Thanks!

US does more of…

OECD Health Data 2015