translating evidence to recommendations within a …...nisha almeida, phd mcgill university health...
TRANSCRIPT
A framework for translating
evidence to
recommendations within a
hospital-based HTA unit
Nisha Almeida, PhD
McGill University Health centre
Disclosure
I have no actual or potential conflict of interest in relation to this topic or
presentation.
2
Background
Proliferation of HTA units
Hospital-based HTA units to support decision-making
Useful tools to adopt cost-effective and clinically pertinent technologies
Yet, while there are numerous guidelines to evaluate and rate the clinical
evidence (e.g. GRADE, Cochrane), there are few comparable tools to
guide decision-makers in framing recommendations
what factors must be considered in this decision-making process?
3
Factors important to decision-making
Everyone agrees that efficacy, safety and cost of the technology are important considerations
Reviewed 24 past TAU reports
Correlated the overall strength of 3 factors (efficacy, cost, safety) with the final recommendation
Approved: overall assessment supports a recommendation for routine use with support through the hospital operating budget
Approved for evaluation: overall assessment does not support permanent approval but is promising enough to warrant an evaluation,
Not approved: The technology is not approved for financial support through the hospital operating budget, but may be re-evaluated based on new evidence
But many more factors/dimensions to consider
4
0
1
2
3
4
5
6
7
8
APPROVED APPROVED FOR
EVALUATION
NOT APPROVED
Fre
qu
en
cy
Recommendation type
Weak Moderate Strong
Previous frameworks
GRADE1 OHTAC2
Domain Comment
Balance between
desirable and
undesirable
effects
The larger the difference between the desirable
and undesirable effects, the higher the
likelihood that a strong recommendation is
warranted. The narrower the gradient, the
higher the likelihood that a weak
recommendation is warranted
Quality of
evidence
The higher the quality of evidence, the higher
the likelihood that a strong recommendation is
warranted
Values and
preferences
The more values and preferences vary, or the
greater the uncertainty in values and
preferences, the higher the likelihood that a
weak recommendation is warranted
Costs (resource
allocation)
The higher the costs of an intervention—that is,
the greater the resources consumed—the lower
the likelihood that a strong recommendation is
warranted
Domain Sub-domain
Overall clinical
benefit
Effectiveness
Safety
Burden of illness
Need
Feasibility of
adoption
Economic feasibility
Organizational feasibility
Value for money Economic evaluations
Consistency with
expected societal
and ethical values
Expected societal values
Expected ethical values
1 Grading of Recommendations Assessment, Development and Evaluation, McMaster University
2 Ontario Health Technology Appraisal Committee
5
Goal
Exhaustively enumerate relevant domains important to the decision-making
process within a hospital setting
To create a checklist that is structured and transparent
to ensure that all relevant domains are given equal consideration
to adequately document the factors and reasons shaping the final
recommendation
That can be applied in a systematic fashion to each technology to be evaluated
6
Objectives
Identify relevant domains that stakeholders in the decision-making process
recognize as important to shape a recommendation.
Propose specific criteria to rate the strength of the evidence, information,
or experience, for each domain.
Estimate the extent to which members of the TAU Policy Committee
endorse the dimensions and criteria, and identify barriers and facilitators to
adoption of the checklist for the decision making process.
7
Methods
Used OHTAC framework as a starting point to define global domains:
Clinical benefit;
Cost;
Feasibility;
Ethics
Performed a review of our past reports, to identify domains that played a role
in the decision-making process
Two members of TAU independently reviewed a random sample of past reports
Considered domains from a survey of hospitals in France, that included 47
determinants (grouped under 8 domains) based on an extensive literature
review
8
Methods
Developed a preliminary list of important domains
Created a survey in order to solicit the opinions of experts within and
outside the MUHC community
To determine the importance of the suggested domains
Gather other relevant domains
Collect information on MUHC values and preferences
9
Survey domains Question: Which of the following domains do you think are most important for consideration by stakeholders when framing recommendations for use of a health technology in a hospital setting?
Domain Sub-domain
Clinical benefit • Magnitude of effectiveness
• Quality of evidence for effectiveness
• Safety
• Burden of illness to patient
• Absence of alternative treatment options
Value for money • Costs related to the technology
• Increased hospital efficiency
• External financial support
Feasibility • Ease of implementation
• Prior hospital experience with the technology
• Need for evidence of effectiveness in the local setting
Ethics and values • Disease is a public health priority
• Disease is rare
• Benefit of technology to society
• Impact on delivery of equitable care
• Impact on patient-important outcomes
Strategic
considerations
• Impact of technology on attracting new patients and/or health professionals
• Impact on creating research opportunities and external collaborations
• Ability to offer a cutting-edge technology or new alternative treatment
• Availability of the technology in other local centres
10
Comments 11
Results
We sent the survey out to 61
panelists, and received 52
completed responses (85%)
12
Type N (Total= 52)
MUHC member 38
MUHC non-member 14
HTA member 16
Administrator 12
0
10
20
30
40
50
60
70
80
90
100
MAGNITUDE OF EFFECTIVENESS
QUALITY OF THE EVIDENCE FOR EFFECTIVENESS
SAFETY OF THE TECHNOLOGY
BURDEN OF ILLNESS TO PATIENT
ABSENCE OF ALTERNATIVES
COST RELATED TO THE TECHNOLOGY
INCREASED HOSPITAL EFFICIENCY
EXTERNAL FINANCIAL SUPPORT FOR THE TECHNOLOGY
EASE OF IMPLEMENTATION
PRIOR HOSPITAL EXPERIENCE WITH THE TECHNOLOGY
NEED FOR EVIDENCE OF EFFECTIVENESS IN THE LOCAL SETTING
DISEASE IS A PUBLIC HEALTH PRIORITY
DISEASE IS RARE
BENEFIT OF TECHNOLOGY TO SOCIETY
IMPACT ON DELIVERY OF EQUITABLE CARE
IMPACT ON PATIENT-IMPORTANT OUTCOMES
IMPACT OF TECHNOLOGY ON ATTRACTING NEW PATIENTS AND/OR HEALTH PROFESSIONALS
IMPACT ON CREATING RESEARCH OPPORTUNITIES AND EXTERNAL COLLABORATIONS
ABILITY TO OFFER A CUTTING-EDGE TECHNOLOGY OR NEW ALTERNATIVE TREATMENT
AVAILABILITY OF THE TECHNOLOGY IN OTHER LOCAL CENTRES
Percent of respondents rating the domain 4 or 5
13
Percent of respondents rating the domain 4 or 5
0
10
20
30
40
50
60
70
80
90
100
MAGNITUDE OF EFFECTIVENESS QUALITY OF THE EVIDENCE FOR
EFFECTIVENESS
SAFETY OF THE TECHNOLOGY
BURDEN OF ILLNESS TO PATIENT
ABSENCE OF ALTERNATIVES
COST RELATED TO THE TECHNOLOGY
INCREASED HOSPITAL EFFICIENCY
EXTERNAL FINANCIAL SUPPORT FOR THE TECHNOLOGY
EASE OF IMPLEMENTATION
PRIOR HOSPITAL EXPERIENCE WITH THE TECHNOLOGY
NEED FOR EVIDENCE OF EFFECTIVENESS IN THE LOCAL SETTING
DISEASE IS A PUBLIC HEALTH PRIORITY
DISEASE IS RARE
BENEFIT OF TECHNOLOGY TO SOCIETY
IMPACT ON DELIVERY OF EQUITABLE CARE
IMPACT ON PATIENT-IMPORTANT OUTCOMES
IMPACT OF TECHNOLOGY ON ATTRACTING NEW PATIENTS AND/OR HEALTH
PROFESSIONALS
IMPACT ON CREATING RESEARCH OPPORTUNITIES AND EXTERNAL
COLLABORATIONS
ABILITY TO OFFER A CUTTING-EDGE TECHNOLOGY OR NEW ALTERNATIVE
TREATMENT
AVAILABILITY OF THE TECHNOLOGY IN OTHER LOCAL CENTRES
MUHC members
MUHC non-members
14
Consensus on domains
Domain Sub-domain Consensus
Clinical benefit
• Magnitude of effectiveness 100
• Quality of evidence for effectiveness 94.2
• Safety 98.0
• Burden of illness to patient 76.9
• Absence of alternative treatment options 57.7
Value for money
• Costs related to the technology 92.3
• Increased hospital efficiency 92.3
• External financial support 21.2
Feasibility • Ease of implementation 78.8
• Prior hospital experience with the technology 34.6
• Need for evidence of effectiveness in the local setting 61.5
Ethics and values
• Disease is a public health priority 69.2
• Disease is rare 46.2
• Benefit of technology to society 73.1
• Impact on delivery of equitable care 78.8
• Impact on patient-important outcomes 90.4
Strategic
considerations
• Impact of technology on attracting new patients and/or health professionals
36.5
• Impact on creating research opportunities and external collaborations 51.9
• Ability to offer a cutting-edge technology or new alternative treatment 69.2
• Availability of the technology in other local centres 63.5
15
Absence of alternatives
Rating Frequency
0 (Don't know) 1.9
1 (Not at all
important) 1.9
2 (Not very
important) 1.9
3 (Somewhat
important) 36.5
4 (Important) 38.4
5 (Indispensable) 19.2
Comments
Depends on context of the patient; would
be important to consider in rare diseases with
serious negative outcomes
Clearly important given financial constraints
The weight to give at this criterion is unclear.
Absence of alternative treatment should not
be used to justify the decision of introduce a
medical technology when there is weak
evidence;
alternative treatment options arguably
belongs in the ethics domain
There are always alternative options. It
should be explicit and understood that
"magnitude of effectiveness" means
magnitude compared to the next best
options
16
Prior hospital experience with
technology
Rating Frequency
0 (Don't know) 0
1 (Not at all
important) 7.7
2 (Not very
important) 23.0
3 (Somewhat
important) 34.6
4 (Important) 26.9
5 (Indispensable) 7.7
Comments
if you are doing a reassessment,
then this item would be important.
Not intrinsically important
although it may help to get going
Not unless this prior experience
generated data on clinical
benefit, cost or other aspects of
interest. Again, something to be
flagged on a "Do not consider" list.
17
Suggestions for new domains
Clarify: comparative effectiveness and safety
Level of innovation
Impact on patient outcomes
No. of patients affected
Cost-effectiveness
Quantify cost-avoidance
Ability to increase cross-institution collaboration
Impact on budget of other units
Impact on services of other units (disruptiveness)
Need for the technology
Impact on attractiveness of the centre
Clinical pertinence
18
Reorganization of domains Old version
Domain Sub-domain
Clinical benefit
1. Magnitude of effectiveness
2. Quality of evidence for effectiveness
3. Safety
4. Burden of illness to patient
5. Absence of alternative treatment options
Value for money
6. Costs related to the technology
7. Increased hospital efficiency
8. External financial support
Feasibility 9. Ease of implementation
10. Prior hospital experience with the technology
11. Need for evidence of effectiveness in the local setting
Ethics and values
12. Disease is a public health priority
13. Disease is rare
14. Benefit of technology to society
15. Impact on delivery of equitable care
16. Impact on patient-important outcomes
Strategic
considerations
17. Impact of technology on attracting new patients
18. Impact on creating research opportunities
19. Ability to offer a cutting-edge technology
20. Availability in other local centres
Domain Sub-domain
Clinical benefit
1. Magnitude of effectiveness
2. Quality of evidence for effectiveness
3. Safety
Impact on
Patient
4. Impact on patient convenience
5. Personal utility: patient values and preference
6. Patient-reported outcomes (QoL)
Value for
money (local setting)
7. Budget impact (Net cost)
8. Costs avoided /increased hospital efficiency
9. No. of patients affected by technology
10. Budget impact on other services
11. Cost-effectiveness
Feasibility
(impact on local setting)
12. Availability of local expertise
13. Disruptiveness
14. Need to generate local evidence
15. Ability to increase cross-institution collaboration
16. Personnel satisfaction
17. Impact of innovativeness of the technology
Impact on
healthcare system
18. Benefit to society (reduces health care costs)
19. Burden on other health care centres
20. Need: unnecessary duplication
Ethical considerations: Disruption of access to care
New version 19
Rating the strength of each criterion 20
DOMAINS Sub-domains RATING, COMPARED TO ALTERNATIVES
Clin
ica
l
be
ne
fit
Magnitude of effectiveness Strong effect Moderate effect Non-inferior
Quality of the evidence Strong Moderate Weak
Safety of the technology Strong effect Moderate effect Non-inferior /Concern for
harm
Imp
ac
t o
n
Pa
tie
nt
Patient perception and
preferences/personal utility
Superior Equivalent Inferior
Impact on patient convenience (pain, side
effects, time, invasiveness)
Superior Equivalent Inferior
Pt-reported outcomes( satisfaction, impact
on QoL/reduction in period of disability)
Superior Equivalent Inferior
Va
lue
fo
r
mo
ne
y
Budget impact (net cost) Low Equivalent High
Costs avoided High Equivalent Low
Number of patients affected Low Equivalent High
Impact on budget of other departments Low Equivalent High
Cost-effectiveness, if available High Equivalent Low
Fe
asi
bili
ty
Availability of local expertise Yes No
Disruptiveness of other services Low Equivalent High
Need to generate local evidence Yes No
Cross-institution collaboration Yes No
Satisfaction of personnel High Equivalent Low
Innovativeness of the technology High Equivalent
Imp
ac
t
on
he
alth
syst
em
Benefit of the technology to society Yes No
Burden on other healthcare services Low Equivalent High
Need for the technology Yes No
Ethical
considerations
Disruption of access to care Low High
An example: IORT for early breast cancer 21 Decision domain Judgment Reason for judgment
Clin
ica
l
be
ne
fit Magnitude of effectiveness Inconclusive Recurrence rate: 3.3% vs. 1.3%. No CIs provided to judge non-inferiority criterion.
Quality of the evidence Low single RCT with serious concerns in the analysis and presentation of results
Safety profile of intervention Equivalent to alternative Complication rate: 17% in IORT vs 15.5% in EBRT;
Pa
tie
nt
imp
ac
t Patient personal utility Superior
Impact on patient convenience Superior Would reduce inconvenience associated with weekly external beam radiation
Pt-reported outcomes Superior
Va
lue
fo
r m
on
ey
Net cost (Budget impact) Moderate If 15 IORT procedures are performed annually instead of EBRT, it would result in
an increase of $11500 in the budget
Costs avoided Balanced Balanced because IORT increases OR time (takes an additional 40-60 minutes
after surgery), but may result in decrease in wait times for EBRT
Number of patients affected Moderate 16 women have been treated with Intrabeam® since November 2013
Budget impact on other services
Cost-effectiveness
Fe
asi
bility
Availability of local expertise Yes
Disruptiveness of other services
Need to generate local evidence Yes single RCT with several concerns and inconclusive results; a field evaluation may
help answer concerns about recurrence rates, and risk
Cross-institution collaboration
Satisfaction of personnel High MUHC radiation oncologists report good experience with using Intrabeam
Innovativeness of the technology High
Imp
ac
t
on
he
alth
syst
em
Benefit of the technology to society Yes
Burden on other health services
Need for the technology Yes
Ethical concerns None
Overall assessment Moderate: Approval for
evaluation
Intrabeam has the potential of reducing unnecessary exposure to radiation, and
the burden on women of repeated EBRT visits.
However, the procedure remains experimental, and without longer term results
on local breast cancer recurrences and safety, this technology can only be
approved within the context of a research study at the MUHC
Future directions
Pilot test the checklist and ratings among members of the MUHC TAU policy
committee: hospital administrators, patient representatives, and quality
and risk management representatives
solicit their input on the practicality and usability of the checklist
Based on feedback, the checklist will be updated and re-tested at a
second meeting, with the eventual goal of pilot testing the checklist at
other hospital-based technology assessment units.
22
Conclusions
Explicit frameworks directed at hospital-based HTAs do not exist
Our framework would
help in creating a systematic and standardized evaluation process by ensuring
all relevant domains are given due consideration
document the decision-making process in an explicit fashion
enable an understanding of the value judgments and reasoning behind
reaching a recommendation
23
Final list of domains DOMAINS
Clinical benefit Magnitude of effectiveness, all clinically important outcomes
Quality of the evidence (including consistency of magnitude and direction of effect between studies)
Safety of the technology
Impact on Patient Impact on convenience (invasiveness, pain, time)
Patient personal utility: Patient perception and preferences
Patient-reported outcomes (QoL)
Value for money
(Impact on the
local setting)
Net cost (includes acquisition, maintenance, procedure and training costs, and accounts for external support)
Costs avoided, measured as increased hospital efficiency
Impact on budget of other departments
Number of patients affected by the technology (budget impact)
Cost-effectiveness, if available
Feasibility
(Impact on the
local setting)
Availability of local expertise (clinical, technical) in the technology
Impact on resources of affected department and/or of services provided by other departments (disruptiveness)
Need to generate evidence of effectiveness in the local setting (due to lack of strong evidence from RCTs)
Ability to increase cross-institution collaboration (through case-sharing or research activities)
Satisfaction of personnel involved with or affected by the technology
Impact of innovativeness of the technology (increase attractiveness of hospital to patients /professionals)
Impact on
healthcare
system /society
Benefit of the technology to society (e.g. technology enables patient to return to work faster, or reduces use of
antibiotics, resulting in cost-savings to healthcare system)
Burden on other healthcare centres: transfers to other centres/ increase in home monitoring
Need for the technology: does it support local innovation and economic growth, or is it unnecessary duplication
of services that increases healthcare costs?
Ethical
considerations
Disruption of access to care by introduction of technology; or by refusal to introduce technology because of cost
alone, when no alternatives available
25
Burden of illness
Rating Frequency
0 (Don't know) 0
1 (Not at all
important) 0
2 (Not very
important) 3.8
3 (Somewhat
important) 19.2
4 (Important) 67.3
5 (Indispensable) 9.6
% rating 4 or 5 76.9
Comments
Mostly important on a national
level
Important but can be very difficult
to capture/compare
In theory this information should
be captured by a cost-
effectiveness metric. A disease
with a low burden of illness should
result in a smaller gain in quality-
adjusted life years and a large
ICER.
burden of illness is
controversial...not sure it belongs
in clinical benefit
important factor but would have
like to had the prevalence of the
problem for patient
26
Survey domains 27 Question: Which of the following domains do you think are most important for consideration by
stakeholders when framing recommendations for use of a health technology in a hospital setting?
Domain Sub - domain
Clinical benefit • Magnitude of effectiveness
• Quality of evidence for effectiveness
• Safety
• Burden of illness to patient
• Absence of alternative treatment options
Value for money • Costs related to the technology
• Increased hospital efficiency
• External financial support
Feasibility • Ease of implementation
• Prior hospital experience with the technology
• Need for evidence of effectiveness in the local setting
Ethics and values • Disease is a public health priority
• Disease is rare
• Benefit of technology to society
• Impact on delivery of equitable care
• Impact on patient-important outcomes
Strategic
considerations
• Impact of technology on attracting new patients and/or health professionals
• Impact on creating research opportunities and external collaborations
• Ability to offer a cutting edge technology or new alternative treatment
• Availability of the technology in other local centres
28
DOMAIN SUB-DOMAIN % OF PARTICIPANTS RATING THE DOMAINS AS IMPORTANT OR INDISPENSABLE (A
RATING OF 4 OR 5)
MUHC MEMBER
(N=38)
MUHC NON-
MEMBER (N=14)
HTA (N=16) ADMINISTRATOR
(N=12)
TOTAL
(N=52)
CLI
NIC
AL
BEN
EFIT
Magnitude of effectiveness 100 100 100 100 100
Quality of the evidence for effectiveness 92 100 100 92 94
Safety of the technology 97 100 100 92 98
Burden of illness to patient 79 71 75 83 77
Absence of alternatives 50 79 81 50 58
VA
LUE
FO
R
MO
NEY
Cost related to the technology 89 100 100 92 92
Increased hospital efficiency 97 79 88 92 92
External financial support for the technology 21 21 25 17 21
FEA
SIB
ILIT
Y
Ease of implementation 79 79 81 75 79
Prior hospital experience with the technology 37 29 25 42 35
Need for evidence of effectiveness in the
local setting
61 64 69 75 62
ETH
ICS A
ND
VA
LUES
Disease is a public health priority 74 57 56 75 69
Disease is rare 42 57 44 33 46
Benefit of technology to society 82 50 56 75 73
Impact on delivery of equitable care 84 64 69 92 79
Impact on patient-important outcomes 92 86 88 100 90
STR
ATE
GIC
CO
NSID
ER
ATI
ON
Impact of technology on attracting new
patients and/or health professionals
39 29 31 33 37
Impact on creating research opportunities 53 50 44 50 52
Ability to offer a cutting-edge technology 74 57 44 83 69
Availability of the technology in other local
centres
63 64 69 58 63