identifying evidence for decision-analytic models suzy paisley doh research scientist in evidence...
TRANSCRIPT
Identifying evidence for decision-analytic models
Suzy PaisleyDoH Research Scientist in Evidence Synthesis
Consensus Working Group on the Use of Evidence in Economic Decision Models : Establishing the Current Situation
MRC HSRC Workshop 25 July 2005
Consensus Working Group on the Use of Evidence in Economic Decision Models
DoH Award
To develop a systematic approach to searching for evidence to inform decision-analytic models
Characteristics and sources of evidence to inform parameter values
Use of evidence in informing other parts of the modelling process
Consensus Working Group on the Use of Evidence in Economic Decision Models
Today’s presentation
Why an issue Current practice
– Identification of evidence– Use of evidence
Implications of current practice
Consensus Working Group on the Use of Evidence in Economic Decision Models
Why an issue?
Environmental issues Own experience
– Undertaking HTA searches– Peer review of HTA reports
Consensus Working Group on the Use of Evidence in Economic Decision Models
‘Typical’ HTA
Decision problem / research question Syntheses of existing evidence Systematic review of clinical-
effectiveness Decision-analytic model of cost-
effectiveness
Consensus Working Group on the Use of Evidence in Economic Decision Models
Decision problemsearch question
P = colorectal cancer
I = irinotecan
C = any relevant
O = survival
#1 colorectal
#2 irinotecan
#3 rct
#4 #1 and #2 and #3
Consensus Working Group on the Use of Evidence in Economic Decision Models
Modelling decision problems
Effectiveness Costs Resource use / activity Health states Utility values Indirect comparators Longer term outcomes ‘Other’ interventions Natural history Epidemiology
Consensus Working Group on the Use of Evidence in Economic Decision Models
Current practice
Analysis of 27 technology assessment reports (TARs)
Review of HTA guidelines Content analysis of modelling guidelines
Consensus Working Group on the Use of Evidence in Economic Decision Models
Current practice:
Reports of models
Use evidence which cannot be identified systematically by PICO focused searches(21/27 (78%) assessments for NICE Paisley, 2001)
Report PICO focussed searches(4/21 (19%) reported additional ‘ad hoc’ searches Paisley, 2001)
Issues of transparency
Consensus Working Group on the Use of Evidence in Economic Decision Models
Current practice:
Reports of modelsTypes
EffectivenessCostsResource use / activityHealth statesUtility valuesIndirect comparatorsLonger term outcomes‘Other’ interventionsNatural historyEpidemiology
Sources
‘Published papers’Routine dataReference sourcesLocal / clinical /
expert opinionSponsor submissions
Uses
Parameter valuesModel structureSensitivity analysisValidation / consistency / calibration
Consensus Working Group on the Use of Evidence in Economic Decision Models
Current practice:
Methodological guidance
HTA Guidelines Modelling guidelines
Consensus Working Group on the Use of Evidence in Economic Decision Models
Current practice:
HTA guidelines (n=32)
Identification of evidence:
No mention 13 (41%)Systematic review approach 13 (41%)Broad approach 2 (6%)Approach unclear 2 (6%)Not available 2 (6%)
Name data sources 21 (66%)
Consensus Working Group on the Use of Evidence in Economic Decision Models
Current practice:
Modelling guidelines
ISPOR Principles of Good Practice (Weinstein et al, 2003)
1 explicit reference to systematic reviews of the literature
41 implicit references to a process of seeking evidence
30/41 (73%) references form statements within recommendations
Consensus Working Group on the Use of Evidence in Economic Decision Models
Current practice:
Modelling guidelines
Model structure 10 (33%)
Data identification 7 (23%)
Data modelling 7 (23%)
Data incorporation 0 (0%)
Model validation 6 (20%)
Consensus Working Group on the Use of Evidence in Economic Decision Models
Current practice:
Conclusions
Reports of models– Report PICO focussed searches– Use a wide range of evidence from a wide
range of sources throughout the modelling process
Modelling and HTA guidelines– Give limited explicit guidance– Give an implicit indication that range of
sources should be used throughout the modelling process
Consensus Working Group on the Use of Evidence in Economic Decision Models
Implications for searching:
Types of evidence
Effectiveness Costs Resource use / activity Health states Utility values Indirect comparators Longer term outcomes ‘Other’ interventions Natural history Epidemiology
Defining search questions
Defining the evidence base
Transferable / ‘proxy’ data
Grouping questions
Consensus Working Group on the Use of Evidence in Economic Decision Models
Implications for searching:
Sources of evidence
‘Published papers’ Routine data Reference sources Local / clinical /
expert opinion Sponsor
submissions
Prioritising appropriate sources
Consensus Working Group on the Use of Evidence in Economic Decision Models
Implications for searching:
Uses of evidence
Parameter values Model structure Sensitivity analysis Validation /
consistency / calibration
Iterative nature of modelling process
Matching retrieval of evidence to use of evidence
Consensus Working Group on the Use of Evidence in Economic Decision Models
Other issues
Aim of the search– What evidence?– What for?– How much?
Methods of selecting evidence Alternative search methods Reporting standards
Consensus Working Group on the Use of Evidence in Economic Decision Models
Conclusions
The process of identifying evidence for models needs to take account of the following issues:
The range of types, sources and uses of evidence The process of identifying relevant evidence bases
and defining search questions The iterative nature of model development How much evidence is required Reporting standards
Consensus Working Group on the Use of Evidence in Economic Decision Models
Conclusions
Identifying evidence for models is different to identifying evidence for systematic reviews.