developing appropriate evaluation pathways for complex … · 2020-03-06 · ideal framework* a 5...
TRANSCRIPT
Developing appropriate evaluation pathways for complex invasive
therapies: A surgeon’s perspective
Peter McCulloch
Professor of Surgical Science & Practice
Oxford University
Chair, IDEAL Collaboration
What are medical devices?
• Term covers a huge range of artefacts from dressings and condoms to diagnostic kits to operating robots
• Focus of regulation rightly on more invasive technology with greater potential for harm
• Focus of this talk on FDA Class 2b and 3 therapeutic devices
All Medical Devices
More invasive devices
FDA Class 2b and 3 therapeutic devices
Who evaluates these devices and why?
1. Government regulatory bodies tasked with evaluating new therapies (eg FDA, MHRA) require and supervise evaluation mainly for SAFETY and TECHNICAL EFFECTIVENESS (but also for clinical effectiveness)
2. Purchasers of healthcare for large populations (e.g. CDC, Blue Cross, NHS Commissioners) evaluate mainly for RELATIVE EFFECTIVENESS and COST-EFFECTIVENESS (but also for safety) – resulting in the “HTA industry”
3. Where devices represent a new treatment concept, clinical & academic researchers evaluate a range of outcomes, of which EFFICACY and RELATIVE EFFECTIVENESS are usually most important (but also safety and cost-effectiveness)
Why is evaluation of these devices a problem? Ciani O et al, International Journal of Technology Assessment in Health Care, 31:3 (2015), 1-12. Tarricone et al Expert Rev. Pharmacoecon. Outcomes Res. Early online, 1–12 (2014)
• Relatively poor quality of evidence presented by developers
• Repeated modifications of devices
• Problems in generalizing the evidence obtained in a specific setting – very variable results
• The “learning curve” effect of outcomes improving with experience
• Difficulties in defining the correct scope of the evaluation • Technical performance • Reliability • Safety • Ease of use • Market demand, • Likely cost-effectiveness, profitability • Rare and late problems
Why is it so?
• Regulatory answer – because there is no mandatory requirement for specific evidence types or standards
• Environmental Answer – small companies with limited resources and knowledge
• Ontological Answer – the nature of devices makes evaluation more difficult
R
E
O
Developing Evidence for Surgical Therapy
PHARMA PARADIGM
• Theory
• Lab demonstration
• (animal studies)
• First-in-man study
• Toxicity study (Phase I)
• Efficacy Study (Phase II)
• RCT (Phase III)
• Post-Marketing Surveillance (Phase IV)
SURGICAL PARADIGM
• Idea
• (animal/bench trial)
• First-in-man (CS)
• Outcome experience (CS)
• Confirmation/dispute from others (more CS)
• RCT??
2016: Pharma versus Surgery
ANALYSIS: BARRIERS TO SURGICAL RCTs
REAL
• Need for refinement of technique
• Need for definition of technique
• Delivery Variation (learning curves, quality control)
• “equipoise” Clinician • (surgical personality?)
• “equipoise” Patient • Tough choices
• No way back
NOT SO REAL
• Rare conditions
• Heterogeneous patients
• Blinding
• Time and Resources
With enough information, we could surmount all of these
Shared problems of evaluation: Devices & Surgery
• Weak or non-existent legal and regulatory evidence requirements
• Small underfunded groups with weak scientific skills & massive conflict of interest evaluating themselves
• Strong preferences based on heuristic biases*
• Need to modify initial design*
• Need for learning curves*
• Variability in delivery*
Q. “What should an evaluation process for X look like?” A. “What is X?”
“The first step in analysis is description”
IDEAL Framework* A 5 stage description of the journey of surgical innovation
• Stage 1 - IDEA
• Stage 2a - DEVELOPMENT
• Stage 2b - EXPLORATION
• Stage 3 - ASSESSMENT
• Stage 4 – LONG TERM MONITORING Barkun JS, Aronson JK, Feldman LS, Maddern GJ, Strasberg SM, McCulloch P; Balliol Collaboration: Evaluation and stages of surgical innovations. Lancet 2009 Sep 26;374(9695):1089-96. Ergina PL, Cook JA, Blazeby JM, Boutron I, Clavien PA, McCulloch P, Balliol Collaboration, Challenges in evaluating surgical innovation. Lancet. 2009 Sep 26;374(9695):1097-104. McCulloch, P., Altman DG, Campbell WB, Flum DR, Glasziou P, Marshall JC, Nicholl J., No surgical innovation without evaluation: the IDEAL recommendations. Lancet, 2009. 374(9695): p. 1105-12.
Key Questions at each IDEAL Stage
Each stage is defined by one key issue:
STAGE 1: What is the new treatment concept?
STAGE 2a: Has it been optimised to a state of stability?
STAGE 2b: Can we reach agreement that the prerequistes for a randomised trial versus current standard of care have been satisfied?
STAGE 3: Is the new treatment better than current practice?
STAGE 4: Are there any surprises?
Implications of the Key Questions
• Studies at each stage should be designed to answer the key question for that stage
• IDEAL Recommendations describe study formats designed to do this
IDEAL: An integrated evaluation pathway
REGISTRATION OF 1st in MAN
PROSPECTIVE DEVELOPMENT STUDY
PROSPECTIVE EXPLORATION STUDY
RCT
REGISTRY
The IDEAL Recommendations • Idea (First in Man) 1.
• Development 2a.
• Exploration 2b.
• Assessment 3
• Long Term Study 4
• Complete technical description • Explanation of patient selection • Registration of report
• Prospective Cohort Study (PDS) • Transparent Consecutive Reporting of Cases • Explanation of Changes in Technique, Indication
• Prospective collaborative cohort study (PES) • Evaluation of learning curves • Definition of QC parameters • Estimation of power calculations • Early joint analysis leading to RCT • Feasibility/Pilot RCT • Definitive RCT • Removal of investigator bias from recruitment
• Registry to detect late/rare events • Monitoring of indication and performance
creep
Legal Firewall for Stage 1 database
• UK Access to Medical Treatments (Innovation) Act 2016
• Requires innovative treatments to be recorded in database
• Offers opportunity to protect innovators from legal challenge on the basis of what is recorded in the database
Stage 2a: Development
• Prospective Development Studies:
- Detailed description of selection criteria
- Detailed technical description
- Prospective account of ALL cases consecutively
- Clear STANDARDISED definitions of outcomes reported
- Description of ALL modifications and when made in series
- Registration of PROTOCOL before study starts
- Use of STATISTICAL PROCESS CONTROL methods to evaluate progress
PDS Example: Development of Robotic Oesophagectomy
Why do Development studies?
• Techniques in DEVELOPMENT stage are not yet stable
• Reporting changes and their reasons allows others to learn faster and not repeat mistakes
• Ethically superior to current practice
• Should increase speed of development process
Stage 2b: Exploration
Prospective Exploration Study (collaborative non-randomised
prospective study)
• To evaluate technique prospectively and co-operatively
• To agree definition of the procedure, quality standards & patient selection
• To accumulate data for power calculations
• To evaluate learning curves
• To evaluate preferences and values amongst patients and clinicians
• To achieve consensus on future trial question and comparator group
• To develop a multi-centre randomised trial OR demonstrate this is not required
PES example: HIFU for fibroids • Previous papers track progress of
technology through stages (0), 1 and 2a
• Large prospective cohort comparing surgery with HIFU (patient choice)
• Defined parameters of treatment
• Measured learning curves
• Tight quality control
• Used results to plan RCT protocol
20 centre study: checking learning curves
• LC-CUSUM technique shows when improvement stops
• All operators showed satisfactory learning BEFORE trial entry
• Learning curve checked during trial as well
HIFU Exploration study results
Significant Complications Patients treated
0
5
10
15
% MajorComplicatio…
Hospital Stay (days) Return to Work (days)
HIFU 3.63 4.07
Myomectomy 8.96 24.01
Hysterectomy 10.53 29.49
Quality of Life (6 & 12 months) HIFU clearly superior but margin small
Why do Exploration (2b) studies?
• Organising surgical RCTs is difficult: it requires TRUST and UNDERSTANDING between surgeons
• 2b studies improve trust and understanding by improving joint ownership and belief in data
• 2b studies allow questions which worry surgeons to be answered, e.g. • Which variations of the procedure are acceptable? • Are some colleagues still learning? • Which patients are suitable? What is the comparator? • How many patients will we need? • What outcome questions are important to patients – and surgeons?
• In this way 2b studies improve FEASIBILITY of RCTs
Is IDEAL applicable to device evaluation? A Delphi Process to develop IDEAL-D *
• Need Stage 0 with minimum declared dataset • Recommendations ??
• Need flexible approach to mixes of Development (2a) and Exploration (2b) stages • 2a often redundant • 2b only partly relevant
• Controversy over RCTs for “me too” and successor devices
• Need Registries from an early stage, developing and changing with needs • Potential of UDI and Big Data • TWICS approach
IDEAL-D: a rational framework for evaluating and regulating the use of medical devices. Sedrakyan A, Campbell B, Merino JG, Kuntz R, Hirst A, McCulloch P. BMJ. 2016 Jun 9;353:i2372. doi: 10.1136/bmj.i2372. *
Outline of IDEAL-D
• Greater emphasis on pre-clinical testing => Stage 0, Less Stage 2a
• UDI and TPLC concepts => Registries from Stage 1
• Number of relevant issues in Stage 2b reduced
• Potential fusion of Stages 2a and 2b
iBRA Network
• Focus on reconstruction after breast cancer treatment (including implants and devices)
• Plans to collaborate on large prospective cohort studies (2b studies)
• Has adopted the IDEAL recommendations as standard operating procedure
• Aims to use these to define where RCTs are feasible and useful
• One study completed, two in pipeline
Will IDEAL help? What is the potential value?
• Having accepted, logical, standardised study formats for evidence provision would greatly enhance transparency and reduce redundancy
• The provision of standards for early clinical evidence (Development and Exploration studies) would facilitate appropriate timely discussions with purchasers and regulators
• The study formats involved are cheaper and faster than RCTs and would represent a much lower barrier to (provisional) market entry
• Much current evaluation activity by purchasers and regulators, looking at reviews of deeply flawed evidence, could be abandoned.
What did we just say?
• Establish standard study formats
• Have an integrated pathway
• Link graded access to progress
• Rely more on this prospective evidence than on reviews
Radical step or logical progression?
• UK - Commissioning through Evaluation • US – Coverage with Evidence
Development • Germany – new §137 Social Code
ALL LINK COVERAGE TO SPECIFIC EVIDENCE DEVELOPMENT UK AND GERMAN MODELS HAVE IDEAL D AND E-LIKE STAGES ALREADY
• WHY NOT SPECIFY AND REQUIRE
FORMATS? • WHY NOT LINK COVERAGE TO EVIDENCE
PATHWAY PROGRESS?
Current IDEAL Adoption in Rx evaluation
• NHS England specialist commissioning – Commissioning with evidence development programme – pilot studies under discussion
• Research Collaboratives e.g.iBRA
• NHS England NIHR research HTA Programme calls for IDEAL 2b (Exploration) studies for: • Incontinence surgery devices
• Fenestrated aneurysm stenting
• Pilonidal sinus treatments
• Surgery for Crohns’ disease
IDEAL as the core of a broader evaluation framework: • EpiCOR programme in Canada
(Martin J et al) using IDEAL to make purchasing decisions
• Radboud University Medical Centre, Nijmigen, Netherlands (Rovers et al) – using IDEAL to advise on Health Technology Assessment reports.
• EXCITE International – an new international body offering end-end expert 3rd party evaluation for therapeutic devices
• All three seek to add “pre-IDEAL” evaluations to provide a spectrum of evidence to meet the demands of both regulators and purchasers • User forums – qualitative work on
value • HE modelling to predict cost-
effectiveness • Ergonomic evaluation to optimise
usability
Integrating evidence for regulation & purchasing
• Goals and evidence types presented likely to differ – but overlap +++
• All new entrants need to confront both issues – eliminating redundancy would reduce costs and delays, simplify process
• Both regulators and payers face the same types of weakness in the evidence they are given
• Common language and understanding of the evidence development process would be hugely helpful for inter-agency work
Using IDEAL in Regulation and Purchasing
• Where is this procedure on the evaluation pathway?
• What questions need to be answered NOW?
• What studies are appropriate?
• What will happen if we don’t insist on proper evaluation?
• Regulator or payer takes responsibility for decision on RCT
Potential Models for using IDEAL: “Commissioning” & Regulation
COMMISSIONING
• Funding only in context of appropriate (IDEAL 2a or 2b) study
• Mandatory data review
• Decision: Clearance or Coverage within RCT only – or decision one is not required.
• Discussion on funding & curation of Registry activities
REGULATION
• Develop and require common minimum Stage 0 dataset
• Require IDEAL stage 2 studies for Class 2b and 3 devices
• Monitor performance of notified bodies in delivering these
• Insist on greater transparency with public on realities of safety assessement.
Contrast with Claxton framework Informing a decision framework for when NICE should recommend the use of health technologies only in the context of an appropriately designed programme of evidence development K Claxton, Palmer, L Longworth, L Bojke,
• Claxton assumes current NICE decision framework will persist and defines criteria for decisions to impose restrictions on use related to further study
• Also assumes that current binary approach to market approval will persist, hampering good data collection post-market.
1 Is it cost-effective?
2 Are there significant irrecoverable costs?
3 Does more research seem worthwhile?
4 Is the research possible with approval?
5 Will other sources of uncertainty resolve over time?
6 Are the benefits of research greater than the costs?
7 Are the benefits of approval greater than the costs?
• IDEAL assumes the unification of regulatory and purchasing evidence requirements
• Also assumes a system in which marketing is strictly limited to cases entered into a study of the appropriate type for the IDEAL stage i.e graduated approval dependent on evidence collection.
Thorny Issues for an IDEAL-based system • First-in-man Registry: legal firewall needed
• Commissioning research: who pays?
• Oversight & curation of Registries: responsibilities • Defining dataset with clinical community • Ensuring quality control of data • Supervising data protection • Ensuring registry fulfils intended functions
• “Me too” devices • Should we follow Pharma model? • How to deal with “minor” changes – beware the 510K route
• Boats we missed • Accepted treatments with weak evidence: Time-limited IDEAL 2b study?
• Rare & expensive treatments: research only? • Proton beam therapy e.g. • How prescriptive should we be? • Will we create “orphan” treatments?
The IDEAL proposition for devices
• Establish a set of standards for clinical studies related to the stage of development of the technology
• Establish an integrated pathway for devices to pass from one stage to the next through evidence development
• Link market access to stage of progression – in principle restrict access to patients in ongoing studies
• Use IDEAL-D Stage 2 evidence for critical decision: RCT required or not?
• Establish registries from an early stage
• If you’re worried it might slow innovation – pilot it properly.