A methodological approach for assessing the impact of HTAEleanor Grieve, Research AssociateHealth Economics & Health Technology Assessment, University of Glasgow

Section 1•‘Better Decisions’ to ‘Better Health’ & iDSI, •This research - can synthesising some of the vast theoretical literature from social sciences with economic modelling help improve our translation of ‘better decisions’ into practice and impact?Section 2•Case studiesSection 3•Theory driven / realist approach to impact assessment

Outline

• The ultimate value of HTA in a health system depends on its contribution to improved health status or increased efficiency rather than to increased knowledge. In this respect, HTA does not differ much from other health technologies and must be subject to the same rigorous standards of evaluation. Source: Garrido et al 2008.

• Raftery and Powell (2009) argue that the [HTA] programme needs to demonstrate that it is cost-effective through its impact on health service resources and wider public health.

Why evaluate HTA

To what extent dowe evaluate HTA?

The literature on assessment of HTA influence is still quite limited and there is little on longer term effects on clinical practice and health outcomes. INAHTA, 2014. Evidence on the Influence of HTA

“…a review of the existing literature on HTA reveals a startling lack of depth, particularly on the impact HTA has had on health-care budgets, efficiency, and on societal health outcomes…whereas the previous 10 years have been well-spent on building the HTA/EBM infrastructure and evidence base, the next 10 should focus on the outcomes.” Reference: Straus SE (2004) in Value in Health Special Issue, Health Technology Assessment: Lessons Learnt from Around the World – An Overview [Volume 12 Issue s2, Pages S1 - S5 (June 2009)]

To what extent dowe evaluate HTA?

NIHR HTA programme provides NHS with best evidence, NICE provides best guidance – this works…What is not nearly so well understood is how new evidence or guidance impacts on patient care…and what factors influence… Source: Turner et al, 2015. Impact of NIHR HTA Programme funded research on NICE clinical guidelines

“..the Department of Health and NHS England do not have the data needed to assess the impact of the (Cancer Drugs) Fund on patient outcomes, such as extending patients’ lives, or to demonstrate whether this is a good use of taxpayers’ money". Cancer Drug Fund, Public Accounts Committee 25 Jan 2016.

A growing network of partners sharing the same principles

We are committed to partnerships with academic, public sector and international development

groups from across the world, and supporting regional hubs for priority-setting.

A new development, iDSI, uniquely adds value for policy makers around the world

7

Three major funders come together

Better Decisions for Better Health

Practical support and knowledge

products

Evidence-informed, transparent, independent,

consultative decision making processes

More efficient and equitable resource allocation decisions

with trade-offs made explicit

Demand-driven supportPolicy-informed knowledge products

Accountable institutions and processes protect politicians from vested interests and

help defend tough choices

9

There is a complex translation process between “better decisions” and “better health” depending on the link between decisions and budgets, budgets and payments/transfers, transfers and delivery system, readiness and effectiveness of delivery and implementation and also the validity and reliability of the original data informing the analysis.

iDSI

Better decisions are better in terms of both how they are made and what they decide.

iDSI

Changes in behaviour and practice can be subject to resistance at structural, cultural, professional and resource based levels.

What is already known

•How to harness the use of evidence in decision-making: well documented (Lavis et al, 2005. Innvaer et al, 2002. M Drummond…)

•The problem of implementing evidence-based practice: well documented (Garrido et al, 2008. Lavis et al, 2008. Sheldon et al, 2004, Audit commission 2005, Cullum et al. M Drummond…)

•How to develop effective methods of changing this ie solutions – less well documented and focus of this research

What is already known

•Implementation variable – despite being mandatory NHS•Depends on technology, location •Subject to how funding is tied•Redistribution of existing resources within health system• Increased or unfunded costs•Need professional engagement•Lack of data on uptake•Lack of dissemination and implementation strategies•Factors related to the process and quality of reports•Broader external factors that influence opportunities for implementation

What is already known

10 studies analysed provided a potential net-benefit of ￡ 3.0 billion based on a value of ￡ 20,000 per QALY, and ￡ 5.0bn based on a value of ￡ 30,000 per QALY. The cost of the HTA Programme since 1993 was ￡ 317m, with the estimated overall cost of the HTA Programme ￡ 367m. We conclude that 12 per cent of the calculated potential net benefit would cover the total cost of the HTA Programme from 1993 to 2012.Assumption: HTA findings are fully implemented in the NHS

Tensions -CEA, priority settingand implementation

 CEA assumes a single constraint – the budget constraint. In reality, numerous. Constraints that may act as barriers to implementation of cost-effectiveness recommendations: design of the health system; costs of implementing change; system interactions between interventions; uncertainty in estimates of costs and benefits; weak governance; and political constraints. + demand side responses by patients. Source: Hauck et al, 2016

WHO (2011), HTA as a priority setting tool is separate from implementation and management…. Source WHO. 2011. ‘Health technology assessment of medical devices.’ WHO medical device technical series

Tensions: CEA, priority settingand implementation

What is already known

Research agenda on implementation

1.What are the barriers to implementation? In particular, whether some barriers are more important than others in certain situations.2.What are the mechanisms to encourage implementation of HTA findings?3.Are certain mechanisms more suitable in certain circumstances or for particular types of technologies?4.What is the relative cost-effectiveness of alternative implementation mechanisms?5.What is the balance of benefit and harm between aggressive and conservative implementation strategies?Source: Drummond M, Wetherley H. Implementing the findings of HTA. International Journal of Technology Assessment in Health Care. 16:1, 2000.

What might this research add

•Generate insights into what works when decisions translate into outcomes, and why they have not worked when they don’t - and how we explain these effects.

•First realist approach to understand better the mechanisms by which we can improve implementation of HTA findings.

•Provide an explanatory theoretical underpinning as a means to improve our response and optimise the use of HTA (as the institutionalisation of HTA moves forward in many countries moving towards UHC).

Case studies

Case studies With acknowledgment to

MCH, Myanmar

Maternal and Child Voucher Scheme, Myanmar

MCHVS, Myanmar

Timeline

May 2010 – March 2011: Ex-ante evaluation (HITAP/ MoH, WHO)

May 2013: MCHVS pilot programme in Yedarshey Township

Jan 2014: Mid term evaluation (HITAP / MoH, WHO)

2015: Annual review (independent)

2015: Ex-post HTA + impact (University of Glasgow)

MCHVS, Myanmar

Ex-anteGBP £

Ex-post GBP £

Incremental cost 96 94Incremental life year saved 0.2513 0.2782Incremental DALY averted 0.2531 0.2789ICER per DALY averted 384 336

GDP Myanmar (2010) 414Threshold used = 1 GDP

NHBs = ΔH - ΔC/λ

Net health benefits (per person) 0.02 0.05Scaled to pilot (11532 pregnant women) 231 NHBs 576 NHBs

Current (actual) value of implementation

• With Ed = 0.2, full implementation = 76% ANC and 87% SBA coverage

• Mid term review of pilot reported 77% coverage SBA• Current value of the HTA – total patient population

eligible for treatment (n) and a proportion of these patients (p) are already receiving the intervention, then the current value is defined as

N * p * NHB = 11532 * 77% (of 87%) * 0.05 = 510 NHBs

Value of implementation

Net Health Benefit: full implementation = 576 NHBs

Net Health Benefit:Current implementation = 510 NHBs

Additional NHB with full implementation

The value of perfect implementation = 66 NHBs

Incremental net benefit of implementation initiative

An implementation initiative is worthwhile if its benefit in terms of increased utilisation of the intervention (the expected value of actual implementation ) is greater than its cost.

n * α * NHB – n * p * NHB - Implementation Cost / λ

.

HTA: Economic modelling – population

health impact – full implementation

Value of perfect implementation

Theory based approaches

Current /actual uptake

Realised ImpactPotential Impact

Conceptual Framework for impact assessment

HPV vaccination vscreening, Thailand

Potential net population health benefit of a new intervention

HPV vaccination vscreening, Thailand

Strategy Size of eligible

population

NHB Coverage Population NHBs

Existing

screening

strategy

14m 28.01 20% 78m

Recommended

screening

strategy

16m 28.06 80% 359m

Vaccination 19m 28.02 100% 532m

Health technology assessment •Quantifies potential population health benefit of a new intervention

Health system decision•Will the new intervention be recommended/ reimbursed/ covered?

Clinical practice•To what extent does clinical practice change to use/not use intervention?

The impact of HTA on population health Potential plus realised population health benefit

HPV vaccination vscreening, Thailand

“This suggests the need for policy managers to have a better understanding of the processes of policy development, including insight into the roles of stakeholders, their interests, and interactions with the health system context. However, analysis of health policy is rarely recognized and applied in developing countries’ academic institutes and health administrative authorities.”

Source: Gilson L, Raphaely N: The terrain of health policy analysis in low and middle income countries: a review of published literature 1994-2007. Health Policy Plan 2008, 23(5):294-307.

HPV vaccination vscreening, Thailand

Treating process of HTA as a complex intervention.

Synthesis of: 1) economic methods as a quantitative measure of impact 2) theory-driven approaches for explanatory power

Theory driven approach to impact assessment

Complex elementsof HTA

Scheme 7 Determinants of effectiveness – program. Reference: Hailey, D. Elements of Effectiveness for Health Technology Assessment Programs. Alberta Heritage Foundation for Medical Research. 2003

• ‘hta’ = the intervention/health technology

• ‘HTA’ = the process of systematically assessing and appraising evidence-based guidance of a technology

• both potentially complex

• 2 levels and their interaction

Complex elements of HTA

• Complex interventions require social science methods to make sense of them

• Systems thinking used to understand system behaviours and to identify systems principles such as open v closed systems..

• Logic models - illustrate how inputs, activities and outcomes might be logically expected to join together.

• Theories of change - identify and describe the complex relationships between the programme components

• Realist evaluation (Context – Mechanisms – Outcome Configurations) - what works, for whom, under what circumstances and why.

Approaches to aid theory development

Critical/organisational cultural theory - interplay and tensions between knowledge, power and social controlImplementation science: getting evidence into practice, dissemination, reaching the right people.Health policy and systems research - processes of policy development, including insight into the roles of stakeholders, their interests, and interactions with the health system context. Theories of behaviour change – cognitive models

Formal theories

Reference: Realist review – a new method of systematic review designed for complex policy interventions J Health Serv Res Policy July 15, 2005 10: 21-34

Realist approach

Outcome measures

Reference: Garrido et al. HTA and policy making in Europe. 2008

CMOCs

1. Realist synthesis - start with an ‘initial’ programme theory of how HTA works to influence policy and practice (in what circumstances, why…). Use assumptions in the ‘unpacked’ iDSI ToC.

2. Produce a number of CMOCs based on previous literature3. Assess to what extent any pre-existing ‘theory of change’ fits

with these CMOCs to produce a refined programme theory which can then be tested in the subsequent case studies.

4. The sampling framework will be driven by this review and itad’s baseline country data.

Data collection

Fieldwork to further test the programme theory •Empirical work to refine this theory, involving the collection of primary data. •Purpose sampling will provide primary and secondary sources of information for testing the initial programme theory. •Selection of studies (x2) would be decided in consultation with iDSI.

Data collection

• Country level: M&E data - logs, self assessment, deep dive evaluations (itad)

• International level: knowledge products, global donors (itad)• Network level: social, health, connectivity (itad)• Interviews will be undertaken at macro (policy-makers), meso

(institutional) and micro (health professionals, patients) levels.

• Informed by those involved in the wider health and regulatory systems.

Key stakeholders

• Data analysis will test the programme theory informed by the formal theories and literature identified.

• Analysis will be undertaken using realist evaluation principles of extracting CMOC variables at play, and iterative, participative and collaborative approaches to interpretation.

• Culminating in the production of explanatory programme theory that considers individual, interpersonal, and institutional/systems-level components.

Data analysis

• HTA should seek to harness (yet to be identified!) mechanisms via components that function to improve the implementation of findings - leading to impact on health

• Aim to produce ‘policy-friendly outputs’/guidance in terms of ‘what works, where, for whom, and how’

• Improve monitoring/data on the adoption of guidance/HTA findings

• Promote greater synthesis and integration of economic methods (value of implementation) with social science theories and approaches

How might this impact on practice

Summary of conceptualframework

iDSI and its funders (BMGF, DFID, Rockefeller Foundation)Thank you to NI for providing slides:•Kalipso Chalkidou, MD, PhD, Director •Ryan Li, PhD, AdvisorScharr – ‘Real world evaluation’ short course•Colleagues at Glasgow and York for input-Dr Hannah Hesselgreaves, Prof Olivia Wu, Prof Andy Briggs, Minnie Parmiter-Paul Revill, Prof Mark Sculpher

Acknowledgements

• Innvaer S et al. (2002). Health policy-makers’ perceptions of their use of evidence: a systematic review. Journal of Health Services Research & Policy, 7:239–244.

• Lavis JN et al. (2005). Towards systematic reviews that inform health care management and policy-making. Journal of Health Services Research & Policy, 10(Suppl 1):35–48.

• Cullum N et al. (2004). The evaluation of the dissemination, implementation and impact of NICE guidance.

• Lavis et al, 2008. Evidence informed health policy – synthesis of findings from a multi-method study of organisations that support the use of research evidence. Implementation Science 2008, 3:53.

• WHO. 2011. ‘Health technology assessment of medical devices.’ WHO medical device technical series.

References

References

• Gerdhardus, A and Dintsios C. The impact of HTA reports on health policy: a systematic review 2005 GMS Health Technology Assessment GMS Health Technol Assess. 2005; 1: Doc02 http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3011311/

• Guthrie, Susan, Marco Hafner, Teresa Bienkowska-Gibbs and Steven Wooding. Returns on research funded under the NIHR Health Technology Assessment (HTA) Programme: Economic analysis and case studies. Santa Monica, CA: RAND Corporation, 2015.

• How can the impact of health technology assessments be enhanced? Corinna Sorenson, Michael Drummond, Finn Børlum Kristensen and Reinhard Busse WHO Regional Office for Europe and European Observatory on Health Systems and Policies, 2008

• Gilson L, Raphaely N: The terrain of health policy analysis in low and middle income countries: a review of published literature 1994-2007. Health Policy Plan 2008, 23(5):294-307

• Fenwick E, Claxton K and Sculpher M. The value of implementation and the value of information: combined and uneven development. Medical Decision Making, 2008, 28, 21-32