a methodological framework for developing the structure of public health economic models

A methodological framework for developing the structure of Public Health economic models Dr Hazel Squires, ScHARR, University of Sheffield

Prof. Jim Chilcott, Prof. Ron Akehurst,

Dr. Jennifer Burr, Prof. Mike Kelly

Introduction

• Why we need a conceptual modelling framework

specific to Economic Modelling in Public Health

(PHE)

• Methods for developing the PHE conceptual

modelling framework

• An outline of the PHE conceptual modelling

framework

• Conclusions

Background

• Healthcare agencies such as the National

Institute for Health and Care Excellence (NICE)

need to decide how to spend their money.

• Mathematical models are used to provide a

rational, coherent & transparent framework of

the cost-effectiveness of healthcare

interventions.

• Inappropriately simple models and lack of

justification may lead to poor validity and

credibility, resulting in suboptimal allocation of

resources.

Current methods for dealing with structural uncertainty

1) Retrospective: following model implementation

by expressing the impact of uncertainties upon

the model results.

• Scenario analysis

• Model averaging

2) Prospective: considering the process through

which decisions are made around the

conceptualisation, structuring and

implementation of the model.

• Model development practice in health economics is

extremely varied (Chilcott, 2010). Chilcott J, Tappenden P, Rawdin A, Johnson M, Kaltenthaler E, Paisley S et al. Avoiding and identifying errors in health

technology assessment models: qualitative study and methodological review. Health Technol Assess 2010; 14(25).

What is a Conceptual Modelling (CM) Framework?

• ‘A set of steps that help to guide modellers

through the development of a model structure,

from developing and describing an

understanding of the decision problem to the

abstraction and non-software specific

description of the quantitative model, using a

transparent approach which enables each stage

to be shared and questioned.’

Based upon Kaltenthaler E., Tappenden P., Paisley S. and Squires H. Identifying and reviewing evidence to inform the

conceptualisation and population of cost-effectiveness models, No. 14, 2011 and Robinson S. Conceptual

modelling for simulation Part I: definition and requirements. Journal of the Operational Research Society 2008;

59:278-290.

Key potential benefits of a CM Framework

• Aids the development of modelling objectives;

• Provides tools for communication with stakeholders;

• Guides model development and experimentation;

• Improves model validation (developing the right model);

• Improves model verification (developing the model right);

• Allows model reuse;

• Helps to characterise structural uncertainties and identify

primary research needs.

Economic evaluation of Public Health compared with clinical interventions

• A key objective of Public Health is reduction of health

inequities;

• PH interventions tend to:

- Be multi-component with complex causal chains;

- Operate within dynamically complex systems;

- Be dependent upon human behaviour;

- Be dependent upon the social determinants of health,

requiring consideration of non-health costs & outcomes;

• It is much less clear what a ‘good’ outcome is;

• The culture & politics of the system is important in

choosing & assessing interventions

Methods for developing PHE conceptual modelling framework

1) Literature review of key challenges in PHE modelling

(Squires et al., 2016)

2) Review of conceptual modelling frameworks in broader

literature.

3) Qualitative research to understand modellers’ experiences

with developing the structure of PH economic models &

their views about the benefits/ barriers of using a CM

framework.

4) Pilot of the draft CM framework within a case study of

diabetes screening & prevention.

5) Evaluation via qualitative analysis & theory-based

evaluation. Squires H, Chilcott J, Akehurst R, Burr J, Kelly M. A systematic literature review of the key challenges for developing the

structure of public health economic models. Int. J. of Public Health. Epub ahead of print.

http://www.ncbi.nlm.nih.gov/pubmed/26747470

The CM framework: Key principles

(1) A systems approach to Public Health modelling should

be taken (feedback loops & unintended consequences

are important);

(2) Developing a thorough documented understanding of

the problem is imperative prior to and alongside

developing and justifying the model structure;

(3) Strong communication with stakeholders and members

of the team throughout model development is essential;

(4) A systematic consideration of the determinants of health

is central to identifying all key impacts of the

interventions within Public Health economic modelling.

A systems approach

The CM framework: Key principles

(1) A systems approach to Public Health modelling should

be taken (feedback loops & unintended consequences

are important);

(2) Developing a thorough documented understanding of

the problem is imperative prior to and alongside

developing and justifying the model structure;

(3) Strong communication with stakeholders and members

of the team throughout model development is essential;

(4) A systematic consideration of the determinants of health

is central to identifying all key impacts of the

interventions within Public Health economic modelling.

The determinants of health

Dahlgren G., Whitehead M. Policies and strategies to promote social equity in health. 1991. Institute

for Future Studies, Stockholm.

CM framework overview Aligning the framework with

the decision making process

Identifying relevant

stakeholders

Understanding the problem

i) Developing a conceptual model of the problem

ii) Establishing resource pathways

Developing and justifying the model structure

i) Reviewing existing economic evaluations

ii) Choosing specific model interventions

iii) Determining the model boundary

iv) Determining the level of detail

v) Choosing the model type

vi) Developing a qualitative description of the quantitative model

Stakeholders

Stakeholders are identified based upon the

classification from Soft Systems Methodology

(SSM):

• Customers which might include patient

representatives and lay members;

• Actors which might include clinical experts and

epidemiologic experts for all relevant diseases

and methods experts;

• System owners which might include policy

experts (in addition to some of the people

identified as actors).




stakeholders











Example conceptual model of the problem Contraception project example:

Why is this a problem?

_ _

Maximise health Minimise costs to NHS & PSS Minimise costs to other sectors

_ _ _ _ _ _ _ _ _

Poor health outcomes of the mother

Low birth weight babies + Poor long term socioeconomic outcomes

+ + +

Miscarriage/ stillbirth Abortion Unwanted teenage birth

+ + +

What is the problem? +

Unintended teenage pregnancies STIs

_ + +

Poor contraceptive use by sexually active teenagers

(interaction between a male & female)

+ +

Disadvantaged background

Competing risks

Dependent on history & timing of STIs

Questions to help develop diagram: The disease & determinants of health

• Have any relevant disease natural histories been

captured?

• Are the following determinants of health important in

determining outcomes & in what way:

• Age, sex & other inherent characteristics of the population of

interest?

• Individual lifestyle factors?

• Social and community networks?

• Living & working conditions & access to essential goods &

services?

• General socioeconomic, cultural & environmental conditions?

Questions to help develop diagram: Dynamic complexity

• Are there any other (positive or negative)

consequences of each concept?

• Are there any other possible causal links

between the factors? (to establish whether there

are any feedback loops)

• Could there be any other factors which explain

both of these outcomes, for links which may not

be causal, but correlated?

• Are there interactions between different sets of

people?

• Is timing/ ordering of events important?

Questions to help develop diagram: Intervention outcomes

• What is a good outcome?

• What would happen in the absence of the

interventions versus as a result of the interventions

– would behaviour be prevented or delayed?

• What evidence exists to describe the outcomes of

the intervention/ comparator over time? Are

behavioural outcomes important?

• Are there any determinants of health reported by

the effectiveness studies which are not included

within the causal diagram? Can such a relationship

be described?

Questions to help develop diagram: Dynamic complexity of interventions

• Might a third party act to reduce or increase the

impact of interventions?

• Are there any substantial impacts of social and/or

community networks upon intervention

effectiveness? Will these impacts be captured over

the long term within the effectiveness evidence?

• Are there any substantial impacts of the

interventions upon other lifestyle factors?

• Might the interventions have other impacts not

already considered?

Sources of evidence

Starting with high yield sources

Individual stakeholder

assumptions & beliefs

Stakeholder

discussion

Modeller assumptions

& beliefs

Project scope

Literature

sources

Existing diagrams/

previous work

Conceptual model of the problem




stakeholders











Develop understanding of the

problem

Assess whether there is

an existing model which

could be employed

Identify strengths &

limitations of different

model structures

Identify strengths &

limitations of different

model types

Identify key variables which

generally affect model results

(incl. any not already

identified) & key variables

included within the causal

diagram which do not

Identify the sort

of data available

Identify factors with not many

causal links & assess whether

they would have a substantial

impact upon the difference

between outcomes of

interventions & comparators

Identify types of

outcomes reported

Identify long term

evidence & mechanisms

Describe effectiveness of

interventions (to help

choose which to model

& for parameterisation)

Model boundary Model detail Model type

Discuss potential model perspectives,

outcomes, interventions &

populations with stakeholders

Review existing health

economic models

Review effectiveness of

relevant interventions

Review evidence of

relationships between factors

Does the factor have many causal links?

Yes No

Is the factor likely to have a substantial

impact upon the difference between costs &

effects of the interventions? This may be

based upon (though not limited to):

(1) the review of economic evaluations;

(2) the description of resource pathways;

(3) clinical papers describing the causal links;

(4) existing models in similar areas which

describe the impact of the factor;

(5) methodological choices eg. discounting;

(6) expert advice.

Yes No

INCLUDE

Is the factor associated with the interventions,

populations & outcomes being modelled?

EXCLUDE

INCLUDE

EXCLUDE

Yes No

Yes

Is the impact of the factor predominantly

captured by other included factors?

Yes

EXCLUDE

No

Would stakeholders prefer to

include the factor for model

credibility AND is it relatively easy

to incorporate in terms of

modelling skill & data availability?

INCLUDE

No

Will the model only assess one intervention AND

is the intervention likely to be cost-effective AND

does the factor only further increase benefits

Yes

No

Determining

the model

boundary

A systems approach

Determining level of detail

Key types of model assumptions/ simplifications:

1. The relationship between the included factors over time;

2. The extrapolation of study outcomes;

3. The level of detail used to describe each included factor;

4. How interventions will be implemented in practice.

Questions to help the modeller for each of these are

proposed within the framework.

Choosing model type (1)

A B C D

Cohort/ aggregate level/ counts Individual level

Expected value, continuous state, deterministic

Markovian, discrete state, stochastic

Markovian, discrete state

Non-Markovian, discrete state

1 No interaction

Untimed Decision tree rollback

Simulation decision tree

Individual sampling model: Simulated patient-level decision tree

2 Timed Markov model (deterministic)

Simulation Markov model

Individual sampling model: Simulated patient-level Markov model

3 Interaction between entity and environment

Discrete time

System dynamics (finite difference equations)

Discrete time Markov chain model

Discrete-time individual event history model

Discrete individual simulation

4 Continuous time

System dynamics (ordinary differential equations)

Continuous time Markov chain model

Continuous time individual event history model

Discrete event simulation

5 Strong interactions between entities. Spatial aspects important.

X X X Agent-based simulation

Revised version of taxonomy by Brennan A, Chick SE, Davies R. A taxonomy of model structures

for economic evaluation of health technologies. Health Econ 2006; 15(12):1295-1310.

Choosing

model type

(2)

Determine the most appropriate model type for the characteristics of the problem.

Is this feasible within the time and resource constraints of the decision making process given:

(i) the data available?

AND

(ii) the accessibility of any existing relevant good quality economic evaluations for use as

a starting point?

AND

(iii) the expertise of the modeller?

Are you intending to use the

model again for other

projects?

Can you answer the decision makers’ question

with a few provisos and uncertainties with a

simpler model type?

Yes No

Explore with the

decision maker the

most useful purpose of

the modelling given the

project constraints

Develop the simpler model

type, documenting the

provisos, uncertainties &

implications of the

simplifications

No Yes

Do you think a

simpler model type

would lead to the

same conclusions?

Develop

the

model

Yes No

Develop the more

complex model

Develop the simpler model,

documenting the provisos,

uncertainties & implications

of the simplifications

Yes No

Conclusions

• Very little research has been undertaken

around conceptual modelling in health

economic evaluation.

• We have developed a conceptual modelling

framework for use within Public Health

economic models (paper under review).

• This has yet to be tested on a range of case

studies.

• If anybody wants to know more about the

framework or potentially use it, let me know!

Email: [email protected]

PhD thesis: http://etheses.whiterose.ac.uk/5316/

a methodological framework for developing the structure of public health economic models

Health & Medicine