1.3 Modeling process and outcomes

Walter Sermeus, RN, PhD

Catholic University Leuven

Belgium

Witten, Fri 02.07.10Session 1: 11:00-12:30Session 2: 13:30-15:00

The European Academy of Nursing Science 2

Background: MRC CI-framework

Modeling

The European Academy of Nursing Science 3

What does the MRC-CI framework say on Modelling process and outcomes?

• Modelling a complex intervention prior to a full scale evaluation can provide important information about the design of both the intervention and the evaluation.

• One useful approach to modelling is to undertake a pre-trial economic evaluation. This may identify weaknesses and lead to refinements, or it may show that a full-scale evaluation is unwarranted, for example because the effects are so small that a trial would have to infeasibly large.

• Formal frameworks for developing and testing complex interventions, such as MOST or RE_AIM may be a good source of ideas, and the National Institute for Health and Clinical Excellence has produced detailed guidance on the development and evaluation of behaviour change

The European Academy of Nursing Science 4

What will we do in this session ?

• What & why

• Models – Tools

• Economic evaluation

• Examples

• Excercises

The European Academy of Nursing Science 5

What kind of models ?

• No scale models

• Rather causal models:– A causal model is an abstract model that

uses cause and effect logic to describe the behaviour of a system.

– Focus on understanding, relationships, which effects might be expected

– Using theories, from various disciplines

Why models ?

• Refining conceptual models– influences, components, relations,

consequences

• Generating (tentative) estimates of effect size

• Identifying barriers

• Optimising combinations of components of the intervention

The European Academy of Nursing Science 6

(Source: Campbell et.al., BMJ, 2007)

Relation between context, problem definition, intervention and evaluation

The European Academy of Nursing Science 7

(Source: Campbell et.al., BMJ, 2007)

MODELS: PRIME

• PRocess Modelling in ImpleMEntation research– Identifying active ingredients in professional behaviour

change– Process modelling: understanding of factors underlying

clinical practice, in order to identify what sorts of processes should be targeted in implementation interventions

– 4 levels: individuals, teams, organisations, systems– Different kind of theories

The European Academy of Nursing Science 8

(Source: Walker AE et.al., BMC HSR, 2003)

Example of PRIME Theories - individual level

The European Academy of Nursing Science 9

Normalization process model

• Theoretical framework for understanding complex interventions

• Normalization vs adoption / rejection• 4 dimensions:

– Interactional workability: effect on interactions between people and practices

– Relational integration: relation to existing knowledge and relationships

– Skill-set workability: effect on current division of labour– Contextual integration: relation to the organisation

The European Academy of Nursing Science 10

May C, BMC Health Services Research, 2006

Example of NPM to telehealth services

The European Academy of Nursing Science 11May C, BMC Health Services Research, 2006

Tools: RE-AIM

Dimension Level Description

Reach Individual Proportion of target population that participated in the intervention

Efficacy / effectiveness

Individual Success rate; positive / negative outcomes

Adoption Organization Proportion of settings that will adopt the intervention

Implementation Organization Extent to which intervention is implemented as intented in real world

Maintenance I&O Extent to which a program is sustained over time

The European Academy of Nursing Science 12

Source: Glasgow R. et.al., 1999 (www.re-aim.org)

The goal of RE-AIM is to encourage program planners, evaluators,funders, and policy-makers to pay more attention to essential program elements including external validity that can improve the sustainable adoption and implementation of effective, generalizable, evidence-based interventions.

TOOLS : MOST

• Multiphase Optimization Strategy– Screening phase:

• Components that are candidates for inclusion in an intervention (active components)

• Based on significant effect, effect size, cost,…

– Refining phase:• Fine tuning e.g. optimal level

– Confirming phase:• Evaluating the intervention as a package

The European Academy of Nursing Science 13

(Source: Collins et.al. Am J Prev Med, 2007)

Economic evaluation

Are Costs and Outcome Examined? outcomes costs Costs &

outcomes Comparison

of two alternatives?

No Outcome description

Cost description

Cost-outcome description

Yes Effectiveness evaluation

Cost-analysis

Full economic evaluation

Source: R. Taylor, Masterclass Complex Interventions, Exeter 2010

Incremental Cost Effectiveness Ratio (ICER)

cost [complex intervention – usual care]

outcome [complex intervention – usual care]

ICER =

Quality adjusted life years (QALYs)

Cost utility analysis (CUA)

Utility

Life expectancy

10

0.7

1

A

B

Intervention A = 5 QALYs

perfect health

death 0

Intervention B = 7 QALYs

QALY difference = +2 QALYs

0.5

EQ-5D(EuroQoL)

(www.euroqol.org)

EQ-5D scoring guide

Example of Modelling exercise

• ProActive causal model (Hardeman et.al., Health Education Research, 2005)

• Aim: increase physical activity among individuals at risk of Type 2 diabetes

• The model provides:– Rational guide for appropriate measures– Intervention points– Intervention techniques

The European Academy of Nursing Science 19

Generic causal model

• Epidemiology– Defining health outcomes and precise objective measures,

target group, likely impact of achievable behaviour change on physiological and biochemical variables

• Psychology– Theory-based determinants, intervention points, techniques

to support behaviour, measures of change in behavioural determinants

The European Academy of Nursing Science 20

Methods used

The European Academy of Nursing Science 21

Theory of Planned Behaviour (Ajzen 1991)

belie

fs

Results : ProActive causal model

The European Academy of Nursing Science 23

… continued

The European Academy of Nursing Science 24

Hardeman causal modelling case study: extending the MRC framework

• Concise one-page representation of causal pathways• Guides the choice of intervention points and

measures• Assists in choice of behaviour change techniques• Informs the assessment of ‘fidelity’ to theories• Enables statistical modelling of the relationships

between behaviours and health outcomes

Exercises

• Exercise 1: Joint arthoplasty surgery

• Exercise 2: diabetes

The European Academy of Nursing Science 26