extending mrc guidance on developing and evaluating complex health interventions · 2019-05-20 ·...

MRC/CSO Social and Public Health Sciences Unit, University of Glasgow

Extending MRC Guidance on Developing

and Evaluating Complex Health

Interventions

Professor Sharon Simpson

MRC/CSO Social and Public Health Sciences Unit

University of Glasgow

MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.

The Team

Dr Kathryn Skivington

Dr Lynsay Matthews

Dr Peter Craig

Professor Laurence Moore

Thanks to the Medical Research Council and the National Institute for Health Research for funding the work and to our advisory group.


Caution…… work in progress…


Plan for this talk

1. Introduction – 3 versions of the guidance

2. Shortcomings of the current 2008 MRC guidance

3. Methods for developing the new guidance

4. The updated draft guidance

• Core Elements

• The 4 phases of development and evaluation

5. Conclusions and further reading


Why the original guidance was needed?

• Recognition that complex (non-pharmacological) interventions need to more be rigorously evaluated

• Specific challenges associated with defining, developing and reproducing complex interventions with multiple interacting components

• The aim was to produce guidance to help researchers and funders recognise the unique challenges of evaluating complex interventions and strategies for addressingthese.


2000 Guidance

“Complex interventions are built up from a number of components, which may act both

independently and inter-dependently.”


Phases of RCTs of complex interventions: MRC April 2000


The 2008 MRC Guidance


Why was an update needed?

• Incorporate developments since 2000

• Address limitations in the 2000 framework

• Better definition of complexity

• A less linear model of the evaluation process

• More attention to piloting/development work

• Randomised trials and alternative methods

• Need to understand process as well as outcomes

• Importance of context

• Case studies more persuasive than didactic recommendations


2008 Definition

Complex interventions are those that contain several interacting

components.

What makes an intervention complex?

• Number of interacting components

• Number and difficulty of behaviours required by those delivering or receiving the intervention

• Number of groups or organisational levels targeted by the intervention

• Number and variability of outcomes

• Degree of flexibility or tailoring of the intervention permitted


A less linear model

MRC/CSO Social and Public Health Sciences Unit, University of Glasgow

Citations per annum


The latest update to the MRC guidance

Previous guidance highly influential on researchers and funders and improved quality and quantity of CI research. However there

are gaps in the existing guidance – thinking and science has moved

on and:

•Too many studies answering narrowly focussed questions that aren’t really useful to policy and practice (in tightly controlled expts)

•Too much focus on effect size estimates

•Not enough focus on theory or involving stakeholders in answering questions important to policy and the public

•Too many studies with limited development work going to full evaluations

•Too many not considered context

•Etc etc……


New Guidance

1.Convene a steering group and a writing group

2.Complete a gap analysis/scoping reviews

3.Run expert workshop

4.Draft the update

5.Consultation process

6.Finalise guidance


Issues identified in the gap analysis

• Outdated definition of complex interventions = need to draw upon a wider understanding of the complex contexts (systems) in which interventions are delivered and evaluated.

• The predominant focus was on designing evaluations to minimise bias (RCTs)

• The limitations of RCTs and the importance of other methods, - qualitative approaches, natural experiments and novel methods such as agent based modelling, has been recognised in order to advance science and address useful questions.

• Rather than a focus on achieving a statistically significant difference in tightly controlled RCTs with limited external validity - the outcome of an evaluation may be to advance theory or produce evidence that can be used by decision makers and practitioners.


Identified Gaps

•Process still too stage based

•Lack of detail on:

•Intervention development phase

•Modification of interventions

•The use of theory

•Economic considerations

•Stakeholder involvement

•Implementation

•How to research complex interventions where interventions are not developed by researchers (eg policy)

•How to approach complexity, or take a systems perspective, in systematic reviews

5 Key Pieces of Work

1. RE-AIM (Glasgow, 1999)

2. Realist critiques & alternatives to RCTs [Pawson & Tilley 1997; Pawson, 2013]

3. Realist Trials (Bonell, 2012)

4. Process Evaluation Guidance (Moore et al 2015)

5. Context Guidance (Craig et al 2018)

http://www.google.co.uk/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&docid=UU7-Zkx6p15FDM&tbnid=UgOFDQqe-64nPM:&ved=0CAUQjRw&url=http://www.betterworldbooks.co.uk/realistic-evaluation-id-0761950095.aspx&ei=doBzU4fuK4Pe7Aax0oGwAw&bvm=bv.66699033,d.ZGU&psig=AFQjCNGDzc4LM9PmpCMpLT480kSAuotLgQ&ust=1400164781621122

http://www.google.co.uk/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&docid=NYBldc1hI-m_fM&tbnid=NyAsTgwGxYvT4M:&ved=0CAUQjRw&url=http://blogs.lse.ac.uk/lsereviewofbooks/2013/06/12/book-review-the-science-of-evaluation-a-realist-manifesto/&ei=ioBzU-W8LKve7AaL0oDACA&bvm=bv.66699033,d.ZGU&psig=AFQjCNGDzc4LM9PmpCMpLT480kSAuotLgQ&ust=1400164781621122


1. RE-AIM – Glasgow et al

Emphasises the importance not only of considering efficacy and effectiveness but also whether the intervention reaches the people targeted, can be adopted in different settings, can be implemented by staff with moderate levels of experience and training and produce replicable long lasting effects at a reasonable cost.

2. Realist Evaluation – Pawson & Tilley,

• Purpose of evaluation “as informing the development of policy and practice”

• Experiments can identify the mean effect but this is rarely, if ever, evenly produced

• “what works, for whom, and in what circumstances?”and also help us to understand “why?”

• Context + Mechanism = Outcome (CMO)

• Need to develop and continually update program theory

3. Realist RCTs – Bonell et al

• Realist evaluation was(is) seen as an alternative to RCTs

• Many trials of complex interventions include some of these elements

• Realist trials’ have been proposed as a way of:

• combining the benefits of trials (minimisation of bias, relatively straightforward causal inference)

• and realist methods (understanding what works for whom, in what circumstances and why).

3 Steps:

• Develop a programme theory or theory of change that identifies the CMO configurations through which the intervention is expected to achieve impact.

• Conduct qualitative process evaluation to refine the CMO hypotheses.

• Test the hypotheses formally ‘ via quantitative analyses of effect mediation (to examine mechanisms) and moderation


4. MRC Process Evaluation Guidance is focused on

“what works for whom in what context & why?”

• Moo

Description of

intervention

and its causal

assumptions

Logic model

Theory of

action/ change

Outcomes

Mechanisms of impact

Participant responses Intervention mediators Unanticipated pathways / consequence

ContextContextual factors which affect (and may be affected by) implementation, intervention mechanisms and

outcomes. Causal mechanisms present within the context which act to sustain the status quo, or lead to change.

Implementation

How delivery is achieved (training, resources etc..)?

What is delivered?FidelityDoseAdaptationsReach

Moore et al 2015


5. Context Guidance 2018 – Craig et al

“Understanding the relationship between intervention and context is critical to understanding success and failure, how interventions achieve impact, why their impacts vary and

whether interventions can be sustained or successfully translated from one context to another.” Craig et al, 2018,

NIHR Journals Library)

CONTEXT

MATTERS


2019 – A new approach to defining complex

interventions

Complex interventions involve “multiple, synergistic components”[Bonell et al, 2012]. Their sum is more than their parts.

They often address both social / environmental determinants and individuals’ knowledge and attitudes (i.e. “multi-level” approaches)

Complex interventions interact with context

• Intervention characteristics are only one dimension of complexity, complexity also arises from the context in which the intervention is placed, and the interaction between intervention and context.

• CIs involve emergent, unpredictable, and non-linear associations between components and outcomes (Moore, 2018)


Importance of context

A simple solution to a complex problem

http://www.luckyironfish.com/research

Program theory for iron supplement tablets

Iron deficiency

Iron tablets

Increased iron intake

Deficiency, anaemia reduced

Supply mechanisms

Access to and use of primary

care

Adherence

Reach to those most deficient

Program theory for iron supplement tablets applied in Cambodia

Iron deficiency

Iron tablets

Increased iron intake


No supply mechanisms

Suspicious of western medicine

Adherence?

Reach? –Rural? Poor?

Children?

X X

Program theory for lucky iron fish applied in Cambodia

Iron deficiency

Iron fishIncreased iron intake


Made locally,

low tech

Cultural symbol

Add fish to pot, lasts 5 years

High acceptability

& reach


Effectiveness depends on …..…..

• Capacity to manufacture fish locally and cheaply

• Cultural meanings of fish, household cooking and storage practices

• Causes of anaemia (not all dietary)

• Chemical composition of the water used for cooking (may vary seasonally)

• Etc., etc.


Complexity updated - 2019

• We need to develop more effective and sustainable interventions that either work well in a specific context or have sufficient flexibility to work across a range of contexts.

• Understanding how interventions relate to context is key to understanding how they work. Failure to do this is one reason why interventions shown to be effective in one setting fail to achieve similar impacts elsewhere.

• Understanding the relationship between intervention and context helps to explain why impacts vary and whether an intervention is likely to narrow or widen inequalities in health.

• Including detailed accounts of context will make trial results more useful to decision-makers interested in implementing, adapting, transferring and scaling up interventions and to researchers developing theoretical models of change.


Complexity – updated 2019

• Number of and interactions between components within the experimental and control interventions

• Increasing number and difficulty of behaviours, expertise and skills (including technical e.g. particular surgical techniques, and non-technical e.g. communication) required by those delivering or receiving the intervention.

• Increasing number of groups or organisational levels targeted by the intervention

• The level of flexibility or tailoring of the intervention or its components that is permitted i.e. how dynamic or adaptive the intervention is.

• The nature and range of context-by-intervention interactions


Beyond Context: System Thinking Approaches

System: a set of things that are interconnected in such a way that they produce their own pattern of

behaviour over time (Meadows, 2008). For example a school, hospital, body, government.

•In order to think in a more dynamic way about the interaction between intervention and context, it is helpful to engage with ‘systems thinking’.

•We can theorise interventions as ‘events in systems’ (Hawe et al., 2009).

•Change isnt always predictable, nor necessarily understood by linear causal pathways. In a social system,. people, interact with each other in non-linear and unpredictable ways, and their actions are interconnected so that the actions of one agent alter the context for other agents (Plsek and Greenhalgh, 2001)


Dealing with complexity

1) Calls to implement ‘complex systems approaches’ to evaluation

‘Rhetoric urging complex systems

approaches is only rarely operationalised

in ways that generate relevant evidence or effective policies.’ (Rutter et al, 2017)

In the meantime…

2) Pay more attention to context

3) Do more replication studies in novel contexts (and fewer studies of novel interventions)

4) Do more realist trials and implementation studies


The Updated Guidance – Key Points

1. Importance of the research perspective taken

2. Core elements – these should be considered at all phases of the development/evaluation process

3. 4 key phases

• No defined starting point and movement between phases is notnecessarily linear or cyclical

• It may not be possible due to time or resource constraints to address all relevant issues but it is valuable to at least consider them


1. Complexity and the research perspective

• When approaching the research a decision needs to be made on whether to attempt to understand or explain complexity, or rather to ignore or control for it.

• The guidance aims to support more ‘complexity-informed’research (Greenhalgh and Papoutsi, 2018 ), i.e. which gives sufficient and appropriate consideration of complexity.

• Complexity arises from the intervention and its interaction with context, but how you approach this complexity depends on the research perspective taken.

• The research perspective taken is informed by the questions that the research aims to address, and needs to be considered at the start of the research process.


1. Research Perspectives

Research perspectives sit on a continuum and are not mutually exclusiveEfficacy perspective: To what extent does the intervention produce the intended outcome(s) in experimental settings? Effectiveness perspective: To what extent does the intervention produce the intended outcome in real-world settings? Theory-based perspective: What works, for whom, under which circumstances, and why? Systems perspective: How does the intervention interact with the system to produce change?

Each of these perspectives has pros and cons, e.g. internal v’s external validity, risk of bias, effect size. Each can be therefore be appropriate depending on the question.


Intervention characteristics are only one dimension of complexity, complexity also arises from the context in which the intervention is placed, and the interaction between intervention and context and the evaluation perspective taken.


Perspectives Examples

• Vaccines

• Smoking


1. Research Perspectives

• Start with considering the questions you want to answer to help guide the focus of the evaluation. Is it to assess: whether the intervention is efficacious within a specific setting?; or to assess the mechanisms of the intervention; or what happens when the intervention is delivered within different contexts/settings?; or all of these?

• Consider the research perspective with regards to the intervention, its place in the wider system in which it is implemented, and the interaction between the two.

• The choice of research perspective is dependent on the uncertainties surrounding the intervention, (i.e. there is no right perspective for all circumstances).


2. New Diagram


2. Six Core Elements

• Context – an understanding of the context of the intervention should be developed at the outset and revisited and documented throughout the research process. Good understanding, description and explanation of context enables judgements to be made about transferability.

• Stakeholders - anyone who has a stake in the intervention and/or its evaluation or implementation; careful consideration at the outset of the relevant stakeholders is important. Involvement of appropriate stakeholders is crucial to each phase of research, involving them in the choice of outcomes and design ensures the outcomes matter to the stakeholders and ensuring that useful data are produced for policy/practice/patients.


2. Core Elements

• Economic considerations - Include economists in the early stages. Economic considerations are relevant at each of the phases

• Adapting the intervention - Modification is permitted, may be helpful, and should be done systematically and in a theory-informed way. On-going modification can speed up the implementation of research findings. Agree acceptable boundaries of modification beforehand.

• Identify key uncertainties and formulate research questions accordingly. Uncertainties may change throughout, therefore it is important to revisit and document them and any resulting changes in the research study. Engage multiple perspectives to develop a detailed understanding of where uncertainty lies for different stakeholder groups.


2. Core Elements

• Programme theory – develop at the outset, as a collaborative process involving researchers and stakeholders.

• The development of programme theory is an iterative process. Produce an updated programme theory at the end of each phase of development and evaluation.

• Logic models can be useful for summarising programme theory and communicating with stakeholders. However, programme theory should always be articulated well in text.


3. Phases of the research: Identifying or Developing an Intervention

• Identify or develop theory of the problem and proposed solutions.

• Consider existing scientific evidence.

• Establish theory-led research questions to drive choice of evaluation methods.

Specific considerations for intervention development

• Model process and outcomes.

• Develop and refine programme theory.

Specific considerations for intervention identification

• Undertake evaluability assessment to determine whether and how an evaluation should be undertaken.

• Consider which natural experimental methods could be used to evaluate the intervention.

• Develop and refine programme theory if one does not exist.


3. Phases of the research: Feasibility

• Assess the feasibility of the intervention and the evaluation design.

• Use pre-defined progression criteria to guide decisions about proceeding to a full evaluation.

• Use qualitative and quantitative methods.

• Avoid using data to generate effect estimates (DELTA2).

• If needed, adapt the intervention and/or evaluation design (within agreed boundaries) and report changes transparently.


3. Phases of the research: Evaluation

• Consider assessing ‘usefulness’ of data versus only effectiveness outcomes.

• Consider a range of experimental or non-experimental designs.

• Include methods of evaluation to explore the change process (i.e., mechanisms, process, context, theory).

• Conduct economic evaluation using a broad perspective.

• The main outcome of evaluation is the collation of useful data to inform future iteration and implementation.

• Report the evaluation, including updated programme theory


3. Phases of the research: Implementation

• Consider implementation during the early development and feasibility phases.

• Consider a wide range of methods to capture useful data.

• Incorporate theories, models or frameworks of implementation.

• Actively present evidence that is useful, accessible and convincing for decision-makers.

• Learn from where others have identified a ‘failure of implementation’.


Next steps

Complete first draft

and submit to Study Advisory Group

Revise guidance

and circulate for wider

consultation

Consultation period

Revise and re-submit to Study Advisory

Group

Submit to MRC and

NIHR

Publish guidance

and journal paper

Jan 2019 November 2019


Conclusion

Key points are that the perspective taken and methods used are guided by the research questions you wish to answer.

Considering context and the wider system is likely to improve the likelihood of your intervention having impact.

We hope that this guidance will help researchers and research funders recognise and adopt appropriate methods to undertake robust evaluations, which advance knowledge and provide useful data to inform decision making and practice - a crucial step to producing research with impact.


Other useful guidance

• Numerous examples of other guidance that provide more detail than main guidance

• For the development and feasibility phases:

• Recent work by Alicia O’Cathain on the INDEX study

• GUEST study – guidance on feasibility trials to be published soon by NIHR– Moore, Simpson, Craig et al

• Evaluability Assessment – Craig, P. and Campbell, M. (2015) Evaluability Assessment: A Systematic Approach to Deciding Whether and How to Evaluate Programmes and Policies. What Works Scotland.

• Evaluation and Implementation phases

• Natural experiments (Craig et al, Annu Rev Public Health. 2017 Mar 20;38:39-56)

• RE-AIM (Am J Public Health 1999 Sep;89(9):1322-7.)

• Normalisation Process Theory (May, Implement Sci. 2009, 4: 29-10.1186/1748-5908-4-29.)


THANK YOU

FOR

LISTENING

Contact: [email protected]

extending mrc guidance on developing and evaluating complex health interventions · 2019-05-20 ·...

Documents