phm analyst academy · 2020-03-11 · 10:30 –10:45 (ex-ante) design stage evaluations / impact...
TRANSCRIPT
![Page 1: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/1.jpg)
Population Health Management
NHS England and NHS Improvement
PHM Analyst Academy26th February
![Page 2: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/2.jpg)
09:00 -09:30 Arrival and Registration
09:30 – 09:45 Welcome & introduction to the day (Mentimeter) M Mohammed, SU and J Dellar, PHE
09:45 – 10:30 An apple a day: Logic models & Qual Methods P Mason, SU
10:30 – 10:45 (Ex-ante) Design stage evaluations / impact assessments vs
(Ex-post) summative evaluations
A Hood, SU
10:45 – 11:00 Coffee /Tea break
11:00 – 12:30 Modelling approaches (RIGHT Framework) A Hood, SU
12:30 – 13:15 Lunch
13:15 - 14:30 Experimental study designs
- Randomised controlled trials
- Cluster randomised trials
- Stepped wedge designs
P Seamer, SU
14:30 – 14:45 Coffee/Tea Break
14:45 - 15:45 Observational study designs
- Interrupted time series analysis
(exercise)
- Matched cohort design
- Synthetic controls
P Seamer, SU
15:45 - 16:00 Mentimeter questions
Review and look ahead to Session 4
J Dellar, PHE and M A Mohammed, SU
![Page 3: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/3.jpg)
midlandsphmacademy.nhs.uk
![Page 4: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/4.jpg)
Population Health Management
NHS England and NHS Improvement
What’s your theory of change?
A brief introduction to logic models
Paul Mason
![Page 5: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/5.jpg)
Every action has a causal theory
5
“If we do x, then we’ll get y”
Sometimes explicit; usually implicit
![Page 6: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/6.jpg)
“If we deliver our training package, then we will
improve the skills of care homes staff...
If staff are more skilled, then they will be more able to
cope in the event of a crisis...
If staff are more able to cope in a crisis, then there will
be fewer unplanned admissions to hospital....
If there are fewer unplanned admissions, then more
people will die in a setting of their choice. They will have
a better death; we will make better use of resources.”
![Page 7: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/7.jpg)
‘An apple a day keeps the doctor
away…’
Giving free fruit to primary schools in order
to improve health
What’s the causal story? How do you
(logically) get from action to outcome?
![Page 8: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/8.jpg)
Pulling out this ‘theory of change’ is essential for evaluation – and for designing initiatives
![Page 9: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/9.jpg)
We learn via advances in theory
So the unit of analysis for evaluation
should be ‘the theory’
Results then refine, reject, (provisionally)
support the theory
![Page 10: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/10.jpg)
Apples
Delivered
Apples
Eaten
Vitamin
Levels
Raised
Health
Outcomes
Improved
Interpretation
❌ ❌ ❌ ❌ Implementation Failure
☑ ❌ ❌ ❌Engagement Failure
(first causal link)
☑ ☑ ❌ ❌Theory Failure
(early causal link)
☑ ☑ ☑ ☑ Consistent with theory
☑ ☑ ☑/❌ ☑Theory Failure
(later causal link)
☑ ☑ ☑/❌ ☑/❌Partial Theory Failure
Works in some contexts
☑ ☑ ❌ ☑Theory Failure
(different causal path)
“An apple a day...” (Ref: Funnell & Rogers, 2013)
![Page 11: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/11.jpg)
Logic models are one tool for representing these theories. There are different approaches; all share common elements*
* Terminology varies but basic concepts remain constant
Inputs
Resources used
Activities
Things done (measured by outputs)
Outcomes
Effects of activities
Impacts
Broader societal ‘goods’
![Page 12: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/12.jpg)
Simple logic model for refurbishing a house (the lazy way)…
Inputs
£
Time
Activities
Source and
manage
experts
Outcomes
Improved
domestic
environment
Impacts
Increased
house value
Improved area
Inputs
£
Time
Activities
Book place to
stay, find
things to do
together
Outcomes
Reduced
stress (?)
Increased
happiness
Impacts
Improved
family
functioning
…or going on a family holiday
![Page 13: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/13.jpg)
“If we deliver our training package, then we
will improve the skills of care homes staff...
If staff are more skilled, then they will be more
able to cope in the event of a crisis...
If staff are more able to cope in a crisis, then
there will be fewer unplanned admissions to
hospital....
If there are fewer unplanned admissions, then
more people will die in a setting of their choice.
They will have a better death; we will make
better use of resources.”
This shows the
theory that
connects activity…
…to outcomes…
…to impacts
![Page 14: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/14.jpg)
•Impacts are the final effects that you are working towards – e.g. increased life expectancy, reduced health inequality, more sustainable services, etc
•Normally expressed at a high level. Triple / quadruple aim a useful framework
•Changes at this level only indirectly attributable to your intervention –you ‘contribute to’, rather than ‘cause’. Contextual factors a significant influence
Work back: what change do you ultimately want to see?
![Page 15: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/15.jpg)
•These are the changes that you are trying to make / that would (logically / evidence suggests…) result from your activities
•Can be broken down into:
• Intermediate outcomes – changes in knowledge / awareness / skills / access
• Outcomes – changes in behaviour / condition / status
•Language suggesting change is therefore important: ‘reduced, increased, improved, better, worse’
Being more detailed: what outcomes do you need to achieve your impact?
![Page 16: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/16.jpg)
•The things you do (e.g. establish apple scheme, etc)
•Measured by outputs (e.g. # people eating apples; # apples eaten)
•You don’t need to be detailed – just the main activities - the logic model is not a plan
•What ‘mechanism’ links activities to outcome? A causal ingredient?
What will you do to achieve these outcomes?
Often at this point, you’ll find
yourself working backwards
and forwards
![Page 17: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/17.jpg)
•Should be fairly straightforward: these are the resources you have to do the things you do
•Usually measured in £
•For most programmes, cash funding is the largest element – but maybe there are in-kind inputs too, e.g. if partners have assigned staff to your programme, if you have lots of volunteers, if you are given ‘free’ facilities, etc..
And what resources will you use?
![Page 18: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/18.jpg)
“In an Integrated Care System, NHS organisations, in partnership with local councils and others, take collective responsibility for managing resources, delivering NHS standards, and improving the health of the population they serve.”
NHS England
![Page 19: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/19.jpg)
Inputs
Resources used
Activities
Things done (measured by outputs)
Outcomes
Effects of activities
Impacts
Broader societal ‘goods’
![Page 20: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/20.jpg)
Take a step back and reflect on:
1: Assumptions in the model:o Practical (e.g. shows significant reliance on recruitment of…)
o Evidential (e.g. implied connection between activity x and effect y)
o Contextual (e.g. that there is no significant change in regulation of x)
Can this be used in programme planning? Is this showing risks to be managed? Would more evidence help design? What does it mean for evaluation?
2: Your overall theory of change. Policies generally use:o Sticks (beat / regulate things into place)
o Carrots (incentivise / ease the change you want)
o Sermons (eulogise and persuade)
What is the mix in your theory? Does this seem optimal given the task? If not, what is missing and can this be managed?
![Page 21: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/21.jpg)
Parting note 1: Logic models and economic evaluation
Cost
(Economy)
Efficiency (£ per output)
Effectiveness / Benefit (£ per outcome)
![Page 22: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/22.jpg)
Parting note 2: your model should reflect your theory, which might not be linear
![Page 23: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/23.jpg)
Further resources / guidance
1. www.strategyunit.co.uk – search ‘logic model’ for fuller guide to using logic models
2. www.betterevaluation.org
3. HMT ‘Magenta Book’ – good all round guidance on evaluation
![Page 24: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/24.jpg)
Population Health Management
NHS England and NHS Improvement
An incredibly brief introduction to qualitative
methods for evaluation
Paul Mason
![Page 25: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/25.jpg)
Qualitative Vs
Quantitative?
![Page 26: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/26.jpg)
There are many different types of qualitative method / approaches for evaluation…
Individual
interviews
Group
interviews
Focus groupObservation
Document
analysis
Participatory
approaches
Diaries
Social media
analysisQualitative
surveysCase studies
Ethnography
![Page 27: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/27.jpg)
…all underpinned by some common purposes
Understanding things as
they are experienced by
the people involved
(context)
Exploring – what matters, to
whom, why
Explaining (e.g.) why did
events unfold in this way?
Creating (e.g.) what
should we do now?
(Not always, but…) democratising
the evaluation process
Interesting to note what is
not on this list that would
be there for quantitative
method
![Page 28: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/28.jpg)
Choice of method depends on many factors
•The question! First and most important consideration…
•Time and resource. Qualitative methods can be expensive / time consuming
•Ethics. Accessing participants, what you might find, power dynamics (etc)
•Skills. Done well, it looks easy; but so easy to do badly. Analysis of non-quantitative data is a specialist skill too
•Stage of the evaluation process. Drawing out theory? Designing approaches (defining outcomes that matter)? Tracing process / implementation? Explaining effects? Interpreting findings / working up recommendations?
•Related point: where does this fit with quantitative methods?
![Page 29: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/29.jpg)
Population Health Management
NHS England and NHS Improvement
Evaluation and Impact Assessments
Andy Hood
![Page 30: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/30.jpg)
•Before implement - design stage evaluation / ex-ante modelling
•After implement - quantitative (ex-post) evaluation
When should we estimate the effect of a planned change?
![Page 31: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/31.jpg)
31
time
Ex-ante
Ex-post
known
unknown
Intervention
or RiskOutcome
or Impact Applications
• commissioning plans• service plans• opportunity assessment• business case
• roll-out • decommission
• adjust plans / contracts• remedial action
![Page 32: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/32.jpg)
Population Health Management
NHS England and NHS Improvement
Break
![Page 33: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/33.jpg)
Population Health Management
NHS England and NHS Improvement
Design stage evaluationAndy Hood
![Page 34: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/34.jpg)
What it is that makes each of these a model?
What are common characteristics of models?
y ~ x
![Page 35: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/35.jpg)
•Models help us understand things that would otherwise be obscured by the complexity of the real world.•Associations, casual relationships and core dynamics – descriptive / explanatory
•Consequences, forecasts - predictive
•Help us understand what we need to do - prescriptive
•Models make us document our assumptions.
•Models help us test things that would be too costly / risky / impractical / unethical to try in real life.
•Models can act as guides or templates for complicated actions / developments.
Why do we need models in healthcare?
![Page 36: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/36.jpg)
Modelling and simulation techniques for supporting healthcare decision making: a selection framework
A collaboration of 6 universities
•Cambridge Engineering Design Centre, University of Cambridge
•The School of Information Systems, Computing and Mathematics, Brunel University
•Brunel Business School, Brunel University
•The School of Management, University of Southampton
•The Information Engineering Research Group, University of Ulster
•The School of Mathematics, Cardiff University
The RIGHT Frameworkhttps://www-edc.eng.cam.ac.uk/downloads/right.pdf
![Page 37: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/37.jpg)
“This workbook is intended to provide guidance for people who are making decisions in healthcare. It is aimed at anyone who wants to find out more about different modelling and simulation techniques –what they are, when to apply them, and what resources are required to use them. It will not only help decision makers commission more appropriate modelling work, but also assist professional modellers and business consultants to expand their modelling repertoire in order to meet the diverse needs of their clients.
The workbook is not a “how-to-do” guide to modelling and simulation, rather a “what-is-it” introductory guide. That said, the further reading section at the end of the workbook will help locate further details for each technique. The RIGHT research team would also welcome any contact regarding the applications of these techniques.”
![Page 38: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/38.jpg)
![Page 39: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/39.jpg)
![Page 40: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/40.jpg)
![Page 41: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/41.jpg)
![Page 42: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/42.jpg)
Technique characterisation (input requirements)
![Page 43: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/43.jpg)
![Page 44: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/44.jpg)
RIGHT framework: technique selection table
![Page 45: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/45.jpg)
Thinking about a specific project you are involved with and using the RIGHT resources on your tables:
• What stage of the project cycle are you currently (stuck) at?
• What type/s of output are you looking for?
• What constraints are you under - time, costs etc…
When you’ve identified a potential modelling tool, your HOMEWORK is to find about a bit more about that technique and discuss with manager/project lead how you’re going to apply it.
Have a go yourself....(5 mins)
![Page 46: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/46.jpg)
RIGHT Modelling
Framework
Design Stage
Evaluation
![Page 47: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/47.jpg)
•A visual aid to explore how different variables in a system are related.
•A modelling method in its own right – but can also underpin quantitative modelling methods
•Comprised of
•a set of nodes, each representing a quantity or variable
•a set of arrows indicating the influence of one quantity on another
Causal loop diagrams (directed acyclic graphs)
![Page 48: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/48.jpg)
X Y
an increase (decrease) in X tends to cause
an increase (decrease) in Y…
… all other things being equal
Population size
Number of
people with a
mental health
problem
![Page 49: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/49.jpg)
Smoking
prevalance
Population life
expectancy
X Y
an decrease (increase) in X tends to cause
an increase (decrease) in Y…
… all other things being equal
![Page 50: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/50.jpg)
X Y
X influences Y but through Z (i.e. not directly)
Z
Prevalence of
cardio-
vascular
disease
Premature
deaths from
cardio-vascular
disease
Smoking
prevalence
Population life
expectancy
![Page 51: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/51.jpg)
Prevalence of
cardio-
vascular
disease
Premature
deaths from
cardio-vascular
disease
Smoking
prevalence
Population life
expectancy
Incidence of
cancer
Premature
deaths from
cancer
![Page 52: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/52.jpg)
hungerfood
consumptionb
a balancing loop
![Page 53: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/53.jpg)
bank balance interestr
a reinforcing loop
![Page 54: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/54.jpg)
•Balancing and reinforcing loops can be made up of more the 2 nodes
•A loop is
Balancing – if there are an odd number of negative arrows (- - - →)
Reinforcing – if there are no or an even number of negative arrows (- - - →)
Balancing and reinforcing loops
b
r
![Page 55: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/55.jpg)
•Work in groups of 2 or 3
•Join the nodes to indicate influence
•Decide if/where there are balancing or reinforcing loops.
•When discussions concluded, you may want to produce final / tidy version
TIP: In total, we think there are 10 arrows and 3 (+1) loops.
Exercise – Join the dots patients who
need to be
admitted
admissionsaverage
length of stay
admission
threshold
discharge
threshold
unoccupied
beds
all beds
![Page 56: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/56.jpg)
Exercise – Join the dots patients who
need to be
admitted
admissionsaverage
length of stay
admission
threshold
discharge
threshold
unoccupied
beds
all beds
![Page 57: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/57.jpg)
Exercise – Join the dots patients who
need to be
admitted
admissionsaverage
length of stay
admission
threshold
discharge
threshold
unoccupied
beds
all beds
b
bb
![Page 58: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/58.jpg)
Exercise – Join the dots patients who
need to be
admitted
admissionsaverage
length of stay
admission
threshold
discharge
threshold
unoccupied
beds
all beds
b
bb
r
![Page 59: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/59.jpg)
Roemer’s Law
"in an insured population, a hospital bed built is a filled bed“
Shain, M; Roemer, MI (April 1959). "Hospital costs relate to the supply of beds". Modern Hospital. 92 (4): 71–3
Parkinson’s Law
"the number of patients always tends to equality with the number of beds available for them to lie in“
patients who
need to be
admitted
admissionsaverage
length of stay
admission
threshold
discharge
threshold
unoccupied
beds
all beds
b
bb
r
![Page 61: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/61.jpg)
Last year…
•Hip replacements completed – 1,253
•Mean length of stay – 7 days
•Overnight bed occupancy – 80%
•Beds – 30
•Mean waiting time – 91 days
You speak with the service and clinical lead and they say that in 5 years time they expect…
•Demand will have increased by 12%
•Mean length of stay will fall by 14%
•It would like overnight bed occupancy to reduce to 70%
How many more/less beds will we need?
Quantitative modellingan elective inpatient hip replacement service
![Page 62: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/62.jpg)
Current Change parameter Future
Annual admissions 1,253 +12% 1,403
Average LoS 7.0 -14% 6.0
Actual occupied bed
days8,771 8,448
Bed days with
occupancy %10,964 -10% 12,069
Beds required 30 33
Therefore, under this model, we would require an additional 3 beds for our hip replacement service
![Page 63: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/63.jpg)
Last year…
•Hip replacements completed – 1,253
•Mean length of stay – 7 days
•Overnight bed occupancy – 80%
•Beds – 30
•Mean waiting time – 91 days
You speak with the service and clinical lead and they say that in 5 years time they expect…
•Demand will have increased by 12% for all levels of need
•24% of cases have only a moderate level of need. They would like to divert all these cases to receive a non-surgical service in another unit
•Mean length of stay for the non-moderate cases will fall by 14%
•It would like overnight bed occupancy to reduce to 70%
Quantitative modellingan elective inpatient hip replacement service
![Page 64: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/64.jpg)
Last year…
•Hip replacements completed – 1,253•Moderate need - 301•High need - 752•Severe need - 200
•Mean length of stay – 7 days•Moderate need – 3.0 days•High need – 7.0 days•Severe need – 13.0 days
•Overnight bed occupancy – 80%
•Beds – 30
•Mean waiting time – 91 days
You speak with the service and clinical lead and they say that in 5 years time they expect…
•Demand will have increased by 12% for all levels of need
•24% of cases have only a moderate level of need. They would like to divert all these cases to receive a non-surgical service in another unit
•Mean length of stay for the non-moderate cases will fall by 14%
•It would like overnight bed occupancy to reduce to 70%
How many beds will we need?
What assumptions are needed to reach this view?
Quantitative modellingan elective inpatient hip replacement service
![Page 65: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/65.jpg)
Current Change parameter Future
Annual admissions 1,253Shift 'moderate need'
then +12%1,066
Average LoS 7.0-14% for remaining
cases7.1
Actual occupied bed
days8,771 7,575
Bed days with
occupancy %10,964 -10% 10,821
Beds required 30 30
Therefore, under this model, we would require no change in beds for our hip replacement service
![Page 66: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/66.jpg)
Last year…
•Hip replacements completed – 1253•Moderate need - 301•High need - 752•Severe need - 200
•Mean length of stay – 7 days•Moderate need – 3.0 days•High need – 7.0 days•Severe need – 13.0 days
•Overnight bed occupancy – 80%
•Beds – 30
•Mean waiting time – 91 days
You speak with the service and clinical lead and they say that in 5 years time they expect…
•Demand will have increased by 12% for all levels of need
•24% of cases have only a moderate level of need. They would like to divert these cases to receive a non-surgical service in another unit
•Mean length of stay for the non-moderate cases will fall by 14%
•It would like overnight bed occupancy to reduce to 70%
How many beds will we need?•What if we only manage to divert 50% of moderate cases?•What if we live with 80% occupancy?
What assumptions are needed to reach this view?
Quantitative modellingan elective inpatient hip replacement service
![Page 67: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/67.jpg)
Current Change parameter Future
Annual admissions 1,253
Shift 50% of
'moderate need'
then +12%
1,235
Average LoS 7.0-14% for remaining
cases6.5
Actual occupied bed
days8,771 8,009
Bed days with
occupancy %10,964 No change 10,012
Beds required 30 27
Therefore, under this model, we would require 3 fewer beds in our hip replacement service
![Page 68: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/68.jpg)
But what if the question you really need to answer was;
•If we divert moderate need patients from the beginning of year 4, how long will it take before the average waiting time falls below 84 days?
•What proportion of patients will spend more than 15 days in hospital?
•How frequently will occupancy exceed 85%?
What if the length of stay reduction is heavily dependent on the occupancy rate?
In such circumstances, basic rules-based models are not sufficient.
Two options worth considering;
•Systems dynamics modelling
•Discrete event simulation
Type Name Link
Proprietary AnyLogic https://www.anylogic.com/
Simul8 https://www.simul8.com/
Sysdea https://sysdea.com/
Open Source Insight Maker https://insightmaker.com/
Simmer (for R) https://r-simmer.org/
![Page 69: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/69.jpg)
Time dependant
I need to know about the model variables at a number of time points –
not just the start and end state of the model and I expect the changes
to evolve non-linearly over time
⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫
Individual
I need to know about variation between patients (its not sufficient to
know the average effect for groups of patients) and I have patient level
data to populate the model.
⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫
Interactions / dynamic
What happens to one patients strongly and importantly effects what
happens to others.⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫
Stochastic
I need to know about the degree of uncertainty in the model results
(either due to randomness or uncertainty of inputs / parameters) – a
point estimate with / without sensitivity analysis is not sufficient.
⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫
Basic rule based model ⚫
Systems dynamics model ⚫ ⚫ ⚫ ⚫
Discrete event simulation model ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫
![Page 70: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/70.jpg)
Discrete event simulation (DES)
Models of sequences of discrete events within a system where future events are a condition of the current system state.
•e.g. patients flowing through a hospital.
Probability distributions used to describe the arrival and treatment of patients.
https://www.youtube.com/watch?v=4BHBJlzv4RA
Systems dynamics modelling (SD)
Used to model nonlinear behaviour in systems over time.
Causal loop models used to describe stocks and flows within a systems.
Calculus used to define the system.
https://www.youtube.com/watch?v=nTD1SL2qp3o
![Page 71: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/71.jpg)
Evidence is best – but not always available. It’s often necessary to rely on expert opinion.
Establish a reference group
•One opinion is better than none, but multiple opinions are better still (https://en.wikipedia.org/wiki/Wisdom_of_the_crowd)
•Be clear what you need to know
•Provide the reference group with useful context
Ask your reference group members to provide •a central estimate (best guess) is good, but ranges are better still•Ask for best guess, low & high estimate – and use triangular distributions or ask for 90% confidence intervals•You can calibrate you reference group member’s views (https://hubbardresearch.com/publications/how-to-measure-anything-book/)
Delphi methods can be used to aggregate multiple views•Ask reference group members for views –share these and ask or second set of estimates (https://en.wikipedia.org/wiki/Delphi_method)
Practical tips on parameritising* a model?
* yes that really is a word!
![Page 72: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/72.jpg)
Population Health Management
NHS England and NHS Improvement
Lunch
![Page 73: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/73.jpg)
Population Health Management
NHS England and NHS Improvement
Experimental Study Designs
Paul Seamer
![Page 74: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/74.jpg)
Population Health Management
NHS England and NHS Improvement
Break
![Page 75: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/75.jpg)
Population Health Management
NHS England and NHS Improvement
Observational Study Designs
Paul Seamer
![Page 76: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/76.jpg)
Population Health Management
NHS England and NHS Improvement
ReflectionsMohammed A Mohammed and Janine Dellar
![Page 77: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/77.jpg)
![Page 78: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/78.jpg)
![Page 79: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/79.jpg)
![Page 80: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/80.jpg)
![Page 81: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/81.jpg)
![Page 82: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/82.jpg)
![Page 83: PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea](https://reader034.vdocuments.us/reader034/viewer/2022042413/5f2ce9bb209bb75af976e7f5/html5/thumbnails/83.jpg)
Population Health Management
NHS England and NHS Improvement
Many thanks for your participation!