using programme theory for evaluation of complex health interventions at district level
DESCRIPTION
In this presentation, we explain the process through which a realist evaluation could be conducted on complex interventions through the building and refining of programme theories of these interventions.TRANSCRIPT
Using programme theory for evaluation of complex interventions
Prashanth NSFaculty & PhD Scholar
Institute of Public Health, Bangalore
Outline
• Background– Interventions, HS interventions and complexity– Framing evaluation questions? “Did it…”, “What..”,
“Why (or why not?)..” or “How…”– Multiple paths
• How to use a programme theory approach in a complex local health systems intervention
Source: Pd photo/NLM, USA
“Medicine is a social science, and
politics is nothing else but
medicine on a large scale.
Medicine, as a social science, as
the science of human beings, has
the obligation to point out
problems and to attempt their
theoretical solution: the
politician, the practical
anthropologist, must find the
means for their actual solution…”
Randomise and control
Origins in drug testing for effectiveness– Outcomes measurable, verifiable; animal models
available. Environmental conditions can be controlled. • Linearity (linear and observable transitions from inputs
to outcome; environmental conditions known)• No ethical issues in setting up control• Gold-standard for effectiveness
Hazardous journeys
Smith, G. C. S., & Pell, J. P. (2003). Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials. BMJ, 327(7429), 1459–61. doi:10.1136/bmj.327.7429.1459
But health systems are complex…
WHO 2000 WHO AHPSR 2008
Software neglected
Complexity & systems-thinking
• Simple, complicated and complex (Glouberman & Zimmerman)
– Examples: from recipies to rockets to children• Characteristics– Multiple interacting elements (within and outside)– Organisational structure and interactions– Multiple paths/configurations to same outcome;
same structural configuration but different outcome (path dependency)
– Unpredictability due to feedback loops
Complexity in health systems/interventions
Institute of Public Health, Bangalore
Medical research council guidance
Effectiveness or Mechanism?
Existing methods in public health research focus on effectiveness: “Did it work? To what extent”
But, middle-level managers, decision-makers are looking for: “How did it work?”, “Why did it not work for x or y?”
The evaluation question
Ideally, evaluations should be able to inform not only if a given intervention worked, but also how it worked, and why it worked for some (and not for others)
Why did Arogyashri not benefit beneficiaries from SC/ST proportional to
their population in AP?Is there a social phenomenon operating
here that could help us understand implementation of financing or other
schemes in general?
Fan, V. Y., & Mahal, A. (2011). Learning and getting better : Rigorous evaluation of health policy in India. National Medical Journal of India, 24(6), 325–327.
Answering how questions
Various methods possible (MRC guidance)
– CrCT with process evaluations, stepped-wedge designs
– Modeling: causal modelling, mathematical modelling, TdI, economic modelling
– Natural experiments & cohort studies
The programme theory approach
A programme theory is way of representing the expected relationship between the elements of the intervention and its expected outcomes.
Consider the pulse polio programme
But, for complex interventions…
Initial programme assumptions
Learning from general theory
Reconstruction
Empircise
Applications of the PT
• Theorising and understanding “how”: in parallel with RCTs (process evaluation) or preliminary to a realist evaluation (my study design)
• Informing design/development of future similar programmes
• Forms a basis for analysing the qualitative data and for using frameworks (Eg. Using MPF for understanding organisational performance)
Six-step modelVan Belle, S. B., Marchal, B., Dubourg, D., & Kegels, G.
(2010). How to develop a theory-driven evaluation design? Lessons learned from an adolescent sexual and reproductive
health programme in West Africa. BMC public health, 10, 741. doi:10.1186/1471-2458-10-741
A mixed method RE design using PT
Addressing complexity by understanding relationships rather than structural outcomes
Marchal et. al’s representation of Sicotte framework
Using PT for evaluation
Examining the data using the framework helps further refine the PT. Allows for identifying positive and negative configurations of the context-mechanism-outcome that can be emperically verified to answer the question “How does capacity-building work in this intervention?”
Key referencesPrashanth, N. S., Marchal, B., Hoeree, T., Devadasan, N., Macq, J., Kegels, G., & Criel, B. (2012). How does capacity building of health managers work? A realist evaluation study protocol. BMJ open, 2(2), e000882. doi:10.1136/bmjopen-2012-000882
Marchal, B., Van Belle, S., Van Olmen, J., Hoeree, T., & Kegels, G. (2012). Is realist evaluation keeping its promise? A review of published empirical studies in the field of health systems research. Evaluation, 18(2), 192–212. doi:10.1177/1356389012442444
Van Belle, S. B., Marchal, B., Dubourg, D., & Kegels, G. (2010). How to develop a theory-driven evaluation design? Lessons learned from an adolescent sexual and reproductive health programme in West Africa. BMC public health, 10, 741. doi:10.1186/1471-2458-10-741
Craig, P., Dieppe, P., Macintyre, S., Michie, S., Nazareth, I., & Petticrew, M. (2008). Developing and evaluating complex interventions: the new Medical Research Council guidance. British Medical Journal, 337(sep29 1), a1655–a1655. doi:10.1136/bmj.a1655
Marchal, B., Dedzo, M., & Kegels, G. (2010). A realist evaluation of the management of a well- performing regional hospital in Ghana. BMC Health Services Research, 10(October 2000), 24.
On Mendeley, see my reading list on critical realism and realist evaluation at:http://www.mendeley.com/groups/535071/critical-realism-and-realist-evaluation/papers/
AcknowledgementBart Criel, Guy Kegels, Jean Macq, Bruno Marchal & Tom Hoeree at ITM, AntwerpUpendra Bhojani, Tanya Seshadri, Arima Mishra & N Devadasan for discussions and re-discussions