really using (useful) theories of change ipdet 2013 john mayne, advisor on public sector performance...
TRANSCRIPT
Really Using(Useful)
Theories of Change
IPDET 2013
John Mayne, Advisor on Public Sector Performance
Using Theories of Change
1. Understanding and agreeing on interventions2. Addressing equity issues3. Designing interventions4. Ex ante evaluation5. Designing monitoring systems6. Managing adaptively7. Reporting performance8. Designing evaluation questions, methods and tools9. Making causal claims10. Generalizing to other locations
Behaviour changes
Activities and Outputs with respect to beneficiaries
External Influences
Capacity changes in knowledge, attitudes
skills, opportunities & incentives
Reach &Reaction
Direct Benefits
Impacts
A Generic Results Chain
Timeline
Behaviour changes
Activities and Outputs with respect to beneficiaries
External Influences
Capacity changes in knowledge, attitudes
skills, opportunities & incentives
Reach &Reaction
Direct Benefits
Impacts
A Generic Results Chain
Timeline
Note• No labeling of
outcomes types• Focus on program
theory• Timeline• Built on capacity and
practices changes• Explicit recognition of
reach• Feedback loops
Behaviour changes
Activities and Outputs with respect to beneficiaries
External Influences
Capacity changes in knowledge, attitudes
skills, opportunities & incentives
Reach &Reaction
Direct Benefits
Impacts
Assumptions and Risks
Timeline
Behaviour Change Assumptions: How are changes in knowledge, attitudes, skills, opportunities and/or incentives expected to change behaviour? What events and conditions have to happen? What factors influence these processes?
Risks: Risks to the link not occurring.Other Explanatory Factors: Peer or trend pressure; other interventions
An assumption is NOT a description of the causal link
Behaviour changes
Activities and Outputs with respect to beneficiaries
External Influences
Capacity changes in knowledge, attitudes
skills, opportunities & incentives
Reach &Reaction
Direct Benefits
Direct Benefits Assumptions & Risks Other explanatory factors
Behaviour Change Assumptions & Risks
Other explanatory factors
Capacity Change Assumptions & Risks
Other explanatory factors
Reach Assumptions & Risks
Impacts
Impact Assumptions & Risks Other explanatory factors
A Generic Beneficiaries Theory of Change
Timeline
timelineAwareness and skills training provided to
teachers on girls’ education needs
External InfluencesParentsPeers
ReligionCulture
Education resourcesCapacity changes in teachers’ knowledge, attitudes and skills
All relevant teachers get the
training
Girls learning improves
Results Chain for Enhancing Education Outcomes for Girls
Girls more engaged in and wanting an
education
Improved education
outcomes for girls
Teachers provide girls with more empathetic
and supportive teaching in schools
Intervention to improve educational outcomes for girls by providing gender sensitivity training to teachers, in a region where girls education has not been a priority.
Girls learning improves
Other Factors for Enhancing Education Outcomes for Girls
Girls more engaged in and
wanting an education
Teachers provide girls with more empathetic and
supportive teaching in schools
• Parents and community support
• Time for studying
• Access to schools• Accommodation in
schools
timelineAwareness and skills training provided to
teachers on girls’ education needs
External InfluencesParentsPeers
ReligionCulture
Education resources Capacity changes in teachers’ knowledge, attitudes and skills
All relevant teachers get the
training
Girls learning improves
Unintended effect: Boys’ education suffers
Girls’ Behaviour Assumptions & Risks Girls have access to schools Schools can adequately
accommodate girlsOther explanatory factors Internet access becomes widely
available
Theory of Change for Enhancing Education Outcomes for Girls
Girls more engaged in and wanting an
education
Improved education
outcomes for girls
Teachers provide girls with more empathetic
and supportive teaching in schools
Girls’ Direct Benefit Assumptions & Risks Girls have time to study Girls education supported by parents and
community Focus on girls is not disruptive to boy’s
educationOther explanatory factors Internet access becomes widely available
Other factorsSeems clear that more than training teachers may be required, such as:• Getting girls to school• Parental and community support• Schools accommodate girls• Teachers support girls education• Empathetic teaching is rewarded• Focus on and resources for girls’ education are
ongoing (outcomes take time)
timelineAwareness and skills training provided to
teachers on girls’ education needs
External InfluencesParentsPeers
ReligionCulture
Education resourcesCapacity changes in teachers’ knowledge, attitudes and skills
All relevant teachers get the
training
Girls learning improves
Unintended effect: Boys’ education suffers
Teacher Behaviour Change Assumptions & Risks
Training relates to the local conditions Teachers are assessed in part on their
application of the training
Capacity Change Assumptions and Risks• Most teachers want to better education
for girls
Reach Assumptions & Risks Only teachers with a predisposition
to the importance of girls’ education attend
Impact Assumptions & Risks Resources and will continue for
multi-year involvement
Theory of Change for Enhancing Education Outcomes for Girls
Girls more engaged in and wanting an
education
Improved education
outcomes for girls
Teachers provide girls with more empathetic
and supportive teaching in schools
Girls’ Behaviour Assumptions & Risks Girls have access to schools Schools can adequately accommodate girlsOther explanatory factors Internet access becomes widely available
Girls’ Direct Benefit Assumptions & Risks Girls have time to study Girls education supported by parents and
community Focus on girls is not disruptive to boy’s
educationOther explanatory factors Internet access becomes widely available
Equity: Distribution of impacts
• As set out so far, ‘girls’ is an general group• But among ‘girls’, there are worst-off girls, who may
need special attention (e.g., access, income, conservative parents)
• And as already noted, there may be unintended impacts on boys
• To get at these distributional affects, we can build nested theories of reach:– A ToC for a particular reach group within an overall ToC
• And can then tell reach impact stories about the effects and impacts of different reach groups
timelineAwareness and skills training provided to
teachers on girls’ education needs
External InfluencesParentsPeers
ReligionCulture
Education resourcesCapacity changes in teachers’ knowledge, attitudes and skills
All relevant teachers get the
training
Girls learning improves
Unintended effect: Boys’ education suffers
Teacher Behaviour Change Assumptions & Risks
Training relates to the local conditions Teachers are assessed in part on their
application of the training
Capacity Change Assumptions and Risks• Most teachers want to better education
for girls
Reach Assumptions & Risks Only teachers with a predisposition
to the importance of girls’ education attend
Impact Assumptions & Risks Resources and will continue for
multi-year involvement
Theory of Change for Enhancing Education Outcomes for Girls
Girls more engaged in and wanting an
education
Improved education
outcomes for girls
Teachers provide girls with more empathetic
and supportive teaching in schools
Girls’ Behaviour Assumptions & Risks Girls have access to schools Schools can adequately accommodate girlsOther explanatory factors Internet access becomes widely available
Girls’ Direct Benefit Assumptions & Risks Girls have time to study Girls education supported by parents and
community Focus on girls is not disruptive to boy’s
educationOther explanatory factors Internet access becomes widely available
Teachers Theory of Reach
Girls Theory of Reach
Worst-Off Girls Theory of Reach
timelineAwareness and skills training provided to
teachers on girls’ education needs
External InfluencesParentsPeers
ReligionCulture
Education resourcesCapacity changes in teachers’ knowledge, attitudes and skills
All relevant teachers get the
training
Girls learning improves
Unintended effect: Boys’ education suffers
Teacher Behaviour Change Assumptions & Risks
Training relates to the local conditions Teachers are assessed in part on their
application of the training
Capacity Change Assumptions and Risks• Most teachers want to better education
for girls
Reach Assumptions & Risks Only teachers with a predisposition
to the importance of girls’ education attend
Impact Assumptions & Risks Resources and will continue for
multi-year involvement
Theory of Change for Enhancing Education Outcomes for Girls
Girls more engaged in and wanting an
education
Improved education
outcomes for girls
Teachers provide girls with more empathetic
and supportive teaching in schools
Girls’ Behaviour Assumptions & Risks Girls have access to schools Schools can adequately accommodate girlsOther explanatory factors Internet access becomes widely available
Girls’ Direct Benefit Assumptions & Risks Girls have time to study Girls education supported by parents and
community Focus on girls is not disruptive to boy’s
educationOther explanatory factors Internet access becomes widely available
timelineAwareness and skills training provided to
teachers on girls’ education needs
External InfluencesParentsPeers
ReligionCulture
Education resourcesCapacity changes in teachers’ knowledge, attitudes and skills
All relevant teachers get the
training
Girls learning improves
Unintended effect: Boys’ education suffers
Teacher Behaviour Change Assumptions & Risks
Training relates to the local conditions Teachers are assessed in part on their
application of the training
Capacity Change Assumptions and Risks• Most teachers want to better education
for girls
Reach Assumptions & Risks Only teachers with a predisposition
to the importance of girls’ education attend
Impact Assumptions & Risks Resources and will continue for
multi-year involvement
Theory of Change for Enhancing Education Outcomes for Girls
Girls more engaged in and wanting an
education
Improved education
outcomes for girls
Teachers provide girls with more empathetic
and supportive teaching in schools
Girls’ Behaviour Assumptions & Risks Girls have access to schools Schools can adequately accommodate girlsOther explanatory factors Internet access becomes widely available
Girls’ Direct Benefit Assumptions/Risks Girls have time to study Girls education supported by parents and
community Focus on girls is not disruptive to boy’s
educationOther explanatory factors Internet access becomes widely available
Supporting activities
timelineAwareness and skills training provided to
teachers on girls’ education needs
External InfluencesParentsPeers
ReligionCulture
Education resourcesCapacity changes in teachers’ knowledge, attitudes and skills
All relevant teachers get the
training
Girls learning improves
Unintended effect: Boys’ education suffers
Teacher Behaviour Change Assumptions & Risks
Training relates to the local conditions Teachers are assessed in part on their
application of the training
Capacity Change Assumptions and Risks• Most teachers want to better education
for girls
Reach Assumptions & Risks Only teachers with a predisposition
to the importance of girls’ education attend
Impact Assumptions & Risks Resources and will continue for
multi-year involvement
Theory of Change for Enhancing Education Outcomes for Girls
Girls more engaged in and wanting an
education
Improved education
outcomes for girls
Teachers provide girls with more empathetic
and supportive teaching in schools
Girls’ Behaviour Assumptions & Risks Girls have access to schools Schools can adequately accommodate girlsOther explanatory factors Internet access becomes widely available
Girls’ Direct Benefit Assumptions/Risks Girls have time to study Girls education supported by parents and
community Focus on girls is not disruptive to boy’s
educationOther explanatory factors Internet access becomes widely available
Supporting activities
Teachers Theory of Reach
Girls Theory of Reach
Parents Theory of Reach
Worst-Off girls learning improves
Unintended effect: Boys’ education suffers
Worst-Off Girls’ Behaviour & Direct Benefit Change Assumptions & Risks Worst-off girls have
access to schools Worst-off girls education
supported by parents and community
Worst-off girls have time to study
Schools can adequately accommodate girls
Focus on girls is not disruptive to boy’s education
Other explanatory factors Internet access becomes
widely available
Impact Assumptions & Risks Resources and will continue
for multi-year involvement
Theory of Reach for Enhancing Education Outcomes for Worst-Off Girls
Worst-Off girls more engaged in and wanting an
education
Improved education outcomes for worst-
off girls
Teachers provide (all) girls with more empathetic and
supportive teaching in schools
Worst-off girls are provided transportation to schools
Engaged with parents & communities with worst-off girls
Support Activities
intervention activities
results
timeline
timeline
Engagement with governments
Engagement with NGOs
Partner Activities
Nested theory of reach for access by worst-off girls
Reach &Reaction
Capacity Changes
Behaviour Changes
Impact
Direct Benefits Assumptions & Risks Other Explanatory Factors
Behavioural Change Assumptions & Risks Other Explanatory Factors
Capacity Change Assumptions & Risks Other Explanatory Factors
Reach Assumptions & Risks
External Influences
Supporting Activities and O
utputs
Direct Benefit Changes
Impact Assumptions & Risks Other Explanatory Factors
TimelineActivities and Outputs
Timeline
A Basic Theory of Change for Comprehensive Interventions
Reach, C
apacity changeB
ehavioural Change
Reach &Reaction
Capacity Changes
Behaviour Changes
Impact
Direct Benefits Assumptions & Risks Other Explanatory Factors
Behavioural Change Assumptions & Risks Other Explanatory Factors
Capacity Change Assumptions & Risks Other Explanatory Factors
Reach Assumptions & Risks
External Influences
Supporting Activities and O
utputs
Direct Benefit Changes
Impact Assumptions & Risks Other Explanatory Factors
TimelineActivities and Outputs
Timeline
A Basic Theory of Change for Comprehensive Interventions
Reach, C
apacity changeB
ehavioural Change
Different Beneficiary Nested ToC
Different Intermediary Nested ToC
Using Theories of Change
5. Designing monitoring systems– likely need more than just a focus on results—the
whole ToC, including assumptions and risks
6. Adaptive management– Identify what to be monitored– On a regular basis, reflect on what is being
observed and on your ToC – Revise your intervention activities and your ToC as
you learn
Using Theories of Change7. Reporting performance– A verified ToC is your performance story
8. Designing evaluation questions, methods, tools– Identifies questions– Adds power to case studies by confirming the ToC– Basis for surveys et al
Behaviour changes
Activities and Outputs with respect to beneficiaries
External Influences
Capacity changes in knowledge, attitudes
skills, opportunities & incentives
Reach &Reaction
Direct Benefits
Impacts
There are potential Evaluation Questions associated with each box and arrow
Timeline
Behaviour Change Assumptions: How are changes in knowledge, attitudes, skills, opportunities and/or incentives expected to change behaviour? What events and conditions have to happen? What factors influence these processes?
Risks: Risks to the link not occurring.Other Explanatory Factors: Peer or trend pressure; other interventions
Behaviour Change• What behavioural
(practice) changes are observed?
Causal Question• To what extent are the
behavioural changes the result of the capacity changes?
Capacity Change• What capacity) changes
are observed?
Assumption Questions• Did the events/conditions
occur?• Did risks materialize?• Are other explanations
plausible?
Using Theories of Change
9. Making causal claims– Based on generative causality– ToC are models of causality– Approaches such as contribution analysis
demonstrates causality
Theories of Change asCausal Packages
ToC is a model of the intervention as a contributing cause– ToC identifies supporting factors (assumptions)
and confounding factors (risks)– Intervention activities plus the assumptions are
sufficient to bring about the intended impacts– Explains how and why intended impacts are
expected to occur
Reach &Reaction
Capacity Changes
Behaviour Changes
Impact
Direct Benefits Assumptions & Risks Other Explanatory Factors
Behavioural Change Assumptions & Risks Other Explanatory Factors
Capacity Change Assumptions & Risks Other Explanatory Factors
Reach Assumptions & Risks
External Influences
Supporting Activities and O
utputs
Direct Benefit Changes
Impact Assumptions & Risks Other Explanatory Factors
TimelineActivities and Outputs
Timeline
A Basic Theory of Change
Causal package
Contribution AnalysisUsing a generative perspective on causality, CA shows that an intervention is a contributory cause:• The expected result occurred• The ToC (causal package) is sufficient– supporting factors (assumptions) occurred and any other
supporting factors have been included in the ToC– plausible rival explanations have been accounted for
• The intervention is necessary for the ToC to be sufficient
• And can explore the role the intervention played, such as a trigger
Using Theories of Change
10.Generalizing– What works where and when?– ToC explicitly includes the context and the causal
mechanisms– Confirming a ToC provides significant evidence on
if and how the intervention would work elsewhere
Messages
• ToC are worth the effort to develop• Not a throw away, but a powerful tool to use
in evaluation and in managing• Lots of fun … and lots of debate!
ReferencesMayne, J. (forthcoming). Using Theories of Reach to Enhance Equity Considerations in
Evaluation. In Evaluation and Equity. K. Forss and M. Marra, Eds: Transaction.Mayne, J. (2013). Making Causal Claims. Connections, June. Available at
http://www.europeanevaluation.org/images/file/files/ees-newsletter-2013-06-june.pdf.
Mayne, J. (2012). Contribution Analysis: Coming of Age? Evaluation 18(3), 270-280.Mayne, J. (2012). Making Causal Claims, ILAC Brief No. 26: The Institutional Learning
and Change Initiative. Available at http://www.cgiar-ilac.org/files/publications/mayne_making_causal_claims_ilac_brief_26.pdf
Stern, E, Stame, N., Mayne, J., Forss, K, Davies, R. and Befani, B. (2012). Broadening the Range of Designs and Methods for Impact Evaluation, Working Paper 38, Department for International Development, London. Available at http://www.dfid.gov.uk/R4D/Output/189575/Default.aspx
Mayne, J. (2008). Contribution Analysis: An Approach to Exploring Cause and Effect, ILAC Brief 16. Available at http://www.cgiar-ilac.org/files/publications/briefs/ILAC_Brief16_Contribution_Analysis.pdf