capturing and communicating results in complex contributions...09.45 - 10.30 capturing results...
TRANSCRIPT
Capturing and Communicating
Results in Complex
Contributions
8 November 2012
09.15-12.00
Development Talks
Susanne Wadstein Director
Department for Organisational Development
Sida
Development Talks Time Item
09.15 - 09.30 Welcome (Susanne Wadstein, Sida)
09.30 - 09.45 Definition of results (Lennart Peck, Sida)
09.45 - 10.30 Capturing results (Michael Woolcock, World Bank)
10.30 - 10.40 Coffee Break
10.40 - 10.55 “What about the results” (Charlotte Örnemark, Nordic Consulting Group)
10.55 - 11.10 An Example: “Vietnam, Laos & Sri Lanka: Evaluation of long-
term development co-operation” (Annika Nordin Jayawardena, Sida)
11.10 - 12.00 Panel discussion & questions
RESULTS
OECD/DAC Definition of Results
“The output, outcome or impact …
intended or unintended…
positive and/or negative…
of a development intervention”
Results of what?
For example:
• a single project or programme
• Sweden’s cooperation with a country
• the joint efforts of partner country and donors
• a strategy or a policy
• influencing factors in the environment
Results at what point? For example:
• after X number of years
• “primary”, “secondery”, “tertiary” etc. effects
• “output”, “outcome” and “impact”
Results for whom?
For example:
• The individual citizen
• ”Women”, ”rural population”, ”the poor” etc.
• Citizens in a village, region or country
Results in terms of what?
For example:
• Products, goods and services
• Changes of individual or organizational behaviour,
attitudes, knowledge etc.
• Changes in the level of service provision,
protection of human rights etc.
• Changes in terms of individual wellbeing
An example: Results of electrification
.
Results of Results at what
point Results in terms of Results for whom
The results of the
Swedish
contribution…
…after completion… … were new
connections…
… for 2 000
households
The results of joint
donor cooperation
with government…
… in an impact
perspective…
… was access to
television, better
security and
improved business…
… for Mrs. x and her
family
The result of access
to donor funds…
… after some time… … was a reduction of
government’s own
spending…
… on citizens
needing electricity
Degrees of causation
• Necessary and sufficient
• Necessary but not sufficient
• Sufficient but not necessary
• Neither sufficient nor necessary - but contributing
Different causal patterns
One cause – One result
One cause – Multiple results
Multiple causes – One result
Multiple causes – Multiple results
Attribution – Contribution – Confusion
” to contribute to create conditions that will enable poor people to improve their lives”.
Four key questions:
• Did the intervention make a difference?
• How has the intervention made a difference?
• To what extent can a specific result be attributed
to the intervention?
• Will the intervention work elsewhere?
Some of our challenges
• Establishing attribution/contribution
• Capturing both the qualitative and the quantitative
• Aggregating results
• Generalising
• Communicating complex things in a simple way
Development Talks
Michael Woolcock Lead Social Development Specialist
The World Bank
“But How Generalizable is That?” A Framework for Assessing the Internal and External Validity of
Complex Development Interventions
Michael Woolcock World Bank and Harvard University
SIDA, Stockholm 8 November 2012
Theory is when you know everything but nothing works. Practice is when everything works but nobody knows why. We have put together theory and practice: nothing is working… and nobody knows why!
Albert Einstein
Overview
• Background
• From assessing IV to EV in ‘complex’ projects – Importance of trajectories, theories of change
• A framework for integrating – Design elements
– Causal density
– Implementation dynamics
– Context compatibility
• Implications for EV, and for case studies
Primary source material • Bamberger, Michael, Vijayendra Rao and Michael Woolcock (2010)
“Using Mixed Methods in Monitoring and Evaluation: Experiences from International Development”, in Abbas Tashakkori and Charles Teddlie (eds.) Handbook of Mixed Methods (2nd revised edition) Thousand Oaks, CA: Sage Publications, pp. 613-641
• Barron, Patrick, Rachael Diprose and Michael Woolcock (2011) Contesting Development: Participatory Projects and Local Conflict Dynamics in Indonesia New Haven: Yale University Press
• Woolcock, Michael (2009) ‘Toward a Plurality of Methods in Project Evaluation: A Contextualized Approach to Understanding Impact Trajectories and Efficacy’ Journal of Development Effectiveness 1(1): 1-14
• Woolcock, Michael and Arathi Rao (2012) ‘But How Generalizable is That? A Framework for Assessing the External Validity of “Complex” Development Interventions’ Mimeo
Background
• Rising obsession with ‘causality’, RCTs – Pushed by donors, foundations (e.g., Gates) – Yet also serious critique
• In medicine: Rothwell (2005) • In philosophy: Cartwright (2011) • In economics: Deaton (2010), Heckman, Ravallion…
– Especially as it pertains to EV • Engber (2011) on ‘Black 6’ (biomedical research) • Heinrich et al (2011) on ‘WEIRD’ people (social psychology) • Across time, space, groups, scale
• How to assess ‘social’, ‘participatory’ projects? – Barron, Diprose and Woolcock (2011) – Mansuri and Rao (2012) – How to compare roads, irrigation, ‘empowerment’?
22
A typology of interventions…
‘Simple’
Nets, pills,
roads
‘Complicated’
Agriculture,
microcredit
‘Complex’
Education,
health
‘Chaotic’
Local justice
reform, CDD
Theory Predictive precision
Cumulative
knowledge
Subject/object gap
High
Mechanisms # Causal pathways
# of ‘people-based’
transactions
# Feedback loops
Few
Outcomes Plausible range
Measurement
precision
Low
Many
Wide Narrow
23
…or aspects of interventions (e.g., health)
‘Simple’
Build clinics
‘Complicated’
Inoculation
(logistics)
‘Complex’
Ambulatory
care
‘Chaotic’
Adolescent
sexual
behavior
Theory Predictive precision
Cumulative
knowledge
Subject/object gap
High
Mechanisms # Causal pathways
# of ‘people-based’
transactions
# Feedback loops
Few
Outcomes Plausible range
Measurement
precision
Low
Many
Wide Narrow
Contesting Development Participatory Projects and Local
Conflict Dynamics in Indonesia
PATRICK BARRON
RACHAEL DIPROSE
MICHAEL WOOLCOCK
Yale University Press, 2011
Summary of findings
Type of Impact
Context Capacity
Low High
Program Functionality Program Functionality
Low High Low High
Direct
Forums (places) -- ++ --* 0
Facilitators (people) 0 0 0 0
Indirect
Group Relations 0 ++ + +++
Behavioral 0 +++ 0 +
Normative 0 + 0 +++
* While we noted higher rates of KDP-triggered conflict in high capacity areas, such conflict is much less likely to escalate and/or turn violent. Hence negative impacts are greater in low capacity areas, where program functionality is poor.
Lessons for evaluating ‘chaotic’ projects
In Evaluation 101, we assume… Impact = f (Design) | Selection, Confounding Variables Adequate for ‘simple’ interventions with a ‘good-enough’
counterfactual. But this is inadequate for assessing ‘complex’ interventions: * design is multi-faceted (i.e., has high ‘causal density’) * interaction with context is pervasive, desirable * implementation quality is vital (high discretion) * trajectories of change are probably non-linear (perhaps
unknowable ex ante)
Lessons for evaluating ‘chaotic’ projects
Impact = f ([DQ, CD], IE, CC) | SE, CV, RE
DQ = Design quality (weak, strong)
CD = Causal density (low, high), or ‘discretionary mechanisms’ (few, many; tight, loose; seen, unseen)
IE = Implementation effectiveness (low, high)
CC = Context compatibility (resistant, supportive)
SE = Selection effects (non-random placement, participation)
CV = Confounding variables
RE = Reasoned expectations (where by when?)
* In Social Development projects (cf. roads, immunizations):
CD is high, loose, often unseen; IE and CC are variable; RE is often unknown (unknowable?)
Pervasive problem
• SD projects are inherently very complex, thus:
– Very hard to isolate ‘true’ impact
– Very hard to make claims about likely impact elsewhere
– Understanding how (not just whether) impact is achieved is also very important
• Process Evaluations, or ‘Realist Evaluations’, can be most helpful (see work of Ray Pawson, Patricia Rogers et al)
Time
t = 0 t = 1
Net Impact
From IV to EV in complex interventions: Understanding impact trajectories
Time
t = 0 t = 1
Net Impact
Understanding impact trajectories
“Same” impact claim, but entirely a function of when the assessment was done
Time
t = 0 t = 1
Net Impact
Understanding impact trajectories
A
B C
Time
t = 0 t = 1
Net Impact
Understanding impact trajectories
A
B C
?
D
t = 2
Initial thoughts on external validity
• Logic of (elite) research and most ‘development effectiveness’ debates leads to a focus on Design – The better to identify, replicate ‘best practices’, ‘tools’
– Preferably validated via an RCT (the ‘gold standard’)
Initial thoughts on external validity
• Logic of (elite) research and most ‘development effectiveness’ debates leads to a focus on Design – The better to identify, replicate ‘best practices’, ‘tools’
– Preferably validated via an RCT (the ‘gold standard’)
• To better address IV and EV, especially of complex interventions, we need (a) an operational definition of “complexity”, (b) a pragmatic typology of Designs, Implementation and Contexts, integrated with (c) an explicit theory of change – The better to specific the conditions under which certain
outcomes are likely to be observed
– Enhancing frequency and rigor of case studies is crucial
(1) Projects Four analytical questions
Is your activity… Does producing successful outcomes from your policy….
TI: Transaction Intensive? Require many agents to act or few, over extended time periods?
LD: Locally Discretionary? Require that the implementing agents make finely based distinctions about the “state of the world”? Are these distinctions difficult for a third party to assess?
KT: Based on Known Technology?
Require that agents innovate to achieve desired outcomes ?
HS: High Stakes? Require that the agents resist large temptations to do something besides implement the policy that would produce the desired outcome?
Classification of “activities” in health
LD: Locally
Discretionary
TI:
Transaction
Intensive
HS: High
Stakes
KT: Known
‘Technology’
Iodization of
salt
No No No Yes
Vaccinations No Yes No Yes
Ambulatory
curative care
Yes Yes No(ish) Yes
Regulation of
private
providers
Yes Yes Yes Yes
Encouraging
preventive
health
Yes Yes No No
Technocratic (implementation light; policy decree)
Logistical (implementation intensive, but easy)
Implementation Intensive ‘Downstream’ (of services)
Complex (implementation intensive, motivation hard), need (continuous?) innovation
Implementation Intensive ‘Upstream’ (of obligations)
(2) Implementation How things stand…
Time
Design
Implementation
Evaluation Effort
Prestige Resources
(2) Implementation …What we need (esp. for ‘complex’ projects)
Time
Design
Implementation
Evaluation Effort
Prestige Resources
(2) Implementation
• Importance vastly underappreciated, misunderstood – Mosse (2005)… (Hard to name much else in development)
• Usually assume weak implementation is a function of agents’ skills, behaviors… – i.e., inadequate ‘training’, pervasive ‘corruption’, lack of
‘political will’, low ‘work ethic’ • Which can ‘fixed’ by engaging agents in extensive ‘capacity building’
• …but agents are employed by organizations, which are part of systems – Need to understand interactions between all three
– ‘Escaping Capability Traps’ (Andrews, Pritchett, Woolcock 2012)
‘Implementation Quality’ – Strong, Weak
(3) Contexts
• Obviously enormously varied…
• Understanding requires time, patience, analysis – political economy, anthropology, sociology etc
• Key questions, for present purposes: – Do prevailing (most administratively salient) elites
fundamentally support or resist the initiative?
– Is the project design, and the implementation apparatus(domestic and/or international), broadly perceived as legitimate?
‘Context Compatibility’ – Positive, Negative
Putting it all together
Project Design Features
Technocratic Logistical Implementation
Intensive (‘Downstream’)
Implementation Intensive
(‘Upstream’) Complex
Implementation Quality
Strong Weak Strong Weak Strong Weak Strong Weak Strong Weak
Context Compatibility
+ - + - + - + - + - + - + - + - + - + -
External Validity
High
Low
Even with low EV interventions, the ideas and processes behind them may still travel well
Putting it all together
Project Design Features
Technocratic Logistical Implementation
Intensive (‘Downstream’)
Implementation Intensive
(‘Upstream’) Complex
Implementation Quality
Strong Weak Strong Weak Strong Weak Strong Weak Strong Weak
Context Compatibility
+ - + - + - + - + - + - + - + - + - + -
External Validity
High
Low
Even with low EV interventions, the ideas and processes behind them may still travel well
Putting it all together
Project Design Features
Technocratic Logistical Implementation
Intensive (‘Downstream’)
Implementation Intensive
(‘Upstream’) Complex
Implementation Quality
Strong Weak Strong Weak Strong Weak Strong Weak Strong Weak
Context Compatibility
+ - + - + - + - + - + - + - + - + - + -
External Validity
High
Low
High
Low
Even with low EV interventions, the ideas and processes behind them may still travel well
Development Talks
Charlotte Örnemark Institutional development, learning and participatory M&E specialist
Nordic Consulting Group
What about the results?
Lessons from long-term process support to strengthen results-based
management (RBM) for Swedish framework NGOs operating in
Western Balkans
Charlotte Örnemark, [email protected]
”It’s all about the project logic...”
Lesson 1:
RBM in self-contained ’PCM bubbles’ tend to be fictive, responding more to donor or ’aid system needs’ than to social change processes that are embedded in contexts that by nature are chaotic and non-linear.
”It’s all about aggregation ...”
Look! A total of 529 phone calls made to government officials!!
Lesson 2:
Increased centralised demand for results can acutally lead to a decrease in overall system performance, unless accompanied by adapted approaches and investments (more of the same won’t necessarily work).
”It’s all about behavioural change ...”
Lesson 3: We may not be able to make people change,
but we can influence and track how external influence makes patterns of interactions evolve towards positive (or negative) alignment for social change. Given that...
● People influence people. ● Institutions organise people and ideas into
systems. ● Systems are often bounded, context-driven
and interact with other systems...
Social change processes Example:
Change agents (boundary partners)
Partner orgs.
Swedish FOs
Sida /UD
Inter-national commit-ments
FO
National Gov Inter-national commit-ments National reforms / poverty reduction plans and results frame-works UNDAF MDGs
Sphere of interest Sphere of influence Sphere of control Infl./interest
PO
PO
PO
PO
PO
PO
•Core org. values
•Key indicators (reflecting core values)
Country strategy / country programme
•Added value to partners
Rights-holders
Duty-bearers
Media, civil
society
Reducediscrim-ination
Women’s move-ment
Select Partner Organisation (PO) carefully based on joint values +
potential for influence
Mutual /joint accountability commitments
Mutual /joint accountability commitments
Boundary partners
Few but clearly defined / in line with FO mandate
What kind of indicators?
Lesson 4:
It’s always good to be SMART in one’s results assessments. But for interventions in the area of complex social change processes it is even more important to be REAL.
The ’REAL’ results framework for complex social change
Lesson 5: Practical implications • Invest in results frameworks where a
multitude of actors feed in meaningful information in a timely manner (used for system-wide learning, engagement)
• Use flexible/adaptive monitoring tools that can capture emerging results patterns
• ”Give the power back”: Demystify RBM • Shift from SMART to REAL • Make RBM a day-to-day concern, not an
administrative or ’control’ function
Development Talks
Annika Nordin Jayawardena Deputy Director
Department for Programme Co-operation
Sida
Lessons from 136 years of Swedish aid
Evaluation of Swedish aid to Vietnam, Laos and Sri Lanka
• Unique opportunity to analyse more than 130 years of experiences and impacts of long-
term programmes and projects
• Possibility to compare experiences from different countries with different preconditions
Select conclusions from the evaluations
• Swedish development co-operation had an impact on poor people’s lives in all three
countries, but the results varies, and in hindsight not all interventions were strategic.
• Swedish aid contributed to pulling millions of Vietnamese out of poverty.
• The contribution to poverty reduction is likely to have been less successful in Laos and
only marginal in Sri Lanka.
1 Background
The evaluators draw conclusions about long-term
Swedish development co-operation
Period: 1967-2011
• Total aid: 3.45 billion USD1
General conclusions:
• Focus on poverty was
limited, but created
preconditions for effective
future aid
• Strategic support
• Sensitive issues
(“motvindsfrågor”)
• Development co-op
facilitated relations beyond
development co-op
… … …
Period: 1958-2009
• Total aid: 1.2 billion USD1
General conclusions:
• Effective beginning to co-op
• Principled stance was
allowed because of long-
term commitment
• Good intentions but
ineffective in relation to
intended impacts on national
level
Period: 1974-2010
• Total aid: 700 million USD1
General conclusions:
• Slow start due to ineffective
aid (tied aid, limited capacity
and poor co-ordination
between donors)
• Focus on capacity
development
2 Main conclusions
1 Constant prices, 2008.
Overall:
• Only marginal impact on
poverty reduction
Overall:
• Contributed to lifting large part
of population out of poverty
Overall:
• Considerable impact on
poverty reduction
Macro economic analysis • Development and aid trends
• Assessment of changes in quality of life, particularly
income poverty
Qualitative analysis • Country specific needs & aid effectiveness principles
• Thematic & longitudinal case studies
• Interviews, focus groups, analysis of available docs
Longitudinal evaluation makes it possible to highlight
long-term development trends
• Trends identified by quantitative analysis on
macro level to give overall picture of
development in respective countries;
• Qualitative analysis of longitudinal case
studies highlight development on micro level;
• Subsequent analysis of whether observations on
macro level are supported by observations on
micro level and vice versa.
Evaluation method
Phase out
Mic
ro
Ma
cro
Phase in
Validation
3 Methodological overview
Development co-operation with Vietnam: From paper
mill to local democracy
Bai Bang Doi Moi Chia Se
Phase I: 1969-85 Phase II: 1986-99 Phase III: 2000-11
• Largest Swedish aid
project ever
– Did not meet acute needs
– Mixed evidence for
effective aid
– Today privately run paper
mill
• Wider understanding for
sustainable use of natural
resources
• Created pre-conditions for
continued development co-
operation
• Sweden only donor with
good relations
– Could support economic
reform in initial phases
• Swedish economic model
used as tool
– Several examples of
economic models given
– Policy dialogue, education
& technical support
• Experimental & long-term
co-operation for local
democracy
– Last programme in
succession of projects/
programmes in same area
as Bai Bang
– Has affected national
development plans with
rights-based approach,
incl. women’s rights
4 Case study - Vietnam
Experiences and lessons learned from Swedish
development co-operation
• Strike a balance between principles and pragmatism.
• Understand underlying causes of aid impacts and be flexible. Drivers of change
and actors.
5 Experiences & lessons learned
• Long-term approaches build trust and relations that allow for engagement in
sensitive issues.
• Sustainable results require development of competences, capacity and
institutions.
Principles &
pragmatism
1
Long-term
approach &
presence
2
Sustainability
4
Learning &
flexibility
3
Development Talks Panelists Organisation
Michael Woolcock The World Bank
Charlotte Örnemark Nordic Consulting Group
Janet Vähämäki
Pontus Modéer Rightshouse
Angelica Broman Sida
Martina Fors Sida
Moderator
Lennart Peck Sida
