capturing and communicating results in complex contributions...09.45 - 10.30 capturing results...

Capturing and Communicating

Results in Complex

Contributions

8 November 2012

09.15-12.00

Development Talks

Susanne Wadstein Director

Department for Organisational Development

Sida

Development Talks Time Item

09.15 - 09.30 Welcome (Susanne Wadstein, Sida)

09.30 - 09.45 Definition of results (Lennart Peck, Sida)

09.45 - 10.30 Capturing results (Michael Woolcock, World Bank)

10.30 - 10.40 Coffee Break

10.40 - 10.55 “What about the results” (Charlotte Örnemark, Nordic Consulting Group)

10.55 - 11.10 An Example: “Vietnam, Laos & Sri Lanka: Evaluation of long-

term development co-operation” (Annika Nordin Jayawardena, Sida)

11.10 - 12.00 Panel discussion & questions

RESULTS

OECD/DAC Definition of Results

“The output, outcome or impact …

intended or unintended…

positive and/or negative…

of a development intervention”

Results of what?

For example:

• a single project or programme

• Sweden’s cooperation with a country

• the joint efforts of partner country and donors

• a strategy or a policy

• influencing factors in the environment

Results at what point? For example:

• after X number of years

• “primary”, “secondery”, “tertiary” etc. effects

• “output”, “outcome” and “impact”

Results for whom?

For example:

• The individual citizen

• ”Women”, ”rural population”, ”the poor” etc.

• Citizens in a village, region or country

Results in terms of what?

For example:

• Products, goods and services

• Changes of individual or organizational behaviour,

attitudes, knowledge etc.

• Changes in the level of service provision,

protection of human rights etc.

• Changes in terms of individual wellbeing

An example: Results of electrification

.

Results of Results at what

point Results in terms of Results for whom

The results of the

Swedish

contribution…

…after completion… … were new

connections…

… for 2 000

households

The results of joint

donor cooperation

with government…

… in an impact

perspective…

… was access to

television, better

security and

improved business…

… for Mrs. x and her

family

The result of access

to donor funds…

… after some time… … was a reduction of

government’s own

spending…

… on citizens

needing electricity

Degrees of causation

• Necessary and sufficient

• Necessary but not sufficient

• Sufficient but not necessary

• Neither sufficient nor necessary - but contributing

Different causal patterns

One cause – One result

One cause – Multiple results

Multiple causes – One result

Multiple causes – Multiple results

Attribution – Contribution – Confusion

” to contribute to create conditions that will enable poor people to improve their lives”.

Four key questions:

• Did the intervention make a difference?

• How has the intervention made a difference?

• To what extent can a specific result be attributed

to the intervention?

• Will the intervention work elsewhere?

Some of our challenges

• Establishing attribution/contribution

• Capturing both the qualitative and the quantitative

• Aggregating results

• Generalising

• Communicating complex things in a simple way

Development Talks

Michael Woolcock Lead Social Development Specialist

The World Bank

“But How Generalizable is That?” A Framework for Assessing the Internal and External Validity of

Complex Development Interventions

Michael Woolcock World Bank and Harvard University

mwoolcock@worldbank.org

SIDA, Stockholm 8 November 2012

Theory is when you know everything but nothing works. Practice is when everything works but nobody knows why. We have put together theory and practice: nothing is working… and nobody knows why!

Albert Einstein

Overview

• Background

• From assessing IV to EV in ‘complex’ projects – Importance of trajectories, theories of change

• A framework for integrating – Design elements

– Causal density

– Implementation dynamics

– Context compatibility

• Implications for EV, and for case studies

Primary source material • Bamberger, Michael, Vijayendra Rao and Michael Woolcock (2010)

“Using Mixed Methods in Monitoring and Evaluation: Experiences from International Development”, in Abbas Tashakkori and Charles Teddlie (eds.) Handbook of Mixed Methods (2nd revised edition) Thousand Oaks, CA: Sage Publications, pp. 613-641

• Barron, Patrick, Rachael Diprose and Michael Woolcock (2011) Contesting Development: Participatory Projects and Local Conflict Dynamics in Indonesia New Haven: Yale University Press

• Woolcock, Michael (2009) ‘Toward a Plurality of Methods in Project Evaluation: A Contextualized Approach to Understanding Impact Trajectories and Efficacy’ Journal of Development Effectiveness 1(1): 1-14

• Woolcock, Michael and Arathi Rao (2012) ‘But How Generalizable is That? A Framework for Assessing the External Validity of “Complex” Development Interventions’ Mimeo

Background

• Rising obsession with ‘causality’, RCTs – Pushed by donors, foundations (e.g., Gates) – Yet also serious critique

• In medicine: Rothwell (2005) • In philosophy: Cartwright (2011) • In economics: Deaton (2010), Heckman, Ravallion…

– Especially as it pertains to EV • Engber (2011) on ‘Black 6’ (biomedical research) • Heinrich et al (2011) on ‘WEIRD’ people (social psychology) • Across time, space, groups, scale

• How to assess ‘social’, ‘participatory’ projects? – Barron, Diprose and Woolcock (2011) – Mansuri and Rao (2012) – How to compare roads, irrigation, ‘empowerment’?

22

A typology of interventions…

‘Simple’

Nets, pills,

roads

‘Complicated’

Agriculture,

microcredit

‘Complex’

Education,

health

‘Chaotic’

Local justice

reform, CDD

Theory Predictive precision

Cumulative

knowledge

Subject/object gap

High

Mechanisms # Causal pathways

# of ‘people-based’

transactions

# Feedback loops

Few

Outcomes Plausible range

Measurement

precision

Low

Many

Wide Narrow

23

…or aspects of interventions (e.g., health)

‘Simple’

Build clinics

‘Complicated’

Inoculation

(logistics)

‘Complex’

Ambulatory

care

‘Chaotic’

Adolescent

sexual

behavior

Theory Predictive precision

Cumulative

knowledge

Subject/object gap

High

Mechanisms # Causal pathways

# of ‘people-based’

transactions

# Feedback loops

Few

Outcomes Plausible range

Measurement

precision

Low

Many

Wide Narrow

Contesting Development Participatory Projects and Local

Conflict Dynamics in Indonesia

PATRICK BARRON

RACHAEL DIPROSE

MICHAEL WOOLCOCK

Yale University Press, 2011

Summary of findings

Type of Impact

Context Capacity

Low High

Program Functionality Program Functionality

Low High Low High

Direct

Forums (places) -- ++ --* 0

Facilitators (people) 0 0 0 0

Indirect

Group Relations 0 ++ + +++

Behavioral 0 +++ 0 +

Normative 0 + 0 +++

* While we noted higher rates of KDP-triggered conflict in high capacity areas, such conflict is much less likely to escalate and/or turn violent. Hence negative impacts are greater in low capacity areas, where program functionality is poor.

Lessons for evaluating ‘chaotic’ projects

In Evaluation 101, we assume… Impact = f (Design) | Selection, Confounding Variables Adequate for ‘simple’ interventions with a ‘good-enough’

counterfactual. But this is inadequate for assessing ‘complex’ interventions: * design is multi-faceted (i.e., has high ‘causal density’) * interaction with context is pervasive, desirable * implementation quality is vital (high discretion) * trajectories of change are probably non-linear (perhaps

unknowable ex ante)

Lessons for evaluating ‘chaotic’ projects

Impact = f ([DQ, CD], IE, CC) | SE, CV, RE

DQ = Design quality (weak, strong)

CD = Causal density (low, high), or ‘discretionary mechanisms’ (few, many; tight, loose; seen, unseen)

IE = Implementation effectiveness (low, high)

CC = Context compatibility (resistant, supportive)

SE = Selection effects (non-random placement, participation)

CV = Confounding variables

RE = Reasoned expectations (where by when?)

* In Social Development projects (cf. roads, immunizations):

CD is high, loose, often unseen; IE and CC are variable; RE is often unknown (unknowable?)

Pervasive problem

• SD projects are inherently very complex, thus:

– Very hard to isolate ‘true’ impact

– Very hard to make claims about likely impact elsewhere

– Understanding how (not just whether) impact is achieved is also very important

• Process Evaluations, or ‘Realist Evaluations’, can be most helpful (see work of Ray Pawson, Patricia Rogers et al)

Time

t = 0 t = 1

Net Impact

From IV to EV in complex interventions: Understanding impact trajectories

Time

t = 0 t = 1

Net Impact

Understanding impact trajectories

“Same” impact claim, but entirely a function of when the assessment was done

Time

t = 0 t = 1

Net Impact

Understanding impact trajectories

A

B C

Time

t = 0 t = 1

Net Impact

Understanding impact trajectories

A

B C

?

D

t = 2

Initial thoughts on external validity

• Logic of (elite) research and most ‘development effectiveness’ debates leads to a focus on Design – The better to identify, replicate ‘best practices’, ‘tools’

– Preferably validated via an RCT (the ‘gold standard’)

Initial thoughts on external validity

• Logic of (elite) research and most ‘development effectiveness’ debates leads to a focus on Design – The better to identify, replicate ‘best practices’, ‘tools’

– Preferably validated via an RCT (the ‘gold standard’)

• To better address IV and EV, especially of complex interventions, we need (a) an operational definition of “complexity”, (b) a pragmatic typology of Designs, Implementation and Contexts, integrated with (c) an explicit theory of change – The better to specific the conditions under which certain

outcomes are likely to be observed

– Enhancing frequency and rigor of case studies is crucial

(1) Projects Four analytical questions

Is your activity… Does producing successful outcomes from your policy….

TI: Transaction Intensive? Require many agents to act or few, over extended time periods?

LD: Locally Discretionary? Require that the implementing agents make finely based distinctions about the “state of the world”? Are these distinctions difficult for a third party to assess?

KT: Based on Known Technology?

Require that agents innovate to achieve desired outcomes ?

HS: High Stakes? Require that the agents resist large temptations to do something besides implement the policy that would produce the desired outcome?

Classification of “activities” in health

LD: Locally

Discretionary

TI:

Transaction

Intensive

HS: High

Stakes

KT: Known

‘Technology’

Iodization of

salt

No No No Yes

Vaccinations No Yes No Yes

Ambulatory

curative care

Yes Yes No(ish) Yes

Regulation of

private

providers

Yes Yes Yes Yes

Encouraging

preventive

health

Yes Yes No No

Technocratic (implementation light; policy decree)

Logistical (implementation intensive, but easy)

Implementation Intensive ‘Downstream’ (of services)

Complex (implementation intensive, motivation hard), need (continuous?) innovation

Implementation Intensive ‘Upstream’ (of obligations)

(2) Implementation How things stand…

Time

Design

Implementation

Evaluation Effort

Prestige Resources

(2) Implementation …What we need (esp. for ‘complex’ projects)

Time

Design

Implementation

Evaluation Effort

Prestige Resources

(2) Implementation

• Importance vastly underappreciated, misunderstood – Mosse (2005)… (Hard to name much else in development)

• Usually assume weak implementation is a function of agents’ skills, behaviors… – i.e., inadequate ‘training’, pervasive ‘corruption’, lack of

‘political will’, low ‘work ethic’ • Which can ‘fixed’ by engaging agents in extensive ‘capacity building’

• …but agents are employed by organizations, which are part of systems – Need to understand interactions between all three

– ‘Escaping Capability Traps’ (Andrews, Pritchett, Woolcock 2012)

‘Implementation Quality’ – Strong, Weak

(3) Contexts

• Obviously enormously varied…

• Understanding requires time, patience, analysis – political economy, anthropology, sociology etc

• Key questions, for present purposes: – Do prevailing (most administratively salient) elites

fundamentally support or resist the initiative?

– Is the project design, and the implementation apparatus(domestic and/or international), broadly perceived as legitimate?

‘Context Compatibility’ – Positive, Negative

Putting it all together

Project Design Features

Technocratic Logistical Implementation

Intensive (‘Downstream’)

Implementation Intensive

(‘Upstream’) Complex

Implementation Quality

Strong Weak Strong Weak Strong Weak Strong Weak Strong Weak

Context Compatibility

+ - + - + - + - + - + - + - + - + - + -

External Validity

High

Low

Even with low EV interventions, the ideas and processes behind them may still travel well

Putting it all together

Project Design Features

Technocratic Logistical Implementation

Intensive (‘Downstream’)

Implementation Intensive

(‘Upstream’) Complex

Implementation Quality

Strong Weak Strong Weak Strong Weak Strong Weak Strong Weak

Context Compatibility

+ - + - + - + - + - + - + - + - + - + -

External Validity

High

Low

Even with low EV interventions, the ideas and processes behind them may still travel well

Putting it all together

Project Design Features

Technocratic Logistical Implementation

Intensive (‘Downstream’)

Implementation Intensive

(‘Upstream’) Complex

Implementation Quality

Strong Weak Strong Weak Strong Weak Strong Weak Strong Weak

Context Compatibility

+ - + - + - + - + - + - + - + - + - + -

External Validity

High

Low

High

Low

Even with low EV interventions, the ideas and processes behind them may still travel well

Coffee break

10.30-10.40

Development Talks

Charlotte Örnemark Institutional development, learning and participatory M&E specialist

Nordic Consulting Group

What about the results?

Lessons from long-term process support to strengthen results-based

management (RBM) for Swedish framework NGOs operating in

Western Balkans

Charlotte Örnemark, cor@ncg.dk

”It’s all about the project logic...”

Lesson 1:

RBM in self-contained ’PCM bubbles’ tend to be fictive, responding more to donor or ’aid system needs’ than to social change processes that are embedded in contexts that by nature are chaotic and non-linear.

”It’s all about aggregation ...”

Look! A total of 529 phone calls made to government officials!!

Lesson 2:

Increased centralised demand for results can acutally lead to a decrease in overall system performance, unless accompanied by adapted approaches and investments (more of the same won’t necessarily work).

”It’s all about behavioural change ...”

Lesson 3: We may not be able to make people change,

but we can influence and track how external influence makes patterns of interactions evolve towards positive (or negative) alignment for social change. Given that...

● People influence people. ● Institutions organise people and ideas into

systems. ● Systems are often bounded, context-driven

and interact with other systems...

Social change processes Example:

Change agents (boundary partners)

Partner orgs.

Swedish FOs

Sida /UD

Inter-national commit-ments

FO

National Gov Inter-national commit-ments National reforms / poverty reduction plans and results frame-works UNDAF MDGs

Sphere of interest Sphere of influence Sphere of control Infl./interest

PO

PO

PO

PO

PO

PO

•Core org. values

•Key indicators (reflecting core values)

Country strategy / country programme

•Added value to partners

Rights-holders

Duty-bearers

Media, civil

society

Reducediscrim-ination

Women’s move-ment

Select Partner Organisation (PO) carefully based on joint values +

potential for influence

Mutual /joint accountability commitments

Mutual /joint accountability commitments

Boundary partners

Few but clearly defined / in line with FO mandate

What kind of indicators?

Lesson 4:

It’s always good to be SMART in one’s results assessments. But for interventions in the area of complex social change processes it is even more important to be REAL.

The ’REAL’ results framework for complex social change

Lesson 5: Practical implications • Invest in results frameworks where a

multitude of actors feed in meaningful information in a timely manner (used for system-wide learning, engagement)

• Use flexible/adaptive monitoring tools that can capture emerging results patterns

• ”Give the power back”: Demystify RBM • Shift from SMART to REAL • Make RBM a day-to-day concern, not an

administrative or ’control’ function

Development Talks

Annika Nordin Jayawardena Deputy Director

Department for Programme Co-operation

Sida

Lessons from 136 years of Swedish aid

Evaluation of Swedish aid to Vietnam, Laos and Sri Lanka

• Unique opportunity to analyse more than 130 years of experiences and impacts of long-

term programmes and projects

• Possibility to compare experiences from different countries with different preconditions

Select conclusions from the evaluations

• Swedish development co-operation had an impact on poor people’s lives in all three

countries, but the results varies, and in hindsight not all interventions were strategic.

• Swedish aid contributed to pulling millions of Vietnamese out of poverty.

• The contribution to poverty reduction is likely to have been less successful in Laos and

only marginal in Sri Lanka.

1 Background

The evaluators draw conclusions about long-term

Swedish development co-operation

Period: 1967-2011

• Total aid: 3.45 billion USD1

General conclusions:

• Focus on poverty was

limited, but created

preconditions for effective

future aid

• Strategic support

• Sensitive issues

(“motvindsfrågor”)

• Development co-op

facilitated relations beyond

development co-op

… … …

Period: 1958-2009

• Total aid: 1.2 billion USD1

General conclusions:

• Effective beginning to co-op

• Principled stance was

allowed because of long-

term commitment

• Good intentions but

ineffective in relation to

intended impacts on national

level

Period: 1974-2010

• Total aid: 700 million USD1

General conclusions:

• Slow start due to ineffective

aid (tied aid, limited capacity

and poor co-ordination

between donors)

• Focus on capacity

development

2 Main conclusions

1 Constant prices, 2008.

Overall:

• Only marginal impact on

poverty reduction

Overall:

• Contributed to lifting large part

of population out of poverty

Overall:

• Considerable impact on

poverty reduction

Macro economic analysis • Development and aid trends

• Assessment of changes in quality of life, particularly

income poverty

Qualitative analysis • Country specific needs & aid effectiveness principles

• Thematic & longitudinal case studies

• Interviews, focus groups, analysis of available docs

Longitudinal evaluation makes it possible to highlight

long-term development trends

• Trends identified by quantitative analysis on

macro level to give overall picture of

development in respective countries;

• Qualitative analysis of longitudinal case

studies highlight development on micro level;

• Subsequent analysis of whether observations on

macro level are supported by observations on

micro level and vice versa.

Evaluation method

Phase out

Mic

ro

Ma

cro

Phase in

Validation

3 Methodological overview

Development co-operation with Vietnam: From paper

mill to local democracy

Bai Bang Doi Moi Chia Se

Phase I: 1969-85 Phase II: 1986-99 Phase III: 2000-11

• Largest Swedish aid

project ever

– Did not meet acute needs

– Mixed evidence for

effective aid

– Today privately run paper

mill

• Wider understanding for

sustainable use of natural

resources

• Created pre-conditions for

continued development co-

operation

• Sweden only donor with

good relations

– Could support economic

reform in initial phases

• Swedish economic model

used as tool

– Several examples of

economic models given

– Policy dialogue, education

& technical support

• Experimental & long-term

co-operation for local

democracy

– Last programme in

succession of projects/

programmes in same area

as Bai Bang

– Has affected national

development plans with

rights-based approach,

incl. women’s rights

4 Case study - Vietnam

Experiences and lessons learned from Swedish

development co-operation

• Strike a balance between principles and pragmatism.

• Understand underlying causes of aid impacts and be flexible. Drivers of change

and actors.

5 Experiences & lessons learned

• Long-term approaches build trust and relations that allow for engagement in

sensitive issues.

• Sustainable results require development of competences, capacity and

institutions.

Principles &

pragmatism

1

Long-term

approach &

presence

2

Sustainability

4

Learning &

flexibility

3

Development Talks Panelists Organisation

Michael Woolcock The World Bank

Charlotte Örnemark Nordic Consulting Group

Janet Vähämäki

Pontus Modéer Rightshouse

Angelica Broman Sida

Martina Fors Sida

Moderator

Lennart Peck Sida

Thank you!