outcome mapping presentation for donor coordination meeting
DESCRIPTION
Civil society coordination meeting Sarajevo, 6.12.2007TRANSCRIPT
Outcome Mapping
Civil society coordination meeting Sarajevo, 6.12.2007
Steve Powell & Ivona ČelebičićproMENTE social research, Sarajevo
an evaluation of six civil society projects in BiH
www.promente.org/om
Contents
TOR of our evaluation project What is OM? Findings Conclusions Recommendations Discussion
www.promente.org/om
Who are we
Steve Powell & Ivona ČelebičićproMENTE social research, Sarajevo
Sources for OM: International Development Research Centre www.idrc.ca
See: www.promente.org/om
www.promente.org/om
Evaluation TOR
1. Explorative evaluation of six civil society projects
2. Sida-funded civil society programming: lessons on sustainability
3. Exploration of OM as a tool
www.promente.org/om
Our project in B&H: 1 donor, 3 framework partners, 6 implementing partners, 6 projects: 2007Framework organisation
Partner organisation Evaluated activity
Kvinna til Kvinna
Zenski Centar Womens political lobby
Most Village activities (including round tables in towns)
Olof Palme International Center
Helsinki Citizens Assembly (HCA) – Academy for political leaders)
Academy for political leaders
Civil Society Promotion Center – GROZD, "Citizen in action"
"Citizen in action"– Project of community-based advocacy campaigns for solving priority citizens problems from "Civic Platform for 2006 Elections". "Local Government leadership building activities"
Swedish Helsinki Committee for Human Rights
BiH Press Council (funded by SHC)
Work with judges and journalists on Press Code
Vasa Prava "Improvement of access to justice – Raising public awareness on access to rights, legislative changes and legal procedures in BiH
www.promente.org/om
What is Outcome Mapping (OM)?
An approach to project planning, monitoring and evaluation A new, soft alternative to replace or complement LFA A new paradigm/way of thinking. A backlash?
www.promente.org/om
Problems with RBMResults Based Management (RBM) / Logframe (LFA)
OM
Force implementing orgs to try to demonstrate that they caused numerically large impacts
Focus on development/change of key partners; outcomes, not impacts
Focus too much on impact in areas “where their influence … is low and decreasing relative to that of other actors”*
Quality, not quantity of change
Inflexible Flexible
NGOs tend to see it as alien Fits better with what NGOs feel they are doing: stimulating change, not delivering outputs
Attribution (did they really cause the change?) Contribution (what did they do, what worked?)
Mechanistic approach to strategy Strategy maps inspire thinking about different dimensions of planning
Provide only “clueless feedback” Rich, useful feedback
Exclusive focus on impact/results Focus on: strategy, learning as well as outcomes
www.promente.org/om
Vision
Logframe vs. OM
Goal
Objectives
Outcomes
Outputs Implementing agency: strategy
Boundary partners.changes in attitudes, behaviour, relationships
Other actors?
?
www.promente.org/om
Boundary partners ≠ beneficiaries! – (might be, do not have to be)
There can be more than one group of boundary partners Development work is really about leveraging the influence of a
limited number of key partners … … focus is NOT on the stakeholders with whom the boundary
partners interact changing consciousness, ways of working … … in service of a vision
Key outcomes: changes in attitudes, behaviour, relationships in boundary partner groups
www.promente.org/om
VisionThe partners become part of a new, younger, gender-equal generation in politics, who through their influence in their political parties initiate a change in the direction of everyday political issues – away from national/ethnic issues towards themes which respect democracy and human rights including gender rights and which will contribute to social prosperity in the long term. …. … …. …
Example
Implementing agency: Helsinki Citizen’s Assembly
Boundary partners: young politiciansOutcome challenge: Empowered young politicians
influence the change of the everyday politics in their parties. Recognized need for improvement in the areas selected in the party
modules, continuous activities on the empowerment of youth engaged in politics.
Other actors: political parties, other young politicians …
www.promente.org/om
Intentional design Outcome & performance monitoring
Evaluation
1. Vision
2. Mission
3. ...Boundary partners
4. ...Outcome challenges
5. ...Progress markers 9. Outcome journals
6. …Strategy maps 10. Strategy journal
7. Organisational practices 11. Performance journal
8. Monitoring priorities 12. Evaluation plan
OM: 12 steps
Helping the implementing partner to learn
Internal M &E
Considering all the dimensions of strategy
Focus on outcomes
www.promente.org/om
VisionProgress marker ladders Outcome challenge 1
Love to see
Like to see
Expect to see
Outcome challenge 2
Boundary partner 1
Boundary partner 2
www.promente.org/om
Method: timeframeevaluation time-frame captures only a small slice of project implementation
Framework partner support: several years
Implementing partner project: several years
Duration of project activities being evaluated: Jan-Dec 2007
Research time-frame: May-Nov 2007
www.promente.org/om
Method: OM Special use of OM for external evaluation
1. Mission, vision, progress markers, outcome challenge2. Assessment of planning strategies using strategy maps3. OM questionnaires4. OM interview with boundary partners5. OM interview with implementing partners6. Writing evaluation-start Outcome Journals 7. Confirming evaluation-start Outcome Journals8. OM questionnaires (same as evaluation-start)9. OM interview with boundary partners (on the basis of evaluation-start interview)10. OM interview with implementing partners (on the basis of evaluation-start interview)11. Assessment of management progress using existing strategy maps12. Mission and vision: still relevant? Changed focus?13. Gathering contextual information and intervention timeline
Baseline
interimfinal assessm
ent
www.promente.org/om
Findings: OM
Most organisations succeeded in redefining changes in a small group of boundary partners as the main component of their vision (and )
All made progress towards their vision (A lot of information about individual
projects)
www.promente.org/om
Method: relevance interviews
Aim: to assess project and program relevance Sources:
– 50 additional interviews at baseline: stakeholder views of poverty, gender and civil society
– + 50 at final assessment: project relevance. Analysis: qualitative methods, specialised software
-> stakeholders’ own themes.
www.promente.org/om
Findings: relevance interviews: Gender
Strong opinions about gender equality– There is no real gender equality – Women are capable of taking leading roles in society– Women are not really interested in change (especially female respondents).
0
0.2
0.4
0.6
0.8
1
1.2
1.4
primary secondary tertiary
tradition causes genderinequality
www.promente.org/om
tabletype of partnergender project education age = <30 area
boun
dary
impl
emen
ting
fem
ale
mal
e
proj
ect =
HC
-VP
proj
ect =
HC
-PC
proj
ect =
KK
-MV
proj
ect =
KK
-ZC
proj
ect =
OP
-CP
proj
ect =
OP
-HB
prim
ary
or le
ss
finis
hed
seco
ndar
y
beyo
nd s
econ
dary
<30
30-
50
51-
65
>66
urba
n
rura
l
total respondents 34 10 29 15 6 8 10 2 6 2 15 10 19 10 25 7 2 18 26
total references 267 89 237 125 125 125 99 37 72 24 95 99 169 95 222 44 7 160 195references/respondent 7.9 8.9 8.2 8.3 20.8 15.6 9.9 18.5 12.0 12.0 6.3 9.9 8.9 9.5 8.9 6.3 3.5 8.9 7.5
gender total references per themethere is no gender equality 36 0.82 0.79 0.90 0.86 0.87 2.17 1.63 1.00 2.00 1.33 2.00 0.87 1.20 0.68 1.00 0.92 0.57 0.50 0.67 0.92poverty, low education cause inequality 11 0.25 0.24 0.30 0.34 0.07 0.17 0.13 0.20 1.00 0.17 0.40 0.26 0.20 0.36 0.22 0.27tradition causes inequality 33 0.75 0.65 1.10 0.79 0.67 1.67 1.25 0.90 1.50 0.33 1.00 0.33 0.40 1.26 0.80 0.92 0.57 1.00 0.58women are capable of leading 28 0.64 0.53 1.00 0.69 0.60 1.50 1.13 0.50 1.50 1.17 0.73 0.50 0.68 1.00 0.64 0.14 1.00 0.61 0.65women are not interested in change 21 0.48 0.50 0.40 0.59 0.27 0.67 0.50 1.30 1.00 0.33 1.00 0.67 0.70 0.21 0.40 0.64 0.50 0.33 0.58bad strategy in fight for equality 5 0.11 0.15 0.14 0.07 0.17 0.13 0.10 0.50 0.10 0.21 0.16 0.14 0.22 0.04
povertyvictims of poverty
pensioners are hit by poverty 5 0.11 0.12 0.10 0.07 0.27 0.67 0.50 0.67 0.13 0.40 0.05 0.20 0.12 0.29 0.06 0.15young people are hit by poverty 7 0.16 0.18 0.10 0.17 0.20 0.50 0.38 0.20 0.33 1.00 0.20 0.50 0.30 0.08 0.29 0.50 0.11 0.19
reasons for povertysystem does not care for vulnerable 9 0.20 0.18 0.30 0.14 0.33 0.83 0.63 0.10 0.50 0.33 0.50 0.07 0.60 0.11 0.40 0.08 0.43 0.17 0.23poverty is due to the war 11 0.25 0.26 0.20 0.21 0.33 0.83 0.63 0.30 1.00 0.13 0.20 0.37 0.10 0.28 0.43 0.39 0.15poverty is due to unemployment 15 0.34 0.35 0.30 0.41 0.20 0.50 0.38 0.40 1.50 0.33 0.33 0.20 0.42 0.30 0.44 0.14 0.28 0.38
forms of povertythere is much material poverty 26 0.59 0.59 0.60 0.52 0.73 1.83 1.38 0.60 0.50 0.83 1.50 0.53 0.80 0.53 0.60 0.60 0.71 0.44 0.69there is much mental poverty 11 0.25 0.18 0.50 0.24 0.27 0.67 0.50 0.30 0.33 0.50 0.30 0.42 0.50 0.20 0.29 0.56 0.04
possibility of changepolitical leadership
corruption is everywhere 11 0.25 0.29 0.10 0.07 0.60 1.50 1.13 0.10 0.50 1.00 0.50 0.07 0.40 0.32 0.50 0.20 0.50 0.17 0.31the wrong people are in power 5 0.11 0.15 0.10 0.13 0.33 0.25 0.10 0.17 0.50 0.13 0.10 0.11 0.10 0.16 0.17 0.08politics is all wrong 11 0.25 0.32 0.21 0.33 0.83 0.63 0.30 0.50 0.50 0.13 0.60 0.16 0.10 0.28 0.29 0.50 0.33 0.19we need ~strong~ leaders 2 0.05 0.06 0.13 0.33 0.25 0.33 0.20 0.20 0.11
Civil societyindividuals cannot do anything alone 15 0.34 0.38 0.20 0.28 0.53 1.33 1.00 0.40 1.00 1.17 0.50 0.33 0.50 0.32 0.50 0.40 0.14 0.22 0.42there is no ~civic consciousness~ 10 0.23 0.26 0.10 0.10 0.47 1.17 0.88 0.20 0.50 0.50 2.00 0.60 0.21 0.60 0.08 0.29 0.39 0.12change possible if we work together 19 0.43 0.41 0.50 0.52 0.33 0.83 0.63 0.80 1.50 0.67 0.53 0.30 0.47 0.50 0.56 0.29 0.33 0.50individuals can help change society 19 0.43 0.29 0.90 0.66 0.70 1.50 0.27 0.40 0.58 0.40 0.52 0.29 0.67 0.27change is possible 23 0.52 0.44 0.80 0.59 0.33 0.83 0.63 0.60 1.50 0.67 0.50 0.20 0.30 0.84 0.20 0.68 0.43 0.67 0.38change is impossible 7 0.16 0.21 0.21 0.07 0.17 0.13 0.60 0.17 0.27 0.20 0.05 0.20 0.29 0.11 0.19citizens are not interested 5 0.11 0.12 0.10 0.07 0.20 0.50 0.38 0.10 0.17 0.50 0.30 0.11 0.20 0.08 0.14 0.17 0.08civil society is the people 11 0.25 0.21 0.40 0.21 0.33 0.83 0.63 0.10 0.50 0.50 0.10 0.53 0.40 0.28 0.14 0.50 0.08
references per theme per person
www.promente.org/om
Findings: relevance interviews: poverty and change
About one respondent in four brought up the concept of "spiritual" or "intellectual" poverty … Most very sceptical about the possibility of change. Males very frequently mentioned the problem of corruption. Implementing partners much more optimistic
There is no end to it. Here, people are hanging and killing themselves with their own hands. In my village, fifteen days ago, right beneath my window, a man hung himself... He was single. He might have had some small pension, just a little bit... and his health was pretty bad ... that’s what I think.
0
0.2
0.4
0.6
0.8
1
boundary implementing
individuals can change society
www.promente.org/om
Assistance?
A good part of the population is used to some sort of humanitarian aid, some sort of social help, mercy, call it what you will. This means that nobody has to work and, at the same time, they get something. We can set our sights lower but we don’t need to invest anything.
www.promente.org/om
Method: quantitative analysis
For four projects, a customised questionnaire was made on the basis of progress markers.
Questionnaire filled in at baseline and again at final assessment
Results from around 100 baseline and 100 final assessment questionnaires compared on a per-project basis.
www.promente.org/om
Findings: quantitative analysis
Results for one project
Small but significant overall improvement
Women start lower than men and improve more than men
In spite of a possible tendency to “raise the bar” during the project
0
0.5
1
1.5
2
2.5
3
m ž
pre
post
male female
www.promente.org/om
Improvement up the progress marker ladder
at which points of the progress marker ladder do which boundary partners improve?
Not only do women improve more than men but they improve strongly at the upper end of the scale
Progress marker ladder
improvement
female
male
www.promente.org/om
Conclusions: programming
Sida support helps develop civil society and CS activists Vision is not always about boundary partner change Civil society not a mass movement! Small number of partners; handful of key “career” activists Poverty
– Less clear progress with the least educated– Not clear that civil society development has reduction of poverty as a direct outcome.– Most successful projects do not connect directly to poverty– Projects with a stronger connection to poverty have less of a development perspective– Projects are not directly relevant to the really poor; different language; illiteracy– For the poor, not with the poor
Gender equality – is perceived as lacking– is not perceived to be a primary need in rural populations
In CS projects, women may have benefited more than men
www.promente.org/om
Recommendations: programming
Think missing links! Think hard about motivation of “missing links”
– Differential skills are needed (talking to parliament, then villagers)– HR for activists? Business support for NGOs?
Need to demonstrate link between civil society and poverty reduction??
Pay more attention to how the poor conceive poverty
www.promente.org/om
Conclusions: OM
“Thinking OM”, e.g. about different groups of boundary partners, makes a dramatic difference
NGOs enthusiastic!? OM for monitoring -> OM for planning Focus on contribution is a big relief Gives richer feedback Strategy maps inspire thinking about different dimensions of planning Separate quantitative analysis validates OM: both show progress
www.promente.org/om
Reflection: criticisms of RBM for CS programming
RBM: killing the birds of planning, control, monitoring and evaluation with one stone? Data and influence flow upwards.– Planning: Inspires theories of change which are inaccurate and
optimistically mechanistic. Blind to the groups and individuals who implement change.
– Control: gathering data via a chain in which every link has a vested interest in lying
– Monitoring: ticking boxes at the expense of strategic thinking and organisational learning
– Evaluation: Can you calculate impacts by aggregating outputs? Are NGOs competent to measure outcome and impact directly? Is it their job?
Impacts …
outcomes …
outputs …Program
Project BProject A
www.promente.org/om
Recommendations: OM
Better planning: OM could/should be adopted by donors from the project application stage (or as a hybrid).
Better control: release NGOs from exhausting box-ticking and implement fair random checking of outputs.
Better monitoring: in exchange, require elements of strategic planning and organisational learning.
Better evaluation: – Demonstrate donor-relevant results: commission independent, external
investigations of society change and why it happens/happened– Sharpen the focus: help NGOs to do empirical explorations of a selection of
specific issues that really interest them