outcomemapping.ca monitoring in outcome mapping principles, design & practice steff deprez &...

outcomemapping.ca

Monitoring in Outcome Mapping Principles, Design & Practice

Steff Deprez & Kaia Ambrose

OM Lab Dar Es Salaam, Tanzania 22-23 Sept, 2014

Monitoring in Outcome Mapping1. Core principles of monitoring in Outcome Mapping

2. Monitoring Design Issues Experiences from practice

3. Monitoring Practice Approaches, tools & instruments

Steff Deprez & Kaia Ambrose (Sept 2014) 2

outcomemapping.ca

1. Core principles of monitoring in Outcome MappingBased on OM Manual

Outcome Mapping Monitoring• Systematic collection of data on outcomes and

performance• A regular learning & improvement cycle • Credit a program for its contribution to bringing about

change• Encourages the program to challenge itself


Outcome Mapping Monitoring• Flexibility• Participatory• Evaluative thinking• Organisational & social

learning• Power of self-assessment• Regular face-to-face meetings


within the broadest development context or sphere of interest

within the program’s

sphere of influence

design boldly

M&E modestly


Outcome Mapping offers a system/process to gatherdata & encourage reflection on:

1. The progress of external partners towards the achievement of outcomes (progress markers)

2. The internal performance of the program (strategy maps)

3. The program's functioning as an organizational unit (Organisational Practices)


Monitoring in Outcome Mapping

Implementing team

Boundary partner 1

Boundary partner 2

Boundary partner 3

Beneficiary 1 Beneficiary 2

Beneficiary 3

Intervention Strategies

Outcomes

Organisational Practices

Sphere of interest

Sphere of influence

Sphere of control

Focus of M&E

Viability

Efficiency & Relevancy

Behavioral changes


3 types of monitoring journals

Implementing team

Boundary partner 1

Boundary partner 2

Boundary partner 3

Beneficiary 1 Beneficiary 2

Beneficiary 3

Intervention Strategies

Outcomes

Organisational Practices

Focus of M&E

viability

Efficiency & Relevancy

Behavioral changes

Outcome Journal

Strategy Journal

PerformanceJournal

OM Journals


Monitoring Plan


Critical questions?

Programme response

•What should we keep doing?

•What do we need to change in order to improve?

•Are we still working with the right BPs?

•What strategies/practices do we need to add?

•What strategies do we need to end?

•What should be evaluated in more depth?


outcomemapping.ca

2. Monitoring design issuesExperiences from practice

Making learning explicit Use of Spaces & Rhythms (*)

Conventional M&E design •Information needs related to the programme framework (objectives, results, outcomes, … + indicators) •Data collection methods•Reporting

Outcome Mapping (as presented in the OM manual)Based on principles of Utilisation-Focused Evaluation

•Focus on monitoring priorities: Who will use it? Purpose?•Use of outcome, strategy and performance journals

(*) introduced by Guijt & Ortiz (2008)


Monitoring Plan


Assumptions about monitoring (in OM)

• Monitoring process = learning process > reflection and analysis happen automatically

• An actor-centered design leads to a participatory monitoring process

• M&E results will be used• Users have the capacity, time and willingness to participate

or facilitate the monitoring process• Using outcome journals & strategy journals is enough to

pave the way forward • The monitoring process is embedded in organisational or

programme management cycles• …


Making learning explicit

• For learning to happen > data is not the starting point

• Start from the intended use

• Start with defining the spaces that are crucial for debate, sharing, reflection and decision-making

• Make the monitoring integral to the thinking and doing of the organisation and programme



(Source: Seeking Surprise (Guijt, 2008)

Learning-oriented monitoring


1. BE CLEAR ON PURPOSE, USES & USERS

2. DEFINE ORGANISATIONAL SPACES & RHYTHMS

3. DECIDE ON INFORMATIONNEEDS

Which information is required, for who, at what time/event,

in what form, to do what?

Three core steps in the design of alearning-oriented monitoring system


E.g. Intended uses of M&E process (Patton)



e.g. Wheel of Learning Purposes (Guijt, 2008)



Short-term planning

PLANNING

LEARNINGACCOUNTABILITY

Long-term & strategic planning

Programme improvement

Knowledge creation VECO & partners

Negotiation & understanding chain actors

Upward & downward accountability

Programmatic & financial accountability

Evidence building & upscaling

24

e.g. PLA system VECO






• What are the spaces and rhythms central to planning, learning, accountability, debate, decision-making, … (Guijt & Ortiz, 2007)

• How can we ensure that monitoring is integral to the thinking and doing of the organisation and programme?

Organisational spaces Formal and informal meetings and events which bring organisations and programmes to life

RhythmsPatterns in time, the regular activities or processes which provide a structure-in-time, through which an organisation can direct, mobilise and regulate its efforts, i.e. regular weekly, monthly, annual activities that characterise the tempo of organisational functioning.

When do people interact and share information and make

sense of what is happening?




Description of the main spaces & rhythms

EVENT What is thepurpose ofthe event

Time/frequency

Whoparticipates

Whocoordinates

What istheexpectedoutput

Which dataInformationis required

…

…

…

> Group exercise





• Which data & information is required?

• What type of data / information?

• From ‘Nice-to-know’ to ‘Must-know’ information


Information needs linked to the main spaces & processes

EVENT What is thepurpose ofthe event

Time/frequency

Whoparticipates

Whocoordinates

What istheexpectedoutput

Which dataInformationis required

…

…

…

Masterlist of info needs

General info need

Specific information need? (indicators or questions)

FrequencyBy when?

Who has the info? Or where is the data generated?

Data collection method/approach?

Who will use the info at which event?





•Plan how data is used and analysed > make it usable for action

• Focus on social interaction: sharing, reflection, debate, decision

• Should be well-planned & facilitated > it will not happen by itself

Plan for Sensemaking

How to make sure that your monitoring principles and design is translated in an effective monitoring practice?

1. Creating organisational conditions: motives, means & oppportunities

2. The ‘web of institutionalisation’

> Should be reflected in the Organisational Practices


Institutionalising a learning-oriented M&E practice

Creating the motives, means & opportunities

Creating Means-Human capacities-Specialist support-Concepts, methods and tools-Budget

Creating Opportunities-Integration in planning and management-Clear M&E plans and responsabilities-Responsive information management system-Trust and respect – speak out, challenge, feedback

Creating Motives-Guiding ideas-Support by management-Develop a learning Culture-Provide incentives

(Steff Deprez, 2009)


2. The web of institutionalisation (Levy, 2006)

See: THE PROCESS OF INSTITUTIONALISING GENDER IN POLICY AND PLANNING: THE ‘WEB’ OF INSTITUTIONALISATION (Levy, 2006)


outcomemapping.ca

3. Monitoring Practice in OMExperiences from practice

Monitoring Practice in OM

Working with progress markers, boundary partners and organizational practices

Sense-making with boundary partnersOngoing challenges

Thinking through the different aspects of monitoring

– M&E plan (Performance Measurement Framework) – more narrative– Unpack different moments

Working with progress markers

• What is your purpose and use? • What is your monitoring culture?• What resources do you have for monitoring?• What qualitative and quantitative data needs

do you have?

GrOW program• Show matrix

menmen FSPFSP ??leadersleaders

Logic Model

Outcome Mapping

Challenges• Qualitative data collection – informal

interviews, observation (including looking for unexpected – positive and negative)

• Qualitative analysis – looking for patterns and trends

• Critical analysis and sense-making – the need for facilitated, well-constructed (agenda, exercises) spaces and processes

• Usage of information

Evolving Lessons

Monitoring beyond outputsGood Enough (in terms of tools, capacity) and

build from thereM&E – mande – evaluative thinking – - explicit

sense-making spaces

Working with progress markersUse progress markers as a checklist to track progression against pre-defined behavioural changes for a specific partner in a specific period of time; use of scoring (LMH, 1234, colour)Write a qualitative description of change (I.e. every 4-6 months) for each pre-defined PM for a respective periodOther monitoring tools, qual or quan, that are then cross-referenced with pre-defined PMs (new ones added)

Working with progress markers: who?

BP describe their own change - then send to implementing teamImplementing team describes change based on their own observationsMutual reflection process with team and BPsExternal evaluator judges progression in change

Working with progress markers: what?

Every single PM monitored Only PMs that are relevant for a specific periodPMs and / or OCs used to trigger discussion during reflection process; key changes documentedDepth of analysis can varyAcross different BPs (comparison)In combination with SMs (effective intervention?)


Using progress markers at an organisational scale (across programmes)

Different geographical regions

Different thematic foci

Different type and bigger numbers of boundary partners

Use of progress markers for whom?Use of general & standard progress markers for each type of BP

-less (or not) useful for the individual projects and their actors

Tailor-made progress markers for individual BPs

+ relevant to guide and steer local projects and their actors

- less useful for higher levels, overload detailed data & aggregation is difficult


Sensemaking with BPs

Regular (4-6 months) reflection with staff + BP (one or more), external evaluator, other stakeholdersEmbedded in spaces and rhythms of the programme

Working with strategy maps

Useful for monitoring relevance, effectiveness and efficiency of the supporting activities of the implementing teamUnderstand how you spend time and money within a programme teamGet direct feedback from BPs on the required / requested support

e.g. VECO Indonesia - 9 categories of strategy maps

Every activity that is carried out by the VECO can be linked to one of the 9 support strategies.

9 categories of strategy maps

EXAMPLE MONTHLY OVERVIEW

I1 I2 I3 I4 E1 E2 E5

Working with organisational practices

Use as is in manualUse of OCs and PMs at the level of the implementing organizationMotives, Means and OpportunitiesWeb of institutionalisation

Reports and Journals

Simplified journal formatsCombined PM and SM reportsIntegrate elements from journals into regular programme reports Integration in MIS, databases, etc.Used as final report, or as report embedded within another reportOM journal as a databaseSM as an activity guide

outcomemapping.ca

Combined use of OM and the logical framework

Three ways of combining LFA & OM

OM+Modifications of OM framework with elements of LFA

LFA+ Modificiations of an LFA with elements of OM

Combined uses OM frameworks that are linked to/ feed into LFA

OM+Additional layer of

objectives (+indicators) to

pinpoint exactly to what BPs are contributing

Some programmes add an ‘ouput’ layer

between strategy maps and outcomes

OM+Additional layer of

objectives (+indicators) + intermediate

tangible results (+indicators) to

clearly state what BPs are contributing

Inte rventi onlogi c

Indi cators S ou rce ofVer ifi cati on A ss umpti ons

G oal /D evel opm en t O bj

R esult s /Outc om es

Act ivi t ies

Pr oje ct Ob ject ive /Pur po se ≈ Specific objectives + indicators

≈ Outcome Challenge + Progress markers

≈ Activities VECO

Actor-centred results: results describing changes in behaviour

LFA+

Logical Framework of a specific project

OM Framework of a specific project

Inte rventi onlogi c

Indi cators S ou rce ofVer ifi cati on A ss umpti ons

G oal /D evel opm en t O bj

R esult s /Outc om es

Act ivi t ies

Pr oje ct Ob ject ive /Pur po se ≈ Specific objectives + indicators

≈ Chain results + indicators

≈ PMs BPs + Activities VECO

Combined use of LFA & OM

OM frameworks are guiding the PM&E of

specific projects

Logical framework of the over-all programme

Synthesis of the different projects

outcomemapping.ca monitoring in outcome mapping principles, design & practice steff deprez &...

Documents

practice slide

outcome mapping principles

om manual slide

monitoring priorities

core principles of monitoring

monitoring practice

types of monitoring

team boundary partner