challenging evaluation: an introduction to outcome mapping

Challenging Evaluation: Challenging Evaluation: An Introduction to Outcome MappingAn Introduction to Outcome Mapping

Amy Etherington & Rebecca Lee

Mini-training for IDRC Interns & PDAs

December 7 & 8, 2005

Session overview:Session overview:

Evaluation context – where does OM fit?

Introduction to OM methodology – main concepts and tools

OM book – it’s useful!

How is OM being used – when is it appropriate?

Warm-up question:Warm-up question:

What words come to your mind when you think of “evaluation”?

IDRC’s Evaluation SystemIDRC’s Evaluation System

Promotes ownership & use of findings at all levels

Decentralized Focus on evaluation processes Monitors quality

Accountability for results at program & project levels

IDRC’s Evaluation UnitIDRC’s Evaluation Unit

Strategic evaluations Cross-cutting issues

Corporate reporting To BoG on performance and results Information systems

Capacity building With Southern partners Promoting M&E with programs

Tools & methods Organizational Assessment Outcome Mapping

Monitoring & evaluation challenges:Monitoring & evaluation challenges:

1. Establishing cause & effect in open systems

2. Sharing ownership & participation

3. Recognizing the contributions of others

4. Tracking progress

5. Encouraging iterative learning

6. Measuring development results of research

7. Timing

Shifting influence over timeShifting influence over time

Low

High

ProgramBeneficiaries/Indigenous Actors

Infl

uen

ce

Time

Are we having an

impact yet ?

A framework that allows researchers to plot human behaviour and actions and assess their contribution to the aims of research projects and programs

Outcomes as changes in behaviour, relationships, activities or actions of the people, groups and organizations with whom a program works directly

IDRC’s Evaluation Unit (1998-2000)

OM bumper-stickers:OM bumper-stickers:

Be prepared for surprises along the way

The map is not the territory!

Being attentive along the journey… This is as important as the destination

Keep your eyes wide open!

OM t-shirtsOM t-shirts

ATTRIBUTIONCONTRIBU

TION

OM t-shirtsOM t-shirts

IMPACTINFLUE

NCE

OM recognizes that in a OM recognizes that in a development context, change is:development context, change is:

Complex (multiple actors and factors)

Continuous (not limited to the life of the project)

Non-linear (unexpected results occur)

Beyond the control of the project (but subject to its influence)

Two-way (program also changes)

Principles of usePrinciples of use

Flexible: modular - use adapted to circumstances

Participatory: seeks dialogue and collaboration with boundary partners in P,M,&E

Evaluative thinking; culture of reflection: promotes social and organizational learning

Step 1: VisionStep 1: Vision

I have a dream!

Martin Luther King, Jr.August 28, 1963

Vision is a guide:Vision is a guide:

NorthLight house

Distant star

Vision statement:Vision statement:

Reflects the broad human, social, and environmental betterment in which the program is engaged and to which it is contributing

Written in present tense - as if the vision were already a reality

“Those who dream make a difference!”

Ms. Kalpana Pant

-Chaitanya

Women and girls in rural India enjoy full access to appropriate health care, education, food and water security and freedom from violence. They earn their own livelihoods with free access to markets and to banking and municipal services. Women knowledgeably use drudgery reduction technologies and agricultural practices that support human and ecological well-being. Villages are served by public transport, are well lit at night and have police forces that enforce laws fully and equitably. Girls attend school full time and, with their families, have the information and resources to make informed decisions regarding their personal health, safety and social needs. Gender equity is evident in the allocation of household labor and household decision-making; and men in the community support gender-responsive initiatives. Schools reinforce the role of women as educated, valuable and empowered community members. Through community-based organizations, women participate in and influence decisions, policies and programs affecting their well-being and share their experiences with others.

What is the human, social and/or environmental condition that you hope to see in the region in which the program is being carried out?

Who is contributing to this situation - and how?

Vision: facilitation questionsVision: facilitation questions

Dream boldly !

Step 2: MissionStep 2: Mission

The mission is that “bite” of the vision statement on which the program is going to focus.

Written in future tense - as something the program will do

Your mission is your Your mission is your “business”“business”

What do you do?

Who do you do it with?

Why do you do it?

How can the program best support or contribute to the achievement of the vision?

What areas do you need to work in?

Where do you have credibility?

Who can you work with?

Mission: facilitation questionsMission: facilitation questions

Vision

Mission

Step 3: Boundary PartnersStep 3: Boundary Partners

Boundary Partners: Boundary Partners: definitiondefinition

Those individuals, groups, and organizations with whom the program:

interacts directly to effect change anticipates opportunities for influence

engages in mutual learning

Program

The rest of the world

= partners

Sphere of influenceSphere of influence

Program

The rest of the world

= partners

Boundary Partners have Boundary PartnersBoundary Partners have Boundary Partners

program program’s bp bp’s bp

CIDA

IDRC

BAIF

StateNGO

StateNGO

StateNGO

StateNGO

StateNGO

StateNGO

SHG Police CommunityLeaders

Families Banks PHCs

SwayamsiddhaSwayamsiddha

In which individuals, groups, or organizations is our program trying to encourage change so that they can better contribute to the vision?

With whom will we work directly?

Are we choosing X BP because we want to influence their behaviour and actions, or because they will influence others? Or both?

What behavioural changes do we (the project and BPs, collectively) want to see in the BP that will contribute to the vision?

Boundary Partners: Boundary Partners: Facilitation QuestionsFacilitation Questions

Vision

BOUNDARY PARTNER 2

Mission

BOUNDARY PARTNER 3

BOUNDARY PARTNER 1

Step 4 & 5: Step 4 & 5: Outcome Challenges and Outcome Challenges and

Progress MarkersProgress Markers

One OC is about a single boundary partner

Describes the ideal behavioural changes, relationships, actions and interactions in this partner

Describes how these changes will contribute to the vision.

Outcome Challenge: Outcome Challenge: definitiondefinition

One for each boundary partner

Does NOT describe program strategies

Written like this: “The program intends to see [boundary partner] who [description of behaviours in the active present tense]”

Outcome Challenge: characteristicsOutcome Challenge: characteristics

Ideally, how would your boundary partner be acting?

With whom would they be interacting?

What would they be doing?

Outcome Challenges: facilitation Outcome Challenges: facilitation questionsquestions

Progress Markers: Progress Markers: descriptiondescription

A graduated set of statements describing a progression of changed behaviours in the boundary partner

Describe changes in actions, activities and relationships leading to the ideal outcome

Articulate the complexity of the change process

Can be monitored & observed

Permit on-going assessment of partners’ progress (including unintended results)

3 Levels of Progress Markers3 Levels of Progress Markers

The program sets out what it would:

Expect to see the boundary partner doing?

Like to see the boundary partner doing? Love to see the boundary partner

doing?

Progress markers = ladder of changeProgress markers = ladder of change

Truly transformativeSet quite high

More active learning, engagement

Early response to program’s basic activities

Love to see

Like to see

Expect to see

Why graduated progress markers?Why graduated progress markers?

Taken as a set, the progress markers: are graduated from easier to more difficult to achieve

changes in behaviour

describe the change process of a single boundary partner

are more complete than a single indicator

help the program think about how it can intentionally

contribute to the most profound transformation possible

facilitate mid-course corrections and improvement

Vision

OUTCOMES

Behaviors, Actions, Relat

ionships of

Boundary Partners

Progress Markers Love to See

Like to See

Expect to See

BOUNDARY PARTNER 2 Outcome Challenge

Mission


Like to See

Expect to See


Like to See

Expect to See



Influencing OutcomesInfluencing Outcomes

BoundaryPartner

Program relevance & viability

Org. Practices

Program Results

Progress Markers

Program Delivery

Strategies

Program

Step 6: Strategy MapStep 6: Strategy Map

Strategy MapStrategy Map

Outlines the program`s approach in working with the boundary partner

Indicates the relative influence the program is likely to have on boundary partner

Helps pinpoint strategic gaps in the approach or if the program is overextended

6 Types of Strategies6 Types of Strategies

Aimed at the

Boundary Parnter

Aimed at the Boundary Partner`s

Environment

Strategy SupportivePersuasiveCausal

I-1

• Direct Output

E-1

•Alter physical or regulatory

environment

I-2

• Arouse New Skills/ Thinking

I-3

•Supporter who guides change

over time

E-2

•Modify the information system

E-3

•Create / Strengthen a Peer Network

Facilitation Questions:Facilitation Questions:

I-3I-2I-1

E-1 E-3E-2

What will bedone to producean “immediate”

output?

What networks/relationships willbe established

or utilized?

How willsustainedsupport,

guidance, ormentoring be

provided?

How will you usethe media orpublications?

What will bedone to build

capacity?

What will bedone to changethe physical or

policyenvironment?

Strategy imagesStrategy images

Vision

ORGANIZATIONAL

PRACTICES

STRATEGIES

Undertaken to Bring

about Outcomes

OUTCOMES

Behaviors, Actions, Relationships of

Boundary Partners


Like to See

Expect to See


Mission


Like to See

Expect to See


Like to See

Expect to See

Strategy Map 3 Strategy Map 2 Strategy Map 1



Step 7: Organizational Step 7: Organizational PracticesPractices

What the program does to:

look within

stay fresh, sharp, effective, healthy

better serve its partners

learn and change

Organizational Practices: definitionOrganizational Practices: definition

Why Organizational Practices?Why Organizational Practices?

Important to how the program is going to function to effectively fulfill its mission

Supporting change in its boundary partners requires that the program be able to change and adapt

8 Organizational Practices8 Organizational Practices

1. Prospecting for new ideas, opportunities, and resources

2. Seeking feedback from key informants

3. Obtaining the support of your next highest power

4. Assessing and (re)designing products, services, systems, and procedures

8 Organizational Practices8 Organizational Practices

5. Checking up on those already served to add value

6. Sharing your best wisdom with the world

7. Experimenting to remain innovative

8. Engaging in organizational reflection

Vision

ORGANIZATIONAL

PRACTICES

STRATEGIES

Undertaken to Bring

about Outcomes

OUTCOMES

Behaviors, Actions, Relationships of

Boundary Partners


Like to See

Expect to See


Mission


Like to See

Expect to See


Like to See

Expect to See

Strategy Map 3 Strategy Map 2 Strategy Map 1



Step 8-11: Monitoring Step 8-11: Monitoring priorities and Journalspriorities and Journals

Count every " F " in the following text:Count every " F " in the following text:

FINISHED FILES ARE THE RE

SULT OF YEARS OF SCIENTI

FIC STUDY COMBINED WITH

THE EXPERIENCE OF YEARS...

M & E involves making choicesM & E involves making choices

“You can’t always get what you want…But if you try sometimes, you might find

You get what you need… ahhhhhhh, yeah…”

The Rolling Stones

Choosing WHAT to monitorChoosing WHAT to monitor

Who will use the monitoring information?

What will it be used for?

When is it needed?

Choosing HOW to monitorChoosing HOW to monitor

Which components will be monitored?

How and when will data be collected?

Who will collect it?

Who will analyze, collate, package data?

Where and when will it be discussed and used?

Monitoring needs to be:Monitoring needs to be:

Useful

Affordable

Light in work

Understandable

Outcome JournalOutcome Journal

Work Dating From/To:

Contributors to Monitoring Update:

Low =

Medium =

High =

Outcome Challenge:

LMH

Expect to see: Who?

3

1

2

Strategy JournalStrategy Journal

Working Dating From/To:


Strategy to be Monitored:

Effectiveness?

(How did it help the boundary

partner?)

Outputs

Performance JournalPerformance Journal

Working Dating From/To:


Practice 1: Prospecting for New Ideas, Opportunities, & Resources

Practice 2: Seeking Feedback from Key Informants

Example or Indicators:

Sources of Evidence:

Lessons:

Example or Indicators:

Sources of Evidence:

Lessons:

Step 12: Evaluation PlanStep 12: Evaluation Plan

Evaluation Plan:Evaluation Plan:

A short description of the main planning elements of an evaluation study

For Whom? What? Where? How? By Whom? When? How Much?

Elements of an Evaluation PlanElements of an Evaluation Plan

Who Will UseEvaluation,

How, & When?

Questions to be

Answered

Info. Sources

EvaluationMethods

Who WillConduct &

Manage theEvaluation?

Date(Start &Finish)

Cost

Evaluation Issue:

Setting Evaluation PrioritiesSetting Evaluation Priorities

Criteria with which to prioritize include:

Learning needs

Reporting requirements

Partners’ information needs

How can OM be used?How can OM be used? Designing and articulating the program’s logic

Recording internal and external monitoring data

Indicating cases of positive performance and areas for improvement

Evaluating intended and unexpected results

Gathering data on the contributions that a program made to bringing about changes in its partners

Establishing evaluation priorities and an evaluation plan

When is OM best used?When is OM best used?

Once strategic direction or primary program areas are established

Particularly effective for larger projects

Best used at the start, but can also be used as a midway or final assessment tool

Activities must be sufficiently specific to identify key groups who will be influenced

Is OM appropriate? Not always!Is OM appropriate? Not always!

May require changes that are not possible Able to focus on “OM” outcomes Commitment to change and self-assessment Not intended for technical evaluations

(assessing relevance of a programming area or a cost-effectiveness comparison)

Credible, and compatible with donor reporting requirements

Team consensus Resource commitment

challenging evaluation: an introduction to outcome mapping

Documents

evaluation context

challenging evaluation

om fit

human behaviour

om bumperstickers

om methodology main

development context

influencetwoway program