"assessing outcomes in cgiar: practical approaches and methods"

Evaluation of outcomes of CGIAR’s CRPs

ECoP training session 25-26 September 2014

Burt Perrin La Masque Burt@BurtPerrin.com 30770 Vissec FRANCE +33 4 67 81 50 11 1

Purpose of the training session

Consider approaches to the evaluation of outcomes Complex programmes/initiatives Focus on the CRPs

Outcomes of the session Better understanding: what’s involved in evaluation of outcomes in complex environ. Appreciation of challenges – and opportunities Ideas that you can use

2

Topics to explore How to plan evaluations

Complexity: what it is, implications for evaluation of research, CRPs Evaluation vs. other related activities Some tools for evaluation planning (evaluability assessment, TOC, outcome trajectories) Focus on evaluation use Evaluation designs and methods Analysis and interpretation

What this means for evaluation of outcomes of CRPs

3

Characteristics of Evaluation

4

Why do evaluation?

Raison d’être of evaluation Social betterment Sensemaking

More generally, rationale for evaluation To be used! Improved policies, programmes, projects,

services, thinking

5

Evaluation – some key aspects

Systematic, data based, “objective” Evidence can come from multiples sources Can consider any aspect of a strategy, policy, programme, project Major focus on outcomes that follow from the intervention (i.e. attribution, cause) E - valua - tion

6

Different types of evaluation Ex-ante vs. ex-post Process vs. outcome Formative vs. summative Descriptive vs. judgemental Accountability vs. learning (vs. advocacy vs. pro-forma) Short-term actions vs. long-term thinking Etc.

7

Maximising evaluation value & use The right questions! Outcome focus Emergent – not restricted to pre-determined objectives/indicators Respects context, identifies how it interacts with what is done Identifies alignment of activities/projects/programmes with strategy/goal Assesses results orientation as well as actual results achieved

8

What is “complexity”? Emergent vs. predetermined outcomes Feedback loops Indirect, non linear trajectories; tipping points Unpredictability, random events Multiple components: partners, levels, causal package (complicated)

(But: try to explain complex situations as simply as possible!)

9

Nature of intervention and logic chain (e.g. Rogers)

Simple E.g. following a recipe Linear cause-and effect chain Complicated E.g. sending a rocket to the moon Multiple factors happening simultaneously Complex E.g. raising a child Recursive (feedback loops), emergent

outcomes that can’t be identified in advance Tipping points

10

Some characteristics of non-linear change (complexity science)

Cause-effect distance (outcome trajectory): long (or short) in time Depends upon a large number of intervening variables Usually several causes for any effect Change not proportional, incremental; qualitative leaps and bounds Sometimes initial ‘negative’ effects (e.g.

the J-curve) – implications for evaluation? Feedback loops

11

Werner Herzog:

1. “Man is a god when he dreams, but a beggar when he reflects.”

2. “Facts do not constitute the truth. There is a deeper stratum.”

Agree or not? Implications for evaluation?

12

Future orientation - Dilemma

“The greatest dilemma of mankind is that all knowledge is about past events and all decisions about the future.

The objective of this planning, long-term and

imperfect as it may be, is to make reasonably sure that, in the future, we may end up approximately right instead of exactly wrong.”

13

14

Similarities, differences, complementarities

Evaluation and: Research Monitoring Audit

15

Evaluation vs. Research Research Primary objective: long-term knowledge generation (no

single study rarely sufficient) Theory creation/testing/revision Evidence needs: the more the better

Evaluation Reference to a particular type of situation Practical application/utilisation in some form an

essential component Evidence needs: as little as necessary to support

meaningful use (level of confidence required) But: evaluation makes use of research methodologies – from diverse disciplines

16

Monitoring – the concept and common definitions

Tracking progress in accordance with previously identified objectives, indicators, or targets (plan vs. reality) RBM, performance measurement, performance

indicators …

En français: “suivi” vs. “contrôle” Some other uses of the term Any ongoing activity involving data collection and

performance (usually internal, sometimes seen as self evaluation)

17

Monitoring and Evaluation Monitoring

Periodic, using data routinely gathered or readily obtainable, generally internal Assumes appropriateness of programme, activities, objectives, indicators Tracks progress against small number of targets/ indicators (one at a time) Usually quantitative Cannot indicate causality Difficult to use for impact assessment

Evaluation Generally episodic, often external Can question the rationale and relevance of the program and its objectives Can identify unintended as well as planned impacts and effects Can address “how” and “why” questions Can provide guidance for future directions Can use data from different sources and from a wide variety of methods 18

How Monitoring and Evaluation can be complementary Ongoing monitoring Can identify questions,

issues for (in-depth) evaluation

Provide data for evaluation Nature of the intervention

Evaluation Can identify what

should be monitored in the future

19

20

Monitoring, Evaluation and Impact Evaluation

Inputs Outputs Outcomes Impact

Investments (resources, staff…)

and activities

Products Intermediate achievements of

the project

Long-term, sustainable changes

Monitoring: what has been invested, done and produced, and how are we progressing towards the achievement of the objectives?

Evaluation: what occurred and what has been achieved as a result of the project?

Impact evalua-tion: what long-term, sustainable changes have been produced (e.g. poverty reduction)?

Evaluation vs. audit

21

Audit Compliance focus Rules and procedures Divergence: planned vs.

actual

Main attention to process Identify transgressions Standardised approach Outside scrutiny

Evaluation Outcome orientation, context and rationale, attribution Constructive guidance “Why” and “how” as well as “what” considerations Unintended as well as planned impacts and effects Wide range of potential approaches and methods

Evaluability assessment (including theory of change)

22

What is an evaluability assessment (EA)?

Essentially, evidence-based plan for evaluation What aspects of the programme are evaluable – and when? E.g. coherent programme logic, data

availability, conducive environment …

What the programme needs to do Expected outcome trajectories TOC that includes above considerations

23

Elements in an EA (Involve stakeholders – build buy-in) Review/clarify programme intent; identify varying perspectives Help articulate the TOC; identify the soundness of the programme logic, including gaps Identify evaluation priorities and questions Identify evaluation implications for the program Explore feasibility of addressing potential questions (data availability, cost, other considerations) Explore alternative evaluation designs

24

Outcome focus: what is this?

Change that follows from the intervention in some way OECD/DAC: The likely or achieved short-

term and medium-term effects of an intervention’s outputs

Can/should consider other factors/ interventions Consider the “whys”

25

Outcomes (vs. process, impact) Level What is this Example: farmer training

Process Activities, outputs What was done

E.g. programme set up, implemented (as expected, or differently), needs assessment carried out, curriculum developed, outreach, training delivered

Outcomes Changes following from the programme

E.g. learning/expertise, confidence, planting practices, increased yields, new markets, increased revenues

Impact Long-term effects following from intervention Invariably in combination Raison d’être

Sustainability of short-term gains, Poverty, hunger, malnutrition reduction; natural resources sustainability

26

Questions for evaluation Start with the questions Choice of methods to follow How to identify questions Who can use evaluation information? What information can be used? How? Different stakeholders – different questions Consider responses to hypothetical findings Develop the theory of change How many questions?

27

28

Three key evaluation questions

What’s happening? (planned and unplanned, little or big at any level)

Why?

So what?

29

UNEG’s three evaluation questions

Are we doing the right thing?

Are we doing it right?

Are there better ways of achieving the results?

30

OECD/DAC Evaluation Criteria Relevance Effectiveness Efficiency Impact Sustainability

31

• Evaluation criteria vs. evaluation questions • Breadth vs. focus • Intelligent vs. mechanical use

Some uses for evaluation Programme improvement Identify new policies, programme directions, strategies Programme formation Decision making at all levels Accountability Learning Identification of needs Advocacy Instilling evaluative/questioning culture

32

Some priorities for an EA Focus on outcomes Identify expected/potential outcomes Be open to unintended outcomes Outcome trajectories Evaluation priorities and questions Surface and question assumptions Implicit and explicit Be realistic (priorities, expectations of the programme and the evaluation) Don’t set up the programme for failure

33

34

Theory of Change

Why a useful tool for planning an evaluation Alternative terms (intervention logic, logic model, results chain …) Linear vs. models that reflect complexity

35

Results chain Impact

Outcomes

Reach

Outputs

Processes

Inputs

36

Intervention logic model

Inputs

Activities OutputsResults/

Intermediate Outcomes

Ultimate Impacts

37

Generic logic model (simplified)

38

Generic logic model – in context

Inputs Activities Intermediate results (1)

Intermediate results (2)

Impacts

Other results Other

results

Other results

Other results

Other factors

Other factors

Other factors

Needs

Environment et context

Knowledge

Outputs

Other factors

Other interventions

Other interventions

39

IMPACT ON CHILDREN

IPEC/partner

Initiatives

Targeted Interventions

Capacity building

Children

Families and communities

The enabling environment (Institutions,

policies & programmes, legislation, awareness,

mobilization…)

40

41

Outline of factors affecting maternal and child health and nutrition

Fig. from Victora, Cesar G, Robert E Black, J Ties Boerma, Jennifer Bryce. (2010). Measuring impact in the Millennium Development Goal era and beyond: a new approach to large-scale effectiveness. The Lancet. Published Online July 9, 2010

42

AAS Theory of Change

43

AAS Theory of Change: Stakeholder engagement workshop

Design, analysis and method considerations

44

Alternative models of causality (All recognised in the physical and social sciences)

Successionist (factual) causality Counterfactual logic All but one possible explanation ruled out Generative (physical) causality Focus on underlying processes, the

“signature” Simultaneous or alternative causal strands “INUS” conditions: insufficient but necessary,

sufficient but unnecessary Non linear (e.g. “tipping point”) causality

45

Some considerations in choice of design (and methods)

Addresses, somehow, priority questions Simplest approach – at needed confidence Internal/external validity Face validity, construct validity Gets at “the whys” as well as “the whats” Engages stakeholders, partners Practicality (resources, time, data …)

46

Determining attribution – some alternatives

Experimental/quasi-experimental designs (counterfactual, randomisation) Eliminate rival plausible hypotheses Generative (physical) causality, INUS, non linear (“tipping point”) Theory of change approach “Reasonable attribution” “Contribution” vs. “cause”

47

Eliminate rival plausible hypotheses (Donald T. Campbell)

Identify plausible alternative explanations Plausible to multiple stakeholders Anticipate possible questions of sceptics Consider threats to both internal and to external validity Use the simplest means possible to rule out likelihood of alternative explanations

48

Contribution Analysis (Mayne: Using performance measures sensibly)

1. Develop the results chain 2. Assess the existing evidence on results 3. Assess the alternative explanations 4. Assemble the performance story 5. Seek out additional evidence 6. Revise and strengthen the

performance story

49

Further considerations for meaningful outcome evaluation

Need information about inputs and activities as well as about outcomes Check, don’t assume that what is

mandated in (Western) capitals is what actually takes place sur le terrain

Check: are data sources really accurate?

Dealing with responsiveness – a problem or a strength? (Internal vs. external validity) 50

Some alternative approaches Theory based Realist evaluation Most Significant Change, Success Case Method, Appreciative Inquiry Participative Outcome mapping/harvesting Anthropological Etc. etc. etc.

51

To bear in mind “For every complex question, there is a simple answer – and it is wrong.” – H.L. Mencken “One cannot succeed on visible figures alone… The most important figures that one needs for management are unknown or unknowable.” – W. Edward Deming “Not every that can be counted counts, and not everything that counts can be counted.” – Einstein

52

And … “Assessment of many of the most common activities in government requires soft judgment… Measurement often misses the point, sometimes causing awful distortions.” – Mintzberg “Better an approximate answer to the right question than an exact answer to the wrong question that can always be made precise.” – Tukey

53

Methods for data gathering: possible options

Surveys Panel studies/longitudinal (experimental/quasi-experimental) Interviews, group interviews Documentation, analysis of records Observation (quantitative, qualitative) Community members as researchers Alternative methods Multiple methods

54

Making evaluation useful - 1 Be strategic E.g. start with the big picture – identify questions

arising Focus on priority questions and information requirements Consider needs, preferences, of key evaluation users Don’t be limited to stated/intended effects Be realistic, don’t set programs up for failure Don’t try to do everything in one evaluation

55

Making evaluation useful - 2 Primary focus: how evaluation can be relevant and useful Bear the beneficiaries in mind Take into account diversity, including differing world views, logics, and values Be an (appropriate) advocate Don’t be too broad Don’t be too narrow

56

How else can one practice evaluation so that it is useful?

Follow the Golden Rule “There are no golden rules.” (European Commission) Art as much as science

Be future oriented – focused on use Involve stakeholders Use multiple and complementary methods, qualitative and quantitative Recognize differences between monitoring and evaluation

57

Conclusion

Primary focus: helping to make a difference (think strategically!) Requires focus of some form on outcomes What happens when, why, and so what Use evaluation to embrace complexity – as

simply as possible Questions are more important than the

“right” method Thank you / grazie / merci / gracias Burt Perrin

Burt@BurtPerrin.com 58