Evaluating Impacts of Agricultural Research: Lessons and Challenges Based on the Experience of the CGIAR

Mywish Maredia

On behalf ofThe Standing Panel on Impact Assessment (SPIA) of the CGIAR Science Council

Outline

Background on CGIAR Concepts and definitions—ex post impact

assessment (epIA) vs. impact evaluation Overview of the practice of epIA in CGIAR—

experience of SPIA Challenges in assessing impacts of

agricultural research and lessons learned

About CGIAR

CGIAR=Consultative Group on International Agricultural Research

The CGIAR is a strategic alliance of members, partners and international agricultural centers that mobilizes science to benefit the poor

64 members: International organizations, governments, and private foundations

15 centers (also called CG Centers) FAO, IFAD, UNDP and the World Bank serve as co-

sponsors Annual funding: ~$ 500 million in 2007-2008

Mission of CGIAR

To achieve sustainable food security and reduce poverty in developing countries through scientific research and research-related activities in the fields of agriculture, forestry, fisheries, policy, and environment.

What is SPIA?

Independent Impact Assessment and Evaluation Group (IAEG) established in 1996, later renamed the Standing Panel on Impact Assessment (SPIA) when it became part of the CGIAR Science Council

It was established by a Task Force of CGIAR in mid-1990s under pressure from donor constituencies

Motivation:– Strong accountability/resource mobilization rationale– Secondary rationale: feedback for priority setting– Need for independence, objectivity and credibility

Concepts and Definitions

Research for Development (R4D)Results (Value) Chain

Research

Time

Input Output Outcome Impact

Objectives Goals

Different Types of Assessments and Evaluations on Research for Development (R4D) Results Chain

Sca

leP

ilot

/ Sm

all

Glo

bal

Research

Un

it o

f Im

pa

ct

An

aly

sis

Pro

jec

tS

yst

em

Pro

gra

m

Impact evaluation studies that measure

the effect size

Studies that measure the

scale of output adoption/ uptake

Time

Input Output Outcome Impact

Objectives Goals

Program M&E, Impact pathway analysis,

Adoption constraints analysis

Ex post Impact Assessment as a

function of

(effect size * scale)

What is impact in the context of epIA?

Impact is defined as: Positive and negative, primary and secondary, directly or indirectly, intended or unintended long-term effects that can be attributed to outputs generated by agricultural R4D activity

Some Salient Features of epIA

EpIA can only be effectively conducted many years after the completion of the research being assessed (when enough time has elapsed for adoption to take place)

Ex post IA primarily emphasizes the accountability and strategic validation functions:

– To systematically build a strong body of evidence about how (or whether) agricultural research contributes to specific development objectives related to enhance food security, poverty alleviation and environmental sustainability.

– What has been the magnitude and nature of the impact (positive and negative for various producer and consumer groups, etc.).

What epIA is not and does not do?

EpIA is not synonymous with research evaluation (which is broader and occurs earlier in the Results Chain)

It is not appropriate for “real time” feedback and mid-course corrections in research strategies and processes.

Other initiatives within the CGIAR such as those associated with “Institutional Learning and

Change” (ILAC) do these tasks better

Impact Assessments in CGIAR

Done both by centers and SPIA M&E, process evaluation, adoption

constraints analysis – Center Impact evaluation to measure efficacy and

effect size – Center Studies to document adoption – Center and

SPIA (system level) EpIA – Center and SPIA (system level)

SPIA Strategy

SPIA Mandate– To provide CGIAR members with quality information on realized

impacts– To partner with centers to enhance epIA and catalyze novel

methodologies– To encourage strategic feedback from epIA

SPIA Strategy– Commission independent peer-reviewed studies to advance

methodologies, synthesize evidence and draw lessons– Summarize and package information on impacts in digestible

forms that nourish investors– Broaden scope of IA down the impact pathway to CGIAR goals

and emphasize non-economic indicators using non-economists, implying need for this expertise on SPIA

Overview of SPIA Activities

1. Document evidence of overall efficacy of agricultural research as development assistance

Example: SPIA commissioned benefit-cost meta-analysis (Raitzer, 2003). This study:

examined whether the entire investment in the CGIAR over time could be justified on the basis of the benefits derived from its proven (and agreed-upon) major successes.

overall benefit-cost ratio of 9.0 for the $7.12 billion invested (evidence of catalysing substantial additional “multiplier effects” for poor producers and consumers).

study was flagged by a number of donors to the System as the type of assessment required in order to justify continued investments by them into the CGIAR.

Overview of SPIA Activities (cont’d)

2. Expanding coverage of IA beyond crop germplasm improvement

Examples: Impact of Agricultural Research on Poverty and Livelihoods

(IFPRI coordinated study) Synthesis and 7 case studies (Adato and Meinzen-Dick, 2007)

Natural Resource Management Research Impact AssessmentSynthesis and 6 case studies (Waibel and Zilberman, 2007)

Policy Oriented Research Impact AssessmentScoping study (2006) and Synthesis and 7 case studies (2008)

Overview of SPIA Activities (cont’d)

3. Raising and establishing standards and good practice in epIA

Examples: Documenting donor demands for and uses of

evidence of research impact in the CGIAR – study completed in 2005

Development of a document providing strategic guidance for conducting epIA of agricultural research report (Walker et al. 2008)

Overview of SPIA Activities (cont’d)

4. Promote impact assessment culture where quality epIA is regarded essential in research management and planning

Example: Development of an annual performance

measurement system to assess ‘impact culture’ at the Centers

Overview of SPIA Activities (cont’d)

5. Systematize and better expose the Impact Assessments done in the CGIAR

Example: Development of a CGIAR impact website to access

most recent SPIA and Center IA study reports (http://impact.cgiar.org)

Production of short, 2-4 page ‘impact briefs’ specially targeted to donors and key stakeholders of the CGIAR.

Lessons Learned

Good epIA depends on the quality and rigor of analysis that measures the effect size (counterfactual, baseline surveys, methodological rigor)

Documentation of adoption and uptake of research outputs – a necessary step (thus, must be done on a regular basis)

As one moves down the impact pathway, epIA becomes more an analysis of “causal contribution” (rather than “causal attribution”)

Lessons Learned (Cont’d)

Rigorous epIA is more difficult for some types of research (e.g., policy) and program goals (e.g., environment) (there is no “one size fits all” strategy)

EpIA requires knowledge to understand the underlying impact pathways, quantitative skills to convert the observed estimates of effect size and scale into costs and benefits, and qualitative assessment and understanding of other potential impacts not reflected in the quantitative estimates

Challenges Ahead

Clarifying objectives of and priorities for ex post IA

Developing new IA methods in difficult areas Moving further down the impact pathway

beyond economic measures of impact Making ex post IA more utilisation focused Enhance coverage and rigour of Centers’ IA

efforts

How SPIA is planning to address these challenges?

1. Plans a study on the key issue --- learning vs accountability: tradeoffs or complements?Elicit views of research directors and senior research scientists to assess actual and potential use of impact related information for decision making

2. Expand on the recent policy research and NRM research impact studies by blending quantitative and qualitative approaches in rigorously establishing (a) counterfactuals, and (b) attribution

How SPIA is planning to address these challenges? (cont’d)

3. Launch new initiatives to document impacts of some neglected areas of CGIAR investments:

Examples: – Germplasm collection, conservation, characterization and

evaluation (biodiversity)– Training and capacity building

4. Broadening the scope of IA beyond partial assessments (particularly aggregate economic assessments) by advancing further down the impact pathway toward indicators that reflect more closely CGIAR goals (e.g., poverty, food security, environmental sustainability)

How SPIA is planning to address these challenges? (cont’d)

5. Making ex post IA more utilisation focused (providing strategic feedback and operational learning to improve the design and conduct of future research efforts).

How?– Conduct more meta-analyses drawing on individual epIA study

experiences to answer: type of research most effective (greatest impact) validate (or refute) ex ante predictions how can spillovers be maximized

– Closer interactions with evaluation community (NONIE, 3IE, etc.) and ILAC

How SPIA is planning to address these challenges? (cont’d)

6. Helping Centers enhance the coverage and rigour (credibility) of their IA efforts.

How?– Closer interactions between SPIA and Center focal points for

impact assessment via: biennial meetings collaborative projects (NRMR impact study, PORIA study, EIA

study) organizing international symposia (IAAE meetings)

– Further refinement of strategic guidance document for conducting impact assessment

– Assessment of impact culture across the centers (Annual Performance Measurement System exercise)

We Welcome Your Feedback!

Thanks