Download - Monitoring and Evaluation (M&E) Plan for AR
Monitoring and Evaluation (M&E)
Plan for ARAfrica RISING
Research Review & Planning Meeting – East & Southern Africa Project
23-25 October 2012, Tamale, Ghana
OutlineAR Objectives, activities, and expected resultsMonitoringEvaluationM&E ObjectivesPrinciplesIndicatorsMethodsRoles and Responsibilities
Expected results• Expected key, direct results of implementation
Sustainable higher productivityReduced negative environmental impactsIncreased contributions to natural capital & flow of
environmental services, nutrition/genderValue chain conduct and performance improved? /
Improved market efficiency? / Improved extension effectiveness?
• Many other results possible: labor use, WUE, poverty, hunger, etc.
Monitoring“process of systematic collection and analysis of
data on specific” Describes the “what” of implementation Useful for:
Management - to see if project is on trackReporting – to inform client & other stakeholders of
progress Indicators
USAID FTFOthers (custom) useful to AR for monitoring or
evaluation or both
Evaluation“periodic assessment of worth or significance of an
activity, policy or programme”
• Helps understand “how” & “why” of implementation
• Determines & attributes impact
• Qualitative and/or quantitative
• Dimensions Sites (different levels) Development domains Household types Technologies & combinations Implementation processes
M&E Objectives
•Support effective project management•Provide the data for timely reporting to
USAID•Help all stakeholders to learn about the
project’s successes and failures
M&E Commitments• FtF Compliance: M&E standards, best practices, and core indicators
established for the entire FtF initiative.
• Open-access platform: deliver and maintain an open-access, M&E data management and analysis platform to serve the needs of SI implementation partners and other stakeholders.
• Monitoring & projection: generate ex ante evaluations (e.g. project targets) for a range of farming system and livelihood outcome indicators on an annual basis to provide enhanced research management and outcome mapping needs.
• Multi-scale reporting: provide the capability to support multi-scale monitoring and evaluation
SSA-wide: cross-system reporting to serve the needs of SI wide roll-up of indicators across the three investment geographies/system “project sites” (Guinea Savanna, Ethiopian Highlands, Eastern and Southern Africa)
Guinea – Savannah
East and Southern Africa Maize Mixed
Ethiopian Highlands
Africa Rising M&E Components, Activities, and Outputs
Program/Project Site Identification Outputs
FtF Indicators / reports by- Research sites- Country / National level- Project sites- Program / SSA
Perfomance Variables(modeling & validation)-∆Whole farm productivity-Technology performance ∆ Yield ∆ Labor prod.- by gender ∆ NUE, WUE- ∆Revenues, Costs, Profits
M&E Outputs-FtF Indicators-Outcome mapping (incl. nutrition & market effect)-Cost/Benefit analyses-Experimental /RCT evaluation -Adoption studies?
SI Innovation Catalogue- Inventory (cross-site)- Characterization- Open access
Project Planning & ManagementImproved insights into innovations , delivery platforms, and site selection
Learning
Data/Analysis Platform
Contextual Data(national/regional)- Statistics- HH survey & census- Spatial data
Derived Indicators- HH Typologies- Intensification Index- Sustainability Index- Nutrition index?
Ranking domains by key AR attributes
A ________C ________B ________
Project Site Stratification(Development Domains)
B
A
A C
Project/Activity/Partner Inventory- Project DB (& maps)
Action Research Site selection criteria-Site access-Existing activity/platforms-Research design-Intervention type-M&E approach-…….M&E approach
Identify action research sites in
priority domains that satisfy selection
criteria
Site Data - Climate, soils, market access, etc- Community/HH survey data - Experimental data- Model input data
Whole-farm models
Innovation Inventory- Standard metadata- User interfaces
Research Site ActivitiesBaseline survey Set up trialsMonitoring Mid-line survey (?)End-line survey
++
+++
+
Communities/Farms/Plots
USAID-provided IRs and Indicators
Additional suggested IRs and Indicators
FTF Goal: Sustainably Reduce Global Poverty and Hunger
IR 1: Improved agricultural productivity
Outcome Indicators: #10 Gross margin per hectare (whole farm and by system component)#11 Number of hectares under improved technologies or management practices
IR 2: Expanding Markets and Trade
Outcome Indicator:
#14 Value of incremental salesCustom Outcome Indicator:
#15 Farmer satisfaction with quantity, quality and timeliness of extension and input supply services (Sub-IR 2.3: Improved Market Efficiency)
IR 4: Increased employment
opportunities in targeted value chains
Outcome Indicator: Increase in diversification of off-farm income opportunities for households
IR 5: Increased resilience vulnerable
commodities and households
Output Indicator:
#17 No. of vulnerable households benefiting directly from USG interventions
FTF First Level Objective 1: Inclusive agriculture sector growth
IR 7: Improved nutrition-related
behaviours
IR 6: Improved access to diverse and quality foods
FTF First Level Objective 1: Improved nutrition status especially of women & children
Sub-IR 1.1: Enhanced Human and Institutional Capacity Development for Increased Agriculture Sector Productivity
Outcome Indicators:
•#5 Farmers who applied new technologies or management practices•#6 Private enterprises/organizations that applied new management practices•#12 Stakeholders implementing risk-reducing practices/actions to improve resilience to climate change
Output Indicators:
•#1 Individuals who received long-term training•#2 Individuals who received short-term training•#3 Private enterprises/ organizations receiving assistance•#4 Producer/community based reorganizations receiving assistance
Sub-IR 1.2: Enhanced Technology Development, Dissemination, Management, and Innovation
Output Indicators:
#7 No. of new technologies or management practices: 1) Under research, 2) Under field testing, or 3) Made available for transfer
#13 No. of rural households benefiting directly from USG interventions
Improved connectivity to and utilization of markets and input suppliersImproved connectivity to and utilization of markets and input suppliers
Wider dissemination of integrated SI innovations leading to similar impacts beyond the AR Action Research SitesWider dissemination of integrated SI innovations leading to similar impacts beyond the AR Action Research Sites
RO 1&2: Sustainable increase of whole-farm productivity by integrated innovations for targeted households at research sites
RO 1&2: Sustainable increase of whole-farm productivity by integrated innovations for targeted households at research sites
RO 1&2: Increased nutrition and reduced poverty, especially for women and children
RO 1&2: Increased nutrition and reduced poverty, especially for women and children
RO 1: Situation Analysis & Program SynthesisRO 1: Situation Analysis & Program Synthesis
Diagnosis - Site selection & Characterisation1Mega-site stratification by development domains2Prioritizing mega-site strata geographic units3Project Action/Research Site Selection4Action/Research Site Characterisation5Problem diagnosis6Construction of farm household typologies7Identifying constraints and opportunities (disabling environment, options, entry points)8Conducting value chain assessment9Literature review1Baseline survey1Development of common key indicators (biophysical, socio-economic & institutional)1Technology inventory and characterisation1Ex-ante analysis of potential options
1integrated systems priority setting & planning
1Cooperation and collaboration with partners
Diagnosis - Site selection & Characterisation1Mega-site stratification by development domains2Prioritizing mega-site strata geographic units3Project Action/Research Site Selection4Action/Research Site Characterisation5Problem diagnosis6Construction of farm household typologies7Identifying constraints and opportunities (disabling environment, options, entry points)8Conducting value chain assessment9Literature review1Baseline survey1Development of common key indicators (biophysical, socio-economic & institutional)1Technology inventory and characterisation1Ex-ante analysis of potential options
1integrated systems priority setting & planning
1Cooperation and collaboration with partners
Developing approaches for farm level interventions 1Identifying key components of integrated systems
2Identification of intensification trajectories3Sequencing interventions to suit stage of intensification of household types / systems4Developing participatory tech. selection methods5Identifying models and decision support tools to guide ex-ante technology identification6Ex-ante sustainability & resilience evaluation7Identify high impact sweet spots/ best bets/fits
Developing approaches for farm level interventions 1Identifying key components of integrated systems
2Identification of intensification trajectories3Sequencing interventions to suit stage of intensification of household types / systems4Developing participatory tech. selection methods5Identifying models and decision support tools to guide ex-ante technology identification6Ex-ante sustainability & resilience evaluation7Identify high impact sweet spots/ best bets/fits
Participatory evaluation and adaptation of appropriate combinations of technologies and interventions
1Combining improved legumes with improved livestock feeding practices
2Managing soil fertility and experimenting with novel approaches to increase productivity3Developing incentives for better soil management4Developing site specific recommendations5Combining identified technologies
(e.g., Agroforestry / MPT, alternatives to draught power to save feed, CA with a livestock lens, fertilizer trees, fiderbia/ Acacia/ leguminous trees, improved management of seasonal feed resources, kitchen gardens/ continued poultry, legume rotations (effective rhizobia, biological N fix), micro dosing, more effective contribution to livestock to nutrient management, supplemental irrigation, rainwater harvesting, etc.)
Participatory evaluation and adaptation of appropriate combinations of technologies and interventions
1Combining improved legumes with improved livestock feeding practices
2Managing soil fertility and experimenting with novel approaches to increase productivity3Developing incentives for better soil management4Developing site specific recommendations5Combining identified technologies
(e.g., Agroforestry / MPT, alternatives to draught power to save feed, CA with a livestock lens, fertilizer trees, fiderbia/ Acacia/ leguminous trees, improved management of seasonal feed resources, kitchen gardens/ continued poultry, legume rotations (effective rhizobia, biological N fix), micro dosing, more effective contribution to livestock to nutrient management, supplemental irrigation, rainwater harvesting, etc.)
Knowledge and Capacity building
1Testing novel extension models2Establishing a linked system of models3Training on market oriented production4Addressing new research challenges and opportunities emerging from the activities
Knowledge and Capacity building
1Testing novel extension models2Establishing a linked system of models3Training on market oriented production4Addressing new research challenges and opportunities emerging from the activities
RO 3: Scaling and Delivery of Integrated InnovationsRO 3: Scaling and Delivery of Integrated Innovations
Increased R4D community ability to design and implement farm-scale SI action research, outreach and support approaches, and related D&KM systems
Increased R4D community ability to design and implement farm-scale SI action research, outreach and support approaches, and related D&KM systems
RO 1, 2&3: Improved understanding of landscape level ecosystem stability from the aggregate impact of farming practices at the household level
RO 1, 2&3: Improved understanding of landscape level ecosystem stability from the aggregate impact of farming practices at the household level
Scaling up/out successful technologies and interventions
1Assessing the scalability of integrated innovations2Identification and development of scaling approaches for targeted integrated innovations3indicators4Testing approaches for scaling up and scaling out SI innovations in action sites with project area5Developing a costed program for scaling by development investors6Evaluating aggregated impact of household level interventions at landscape level7Evaluation and validation of scaling approaches for integrated systems
Scaling up/out successful technologies and interventions
1Assessing the scalability of integrated innovations2Identification and development of scaling approaches for targeted integrated innovations3indicators4Testing approaches for scaling up and scaling out SI innovations in action sites with project area5Developing a costed program for scaling by development investors6Evaluating aggregated impact of household level interventions at landscape level7Evaluation and validation of scaling approaches for integrated systems
RO 1, 2 &3: Improved community-based & on-farm NRM
RO 1, 2 &3: Improved community-based & on-farm NRM
RO 1, 2 &3: Increased ability of R4D community to design and implement farm-scale action SI research, outreach and support & related D&KM systems
RO 1, 2 &3: Increased ability of R4D community to design and implement farm-scale action SI research, outreach and support & related D&KM systems
RO 2: Integrated Systems ImprovementRO 2: Integrated Systems Improvement
Research Output 4 Activities: M&E• Validation of indicators and impact pathways
• Development of an M&E indicator collection, management,
and sharing platformOutcome mapping
• Assessment of the nutrition/gender and
environment/ecosystems services outcomes of SI
interventions
• Ex-ante assessment of project- and program-scale outcomes,
impacts and spillover potentials
• Adoption and Impact studies
Analytical Approaches & Tools • Participatory M&E (engagement of multi-stakeholder
partners)
• Delineation and characterization of target farming systems
• Development of a consistent web-based geo-referenced M&E data platform (geo-processing)
• Statistical methods and potentially other specialized approaches, e.g., Caroline Moser’s Gender Frameworks
• Change estimation/projection models for selected indicators
• “Whole-farm” simulation models (e.g., NUANCES and APSFARM)
• Econometric approaches
Impact evaluation methodsMethod Pros Cons Likely to
be used
Outcome Mapping
Easier to implement and interpret. Forces impact pathway articulation
Primarily qualitative. Subjective assessment approach
Yes
DD (difference in difference)/RCTs
Provide quantitative evidence
Not as rigorous if RCTs are not carried out
Yes
PSM/IPW (propensity score matching
Provide quantitative evidence, although second-best option
Not as rigorous if RCTs are not carried out
Yes
RDD (regression discontinuity design)
Provide rigorous evidence
Big sample needed, sharp cut-off based on continuous eligibility criterion (which AR is currently not supporting)
No
Adoption Studies Gain insight into factors limiting uptake of innovations
Many factors influence adoption. Care needed in data collection to limit cost
Yes
Stratification & SI Trajectories
Ag. Potential(Rainfall)
Hi-Hi
Lo-HiLo-Lo
Hi-Lo
Market PotentialPop. density
Geographic Stratification
Hi-Hi
Lo-HiLo-Lo
Hi-Lo
Sustainability Index
Intensification Index
Farm/Landscape Stratification
HHt0
HHti(C)
(A)
(B)
Hi-Hi
Hi-Lo
Lo-Hi
Lo-Lo
AR
C
AR
C
AR
C
AR
C
5 “clean” villages x 10 HHs = 50 HHs + …?“ “ ““ “ “
“ “ “
5 “clean” villages x 10 HHs = 50 HHs + …?
“ “ ““ “ “
“ “ “
#1 team
#2 team
#3 team
#4 team
5 “clean” villages x 10 HHs = 50 HHs + …?“ “ ““ “ “
“ “ “
5 “clean” villages x 10 HHs = 50 HHs + …?
“ “ ““ “ “
“ “ “
5 “clean” villages x 10 HHs = 50 HHs + 5 “conv. ARD” x 10 HHs = 100 HHs“ “ ““ “ “
“ “ “
5 “clean” villages x 10 HHs = 50 HHs + 5 “conv. ARD” x 10 HHs = 100 HHs“ “ ““ “ “
“ “ “
5 “clean” villages x 10 HHs = 50 HHs + 5 “conv. ARD” x 10 HHs = 100 HHs“ “ ““ “ “
“ “ “
5 “clean” villages x 10 HHs = 50 HHs + 5 “conv. ARD” x 10 HHs = 100 HHs“ “ ““ “ “
“ “ “
Total sample: 800 AR 800 “clean C” 800 “conv. ARD” 2,400 HHs
Roles and Responsibilities Monitoring implementing
partners (IITA and ILRI and Collaborators),Sister CG centers, NARS, FOs, NGOs,Private sector
Evaluation (IFPRI and partners)
Impact, Monitoring and Evaluation Team (IFPRI)
M&E Coordination Team(IFPRI)
Collaborators(SpatialDev, ABT, MSU, etc.)
West Africa Project M&E
Team(IITA)
Ethiopian Highland Project M&E Team
(ILRI, IFPRI)
ESA Project M&E Team
(IITA, MSU)
AR M&E Learning Task Force 3 month brief (1st Qtr 2013) M&E Task Force (AR, CSISA, FEEDBACK, CRSP, Local
Institutions M&E specialists) Provide CSISA-AR cross-learning and FtF FEEDBACK
best-practice guidelines Visit all sites, meet local teams and refine M&E
strategy that; (1) appropriate for interventions being tested, (2) strikes appropriate balance between rigour and cost/feasibility, (3) lays out strategy for remainder of 5 years.
Africa Research in Sustainable Intensification for the Next Generation
africa-rising.net
Thank you!