a flexible, multi-purpose monitoring, evaluation, and learning system to support australian...

A flexible, multi-purpose monitoring, evaluation, and learning system to support Australian agriculture international aid and development programsSamantha Stone-Jovicich, Andy Hall, Jennifer Kelly, Michaela Cosijn, and Lucy Carter (CSIRO) AES 2015 Conference, Melbourne, 7 September 2015

This presentation

• Journey • Reflections and lessons learnedo Concept to practiceo Scientists doing monitoring,

evaluation and learning

M&E in abstract

M&E in practice

The project context

Flexible, multi-purpose MEL system • Samantha Stone-Jovicich

FSI• 2012-2015

• PARTNERSHIP

o Department of Foreign Affairs and Trade (DFAT)

o Australian Centre for International Agricultural Research (ACIAR) and the Australian International Food Security Centre (AIFSC)

o Commonwealth Science and Industrial Research Organisation (CSIRO)

• IMPACT

• 3 THEMATIC AREAS

DFAT

ACIAR

Improve the impact of ODA-supported agriculture and food security programs in the Indo-Pacific region

Partners-in-country

(Africa, Asia)

Multiple partners, cross-scale

CSIRO

KEY

ACTI

VITI

ES &

OU

TPU

TS(o

rgan

ised

arou

nd 3

them

es: M

arke

ts &

Par

tner

ship

s;

Agric

ultu

ral F

ront

iers

; and

Man

agin

g fo

r Im

pact

)

PEOPLE with different skills &

knowledge

FINANCIAL support

CHAMPIONS from each of the key partner organisations

Formal LINKS/ACCESS to Australian-supported international development arenas, organisations, and individualsKE

Y RE

SOU

RCES

/IN

PUTS

Improved impact of ODA-supported food and agricultural programs in the

Indo-Pacific region

Learning events

What FSI is trying to achieve and how

Knowledge products

Expertise & practice networks

In-country engagement

Outreach & external visibility

FSI governance

Key partnering development programs and projects

Markets and Partnerships

Agricultural Frontiers

Managing for Impacts

FSI themes

INDONESIAARISA

COLOURED AREAS ON MAP = Regions and countries where ACIAR, AIFSRC, and DFAT work

= Agriculture and food security programs and projects engaged with FSI

INDIA/BANGLADESHSRIFSI

EAST AFRICAAAPP Program

TIMOR LESTETOMAK Program

CAMBODIAKAVAC Program

The challengesThe complexity of FSI The complexity of the

broader context

(Source: Batchelor et al. 2011)

New area for CSIRO

(Source: SIL International)

Our experience designing a MEL system for FSIA journey of moving from ‘best practice’ to ‘fit-for-purpose’


Food Systems Innovation for Food Security (FSIFS) Inception Project launched

The Food Systems Innovation (FSI) initiative launched

FSI mid-point review highlights lack of progress with MEL

July2012

May2014

Dec2015 FSI to end

July2013

FSIFS endsJune2013

Social learning

Indicator-based M&E

MEL

APP

ROAC

HES

AN

D T

OO

LS T

RIAL

LED

Realisation that a social learning approach was not most suitable approach

‘Fit-for-purpose’ MEL

KEY EVENTS AND CRITICAL JUNCTURES

Internal team meeting reflecting on achievements and challenges Results in ‘home-grown’ MEL approaches (FSI-at-a-Glance & FSI Learning Trajectories)

‘Home-grown’ MEL approaches

Adoption of M&E approach most familiar to our donors

Realisation that logframe, indicator-based M&E was not sufficient

Impasse with FSI-at-a-Glance and FSI Learning TrajectoriesShift to Rubrics and other complementary approachesDec

2014

What we tried‘Best-practice’: A social learning approach


“Organisations where people continually expand their capacity to create the results they truly desire, where new and expansive patterns of thinking are nurtured, where collective aspiration is set free, and where people

are continually learning how to learn together.” (Senge 1990: 3)

Seminars; facilitated discussions+

Series of reflection and learning activities, tailored to the culture, interests & needs of each partner*

BARRIERS WANT TO FOCUS ON/ABLE TO INFLUENCE

ACTIONS AND COMMITMENTS TO FOLLOW THROUGH

Narrow performance measures

Workforce planning

“Soft structures”

Dominant paradigms and assumptions

Project design and implementation

In-country partners and partnerships

* NOTE: was implemented only with CSIRO


What we tried‘Best-practice’: Indicators

We built on the program logic framework

THEMES Component 1: Analysis & Field Application

Component 2: Knowledge Management and Communications

Component 3: Capacity Building

Theme 1: Theory of Change and Livelihood Impact Pathways

• Activity 1 & outputs• Activity 2 & outputs• Etc.

Theme 2: Market-based Approaches and Public-Private Partnerships

Theme 3: Nutrition-sensitive Agriculture

FSI activities

and outputs

Aid program

outcomes

Development impacts

Outcome 1

Outcome 3

Impact 2 Impact 3

Outcome 2Outcome 4

Impact 1

Monitoring, Evaluation and Learning (MEL): key questions

AID PROGRAM OUTCOMES• What were the aid program outcomes from the FSI project? • Did the project achieve outcomes 1-3? Where there any additional

outcomes?

ACTIVITIES AND PROCESSES• How did the activities, processes, and outputs from each of the project

components, learning themes, and MEL contribute to the outcomes?• What are the key lessons learned that emerged?

DEVELOPMENT OUTCOMES• What were the development outcomes from the FSI project? • Did the project contribute to more effective, evidence-based agriculture,

food security and nutrition interventions? More food? Increased net incomes? Where there any additional outcomes?

FSI p

roje

ct (2

013-

15)

FSI p

roje

ct (2

013-

15)

Futu

re p

roje

ct(s

)

OUTPUTS• What were the outputs generated from the FSI project for each learning

theme? • In particular, what innovative solutions were identified and which were

integrated into projects?

ACTIVITIES AND PROCESSES• How did the activities and processes implemented contribute to those

outputs?• What are the key lessons learned that emerged during these activities and

processes?

…too complicated; too cumbersome; too rigid

THEMESCOMPONENT 1:

Analysis & Field ApplicationCOMPONENT 2:

Knowledge Management and CommunicationsCOMPONENT 3:Capacity Building

THEME 1

ACTIVITIES• Seminar 1• Desktop study• Etc.

OUTPUTS• 1 page Discussion • Report• Etc.

ACTIVITIES OUTPUTS ACTIVITIES OUTPUTS

THEME 2

THEME 3

FSI a

ctiv

ities

an

d ou

tput

s

Aid

prog

ram

ou

tcom

es, i

ndic

ator

s an

d M

EL to

ols f

or

mea

sure

men

t

Deve

lopm

ent

impa

cts

Outcome 1

Outcome 3

Impact 2 Impact 3

Outcome 4

Impact 1

INDICATORS MEASURES OF PERFORMANCE MEL TOOLS

Indicator 1 # of…. Survey

Indicator 2 Perceived changes of … Most Significant ChangeINDICATORS MEASURES OF

PERFORMANCEMEL TOOLS

Indicator 1 XXX XXX

Indicator 2 XXX XXX

Etc.

INDICATORS MEASURES OF PERFORMANCE

MEL TOOLS

Indicator 1 XXX XXX

Indicator 2 XXX XXX

Etc.

Outcome 2INDICATORS MEASURES OF

PERFORMANCEMEL TOOLS

Indicator 1 XXX XXX

Indicator 2 XXX XXX

Etc.

We developed detailed indicators and performance measures…

We attempted to simplify it into tables…

Indicators Means of verification MEL data collection tools Responsibility Presentation of results

Reporting timeframe

Information on food systems innovation is made available to Australian-funded policies, programs, and projects

Database of range of information (e.g. topics, technical info, methodology, etc.) made available

Content analyses of materials produced and made available through different FSI dissemination/communication channels

FSI teammember X

• Time series• Pie charts

Dec 2014May 2015

Etc.

OUTCOME 1: Knowledge about of food systems innovation for food security enhanced among FSI partners and collaborators engaged in Australian-funded policies, programs, and projects

What we tried‘Home grown’ approach: FSI-at-a-Glance and FSI Learning Trajectories


13%

7%

15%

9%

Outputs

Brokering of knowledge and collaborations

Lessons learnt

Trust-building

• Studies/reports• Dossiers• Working papers• Blogs and online dialogues• Guidelines• Practice Notes

FSI AT A GLANCE

FSI LEARNING TRAJECTORIES

FSI

INTAN

GIBLE STU

FF;PRO

CESSES; “INVISIBLE”

Engagement processes

• Seminars, brown paper bag talks• Facilitated workshops• Training modules• Roundtables• Networks

Emergent, unexpected

Balancing M&E of outputs and processes; incorporating learning

TANG

IBLE STUFF;

FORM

AL CO

MM

ITMEN

TS

NOTE: Information shown is indicative of types of outputs and processes monitored

Presentation title • Presenter name19 •

OVERARCHING OUTCOMES GOAL

Evidence, analysis and experience more effectively and routinely applied in agriculture and food security programming and strategy and practice:

Innovative and effective

agriculture and food security

programming, strategy and

practice

BROAD-LEVEL OUTPUTSSPECIFIC ACTIVITIES & OUTPUTS

FSI-AT-A-GLANCE EXAMPLE FOR AGRICULTURAL LINKAGES COMPONENT (Nutrition Sensitive Agriculture, July – Nov 2014)

• 1 invited presentation • Planning for Roundtable initiated

in collaboration with DFAT and USyd

• 1 Dossier completed• 2nd Dossier in preparation• NSA web homepage completed• 1 Discussion Brief• 1 Working Paper • 16 external resources

identified

• Presentation of FSI NSA insights at workshop that brings together group of Australian researchers PILOTED as mechanism for info exchange and networking and future capacity building opportunities

• NSA homepage, Discussion Brief 1, Dossier 1 and Working Paper 1 on FSI web for partners and other audiences to access

• Dossier 1 shared with DFAT, CSIRO and ACIAR

• 16 external resources shared (summaries and links on FSI web)

• Engagement with USyd Food & Nutrition Security Node

• Connections with NSA experts being established and maintained via multiple avenues

• Pool of NSA experts being identified, introduced to FSI, and invited to formally contribute

• FSI being increasingly recognized by partners as an expert in NSA, and contacted for advice and feedback

DFAT• Co-development of Operational

Guidance Note on NSA• Contribution to NSA policy wording in

AgFishWater Strategy• Guidance on description of FSI and NSA

in DFAT presentation at ICN2• Information provided on link b/w NSA

and trade

OVERSEAS PROGRAMS/PROJECTS• NSA advice and feedback to AAPP

program and other projects

ACIAR• Information provided on Nutrition

Foresighting with DAFF• Contribution to Vietnam nutrition

workshop program

• Engagement with DFAT Agricultural Productivity & Food Security Sector

• Engagement with ACIAR Agribusiness Program

• Engagement with AAPP Africa• Engagement with

Timor Leste desk

• Collaborations with DFAT and ACIAR, and associated overseas programs and projects, being established and specific activities/products being identified and advanced

ENG

AGEM

ENT

EXPE

RTIS

E &

PR

ACTI

CE N

ETW

ORK

SKN

OW

LED

GE

PRO

DU

CTS

LEAR

NIN

G

EVEN

TS

3

5 16

72

4

KNOWLEDGE PRODUCTS produced by FSILEARNING NOTES/DOSSIERS Dossier 1 Improving Nutrition through Agriculture Linkages Dossier 2 Food Security and the Double BurdenDISCUSSION BRIEFS Discussion Brief 1 Enhancing the nutritional impact of agricultural

investments for improved development outcomes

WORKING PAPERS Working Paper 1 Enhancing the nutritional-sensitivity of agricultural

development interventions in the Eastern Gangetic PlainsFSI WEBSITE ‘AGRICULTURE LINKAGES’ HOME PAGE Agricultural linkages: Strengthening the linkages between agriculture and

other development priorities

External KNOWLEDGE PRODUCTS sharedEXTERNAL PUBLICATIONS * FAO: Child and maternal nutrition FAO: The state of food and agriculture: food systems for better nutrition FAO & Agriculture-Nutrition CoP: Key recommendations for improving nutrition through

agriculture Global Forum for Rural Advisory Services (GFRAS) & World Bank: A synthesis of experiences,

lessons and recommendations

OUTCOMES: direct contributions to partners’ ag & food security programming/strategy/practice

DFAT (Canberra) Co-development of DFAT Operational Guidance Note on NSA Responding to request on how to incorporate NSA definition in AgFishWater Strategic Framework Responding to request on FSI and NSA wording for DFAT statement at ICN2 and on linking trade

and NSA ACIAR (Canberra) Responding to request for Nutrition Foresighting DAFF project Responding to request, input provided into program of Hanoi nutrition workshop programOVERSEAS PROGRAMS AND PROJECTS Technical advice and feedback to AAPP program (DFAT/CSIRO) on nutrition pathways in their

project designs

NETWORKS AND PLATFORMS * FAO: Global Forum on Food Security and Nutrition (FSN Forum) FAO: Improved Global Governance for Hunger Reduction Programme Scaling Up Nutrition (SUN) SecureNutrition Agriculture-Nutrition Community of Practice

FAO Improved Global Governance for Hunger Reduction Programme: Synthesis of guiding principles on agriculture programming for nutrition Rachel Turner et al. journal paper: Agriculture for improved nutrition: the current

research landscape (Food and Nutrition Bulletin) The Lancet journal: Executive summary on maternal and child nutrition World Bank: Improving nutrition through multisectoral approaches

Action against Hunger (ACF – International): Maximising the nutrition impact of food security and livelihoods interventions

World Bank: Prioritizing nutrition in agriculture and rural development: Guiding principles for operational investments

IFPRI: Working multisectorally in nutrition: principles, practices and case studies

EXPERTISE & PRACTICE NETWORKSU. OF SYDNEY Discussions with U of Syd Food and Nutrition Security NodeNSA-RELATED GLOBAL ALLIANCES/ORGANISATIONS Compilation of global alliances and organisations working on

NSA-related research and practice OTHER AVENUES Links being made with NSA experts via writing, workshops, peer reviews

LEARNING EVENTSSEMINARS, PRESENTATIONS University of Sydney Workshop (Nov 14, 2014) - Taking a

wider view: Health impacts on agricultural productivity in Southeast Asia (co-presentation with ACIAR and DFAT)

REGIONAL DIALOGUES/ROUNDTABLES Planning started for Roundtable Discussion on NSA with

leading nutrition sensitive agriculture experts in Australia and internationally

ENGAGEMENT with partnersDFAT Agricultural Productivity & Food Security SectorACIAR Agribusiness ProgramAAPP AfricaTIMOR LESTE DeskSRFSI

FSI-AT-A-GLANCEEXAMPLE FOR AGRICULTURAL LINKAGES COMPONENT (Nutrition Sensitive Agriculture, July – Nov 2014)

13%

7%

15%

9%

.

13%

7%

15%

9%

(For

mal

) out

puts

Supp

ortin

g (in

form

al) p

roce

sses

FSI A

CTIV

ITIE

S

TIMELINEFY

2013-14July 2014

OTH

ER

SUPP

ORI

NG

EVEN

TS

Aug 2014 Sept 2014 Oct 2014

FSI OU

TCOM

ES

FSI MEL reflection workshop

(May 2014)

June 2014 Nov 2014

Previous FSI discussions with

DFAT Timor Lestedesk about NSA

FSI MEL NSA Learning Trajectory for 2012-2014

International movement to bring NSA to the fore in development (has helped put NSA in the spotlight in Australia)

FSI Work Plan process FSI invited to review manuscript in Food and

Nutrition Bulletin

Request from ACIAR

Published on FSI website

FSI-NSA presentation at U Sydney Workshop

FSI-NSA meeting w/ACIAR Agribusiness

Program

ACIAR Nutrition workshop program for

Vietnam (Hanoi, Nov 2014)ACIAR-DAFF Nutrition

Foresighting web tool POTENTIAL: input on strategic

approach to NSA in ACIAR

WORKING PAPER 1

Contribution to post-

workshop position paper

UNIVERSITY OF SYDNEY Input into development of the

Food Security & Nutrition Security Node

FSI-NSA meeting

w/DFAT Ag & FS Sector

DFAT- Ag & Food Security SectorOperational Guidance NoteGuidance on linkages b/w NSA

and trade DFAT- Whole of ODA StrategyNSA policy wording for the

AgWaterFish Strategy

FSI NSA-related engagement

w/AAPP

Report produced for AAPP

FSI-NSA meeting with DFAT Timor

Leste desk

DFAT Ag & FS Sectorprovides feedback

and distributes internally

FSI-NSA linkages established w/ U of

Sydney Food & Nutrition Security Node

DFAT- Timor Leste Program POTENTIAL: NSA input - TOMAK

DFAT- AAPP (Africa) NSA advice to AAPP

DOSSIER 1

Draft completed

FSI Partnership

Building Workshop

FSI contributing to (as of August 2014):

Re-engagement w/DFAT Timor

Leste desk

= knowledgeproducts

= learning events = engagement

with FSI partners = fostering of

networks with experts

= partner outcomes = other outcomes

LEGEND

NSA = nutrition sensitiveagriculture

WORKING PAPER 1: Enhancing NSA interventions in the Eastern Gangetic Plains

DOSSIER 1: Improving Nutrition through Agriculture Linkages

FSI LEARNING TRAJECTORIESEXAMPLE FOR AGRICULTURAL LINKAGES COMPONENT (Nutrition Sensitive Agriculture, July – Nov 2014)

What we tried‘Fit-for-purpose’: A rubrics approach & more


INTERMEDIATE OUTCOME

GOALSPRIMARY OUTCOMES

Relevant, timely, appropriately-designed learning events that involve Australian and/or in-country and partners and their networks, and are perceived as worthwhile

An expanded range of relevant experts and development practitioners who actively contribute to FSI Australian and in-country partners’ and their networks’ international development discussions, designs, and practices in ways that are perceived as collaborative, salient, credible and useful

Activities and a web-based platform that bring together analyses, opinions, lessons, experiences and capacity building resources derived from FSI and its Australian and regional partners; are easily accessed and easy to navigate; and audience-appropriate and reader-friendly; and updated on a regular and in a timely fashion

Improved impact of ODA-supported food and agricultural programs in the Indo-Pacific region

FSI recognised internationally as a trusted and credible source of analysis and operational guidance on practices and policies that promote food systems innovation

Knowledge products are relevant, practice-based and practice-oriented, and aligned with current and emerging (ag, food, and nutrition) international development thinking, practices, and needs within Australia and overseas; collaboratively-created, reader-friendly and audience-appropriate, and produced and delivered in a timely manner

FSI creates opportunities for in-country partners to participate in reflection and learning in food systems innovation

FSI key partners (DFAT, ACIAR, CSIRO) work together collaboratively, respecting the agreed-upon partnership principles, to collectively learn and manage FSI in a responsive and agile manner

KEY ACTIVITIES & OUTPUTS

organized around Markets and Partnerships, Agriculture Linkages,

and Managing for Impact

LEARNING EVENTS

KNOWLEDGE PRODUCTS

EXPERTISE & PRACTICE

NETWORKS

FSI GOVERNANCE

OUTREACH &EXTERNAL VISIBILITY

IN-COUNTRY ENGAGEMENT

Enhanced knowledge-exchange; learning; and networking among FSI partners and other stakeholders in Australia and overseas, thereby strengthening capacity to progress food systems innovation

Innovation in ODA-supported food and agricultural program design and implementation

Rubrics approach focused on the outcomes

LEARNINGEVENTS

EVALUATION CRITERIA & RATINGS

Rele

vant

Tim

ely

Appr

opria

tely

de

signe

d

Invo

lve

part

ners

Wor

thw

hile

Enha

nce

kn

owle

dge

Enha

nce

lear

ning

Enha

nce

netw

orki

ng

Workshop 1

Seminar

Lunch seminar

Reflection event

Workshop 2

Training event 1

Training event 2

Presentation

OVERALL

FSI RUBRICS APPROACH: An example for Learning Events (fabricated data)

FSI OUTCOME: Learning events are relevant, timely, appropriately designed; involve Australian and/or in-country partners and their networks; are perceived as worthwhile; and are effective in progressing FSI’s primary outcome (enhanced knowledge-exchange, learning, and networking among FSI partners and other stakeholders in Australia and overseas, thereby strengthening capacity to progress food systems innovation)

3. SUMMARY REPORT2. SUMMARY RUBRICS OF ALL EVENTS1. INDIVIDUAL RUBRICS FOR EACH EVENTEVALUATION

CRITERIARATINGS EVIDENCE

Relevant FEEDBACK FROM PARTICIPANTS WHO FILLED POST-EVENT EVALUATION FORM (N=13)• Aligned with my current work requirements or needs (85%)• Knowledge I gained can be used to improve my work (77%)• Structured in a way that supported my learning style (100%)• Benefits of attending the seminar outweighed the time away from the office

(85%)• Will share the information I learnt at the seminar with colleagues (77%)• I met people who have the potential to be valuable in my work (100%)

SELECTION OF ADDITIONAL COMMENTS FROM PARTICIPANTS:• “I learn some new things in this presentation. I will use some of the things

learned to my work”• “The information presented was very useful for me as a practitioner and

researcher. The example gave me a clear idea on how important it is to consider these issues”

• “I gained great benefit from the workshop program and specifically from the role-playing exercise”

OTHER EVIDENCE OF SUCCESS• Since the training, participant X and Y have re-designed their program to

incorporate the ideas and practices shared in the event• Participant Z wrote the following, unsolicited e-mail: “I learned a lot from the

event and was wondering if FSI will be offering a follow-on training course”

Timely

Appropriately designed

Involve Australian and/or in-country partners and their networksWorthwhile

Enhance knowledge-exchange

Enhance learning

Enhancenetworking

OVERALL RATING – GOOD

A series of training events on private-public partnerships and Theories of Change have been delivered and well received. There is a growing demand from in-country programs for similar events and 3 are being planned in the next quarter.

SOURCES OF EVIDENCE:• Participants’ responses to evaluation feedback form• Reflections from FSI team member leading event • Feedback from collaborators and FSI steering committee

= Excellent

= Good

= Adequate

= Insufficient information

Rubrics is one of 3 complimentary MEL approachesREPORTING APPROACH/TOOL

M, E, or L? WHAT? FREQUENCY & FORMAT

PRIMARY PURPOSE PRIMARY AUDIENCES

FSI This Past Month Monitoring • ‘Live’ tracking of FSI activities and outputs

• Approx. every month• Succinct e-mail

message

• Evidence-base for FSI Steering Committee & funders

• Accountability• Sharing where everyone

is at• Day-to-day management

Internal (FSI team andimmediate partners)

FSI Rubrics Reports • Evaluation• Learning

• Evaluation of completed FSI activities and outputs

• Reflections of processes,and progress towards short and medium-term outcomes

• Deviations from plan; wins & challenges

• Lessons learned

• Approx. every 3-4 months: FSI Steering Committee meetings

• Succinct reportcomprised of rubrics tables & narratives

• Evidence-base for FSI Steering Committee & funders

• Accountability• Critical decisions and

directions (management)

Internal (as above)

FSI Practice Notes Learning • Practice-based experiences and lessons learned

• As ‘critical mass’ of practices and associated lessons learned gathered

• Succinct, reader-friendly ‘notes’

• Synthesis and sharing of key lessons learned and insights from practice

Internal and externalaudiences

Reflections and lessons learned

Learning from our experience with MEL• It takes considerable time• No definitive, perfect MEL

system• ‘Fit-for-purpose’ key• Flexibility and adaptability• Light, easy to communicate• Meet people where they are at• Involve the people/groups

whom will make use of information

• Science and practice balance

Limestone Avenue, Campbell ACT 2601 AustraliaTEL. +61 2 6276 6621hello@foodsystemsinnovation.org.auwww.foodsystemsinnovation.org.au

• ADD BUSINESS UNIT/FLAGSHIP NAME

Thank youFSI leader: Andy Hall, [email protected]

MEL coordinator: Samantha [email protected]