A flexible, multi-purpose monitoring, evaluation, and learning system to support Australian agriculture international aid and development programsSamantha Stone-Jovicich, Andy Hall, Jennifer Kelly, Michaela Cosijn, and Lucy Carter (CSIRO) AES 2015 Conference, Melbourne, 7 September 2015
This presentation
• Journey • Reflections and lessons learnedo Concept to practiceo Scientists doing monitoring,
evaluation and learning
M&E in abstract
M&E in practice
The project context
Flexible, multi-purpose MEL system • Samantha Stone-Jovicich
FSI• 2012-2015
• PARTNERSHIP
o Department of Foreign Affairs and Trade (DFAT)
o Australian Centre for International Agricultural Research (ACIAR) and the Australian International Food Security Centre (AIFSC)
o Commonwealth Science and Industrial Research Organisation (CSIRO)
• IMPACT
• 3 THEMATIC AREAS
DFAT
ACIAR
Improve the impact of ODA-supported agriculture and food security programs in the Indo-Pacific region
Partners-in-country
(Africa, Asia)
Multiple partners, cross-scale
CSIRO
KEY
ACTI
VITI
ES &
OU
TPU
TS(o
rgan
ised
arou
nd 3
them
es: M
arke
ts &
Par
tner
ship
s;
Agric
ultu
ral F
ront
iers
; and
Man
agin
g fo
r Im
pact
)
PEOPLE with different skills &
knowledge
FINANCIAL support
CHAMPIONS from each of the key partner organisations
Formal LINKS/ACCESS to Australian-supported international development arenas, organisations, and individualsKE
Y RE
SOU
RCES
/IN
PUTS
Improved impact of ODA-supported food and agricultural programs in the
Indo-Pacific region
Learning events
What FSI is trying to achieve and how
Knowledge products
Expertise & practice networks
In-country engagement
Outreach & external visibility
FSI governance
Key partnering development programs and projects
Markets and Partnerships
Agricultural Frontiers
Managing for Impacts
FSI themes
INDONESIAARISA
COLOURED AREAS ON MAP = Regions and countries where ACIAR, AIFSRC, and DFAT work
= Agriculture and food security programs and projects engaged with FSI
INDIA/BANGLADESHSRIFSI
EAST AFRICAAAPP Program
TIMOR LESTETOMAK Program
CAMBODIAKAVAC Program
The challengesThe complexity of FSI The complexity of the
broader context
(Source: Batchelor et al. 2011)
New area for CSIRO
(Source: SIL International)
Our experience designing a MEL system for FSIA journey of moving from ‘best practice’ to ‘fit-for-purpose’
Flexible, multi-purpose MEL system • Samantha Stone-Jovicich
Food Systems Innovation for Food Security (FSIFS) Inception Project launched
The Food Systems Innovation (FSI) initiative launched
FSI mid-point review highlights lack of progress with MEL
July2012
May2014
Dec2015 FSI to end
July2013
FSIFS endsJune2013
Social learning
Indicator-based M&E
MEL
APP
ROAC
HES
AN
D T
OO
LS T
RIAL
LED
Realisation that a social learning approach was not most suitable approach
‘Fit-for-purpose’ MEL
KEY EVENTS AND CRITICAL JUNCTURES
Internal team meeting reflecting on achievements and challenges Results in ‘home-grown’ MEL approaches (FSI-at-a-Glance & FSI Learning Trajectories)
‘Home-grown’ MEL approaches
Adoption of M&E approach most familiar to our donors
Realisation that logframe, indicator-based M&E was not sufficient
Impasse with FSI-at-a-Glance and FSI Learning TrajectoriesShift to Rubrics and other complementary approachesDec
2014
What we tried‘Best-practice’: A social learning approach
Flexible, multi-purpose MEL system • Samantha Stone-Jovicich
“Organisations where people continually expand their capacity to create the results they truly desire, where new and expansive patterns of thinking are nurtured, where collective aspiration is set free, and where people
are continually learning how to learn together.” (Senge 1990: 3)
Seminars; facilitated discussions+
Series of reflection and learning activities, tailored to the culture, interests & needs of each partner*
BARRIERS WANT TO FOCUS ON/ABLE TO INFLUENCE
ACTIONS AND COMMITMENTS TO FOLLOW THROUGH
Narrow performance measures
Workforce planning
“Soft structures”
Dominant paradigms and assumptions
Project design and implementation
In-country partners and partnerships
* NOTE: was implemented only with CSIRO
Flexible, multi-purpose MEL system • Samantha Stone-Jovicich
What we tried‘Best-practice’: Indicators
We built on the program logic framework
THEMES Component 1: Analysis & Field Application
Component 2: Knowledge Management and Communications
Component 3: Capacity Building
Theme 1: Theory of Change and Livelihood Impact Pathways
• Activity 1 & outputs• Activity 2 & outputs• Etc.
Theme 2: Market-based Approaches and Public-Private Partnerships
Theme 3: Nutrition-sensitive Agriculture
FSI activities
and outputs
Aid program
outcomes
Development impacts
Outcome 1
Outcome 3
Impact 2 Impact 3
Outcome 2Outcome 4
Impact 1
Monitoring, Evaluation and Learning (MEL): key questions
AID PROGRAM OUTCOMES• What were the aid program outcomes from the FSI project? • Did the project achieve outcomes 1-3? Where there any additional
outcomes?
ACTIVITIES AND PROCESSES• How did the activities, processes, and outputs from each of the project
components, learning themes, and MEL contribute to the outcomes?• What are the key lessons learned that emerged?
DEVELOPMENT OUTCOMES• What were the development outcomes from the FSI project? • Did the project contribute to more effective, evidence-based agriculture,
food security and nutrition interventions? More food? Increased net incomes? Where there any additional outcomes?
FSI p
roje
ct (2
013-
15)
FSI p
roje
ct (2
013-
15)
Futu
re p
roje
ct(s
)
OUTPUTS• What were the outputs generated from the FSI project for each learning
theme? • In particular, what innovative solutions were identified and which were
integrated into projects?
ACTIVITIES AND PROCESSES• How did the activities and processes implemented contribute to those
outputs?• What are the key lessons learned that emerged during these activities and
processes?
…too complicated; too cumbersome; too rigid
THEMESCOMPONENT 1:
Analysis & Field ApplicationCOMPONENT 2:
Knowledge Management and CommunicationsCOMPONENT 3:Capacity Building
THEME 1
ACTIVITIES• Seminar 1• Desktop study• Etc.
OUTPUTS• 1 page Discussion • Report• Etc.
ACTIVITIES OUTPUTS ACTIVITIES OUTPUTS
THEME 2
THEME 3
FSI a
ctiv
ities
an
d ou
tput
s
Aid
prog
ram
ou
tcom
es, i
ndic
ator
s an
d M
EL to
ols f
or
mea
sure
men
t
Deve
lopm
ent
impa
cts
Outcome 1
Outcome 3
Impact 2 Impact 3
Outcome 4
Impact 1
INDICATORS MEASURES OF PERFORMANCE MEL TOOLS
Indicator 1 # of…. Survey
Indicator 2 Perceived changes of … Most Significant ChangeINDICATORS MEASURES OF
PERFORMANCEMEL TOOLS
Indicator 1 XXX XXX
Indicator 2 XXX XXX
Etc.
INDICATORS MEASURES OF PERFORMANCE
MEL TOOLS
Indicator 1 XXX XXX
Indicator 2 XXX XXX
Etc.
Outcome 2INDICATORS MEASURES OF
PERFORMANCEMEL TOOLS
Indicator 1 XXX XXX
Indicator 2 XXX XXX
Etc.
We developed detailed indicators and performance measures…
We attempted to simplify it into tables…
Indicators Means of verification MEL data collection tools Responsibility Presentation of results
Reporting timeframe
Information on food systems innovation is made available to Australian-funded policies, programs, and projects
Database of range of information (e.g. topics, technical info, methodology, etc.) made available
Content analyses of materials produced and made available through different FSI dissemination/communication channels
FSI teammember X
• Time series• Pie charts
Dec 2014May 2015
Etc.
OUTCOME 1: Knowledge about of food systems innovation for food security enhanced among FSI partners and collaborators engaged in Australian-funded policies, programs, and projects
What we tried‘Home grown’ approach: FSI-at-a-Glance and FSI Learning Trajectories
Flexible, multi-purpose MEL system • Samantha Stone-Jovicich
13%
7%
15%
9%
Outputs
Brokering of knowledge and collaborations
Lessons learnt
Trust-building
• Studies/reports• Dossiers• Working papers• Blogs and online dialogues• Guidelines• Practice Notes
FSI AT A GLANCE
FSI LEARNING TRAJECTORIES
FSI
INTAN
GIBLE STU
FF;PRO
CESSES; “INVISIBLE”
Engagement processes
• Seminars, brown paper bag talks• Facilitated workshops• Training modules• Roundtables• Networks
Emergent, unexpected
Balancing M&E of outputs and processes; incorporating learning
TANG
IBLE STUFF;
FORM
AL CO
MM
ITMEN
TS
NOTE: Information shown is indicative of types of outputs and processes monitored
Presentation title • Presenter name19 •
OVERARCHING OUTCOMES GOAL
Evidence, analysis and experience more effectively and routinely applied in agriculture and food security programming and strategy and practice:
Innovative and effective
agriculture and food security
programming, strategy and
practice
BROAD-LEVEL OUTPUTSSPECIFIC ACTIVITIES & OUTPUTS
FSI-AT-A-GLANCE EXAMPLE FOR AGRICULTURAL LINKAGES COMPONENT (Nutrition Sensitive Agriculture, July – Nov 2014)
• 1 invited presentation • Planning for Roundtable initiated
in collaboration with DFAT and USyd
• 1 Dossier completed• 2nd Dossier in preparation• NSA web homepage completed• 1 Discussion Brief• 1 Working Paper • 16 external resources
identified
• Presentation of FSI NSA insights at workshop that brings together group of Australian researchers PILOTED as mechanism for info exchange and networking and future capacity building opportunities
• NSA homepage, Discussion Brief 1, Dossier 1 and Working Paper 1 on FSI web for partners and other audiences to access
• Dossier 1 shared with DFAT, CSIRO and ACIAR
• 16 external resources shared (summaries and links on FSI web)
• Engagement with USyd Food & Nutrition Security Node
• Connections with NSA experts being established and maintained via multiple avenues
• Pool of NSA experts being identified, introduced to FSI, and invited to formally contribute
• FSI being increasingly recognized by partners as an expert in NSA, and contacted for advice and feedback
DFAT• Co-development of Operational
Guidance Note on NSA• Contribution to NSA policy wording in
AgFishWater Strategy• Guidance on description of FSI and NSA
in DFAT presentation at ICN2• Information provided on link b/w NSA
and trade
OVERSEAS PROGRAMS/PROJECTS• NSA advice and feedback to AAPP
program and other projects
ACIAR• Information provided on Nutrition
Foresighting with DAFF• Contribution to Vietnam nutrition
workshop program
• Engagement with DFAT Agricultural Productivity & Food Security Sector
• Engagement with ACIAR Agribusiness Program
• Engagement with AAPP Africa• Engagement with
Timor Leste desk
• Collaborations with DFAT and ACIAR, and associated overseas programs and projects, being established and specific activities/products being identified and advanced
ENG
AGEM
ENT
EXPE
RTIS
E &
PR
ACTI
CE N
ETW
ORK
SKN
OW
LED
GE
PRO
DU
CTS
LEAR
NIN
G
EVEN
TS
3
5 16
72
4
KNOWLEDGE PRODUCTS produced by FSILEARNING NOTES/DOSSIERS Dossier 1 Improving Nutrition through Agriculture Linkages Dossier 2 Food Security and the Double BurdenDISCUSSION BRIEFS Discussion Brief 1 Enhancing the nutritional impact of agricultural
investments for improved development outcomes
WORKING PAPERS Working Paper 1 Enhancing the nutritional-sensitivity of agricultural
development interventions in the Eastern Gangetic PlainsFSI WEBSITE ‘AGRICULTURE LINKAGES’ HOME PAGE Agricultural linkages: Strengthening the linkages between agriculture and
other development priorities
External KNOWLEDGE PRODUCTS sharedEXTERNAL PUBLICATIONS * FAO: Child and maternal nutrition FAO: The state of food and agriculture: food systems for better nutrition FAO & Agriculture-Nutrition CoP: Key recommendations for improving nutrition through
agriculture Global Forum for Rural Advisory Services (GFRAS) & World Bank: A synthesis of experiences,
lessons and recommendations
OUTCOMES: direct contributions to partners’ ag & food security programming/strategy/practice
DFAT (Canberra) Co-development of DFAT Operational Guidance Note on NSA Responding to request on how to incorporate NSA definition in AgFishWater Strategic Framework Responding to request on FSI and NSA wording for DFAT statement at ICN2 and on linking trade
and NSA ACIAR (Canberra) Responding to request for Nutrition Foresighting DAFF project Responding to request, input provided into program of Hanoi nutrition workshop programOVERSEAS PROGRAMS AND PROJECTS Technical advice and feedback to AAPP program (DFAT/CSIRO) on nutrition pathways in their
project designs
NETWORKS AND PLATFORMS * FAO: Global Forum on Food Security and Nutrition (FSN Forum) FAO: Improved Global Governance for Hunger Reduction Programme Scaling Up Nutrition (SUN) SecureNutrition Agriculture-Nutrition Community of Practice
FAO Improved Global Governance for Hunger Reduction Programme: Synthesis of guiding principles on agriculture programming for nutrition Rachel Turner et al. journal paper: Agriculture for improved nutrition: the current
research landscape (Food and Nutrition Bulletin) The Lancet journal: Executive summary on maternal and child nutrition World Bank: Improving nutrition through multisectoral approaches
Action against Hunger (ACF – International): Maximising the nutrition impact of food security and livelihoods interventions
World Bank: Prioritizing nutrition in agriculture and rural development: Guiding principles for operational investments
IFPRI: Working multisectorally in nutrition: principles, practices and case studies
EXPERTISE & PRACTICE NETWORKSU. OF SYDNEY Discussions with U of Syd Food and Nutrition Security NodeNSA-RELATED GLOBAL ALLIANCES/ORGANISATIONS Compilation of global alliances and organisations working on
NSA-related research and practice OTHER AVENUES Links being made with NSA experts via writing, workshops, peer reviews
LEARNING EVENTSSEMINARS, PRESENTATIONS University of Sydney Workshop (Nov 14, 2014) - Taking a
wider view: Health impacts on agricultural productivity in Southeast Asia (co-presentation with ACIAR and DFAT)
REGIONAL DIALOGUES/ROUNDTABLES Planning started for Roundtable Discussion on NSA with
leading nutrition sensitive agriculture experts in Australia and internationally
ENGAGEMENT with partnersDFAT Agricultural Productivity & Food Security SectorACIAR Agribusiness ProgramAAPP AfricaTIMOR LESTE DeskSRFSI
FSI-AT-A-GLANCEEXAMPLE FOR AGRICULTURAL LINKAGES COMPONENT (Nutrition Sensitive Agriculture, July – Nov 2014)
13%
7%
15%
9%
.
13%
7%
15%
9%
(For
mal
) out
puts
Supp
ortin
g (in
form
al) p
roce
sses
FSI A
CTIV
ITIE
S
TIMELINEFY
2013-14July 2014
OTH
ER
SUPP
ORI
NG
EVEN
TS
Aug 2014 Sept 2014 Oct 2014
FSI OU
TCOM
ES
FSI MEL reflection workshop
(May 2014)
June 2014 Nov 2014
Previous FSI discussions with
DFAT Timor Lestedesk about NSA
FSI MEL NSA Learning Trajectory for 2012-2014
International movement to bring NSA to the fore in development (has helped put NSA in the spotlight in Australia)
FSI Work Plan process FSI invited to review manuscript in Food and
Nutrition Bulletin
Request from ACIAR
Published on FSI website
FSI-NSA presentation at U Sydney Workshop
FSI-NSA meeting w/ACIAR Agribusiness
Program
ACIAR Nutrition workshop program for
Vietnam (Hanoi, Nov 2014)ACIAR-DAFF Nutrition
Foresighting web tool POTENTIAL: input on strategic
approach to NSA in ACIAR
WORKING PAPER 1
Contribution to post-
workshop position paper
UNIVERSITY OF SYDNEY Input into development of the
Food Security & Nutrition Security Node
FSI-NSA meeting
w/DFAT Ag & FS Sector
DFAT- Ag & Food Security SectorOperational Guidance NoteGuidance on linkages b/w NSA
and trade DFAT- Whole of ODA StrategyNSA policy wording for the
AgWaterFish Strategy
FSI NSA-related engagement
w/AAPP
Report produced for AAPP
FSI-NSA meeting with DFAT Timor
Leste desk
DFAT Ag & FS Sectorprovides feedback
and distributes internally
FSI-NSA linkages established w/ U of
Sydney Food & Nutrition Security Node
DFAT- Timor Leste Program POTENTIAL: NSA input - TOMAK
DFAT- AAPP (Africa) NSA advice to AAPP
DOSSIER 1
Draft completed
FSI Partnership
Building Workshop
FSI contributing to (as of August 2014):
Re-engagement w/DFAT Timor
Leste desk
= knowledgeproducts
= learning events = engagement
with FSI partners = fostering of
networks with experts
= partner outcomes = other outcomes
LEGEND
NSA = nutrition sensitiveagriculture
WORKING PAPER 1: Enhancing NSA interventions in the Eastern Gangetic Plains
DOSSIER 1: Improving Nutrition through Agriculture Linkages
FSI LEARNING TRAJECTORIESEXAMPLE FOR AGRICULTURAL LINKAGES COMPONENT (Nutrition Sensitive Agriculture, July – Nov 2014)
What we tried‘Fit-for-purpose’: A rubrics approach & more
Flexible, multi-purpose MEL system • Samantha Stone-Jovicich
INTERMEDIATE OUTCOME
GOALSPRIMARY OUTCOMES
Relevant, timely, appropriately-designed learning events that involve Australian and/or in-country and partners and their networks, and are perceived as worthwhile
An expanded range of relevant experts and development practitioners who actively contribute to FSI Australian and in-country partners’ and their networks’ international development discussions, designs, and practices in ways that are perceived as collaborative, salient, credible and useful
Activities and a web-based platform that bring together analyses, opinions, lessons, experiences and capacity building resources derived from FSI and its Australian and regional partners; are easily accessed and easy to navigate; and audience-appropriate and reader-friendly; and updated on a regular and in a timely fashion
Improved impact of ODA-supported food and agricultural programs in the Indo-Pacific region
FSI recognised internationally as a trusted and credible source of analysis and operational guidance on practices and policies that promote food systems innovation
Knowledge products are relevant, practice-based and practice-oriented, and aligned with current and emerging (ag, food, and nutrition) international development thinking, practices, and needs within Australia and overseas; collaboratively-created, reader-friendly and audience-appropriate, and produced and delivered in a timely manner
FSI creates opportunities for in-country partners to participate in reflection and learning in food systems innovation
FSI key partners (DFAT, ACIAR, CSIRO) work together collaboratively, respecting the agreed-upon partnership principles, to collectively learn and manage FSI in a responsive and agile manner
KEY ACTIVITIES & OUTPUTS
organized around Markets and Partnerships, Agriculture Linkages,
and Managing for Impact
LEARNING EVENTS
KNOWLEDGE PRODUCTS
EXPERTISE & PRACTICE
NETWORKS
FSI GOVERNANCE
OUTREACH &EXTERNAL VISIBILITY
IN-COUNTRY ENGAGEMENT
Enhanced knowledge-exchange; learning; and networking among FSI partners and other stakeholders in Australia and overseas, thereby strengthening capacity to progress food systems innovation
Innovation in ODA-supported food and agricultural program design and implementation
Rubrics approach focused on the outcomes
LEARNINGEVENTS
EVALUATION CRITERIA & RATINGS
Rele
vant
Tim
ely
Appr
opria
tely
de
signe
d
Invo
lve
part
ners
Wor
thw
hile
Enha
nce
kn
owle
dge
Enha
nce
lear
ning
Enha
nce
netw
orki
ng
Workshop 1
Seminar
Lunch seminar
Reflection event
Workshop 2
Training event 1
Training event 2
Presentation
OVERALL
FSI RUBRICS APPROACH: An example for Learning Events (fabricated data)
FSI OUTCOME: Learning events are relevant, timely, appropriately designed; involve Australian and/or in-country partners and their networks; are perceived as worthwhile; and are effective in progressing FSI’s primary outcome (enhanced knowledge-exchange, learning, and networking among FSI partners and other stakeholders in Australia and overseas, thereby strengthening capacity to progress food systems innovation)
3. SUMMARY REPORT2. SUMMARY RUBRICS OF ALL EVENTS1. INDIVIDUAL RUBRICS FOR EACH EVENTEVALUATION
CRITERIARATINGS EVIDENCE
Relevant FEEDBACK FROM PARTICIPANTS WHO FILLED POST-EVENT EVALUATION FORM (N=13)• Aligned with my current work requirements or needs (85%)• Knowledge I gained can be used to improve my work (77%)• Structured in a way that supported my learning style (100%)• Benefits of attending the seminar outweighed the time away from the office
(85%)• Will share the information I learnt at the seminar with colleagues (77%)• I met people who have the potential to be valuable in my work (100%)
SELECTION OF ADDITIONAL COMMENTS FROM PARTICIPANTS:• “I learn some new things in this presentation. I will use some of the things
learned to my work”• “The information presented was very useful for me as a practitioner and
researcher. The example gave me a clear idea on how important it is to consider these issues”
• “I gained great benefit from the workshop program and specifically from the role-playing exercise”
OTHER EVIDENCE OF SUCCESS• Since the training, participant X and Y have re-designed their program to
incorporate the ideas and practices shared in the event• Participant Z wrote the following, unsolicited e-mail: “I learned a lot from the
event and was wondering if FSI will be offering a follow-on training course”
Timely
Appropriately designed
Involve Australian and/or in-country partners and their networksWorthwhile
Enhance knowledge-exchange
Enhance learning
Enhancenetworking
OVERALL RATING – GOOD
A series of training events on private-public partnerships and Theories of Change have been delivered and well received. There is a growing demand from in-country programs for similar events and 3 are being planned in the next quarter.
SOURCES OF EVIDENCE:• Participants’ responses to evaluation feedback form• Reflections from FSI team member leading event • Feedback from collaborators and FSI steering committee
= Excellent
= Good
= Adequate
= Insufficient information
Rubrics is one of 3 complimentary MEL approachesREPORTING APPROACH/TOOL
M, E, or L? WHAT? FREQUENCY & FORMAT
PRIMARY PURPOSE PRIMARY AUDIENCES
FSI This Past Month Monitoring • ‘Live’ tracking of FSI activities and outputs
• Approx. every month• Succinct e-mail
message
• Evidence-base for FSI Steering Committee & funders
• Accountability• Sharing where everyone
is at• Day-to-day management
Internal (FSI team andimmediate partners)
FSI Rubrics Reports • Evaluation• Learning
• Evaluation of completed FSI activities and outputs
• Reflections of processes,and progress towards short and medium-term outcomes
• Deviations from plan; wins & challenges
• Lessons learned
• Approx. every 3-4 months: FSI Steering Committee meetings
• Succinct reportcomprised of rubrics tables & narratives
• Evidence-base for FSI Steering Committee & funders
• Accountability• Critical decisions and
directions (management)
Internal (as above)
FSI Practice Notes Learning • Practice-based experiences and lessons learned
• As ‘critical mass’ of practices and associated lessons learned gathered
• Succinct, reader-friendly ‘notes’
• Synthesis and sharing of key lessons learned and insights from practice
Internal and externalaudiences
Reflections and lessons learned
Learning from our experience with MEL• It takes considerable time• No definitive, perfect MEL
system• ‘Fit-for-purpose’ key• Flexibility and adaptability• Light, easy to communicate• Meet people where they are at• Involve the people/groups
whom will make use of information
• Science and practice balance
Limestone Avenue, Campbell ACT 2601 AustraliaTEL. +61 2 6276 6621hello@foodsystemsinnovation.org.auwww.foodsystemsinnovation.org.au
• ADD BUSINESS UNIT/FLAGSHIP NAME
Thank youFSI leader: Andy Hall, [email protected]
MEL coordinator: Samantha [email protected]