Monitoring & Evaluation System“Learning to Improve”

Making evidence work for development

Contents• Rationale of the M&E• Users and needs • Principles of the M&E• Units of analysis and dimensions of study• Elements of the M&E• Products of the M&E• M&E share of responsibilities• M&E constraints • M&E linkages: KM, decision making and learning

Rationale: Why should the MDGF have a M&E system?

• It is a requirement included in the legal agreement between the Donor (Spain) and UNDP

• It is an obligation, included in all signed joint programs

• It is necessary: if we want, scaling up programs into policies and spread solutions to achieve MDGs at Global level

• It is useful: As it is part of the program managing cycle and is the best way to measure progress, detect problems, correct them, improve performance and learn at local and global level

Who are the users of the M&E system

what are their needs in terms of Information?

For 5 minutes discussion:

Are we missing anyone?

Are most of the views of stakeholders included?

This is an attempt to let your tacit knowledge flow

What kind of principles should the MDGF’s M&E system incorporate?

PRINCIPLES

What kind of principles should the MDGF’s M&E system incorporate?

• Accordance with UNEG and DAC/OECD standards

• Oriented to well balanced learning and accountability purposes

• Evidence-based: consistent data, information or knowledge to support judgments and conclusions of monitoring and evaluation

What kind of principles should the MDGF’s M&E system incorporate?

• Built on an aggregation scheme: Elements of M&E (indicators + evaluations, etc) in lower levels add up in higher level of inquire

• Measure (DELTA), describe, analyze understand the object of study (JP+C+W+MDGs) and use results to improve program and policy performance

What dimensions should the MDGF M&E system cover?

What questions should it answer?

Joint Programs Countries

1st M&E LevelJoint programs

2nd M&E LevelCountries

Environment and Climate Change

Gender Equality Women’s empowerment

Culture & development

Economic democratic Governance

Youth, Employment and Migration

Conflict prevention & Peace Building

Children Food Security and Nutrition

Development and the Private Sector

3th M&E LevelWindows

Monitoring AspectsInput-Products-Results-ProcessesEvaluation Dimensions-Quality of the program formulation-Program objectives attained-Contribution to MDGs &other development indicators, gender-Replication: Scale up-Innovation -UN system Coordination-Delivering as One -Ownership-Alignment-Harmonization-MFDR-Mutual Accountability-Delta change, effects incitizens’ life

Monitoring Aspects-Results (UNDAF+PRS)-Processes (Coordination)Evaluation DimensionsInduced effects on:

-MDG at country level-Other Development indicators-UN system Coordination -Delivering as one-Ownership-Alignment-Harmonization-MFDR-Mutual Accountability-UN country pilots

4th M&E LevelMDG Achievement Fund

Monitoring Aspects-ResultsEvaluation DimensionsQuality of partnership Spain/UNDP:Added Value as a mechanism to Progress MDGs achievementThe Secretariat’s role and added valueInduced effects on:-Linkage of windows and MDGs-Other Development indicators-UN system Coordination -Delivering as one-Ownership-Alignment-Harmonization-MFDR-Mutual Accountability-Effects on citizens' life

Joint Programs

Joint Programs

Countries

Countries

Monitoring Aspects-Results as an aggregate of the JP-HR/Gender/EnvironmentEvaluation DimensionsNew themes:: Culture & migrationInduced effects on:-MDG at country level-Other Development indicators(Peace and Culture)-UN system Coordination-Delivering as one -Ownership-Alignment-Harmonization-MFDR-Mutual Accountability

Please in 5 minutes

What are the 3 most burning questions an M&E system should it answer?

What elements comprise the MDGF M&E system• Monitoring indicators: to measure progress and trends in the short

and medium-term at the (inputs, activities, outputs, outcomes)• Field visits: to monitor JP in depth and prepare and manage

evaluations as well as disseminate results and provide feedback from recommendations

• Evaluations: to review programs and value the worth of the dimension of study JP, at country level, in thematic windows and the MDGF as a whole

• Meta-evaluations: to review the quality of the evaluations (JP + Country) conducted and produce robust evidence at window level to link these evidence to MDG achievement

• Desk reviews and data collection & analysis: from a variety of sources to contribute with information and knowledge to the M&E+KM system

What products will the MDGF M&E system create and offer?

• Field monitoring reports• Country monitoring reports• Mid-term evaluations reports• Impact evaluation reports• Country evaluations (case studies)reports• Meta-evaluations reports• In depth review reports • MDGF global reports (midterm + final)• Special activities under the M&E+ Information & Advocacy

focus country initiative

MDG F Achievement Fund

M&E USER Responsibilities to be taken Deliver the products

MDG-F

Secretariat

Support the UN country teams in the collection of the information and the reporting procedure for monitoring indicators and activities

Design 24 thematic and process indicators to feed MDGF M&E system.

Design generic terms of reference for mid-term evaluations.

Work with UN country teams customizing mid-term evaluation’s terms of reference

Design 8 meta evaluations studies 1 for each window of activity

Support and facilitate the ongoing evaluations, assuring quality and participation through all their phases: desk review, field work report drafting, publication and dissemination.

Assure that recommendations from monitoring system and evaluations mainstream the manager’s decisions and correct programs’ deviations, guarantee that knowledge created serves as a input for the knowledge management system.

Implement a calendar of field missions to carry out M&E activities

Elaborate a protocol guide to monitoring visits

Implement the calendar of field visits to carry out the above mentioned activities

The strategy of M&E for MDGF

Country monitoring reports+ special countries with high learning and knowledge transfer potential

The MDGF mid-term evidence-based report

Mid-term evaluations

Country evaluations

At least 3 Impact evaluations ( experimental or/and quasi-experimental)

meta evaluation studies

The MDGF evidence-based final report

MDG F Achievement Fund

M&E USER Responsibilities to be taken Deliver the following products

UN Country Team

Collect and report information on the following indicators:

On joint program implementation at 3 levels :(3 input, 3 output, 3 outcome) (best, medium and worst indicators, to observe trends) (quarterly reported)

3 indicators on joint program coordination (supplied by MDGF Secretariat) (quarterly reported)

3 thematic indicators per active window in the country (every six months, supplied by MDGF Secretariat)

Report on all Joint Program indicators designed by themselves ( annually)

Facilitate the M&E processes and activities according to the guidance and principles supplied by the MDGF Secretariat

Integrate recommendations from M&E to improve programs

Adapt the templates for TOR for final evaluations joint program and country evaluations as well as financing t final evaluations

Special activities under the initiative focus countries (Ethiopia, Mauritania, Morocco,Timor Leste, Philippines, Bosnia Herzegovina, Brazil, Ecuador, Honduras)

Monitoring Quarterly reports

Final evaluations

M&E framework activities

Report in coordination indicators

And thematic indicators

MDG F Achievement Fund M&E Timeframe

Activities 2009 Jan Feb Mar Apr May Jun jul Aug Sep Oct Nov DecM&E Document finishedSocialization process of the M&E strategy: consensus reachedDesign of common indicators: 24 thematic + 3 on coordinationSelect 9 pilot countries for M&E and communication purposesSpecific m&E activities designed and implemented9 Pilot country monitoring reportsselect and design 3 impact evaluationsConduct baseline study for impact evaluationDraft M&E Guidelines document elaborated M&E Guidelines document testingM&E Guideline document in operationTraining sessions for Secretariat Collaborative work with MDTF for monitoring and reportingCollection of information on indicatorsReview of the JP quaterly reportsField visits for monitoring and evaluationElaborate general terms of reference template for mid-term evaluationsElaborate customized terms of reference template 5 mid-term evaluationsProcurement process for 2009 midterm evaluations consultants2 regional Workshops intorducing M&E and trainingActivities 2010 Jan Feb Mar Apr May Jun jul Aug Sep Oct Nov DecElaborate customized terms of reference for midterm evaluationsHire consultants and procurement processmanagement and coordination of Mid-term evaluations 5management and coordination of Mid-term evaluations 4management and coordination of Mid-term evaluations 10management and coordination of Mid-term evaluations 6management and coordination of Mid-term evaluations 3management and coordination of Mid-term evaluations 3management and coordination of Mid-term evaluations 7management and coordination of Mid-term evaluations 7management and coordination of Mid-term evaluations 6management and coordination of Mid-term evaluations 8management and coordination of Mid-term evaluations 66 evaluations started and 45 reports are completed 7Colborative work with MDTFCollection of information on common indicatorsConduct a second baseline study for impact evaluationReview of the JP quaterly reportsField visits for monitoring and evaluationKnowledge management activities, compilation, digestion and disseminationDesign and conduct the meta anlaysis evaluation round 1Meta analysis reportThematic reports on the windows round 1Pilot Country monitoring reportsEvidence based mid-term report of the MDGF FundActivities 2011 Jan Feb Mar Apr May Jun jul Aug Sep Oct Nov DecElaborate customized terms of reference for midterm evaluationsHire consultants and procurement processmanagement and coordination of Mid-term evaluations 2management and coordination of Mid-term evaluations 7management and coordination of Mid-term evaluations 10management and coordination of Mid-term evaluations 10management and coordination of Mid-term evaluations 10management and coordination of Mid-term evaluations 12management and coordination of Mid-term evaluations 5management and coordination of Mid-term evaluations 3management and coordination of Mid-term evaluationsmanagement and coordination of Mid-term evaluationsmanagement and coordination of Mid-term evaluationsmanagement and coordination of Mid-term evaluationsCollaborative work with MDTFCollection of information on common indicators

59 evaluations started and 80 reports are completed

real data as december 2008

What are the constraints we would face in such a enterprise? budget, time, data, political constraints

• What level of involvement and what work load can the staff from the JP take?

• Organizational issues, Who is responsible for what part of the system?

• Joint program, for real? Do joint programs have the necessary coordination mechanisms to allow for joint monitoring and evaluation?

• Will the users of the M&E system willing to collaborate to implement it ?

• Is data available at a reasonable cost and on time?• Will we have the political support to implement it?

M&E linkages: KM, decision making and learning• M&E is an enormous source of knowledge and will generate an

extraordinary amount of valuable explicit and tacit knowledge as well as organizational and general knowledge that will have to be delivered at the right moment, to the right persons in the adequate format for the purposes needed.

• KM system could also become a source of data, information and

knowledge for M&E system, is a reciprocal 2 way relation. • The ultimate goal of evaluation is recommend actions to improve

decision making and improve performance of programs and policies so all evaluations should focused on utilization of their findings and recommendations