MONITORING & EVALUATION: A MANAGEMENT TOOL

Presented by

Nina Blöndal Impact Evaluation Specialist

nina@endeleza.com Guangzhou, PRC 19-21 June 2013

This presentation

1.  Monitoring & Evaluation: What, when & why?

2.  Results-Based M&E: Placing M&E within the public management cycle

3.  Improving policy through M&E: Country case studies

PART ONE Monitoring & Evaluation:

What, when & why?

But often it is:

•  Not well understood •  Not well done •  Not used for project management •  Not used for policy-making ➞ That looks an aweful lot like wasted resources!

There’s a lot of talk about M & E…

It’s all about measuring results •  If you don’t measure your results, how can you tell success from failure?

•  If you can’t tell success from failure

- how can you reward success? - how do you know you’re not rewarding failure? - how can you learn? - How can you demonstrate results for public support?

(Inspired by Kusek & Rist, 2004).

CASE STUDY: The FARM Project, South Sudan

CASE STUDY: The FARM Project, South Sudan •  Goal: Improve agricultural sector productivity and

marketing in Greenbelt Area through support for Farmer Based Organizations

•  Components: 1) Agricultural Productivity

-  Grants/seeds -  Training and demonstration

2) Agricultural Trade -  Linking farmers to traders

3) Capacity Building

-  Post-harvest storage, pest management, business skills

CASE STUDY: The FARM Project, South Sudan

• Expected Impacts

→ Increased agricultural productivity in project regions → Increased food security → Increased family health and nutrition → Expansion of market led agriculture → Increased small-holder business profitability/farmer

incomes

M… for what?

Monitoring

→  “A continuing function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds” (OECD DAC)

Monitoring Gives ongoing information via select indicators, on the direction of change, the pace of change, and the magnitude of change. It can also identify unanticipated changes. → Are targets being reached? → Is the program/policy/country on track?

Monitoring Can happen at different levels → Project/program → Sector → National / sub-national

Monitoring → Program specific indicators → National development indicators

• Poverty Reduction Strategies • Millennium Development Goals • Cross-cutting issues (gender, environment)

What can we monitor? • Monitoring may include indicators for: →  Spending

→  Inputs

→  Activities

→  Outputs

→  Outcomes

What can we monitor? • 

Examples of project level monitoring indicators: FARM Project • Component 1

-  Number of farmers, processors, others who adopted new technologies

-  Hectares under improved technologies -  Number of individuals that have received training, by sex

• Component 2 -  Number of producer organizations receiving assistance -  Number of women farmers organizations assisted

• Component 3 -  Number of training events -  Number of individuals trained

Examples of national level monitoring indicators – UNAIDS

1. Indicators of national commitment and action →  E.g. Amount of national funds spent on HIV/AIDS

2. Indicators of national programme and behaviour →  Percentage of large enterprises/companies that have HIV/

AIDS workplace policies and programme →  E.g. Percentage of patients with STIs at health-care

facilities who are appropriately diagnosed, treated and counselled

3. National-level programme impact → E.g. Percentage of young people aged 15–24 who are HIV-

infected

Who monitors? • Monitoring can be anchored and carried out at various institutional levels →  National M&E agency →  National statistics agency →  Line Ministries →  Local government →  Project staff →  Third sector →  Donor agencies →  A combination of these

• 

Monitoring - what’s it good for? • Critical to knowing whether policies, programs, and projects are moving in the intended direction.

• Building a monitoring system to continuously track performance ought to be essential for managers.

•  If used properly, can improve service delivery and provide data for evaluation

Monitoring - challenges • Strategy: What to monitor – and why? • Capacity: Building capacity for data collection.

•  Incentives: Making it happen – collecting the data. • Building on what’s there: Integrating information between different agencies/ministries/organizations

• Using the data – returning to why.

Monitoring - challenges

Monitoring - what doesn’t it tell us? Relevance

Implementation Efficiency

Cost-effectiveness Causality

Attribution Sustainability

Monitoring

How is monitoring used in your country?

Checklist?

Management? Not used?

Well-defined system and

use?

Evaluation

→  “The systematic and objective assessment of an on-going or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfillment of objectives, development efficiency, evaluation effectiveness, impact and sustainability” (OECD DAC, 2010)

Evaluation → “In contrast to monitoring, evaluations are carried out at discrete points in time and often seek an outside perspective from technical experts” (World Bank, 2011)

→ Evaluation assesses results achieved and informs succeeding plans (APCoP, 2011)

Process Evaluation

→  “A study of the implementation of a program or policy focusing on inputs, activities, and outputs, and often used to assess the performance of implementers.”

→ Often focussed on the internal dynamics of implementing organizations, policy instruments, service delivery mechanisms, management practices, and the linkages among these.

Process Evaluation → Often asks:

• Whether services and goals are properly aligned • Whether services are delivered as intended to the

appropriate recipients.

• How well service delivery is organized.

•  The effectiveness of program management. • How efficiently program resources are used.

Program Evaluation

→  “A study, using research methods to collect and analyze data of the outcome of a program, that is whether and how well a program is working”

→ Examines whether specific objectives were reached within a given budget and time period.

Performance Evaluation “Mixing it up - process and outcome focus”: •  What a project or program has achieved. •  How it is being implemented •  How it is perceived and valued; •  Whether expected results are occurring; •  Other questions pertinent to program design, management,

and operational decision making •  Often include before and after comparisons, but generally lack

a rigorously defined counterfactual.

Impact Evaluation

→  An evaluation, which examines the final welfare outcomes of a project, program or policy.

→ A counterfactual analysis, which seeks to establishing causality and attribution.

→ Much more on this later today!

The FARM Project South Sudan: PE vs. IE – Performance Evaluation

•  To what extent did the project meet deliverables of the contract, incl results on performance indicators?

•  How cost effectively has the project implemented its

components? •  To what extent has FARM contributed to agricultural

productivtity, increased trade, increased capacity? •  To what extent has project engaged men and women equally

in project activities? •  How well has the project coordinated with local institutions

and other projects/organizations? •  Has the contractor managed implementation effectively?

The FARM Project South Sudan: PE vs. IE - Impact Evaluation •  Do FARM project activities improve livelihoods for FBO members

participating in the project? •  In particular, does the FARM project affect: Use of improved inputs,

Yields, Post-harvest losses, Area farmed in total and with improved practices, Sales, Income/expenditure, Nutrition, Time to market.

•  How do outcomes and impact vary according to the amount and combination of FARM support farmers receive? (E.g. amount and types of training, inputs, and grants)

•  To what extent are outcomes and impact affected by access to

markets, roads, and towns? •  How do outcomes and impact vary by socio-economic status, gender,

and geographic location?

Complementarity of M & E •  The main difference between monitoring and evaluation is

their timing and focus of assessment. • Monitoring and evaluation are integrally linked; monitoring

typically provides data for evaluation, and elements of evaluation (assessment) occur when monitoring.

• E.g. Monitoring looks at how many farmers were training

(what happened), but may also include post-training tests (assessments) on how well they were trained

• More monitoring than evaluation going on – but quality

and utility is not always high

Qualitative

vs Quantitative

Quantitative Evaluation • Large numbers – statistics. • Household surveys, business surveys, teacher surveys, health staff surveys.

• Answers are coded, personal accounts excluded

• Primary or secondary data – or both.

Quantitative Evaluation

Qualitative Evaluation Narrative, visual, participatory

Qualitative Evaluation • Fewer observations – no statistical validity • Data sources can include

• Historical records, administrative reports, letters, legal documents

• Media (print, radio, TV) • Open-ended responses to survey questions • Observations (ethnography) •  Interviews – key informants, focus groups • Participatory approaches

Mixed methods • Using both quantitative and qualitative evaluation methods together

• Different methods help to explain different elements of the causal chain

• Triangulation

Triangulation

The FARM Project South Sudan: Mixed methods

• Quantitative →  HH survey →  Community survey →  Key informant survey

• Qualitative → Focus groups → Homestead visits → Field visits → Walking & talk

The FARM Project South Sudan: Mixed methods – Quantitative gender info

The FARM Project South Sudan: Mixed methods – Qualitative gender info The group insisted, and it was indeed observed prior to sitting down and interviewing the group, that men and women shared FBO farm tasks equally. But a clear line was drawn between the FBO social space and the domestic social space. “At home, the work is different,” a man said, explaining that he was willing to do weeding for the FBO but not at home, where he and his wife—both FBO members—used more traditional methods. When asked why, his wife said, “It depends on the deal you have with your husband,” and the whole group burst out laughing, nodding in agreement. A few women in the group shook their heads and said if their husbands didn’t help like they did in the FBO, “I wouldn’t feed him….”

M&E – Summing up

• It’s all about measuring results

• Monitoring and evaluation are two separate but interrelated strategies to collect data and report the findings on how well (or not) the public sector is performing

• If used strategically they can be powerful management and policy-making tools.

PART TWO Results-Based M&E:

Placing M&E within the public management cycle

A good Results-Based M&E system is central to evidence-based policy-making

Results-Based Framework

”A results framework is an explicit articulation (graphic display, matrix, or summary) of the

different levels, or chains, of results expected from a particular intervention—project, program, or

development strategy”

Results-Based M&E 1.  A management system 2.  A performance reporting system 3.  Essentially about laying out a stategy for

measuring the right things, measuring them well, and using the results for better management and better policy.

Results-Based M&E

Results-Based M&E Often used in national level context:

”A results-oriented framework refers to a country-level results-based monitoring and evaluation (M&E) system that, building on sound statistical data and open access to information, produces

data on progress toward desired inputs, outputs, and outcomes that are identified in the national

development strategy” (World Bank, 2007)

Results-Based M&E: 10 steps • 

Results-Based M&E: 10 steps • 

Results-Based M&E: 10 steps • 

Results-Based M&E: 10 steps • 

Results-Based M&E: 10 steps • 

Results-Based M&E: 10 steps • 

Results-Based M&E: 10 steps • 

Results-Based M&E: 10 steps • 

PART THREE Results-Based M&E in practice

Case study: Australia •  Late 1980s the government created a whole-of-

government evaluation system, managed by the Department of Finance (DoF).

• All ministries were required to evaluate each of their

programs every three to five years. • Evaluations were conducted by the line ministries

themselves, but they were overviewed by the DoF and other central departments.

Case study: Australia •  Focus mainly on evaluation - defined as a form of

disciplined inquiry, including rapid evaluations, formal policy reviews, rigorous impact evaluations, and performance audits conducted by the national audit office.

• By 1994, almost 80 percent of new spending proposals

relied on evaluation findings, usually to a significant degree.

•  About two-thirds of savings options also relied on

evaluation findings.

• System replaced by a weaker performance monitoring framework in 1997 with a change of government.

Case study: Colombia •  The government’s M&E system, SINERGIA was set up in

1991 and is managed by the Department of National Planning (DNP).

• Main component: → A performance information database containing 500 performance

indicators to track the government’s performance against all of the 320 presidential goals → Where performance targets are not met, the manager responsible

for meeting the target is required to prepare a statement explaining this underperformance. → The president uses this information, in his monthly management

control meetings with each minister and in his weekly town hall meetings in municipalities around the country

Case study: Colombia • Second main component: → Series of evaluations → The three main types of evaluation are rigorous impact

evaluations, “institutional” evaluations, and “management” evaluations—the latter two focus on management and process issues.

•  Final components are weaker and much less fully

institutionalized - including: →  An effort to partner with civil society →  Preparation of performance budget reports to the Congress. They

present government spending on a programmatic basis and report the available M&E information on the performance of these programs. But weak implementation.

Mexico: A centralized evaluation agency •  New Institutional Framework 2006 •  Creation of the National Evaluation Council (CONEVAL):

Autonomous institution in charge of poverty measurement and social policy and program evaluation

•  Performance Evaluation System: Formalizes linkage between

planning, budgeting, and monitoring and evaluation

•  New Evaluation Guidelines •  Better institutional coordination (Ministry of Finance, Ministry of Public

Management and CONEVAL) •  Institutionalizes use of logframes (results matrix) •  More realistic evaluation requirements •  Multi-year evaluation agendas

Mexico: A centralized evaluation agency

Mexico: A centralized evaluation agency

• Program manager together with evaluation and planning and budget units develop action plan to address evaluation recommendations at the program and institutional (ministry) level.

• CONEVAL prepares a report to Congress on evaluation results and government-wide recommendations

South Africa : A Government-wide Monitoring and Evaluation System • Major shift in emphasis concerning M&E since 2009: Review of 14 international M&E systems: Ministry of Performance M&E created in 2009, and the Department of Performance M&E (DPME) in January 2010

• Focus on a limited number of cross-government outcomes in attempt to focus efforts to promote change

• Moving government towards an impact focus, rather than an ”activity” focus.

South Africa : A Government-wide Monitoring and Evaluation System

• Regular monitoring and reporting to Cabinet of performance on delivery agreements and progress.

• DPME increasingly involved in the budget process: budget guidelines ask departments to include links with the delivery agreements in their plans

• But many challenges remain - system not yet institutionalized.

Improving policy through Results-based M&E

M&E must be driven by a demand for improved public sector performance

”In countries where there is no drive for improved public sector performance, there will be no

motivation for M&E. In such cases, a more basic examination of institutional conditions for public

sector reform is required” (Shepherd, 2011)

Improving policy through Results-based M&E ➞  Management and maintenance of M&E systems require

creating the right incentives and providing sufficient financial, human, and technical resources for organizations, managers, and staff to carry out monitoring tasks.

➞  Individual and organizational responsibilities should be

delineated, and a clear “line of sight” established—meaning that staff and organizations should understand their connections to common goals. Clear relationships need to be established between actions and results.

➞ Individuals and organizations need to understand how

their specific tasks contribute to the big picture.

Incentives

Resources

Responsibility

The Big Picture

Improving policy through Results-based M&E ➞ Management and maintenance of M&E systems require

creating the right incentives and providing sufficient financial, human, and technical resources for organizations, managers, and staff to carry out monitoring tasks.

➞  Individual and organizational responsibilities should be

delineated, and a clear “line of sight” established—meaning that staff and organizations should understand their connections to common goals. Clear relationships need to be established between actions and results.

➞ Individuals and organizations need to understand how

their specific tasks contribute to the big picture

Questions?

Thank you!

References •  Asia-Pacific Community of Practice on Managing for Development

Results (2011) Framework for Results-Based Public Sector Management.

•  Brown, A. (2012) Evaluation in the M&E context. Presentation for SHIPDET Special Topic Course on Evaluation. Kunming, PRC.

•  Capuno, J. (2012) Introduction to Evaluation Techniques. Presentation for AFDC-APCoP-ADB Special Course on Results-based Planning, Budgeting and Evaluation, Urumqi, PRC.

•  Castro, Manuel Fernando, Gladys Lopez-Acevedo, Gita Beker Busjeet, and Ximena Fernandez Ordonez. 2009. “Mexico’s M&E System: Scaling Up from the Sectoral to the National Level.” Independent Evaluation Group, ECD Working Paper Series 20, World Bank, Washington, DC

References •  Engela, Ronette, and Tania Ajam (2010). Implementing a

Governmentwide Monitoring and Evaluation System in South Africa. Independent Evaluation Group, ECD Working Paper Series 21, World Bank, Washington D.C.

•  Kusek & Rist (2004) Ten steps to a results-based monitoring and evaluation system : a handbook for development practitioners, World Bank, Washington D.C.

•  OECD DAC (2010). Glossary of Key Terms in Evauation and Results Based Management.

•  Shepherd, Geoffrey (2011). Conducting Diagnoses of M&E Systems and Capacities, PREM Notes Number 16, World Bank, Washington D.C.

•  Soto, G. (2010) Integrating Evaluation into Decision Making: The Mexico Experience. Presentation at World Bank Regional Impact Evaluation Workshop, Ghana.