carrying out a monitoring and evaluation diagnostic study geoffrey shepherd april 28, 2011 1
TRANSCRIPT
1
Carrying out a Monitoring and Evaluation Diagnostic Study Geoffrey Shepherd
April 28, 2011
2
Introduction: the role and organization of M&E
• M&E provides information allowing us to judge the efficiency and effectiveness of public actions and their consistency with other public actions.
• An M&E system is a set of linked activities to carry out the regular, systematic, and planned monitoring and evaluation of government activities.
• M&E is a building block of a performance-oriented (or development-oriented) policy cycle.
3
What is an M&E diagnosis?
• An M&E diagnosis is an analysis of what is and is not working in a country’s (or a ministry’s, or a sub-national government’s) M&E activities, with recommendations for improving activities and systems.
• The diagnosis primarily involves:– Institutional analysis: what are the rules and practices
that drive demand for M&E?– Analysis of technical capacity: what are the data-
collection practices, analytical instruments, and skills that supply the M&E?
4
Constructing a checklist for a diagnosis
• Different country situations (levels of development, institutional attributes, etc.) mean that there is no single blueprint for a “good” M&E system or for preparing a diagnostic study.
• On the other hand, different studies of national M&E systems cover similar topics.
• I have derived a checklist of topics from: – existing diagnostic guides; and – a review of recent M&E cases studies and diagnoses.
• I shall illustrate some of the main practical issues under each topic with brief reference to country cases.
5
A checklist: nine topics for preparing an M&E diagnosis
Blocks TopicsA. The national environment for M&E
A.1 The national policy and institutional framework
B. M&E systems(in principle, each important M&E system needs to be separately analyzed along these dimensions)
B.1 Historical developmentB.2 ObjectivesB.3 Processes, tools, and productsB.4 Relationship with other systemsB.5 The institutional architectureB.6 Characteristics of organizationsB.7 Results
C. Findings C.1 Conclusions and recommendations
6
Topic A.1. The national policy and institutional framework
The questions
How are policies made? (- and what role do donors play?) is political power wielded in the public interest? do policies create a demand for M&E? how decentralized is the country? how has the relevant policy environment evolved over time?
The main issue
Identifying the “big” drivers of M&E (a central topic): “Performance” regimes: Chile, Mexico, Australia,
Canada, UK, US. Poverty-reduction agenda: Uganda, South Africa. Linking central and sub-national governments:
South Africa, Brazil.
7
Topic B.1. M&E systems: historical development
The questions
How and why did the system development? who championed it and who opposed it? what kind of implementation strategy was adopted?
Some issues
Present structures and reform options are the culmination of past events and decisions. Why Chile has a centralized M&E system and
Brazil does not. Administrative systems do not start out fully
formed, but are perfected through a learning process. There are constant adjustments to improve
systems in OECD countries. But sometimes developments are more cyclical.
Budget cuts and changes in political ideology affect the pursuit of M&E in OECD countries.
8
Topic B.2. M&E systems: objectives (announced, implicit, or revealed)
Objectives National examples
Solving problems of implementation
Brazil and UK: center-of-government monitoring and “trouble-shooting” programs with political priority.
Accountability within the government
Brazil: compliance and process audits Canada, South Africa, and UK: departmental reporting to the center of
government.
Accountability & information to the public
Colombia and South Africa: government reporting results to the public. Brazil: Supreme Audit Institution reporting program results to the public
(and imposing corrective measures on agencies). UK: education and health “score-cards” (or “league tables”) for consumer
choice.
Improving the design of programs
Chile: Budget Office evaluating programs (also budget impacts). Colombia, Mexico, and Spain: specialized agency evaluating programs and
policies. Ireland: ministry evaluations of projects using European-Union Structural
Funds (also for accountability).Coordinating and prioritizing among programs
Australia (1987-97): evaluation system for policy and budget prioritization. Canada, Chile, Ireland, UK, US: spending reviews that feed into budget
decisions.
9
Topic B.3. M&E systems: processes, tools, and products
The questions
What is produced (indicators and evaluations by type and numbers, and so on)? what are the selection criteria? what is the production cycle? how is the information used (dissemination; reward, sanction, or correction)? how is the quality of the information controlled? what are the tools used to collect, manage, and analyze information and are they appropriate?
Some issues
Problems differ between sectors – for instance social protection vs. infrastructure – hence optimal M&E tools may differ.
Different tools have different costs and benefits – for instance, impact evaluation provides rigorous answers, but it is expensive and time-consuming.
Different objectives require different tools.
10
Topic B.4. M&E systems: relationship with other systems
The questions
How, if at all, are systems interconnected? – monitoring with evaluation? - M&E with the budget? - ministry or sub-national monitoring systems with national systems? - monitoring with information systems? - M&E with quality-management systems?
Some issues
Monitoring and evaluation functions are not necessarily well integrated with each other • For example, integration is high in Canada, low in the UK.
M&E activities overlap with quality-management, information-management, and policy-analysis activities, though these overlaps are often under-exploited.• M&E systems sit within a broader national knowledge-
management infrastructure (statistics offices, national IT systems, public and private think-tanks and research institutes, etc.).
• This infrastructure is characterized in OECD countries by multiple and diverse sources of policy-relevant knowledge.
11
Topic B.5. M&E systems: institutional architecture
The questions How do the system’s components fit together? how is cooperation (in the exchange of information, in willingness to act on results) achieved within the system? how centralized is the system?
Examples of different types of architecture
Chile, Colombia, and Mexico: a central agency carries out the evaluations (the Mexican agency also oversees M&E practices in the line secretariats).
Australia (1987-97), Canada, and South Africa: the center sets general rules and provides advice, while the line departments carry out the evaluations.
UK: for many ministries, specialized, “arms-length” inspectorates carry out the monitoring function.
Brazil and UK: supreme audit institutions are leaders in performance evaluation (and are able to exercise authority over ministries).
12
Topic B.6. M&E systems: characteristics of organizations that are part of the system
The questions
What is/are: the historical reform/policy-change process? the tasks of the agency? its resources (authority, budget, incentives, expertise, training, donor support, etc.)? its “authorizing environment” (the legal framework, roles of main stakeholders)? the obstacles it faces (information, coordination problems)?
Some issues Brazil is an example of a country where the incentives (policy framework and authorizing environment) and capacity to carry out M&E vary significantly between ministries.
Mexico: organizational characteristics of political independence and technical capacity have made the M&E agency into a credible and influential evaluator of social programs.
South Africa: a culture of information hoarding, as well as other differences among public agencies, impede cooperation.
13
A recapitulation en route: where are incentives in the checklist?
TopicsA.1 The national policy and institutional frameworkB.1 M&E system: historical developmentB.2 M&E system: objectivesB.3 M&E system: processes, tools, and productsB.4 M&E system: relationship with other systemsB.5 M&E system: the institutional architectureB.6 M&E system: organizational characteristicsB.7 M&E system: resultsC.1 M&E system: conclusions and recommendations
The major policy frameworks that drive M&E
- Incentives to provide good information.
- Incentives to use findings
Organizational incentives (usually differing between agencies): - to perform, - to demand M&E, - to cooperate with
other organizations
14
Topic B.7. M&E systems: resultsThe questions
What are the quality, credibility, and accessibility of the products of M&E? what is the impact of these products? where there are multiple objectives, are there multiple impacts?
Some issues
It is not at all easy to measure the results of M&E systems (goals fulfilled, programs and policies improved, budgets rationalized, or a public better informed) and to attribute them to specific causes.
Especially where resources are limited, most M&E studies and diagnoses have had to be content (at best!) with author inferences based on knowledge of the system and interviews and focus groups with participants.
15
Topic C.1. Conclusions and recommendations
The questions
What is working and not working and why? what reforms are underway? how can M&E activities be improved?
Some suggestions
No one model will fit all circumstances, given different historical starting points, national institutional environments, objectives, architectures, and difference between public agencies.
See what has worked in other countries – and under what circumstances.
Remember that public-sector reform is subject to bounded rationality : we do not have enough information to know how to get it all right – especially at the first go.
Build on what is in place and on what, under local circumstances, is more likely to work. (Hence the importance of the assessment of the policy and institutional framework and the organizational framework.)
16
Organizing the diagnosis: some modest advice
• An M&E diagnosis will not make sense in every country: there must be a policy framework that drives M&E.
• Involve the government (c.f. Topic B.5 on institutional architecture – incentives to provide good information and to use the findings).
• A diagnosis can be carried out at different levels – it can range from good journalism (with limited resources) to a research project (with greater resources). In any case, you cannot cover all topics in equal depth.
• Prepare as much as you can before you go into the field – it’s amazing what you can find on the Internet.
• Do not duck the challenge of making some judgment of results (Topic B.7) – with whatever caveats are necessary: – interviews and focus groups involving a diversity of actors will help.
• The checklist is better at helping you identify the right information to collect than it is at indicating how to tell the story: – often, a historical narrative works best, but sometimes the need to compare
systems may dictate a more instrumental approach.
