1
Monitoring and Evaluation of Social Programs
Kathy Lindert, World BankQualidade do Gasto Publico no Brasil
June 26-27, 2003
2
OutlineObjectives of M&E
Conceptual Framework, example
Monitoring (role, steps, etc.)
Evaluation (role, steps, etc.)
Dissemination and Uses
Institutional Issues and Sustainability
3
Objectives of M&EMonitoring and evaluation can help policy makers in many ways:
As planning and budgeting tool: to ensure effective allocations of government resources – planning, decision-making and prioritization
As a management tool by revealing the performance of ongoing activities at the sector, program or project level, to inform decisions about expansion, modification, or elimination of programs depending on their effectiveness
As an accountability mechanism, so that managers can be held accountable for the performance of their projects/programs and so that government can be held accountable for its performance
To obtain and maintain political support, institutional resilience (even across administrations), and financing for successful activities
4
Marco Conceitual
Resultados • Efeitos intermediários dos produtos sobre os clientes
Produtos • Produtos e serviços produzidos
Atividades• Tarefas empreendidos
para transformar insumos em produtos
Insumos • Financeiros, humanos, e recursos materiais
Metas(Impactos)
• Longo prazo, ampla melhoria na sociedade
Impl
emen
taçã
oO
bjet
ivos
5
Example: Secondary Education
Outcomes • Increased skills; more employment opportunities
Outputs • Number of youths completing secondary school
Activities • Schooling
Inputs • $, facilities, teachers, materials
Goal(Impacts)
• Higher income levels; increase access to higher skill jobs
6
Monitoring
7
Monitoring of Implementation and ResultsMonitoring is a continuous process of collecting and analyzing information to compare how well a program is being implemented against expected resultsMonitoring tracks both implementation and resultsEnfoque tradicional: Monitoring of activities Definir atividades e responsabilidades Monitoring activities is important
No entanto: É possível executar as atividades, cumprir as
responsibilildades, mas não fazer progresso até as metas
Question: how will you know when you have been successful?
“Se não se reconhecer o sucesso, vai se valorizar o fracasso”
8
Main Types of Monitoring
Output
Activity
Input
Outcome
Impact
Results Monitoring
Implementation Monitoring (enfoque tradicional)
Imp
lem
enta
tio
nR
esu
lts
9
Key Steps for Monitoring
Acordo sobre os resultados a monitorar
Seleção de indicadores de estos resultados
Medição de “baseline”
Eleição de metas quantitativas
Gerenciamento com relação às metas
10
1. Acordo sobre os resultados a monitorar
Outcomes make explicit the intended objectives of governmental action – the visionConsensus about these outcomes, and this vision, is important Participation in agreeing on outcomes is key Key stakeholders should be involved (e.g., government, civil society,
donors, etc.)
Desenhar para sucesso Evitar resultados impossíveis
Motivar pessoal com uma visão inspiradora Criar linha de vista (“line of sight”) para todos que trabalham no
sistema entre as suas ações e o resultado desejado
“Se você não sabe para onde vai, todos os caminhos estão certos”
11
2. Seleção de indicadoresOutcome indicators are not the same as outcomes
Indicadores devem medir resultados mas tambem processos e productos
Um resultado corresponderá típicamente a mais de um indicador
Respondem à pergunta:
“Cómo conheceremos o sucesso, se o vermos?”
12
2. Seleção de indicadores, cont.
A good performance indicator must be:
Clear (Precise and unambiguous)
Relevant (Appropriate to subject at hand)
Economic (Available at reasonable cost)
Adequate (Must provide a sufficient basis to assess performance)
Monitorable (Must be amenable to independent validation)
13
3. Establishing Baseline Data on Indicators
A performance baseline is…Information (quantitative or qualitative) that provides data at the beginning of, or just prior to, the monitoring period. The baseline is used to: Learn about recent levels and
patterns of performance on the indicator; and to
Gauge subsequent policy, program, or project performance
14
Data Sources May Be Primary or Secondary
PRIMARY data are collected directly by the organization, for example, through surveys, direct observation, and interviews.SECONDARY data have been collected by someone else, initially for a purpose other than that of the program. Examples include survey data collected by another agency, a Demographic Health Survey, or data from a financial market. Secondary data often can save money in acquiring
data, but be careful!
15
Dados de “Baseline”
Conversascom
pessoasafetadas
Análisesdos
registrosofficiais
Métodos informais/menos estruturados
Visitas de
campo
Entrevistasdas
comunidades
Observação de participantes
Entrevistas cominterlocutores
chaves
“Gruposde foco”
Pesquisasúnicas
Questionários
Pesquisas de painel
Experiências controladas
Census
Métodos formais/mais estruturados
Observação direta
16
Practicality
Are the data associated with the indicator practical?
Ask whether…Quality data are currently availableThe data can be procured on a regular and
timely basisPrimary data collection, when necessary, is
feasible and cost-effective
17
4. MetasTargets are the quantifiable levels of the indicators that a country wants to achieve at a given point in timePrecondições Clara compreensão do passado recente Conhecimento dos recursos disponíveis
Considerações políticas Talvez o mais importante sendo o custo de não cumprir a
meta… Gerenciamento para sucesso
Seja realista! Faixas? Nota: gestão por resultados não depende críticamente das metas
18
5. Monitoria e gestão por Metas
“Abertura” faz parte do enfoque Desenvolver estratégias de divulgação para o
público, internet, etc.
Monitoramento deve ser contínuoInformações devem correr “verticalmente” e “horizontalmente”A nenhum ponto do sistema devem ser coletados dados sem ser analisados Cria incentivos em favor da qualidade dos dados
19
5. MonitoriaFidelida
de
Validade
Pontualidade
Metodologia estávele consistente
Medida clara e diretado desempenho relevante
Dados atualizados esuficientemente frenqüentepara influenciar decisões gerenciais
20
5. Monitoria: FreqüênciaQuanto mais observações tem, tanto mais os diagnósticos serãoSegurosSofisticados
?Tempo
Melhorar acesso aos mercados
Acc
esso
TempoMelhorar acesso
aos mercados
Acc
esso
21
Evaluation
22
O papel das avaliaçõesResults-Based Evaluation
• An assessment of a planned, ongoing, or completed intervention to determine its relevance, efficiency, effectiveness, impact and sustainability. The intent is to incorporate lessons learned into the decision-making process.
Evaluation Addresses
“Why” Questions: What caused the changes we are monitoring
“How” Questions: What was the sequence or processes that led to successful (or not) outcomes
“Compliance/ Accountability Questions”: Did the promised activities actually take place and as they were planned?
23
Uses of EvaluationTo make resource decisionsTo re-think about the causes of a problemTo identify issues around an emerging problem, i.e. children dropping out of schoolTo identify alternativesTo build support of public sector reform / innovationTo help build consensus among stakeholders on how to respond to a problem
24
Evaluation Means Information on:
Strateg
y
Whether we are doing the right thingsRationale/justificationClient satisfaction
Operation
Whether we are doing things right Effectiveness in achieving expected
outcomes Efficiency in optimizing resources
Learning
Whether there are better ways of doing it
AlternativesBest practicesLessons learned
25
General Types of Evaluations & PERs
Evaluations can occur at many levels: Evaluations of individual staff performance
(including project managers) Evaluations of organizational units Evaluations of programs (focus of this
presentation) Evaluations of sectors/policy areas (e.g., cross-
sectoral issues, fiscal policy, overall education policy)
Public expenditure reviews generally evaluate the broadest level (evaluations of sectors/policy areas), but they also draw on evaluations of programs
26
Types of Program EvaluationsProcess Implementation Evaluations: Provides detailed information on whether program is
operating as intended (are we doing things right?) Provides continuous feedback to assist managers Provides detailed information to those interested in
scaling up or replicating Identifies bottlenecks
Cost-Benefit Evaluations: Calculates the cost of the intervention relative to
results Are we doing things efficiently?
27
Types of Program Evaluations, cont.
Impact Evaluations: Measure net effect of program to determine whether or
not objectives are being achieved Tries to establish causality between activities and impacts Impacts include both those that were intended and those
that were not Impact evaluations require a clearly established counter-
factual: What would have happened without the project? Generally need to compare those “with” the project (beneficiaries,
“treatment group”) and those “without” the project (“control group”) both before and after project
28
Stages of EvaluationClarify objectives of evaluationExplore available informationDesign evaluationForm evaluation team (capacity?)Collect information (sample, questionnaire, pilot, data collection, entry, quality control, etc.)AnalysisReporting, disseminating, discussing resultsUsing results: feedback to policies, budgets, planning
29
Dissemination and Uses of Information
30
Divulgação
Informar sobre o estado dos projetos
Prover índices sobre problemas
Explicar problemas
Criar oportunidades para considerar melhoras o alternativas
Prover informação sobre tendências
31
Usando as ConclusõesAs a planning and budgeting tool: Formular/justificar orçamentos Ajudar alocação operacional de recursos
As a management tool: Melhorar os serviços
As a performance/accountability tool: Responder às demandas para “accountability” Motivar pessoal Monitorar fornecedores Provocar debate sobre problemas
Reforçar confiança pública
32
Institutional Issues and Sustainability
Demand for M&E is crucial Structural requirements (ex. national law that mandates M&E of all
federal programs) Budgetary and planning links
Papéis claros Papel formal para organismos de planejamento e finanças
Credibilidade Sistema deve entregar boas e más notícias! Proteção dos gerentes contra conseqüências gravas
Responsabilidade (“Accountability”) Papéis da sociedade civil, mídia…
Capacidade técnica