Evaluation in Latin America. Evaluation for

Transparency, Accountability and Democracy

Gabriela Pérez Yarahuán Claudia MaldonadoTrujillo

Some facts:

It is one of the most unequal

regions in the world (measure by

its Gini Coefficient on income).

The percentage of people living

in poverty has been declining

but is close to a third of the

population and much higher in

some countries.

The mean rate of growth of real

GDP, 2005-2012 for the region

was 3.9. With important variance

among countries.

Most of our nations transitioned

into full electoral democracy

between the mid 90’s and the

early 2000´s.

• By the late 1980´s – 1990´s disruptive political,

economic and social processes that motivated

government change/innovation.

• Accountability, transparency, evaluation

institutions

• New social policy interventions (CCTs)

• Decentralization

• Evaluation for different purposes

–Legitimacy

– Accountability

– Budgeting

– Improvement

Monitoring and Evaluation

Systems

Our concern…

• What do we know about these evaluation

initiatives?

• Are they converging, innovating, providing

good evidence for decision making?

• Are these systems developing evaluation

for responsible innovation?

Studies/Research on M&E Systems

Supply and Demand; Policy Cycle integration; Utilization; and Sustainability

Monitoring and Evaluation

Systems

References Components that characterize the Evaluation Systems

Mackay, K. (2007). How to Build M&E Systems to Support Better Government. Washington D.C.: Banco Mundial.

• Key issues (political, economic or social ) that give rise to the system • Roles and responsibilities of primary stakeholders • Use of the M&E information • Types of monitoring and evaluation • Clear and defined responsibilities for the data collection for the evaluation • Donor support

Cunill-Grau, N., & Ospina, S. M. (2008). Fortalecimiento de los Sistemas de Monitoreo y Evaluación (M&E) en América Latina. Informe Comparativo 12 países. Washington D.C.: Banco Mundial/CLAD.

• Formal existence (legal) • Managing of the system by an entity with authority and functions in the Public

Administration • Steadiness in the M&E activities • Intent global coverage • Use of the information • Explicit articulation of the users and the functions of the system • Placing of the system under the executive branch • Regulatory role • Systematic developed tools and instruments for the execution of monitoring and evaluation

Various studies …

References

Components that characterize the Evaluation Systems

García, R., & García, M. (2010). La gestion para resultados en el desarrollo. Avances y desafios en America Latina y El Caribe. Banco Interamericano de Desarrollo.

Monitoring institutions Scope of programs and projects monitoring Use and dissemination of monitoring information Statistic information systems Legal and institutional framework of the evaluation Actions derived from unattained goals Dissemination of evaluation results

López, G., Mackay, K., & Krause, P. (Eds.). (2012). Building Better Policies: The nuts and bolts of monitoring and evaluation systems. Washington D.C.: Banco Mundial.

• Government demand for information Use of information Availability and quality of government information Officials and consultants skills to evaluate and analyze the system’s

information Available funds for the system

References

Components that characterize the Evaluation Systems

Feinstein, O. (2012). La institucionalización de la evaluación de políticas públicas en América Latina. Presupuesto y Gasto Público, 68(2012), 41-52.

Demand for evaluations o Explicit and periodic requests for evaluations, explicit

stakeholders and engaged funds Supply of evaluation

o Institutions and professionals with experience in hiring evaluations, capacity building activities in evaluation, institutions with potential to hire and perform evaluations

Linkage o Existence of link-up mechanisms and instruments, adjustment

mechanisms between supply and demand, market development of evaluators

Use of evaluations o Use of evaluation results, identification of users, motivations to

use evaluation results

M&E System

Recognition from different stakeholders

Clear planning of monitoring and

evaluation activities, that is transparent

and public

Monitoring and evaluation is sound

and methodologically rigorous

Outputs of the system (monitoring tools and evaluation studies) are used by stakeholders

A developed, institutionalized, successful

system should have the following characteristics:

Recognition from different stakeholders

Clear planning of monitoring and evaluation activities, that is transparent and public

Monitoring and evaluation is sound and methodologically rigorous

Outputs of the system (monitoring tools and evaluation studies) are used by stakeholders

Path towards an adequate system of evaluation

Methodology

• Case studies of 10 Latin American countries–open call

• Qualitative Method of Research: Grounded Theory

• Emphasis in evidence

• Collaborative work with feedback

Country

Argentina

Brazil

Chile

Colombia

Costa Rica

Ecuador

Mexico

Peru

Uruguay

Venezuela

Recognition from stakeholders 2014

Venezuela

Low Argentina

Medium

Brazil, Chile Colombia Costa Rica Ecuador

Mexico, Peru, Uruguay

High

• Legitimate formal rules, norms, regulations for government evaluation functions.

• Rules include relevant government stakeholders to compose a system. • Stakeholders comply, there is evidence that the rules are carried out.

Planning of monitoring and evaluation activities 2014

Ecuador

Venezuela Costa Rica,

Low

Argentina, Brazil Peru,

Uruguay

Medium Chile

Mexico Colombia

High

• Criteria by which programs to be evaluated are selected. • Important decisions which may influence evaluation use • Open access to decision making on evaluation planning. • Explicit evaluation methodologies

Outputs of the system are used by stakeholders

•

•

Use

Conceptual

Symbolic

Instrumental

Argentina, Colombia, Costa

Rica, Ecuador, Nicaragua, Peru,

Dominican Republic, Uruguay,

Venezuela

Brazil, Chile,

Mexico

Low

Medium High

So far…

• There are different models to understand national M&E

systems.

• How to measure progress in M&E systems?

• There is no unique path or recipe of development, but

the common objective is to influence decision making by

providing more and better evidence.

• In order for evidence to influence decision making the

process by which this evidence is collected has to be

institutionalized, this includes the recognition of the

functions of evaluation and M, planning of M and

evaluations, the assurance of methodological rigor and

consistency.

Debates and tradeoffs

Macro institutions

Micro institutions

Another concern…

Institutionalization

Innovation

To conclude…

• M&E systems are developing as a innovation/response to various disruptive

political, economic and social processes…

• In the last 10 to 15 years there have been important developments in the

conformation of M&E systems in Latin America, but this is not true for all

countries.

• Different stages of development

• A need to understand the different aspects of this development and draw

lessons

• Given the recent information we have on M&E systems we can assess

system´s adequacy based on:

– formal and de facto recognition of evaluation functions,

– capacities to develop adequate evaluation planning,

– adequate conduction of evaluations that have methodological consistency and

rigor

– explicit mechanisms to promote different uses of monitoring and evaluations

• Debate on the developments of the

systems based on the best tools to

influence decision making (big institutional

changes vs. micro/demostrations to

promote change).

• Trade off between innovation and

institutionalization?