the maturity of monitoring and evaluation systems …€¦ · capacity learning focus iii...
TRANSCRIPT
![Page 1: THE MATURITY OF MONITORING AND EVALUATION SYSTEMS …€¦ · capacity learning Focus III Evaluation use Focus IV. Evaluation Systems Maturity Levels Maturity Level Description Score](https://reader033.vdocuments.us/reader033/viewer/2022042310/5ed7347ad37f9f58ca6a77a4/html5/thumbnails/1.jpg)
THE MATURITY OF
MONITORING AND EVALUATION SYSTEMS OF
THE BRAZILIAN PUBLIC ADMINISTRATION
TEAM
Selma Maria Hayakawa Cunha Serpa
Hiroyuki Miki
SUPERVISION
Glória Maria Merola da Costa Bastos
Maridel Piloto de Noronha
ADVISOR
Paulo Carlos Du Pin CalmonUniversity of Brasília
Bangkok, October 2015
FEDERAL COURT OF ACCOUNTS - BRAZIL
4th International Conference on National Evaluation Capacities
Support
![Page 2: THE MATURITY OF MONITORING AND EVALUATION SYSTEMS …€¦ · capacity learning Focus III Evaluation use Focus IV. Evaluation Systems Maturity Levels Maturity Level Description Score](https://reader033.vdocuments.us/reader033/viewer/2022042310/5ed7347ad37f9f58ca6a77a4/html5/thumbnails/2.jpg)
Objectives
Evaluate Brazilian Ministries` (Federal Departments) capacity to demand, do and use evaluation
Describe their evaluation systems
![Page 3: THE MATURITY OF MONITORING AND EVALUATION SYSTEMS …€¦ · capacity learning Focus III Evaluation use Focus IV. Evaluation Systems Maturity Levels Maturity Level Description Score](https://reader033.vdocuments.us/reader033/viewer/2022042310/5ed7347ad37f9f58ca6a77a4/html5/thumbnails/3.jpg)
Conceptual Framework
![Page 4: THE MATURITY OF MONITORING AND EVALUATION SYSTEMS …€¦ · capacity learning Focus III Evaluation use Focus IV. Evaluation Systems Maturity Levels Maturity Level Description Score](https://reader033.vdocuments.us/reader033/viewer/2022042310/5ed7347ad37f9f58ca6a77a4/html5/thumbnails/4.jpg)
Analysis Framework
Evaluative
Demands
Focus I
Focus I - Do governmental decision-makers
have well defined program objectives, targets, goals, budget and other resources?
know which, when and for what purpose evaluative information is needed ?
know who needs the evaluative information to be produced ?
Production of
evaluative knowledge –
supply
Focus II
Focus II – Do Brazilian Ministries have (to produce evaluative information)
Institutionalized organizational process?
Evaluative procedures and practices?
Skilled resources to develop or to ask for evaluation?
Evaluative rules and organizational support (resources to do evaluation) ?
Organizational
capacity learning
Focus III
Evaluation
use
Focus IV
![Page 5: THE MATURITY OF MONITORING AND EVALUATION SYSTEMS …€¦ · capacity learning Focus III Evaluation use Focus IV. Evaluation Systems Maturity Levels Maturity Level Description Score](https://reader033.vdocuments.us/reader033/viewer/2022042310/5ed7347ad37f9f58ca6a77a4/html5/thumbnails/5.jpg)
Evaluation Systems Maturity LevelsMaturity Level Description Score
Not structured
A system or focus is considered not structured when the mechanisms and instruments needed to characterize the evaluation systems are partially present and they are not enough and do not regularly meet the decision-makers` needs of evaluative knowledge
> 50
Incipient
A system or focus is considered incipient when the mechanisms and instruments needed to characterize the evaluation systems are present, but they are partially sufficient and do not regularly meet the decision-makers` needs of evaluative knowledge
≥ 50 e ≤ 60
Intermediate
A system or focus is considered intermediate when the mechanisms and instruments needed to characterize the evaluation systems are present, partially sufficient and regularlymeet the decision-makers` needs of evaluative knowledge
> 60 e ≤ 70
Improved
A system or focus is considered Improved when the mechanisms and instruments needed to characterize the evaluation systems are present, sufficient and satisfactorily meet the decision-makers` needs of evaluative knowledge
> 70 e ≤ 80
Advanced
A system or focus is considered advanced when the mechanisms and instruments needed to characterize the evaluation systems are present and entirely adequate to meet the decision-makers` needs of evaluative knowledge
> 80
![Page 6: THE MATURITY OF MONITORING AND EVALUATION SYSTEMS …€¦ · capacity learning Focus III Evaluation use Focus IV. Evaluation Systems Maturity Levels Maturity Level Description Score](https://reader033.vdocuments.us/reader033/viewer/2022042310/5ed7347ad37f9f58ca6a77a4/html5/thumbnails/6.jpg)
Methodology
Web-survey (as of 26, August to October 4th, 2013)
Target group - 2,062 governmental decision-makers at 28 ministries (Secretariats
(DAS 6), Directors (DAS 5) and Coordinators (DAS 4 and DAS 3))
750 respondents (36.4%)
Descriptive statistics and content analysis
Multi-Criteria Decision Analysis (constructivist approach) – MCDA-C (index – ISA-
Gov)
Minister
Director B1
Secretariat A Secretariat B Secretariat C
Director B2
Coordinator General Coordinator Project Manager
Executive Secretariat
73.1%
18.3%
7.1%
1.1%
![Page 7: THE MATURITY OF MONITORING AND EVALUATION SYSTEMS …€¦ · capacity learning Focus III Evaluation use Focus IV. Evaluation Systems Maturity Levels Maturity Level Description Score](https://reader033.vdocuments.us/reader033/viewer/2022042310/5ed7347ad37f9f58ca6a77a4/html5/thumbnails/7.jpg)
Survey Target Group
0.4%
0.5%
0.8%
1.3%
1.5%
5.9%
10.3%
15.7%
0.8%
0.3%
0.5%
5.6%
11.1%
22.6%
22.7%
0.0% 10.0% 20.0% 30.0% 40.0% 50.0%
Branco (sem resposta) → 0,4%
Secretário(a) Executivo(a) → 1,1%
Diretor(a) Geral → 1,4%
Coordenador(a) → 1,8%
Secretário(a) → 7,1%
Diretor(a) → 16,9%
Outros → 32,9%
Coordenador(a) Geral → 38,4%
Percentage - per job position
% Respondentes % Ñ Respondentes
Others
Directors
Secretariats
Coordinators
General Directors
Executive Secretariats
Non identified
Respondents Non Respondents
General Coordinators
![Page 8: THE MATURITY OF MONITORING AND EVALUATION SYSTEMS …€¦ · capacity learning Focus III Evaluation use Focus IV. Evaluation Systems Maturity Levels Maturity Level Description Score](https://reader033.vdocuments.us/reader033/viewer/2022042310/5ed7347ad37f9f58ca6a77a4/html5/thumbnails/8.jpg)
17.65%
6.60%
19.14%
33.56%
23.05%
0% 5% 10% 15% 20% 25% 30% 35% 40%
Mais de 6 anos
5 a 6 anos
3 a 4 anos
1 a 2 anos
Menos de um ano
Period of time at the current job position
< 1 year
1 to 2 years
3 to 4 years
5 to 6 years
> 6 years
Respondents Profile
57 %
![Page 9: THE MATURITY OF MONITORING AND EVALUATION SYSTEMS …€¦ · capacity learning Focus III Evaluation use Focus IV. Evaluation Systems Maturity Levels Maturity Level Description Score](https://reader033.vdocuments.us/reader033/viewer/2022042310/5ed7347ad37f9f58ca6a77a4/html5/thumbnails/9.jpg)
ResultsMaturity Level per Focus
Capacity to do evaluation
0
10
20
30
40
50
60
70
80
90
Pontuação noFoco 1
(Demanda)
Pontuação noFoco 2 (Oferta)
Pontuação noFoco 3
(Aprendizado)
Pontuação noFoco 4
(Utilização)
GERAL
59.17
49.57
82.50 80.00
63.96
Focus I score(Demand)
Focus II score(Supply)
Focus III score(Organizational
Capacity Learning)
Focus IV score(Evaluation use)
General score
![Page 10: THE MATURITY OF MONITORING AND EVALUATION SYSTEMS …€¦ · capacity learning Focus III Evaluation use Focus IV. Evaluation Systems Maturity Levels Maturity Level Description Score](https://reader033.vdocuments.us/reader033/viewer/2022042310/5ed7347ad37f9f58ca6a77a4/html5/thumbnails/10.jpg)
ResultsBreakdown of Foci - Dimensions of Analysis
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
60.48
74.00
32.46
72.46
41.3346.00
50.00
59.11
82.50 80.00 80.00 80.00 80.00
63.96
![Page 11: THE MATURITY OF MONITORING AND EVALUATION SYSTEMS …€¦ · capacity learning Focus III Evaluation use Focus IV. Evaluation Systems Maturity Levels Maturity Level Description Score](https://reader033.vdocuments.us/reader033/viewer/2022042310/5ed7347ad37f9f58ca6a77a4/html5/thumbnails/11.jpg)
Main evaluation practices
![Page 12: THE MATURITY OF MONITORING AND EVALUATION SYSTEMS …€¦ · capacity learning Focus III Evaluation use Focus IV. Evaluation Systems Maturity Levels Maturity Level Description Score](https://reader033.vdocuments.us/reader033/viewer/2022042310/5ed7347ad37f9f58ca6a77a4/html5/thumbnails/12.jpg)
ResultsComparative analysis between evaluation demand x supply – Focus I x Focus II
Focus I
I -
Supply
Focus I - Demand
![Page 13: THE MATURITY OF MONITORING AND EVALUATION SYSTEMS …€¦ · capacity learning Focus III Evaluation use Focus IV. Evaluation Systems Maturity Levels Maturity Level Description Score](https://reader033.vdocuments.us/reader033/viewer/2022042310/5ed7347ad37f9f58ca6a77a4/html5/thumbnails/13.jpg)
Focus Radar
![Page 14: THE MATURITY OF MONITORING AND EVALUATION SYSTEMS …€¦ · capacity learning Focus III Evaluation use Focus IV. Evaluation Systems Maturity Levels Maturity Level Description Score](https://reader033.vdocuments.us/reader033/viewer/2022042310/5ed7347ad37f9f58ca6a77a4/html5/thumbnails/14.jpg)
Ministry
Governmental Evaluation Systems Purposes
Organizational Learning
Performance Monitoring
Evidence-Based Policy
Accountability
6 X X
8 X X X X
12 X X
14 X X
17 X X
21 X X X X
26 X X X
27 X X X X
28 X X X
Purposes of Evaluation Systems
![Page 15: THE MATURITY OF MONITORING AND EVALUATION SYSTEMS …€¦ · capacity learning Focus III Evaluation use Focus IV. Evaluation Systems Maturity Levels Maturity Level Description Score](https://reader033.vdocuments.us/reader033/viewer/2022042310/5ed7347ad37f9f58ca6a77a4/html5/thumbnails/15.jpg)
Maturity Index of Governmental Evaluation Systems - iSA-Gov
Advanced
Improved
Intermediate
Incipient
Not Structured
![Page 16: THE MATURITY OF MONITORING AND EVALUATION SYSTEMS …€¦ · capacity learning Focus III Evaluation use Focus IV. Evaluation Systems Maturity Levels Maturity Level Description Score](https://reader033.vdocuments.us/reader033/viewer/2022042310/5ed7347ad37f9f58ca6a77a4/html5/thumbnails/16.jpg)
Thank you!
Selma Maria Hayakawa Cunha Serpa
Federal Court of Accounts - Brazil