pcr evaluation note for public sector operations · knowledge and results management strengthening...

1. BASIC INFORMATION

a. Basic project data

Project title: Institutional Capacity Building for Poverty Reduction

Project code: P-AO-IE0-001 Instrument number(s): ADF Grant 2100155019816

Project type: Social Sector: OSHD

Country: Angola Environmental categorization (1-3) : 3

Processing Milestones Key Events Disbursement and Closing date

Date approved: 23.02.2011 Cancelled amount: 0 Original disbursement deadline:

31.12.2013

Date signed: 01.04.2011 Supplementary financing: 0 Original closing date: 31.12.2013

Date of entry into force :

01.04.2011

Restructuring: 0 Revised disbursement deadline:

30.06.2016

Date effective for 1st

disbursement: 14.01.2012

Extensions (specify dates): 1)

31.12.2014, 2) 31.12.2015

Revised closing date: 30.06.2016

Date of actual

1st disbursement: 14.02.2012

b. Financing sources

Financing source/ instrument

(MUA)

Approved amount

(UA) :

Disbursed amount

(UA) :

Percentage disbursed

(%):

Loan:

Grant: 4.920.000 4.487.110 91.2%

Government: 250,000 300,000 120%

Other (ex. Co-financiers):

TOTAL : 5.170.000 4.787.110 92.59%

Co-financiers and other external partners: N/A

Execution and implementation agencies: Ministry of Planning and Territorial Development (MPDT), National

Directorate of Public Investments (DNIP)

c. Responsible Bank staff

Position At approval At completion

Regional Director Frank Black Kennedy Mbekeani

Sector Director Agnes Soucat Sunita Pitamber

Sector Manager Sunita Pitamber Justin Murara

Task Manager João Cunha Yolanda Arcelina

Alternate Task Manager Raymond Besong

PCR Team Leader Yolanda Arcelina

PCR Team Members Raymond Besong

d. Report data

PCR Date : 29 July 2016

PCR Mission Date: From: 27 June 2016 To: 7 July 2016

PCR-EN Date:

Evaluator/consultant : Arthur Neiland - IDDRA Peer Reviewer/Task Manager:

PCR EVALUATION NOTE FOR PUBLIC SECTOR OPERATIONS

2. PROJECT DESCRIPTION

The Institutional Capacity Building for Poverty Reduction Project was a response to a Government of Angola (GoA) request

for technical assistance from the African Development Bank (AfDB) to help build capacity of the National Directorate of

Public Investment (NDPI) in the Ministry of Planning (MINPLAN/MPTD). The initiative was part of GoA’s efforts to improve

execution of the Public Investment Program (PIP) for economic growth and poverty reduction. The project is expected to

strengthen institutional capacity for improved planning, appraisal, implementation, monitoring and evaluation (M&E) of

publicly-financed projects. The duration of the project at inception was 30 months. The total cost was UA 5.17 million, of

which UA 4.92 million was a Bank grant and UA 0.25 million GoA counterpart funding.

A number of changes were introduced at MTR in order to speed up project implementation and in response changes in the

priorities of the GoA. The project deadline was extended twice beyond its original time limit.

a. Rationale and expected impacts:

The rationale of the project was to improve the perceived acute institutional and human resource weaknesses confronting the country which have resulted in inefficient public administration and bottlenecks in implementation of development programs, including those financed by donors. This capacity constraint has resulted in an environment of poor programming and low levels of execution and implementation of public investments, and ultimately poor infrastructure for supporting economic activities and the delivery of social services. The project would therefore provide assistance to ministries, departments and agencies (MDAs) of GoA on adopting procedures, methods and criteria at all stages of the Project Implementation Process (PIP). The project would also contribute to improved donor harmonization and coordination. Additionally, the project was expected to: i) improve the overall performance of the Bank’s portfolio in Angola, ii) reinforce the Bank’s new Country Strategy, iii) improve fiscal effectiveness, and iv) promote balanced and improved development. While the long-term indirect project beneficiaries of the project is the population of Angola, the short and medium-term direct beneficiaries are the staffs of the EA, the NDPI and of the Offices of Studies and Planning (GEPs) at ministerial and provisional level: all country staff involved in the PIP. The rationale remained unchanged at the MTR. The changes occurred at a lower level of the RLF hierarchy relating to project Outputs.

b. Objectives (Project purpose RLF)/Expected Outcomes: The project objective was to build an integrated project management and coordination system at the NDPI within MINPLAN/MPTD, for improving the results of public sector projects, especially with respect to their impact on poverty reduction. The expected outcome of the project is improved programme planning and improved implementation and M&E of public sector development projects. The intended direct beneficiaries are the staff of the NDPI, MINPLAN and GEPs throughout the country.

c. Outputs and intended beneficiaries:

At appraisal, the project RLF sited 13 outputs (under 3 components). The intended beneficiaries are the staff of the NDPI and the GEPs in the MINPLAN and at provincial level. Human Resource Development

i. Capacity Building Master Plan (w/ skills analysis and training framework)

ii. Manual developed for key areas of PIP operations

iii. Staff involved in PIP process trained in project management topics iv. Technical assistance and “on job” training provided on PIP preparation

Knowledge and results management strengthening

i. M&E system in place and key performance indicators (KPIs) for each sector developed ii. Projects supervision improved

iii. Project management ICT tools implemented at GEPs

iv. Report and recommendations of the Economic Diversification Study

v. PIP design targets impact on various segments of population (youth, gender, rural/urban, etc.)

vi. Stakeholder forums convened

Project Management

i. Operational team established ii. Qualified consultants recruited

iii. Facilities, equipment and transport available for project implementation

d. Principal activities/Components: At appraisal, the project RLF sited 3 components and 12 activities: Human Resource Development

i. Conduct institutional capacity analysis and develop remedial training framework ii. Develop operational learning and support materials

iii. Train NDPI and GEPs staff on project management and English language proficiency iv. Participate in regional study tours and international conferences

Knowledge and results management strengthening

i. Develop an M&E framework for PIPs

ii. Conduct field monitoring visits

iii. Establish a Government-wide ICT network for PIP management

iv. Carry out strategic studies to inform design and management of multi-year investment programs v. Organize stakeholder forum

Project Management

i. Recruit Coordinator and project management support team ii. Recruit external consultants

iii. Procure equipment for operations At the MTR, a new set of ‘priority activities’ were agreed between the Bank and the GoA. These comprised of 2 components with 12 activities. These new components and activities are not actually integrated into the revised RLF in the MTR. The

revisions in the RLF are in the targets for the Outputs of the Appraisal RLF. Institutional strengthening

a. Supporting the preparation of the Public Investment Law and Regulations b. Supporting the conducting of a Staff profile & Careers (survey) for the MPTD; c. Training of MPTD staff d. Supporting the recruitment of young professional staff for the duration of the Project that the MPTD will take on after

the end of the Project e. Support the organization of two stakeholder forums to discuss the PIP process f. Support the preparation of Procurement Operations Manual.

SIPIP software and system improvements

a. Support the preparation of User and Technical Manuals of PIP operations b. Support the setting up of the M&E system for PIP operations c. Support the setting up of a system for the analysis and implementation of Structuring (strategic/large) projects d. Support the setting up of a database of Reference Prices to improve the reliability and accuracy of PIP budgets e. Support the preparation of a Risk Analysis and Prevention Manual for the PIP process f. Support the development of a Methodology for setting KPIs for PIP;

At MTR, as the new activities are not actually integrated into the RLF, they are not specifically linked to outputs, output indicators and targets. This more ‘open-ended’ approach of the revised project probably reflects the reality of low institutional capacity in the Angolan institutions encountered by the project up until the MTR and a pragmatic response to a change at political level in the host ministry.

3. PROJECT PERFORMANCE ASSESSMENT

RELEVANCE

a. Relevance of the project development objective:

The PCR scores the project at 4,’Highly Satisfactory’. The current evaluation agrees.

Angola does not have a Poverty Reduction Strategy Paper, as poverty reduction is mainstreamed in various documents outlining Government of Angola’s (GoA’s) growth and development strategies, such as: (a) the Long-Term Strategy entitled Angola Visão 2025; (b) the Government Program (2009-2012); (c) the National Plan for 2009 and (d) the National Biannual Plan for 2010-2011. The main objectives of the broad economic and development strategies pursued by GoA for the period 2006-2015 are: (i) promotion and acceleration of growth and competitiveness through diversification of the economy; (ii) poverty reduction through human capital development, especially private sector job creation; (iii) balanced growth and development alongside natural resource protection; and (iv) an efficient and accountable government underlined by strong institutions. The project’s development objectives are particularly relevant to objectives (ii) and (iv).

The proposed project is indicated under Pillars I (Stimulus to the Competitiveness of the Angolan Economy and Private Sector Development) and II (Support Private Sector Development through the Improvement of Economic Infrastructure) of Angola Country Strategy Paper (CSP, 2011-2015) as a technical assistance and capacity building intervention geared towards supporting GoA’s Public Investment Program (PIP).

The proposed project is also referenced by GoA in the Memorandum of Economic and Financial Policies for 2010/11 of

the Stand-by-Agreement (SBA) with the International Monetary Fund (IMF) as a means to “enhance the project appraisal framework at the MINPLAN (with the assistance of the AfDB) to ensure that projects are soundly appraised before they are allocated in the budget”.

While the alterations of the project at MTR did not change the overall project development objective, they were essential to maintain project relevance with respect to changes in the GoA’s development priorities and strategies and needs. These stressed the need for the GoA to measure the impact of public sector projects and so the need to have appropriate tools and staff with appropriate skills in order to do this. Angola’s increasingly difficult fiscal position and heightened need for international financial assistance only underlies the relevance and importance of the project’s objectives.

b. Relevance of project design (from approval to completion):

The PCR scores the project at 3,’Satisfactory’. The current evaluation agrees.

The initial project design was relevant to the request from the GoA for technical assistance to build capacity in the DNIP of the MINPLAN and other offices charged with implementing the Public Investment Programme (PIP). This was highly relevant to CPIA 13: Quality of Budgetary and Financial Management, which… assesses the extent to which there is: (a) a comprehensive and credible budget, linked to policy priorities; (b) effective financial management systems to ensure that the budget is implemented as intended in a controlled and predictable way; and (c) timely and accurate accounting and fiscal reporting, including timely audit of public accounts and effective arrangements for follow up (The World Bank Group, CPI2013 Criteria). The project outcome indicator ‘public projects completed on schedule’ was found to lack relevance as the project had no direct responsibility for implementing or completing public sector projects in Angola. Changes at the political level in Angola which saw the formation of the Ministry of Planning and Territorial Development (MPTD) brought new development priorities to which the Bank responded in the MTR, thereby maintaining and enhancing relevance of the project to client needs in the new situation. The PCR identifies 3 lessons pertaining to the relevance of the original project design: i) the implementation and M&E systems of the PIP should have been developed prior to the training needs assessment and the Capacity Building Master Plan, ii) the gender and environmental studies should have been completed prior to development of the PIP systems, so that the results could have informed the design of the PIP systems, and iii) trainees should have been ‘mentored’ after training to ensure the practical effectiveness of the training. One lesson related to the revised design adopted at the MTR. That was, while the Economic Diversification Study was directly relevant to a borrower need, the PCR team felt it lacked relevance to the rest of the project. The project re-design at MTR recognised the over-ambition of the outcomes and timetable of the original project design given the weaknesses in capacity in Angola at the time.

EFFECTIVENESS

EVALUATOR’S NOTE: Sections c) and d) are reversed in order here so that they are in the same logical order as in the RLF

and the PCR.

d. Effectiveness in delivering outcomes:

The PCR scores the project at 4,’Highly Satisfactory’. The current evaluation scores the project at 3, ‘Satisfactory’.

Outcome 1. – CPIA 13 – Quality of budgetary and financial management. The outcome indicator was 3.8 at end of project (2015), close to the target of 4.0. The PCR is unclear on whether this was from a baseline in 2009 of 2.5 or 3.0. Either way, it is a successful outcome of the project. Outcome 2. – The percentage of Bank portfolio at risk. Only a one percentage point reduction was recorded during the project, from 45% down to 44%. This is only a 5% movement towards the target of 25%. The project was not successful in achieving this outcome. The PCR sites the continued need to improve project selection and implementation. Outcome 3. – The percentage of public projects completed on schedule. The Outcome indicator was found not to be relevant as execution of PIP projects is outside the scope of the Bank project. No data is supplied. Changes to the project at the MTR lead to a number of other project outcomes not envisaged in the original project design. As noted above, they have no specific output indicator or target attached to them, only a narrative assessment. These project outcomes are:

i. Improved communication between PIP preparation stakeholders ii. Improved quality of information and reporting system

iii. Improved alignment of public investment projects with the NDP iv. Improved national project portfolio v. Improved PIP preparation performance

c. Effectiveness in delivering outputs:

The PCR scores the project at 4,’Highly Satisfactory’. The current evaluation agrees.

The project achieved a large number of administrative project outputs (‘things done’). These are:

1. The Capacity Building Master Plan was prepared. 2. Project database (SIPIP) was set up. 3. SIPIP was web-enabled. 4. ICT systems developed. 5. EDS carried out. 6. Multi-year investment programme finalised. 7. Staff recruited to manage project implementation. 8. Consultants recruited. 9. ICT equipment and vehicles procured. 10. An electronic framework for registering infrastructure was developed. 11. DP3 revised and pending approval by Cabinet.

12. All project reports completed. Several other quantifiable output indicators met or exceeded target. These were:

13. Development of manuals (233% of target) 14. Staff training (59% PAR target, 99% MTR target) 15. Timely and improved processing of PIP proposals (100% - 150%) 16. Number of events in which DNIP counterpart staff participated (100%) 17. Number of field visits (150%)

Additional outputs subsequent to the alterations of the MTR were:

18. supporting the preparation of the Public Investment Law and Regulations 19. supporting the conducting of a Staff profile & Careers (survey) for the MPTD

20. training of MPTD & GEPs staff 21. supporting the recruitment of young professional staff for the rest of the duration of the Project that the MPTD will

take on after the end of the Project 22. support the preparation of User and Technical Manuals of PIP operations

The project failed to establish and convene a stakeholder’s forum.

e. Project development outcome:

The PCR scores the project at 3,’Satisfactory’. The current evaluation agree.

f. Beneficiaries: The direct beneficiaries of the project were:

1. Staff.* 169. 24% female. Benefitted from: training (1,429,000 UA, 18% of budget), field trips (409,774 UA, 5% of budget) and use of equipment (658, 208 UA, 8% of budget). Total benefit 2,496,982 UA, 31% of budget.

2. Young professionals. 6. 33% female. Benefitted from: work experience (252,000 UA, 3% of budget).

3. Project Management Team. 1. Benefitted from contract awarded (3,412,800 UA, 43% budget).

4. Consultants. 3. Benefitted from contracts awarded (260,000 UA, 3% of budget).

*The PCR Mission reports a total number trained of 175 which includes the 6 young professionals. It is unclear to the evaluation if the 16 staff trained in English were part of the 148 who received other professional training or were different staff.

g. Unanticipated additional outcomes (positive or negative, not taken into consideration in the project

logical framework):

The PCR provides the following list of unexpected outcomes.

1. Improved communication between PIP preparation stakeholders: Social / Positive / Medium. 2. Improved Quality of information and Reporting system: Social / Positive / Medium. 3. Improved Alignment of public investment projects with the NDP: Social / Positive / Medium. 4. Improved National Project Portfolio: Social / Positive / Medium. 5. Improved PIP preparation Performance: Social / Positive / Medium.

EFFICIENCY

h. Timeliness:

The PCR scores the project at 2,’Unsatisfactory’. The current evaluation agrees.

The planned duration of the project was 3 years, but it was extended to 4.3 years, a 43% over run. This was largely due to personnel problems on both project and GoA sides. The PCR accepts that in retrospect, the original time scheme for the project was over-ambitious in the context of the institutional weaknesses of MINPLAN at the time of project inception.

i. Resource use efficiency:

The PCR scores the project at 4,’Highly Satisfactory’. The current evaluation gives a slightly lower score of 3, ‘Satisfactory’.

The PCR calculates the Resource use efficiency coefficient at 91%. This includes physical implementations resulting from the alterations following the MTR and achieved in the extended period of the project. The disbursement rate was 91.6% which, although high in the context of AfDB projects in Angola, is still short of 100% disbursal. There was a 43% time overrun on the project. Also, the Peer Review notes that, “Even though some outputs exceeded the planned activities, some others were not delivered”. According to the PCR, “At the closure of the project a number of key activities remain outstanding and need to be carried out to reinforce the gains made under the project and to improve sustainability”.

j. Cost-benefit analysis: No CBA was undertaken as part of the project, at appraisal or otherwise.

k. Implementation progress:


The PCR calculates that the project has implemented at least 90% of project activities. The exact calculation is difficult due to the changes brought about by the MTR. The PCR states that, ‘the Project has contributed significantly in developing the Public Investment Program Management Information System (SIPIP) at the MPDT as well as in strengthening the capacity of the Ministry and other users of the system in its operation.’ There were no significant environmental or social safeguards required for the project. All the procurement goals for the project were met (IT equipment, vehicles). The financial management arrangements and M&E were judged adequate. The Bank funds were disbursed within the project to a level of about 92%, which is very good anyway but very much higher than the disbursal rates for other Bank projects in Angola. Counterpart funds were provided by the GoA to 120% of planned, which is excellent, especially in the difficult fiscal environment at the time.

SUSTAINABILITY

l. Financial sustainability:


The on-going financial sustainability of the work that the project has begun was entirely dependent at the time of the PCR on the GoA’s commitment and fiscal ability to allocate sufficient budgetary resources to the MPDT. In the short term, this may in turn depend on the GoA relationship with the IMF. The other possibility is further project specific funding from the Bank. The provision of the Bank MIC Grant is viewed at crucial to making financial sustainability ‘Satisfactory’.

m. Institutional sustainability and strengthening of capacities:


The project outputs in terms of setting up systems (SIPIP) and staff training will continue to provide a flow of benefits to the GoA as long as their ongoing costs are met by the Government budget. However, the PCR identifies 3 tasks which are important to carry forward the existing gains and momentum the project has generated, i) the development of a Reference Prices database, ii) the development of a full Infrastructure Inventory database, and iii) the extension and widening of training activities.

n. Ownership and sustainability of partnerships:

The PCR scores the project at 4,’Highly Satisfactory’. The current evaluation gives a score of 3, ‘Satisfactory’.

An important part of the project design was inclusion of staff not only of the MPTD, but other PIP related staff in the Ministry and other offices throughout Angola. However, a project lesson listed in the PCR is that other departments in the MPTD should also be identified as stakeholders and included in project activities: IT, Territorial and Planning. Another lesson is that the GEPs should have electronic access to SIPIP from their offices. While this is essential for the running of PIP, it is also a strong ‘ownership’ measure that is countrywide. The project also developed a relationship with the Microsoft Africa NPO which will be important for continued technical services to SIPIP.

o. Environmental and social sustainability:

N/A

4. PERFORMANCE OF STAKEHOLDERS

a. Bank performance:


The evaluation finds that the bank has performed well with regard to all 7 criteria in the Guidance note and that the PCR team have done a competent, conscientious and objective job with the PCR; for which they are to be commended. In particular, the ‘lessons learned’ identified in the PCR will help to cement the project impact if the Bank continues support to

the GoA in this area, and will inform similar Bank projects in other countries.

b. Borrower performance:


The PCR provides a well-balanced assessment of borrower performance.

c. Performance of other stakeholders:


The PCR found the service of the other stakeholders satisfactory, with the exception of that of the auditors.

5. SUMMARY OF OVERALL PROJECT PERFORMANCE

a. Overall assessment:

The PCR gives the project an overall score of 3.25 – Satisfactory. The PCREN-R agrees that the project is Satisfactory but with a slightly lower average score of 3.04 The evaluation finds that overall the project achieved a high degree of success over its extended lifetime in a difficult environment for implementation. The performance is summarised below in relation to the four key criteria. Relevance: The evaluation finds that the original project was highly relevant to the context and to the GoA’s weaknesses in capacity for public project implementation. The project design was thought relevant, but quickly ran into operational and then political problems. This necessitated a revised project design at mid-term which was relevant to the new priorities of the MPTD and the needs of the GoA. Effectiveness: Despite the initial operational difficulties and the changes at MTR, the project adopted a flexible approach which helped it to achieve an overall high level of effectiveness in reaching both the original and the new project Outcomes and Outputs. For Outcomes, it was effective in raising Angola’s CRIA 13 rating, and a number of new outcomes after MTR, although these lacked specific indicators and targets. It was not effective in reducing the Ban’s exposure to risk in its Angolan portfolio. For Outputs, the project effectively achieved 12 administrative outputs, 5 outputs at target or better, and 5 new outputs after the MTR. It failed to convene a stakeholder conference. Efficiency: The project was beset by implementation problems in the first year which meant that timeliness was ultimately unsatisfactory with a 43% overrun on time. However, the project achieved a high percentage of its goals over its extended lifetime, met all procurement goals and disbursed a very high proportion of funds for a Bank project in Angola. Thus, overall efficiency was satisfactory. Sustainability: At the PCR, the medium-term financial sustainability of the advances made by the project remained ‘at risk’

in the fragile fiscal environment in Angola without outside support from the Bank (or other) at project level and the world economy/IMF at macro-economic level. The Bank MIC grant will secure project gains in the short to medium term.

b. Design, implementation and utilization of the M&E (appreciation of the evaluator):

The project had extensive M&E from start to finish of a high standard. It did a commendable job in keeping track of a project which underwent almost constant change in its lifetime. The good M&E ensured that the project made real achievements despite this environment of change.

6. EVALUATION OF KEY LESSONS LEARNED AND RECOMMENDATIONS

The evaluation finds the comments in this section of the PCR are honest, conscientious and analytically astute reflections on the project experience. However, in response to advice from the peer reviewer, the evaluator has re-written the PCR comments to better fit the AfDB protocol. The numbering is consistent with the PCR. a. Lessons learned:

1. NOT VALIDATED: ISSUE: Identification and enfranchisement of stakeholders. REVISED LESSON LEARNED: Project design needs to take a broad view of which groups of people or institutions are stakeholders in the project and then enfranchise them in the project from that stage forward. EXAMPLE: The example in this project is that various departments, ministries and other entities - all of whom have a role to play in the implementation of public projects in Angola - were not identified as stakeholders and enfranchised in the project design. AUDIENCE: AfDB, GoA, other country governments.

2. NOT VALIDATED: ISSUE: Mandate (legal status) of the implementing agency. REVISED LESSON LEARNED: An important consideration at design and appraisal stages of the project cycle is to assess whether the legal status and mandates of the implementing agency(ies) are suitable and sufficient for that agency(ies) to conduct the activities of the project in the implementation phase. EXAMPLE: The example in this project was that the planning ministry had a weak mandate to coordinate inter-agency institutional development. AUDIENCE: AfDB, GoA, other country governments.

3. NOT VALIDATED: ISSUE: Sequencing and flexibility of project activities. REVISED LESSON LEARNED: The successful implementation of a project depends in part on the correct sequencing of project activities. Project activities also need to be flexible enough to respond to changing requirements (where necessary), whether requirements change as a direct result of project activities or due to factors external to the project. EXAMPLE: The example in this project was that training needs (capacity building) assessment should have been made after, not before, the systems and instruments of the public investment programme were developed. The training needs should then have been revised in response to the changed requirements brought about by the project. AUDIENCE: AfDB, GoA, other country governments.

4. NOT VALIDATED:

ISSUE: Training monitoring. REVISED LESSON LEARNED: The success of training components in projects would be assisted by post-training

monitoring, i) to check the effectiveness of the training, and ii) to support the further professional development of the trainees. AUDIENCE: AfDB, GoA, other country governments.

5. NOT VALIDATED: ISSUE: Devolved decision making. REVISED LESSON LEARNED: Effectiveness in the implementation phase of a project is partly dependent on devolving decision-making power down the hierarchy, especially to the Project Co-ordinator, so that decisions can be made quickly in response to circumstances. EXAMPLE: In this project, implementation was delayed by decision-making being vested too high-up the project hierarchy. AUDIENCE: AfDB, GoA, other country governments.

b. Recommendations:

The evaluation finds that the recommendations of the PCR were relevant and sensible at the time of the PCR (July 2016). However, they are re-presented here to better fit the AfDB protocol. The numbering is consistent with the PCR.

1. VALID (AT TIME OF PCR) RISK: That project outcomes will be lost without further work to support their sustainability. REVISED RECOMMENDATION: An MIC grant should be extended to the GoA to enable, i) the completion of project outputs related to the new PIP systems, and ii) a concept note and feasibility study of a new project to support the PIP into the future. AUDIENCE: AfDB, GoA.

2. INVALID (AT TIME OF PCR) – too similar to (1.) above.

RISK: That factors not addressed by the current project will undermine the future efficiency of the PIP if not addressed. REVISED RECOMMENDATION: Included in (1.) above. AUDIENCE:

3. VALID (AT TIME OF PCR) RISK: That staff trained (capacity built) under the project will be lost. REVISED RECOMMENDATION: Arrange on-going funding for the 6 young professionals recruited and trained under the project. AUDIENCE: GoA.

4. VALID (AT TIME OF PCR) RISK: That poor financial monitoring procedures in Angola will undermine the possibility of future AfDB support to the PIP in Angola. REVISED RECOMMENDATION: A revised audit report for the period ending 31 Dec 2015 and a final audit report for the period to June 2016 should be submitted (by 31 Dec 2016). AUDIENCE: GoA.

7. COMMENTS ON PCR QUALITY AND TIMELINESS

The evaluation finds that the PCR is a comprehensive, conscientious and competent piece of work and gives it an overall score of 3.56. The main weakness, as noted by the Peer Review of the PCREN, is that, ‘Even though the PCR is comprehensive and all sections (have been) completed, there were some weaknesses. None of the over 30 ‘lessons learned’ are lessons, rather these are either findings/conclusions drawn or recommendations. It is essential that the right lessons are drawn as these will serve as pointers for future interventions both for the Government of Angola, the Bank and other Bank projects elsewhere.’ The evaluator accepts this comment, but feels that the issue of ‘lessons learned’ is primarily a matter of language rather than substance. Accordingly, the evaluation has made an effort to re-present the summary observations of the PCR as ‘lessons learned’ in the PCREN, without diminishing the validity of the observations made in the PCR. This is a matter that the AfDB could perhaps address through training. The PCR is rated as ‘On time’ within the extended time-frame of the project, and so scores 4 and the participation is excellent also ranked 4 giving an overall score of 4 for conformity

8. SUMMARY OF THE EVALUATION

Criteria PCR PCREN Reason for disagreement/ Comments

DIMENSION A: RELEVANCE 3.5 3.5

Relevance of project development objective 4 4

Relevance of project design 3 3

DIMENSINO B: EFFECTIVENESS 3 3

Development objective (DO) 3 3

DIMENSION C: EFFICIENCY 3 2.67

Timeliness 2 2

Resource use efficiency 4 3 Some project activities not completed.

Cost-benefit analysis N/A N/A

Implementation progress (IP) 3 3

DIMENSION D: SUSTAINABILITY 3.5 3

Financial sustainability 3 3

Institutional sustainability and strengthening of capacities

3 3

Ownership and sustainability of partnerships 4 3

Environmental and social sustainability 4

AVERAGE OF THE DIMENSION RATINGS 3.25 3.04

OVERALL PROJECT COMPLETION RATING S S

Bank performance 3 3.71

Borrower performance 3 2.83

Performance of other stakeholders 3 3

Overall PCR quality 3.56

9. PRIORITY FOR FUTURE EVALUATIVE WORK: PROJECT FOR PERFORMANCE

EVALUTION REPORT, IMPACT EVALUTION, COUNTRY/SECTOR REVIEWS OR

THEMATIC EVALUATION STUDIES:

- Project is part of a series and suitable for cluster evaluation

- Project is a success story

- High priority for impact evaluation

- Performance evaluation is required to sector/country review

- High priority for thematic or special evaluation studies (Country)

- PPER is required because of incomplete validation rating

Major areas of focus for future evaluation work:

a) Performance evaluation is required for sector/ country review

b) Cluster evaluation (institutional support)

c) Sector evaluation (budgetary support or public finance management reforms)

Follow up action by IDEV: Identify same cluster or sector operations; organize appropriate work or consultation mission to

facilitate a), b) and/or c).

Division Manager clearance Director signing off

Data source for validation:

Task Manager/ Responsible bank staff interviewed/contacted (in person, by telephone or

email)

Documents/ Database reports

Attachment:

PCR evaluation note validation sheet of performance ratings

List of references

Appendix 1

PROJECT COMPLETION REPORT EVALUATION NOTE Validation of PCR performance ratings

PCR rating scale:

Score Description 4 Very Good – Fully achieved with no shortcomings 3 Good – Mostly achieved despite a few shortcomings 2 Fair – Partially achieved. Shortcomings and achievements are roughly balanced 1 Poor – very limited achievement with extensive shortcomings

UTS Unable to score/rate NA Non Applicable

Criteria Sub-criteria PCR

work

score

IDEV

review Reasons for deviation/comments

RELEVANCE Relevance of the project

development objective

(DO) during

implementation

4

Relevance of project

design (from approval to

completion)

3

OVERALL RELEVANCE SCORE

EFFECTIVENESS* Effectiveness in delivering outcomes

Outcomes 4 3

Effectiveness in delivering output

Outputs 4 4

Development objective (DO)

Development objective

rating

3 3

Unanticipated outcomes (positive or negative not considered in the project logical

framework) and their level of impact on the project (high, moderate, low)

Institutional

development

Gender

Environment & climate

change


work

score

IDEV


Poverty reduction

Private sector

development

Regional integration

Other (specify)

EFFECTIVENESS OVERALL SCORE 3 3

EFFICIENCY Timeliness (based on the

initial closing date) 2 2

Resource used

efficiency 4 3

Cost-benefit analysis N/A N/A

Implementation progress

(from the IPR) 3 3

OVERALL EFFICIENCY SCORE 3 2.67

SUSTAINABILITY Financial sustainability

3 3

Institutional

sustainability and

strengthening of

capacities

3 3

Ownership and

sustainability of

partnerships

4 3

Environmental and

social sustainability 4

*The rating of the effectiveness component is obtained from the development objective (DO) rating in the latest IPR of the

project (see Guidance Note on the IPR).

The ratings for outputs and outcomes are determined based on the project’s progress towards realizing its targets, and the

overall development objective of the project (DO) is obtained by combining the ratings obtained for outputs and outcomes

following the method defined in the IPR Guidance Note. The following method is applied: Highly satisfactory (4),

Satisfactory (3), Unsatisfactory (2) and Highly unsatisfactory (1).


Work

score

IDEV


BANK

PERFORMANCE Proactive identification and

resolution of problems at different

stage of the project cycle

4

Use of previous lessons learned

from previous operations during

design and implementation

3

Promotion of stakeholder

participation to strengthen

ownership

3

Enforcement of safeguard and

fiduciary requirements

4

Design and implementation of

Monitoring & Evaluation system

4

Quality of Bank supervision (mix

of skills in supervisory teams,

etc.)

4

Timeliness of responses to

requests

4

OVERALL BANK PERFORMANCE SCORE 3 3.71

BORROWER

PERFORMANCE

Quality of preparation and

implementation 2

Compliance with covenants,

agreements and safeguards 3

Provision of timely counterpart

funding 3

Responsiveness to supervision

recommendations 3

Measures taken to establish basis

for project sustainability 3

Timeliness of preparing requests 3

OVERALL BORROWER PERFORMANCE SCORE 3 2.83

PERFORMANCE

OF OTHER

STAKEHOLDERS

Timeliness of disbursements by

co-financiers N/A

Functioning of collaborative

agreements 3

Quality of policy dialogue with

co-financiers (for PBOs only) N/A

Quality of work by service

providers 3

Responsiveness to client demands 3

OVERALL PERFORMANCE OF OTHER

STAKEHOLDERS

3 3

The overall rating is given: Very Good, Good, Fair and Poor.

(i) Very Good (HS) : 4

(ii) Good (H) : 3

(iii) Fair (US) : 2

(iv) Poor (HUS): 1

DESIGN, IMPLEMENTAION AND UTILIZATION OF MONITIRING AND

EVALUATION (M&E)

Criteria Sub-criteria IDEV

Score Comments

M&E DESIGN M&E system is in place, clear,

appropriate and realistic 4

Monitoring indicators and

monitoring plan were duly

approved

4

Existence of disaggregated gender

indicator 2

Baseline data were available or

collected during the design 3

Other, specify

OVERALL M&E DESIGN SCORE 3

M&E

IMPLEMENTA-

TION

The M&E function is adequately

equipped and staffed 3

OVERALL M&E IMPLEMENTATION SCORE 3

M&E

UTILIZATION

The borrower used the tracking

information for decision 3

OVERALL M&E UTILIZATION SCORE 3

OVERALL M&E PERFORMANCE SCORE 3

PCR QUALITY EVALUATION

Criteria PCR-EVN

(1-4) Comments

QUALITY OF PCR

1. Extent of quality and completeness of the PCR

evidence and analysis to substantiate the ratings of

the various sections

4

2. Extent of objectivity of PCR assessment score 4

3. Extent of internal consistency of PCR assessment

ratings; inaccuracies; inconsistencies; (in various

sections; between text and ratings; consistency of

overall rating with individual component ratings)

4

4. Extent of identification and assessment of key

factors (internal and exogenous) and unintended

effects (positive or negative) affecting design and

implementation

4

5. Adequacy of treatment of safeguards, fiduciary

issues, and alignment and harmonization

4

6. Extent of soundness of data generating and

analysis process (including rates of returns) in

support of PCR assessment

3

7. Overall adequacy of the accessible evidence (from

PCR including annexure and other data provided)

3

8. Extent to which lessons learned (and

recommendations) are clear and based on the PCR

assessment (evidence & analysis)

2 Analysis and observations of the PCR were good,

but needed re-working to fit the ‘lessons learned’

and ‘recommendations’ of the AfDB protocol

(Guidance Notes). It is an issue for training.

9. Extent of overall clarity and completeness of the

PCR

4

Other (specify)

PCR QUALITY SCORE 3.55

PCR compliance with guidelines (PCR/OM ; IDEV)

1. PCR Timeliness (On time = 4; Late= 1) 4

2. Extent of participation of borrower, Co-financiers

& field offices in PCR preparation

4

3. Other aspect(s) (specify)

PCR COMPLIANCE SCORE 4

*** rated as Very Good (4), or Good (3), or Fair (2), or Poor (1)

References aid memoire June 2016r.docx. 29 Feb – 4 Mar 2016. aid memoire draft November 20132.doc. 18 – 22 Nov 2013. Aide-Memoire for Project Completion. 27 June – 7 July 2016. Back to office March 2016.docx. 29 Feb – 4 March 2016. Back to office May 2014 (2).docx. 11-16 May 2014. BTOR PCR Mission June 27-7 July.docx. 9 July 2016. feedback report consultant.docx. 4 May 2012. Final Report 28.4.12.docx – 22 Feb. 19 Mar 2012. ICBPR MTR Report.docx. Oct 2014. IPR-Angola- ICBPIR PCR 2016.docx. 27 June – 7 July 2016. Plano Formação_Versão Final… 3 Mar 2014. Relatório BAD 3° Trim2015.Port.docx. Q3 2015, 1 Jul – 30 Sep. Aide Memoire_ICBPIP.pdf. 24 Aug – 4 Sept 2014. AIDE-MEMOIRE February2012.pdf. 13 – 20 Feb 2012. AM Missao Setembro 2012.pdf. 10 – 15 Sept 2012. Angola – APPROVED – ICBPR.pdf. Dec 2010. ANGOLA (2)FFCO3.pdf. 16 April 2014. Estudo de Diversificação da Econ… June 2015. PCR.pdf. 29 July 2016. Review of the audit financial statements… 8 Jul 2015.

List of Acronyms ADF African Development Fund AfDB African Development Bank ASF Annual Stakeholders Forum BNA Banco National de Angola (Angolan Central Bank) CSP Country Strategy Paper EMTA Economic Management Technical Assistance EU European Union GDP Gross Domestic Product GEP Gabinete de Estúdios e Planificação (Office of Studies and Planning) GoA Government of Angola ICT Information and Communication Technology IMF International Monetary Fund KPI Key Performance Indicator MDA Ministry, Department and Agency MDG Millenium Development Goal M&E Monitoring and Evaluation MINEC Ministry of Economic Coordination MINFAMU Ministry of Family and Promotion of Women MINPLAN Ministry of Planning NDPI National Directorate of Public Investments ODA Official Development Assistance PCR Project Completion Report PD Presidential Decree PFM Public Financial Management PIP Public Investment Programme/Project PIU Project Implementation Unit PMU Project Management Unit PMT Project Management Team RMC Regional Member Country SBA Stand-By-Agreement SIPIP System for Public Investment Management and Programming UA Unit of Account UNDP United Nations Development Programme