pcr evaluation note for public sector operations · the project’s rationale is that in a tropical...

1. BASIC INFORMATION a. Basic project data Project title: Updating the Studies of the Gue-Gue, Lowe-IAI and Terre NouvelleWatersheds Project code: P-GA-EB0-002

Instrument number(s): 56001 55001 702

Project type: Studies Sector: Storm water purification Country: Gabon Environmental categorization (1-3) : 1Processing Milestones Key Events Disbursement and

Closing date Date approved: Oct 2009

Cancelled amount: - Original disbursement deadline: 30/06/2011

Date signed: Feb 2010 Supplementary financing: - Original closing date: 31/12/2011

Date of entry into force :

Restructuring: - Revised disbursement deadline: 30/09/2013

Date effective for 1st disbursement: Feb 2010

Extensions (specify dates): from 31/12/2011 to 25/04/2014

Revised closing date: 25/04/2014

Date of actual 1st : 02 Feb 2010b. Financing sources

Financing source/instrument (MUA)

Approved amount(MUA) :

Disbursed amount(MUA) :

Percentage disbursed(%):

Grant : EUR 1,579,100 EUR 1,579,100 98%Government: (in kind) EUR 297,112 EUR 297,112 197% TOTAL : EUR 1,876,212 EUR 1,876,212 106% Co-financiers and other external partners: -

Execution and implementation agencies: Studies and Works Coordination Unit (UCET) at the Ministry of Public Worksc. Responsible Bank staf

Position At approval At completion Regional Director Marlène KANGA Sector Director Sering JALLOW Mohamed El AZIZI Sector Manager Freddie KWESIGA Akissa BAHRI Task Manager Francis Daniel BOUGAIRE Francis Daniel BOUGAIRE Alternate Task Manager

Jean-Michel OSSETE Jean-Michel OSSETE

PCR Team Leader - Francis Daniel BOUGAIRE PCR Team Members - Maher MANI (Consultant)d. Report data PCR Date : 10/03/2015 PCR Mission Date: From: 17/01/2015 To: 24/01/2015 PCR-EN Date: 08/06/2018 Evaluator/consultant : Daniel VAN ROOIJEN

Peer Reviewer/Task Manager:

PCR EVALUATION NOTE FOR PUBLIC SECTOR OPERATIONS

2. PROJECT DESCRIPTIONSummary from Appraisal Report including addendum/corrigendum or loan agreement, and taking into account any modification that occurred during the implementation phase.

The project discussed here is on urban catchment water management in Gabon, focussing on three watersheds in Gabon’s capital Libreville. The projects main focusing is the updating of drainage and environmental maps for these watersheds. A second component focusses on capacity strengthening of institutions working in the urban water sector. The project, with a design duration of 15 months, was funded by the African Development Bank and the Government of Gabon et a été executé sur une durée de 27 mois, sans considérer l’étude complémentaire portant sur la problématique de la gestion des ordures ménagères et déchets solides d’une durée de 28 mois. The project was executed and implemented by the Studies and Works Coordination Unit (UCET) at the Ministry of Public Works of the Government of Gabon. Some delays have occurred during the implementation phase, leading to an extension of the originalend data of 31/12/2011 to 25/04/2014 (environ 28 mois). The contractual amount of a study contract amountwas less than estimated (EUR 1,225,350 estimated against actual EUR 769,525). This has provided room forsome flexibility to adjust and develop some components including the addition of an important study on solid waste disposal.

a. Rationale and expected impacts:

The project’s rationale is that in a tropical climate with high annual rainfall, and despite relatively good access to water and sanitation, the poor state of the urban drainage system is a major cause for the high incidence of water-borne diseases and outbreaks which in turn has a negative effect on the urban environmental and human health for the urban population. A lack of knowledge and capacity exists with regards to storm water management which has kept local government from properly addressing the above issue. The project aims to reduce the health burden of the urban population by addressing the poor state of the drainage system at watershed level through building knowledge and capacities which will form a basis for major infrastructural investments and securing financial support.

No major changes have been made during the implementation phase, with regards to the project rationale and intended beneficiaries.

b. Objectives/Expected Outcomes:

The overall project objective was to improve the living conditions of the populations of the Gué-Gué, Lowe-IAI and Terre Nouvelle basins by improving rainwater drainage in the three watersheds, following the creation of the necessary conditions for the execution of works. The specific project objective was to enhance the technical, socio-economic and environmental knowledge of the Gué-Gué, Lowe-IAI and Terre Nouvelle basins and the capacities of the stakeholders in the development of the watersheds.

Expected outcomes were, as reported in the logframe:

In the short term: - Feasibility studies including ESIA and ESMP prepared- Preliminary detailed plans updated - Bidding Documents for the works developed - Executives of CIPA Member governments and associations working in the basins have been

trained - Computer software and hardware acquired- Sanitation database has been established- Feasibility studies, DD and bidding documents approved- Donors mobilized

In the medium term:- Technical, socio-economic and environmental knowledge is improved- Capacities of stakeholders for the development of the three watersheds are strengthened through

training and procurement of appropriate logistics means.

In the long term: - Improved living conditions through the development of rainwater drainage schemes- Poverty reduced through the drainage of the three watersheds

Intended beneficiaries were the Ministry of Equipment, Infrastructure and Construction, several other ministries, directorates within these ministries and the Interministerial Commission for Sanitation (CIPA)),Development agencies and associations working in the three watersheds in sanitation and health and related development sectors, donors, consulting firms and the estimated 265,000 people living in the three watersheds.

c. Outputs and intended beneficiaries:

Projects outputs were (by component):

Component A : Updating of Studies, benefiting project staff and consulting firms- Feasibility studies including ESIA and ESMPs developed - Detailed preliminary draft (DPD) updated- Works bidding documents (BD) prepared

Component B : Capacity-building, benefiting mainly staff at the Ministry and other stakeholders working in the sector

- Officials of CIPA administrative bodies and associations involved in the basins trained- Computer software and hardware for studies acquired - Sanitation database established

Component C : Project management, benefiting project staff, donors, consulting firms- Feasibility study, DPD and BD validated. - Donors mobilized

d. Principal activities/Components:

The principal project activities were the following, as listed by project component:

Component A: Update of studies

- Feasibility studies, which will include:o a socio-economic studyo the preparation of the preliminary draft plans (APS)o an Environmental and Social Impact Assessment (ESIA) and an Environmental and Social

Management Plan (ESMP)o a land surveyo an institutional study on the management of primary and secondary drains.o an assessment of the capacity building needso a review of the design criteria to analyse the technically and economically appropriate

solutions proposed in the existing studies.

- detailed preliminary design studies (DPAs), which will include:

o topographic surveys;o geotechnical reconnaissance;o hydraulic studies and water analyses;o APD studies of sanitation works (main outfall from its origin until its completion at sea,

protection of the banks against erosion, canals, scuppers, culverts, looks etc ...);o ODA studies of crossing structures (small structures such as scuppers);o studies of general earthworks;o studies of expropriation and network relocation (water, telephone and electricity) and;o the detailed pre - survey, the estimate, the modalities and the schedule of execution of the

works.

- Bidding Documents (DAO)

Component B: Capacity building of stakeholders- Training in project management and stormwater sanitation management;- Training of information, education and communication (IEC) relays on sanitation;- Training in database management on sanitation and;- Training in environmental and social impact monitoring.

Component C: Study Management- establishment of a Pilot Steering Committee of the Study (CPE)- establishment within the DGCE of the Coordination Unit of the (UCE)- acquisition of an all-terrain vehicle- acquisition of office furniture- organization of the study launching workshops, validation of feasibility studies, ODA and OAD

work - organization of the Round Table of Donors.

3. PROJECT PERFORMANCE ASSESSMENT

RELEVANCE

a. Relevance of the project development objective:

The project was very well embedded in national strategies and sectoral priorities of the government of Gabon as well as strategies and priorities of the Bank. The appraisal reports states that improving the health situation of the population has always been a priority area of intervention for the Gabonese Government. Since 1977, a sanitation master plan has been in effect for the whole country including urban areas, which is currently being updated with support from the Bank, in order to establish a more comprehensive framework that includes other issues such as storm water, solid waste, wastewater etc. The project fits very well into the 2006-2012 Priority Sanitation Program which aims to improve stormwater drainage conditions in the country's two main cities including Libreville. The Country Strategy Paper 2011-2015 published in 2011 also mentions water and sanitation as a key area of intervention in Gabon.

The projects fits well into the second pillar of the Gabon joint 2016-2020 country strategy paper and country portfolio performance review report published by the African Development Bank in 2016, which states that “the improvement of the living environment, particularly through access to adequate drinking water supply and sanitation services, contributes to such development. It also fits well into the Banks Policy for integrated water resources management, which has prioritised attention to the linkages between water resources, health, environmental sustainability, watershed degradation and urbanisation. On watershed degradation, the project is directly relevant in addressing the issue of watershed degradation,stated in the policy as: “uncontrolled urbanization negatively affect the state of water resources occurrences

and the natural drainage systems. The results are poor water quality, land degradation, increasing incidence and severity of floods, soil and gully erosion and reduced retention of water in river basins”. Environmental degradation from domestic, industrial and agricultural waste is mentioned in the Africa Water Vision 2025 as one of the environmental factors influencing future water developments. Durant l’exécution, l’objectif du projet est resté parfaitement aligné sur les stratégies sectorielles et d’intervention de la Banque, les priorités et les stratégies de développement du pays et les besoins prioritaires des bénéficiaires

Ainsi la revue confirme comme très satisfaisante, la pertinence de l’objectif de développement du projet avec la note de 4 sur 4 conforme à celle du RAP. The review and the Evaluator concur with the PCR rating and also rate this criterion as highly satisfactory (4).

b. Relevance of project design (from approval to completion):The design of the project is straightforward and logical, consisting of 3 main components of carrying out technical studies, capacity strengthening and project management. The time period could have been designed a bit longer, to allow for sufficient time to carry out the studies, the trainings and prepare the bidding documents. The log frame follows a clear logic from outputs to outcomes to impact, and is supported by solid assumptions and realistic risks. The institutional arrangement of implementations has assured that all stakeholders are involved in decision making aiming to leverage participation and ownership in the project. There are a few side notes to make. It is unclear why the project has not included hydraulic modeling of watersheds for different land use/cover and climate scenarios. It is not clear based onwhat criteria the 3 watersheds were selected (in comparison to the other urban watersheds). The project was executed and implemented by the Studies and Works Coordination Unit (UCET) at the Ministry.

Le RAP juge la conception du projet bonne et satisfaisante car elle s’est adaptée à la situation du pays et à l’urgence de l’intervention, en mettant en place un montage flexible qui garantit en même temps la réalisation des activités et l’appropriation des résultats pour enchaîner avec la réalisation des travaux. De laphase de conception à la phase de clôture, la conception initiale du projet a montré des insuffisances et des ajustements techniques majeurs se sont avérés nécessaires pendant l’exécution du projet. Ainsi, la conception du projet n’a pas pris en compte l’intégralité des enjeux de tous les sous-secteurs de l’assainissement (eaux usées, eaux pluviales et déchets solides). Cette situation a conduit à la réalisation des études complémentaires portant sur la problématique de la gestion des ordures ménagères et déchets solides d’une durée de 28 mois. En plus, le calendrier initial d’exécution de l’étude s’est avéré trop court. Prévue pour 16 mois à l’évaluation, l’étude s’est réalisée sur délai global de 49 mois y compris le délai de 28mois pour les études complémentaires. Par ailleurs, en matière de mise en œuvre du projet, les différents rapports de suivi et supervision consultés par la revue relèvent des insuffisances au niveau de la gestion financière du projet. La première mission d’audit est intervenue très en retard (décembre 2012), ce qui correspond à la fin de l’exécution de l’étude.

Au vu de ce qui précède, la revue contrairement à l’évaluateur, confirme comme satisfaisante, la pertinence de la conception du projet avec la note de 3 sur 4 conforme à celle du RAP.

EFFECTIVENESS

c. Effectiveness in delivering outputs :

The outputs expected in the short term have overall been delivered within the expected time frame despite afew delays during implementation and a few issues under capacity building. Under component A, the carrying out of the feasibility studies on the development of the Gué-Gué, Lowé-IAI and Terre Nouvelle watershed, as well as the various design studies and bidding documents (BDs) have been effectively prepared. Under component B, all the training courses were carried with a few less participants than expected. Computer software and hardware for carrying out the studies have been delivered as planned. An

additional request for a training for logician in the project was never met, however this was considered outside the project design, so it has not affected performance for this output. Regarding the setting up of a functional database, this was only partly achieved; data is available and associated trainings have been carried out but database management was not achieved due to the reported lack of guidelines in the appraisal report. Under component C, project management, a donor round table meeting was held and 4 workshops were organised to kick-off the project and to validate important project outputs such as the feasibility study, DPD and BD documents. Actuellement, les études permettent de produire des données avec une certitude plus précise sur les caractéristiques des aménagements projetés dans les trois (3) bassins.Le choix et les caractéristiques des ouvrages conçus tiennent compte des conditions réelles des trois bassins versants. The review and the evaluator rate this criterion with satisfactory (3), which is line with the PCR rating.

d. Effectiveness in delivering outcomes:

The project has generally delivered reasonably well in terms of delivery on outcomes. Some critical technical aspects were not addressed in the watershed studies (as reported in the PCR) such as climate change, which would be important in relation to the associated climatic scenarios on the hydrology. Solid waste (leading to clogging of drainage canals hereby worsening the flooding) was initially not addressed butat a later stage it was included as an initial study. The ToR’s for carrying out the watershed studies should have included all assessing technical aspects of relevance to the stated issues of drainage/flooding/sanitation.Also the institutional framework was not well addressed. The PCR reports that pubic consultations were held with the participation of all stakeholders, however it is not clear how communities in the watersheds were consulted and how their perspectives were considered in the watersheds development plans. Communication with the communities is considered critical for social sustainability.The review and the evaluator rate this criterion with satisfactory (3), which is line with the PCR rating.

e. Project development outcome:

The overall project development objective was to improve the living conditions of the populations in the three target watersheds, by improving rainwater drainage, following the creation of the necessary conditions for the execution of works. Very clear and tangible progress have been made at project output and outcome level, which have provided an important prerequisite to accomplish the project development outcome at post implementations stage. This progress lies in the building of data and knowledge through studies and capacity building of relevant stakeholders. Physical infrastructural development works have been fully prepared. La table ronde des bailleurs de fonds a enregistré la participation de six bailleurs et a abouti à un engagement de deux bailleurs (BID pour financer terre nouvelle et AFD pour financer Gué-Gué). The PCR reports that donors have shown commitment to financially support most parts of the infrastructural development plans, however this still needs to be secured.

The review rates this criterion with satisfactory (3), which is line with the PCR rating.

f. Beneficiaries:

The PCR has listed 4 groups of beneficiaries which are the same under actual as planned. These are; Gabonese Government, Ministry of Public Works, DGCE, DGE, Town Councils, Donors, development agencies, the population affected by the work (all direct beneficiaries) and the entire population in the three watersheds (indirect beneficiaries).

While the type of beneficiaries are sound, there is a lack of quantification as to how many people have actually benefited and how many people were anticipated to benefit. There is no disaggregation by sex. The progress reporting (100, 50, 0%) seems arbitrary and cannot be verified by the evaluator. The poor reporting on beneficiaries should be considered a missed opportunity when it comes to disseminating project achievements and demonstrating the impact it has made on people. Les données sur les bénéficiairesau niveau des 3 bassins versants auraient pu être désagrégées par genre. La revue tout comme l’évaluateur considèrent la couverture des bénéficiaires du projet est jugée insatisfaisante (2). This criterion is rated as

unsatisfactory (2).

g. Unanticipated additional outcomes (positive or negative, not taken into consideration in the project logical framework) :

The PCR list a single unanticipated additional outcome which is the socio-economic / commercial opportunities arising from the construction of roads along the drainage canals. The evaluator agrees that this is a real positive outcome of the project which was not anticipated in the appraisal report.

Additional outcomes may exist in relation to business opportunities for solid waste disposal and recycling aswell as for faecal sludge treatment. Depending on the exact layout of the watershed development plans there may be opportunities for expansion of urban agriculture boosting income generation of farmers and urban food security. Possible green zones or zones designed for flood control may hold potential for other ecosystem services such as increasing biodiversity and supporting carbon absorption. The review and the evaluator rates this criterion as satisfactory (3).

EFFICIENCY

h. Timeliness:

Some discrepancies exist in the reporting of project duration. While the appraisal report mentions 16 months project duration, the PCR noted under the section timeliness a 15 months designed implementation period against a 27 months actual implementation period. Initialement prévue pour 16 mois à l’évaluation (accord de don), l’étude s’est réalisée sur délai global de 49 mois y compris le délai de 28 mois pour les études complémentaires pour prendre en compte l’intégralité des enjeux de tous les sous-secteurs de l’assainissement (eaux usées, eaux pluviales et déchets solides). The various reasons for delays for particularactivities have sufficiently been justified. Le rapport entre délai d’exécution prévisionnel et délai réel d’exécution à compter de la date de satisfaction des conditions de premier décaissement est de 0,30, et de cefait jugé très insatisfaisant car inférieur à 0,50. The ratio of planned and actual implementation time is 0.30, which results in a rating of highly unsatisfactory (1) for this criterion.

i. Resource use efficiency:

The ratio of the median percentage physical implementation of the project outputs and commitment rate is 0.94 which is within the category of 1-0.75. This corresponds with a rating of satisfactory (3). The project largely delivered the outputs expected within the available budget. Pour la revue et l’évaluateur, tout comme pour le RAP, la performance en matière d’efficience dans l’utilisation des ressources du projet est jugée satisfaisante avec la note de 3 sur 4.

j. Cost-benefit analysis:

N/A

k. Implementation progress:

Based on the available information, the project has complied well with procedures and protocols set by the Bank. Quarterly reports were prepared well and have been timely submitted to the Bank. Requests for disbursements and other financial procedures have been followed. Monitoring and evaluation of beneficiaries was not properly done which is evident from the lack of qualitative and quantitative data. Les processus d’exécution sont jugés satisfaisants pour la plupart, et ont permis d’obtenir les résultats escomptés. The review and the evaluator rate this criterion as satisfactory (3), in line with the PCR score for this item.

SUSTAINABILITY

l. Financial sustainability:

As part of the project, substantial efforts were made to attract donors for financing of the 3 watershed development plans. The PCR reports that donors have shown commitment to financially support most parts of the infrastructural development plans, however this still needs to be secured. The management of the urban watersheds with regards to the drainage network and solid waste disposal is extremely critical for controlling risks associated to urban flooding. There is lack of information on the financial health of the local authorities responsible for operation and maintenance of the drainage network and solid waste disposal. This makes it difficult to assess the financial sustainability of integrated urban watershed planning and management. Despite the lack of information, this criterion is rated as satisfactory (3).

m. Institutional sustainability and strengthening of capacities:

The project has demonstrated well to strengthen institutions working in the sector through a number of activities, including trainings and other events. Based on the documentation it has become clear that institutional responsibilities with regards to operation and maintenance of solid waste, sanitation, storm water drainage and road infrastructure still needs to further advanced and requires continued capacity strengthening through trainings and support to operationalisation of a functional database. It must be noted that project achievements are not well supported by data. It would have been useful if actual benefits from capacity strengthening had been measured and recorded in simple surveys to identify remaining knowledge gaps and priorities. Le montage institutionnel prévu pour cette étude est jugé cohérent et favorise l’implication de toutes les parties prenantes nationales (UCET, CIPA, services techniques, mairies, ONG, populations) en harmonie avec la Banque (FAE), les PTF et les prestataires de services. This criterionis rated as satisfactory (3) which is in line with the respective PCR score.

n. Ownership and sustainability of partnerships:

The project has ensured ownership from relevant stakeholders including service providers through their participation in various events and consultations. As mentioned earlier, it is unclear to what extend communities living in the watershed have been involved in the project, in order to ensure their future participation as well. The institutional framework was not well addressed in the feasibility studies for the watersheds, which is considered a critical omission with regards to ensuring ownership and sustainability ofpartnerships. Despite the above concerns, the evaluator would rate this criterion as satisfactory (3).

o. Environmental and social sustainability:

The PCR reports that pubic consultations were held with the participation of all stakeholders, however it is not clear how communities in the watersheds were consulted and how their perspectives were considered in the watersheds development plans. Communication with the communities is considered critical for social sustainability.

The available documentation has not convincingly demonstrated that environmental values have sufficiently been taken into account. There could have been opportunities for environmental development agencies to play a role in watershed management. Some of the possible unanticipated positive outcomes of the project (urban agriculture, ecosystem services, resource recovery) were not at all considered, but could have added value to ensuring environmental sustainability. The review and the evaluator would rate this criterion as unsatisfactory (2), which is lower than the PCR rating of satisfactory.

4. PERFORMANCE OF STAKEHOLDERS

a. Bank performance:

The Bank has supervised this project with satisfaction. It has complied with all the procedures and

protocols during design implementation and completion phase of the project. Sufficient supervisory missionhave been carried out. As noted in the PCR, at the design stage could perhaps have done a better job in making sure that the drainage and flooding issues are understood in integrated manner, namely including the issues of solid waste. Comments from the Bank reflecting on its own performance were lacking in the PCR. Le RAP confirme la conception pragmatique du projet pour répondre à des besoins urgents de la population. En plus, la recherche de complémentarité avec d’autres bailleurs de fond pour concrétiser le financement des travaux, définis par les études élaborées, représente un autre atout de la bonne performance de l’action de la Banque. Cependant, des insuffisances ont été relevées au niveau de la conception initiale du projet qui n’a pas tenu compte des interactions existantes entre l’assainissement pluvial et le système de collecte des déchets solides. Cette situation a nécessité des études complémentaires d’une durée de 28 mois. The review and the evaluator would rate the Bank performance as satisfactory (3) which is slightly lower than the PCR score as highly satisfactory (3.5).

b. Borrower performance:

The project was essentially managed by the Study and Works Coordination Unit (Unité de Coordination del’Etude et des Travaux), placed in the Ministry of Public Works. Overall, the unit has done a good job in running the project during all its phases. The Borrower’s performance was efficient and effective and the Borrower made sure that all important stakeholder were included at consultation and other events. In terms of monitoring and evaluation a better job could have been with regards to measuring the capacity built and identifying knowledge gaps that have remained among stakeholders. The review and the evaluator rates the bank performance as satisfactory (3) which is the same as the PCR score (3).

c. Performance of other stakeholders:

Other stakeholders are considered the consulting firms contracted to carry out the major watershed studies. Based on the observations made in the PCR narrative, the quality of their products is considered good. Due to the unavailability of the major outputs (reports etc.) delivered by the consulting firms, the evaluator is unable to fully evaluate their performance. The review and the evaluator would therefore concur with the PCR rating and would rate the performance of the consulting firms as satisfactory (3).

5. SUMMARY OF OVERALL PROJECT PERFORMANCE

a. Overall assessment:

In conclusion, the project has performed well with regards to the key dimensions of this evaluation (relevance, effectiveness, efficiency and sustainability). The project was very well embedded in national strategies and sectoral priorities of the government of Gabon as well as strategies and priorities of the Bank. The design of the project is straightforward and logical, consisting of 3 main components of carrying out technical studies, capacity strengthening and project management. The outputs expected in the short term have overall been delivered within the expected time frame despite a few delays during implementation and a few issues under capacity building. The clear and tangible progress made at output and outcome level has provided an important prerequisite to accomplish the project development outcome at post implementations stage. Regarding sustainability, some issues have been noted regarding financial, social and environmental aspects. An M&E framework was not fully functional which has resulted in some missed opportunities firstly in terms of showing impact and secondly in terms of capturing progress made and remaining gaps with regards to capacity strengthening of staff and relevant institutions in the sector. Dans l’ensemble, et malgré les insuffisances du dispositif de suivi-évaluation préconisé pour la présente étude, la performance globale du projet est satisfaisante avec la note de 3 sur 4

b. Design, implementation and utilization of the M&E (appreciation of the evaluator):

Au niveau de la conception, le dispositif prévu à l’évaluation de l’étude était classique comprenant le suivi interne et externe, les missions de supervision de la Banque, les ateliers de validation des résultats des différentes phases de l’étude et une évaluation terminale incluant le rapport d’achèvement. Un plan de suivi-évaluation de l’exécution de l’étude devrait être mis en place par l’UCE sur la base de la matrice du cadre logique du projet. Ce plan de suivi-évaluation s’inscrira dans le cadre du suivi-évaluation des projets du MITPC. Au vu de ce qui précède, la conception du système de suivi-évaluation est jugée insatisfaisante par la revue avec la note 2 sur 4.

En terme de mis en œuvre, l’UCE a présenté des rapports trimestriels et annuels sur l’état d’avancement del’étude. Ces rapports devaient préciser l’efficacité des interventions et étaient élaborés conformément au format usuel de la FAE. Un rapport de fin de l’étude portant les activités réalisées et la situation financière du projet sera élaboré par le donataire à sa clôture. Monitoring and evaluation of project performance and beneficiaries reached could have been done in a better way. Monitoring and evaluation of beneficiaries was not properly done which is evident from the lack of qualitative and quantitative data. Most of the performance indicators used are considered relatively limited (no. of studies, no. meetings) and do not capture quality aspects of performance. Au vu de ce qui précède, la revue confirme la mise en œuvre du S&E insatisfaisante avec la note 2 sur 4.En termes d’utilisation des résultats du suivi-évaluation, les rapports trimestriels, annuels d’avancement du projet et les rapports d’audit étaient produits régulièrement et souvent avec retard. Le RAP ne confirme pas l’utilisation régulière du dispositif de suivi-évaluation la prise de décision pour la bonne exécution du projet. Au vu de ce qui précède, la revue considère l’utilisation du S&E dans la prise de décisions sur la bonne marche du projet insatisfaisante avec la note 2 sur 4. Based on the above issues, the review and the evaluator rate this criterion with unsatisfactory (2), which is in line with the PCR rating (2).

6. EVALUATION OF KEY LESSONS LEARNED AND RECOMMENDATIONS

a. Lessons learned:

The Evaluation’s opinion of each lesson is given in summary below:

1. NOT VALIDATED, TOO PROJECT SPECIFIC2. VALIDATED3. NOT VALIDATED, MOVED TO RECOMMENDATIONS4. NOT VALIDATED, TOO PROJECT SPECIFIC

The evaluator would suggest to add the following lesson learned.

The lack of a robust M&E framework has not enabled the project to report progress on important outcomes such as the quality of performance of consulting firms and capacity strengthening.

b. Recommendations:

The Evaluation’s opinion of each recommendation is given in summary below:

1. NOT VALIDATED, TOO PROJECT SPECIFIC2. VALIDATED3. NOT VALIDATED, TOO PROJECT SPECIFIC4. NOT VALIDATED, TOO PROJECT SPECIFIC

The evaluator would suggest to add the following 2 recommendations:

For this project and similar ones, it is recommended to distinguish between on one hand a project steering committee for the validation of progress documents and other project outputs as well as project decisions made and main choices of the study and on the other hand a technical committee for project monitoring and providing assistance to consulting firms. In this regard it is recommended to attach the steering committee to an appropriate already existing institution.

In development projects like this one, it is highly recommended to include an M&E framework that captures the quality of performance of contracted consulting firms and progress made and remaining gaps with regards to capacity strengthening of institutions.

7. COMMENTS ON PCR QUALITY AND TIMELINESSThe overall PCR rating is based on all or part of the criteria presented in the annexe and other: The quality of the PCR is rated as highly satisfactory (4), satisfactory (3), unsatisfactory (2), and highly unsatisfactory (1). Le timeliness of the PCR is rated as on time(4) or late (1). The participation of the Borrower, co-financier, and the bank’s external office(s) are rated as follows: Very Good (4), Good (3), Fair (2), Poor (1).

The PCR quality is rated 3.11 based on the scoring of the 9 criteria listed in Annex 1. The extent of quality and completeness of the PCR evidence and analysis to substantiate the ratings of the various sections was considered satisfactory. The extent of objectivity of PCR assessment score was considered satisfactory. The extent of internal consistency of PCR assessment ratings; inaccuracies; inconsistencies was considered as satisfactory. The extent of identification and assessment of key factors (internal and exogenous) and unintended effects (positive or negative) affecting design and was considered satisfactory. The adequacy of treatment of safeguards, fiduciary issues, and alignment and harmonization was considered very satisfactory. The extent of soundness of data generating and analysis process (including rates of returns) in support of PCR assessment was considered unsatisfactory. Some of the data presented in the PCR was incomplete and was missing a clear methodology on how the data was calculated or collected. The overall adequacy of the accessible evidence (from PCR including annexure and other data provided was considered unsatisfactory. The extent to which lessons learned (and recommendations) are clear and based on the PCR assessment (evidence & analysis) was considered highly satisfactory. The extentof overall clarity and completeness of the PCR was considered satisfactory.

PCR timeliness was considered late (1) since the PCR report was completed more than 6 months after project completion.

8. SUMMARY OF THE EVALUATION

Criteria PCR PCREN Reason for disagreement/ Comments

RELEVANCE 3.50 3,50 HIGHLY SATISFACTORY

Relevance of project development objective 4 4 Très satisfaisante. Durant l’exécution, l’objectif du projet est resté parfaitement aligné sur les stratégies sectorielles et d’intervention de la Banque, les priorités et les stratégies de développement du pays et les besoins prioritaires des bénéficiaires. Le projet s’inscrit parfaitement dans le cadre dela mission de la FAE, notamment les domaines d’intervention de sa stratégie opérationnelle relative au renforcement de lagouvernance et la gestion des ressources en eau, le renforcement de la base financière et l’amélioration des connaissances du secteur de l’eau. Le Projet s’inscrit bien dans les objectifs du Document de Croissance et de Réduction de la Pauvreté (DSCRP), de la Stratégie de développement urbain et dans les axes prioritaires du programme opérationnel 2008-2010 de la FAE ainsi que dans ceux du Document de la Stratégie Pays(DSP) du Gabon. Il contribuera à l’atteinte des Objectifs de Développement du Millénaire (OMD-2015) et de la Vision Africaine de l’Eau (VAE) pour l’horizon 2025.

Relevance of project design 3 3 Validated satisfactory. The design of the project is straightforward and logical, consisting of 3 main components of carryingout technical studies, capacity strengthening and project management. The log frame follows a clear logic from outputs to outcomes to impact, and is supported by solid assumptions and realistic risks. The institutional arrangement of implementations has assured that all stakeholders are involved in decision making aiming to leverage participation and ownership in the project. Cependant, la conception du projet n’a pas pris en compte l’intégralité des enjeux de tous les sous-secteurs de l’assainissement (eaux usées, eaux pluviales et déchets solides). Cette situation a conduit à la réalisation des étudescomplémentaires portant sur la problématique de la gestion des ordures ménagères et déchets solides d’une durée de 28 mois. En plus, le calendrier initial d’exécution de l’étude s’est avéré trop court.Prévue pour 16 mois, l’étude s’est réalisée sur délai global de 49 mois y compris le délai de 28 mois pour les études complémentaires. Le dispositif de suivi-évaluation préconisé à l’évaluation présente des insuffisances importantes.

EFFECTIVENESS 3 3 Satisfactory Development objective (DO) 3 3 Very clear and tangible progress have been

made at project output and outcome level, which have provided an important prerequisite to accomplish the project development outcome at post

implementations stage. This progress lies in the building of data and knowledge through studies and capacity building of relevant stakeholders.

EFFICIENCY 2.00 2.33 UnsatisfactoryTimeliness 1 1 Highly UnsatisfactoryResource use efficiency 2 3 The ratio of the median percentage physical

implementation of the project outputs and commitment rate is 0.94 which is within the category of 1-0.75. This corresponds with a rating of satisfactory (3). The project has largely delivered the outputs expected and remained within the available budget.

Cost-benefit analysis N/A N/AImplementation progress (IP) 3 3 SatisfactorySUSTAINABILITY 3.00 2.75 Satisfactory Financial sustainability 3 3 SatisfactoryInstitutional sustainability and strengthening of capacities

3 3 Satisfactory

Ownership and sustainability of partnerships 3 3 SatisfactoryEnvironmental and social sustainability 3 2 Unsatisfactory It is not clear how

communities in the watersheds were consulted and how their perspectives were considered in the watersheds development plans. Communication with the communitiesis considered critical for social sustainability.The available documentation has not convincingly demonstrated that environmental values have sufficiently been taken into account. There could have been opportunities for environmental development agencies to play a role in watershed management. Some of the possible unanticipated positive outcomes of the project (urban agriculture, ecosystem services, resource recovery) were not at all considered, but could have added value to ensuring environmental sustainability.

OVERALL PROJECT COMPLETION RATING

2.9 2.9 SATISFACTORY

Bank performance: 3.5 3 SatisfactoryBorrower performance: 3 3 SatisfactoryPerformance of other shareholders: 3 3 SatisfactoryOverall PCR quality: - 3.11 Satisfactory

9. PRIORITY FOR FUTURE EVALUATIVE WORK: PROJECT FOR PERFORMANCE EVALUTION REPORT, IMPACT EVALUTION, COUNTRY/SECTOR REVIEWS OR THEMATIC EVALUATION STUDIES:

- Project is part of a series and suitable for cluster evaluation

- Project is a success story

- High priority for impact evaluation

- Performance evaluation is required to sector/country review

- High priority for thematic or special evaluation studies (Country)

- PPER is required because of incomplete validation rating

Major areas of focus for future evaluation work:

a) Performance evaluation is required for sector/ country review

b) Cluster evaluation (institutional support)

c) Sector evaluation (budgetary support or public finance management reforms)

Follow up action by IDEV: Identify same cluster or sector operations; organize appropriate work or consultation mission to facilitate a), b) and/or c).

Division Manager clearance Director signing off

Data source for validation: Task Manager/ Responsible bank staff interviewed/contacted (in person, by telephone or

email) Documents/ Database reports

Attachment:

PCR evaluation note validation sheet of performance ratings

List of references

Appendice 1

PROJECT COMPLETION REPORT EVALUATION NOTE Validation of PCR performance ratings

PCR rating scale:

Score Description4 Very Good – Fully achieved with no shortcomings3 Good – Mostly achieved despite a few shortcomings2 Fair – Partially achieved. Shortcomings and achievements are roughly balanced1 Poor – very limited achievement with extensive shortcomings

UTS Unable to score/rateNA Non Applicable

Criteria Sub-criteriaPCRworkscore

IDEVreview

Reasons for deviation/comments

RELEVANCE Relevance of the project development objective (DO) duringimplementation

4 4 Validée très satisfaisante. Durant l’exécution, l’objectif du projet est resté parfaitement aligné sur les stratégies sectorielles et d’intervention de la Banque, les priorités et les stratégies de développement du pays et les besoins prioritaires desbénéficiaires. Le projet s’inscrit parfaitement dans le cadre de la mission de la FAE, notamment les domaines d’intervention de sa stratégie opérationnelle relative au renforcement de la gouvernance et la gestion des ressources en eau, le renforcement de la base financière et l’amélioration des connaissances du secteur de l’eau.

Relevance of project design (from approvalto completion)

3 3 Validée satisfaisante. The design of the project is straightforward and logical, consisting of 3 main components of carrying out technical studies, capacity strengthening and project management. The log frame follows a clear logic from outputs to outcomes to impact, and is supported by solid assumptions and realistic risks. The institutional arrangement of implementations has assured that all stakeholders are involved in decision making aiming to leverage participation and ownership in the project. Cependant, la conception du projet n’a pas pris en compte l’intégralité des enjeux de tous les sous-secteurs de l’assainissement (eaux usées, eaux pluviales et déchets solides). Cette situation a conduit à la réalisation des études complémentaires portant sur la problématique de la gestion des ordures ménagères et déchets solides d’une durée de 28 mois. En plus, le calendrier initial d’exécution de l’étude s’est avéré trop court. Le dispositif de suivi-évaluation préconisé à l’évaluation présente des insuffisances importantes.

OVERALL RELEVANCE SCORE 3.50 3.50 TRES SATISFAISANTE


IDEVreview


EFFECTIVENESS*

Effectiveness in delivering outcomes

Outcome 1: Improving the technical, socio-economic and environmental knowledge of the Gué-Gué, Lowe-IAI and Terre Nouvelle basins

Noscore

3Some critical technical aspects were (initially) not addressed in the watershed studies.

Outcome 2: Improvement of stakeholders’ capacityfor the development of the said basins

Noscore

3

The PCR reports that pubic consultations were held with participation of all stakeholders. it is not clear how communities in the watersheds were consulted and how their perspectives and demands were incorporated in the watersheds development plans. Communication with the communities is considered critical for ensuring social sustainability.

Moyenne 3 3 Validée satisfaisante

Effectiveness in delivering output

Component A : Updating of Studies

Output 1: Feasibility studies including ESIA and ESMPs developed

Noscore

3

The carrying out of the feasibility studies on thedevelopment watersheds was done well.

Output 2: Detailed preliminary draft (DPD) updated

Noscore

3The detailed preliminary drafts (DPD) have been effectively prepared.

Output 3: Works bidding documents (BD) prepared

Noscore

3The bidding documents (BDs) have been effectively prepared.

Component B : Capacity-building

Output 1: Officials of CIPA administrative bodies and associations involved in the basins trained

Noscore

3

All the training courses were carried with a few less participants than expected. An additional request for a training for logician in the project was never met, however this was considered outside the project design, so it has not affected performance for this output.

Output 2: Computer software and hardware for studies acquired

Noscore

3

Computer software and hardware for carrying out the studies have been delivered as planned

Output 3: Sanitation database established

Noscore

2 The setting up of a functional database was onlypartly achieved; data is available and associated trainings have been carried out but database


IDEVreview


management was not achieved due to the reported lack of guidelines in the appraisal report.

Component C: Project management

Output 1: Feasibility study, DPD and BD validated.

Noscore

3A donor round table meeting was held and 4 workshops were organized to kick-off the project and to validate important project outputssuch as the feasibility study, DPD and BD documents.

Output 2: Donors mobilized

Noscore

3

Moyenne 3 3 Validée satisfaisante

Development objective rating

3 3

Very clear and tangible progress has been made at project output and outcome level, which haveprovided an important prerequisite to accomplish the project development outcome at post implementations stage. This progress lies inthe building of data and knowledge through studies and capacity building of relevant stakeholders.

Beneficiaries

Directly: Gabonese Government, Ministryof Public Works, DGCE, DGE, Town Councils

100% 3

While the type of beneficiaries are sound, there is a lack of quantification as to how many people have actually benefited and how many people were anticipated to benefit. There is no disaggregation by sex. The progress reporting (100, 50, 0%) seems arbitrary and cannot be verified by the evaluator. The poor reporting on beneficiaries should be considered a missed opportunity when it comes to disseminating project achievements and demonstrating the impact it has made on people.

Donors, NGOs 100% 3

The population affected by the work

50% 1

Indirectly: the entire population in the threeWatersheds

0% 1

NOTE GLOBALE DU DEGRE DE COUVERTURE DES BENEFICIAIRES

n/a 2 Insatisfaisante. La revue tout comme l’évaluateur considèrent la couverture des bénéficiaires du projet estjugée in satisfaisante

Unanticipated outcomes (positive or negative not considered in the project logical framework) and their level of impact on the project (high, moderate, low)Institutional development

Noscore

-Not considered as an additional unanticipated outcome.

GenderNo

score-

Not considered as an additional unanticipated outcome.

Environment & climate change

Noscore

Moderate

Possible green zones or zones designed for flood control may hold potential for other ecosystem services such as increasing biodiversity and supporting carbon absorption.

Poverty reduction High Moderate

The PCR list a single unanticipated additional outcome which is the socio-economic /


IDEVreview


commercial opportunities arising from the construction of roads along the drainage canals. This could indirectly contribute to poverty reduction. The evaluator agrees that this is a positive outcome of the project which was not anticipated in the appraisal reportIn addition, depending on the exact layout of thewatershed development plans there may be opportunities for expansion of urban agriculture boosting incomes and possibly reducing poverty.

Private sector development

Noscore

Moderate

Additional outcomes may exist in relation to business opportunities for solid waste disposal and recycling as well as for fecal sludge treatment.

Regional integrationNo

score-

Not considered as an additional unanticipated outcome.

Other (specify)

Moyenne n/a 3 Validée satisfaisante

EFFECTIVENESS OVERALL SCORE 3 3 Satisfaisante

EFFICIENCYTimeliness (based on the initial closing date)

1 1

Validée très insatisfaisante. Le rapport entre délai d’exécution prévisionnel et délai réel d’exécution à compter de la date de satisfaction des conditions de premier décaissement est de 0,30, et de ce fait jugé très insatisfaisant car inférieur à 0,50.

Resource used efficiency

2 3

The ratio of the median percentage physical implementation of the project outputs and commitment rate is 0.94 which is within the category of 1-0.75. This corresponds with a rating of satisfactory (3). The project has largelydelivered the outputs expected and remained within the available budget.

Cost-benefit analysis N/A N/A

Implementation progress (from the IPR)

3 3Les processus d’exécution sont jugés satisfaisants pour la plupart, et ont permis d’obtenir les résultats escomptés

Other (specify)

OVERALL EFFICIENCY SCORE 2.00 2.33 Unsatisfactory

SUSTAINABILITY

Financial sustainability

3 3 Validée satisfaisante. La viabilité financière du projet est prise en compte dès sa conception. Les PTF ont été impliqués tout le long du projet, en participant aux ateliers de validation des différentes phases. La table ronde organisée à la fin de l’étude en avril 2013 a permis l’engagement de plusieurs bailleurs pour financer les travaux définis par les études.


IDEVreview


Institutional sustainability and strengthening of capacities

3 3 Validée satisfaisante L’UCET, actuellement considérée comme le leader de l’opération, achèvera sa mission après la fin des travaux. La Direction Générale des Infrastructures de Transport (DGIT) se chargera ensuite des tâches d’entretien et de maintenance de l’infrastructure. Le schéma institutionnel est prévu au niveau de la collecte des déchets solides. Les missions des mairies restent limitées, en attendant l’adoption de la loi de décentralisation.

Ownership and sustainability of partnerships

3 3 Validée satisfaisante. Le montage institutionnel du projet est une expérience fortement appréciée par les différents intervenants. Les parties prenantes ont mené le projet en harmonie avec la FAE, les PTF et les prestataires de service. Les différentes parties prenantes se sont approprié le processus de l’étude : choix techniques, outils utilisés, plan de réinstallation, EIES, consultation publique, PGES, etc.

Environmental and social sustainability

3 2 Unsatisfactory. It is not clear how communities in the watersheds were consulted and how their perspectives were considered in the watersheds development plans. Communication with the communities is considered critical for social sustainability.The available documentation has not convincingly demonstrated that environmental values have sufficiently been taken into account.There could have been opportunities for environmental development agencies to play a role in watershed management. Some of the possible unanticipated positive outcomes of the project (urban agriculture, ecosystem services, resource recovery) were not at all considered, but could have added value to ensuring environmental sustainability.

NOTE GLOBALE DE DURABILITE 3 2.75 Satisfaisant

*The rating of the effectiveness component is obtained from the development objective (DO) rating in the latest IPR of the project (see Guidance Note on the IPR). The ratings for outputs and outcomes are determined based on the project’s progress towards realizing its targets, and the overall development objective of the project (DO) is obtained by combining the ratings obtained for outputs and outcomes following the method defined in the IPR Guidance Note. The following method is applied: Highly satisfactory (4), Satisfactory (3), Unsatisfactory (2) and Highly unsatisfactory (1).

Criteria Sub-criteriaPCRWorkscore

IDEVreview


BANK PERFORMANCE

Proactive identification and resolution of problems at different stage of the project cycle

No score

3 Satisfaisante

Use of previous lessons learned from previous operations during design and implementation

No score

4 Très satisfaisante

Promotion of stakeholder participation to strengthen ownership

No score

3 Satisfaisante

Enforcement of safeguard and fiduciary requirements

No score

3 Satisfaisante

Design and implementation of Monitoring & Evaluation system

No score

2 Lack of evidence that a comprehensiveand functional M&E framework was put in place.

Quality of Bank supervision (mix of skills in supervisory teams, etc)

No score

3 Satisfaisante

Timeliness of responses to requests

No score

3 Satisfaisante

OVERALL BANK PERFORMANCE SCORE 3 Satisfaisante

BORROWER PERFORMANCE

Quality of preparation and implementation

No score

3 Satisfaisante

Compliance with covenants, agreements and safeguards

No score

3 Satisfaisante

Provision of timely counterpart funding

No score


Responsiveness to supervision recommendations

No score

3 Satisfaisante

Measures taken to establish basis for project sustainability

No score

3 Satisfaisante

Timeliness of preparing requests

No score

3 Satisfaisante

OVERALL BORROWER PERFORMANCE SCORE

3 Satisfaisante

PERFORMANCEOF OTHER STAKEHOLDERS

Timeliness of disbursements by co-financiers

No score

3 Satisfaisante

Functioning of collaborative agreements

No score


Quality of policy dialogue with co-financiers (for PBOs only)

No score

UTR No information available.

Quality of work by service providers

No score

UTR Reports were not made available to theevaluator.

Responsiveness to client demands

No score

3 Satisfaisante

OVERALL PERFORMANCE OF OTHER STAKEHOLDERS

3 Satisfaisante

The overall rating is given: Very Good, Good, Fair and Poor.

(i) Very Good (HS) : 4(ii) Good (H) : 3(iii) Fair (US) : 2(iv) Poor (HUS): 1

DESIGN, IMPLEMENTAION AND UTILIZATION OF MONITIRING ANDEVALUATION (M&E)

Criteria Sub-criteriaIDEVScore

Comments

M&E DESIGN

M&E system is in place, clear, appropriate and realistic

2

Monitoring and evaluation of beneficiaries was not properly done which is evident from the lack of qualitative and quantitative data. Most of the performance indicators used are considered relatively limited (no. of studies, no. meetings) and do not capture quality aspects of performance.

Monitoring indicators and monitoring plan were duly approved

3

Satisfaisante. Certains indicateurs de performance disponibles dans la matrice du cadre logique du projet. Les études ontpermis de produire des données avec une certitude plus précise sur les caractéristiques des aménagements projetés dans les trois (3) bassins en termes d’indicateurs de performance du projet.

Existence of disaggregated gender indicator

1 Not available and/or used.

Baseline data were available or collected during the design

2Validée insatisfaisante. Quelques données sont disponibles sur la situation de départ de l’étude.

Other, specify

OVERALL M&E DESIGN SCORE 2 Validée insatisfaisante.

M&E IMPLEMENTA-TION

The M&E function is adequately equipped and staffed

2 Validée insatisfaisante.

OVERALL M&E IMPLEMENTATION SCORE


M&E UTILIZATION

The borrower used the tracking information for decision


OVERALL M&E UTILIZATION SCORE 2 Validée insatisfaisante.

OVERALL M&E PERFORMANCE SCORE 2 INSATISFAISANTE

PCR QUALITY EVALUATION

CriteriaPCR-

EVN (1-4)

Comments

QUALITY OF PCR

1. Extent of quality and completeness of the PCR evidence and analysis to substantiate the ratings of the various sections

3 Satisfaisante

2. Extent of objectivity of PCR assessment score

3 Satisfaisante

3. Extent of internal consistency of PCR assessment ratings; inaccuracies;

3 Satisfaisante

inconsistencies; (in various sections; between text and ratings; consistency of overall rating with individual component ratings) 4. Extent of identification and assessment of keyfactors (internal and exogenous) and unintendedeffects (positive or negative) affecting design and implementation

3 Satisfaisante

5. Adequacy of treatment of safeguards, fiduciary issues, and alignment and harmonization


6. Extent of soundness of data generating and analysis process (including rates of returns) in support of PCR assessment

3 Satisfaisante

7. Overall adequacy of the accessible evidence (from PCR including annexure and other data provided)

2 Insatisfaisante

8. Extent to which lessons learned (and recommendations) are clear and based on the PCR assessment (evidence & analysis)


9. Extent of overall clarity and completeness of the PCR

3 Satisfaisante

Other (specify)

PCR QUALITY SCORE 3.11 Satisfaisante

PCR compliance with guidelines (PCR/OM ; IDEV)

1. PCR Timeliness (On time = 4; Late= 1) 1 PCR report was completed more than 6 months after project completion.

2. Extent of participation of borrower, Co-financiers & field offices in PCR preparation

UTS No information available that this was done.

3. Other aspect(s) (specify)

PCR COMPLIANCE SCORE

*** rated as Very Good (4), or Good (3), or Fair (2), or Poor (1)

References

African Development Bank Group, Policy for integrated water resources management, April 2000

African Development Bank Group, Staff Guidance on Project Completion Reporting and Rating,Quality Assurance and Results Department (ORQR), Quality Assurance Division (ORQR.2),August 2012

African Development Bank Group, Gabon joint 2016-2020 country strategy paper and country portfolio performance review report. Published in 2016

African Development Bank Group. Country Strategy Paper 2011-2015. Published in 2011

Project Completion Report for project entitled Updating the Studies of the Gue-Gue, Lowe-IAI and Terre Nouvelle Watersheds

UNECA, AU, AfDB (2000) Africa Water Vision 2025. Equitable and sustainable use of water forsocio-economic development.

Various other internal reports, memos, including trip reports, progress reports, etc