the world bankdocuments.worldbank.org/curated/pt/176581468191348459/pdf/icr3443-p... · bpkp badan...

Document of

The World Bank

Report No: ICR00003443

IMPLEMENTATION COMPLETION AND RESULTS REPORT

( TF-98819 AND TF-98862)

ON

GRANTS

IN THE AMOUNT OF US$ 14.1 MILLION

TO THE

REPUBLIC OF INDONESIA

FOR THE

THIRD NATIONAL PROGRAM FOR COMMUNITY EMPOWERMENT IN RURAL AREAS - DISASTER MANAGEMENT SUPPORT PROJECT

May 1, 2015

Global Practice for Social, Urban, Rural and Resilience Indonesia Country Management Unit East Asia and Pacific Region

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

CURRENCY EQUIVALENTS (Exchange Rate Effective November 30, 2014

Currency Unit = Rupiah IDR 1.00 = US$ 0.0008 US$ 1.00 = IDR 12,204

FISCAL YEAR

January 1 – December 31

ABBREVIATIONS AND ACRONYMS

BPD Badan Permusyawaratan Desa (Village Council) BPKP Badan Pengawasan Keuangan dan Pembangunan (Finance and Development

Supervisory Agency) BPS Badan Pusat Statistik (Central Bureau of Statistics) BPMD Badan Pemberdayaan Masyarakat Desa (Village Community Empowerment

Agency) CDD Community-Driven Development CDRM Community-Based Disaster Risk Management CPS Country Partnership Strategy DIY Daerah Istimewah Yogyakarta (Yogyakarta Special Region) EIRR Economic Internal Rate of Return FM Financial Management GoI Government of Indonesia IBRD International Bank for Reconstruction and Development ICR Implementation Completion and Results IDR Indonesian Rupiah IRI Intermediate Results Indicator ISR Implementation Status and Results Report JMC Joint Management Committee Kabupaten District KDP Kecamatan Development Project Kecamatan Sub-district M&E Monitoring and Evaluation MoHA Ministry of Home Affairs MIS Management Information System NGO Non-Government Organization NMC National Management Consultant O&M Operations and Maintenance PAD Project Appraisal Document PDO Project Development Objective PMD Pemberdayaan Masyarakat Desa (Village Community Empowerment) PMU Project Management Unit PNPM Program Nasional Pemberdayaan Masyarakat (National Community

Empowerment Program) PSF PNPM Support Facility PTO Petunjuk Teknis Operasional (Technical Operational Guidelines)

ii

RMC Regional Management Consultant Satker Unit Satuan Kerja (local government coordination team) UPK Sub-District Activity Management Unit

Regional Vice President: Axel van Trotsenburg Country Director: Rodrigo A. Chaves Senior Global Practice Director

Ede Jorge Ijjasz-Vasquez

Acting Practice Manager: Kevin Tomlinson Project Team Leader: Sonya Woo ICR Team Leader: Sonya Woo

iii

INDONESIA THIRD NATIONAL PROGRAM FOR COMMUNITY EMPOWERMENT

IN RURAL AREAS – DISASTER MANAGEMENT SUPPORT

CONTENTS

Data Sheet A. Basic Information ........................................................................................................... v B. Key Dates ....................................................................................................................... v C. Ratings Summary ........................................................................................................... v D. Sector and Theme Codes ............................................................................................... vi E. Bank Staff ...................................................................................................................... vi F. Results Framework Analysis ........................................................................................ vii G. Ratings of Project Performance in ISRs ....................................................................... ix H. Restructuring (if any) ..................................................................................................... x I. Disbursement Profile ...................................................................................................... x 1. Project Context, Development Objectives and Design ................................................. 1 2. Key Factors Affecting Implementation and Outcomes ................................................ 4 3. Assessment of Outcomes ............................................................................................ 13 4. Assessment of Risk to Development Outcome ........................................................... 19 5. Assessment of Bank and Borrower Performance ....................................................... 20 6. Lessons Learned.......................................................................................................... 23 7. Comments on Issues Raised by Borrower/Implementing Agencies/Donors .............. 24 Annex 1. Project Costs and Financing .............................................................................. 25 Annex 2. Outputs by Component...................................................................................... 25 Annex 3. Economic, Financial and Technical Analysis ................................................... 33 Annex 4. Loan Preparation and Implementation Support/Supervision Processes ............ 36 Annex 5. Beneficiary Survey ........................................................................................... 38 Annex 6. Stakeholder Workshop Report and Results ....................................................... 39 Annex 7. Borrower's Completion Report ......................................................................... 41 Annex 8. Comments of Co-financiers .............................................................................. 43 Annex 9. List of Supporting Documents .......................................................................... 44 Annex 10. Photographs ..................................................................................................... 45 MAP IBRD33420

iv

A. Basic Information

Country: Indonesia Project Name:

Third National Program for Community Empowerment in Rural Areas - Disaster Management Support

Project ID: P125600 L/C/TF Number(s): TF-98819,TF-98862 ICR Date: 05/26/2015 ICR Type: Core ICR

Lending Instrument: SIL Grantee: REPUBLIC OF INDONESIA

Original Total Commitment:

USD 14.10M Disbursed Amount: USD 13.88M

Revised Amount: USD 13.90M Environmental Category: B Implementing Agencies: Directorate General of Community and Village Empowerment, Ministry of Home Affairs Cofinanciers and Other External Partners: US Agency for International Development (USAID) B. Key Dates

Process Date Process Original Date Revised / Actual Date(s)

Concept Review: 12/21/2010 Effectiveness: 03/23/2011

Appraisal: 01/29/2010 Restructuring(s): 12/17/2012 12/05/2013

Approval: 01/19/2011 Mid-term Review: Closing: 12/31/2012 11/30/2014 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Moderately Satisfactory Risk to Development Outcome: Moderate Bank Performance: Moderately Satisfactory Grantee Performance: Moderately Satisfactory

C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings

Quality at Entry: Moderately Satisfactory Government: Moderately Satisfactory

Quality of Supervision: Moderately Satisfactory Implementing Agency/Agencies: Moderately Satisfactory

v

Overall Bank Performance: Moderately Satisfactory Overall Borrower

Performance: Moderately Satisfactory

C.3 Quality at Entry and Implementation Performance Indicators

Implementation Performance Indicators QAG Assessments

(if any) Rating

Potential Problem Project at any time (Yes/No):

Yes Quality at Entry (QEA):

None

Problem Project at any time (Yes/No):

No Quality of Supervision (QSA):

None

DO rating before Closing/Inactive status:

Moderately Satisfactory

D. Sector and Theme Codes

Original Actual Sector Code (as % of total Bank financing) General agriculture, fishing and forestry sector 3 3 Irrigation and drainage 56 56 Other social services 3 3 Rural and Inter-Urban Roads and Highways 32 32 Water supply 6 6

Theme Code (as % of total Bank financing) Natural disaster management 53 53 Other social development 1 1 Participation and civic engagement 15 15 Rural markets 5 5 Rural services and infrastructure 26 26 E. Bank Staff

Positions At ICR At Approval Vice President: Axel van Trotsenburg James W. Adams Country Director: Rodrigo A. Chaves Stefan G. Koeberle Practice Manager/Manager:

Kevin A Tomlinson Jan Weetjens

Project Team Leader: Sonya Woo Kevin A Tomlinson ICR Team Leader: Sonya Woo ICR Primary Author: Jacqueline L. Pomeroy

vi

F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) For villagers in PNPM-Rural locations to benefit from improved socio-economic and local governance conditions. Revised Project Development Objectives (as approved by original approving authority) For villagers in PNPM-Rural locations to benefit from improved socio-economic and local governance conditions in disaster affected areas. (a) PDO Indicator(s)

Indicator Baseline Value

Original Target Values (from

approval documents)

Formally Revised Target Values

Actual Value Achieved at

Completion or Target Years

Indicator 1 : More than 50% of villages replace infrastructure lost to disaster Value quantitative or Qualitative)

50% 50% 96%

Date achieved 11/30/2012 11/30/2012 11/30/2014 Comments (incl. % achievement)

Indicator 2 : More than 80% satisfaction levels from beneficiaries regarding improved services and local level governance

Value quantitative or Qualitative)

80% more than 80% more than 80%

72% or 83% with adjusted weightings


Indicator 3 : Economic internal rates of return (EIRRs), more than 30% for major rural infrastructure types

Value quantitative or Qualitative)

39% more than 30% more than 30% 40%

Date achieved 12/30/2010 11/30/2012 11/30/2012 11/30/2014 Comments (incl. % achievement)

Indicator 4 : Project beneficiaries Value quantitative or Qualitative)

0 more than 70% 74%

Date achieved 11/30/2012 11/30/2012 11/30/2014 Comments

vii

(incl. % achievement) Indicator 5 : of which female (beneficiaries) Value quantitative or Qualitative)

0 50% 50.42%


(b) Intermediate Outcome Indicator(s)

Indicator Baseline Value

Original Target Values (from

approval documents)

Formally Revised

Target Values

Actual Value Achieved at

Completion or Target Years

Indicator 1 : Minimal 50% participation rate of women and poorest community members in planning and decision-making meetings

Value (quantitative or Qualitative)

50% 42.4% women and 28.5% poorest

Date achieved 11/30/2012 11/30/2014 Comments (incl. % achievement)

Indicator 2 : Number of type of infrastructure works, economic, and education and health subprojects or activities completed in selected disaster areas by 2012


98%

Date achieved 11/30/2014 Comments (incl. % achievement)

Indicator 3 : More than 50% of infrastructure works are evaluated as of high quality Value (quantitative or Qualitative)

more than 50% 86-91%


Indicator 4 : O&M arrangement are in place and functioning for more than 50% of infrastructure works


more than 50% 86-87%

viii


Indicator 5 : More than 70% of agreed additional consultants are recruited and trained Value (quantitative or Qualitative)

more than 70% 100%


Indicator 6 : 90% of disaster-affected sub-districts get additional facilitator and training for disaster assessment


90% 100%


Indicator 7 : Audit carried out in more than 25% of disaster affected sub-district Value (quantitative or Qualitative)

more than 25% 39%


G. Ratings of Project Performance in ISRs

No. Date ISR Archived DO IP

Actual Disbursements (USD millions)

3 04/13/2012 Moderately Satisfactory Moderately Satisfactory 6.04 4 02/10/2013 Moderately Satisfactory Moderately Satisfactory 11.55 5 11/02/2013 Moderately Satisfactory Moderately Satisfactory 13.00 6 06/11/2014 Moderately Satisfactory Moderately Satisfactory 13.00 7 01/05/2015 Moderately Satisfactory Moderately Satisfactory 13.54

ix

H. Restructuring (if any)

Restructuring Date(s)

Board Approved

PDO Change

ISR Ratings at Restructuring

Amount Disbursed at

Restructuring in USD millions

Reason for Restructuring & Key Changes Made DO IP

12/17/2012 N MS MS 11.55 Extend the closing date to 31 December 2013

12/05/2013 MS MS 13.00 Extend the closing date to 30 November 2014

I. Disbursement Profile

x

1. Project Context, Development Objectives and Design

1.1 Context at Appraisal In October 2010 Indonesia was struck by two major natural disasters: an earthquake which triggered a tsunami hit Mentawai (West Sumatra); and Mt. Merapi volcano erupted (Central Java and Yogyakarta). Nearly a thousand people died in these disasters and more than 350,000 were displaced in Java alone. Nearly three thousand homes were heavily damaged or destroyed in Java, along with roughly 20% of all housing in four sub-districts in Mentawai. Already-poor fishing and farming families suffered catastrophic losses to their income-generating ability. (See Annex 10 for photographs of both Java and Mentawai.) The Government of Indonesia (GoI) requested emergency assistance to use the ongoing Program Nasional Pemberdayaan Masyarakat (National Community Empowerment Program or “PNPM”) platform and aligned community driven development (CDD) mechanisms as a key pillar of its recovery strategy for these three areas, Mentawai (West Sumatra), Central Java and the Yogyakarta Special Region (D.I. Yogyakarta or DIY). The Disaster Management Support Project (“project”) was subsequently proposed as an additional grant financing operation linked to PNPM Rural III (Ln. 7867-ID). The grant would finance the costs associated with GoI disaster management and recovery efforts at the community level. PNPM provided the platform and implementation mechanism to: (a) access financing through the PNPM Support Facility (PSF), a multi-donor trust fund; and (b) transfer these resources directly to communities through block grants to finance disaster recovery activities. Allocating the proposed US$14.1 million through the PSF was intended to promote economies of scale and continuity with on-going disaster management support activities by using the facilitation and other project support structures already in place. Rationale for Bank Involvement. The CDD approach of PNPM was a key component of the Bank’s Country Partnership Strategy (CPS) for 2009-2012 1 which emphasized engagements with government counterparts and other stakeholders who were committed to addressing critical governance and institutional challenges. The Bank had extensive experience with CDD-based disaster recovery and had delivered programs following the 2004 earthquakes and tsunami in Aceh and Nias, and after the 2006 earthquake that devastated Yogyakarta. See Section 2.1, footnote 6. The additional financing supported three of the five thematic areas of the CPS: (a) Infrastructure; (b) Community Development and Social Protection; and (c) Environmental Sustainability and Disaster Mitigation. The project was also consistent with Indonesia’s Disaster Management Law (Law No. 24 of 2007) which included community-based disaster risk management (CDRM). CDRM focused on ways to encourage and invite more active participation from local government

1 “Investing in Indonesia’s Institutions” (Report No. 44845-IND) dated July 22, 2008 and approved by the Board on September 11, 2008.

and community members in the planning, implementation, and monitoring of disaster management programs. The proposed additional financing was intended to make a direct contribution to the CDRM component of the National Disaster Management Framework.

1.2 Original Project Development Objective (PDO) and Key Performance Indicators The PDO as stated in the PAD was for villagers in PNPM-Rural locations to benefit from improved socio-economic and local governance conditions in disaster-affected areas. The PDO in the Grant Agreement was consistent with the PAD version. Key performance indicators were based on PNPM-Rural III indicators and modified to meet the project requirements. This included adding an outcome indicator specific to the disaster-affected areas, revising four PNPM-Rural III intermediary results indicators, and dropping two which were inappropriate. (See the datasheet for full set of indicators.) Specifically, PDO indicators were:

• >50% of villages replace infrastructure lost to disaster; • >80% satisfaction levels from beneficiaries regarding improved services and local

level governance; • EIRRs >30% for major rural infrastructure types; and • > 70% Project beneficiaries (and percentage female beneficiaries)2.

1.3 Revised PDO and Key Indicators The PDO and key indicators remained unchanged during project implementation.

1.4 Main Beneficiaries The intended primary beneficiaries of the project consisted of the inhabitants of 18 disaster-affected sub-districts located in three provinces.

Population West Sumatra (Mentawai District, four sub-districts) 31,985 Central Java (3 districts, 13 sub-districts) 612,502 DI Yogyakarta (Selman District, one sub-district) 27,657 Total Population 672,144

It was intended that the population of these kecamatan (“sub-districts”) would benefit both directly and indirectly from the repair and reconstruction of community infrastructure and services, grant assistance for those affected by the disasters, especially the poorest and most vulnerable groups, supporting the return of families to their villages and basic clean up (cash-for-work programs), and support for displaced populations (preparation of resettlement sites), training, facilitation and implementation of disaster risk reduction

2 While this indicator was not included in the Results Framework, it was featured in the Monitoring Table. In accordance with the Bank’s guidance on core sector indicators, all investment projects which had an approval date of July 1, 2009 or later were required to identify and measure the number of project beneficiaries.

strategies (improved communications, signage and infrastructure for evacuation, disaster response strategies), and support for economic recovery strategies (infrastructure and revolving credit)–all planned and implemented through a CDD process. Secondary beneficiaries were: (i) the Directorate General for Community and Village Empowerment (PMD) in the Ministry of Home Affairs (MOHA) (the executing agency), which would benefit from increased capacity in post-disaster management; (ii) village level cadres and partner agencies providing implementation support for disaster-related activities; (iii) district governments in which the 18 target sub-districts were located, who would benefit from training and institutional strengthening aimed at improving their planning and disaster response capabilities, including supporting communications, and coordination with the village level underpinned by transparent and accountable governance processes; and (iv) village representative councils (Badan Permusyawaratan Desa or BPD) in disaster-affected areas, which would benefit from technical assistance to establish and strengthen their role in channeling CDD funds to appropriate disaster response and recovery-related community activities and investments.

1.5 Original Components Component I: Disaster Management Kecamatan Grants (US$12.9 million). The additional financing provided grants to beneficiaries for: (i) planning for community development, including the community’s assessment of damages and the preparation of sub-grant proposals, (ii) training and capacity building, including development planning and investment, (iii) investing in social and economic infrastructure identified through community development planning, including the initial cash-for-work recovery activities3 followed by infrastructure planning, (iv) investing in activities identified through community development planning using Revolving Loan Funds (RLF), and (v) preparing for and responding to disasters, emergency or catastrophic events, as needed. The intent was for community-driven investments in infrastructure and economic recovery activities to lead to improvements in socio-economic conditions, and community collaboration with local government in planning and implementing both the projects and the disaster risk management strategies with transparent and accountable mechanisms for all participants contributed to improved local governance conditions. Component II: Facilitation Support (US$1.0 million). The additional financing supported technical advisory services, general and disaster-specific training and other material support, through facilitators, to strengthen the capacity of Kabupaten and Kecamatan government institutions and communities in development planning and investment. The PAD noted that facilitation is instrumental for sound community planning and budgeting, which affects the number and quality of the infrastructure investments and hence their socio-economic impact. In addition, the PAD identified facilitation as the communications conduit between local government and the communities, and serves to reinforce the transparent and accountable governance processes.

3 Cash-for-work programs were implemented in all locations, but there were enough unspent block grant resources in Mentawai to fund these directly. Grant funds were used to support these activities in Central Java and DI Yogyakarta.

Component III: Implementation Support and Technical Assistance (US$0.2 million). This component would provide technical advisory services, training and other material support for PNPM implementation at national-, Province-, District-, and Sub-district-levels, including training of facilitators, monitoring and evaluation, and enhanced technical and financial audits. These measures were identified as have substantial impact on improved local governance, both directly through the technical and financial audits, plus monitoring and evaluation, and indirectly through training and capacity building.

1.6 Revised Components The components remained unchanged during project implementation.

1.7 Other significant changes Two Level II restructurings were undertaken to extend the project closing dates, primarily to meet the needs for additional assistance to complete activities in the remote area of Mentawai. The unspent funds remaining in the trust fund at the end of the project were reallocated to fund similar post-disaster activities for communities in North Sumatra and East Java, as anticipated in the PAD. Date Change December 17, 2012 Level II restructuring: closing date for TF98819 (US$13.9

million) extended from December 31, 2012 to December 31, 2013. TF98862 (US$.2 million) was allowed to close on December 31, 2012 without disbursement. See section 2.2 for details.

November 26, 2013 Level II restructuring: closing date extended from December 31, 2013 to November 30, 2014

August 2014 Bank provided no objection to PMD request to reallocate the anticipated undisbursed Grant funds to reimburse eligible expenditures already incurred in disaster-affected areas of North Sumatra and East Java. The final amount reallocated to these areas was US$353,000.

2. Key Factors Affecting Implementation and Outcomes

2.1 Project Preparation, Design and Quality at Entry Soundness of project preparation (background analysis and lessons learned from previous projects). The design of the DMS project was built on the existing PNPM Rural III program (IBRD-78670, TF-99616), but with several modifications to reflect the needs and risks inherent in a post-disaster environment. As described below, these modifications/elements guided the organization and implementation of the project. Key attributes of PNPM that were in place and lent themselves well to disaster recovery included:

• A well-established network of community facilitators, based and operating in every sub-district of the country, able to carry out damage and loss assessments and to facilitate communities in the identification, planning and implementation of disaster recovery activities;

• Existing social capital and institutional mechanisms developed through PNPM and its predecessors the Kecamatan Development Program (KDP) 4 , allowing communities to efficiently prioritize, plan, implement and manage disaster recovery activities at the community level;

• Established financial architecture and efficient management practices that enable the transfer of resources directly to the sub-districts with robust fiduciary controls; and

• A comprehensive set of PNPM operational procedures to expedite and support disaster recovery, which were used with success in Aceh, Nias, Papua, Nusa Tenggara Timur (NTT), West Sumatra, and West Java.

The project was initially supported by two trust funds from different sources of funding: TF098819 (US$13.9 million) for Components I & II, and TF098862 (US$0.2 million) for Component III. Assessment of project design. The foundation design of the PNPM Rural III5 program was verified through previous studies and its design incorporated lessons learned from previous PNPM Rural (KDP) I and II projects (see footnote 4), as well as extensive experience utilizing the CDD model to rebuild and support recovery in Aceh, Nias, and other post-disaster locations6. Key aspects of success from these earlier interventions were incorporated into the design of the DMS project and included: (i) the value of using the same transparent processes communities already understood and had successfully managed; (ii) initial short term cash-for-work programs to facilitate residents’ re-entry to the affected areas; (iii) the need to strengthen the local-level implementation structure by providing additional facilitators to handle the increased work load (in many cases, communities quadrupled their block grant funds and workload), together with post-disaster training; (iv) modest relaxation of the quality requirements while increasing audits to reflect the higher fiduciary risks in disaster-affected areas. These modifications incorporated into the DMS project design provided some appropriate flexibility in anticipation of a more fluid and complex immediate post-disaster environment and the higher risks involved in implementing an ambitious medium-term work agenda.

4 Kecamatan Development Fund (SCL-4330; IDA-3430); The Second Kecamatan Development Project (Loan 4627/Credit 3525). 5 The Third National Program for Community Empowerment in Rural Areas (IBRD-78670, TF-99616). 6 For lessons learned, see the Project Appraisal Documents (PAD) for the Second Kecamatan Development Project (IBRD-46270 IDA-35350), and the Third National Program for Community Empowerment in Rural Areas (IBRD-78670, TF99616), and numerous pieces of analysis emerging from the reconstruction experience in Aceh and Java. These form the basis of the extensive MDF-JRF Working Paper Series: Lessons Learned from Post-Disaster Reconstruction in Indonesia, especially Working Paper #1: Adapting Community Driven Approaches for Post-Disaster Recovery: Experiences from Indonesia.

The main criterion for selection of sub-districts to participate in the DMS project was physical and economic damage resulting from the October 2010 earthquake and volcanic eruption. Two sub-districts were added by the GoI to the five target locations proposed by the local government in Magelang District, Central Java, and while they had extensive economic impacts from the eruption, they suffered no physical damage. Target areas with no physical damage could not contribute to KPIs focused on rebuilding damaged infrastructure, while the KPIs did not capture needed economic recovery. The original planned two year implementation period was ambitious, but acceptable for implementation in Java. The project disbursed 98% of the funds allocated to Central Java and DI Yogyakarta during 2011-2012, and all activities were completed there during the fiscal year roll over period (by end-March) of 2013. However, for Mentawai, this timeframe ended up being unfeasible because of the extremely remote location coupled with weak local management. Forty percent of the funds were disbursed during 2012, 60% followed in 2013, and project activities were largely (but not fully) completed by the November 30, 2014 closing date. The PDO and intermediate level indicators were aligned with underlying PNPM Rural III indicators while focusing on the achievements of the supplementary DMS project. An Economic Internal Rate of Return (EIRR) analysis was undertaken to support the achievement of the PDOs, in addition to a beneficiary survey to assess community satisfaction with the outcomes. The intermediate indicators for high quality and sub-project completion rates were relaxed somewhat to anticipate the pressures of a post-disaster environment, while others were indicators for facilitation resources and audit support were increased to ensure proper implementation and governance measures in a risky environment. The adjustments to the intermediate indicators were appropriate for the circumstances. The KPIs for the PDO were adequate, but would have been stronger with the addition of an indicator that captured economic recovery. Government commitment, stakeholder involvement, and participatory processes. GoI played a central role in helping to prepare the project by sending a team to the affected districts to assist district governments to identify the hardest-hit areas for participation. The national team took an active role in launching and monitoring implementation. Despite weaknesses in local government capacity in Mentawai, district and sub-district governments actively participated to ensure coordination in the planning process and links to BPMD and other disaster-related activities. Utilizing existing platforms and familiar mechanisms made it possible for communities to complete the larger number of projects enabled by the substantially larger block grants than they had previously managed under the regular block grant. Risks and risk mitigation measures. The overall project risk was rated “Medium-I” (low likelihood-high impact), which may have been somewhat under-estimated owing to the absence of several key factors. Much of the identified risk in the ongoing PNPM Rural III project remained relevant, with some additional risks specific to the post-disaster environment. Key risks included in the Risk Assessment Framework included: (i) marginalization of some villagers and heavy-handedness by village elites and local

government over use of the block grants owing to the need for speedy implementation in post-disaster situations; (ii) potential for duplication, overlap and poor coordination among government and NGO programs to support reconstruction; (iii) institutional and managerial capacity of PMD, which was already overseeing the national scale up of the PNPM Rural III program; (iv) potential delays in disbursements, which was the case in Mentawai; and (v) the need for additional monitoring and evaluation support and oversight. Two additional risks should have been included in the risk assessment and mitigation plan: (i) Risks stemming from inadequate facilitation. Community participation,

identification of projects, planning, budgeting, implementation and project quality would all have benefited from two additional experienced facilitators (one social and one technical) in each sub-district, instead of the one newly-recruited assistant facilitator and local cadre they received to handle the increased workload.

(ii) Risks in working in very remote and isolated areas. See section 2.2 below for more detailed discussion.

The project’s Quality at Entry is rated Moderately Satisfactory. While weaknesses were noted in the section above, the inclusion of the two key factors in the risk matrix noted above – together with actions to mitigate them – would have effectively anticipated the additional challenges with the project.

2.2 Implementation The project was affected by factors that supported and challenged implementation efforts, and which were both within and outside government control. Factors outside the control of the government or the implementation agency: Supporting: The disasters in the targeted areas were relatively short-lived The earthquake and tsunami in Mentawai were one-time events, and the intensive danger period for the eruption affecting Central Java and Yogyakarta lasted from October–December, 2010 and residents began returning to their villages in early 2011.7 This meant that communities could begin damage assessments and recovery efforts relatively quickly. Challenging: Mentawai’s remote location and the isolation of target sub-districts A number of factors contributed to lengthy delays in project implementation and were the primary reasons for the extensions of the closing dates noted in Section 1.7. The population was scattered across four islands. There was a high turnover in facilitators as many did not want to live or stay for extended periods in Mentawai. This resulted in a lack of continuity in facilitation and added to delays. Facilitators were already struggling to complete their regular PNPM workload, and the additional DMS funds spread facilitation resources even more thinly. Basic construction materials were not available locally in sufficient quantities and were difficult to procure. Transportation was problematic: only one boat per week from the provincial capital, Padang, infrequent and irregular transport between the four islands,

7 In comparison, Mt. Sinabung in North Sumatra started large-scale eruptions in September 2013 and again in February 2014, and residents were displaced for seven months or longer.

and limited, poor quality and expensive land transportation. Telecommunications capacity was limited and irregular. Factors subject to the control of the government or the implementation agency: The key activity planned for Component III was the hiring of disaster recovery specialists (special technical advisory services) to work with the communities in each of the disaster-affected areas. However, in 2012 the GoI established permanent disaster coordination bodies at the provincial and district levels as part of the implementation of the Law on Disaster Management (Law No. 24 of 2007), which changed the local institutional and operational environment for the DMS project. In response, MoHA determined that existing facilitators with some assistance could effectively implement the project activities in coordination with the permanent local bodies. This resulted in two further implementation adjustments to Components II and III: (i) PMD hired one post-disaster specialist to work with the NMC to provide technical support including training and advice to subnational facilitators in the affected areas and to liaise with the Badan Pemberdayaan Masyarakat Desa (Village Community Empowerment Agencies or BPMDs), and (ii) the Provincial Regional Management Consultants (RMCs) hired one additional assistant facilitator and one additional local cadre for each sub-district team. Given the changes in implementation, the GoI and WB agreed to close the Trust Fund supporting Component III without disbursement as part of the December 2012 restructuring. These responses were generally appropriate, although implementation would have been better supported by ensuring additional facilitators hired in post-disaster areas were experienced, as detailed in Section 6. Supporting: Strong institutional commitment The project had strong political commitment at the central level of government, which translated into appropriate adaptation in implementation. This is reflected in strengthened oversight and visits from the joint task force established by PMD in response to the gaps in local management and implementation delays experienced in Mentawai. PMD’s commitment to assist recovery in these areas is also seen in the flexible development of initial cash-for-work programs in Central Java and Yogyakarta to facilitate the quick return of residents to their villages. In addition, to address the extensive delays in Mentawai, beginning in mid-2013, PMD intensified supervision and support from NMC and the Provincial Consultants, established a special task force focused solely on project completion in Mentawai, and performance in that location improved from 8% project completion to 84% of projects completed by November 30, 2014. Supporting: Effective anti-corruption strategy Individuals and independent organizations in beneficiary communities did use channels of communication established by the project (such as the complaints handling unit or individual facilitators) to report four incidences of corruption as detailed in Section 2.4 (Fiduciary and Governance). Challenging: Centralized program management

The implementing agency’s capacity to take corrective and timely actions was hampered by its centralized management structure – issues had to be passed up the administrative chain for resolution. Coupled with weak local government and consultant team overseeing implementation in remote Mentawai, communications both upwards (to PMD) or downwards (to the sub-district) were inadequate and delays began building early on: at end-2012 (the original closing date), 40% of funds were disbursed in Mentawai and no projects completed (accounting for 10% of total sub-projects and 19% of the total block grants), compared to 98% disbursement and nearly complete sub-projects in Central Java and D.I. Yogyakarta (90% of total sub-projects and 81% of the block grants). Highly centralized management also underlined the challenges stemming from the “flea versus elephant” problem especially during the early period of implementation: the DMS project (US$14.1 million in 18 sub-districts) and PNPM Rural III/IV (US$2,361.7 million combined GOI and WB resources in 4,791 sub-districts over a roughly similar period) competed for scarce management attention, making it difficult for PMD/NMC to focus on remedies for human resources and other gaps in the disaster-affected areas (a risk identified at the time of appraisal). However, this effect diminished over time as the PNPM scale-up became more fully established. Challenging: Weak local government commitment and capacity in Mentawai Mentawai was established as a district in 1999 and already-weak government capacity was strained by the earthquake and tsunami. Support from the district government’s coordination team (Satker) was limited, and they were unable to carry out coordination tasks with individuals and offices at the sub-district level. Midterm Review. There was no midterm review because the project was viewed as a short term response to the natural disasters and expected to be implemented in approximately 20 months. Indeed, the Java portion of the project (81% of the Grant) was completed within the original 20 month timeframe. Only the Mentawai portion required extra time due to the weak implementation capacity, geographical remoteness, and unavailability of construction materials as cited above. Effectiveness of risk mitigation. With the exception of the two residual risks identified in Section 2.1, mitigation of the risks identified in the PAD was generally satisfactory. Improvements to PNPM Rural’s fiduciary and governance systems were largely effective in minimizing corruption, and communities expressed strong confidence in the system’s transparency. PMD’s capacity was stretched, especially in Mentawai, but the fiduciary systems and procedures were generally satisfactory. The Environmental Management Framework, the Land Acquisition and Resettlement Policy Framework, and the Isolated and Vulnerable Peoples’ Planning Framework were integrated into the technical operations manual (PTO) for successful safeguards implementation.

2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization M&E design. The project measured progress against key performance indicators using PNPM Rural’s standard management information system (MIS), augmented by several quantitative surveys and the findings of regular implementation support missions. At

appraisal, the MIS in place to monitor and evaluate PNPM Rural III was used to monitor and evaluate the implementation of the DMS Project. Additional outcome indicators were added to reflect the post-disaster nature of sub-projects and augmented facilitation linked to the additional financing – these were feasible, easily monitored, and illustrated how the project implementation was strengthened to deal with the post-disaster conditions. Additional information was to be gathered through regular field visits by the facilitators, RMC, NMC, PMD and PSF. The PAD also identified technical evaluations to be carried out at the end of the project to measure beneficiary satisfaction and an EIRR. There were some weaknesses in the M&E design. Specifically, one (of the four) units of measurement and the targets value did not match the indicator: “Project beneficiaries” was mismatched to “% of disaster-affected villages assisted;” and one (out of six) of the IRIs relating to the number of subprojects completed had no target value against which to measure progress or success. However, overall, both cases did not have detrimental effects on the project. The indicators were clarified through supervision and it was agreed that the baseline value at the project’s outset would be established through the total number of subprojects proposed. M&E implementation. In practice, routine project monitoring suffered from two types of information collection problems. Firstly, the project encountered problems with the MIS when the project website (http://www.pnpm-perdesaan.co.id) that updated data on a monthly basis was unable to handle the PNPM Rural scale-up and become inoperative in December 2011, just before community planning began for the 2012 block grants. Good communications infrastructure meant that DMS project reporting from Central Java and Yogyakarta was maintained through the completion of the sub-projects in December 2012, supplemented by field visits. Secondly, reporting from Mentawai was delayed and incomplete due to the lack of telecommunications at the village level, irregular telecommunications access at the sub-district and district level, high turnover and frequent absence of facilitators, and weak local program management, which affected overall project implementation in this remote location, not just M&E. Once these and other issues were identified, PMD using its PNPM Rural and disaster experience and reporting, revised its support and supervision strategy in Mentawai. Additional GoI technical missions, followed by monthly visits by the Jakarta-based disaster response specialist and Provincial consultants, and intensive GoI-Bank support missions were critical for improving project performance, especially during the last 18 months. The data collected from these missions to Mentawai were essential to: (i) identify and remedy facilitation problems and to incorporate better controls as the project progressed, (ii) improve accountability for sub-grants, (iii) escalate issues to PMD management, as appropriate, and (iv) document progress against the indicators. In addition, the EIRR analysis and beneficiary survey defined in Annex 1 of the PAD were completed, and summary details of the results are contained in this report in Annex 3 and Annex 5, respectively. M&E utilization. As explained above, the utilization of the M&E data was compromised by the fact that the MIS was unable to generate timely information throughout implementation. To compensate for this shortcoming, both GoI and the Bank increased their reliance on the findings of supervision and technical support missions (funded from

Bank Budget and trust funds supporting PNPM supervision activities) to monitor, reconcile, verify and evaluate project implementation.

The overall M&E quality rating is rated as Modest.

2.4 Safeguard Compliance Overall, safeguards compliance is rated Moderately Satisfactory. The project acquired the same Category B classification as PNPM Rural III. and triggered the same safeguard policies related to environmental assessment (OP/BP 4.01), natural habitats (OPBP 4.04), indigenous peoples (OP/BP4.10), and involuntary resettlement (OP/BP 4.12). The Implementation Guidelines for Social and Environmental Safeguards (IGSES) were satisfactorily prepared and disclosed in 2013, and were disclosed again in March 2014 as part of the revised PTO. Environmental Safeguards: Implementation of Environmental Assessment (OP/BP4.01) is rated moderately satisfactory. Appropriate environmental safeguards were included in the operations manual, but with a cap of $35,000 and average sub-project size of $25,000, it was not possible to reach the threshold to trigger a formal assessment required under the Bank’s OP/BP 4.01. However, while there were no significant negative impacts noted in all project locations, there was weak awareness of general environmental safeguards in Mentawai, mainly as a result of weak facilitation noted elsewhere in this report. Supervision missions noted that communities used dead coral rubble for building material in a remote area where there was no access to alternatives. The rubble consisted of already-dead coral washed up on the shore, and the teams found there was no significant environmental impact from this traditional practice, especially considering the small number and size of the projects. While damage to communities from the earthquake and tsunami required a steadfast response, all sub-projects were implemented in accordance with the plan for the construction of permanent residences (Hunian Tetap) implemented by the Government. Implementation of Natural Habitats (OP/BP 4.04) is rated as moderately satisfactory for the same reason of weak implementation stemming from weak facilitation. A mission identified sub-projects in a new settlement away from coastal area on land formally provided by the Ministry of Forestry with appropriate documentation, but still classified as protected forest. This was eventually resolved by the Ministry of Forestry. There were no issues in Central Java or Yogyakarta, accounting for 90% of the sub-projects and 81% of block grants. Social Safeguards: Implementation of Involuntary Resettlements Policy (OP/BP 4.12 is rated moderately unsatisfactory. The PNPM operational manual includes a planning and verification framework to ensure that any land acquisition is well-documented both in terms of its voluntary nature and the lack of any significant economic impact on villagers. Field engineers are trained in the land donation procedures and failure to record voluntary contributions in project documents is grounds for dismissal. However, supervision missions confirmed there were instances of absent documentation or

certificates for donated land for sub-projects and this generated the potential for future conflict. On the other side, the project generated substantial social benefits through investments in disaster risk management strategies, and by facilitating villagers’ return to their homes and communities through cash-for-work activities. Implementation of Indigenous Persons (OP/BP 4.10) is rated satisfactory. PNPM follows all best-practice principles for indigenous people’s involvement, and participation is entirely voluntary. While there are indigenous persons in Mentawai, none were identified in the affected project areas. Fiduciary & Governance: The project’s fiduciary and governance compliance is rated as satisfactory. The PNPM Rural III audit target of 20% of sub-districts was increased to 25% to address the risks associated with implementation in disaster-affected areas. The actual external audit rate was 39%, with all four sub-districts in Mentawai (an additional 22%) completing internal audits. External audits for 2011, 2012 and 2013 were submitted with unqualified opinions. Planned external audits of 2014 activities in Mentawai will be submitted in 2015. In addition to external audits, good governance was supported by the complaints handling mechanism: two cases were reported of misuse of funds by a sub-district UPK worth a total of Rp. 82 million (about US$10,000 at the then-prevailing exchange rate) were fully repaid, while two remaining cases of suspected mark up of invoices for heavy equipment services were under investigation at the time the project closed. These were subsequently resolved after the closing date with assistance from the District Facilitator and Provincial Complaint Handling Specialist, the projects were completed and have been formally handed over to the communities. Four complaints out of 781 community projects is evidence of an effective governance strategy. Procurement: The project’s procurement compliance is rated as moderately satisfactory. The June 2014 ex-post review of DMS community procurement activities from 2013 and early 2014 noted they were carried out in accordance with the old PTO, as the revised PTO was not finalized until April 2014. Despite weaknesses in the old PTO that fostered non-transparent processes8, no major noncompliance issues were identified. The overall rating is based on the ex-post review and the finalization of the revised PTO in April 2014. The Bank also recommended PMD develop mechanisms for combining contracts and consolidating procurement procedures to overcome barriers to materials access in remote areas.

2.5 Post-completion Operation/Next Phase Operations and Maintenance. No transitional arrangements are necessary. Following standard PNPM Rural mechanisms, the responsibility for operation and maintenance (O&M) has been assumed by the community through the establishment of O&M teams during the accountability meetings. These teams are responsible for collecting fees for operating costs as well as for the provision of regular infrastructure maintenance. Out of

8 The new PTO simplified the procurement language to make it more accessible to villagers, and established a separate community procurement team where previously procurement had been done by the village-level project implementation team (TPK).

781 total sub-projects, 97% have been officially handed over to communities together with O&M provisions. PMD reported that 12 projects in Mentawai were incomplete by the project closing date and an additional 10 were completed but not yet formally handed over to the communities. These 22 (13%) would not yet have active O&M plans. GoI documented 100% of the projects in Central Java and D.I. Yogyakarta were operational in May 2013 and 87% had good quality O&M. The consultant survey in Central Java in December 2014 reported that 86% of projects were managed well or very well, including ongoing O&M. (See Annex 3). Follow-up projects. The PTO used for activities in disaster-affected areas will be retained for use in any future emergencies.

3. Assessment of Outcomes

3.1 Relevance of Objective, Design and Implementation Rating for Relevance of Objective, Design and Implementation: Relevance of Objective: Substantial This project was well-aligned with the Bank’s CPS of 2009-2012 (Report No. 45381, September 11, 2008 and remained aligned with the Bank’s 2013-2015 CPS (Report No. 74422, December 11, 2012), where it fits three key engagement areas: (i) Promoting Communities, Protecting the Vulnerable and Improving Health Outcomes, (ii) Ensuring Sustainable Development and Improving Disaster Resilience, and (iii) Gender and Governance. At completion, the project was still highly relevant to the objectives of both the Bank and the GoI: it targeted support to disaster-affected communities through PNPM -- the key GoI strategy for improving existing government programs, strengthening institutions, and enhancing the links between different levels of government. The project also supported the practical implementation of community-based disaster risk management under the GOI’s Disaster Management Law (Law No. 24/2007), providing active participation from local government and community members in the planning, implementation, and monitoring of disaster management programs. Relevance of Design and Implementation: Substantial The design and implementation of the DMS project remained relevant as they were built upon the previous PNPM Rural projects; built on experience implementing disaster recovery activities in other affected regions; and according to field discussions with local disaster management agencies and communities, the project was a central mechanism in realizing the CDRM approach. There is a convincing causal chain between project activities and socio-economic development – the first element of the PDO. Recovery of economic and social infrastructure are expected to provide access to villages, restore access to markets, town centers, education and health facilities, and clean water supplies, while also restoring business opportunities and employment for villagers.

Seventy-five percent of beneficiaries surveyed were satisfied with the project and resulting infrastructure (see the Beneficiary Survey results in Annex 5). The implementation strategy–community prioritization, planning and implementation of projects with technical facilitation support–remains also relevant in a country of 17,000 islands where the government often finds it challenging to provide routine basic services. Furthermore, the delays experienced in implementation affecting only 10% of the total sub-projects stem from the difficulties in serving extremely remote areas—not necessarily stemming from the disaster—and were overcome by making available supporting resources sufficient to enable communities to complete the planned tasks. These results suggest that the design and approach remained relevant, but require stronger or customized support to overcome the specific challenges demonstrated in Mentawai.

3.2 Achievement of Project Development Objective Rating: Substantial Communities in regions of Java and Sumatra suffered greatly from natural disasters in late 2010, and while the nature of those disasters differed widely, the impact was the essentially the same: their lives were upended, they lost vital infrastructure and incomes were cut or disappeared. The project was designed to help them regain some control over their lives and livelihoods by providing them the opportunity and the tools to assess the damages in their own communities, determine ways they could reduce future risk, and prioritize and build sub-projects such as roads, bridges, retention walls and evacuation shelters that met their own expressed needs for restoring or improving their social and economic conditions. The project also provides the governance tools to ensure the funds are spent efficiently. Achieving improved socio-economic conditions: Substantial Earlier PNPM Rural research results 9 have demonstrated that investments in rural infrastructure have improved basic socio-economic conditions for communities by providing access to transportation, access to basic services (such as health and clean water), and opportunities for economic recovery for villagers in the project areas. Access to transportation and basic services have helped households to reduce their transportation costs while new economic opportunities, along with temporary employment provided by sub-project implementation and access to the RLF, contributed to project outcomes of increased household expenditure and improved access to economic and social services. In a post-disaster environment, these benefits were augmented by the quick income supplements from cash-for-work activities for 160,000 families returning to their homes amounting to US$2.543 million which helped defray the costs families incurred returning to their communities, replacing lost household goods, making basic repairs, or general consumption smoothing during an uncertain period. Communities also reported important intangible benefits from participation in strategy development for risk mitigation and disaster response, and psychological benefits from having clear disaster responses prepared

9 PNPM-Rural Impact Evaluation. PSF. April 2012, A Qualitative Study: Impact of PNPM Rural in East Java, West Sumatera, and South Sulawesi. PSF. April 2012. Economic Impact Analysis of Kecamatan Development Program Infrastructure Projects. DSF. 2005, Ex-post Evaluation of KDP Infrastructure Projects. World Bank. 2001.

and ready to implement, along with simply restoring routines of work and community life that came with cash-for-work activities.10 Key performance indicators that measure whether or not the project achieved its objectives are: • >50% of villages replace infrastructure lost to disaster. Infrastructure damage in 96%

of the (204 of 213) villages 11 was repaired, with the exception of some coastal Mentawai villages that were relocated to higher ground and built new replacement infrastructure. Sixty-eight percent of all activities were related to: (i) rehabilitation & reconstruction; (ii) disaster mitigation & risk reduction; and (iii) economic and social recovery, as outlined in the technical operations manual, and contributed to improved socio-economic well-being in the target communities reflected in the satisfaction levels (below).

• Minimum 80% satisfaction levels from beneficiaries regarding improved services and

local level governance. PMD reports 12 that the sub-projects reached 491,699 beneficiaries, in addition to providing 700,000 paid cash-for-work days. Seventy-two percent of residents were satisfied or very satisfied with the infrastructure sub-projects,13 which is slightly below the 80% target. However, if weights based on proportion of projects built are applied to the satisfaction ratings found in the survey, the overall level of satisfaction rises to 83%, meeting the project target.

Type of Infrastructure

Positive Satisfaction

Level

% of Sub-projects

built Weight Roads & Bridges 90% 51% .74 Irrigation 67% 15.1% .22 Clean water systems 51% 2.9% .04 Weighted average 83%

An additional proxy for satisfaction levels can be measured by the extent to which communities benefit from, actively use and maintain sub-projects, is reflected in their active maintenance of the sub-projects. The technical evaluation of infrastructure projects undertaken as part of the DMS EIRR evaluation showed that 93% of the sub-projects evaluated (built in 2012-2013) have a functioning community O&M team, reflecting the continued flow of services from the infrastructure, and socio-economic benefits accruing to the community.

10 Based on comments from community members during discussions held as part of the ICR preparation process. 11 PMD’s December 2014 Monthly Report and Project Completion Reports from the National Management Consultant; and Klasifikasi Kegiatan Pasca Bencana, 2012. 12 Ibid. 13 Laporan Akhir Studi Kecil Dampak PNPM Mandiri Perdesaan Pola Khusus Pasca Bencana Wilaya Jawa Tengah, January 2015.

• Economic Internal Rates of Return (EIRRs) >30 % for major rural infrastructure types. The EIRR study (see Annex 3) found the economic internal rate of return (EIRR) of individual sub-projects ranged from zero (for two non-functioning sub-projects) to 341%. After adjustment for the outliers and to reflect a 5% annual decline in the benefit stream, the resulting 40% EIRR is still higher than the targeted 30%.14 These results are consistent with the previous EIRR assessment of similar sub-projects built as part of PNPM Rural IV, which showed EIRR rates of between 35% to 50%.15

• > 70% Project Beneficiaries (and percentage female beneficiaries). PMD reported a total of 491,699 beneficiaries or 74% of the total population (662,706) residing in the in the 18 affected sub-districts.16 Although no target value was identified for female beneficiaries, the percentage of female beneficiaries (50.42%) exceeds the targets established for the PNPM Rural program (at 49%).17 This is in addition to the 700,000 work days for 160,000 people (half women) created under the initial cash-for-work program in 2011.

Table 2: Number and category of DMS Sub-projects

Type of DMS sub-projects Mentawai

Central Java &

DIY Total Percentage

of Total Rehabilitation & reconstruction 9 169 178 23% Disaster mitigation & risk reduction 17 256 273 35% Supporting infrastructure for relocation sites 19 0 19 2% Economic & social recovery 5 57 62 8% General infrastructure development & non-infrastructure 25 224 249 32% Total 75 706 781 100% Disaster-related 50 482 532 68%

The physical outputs of the project were significant. Sixty-eight percent of sub-projects were directly related to the disasters, including 23% of sub-projects which replaced or repaired lost or damaged infrastructure. Overall, communities built or repaired more than 188 km of roads, 98 irrigation systems, 63 schools and health facilities, 17 bridges and 19 water and sanitation systems (see Table A2.2 in Annex 2 for more detail). Achieving improved governance: Substantial

14 Ibid. 15 Laporan Akhir Studi Skala Kecil Analyisis Manfaat Ekonomi Proyek Infrastruktur PNPM Mandiri Perdesaan. PSF. July 2012. 16 PMD’s Project Completion Report, 2014, and Central Bureau of Statistics (BPS). 17 No targets for female and poor beneficiaries were established in the PNPM Rural III Program but PNPM Rural IV is used as a comparison.

Because the DMS project was layered onto PNPM Rural and was identified as a part of that program, the project acquired the same attributes as documented for PNPM Rural, for example, the perception that the project was more transparent than other projects at village level demonstrated the level of satisfaction beneficiaries have regarding the project’s local governance conditions.18 While one of the project’s IRIs fell slightly short of its intended targets, improvements in local governance conditions resulting from the project can be reflected in the following achievements:

• The project’s external audits, undertaken by GOI’s auditing entity BPKP, achieved an audit coverage of 39% in the higher-risk post-disaster environment, exceeding its project target of 25%, and substantially higher than the PNPM Rural target of 20%. An additional 22% of sub-districts completed internal audits. Audit results for 2011, 2012 and 2013 were unqualified, and the final audit report for 2014 will be completed during 2015;

• The complaints handling system identified two modest cases of misuse of funds, both of which were fully repaid, and two possible cases of over-invoicing (see section 2.2 for more detail). This helped to reinforce the message to the public and communities that corruption would not be tolerated; and

• Seven hundred and fifty-nine (out of 781 projects) accountability meetings have been held at the village level at which local implementing teams have handed over projects to the community.

3.3 Efficiency Rating: Substantial Efficiency of infrastructure sub-projects. Efficiency was gauged by the economic rate of return, and the unit rate norms of subprojects financed by the predecessor projects of PNPM-Rural. As mentioned in Section 3.2, a study undertaken after the project closed in November 2014 estimated the EIRRs of sub-projects financed by the DMS project averaged 52%, which is higher than the targeted EIRR of 30% (see Annex 3). It is reasonable to assume that the EIRRs of subprojects financed by PNPM Rural III-DMS could be lower than the average presented in the study because the DMS study assumes that benefits generated by a subproject remains constant over time until the end of its service life, even though benefits will in practice gradually decline because of maintenance. However, even if more conservative assumptions are used and benefits are estimated to decline by 5% per year, the EIRR of most subprojects will, in all likelihood, remain above the economic opportunity cost of capital (which the Bank estimates at 12% for developing member countries), and are likely to remain higher than the project target of 30% (see Annex 3 for details). In addition, more than 90% of the planned projects were completed within the original timeframe (2011-2012) along with 87% of the block grants – 98% of the block grants had disbursed in Java, compared to just 40% in Mentawai. The additional 23 months to provide necessary support enabled communities in the most remote areas to

18 The Local Level Institutions III: Overview Report. World Bank. 2013, and re-stated during ICR discussion groups at the community level.

continue work, and 98% of all sub-projects were completed by the time the project closed on November 30, 2014.

3.4 Justification of Overall Outcome Rating Rating: Moderately Satisfactory While the relevance, efficacy and efficiency of the project are all rated substantial and the four outcome indicators confirm it has achieved its development objectives, the overall rating is moderately satisfactory. The overall rating is reduced to moderately satisfactory owing to weaknesses in the results framework, the underestimation of key risks in the design, and the resulting delays experienced in Mentawai together with the twelve incomplete sub-projects there.

3.5 Overarching Themes, Other Outcomes and Impacts After more than one and a half decades of implementation of CDD projects in Indonesia which have been used with success to expedite and support disaster recovery in Aceh, Nias, Papua, Nusa Tenggara Timur, West Sumatra Central Java and West Java, lessons learned from various evaluations and studies have shown that CDD projects have had positive impacts on community recovery through rehabilitation and reconstruction, development and implementation of disaster mitigation strategies, and economic and social recovery. The underlying foundation of PNPM Rural (and its predecessor, KDP) have demonstrated the ability to help increase household consumption and improve access to economic and social services. Evaluations also have shown that PNPM Rural and KDP are more effective in reaching poor households and households in poor sub-districts19 – all of which are key in effective disaster response. (a) Poverty Impacts, Gender Aspects, and Social Development

While there was no direct measurement of the poverty impact on communities, there are two facts that indicate the project was important for the poor: a) roads, bridges and irrigation – infrastructure with a critical impact on rural incomes via access to markets and increased agricultural productivity – accounted for 55% of all community sub-projects, and b) the satisfaction level for roads and irrigation was 90% and 67% (respectively). The DMS project saw women benefit from participation, accounting for 23% of approved proposals and just over half of the total beneficiaries–with women’s project satisfaction levels being somewhat higher than those of men. The lower levels of participation in project planning meetings as well as the lower percentage of women’s proposals being funded each year (they typically account for more than half of proposals funded in PNPM Rural) may reflect different stresses and constraints placed on women and families during times of crisis, or the tendency for families to return

19 PNPM Rural Impact Evaluation. PSF. 2012, Second Phase KDP Impact Evaluation. World Bank. 2008.

later in the recovery process20 and highlight an opportunity to improve future designs of this nature.

(b) Institutional Change/Strengthening A 2013 study on local level institutions in Indonesia indicates that the project is associated with good transparency and in general has higher satisfaction rates compared to other development projects at the local level21, views which were confirmed during group discussions with beneficiaries at the village level during ICR field visits. These positive achievements have contributed to the GOI’s decision to incorporate CDD principles such as participation/inclusion, community empowerment and accountability into the Disaster Management Law (Law Number 24 of 2012) and subsequently into the Village Law (Law Number 6 of 2014) as noted in Section 3.1. Moreover, communities visited during the ICR field mission consistently indicated that community-based disaster management strategies had an important impact on the communities – they were better prepared to act effectively when disaster next struck, and suggested updates on disaster preparedness needed to be included in the annual village planning meetings. This indicates an organic strengthening of local institutions as a result of processes supported in the DMS project.

3.6 Summary of Findings of the EIRR, Beneficiary Survey and Stakeholder Workshops EIRR and Beneficiary survey. As the project closed, two surveys were carried out: (i) a review of the economic performance (EIRR) of sub-projects including assessments of the technical and management quality of implementation, and (ii) a beneficiary survey of sub-projects. The key findings of these studies are summarized in Section 3.2 (see Annex 3 and Annex 5, respectively, for details on each). Stakeholder workshop. On April 30, 2015, PSF met with key stakeholders in Jakarta. In light of the transition of PNPM from MOHA implementation to the MoV, together with the ongoing design of Village Law implementation, PSF and relevant stakeholders felt it would be useful to focus the workshop on lessons learned and future disaster management. To this end, representatives from MOHA/PMD, MoV, and the National Board for Disaster Management (BNPB) met to discuss key issues for moving forward. Annex 6 contains a summary and the key conclusions of this discussion.

4. Assessment of Risk to Development Outcome Rating: Moderate The political risk is considered to be “Low”. While it is true that the achievements of PNPM Rural and its associated post-disaster additions would not be sustainable in the event of a withdrawal of local and national governmental support for CDD, such a withdrawal is considered highly unlikely. Both the central and local governments have indicated a strong

20 Research and documentation on the behavior and role of women and families during disaster recovery would be useful input for future projects. 21 The Local Level Institutions III: Overview Report. World Bank/PSF. 2013.

and long-term commitment to the financing of community-based poverty alleviation programs, the new Village Law is a strong testament to the success of PNPM, and the Disaster Management Law relies on CDD mechanisms at the local level. The technical risk is rated “Moderate.” The results of the technical evaluation conducted in Central Java at the end of the project, together with the findings from the last supervision mission to Mentawai in June 2014, indicate that O&M remains generally satisfactory, although it is a bit weaker in Mentawai (where 10% of the sub-projects are located) and 76% of those sampled had adequate O&M in place. The social risk is rated as “Moderate.” While local governance remains relatively strong as evidenced by the high numbers of beneficiaries in the DMS project. This is further reinforced by the high proportion of PNPM Rural infrastructure still being used by communities after a number of years. However, there is a marginal risk that the sustainability of community ownership of the project could be undermined if issues are not clearly addressed relating to transfer of project assets to the community as part of the transition to the new Village Law. The natural disaster risk is rated as “Substantial.” West Sumatra is adjacent to a major tectonic thrust and scientists agree that a large earthquake off the west coast of Sumatra is likely in the coming decades that will cause substantial destruction. Mt. Merapi in Java has erupted regularly over the past 500 years. Communities may be better able to manage the risk of recurrence as a result of the project, but will require additional assistance when it happens. Based on the above, the overall risk to development outcome is rated Moderate.

5. Assessment of Bank and Borrower Performance

5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Satisfactory The design of the DMS project was based on the design of PNPM-Rural III, its predecessor rural programs, and predecessor PNPM-linked disaster recovery projects, which were considered at the time of appraisal to be successful, but with some corrective actions to reflect lessons learned from these projects (notably the need for additional support for facilitation, and governance improvements) and otherwise using the same implementation arrangements. The Bank kept design changes to a minimum in order to ease implementation in disaster-affected communities as it would have been imprudent to introduce design changes along with substantial additional resources into a period of emergency and disaster recovery. Utilizing the existing project structure was a factor in the quick turnaround time for this project – four months from events to Board approval. However, relevant risks for Mentawai that were not identified at the outset could have improved the quality and speed of implementation; and certain weaknesses in the

formulation of indicators in the results framework could have been addressed. However, in view of the Bank’s demonstrated efficiency in processing the project especially under extenuating circumstances, and given that the majority of the Grant funds were implemented as planned, Bank performance in ensuring quality at entry is rated as “Moderately Satisfactory.” (b) Quality of Supervision Rating: Moderately Satisfactory Administrative supervision was effective, including extending closing dates as appropriate, identifying issues such as the unused trust fund which the GoI elected to cancel, and facilitating the reallocation of remaining funds to use in disaster-affected areas in East Java and North Sumatra. GoI issues and requests received timely responses; Aides-Memoire identified issues early in the project cycle, were discussed constructively with GoI and action plans were followed up. Evaluations in ISRs were generally consistent with the ICR assessment. The Bank mobilized a series of multi-disciplinary supervision teams with substantial experience in supervising Bank-financed CDD projects in Indonesia, and experience in remote areas. Regular implementation support missions were carried out in addition to specific technical missions in the effort to expedite progress in Mentawai. The team was able to identify and help GoI proactively address key issues adversely affecting achievement of the PDO, notably problems in Mentawai with facilitation staffing and performance, management, community procurement and safeguards issues, and flagged this for attention by PMD. These missions resulted in a series of important recommendations, including using NMC to strategically fill gaps in support from the province and district, which was key to improving performance in Mentawai during the last 18 months. Furthermore, in recognizing some of the shortcomings described in Section 5.1 (a), the evaluation studies implemented by the Bank, were designed to examine not just the technical efficiency of the project’s investments, but more specifically, beneficiary perceptions regarding the improved socio-economic and local governance conditions created by the project. Procurement and financial management issues were identified in Mentawai, and most were addressed with the revised PTO issued in March 2014. The remaining FM issues stemmed from high facilitator turnover and limited capacity building at the community level. Lack of data on Mentawai’s tsunami-affected resettled communities, late identification of issues for the communities resettled on Ministry of Forestry land (protected areas), and lack of progress on documentation of donated land despite potential conflicts indicates that safeguards supervision could have been strengthened. (c) Justification of Rating for Overall Bank Performance Rating: Moderately Satisfactory

5.2 Borrower Performance

(a) Performance of the Government Rating: Moderately Satisfactory Throughout project design and implementation, the project enjoyed the strong support of GoI. GoI responded quickly to the disasters, 22 working with district governments to identify heavily affected areas, complete the identification of project locations by early January, and launch preliminary activities by mid-January23. GoI ensured that funds were generally provided in a timely manner to the community – 84% of the funds were disbursed by December 2012 – although rigidities in mid-year budget processes and timing caused initial delays. Despite this, the Government was still able to ensure quality of preparation, relatively smooth implementation, and compliance with the Grant Agreement. (b) Implementing Agency Performance Rating: Moderately Satisfactory As the Implementing Agency, PMD played an important role throughout the life of PNPM Rural III-DMS for the overall implementation of the 781 sub-projects. During the same period it was managing the massive scale-up of PNPM Rural from 2,800 sub-districts in 2007 to more than 4,500 in 2012. Ninety percent of the sub-projects accounting for 81% of the block grants were completed on schedule in Java. The delays experienced in Mentawai were due to the very remote location and inadequate facilitation, compounded by lack of resources and weak local government. Weak oversight of activities in Mentawai during the initial project period resulted in problems in financial management, lack of transparency in implementation indicated low participation, and lack of safeguards training and awareness among facilitators and communities was evident throughout the project despite their inclusion in the project materials. These weaknesses were clearly evident in safeguards, and could set the stage for future conflicts. The fact that there were enough unused and uncommitted block grant funds in the Mentawai sub-districts to fund cash-for-work programs without having to draw on DMS funds should have indicated there were pre-existing capacity issues. Despite the difficult start, PMD marshaled their available resources, reorganized, and turned the Mentawai program around during the last 18 months of implementation – the result being 98% overall DMS sub-project completion. The revision of the PNPM PTO and increase in facilitator benefits for the overall PNPM Rural program also supported improvements during the later stage of the program, but did not solve the problems with extremely high costs for facilitators in remote areas. (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Satisfactory

22 The earthquake and tsunami hit Mentawai on October 25, 2010, and Mt. Merapi erupted on the same day. The volcanic activity lasted until late November. 23 PNPM guidelines allow the reallocation of regular block grant funds to community-designated cash-for-work programs during declared emergencies, and this was done in anticipation of the DMS project already under preparation and informally endorsed by the Joint Management Committee of PSF.

Based on the performance of both the Government and the Implementing Agency, the overall performance of the Borrower was also rated “Moderately Satisfactory.”

6. Lessons Learned

a. A clear and measurable results framework is essential. The results framework is intended to set clearly articulated targets for outcomes that the project is expected to achieve over time, and which can be periodically measured to indicate to what extent the targets have been achieved. Measurements or studies undertaken after the project has been completed may be useful for long-term assessment, but cannot be relied on for management purposes as they do not enable early detection of implementation weaknesses requiring management attention.

b. Good community targeting is key, along with flexibility to meet local needs as they

emerge. Two sub-districts receiving DMS block grants had no physical damage from the disaster, but suffered severe economic downturns as a result. For logical reasons, these communities had difficulty developing sub-projects linked to the disaster. Moreover, while the project’s basic operating system and facilitation platform are well known and have demonstrated their ability to serve communities under special circumstances, it requires substantial community effort to assess, plan, prioritize and implement activities. For the very early stages of recovery community-administered cash-for-work programs may be more appropriate, and adequate flexibility should be built into the project design to adapt to this.

c. The rigidity of Government bureaucratic and budgetary processes, including the

PNPM Rural annual project cycle means that it often takes months before community-based disaster response activities can begin. It would be useful to explore options under the new Village Law that could provide some flexibility during times of officially declared disasters.

d. Remote areas present unique challenges for PNPM Rural – and those challenges

are intensified during periods of disaster. Facilitation is critical to enable communities to implement good quality activities, and Mentawai suffered from vacancies and high turnover even before the earthquake and tsunami hit. Coupled with the scarcity of building materials and lack of infrastructure, this was a very difficult implementing environment. Special strategies are needed to ensure that adequate resources are available to reflect real scarcity and high costs, including the cost of good facilitation.

e. Disaster-affected communities need adequate facilitation resources. The additional

block grants from the DMS Grant funds sometimes quadrupled and quintupled the workload for facilitators, often while they faced broken infrastructure needed to do their jobs. Communities need adequate support – in the form of additional, experienced social and technical facilitators – to ensure sound implementation and to avoid disempowering communities through failed projects. Protocols to enable fast and effective Central-level support for such locations (e.g. by way of National Disaster Management Facilitators/Consultants) as a stop-gap until adequately

qualified personnel are recruited may also be a consideration going forward. Incentives (e.g. emergency allowances, the addition of “special skills” in professional facilitation qualifications, or priority for future advancement) should be developed that encourage experienced, skilled workers to step up in times of disasters, and for any new recruits to be allocated to more stable environments.

f. In recurrent, high-risk areas (such as around volcanoes), CDRM mechanisms can

be integrated into the village planning process. Communities would benefit from closer coordination with the local Disaster Management Agency (BPBD) which requires continuous awareness and planning. The Disaster Management Law and the Village Law can be mutually supportive in these areas, and good community participation as part of the annual process can help provide pro-active risk reduction & disaster management strategies.

7. Comments on Issues Raised by Borrower/Implementing Agencies/Donors (a) Borrower/Implementing agencies (b) Cofinanciers/Development Partners (c) Other partners and stakeholders

Annex 1. Project Costs and Financing

a) Project Cost by Component (in US$ million equivalent)

Components Appraisal Estimate (US$ millions)

Actual/Latest Estimate (US$ millions)*

Percentage of Appraisal

Kecamatan Grants 12.9 13.9 107.7 Facilitation Support 1.0 0 0 Implementation Support and Technical Assistance 0.2 0 0 Total Baseline Cost 14.1 13.9 98.5 Physical Contingencies 0.0 - 0 Price Contingencies 0.0 - 0 Total Project Costs 14.1 13.9 98.5 Front-end fees 0.0 - 0 Total Financing Required 14.1 13.9 98.5 *Provisional figures (minor differences may arise between these data and final loan closing data)

b) Financing

Source of Funds Appraisal Estimate (US$ millions)

Actual/Latest Estimate (US$ millions)*

Percentage of Appraisal

Borrower 0 0 0 Communities 0 0 0 International Bank for Reconstruction and Development (IBRD) 14.1 13.9 98.5 Co-Financiers 0 0 0 Total 14.1 13.9 98.5 *Provisional figures (minor differences may arise between these data and final loan closing data) Two Trust Funds were initially established to fund these activities:

• TF-98819 in the amount of US$13.9 million was intended to fund Component I: Kecamatan Grants (US$12.9 million), and Component II: Facilitation Support (US$1.0 million)

• TF-98862 in the amount US$0.2 was intended to fund Component III: Implementation Support and Technical Assistance

When the project extension was negotiated in December 2012, it was judged that resources under the larger TF-98819 would be adequate to fund all activities, and the GOI and World Bank agreed to allow TF-98862 to close without disbursements. As a result, the total value of the Grant was reduced to US$13.9 million.

Annex 2. Outputs by Component

Component I: Kecamatan Grants Overview. This component first provided grants for cash-for-work programs during 2011 in Central Java and Yogyakarta24. As detailed in the operational guidelines, one member of a household could participate in these activities, resulting in communities implementing 2,709 work activities with more than 700,000 paid work days benefitting a reported 160,000 people/households as they returned to their home villages. Block grants were provided to 18 disaster-affected kecamatan in West Sumatra, Central Java and Yogyakarta beginning in 2012 to finance 781 competitive grant proposals selected by communities according to priorities resulting from the community disaster assessments, and benefitting 491,699 people (73% of the population, more than half women). Block grant amounts ranged from IDR 4 billion (US$453,000) for less-severely affected areas, to IDR 6 billion (US$679,000) for those areas suffering the heaviest impact (Table A2.1). These disaster-related block grants were substantially larger than standard PNPM Rural block grants, which ranged from IDR 750 million to IDR 3 billion per kecamatan per year, depending on population, poverty incidence and location. Investments took place in: (i) physical infrastructure, (ii) social programs, and (iii) revolving loan funds. As with PNPM Rural block grants, communities were allowed to allocate a maximum of 25% of the total DMS kecamatan block grant amount to revolving loan funds to support economic recovery (provided that the recipient RLF had a repayment rate of at least 80%), although only five out of the 18 sub-districts chose this option and the RLF only amounted to just 4.2% of the total block grant allocation. Women proposed 21% of the sub-projects selected for use of the block grants. Most block grants for infrastructure and social programs were allocated to village roads, retaining walls and irrigation systems, followed by investments in education facilities and revolving loan funds (Table A2.2). Tracking the projects as shown in Table A2.3 reveals that 68% of the activities were classified into three disaster-related categories: (i) rehabilitation & reconstruction; (ii) disaster mitigation & risk reduction; and (iii) economic and social recovery. These categories follow the instructions in the technical operations manual provided to facilitators, and is a more appropriate indicator of PDO achievement than simply rehabilitation & reconstruction as entered in the results framework. This is particularly true in areas such as Mentawai where the nature of the disaster (earthquake and tsunami) generated activities that often supported communities moving to new locations on higher ground, with no rehabilitation or reconstruction at the original locations. Overall, female beneficiaries were calculated at 50.42% of the total. These included women in 127 revolving loan groups consisting of 10 – 15 members each (1,250-1,900 women) who received block grant allocations in five sub-districts. The sub-districts which chose this option were among the larger, better-performing revolving loan funds – all were larger than the median RLF in Central Java with the largest assets nationally, Yogyakarta was the best performing province nationally with an NPL of 5% at end-2013, and Central

24 West Sumatra cash-for-work programs were funded from unspent block grant resources.

Java was the fourth-best with an NPL of 12%, compared to the national average of 30%. This data is used as a proxy for local performance because detailed performance data on the revolving loan fund by subdistrict is not available now that PNPM Rural activities have ceased and the NMC has closed. The project’s investments in rural infrastructure mentioned above have provided many of the same benefits as regular PNPM Rural project activities, such as access to transportation, access to basic services (such as health and education), and opportunities for economic activities for villagers in the project areas. While access to transportation helps households reduce their transportation costs and expand economic opportunities, in disaster-affected areas it also means evacuation routes have been prepared and clearly marked along with supporting infrastructure along the way. Rehabilitation of basic health and education facilities together with the temporary employment provided by the projects and the revolving loan funds, all enabled families to return to their villages and re-establish normal community life. Table A2.1: Locations and number of proposals funded by the Project

Location damage level

# sub- projects

West Sumatra Kab. Kepulauan Mentawai Pagai Utara very heavy 18 Sipora Selatan very heavy 16 Sikakap very heavy 20 Pagai Selatan very heavy 21 Central Java Kab. Magelang Borobudur heavy 62 Ngluwar medium 29 Salam heavy 44 Srumbung very heavy 61 Dukun very heavy 58 Sawangan heavy 50 Mungkid heavy 47 Kab. Boyolali Selo heavy 29 Cepogo heavy 29 Musuk heavy 38 Kab. Klaten Manisrenggo very heavy 56 Karangnongko very heavy 75 Kemalang very heavy 84 Yogyakarta

Location damage level

# sub- projects

Kab. Sleman Cangkringan very heavy 44 781

Table A2.2: Types and Number of Sub-projects Financed by the Project

Type Mentawai

Central Java &

DIY Total % Roads* 42 274 316 40% Retaining walls 0 154 154 20% Irrigation 1 97 98 13% RLF Credit 0 51 51 7% Education facilities 9 42 51 7% Health facilities 6 6 12 2% General infrastructure 1 21 22 3% Drains & culverts 1 19 20 3% Water & sanitation 3 16 19 2% Bridges 5 12 17 2% Weirs & flood control 0 11 11 1% Education & training 5 2 7 1% Relocation site prep 2 0 2 0% Markets 0 1 1 0% Total 75 706 781 100% *totaling 188 km Source: Project MIS System (31 December 2014)

188 km of roads, 98 irrigation systems, 63 schools and health facilities, 17 bridges and 19 water and sanitation systems (see Table A2.2 in Annex 2 for more detail).

Table A2.3: Disaster-related Sub-projects

Mentawai

Central Java &

DIY Total % Rehabilitation & Reconstruction 9 169 178 23% Disaster mitigation & risk reduction 17 256 273 35% Supporting infrastructure for relocation sites 19 0 19 2% Economic & social recovery 5 57 62 8% General Infrastructure development 25 224 249 32% Total 75 706 781 100% Disaster related 50 482 532 68% Source: Project MIS System (31 December 2014)

Sustainability. Based on a 2013 exercise on infrastructure quality mapping covering all of Central Java and Yogyakarta, 91% of the infrastructure projects built under the DMS project included in the evaluation were considered high quality. The challenges of good O&M were mitigated in PNPM Rural by formally appointing village-level O&M committees at the time the project is completed and formally handed over to the community, with the objective of improving governance by instilling local ownership. According to NMC technical evaluations in Central Java and Yogyakarta in 2014 (more than one year after sub-project completion), 100% of the infrastructure sub-projects were operational, and 87% were being well-maintained. Quality and O&M information is more difficult to obtain from Mentawai, but the Central Java and Yogyakarta figures alone yield 78% satisfactory maintenance of infrastructure sub-projects over the entire project, which exceeds the 50% target. Component II: Facilitation Support Each province, district and sub-district has an established structure of PNPM Rural facilitators, including technical, financial and social facilitators at the sub-district (kecamatan) level, and technical facilitators at the district (kabupaten) level. With the assistance of local cadres in each village, these facilitators help strengthen inter-village organizations and forums, and assist local government coordination and oversight. The original intent of this component under the DMS project was to finance additional facilitators, including disaster management specialists, in each of the target districts and sub-districts to support district and sub-district government institutions, assist communities with the disaster assessment process, provide training, etc. Shortly after this series of natural disasters struck, the GOI made major revisions to its disaster response mechanisms in the Disaster Risk Management Law (Law No. 12 of 2007), establishing permanent national and provincial coordinating bodies, and permanent,

dedicated agencies at the district level. 25 Considering these changes, GOI chose an alternative strategy of adding an assistant facilitator and a local assistant for each of the 18 disaster-affected sub-districts, plus a national disaster management specialist based in Jakarta. This strategy worked reasonably well, although sub-district facilitators and village implementation teams emphasized the need for experienced facilitators – both social and technical – when faced with such dramatic increases in workloads (some sub-districts quadrupled or quintupled their project implementation as a result of the DMS project). The strategy was less successful in Mentawai, where weak government management combined with very challenging working conditions in a remote location meant that there was a near-constant shortage of facilitators and very high turnover. Facilitators were having a difficult time meeting program requirements before the earthquake and tsunami hit the islands (where the block grants had ranged from IDR 1.75-3 billion), and the increased workload coming from the additional DMS block grant of IDR 5 billion, complicated by weak management support, meant that the full program was beyond their capacity to implement. Additional skilled assistance was required, and the national disaster management specialist played a key role in supporting facilitators in Mentawai during 2013-2014, including monthly visits during 2014 to help resolve problems and sustain implementation momentum. Ultimately, Components I & II were consolidated as part of the extension process in December 2012, and all costs were drawn from this consolidated Component 1 budget. Component III: Implementation Support and Technical Assistance PNPM Rural is managed by a Project Management Unit (PMU) in PMD, with technical assistance provided through a National Management Consultant (NMC) at the central level, and Regional Management Consultants (RMCs) at the central and provincial level. The activities under Component III were intended to finance technical advisory services, training and other material support for DMS project implementation at the national, provincial, district, and sub-district levels, including training, monitoring and evaluation and enhanced technical and financial audits. The Jakarta-based national disaster management specialist was hired by the NMC in 2013, who was strategic in organizing increased support to Mentawai from the provincial and district consultants in addition to his own monthly support visits. Ultimately, the financial audits were fully integrated into the main PNPM Rural activities, and the additional costs for technical assessments and the national disaster management specialist engaged by the National Management Consultant (NMC) were drawn from the resources in the main Trust Fund (TF-98819). For these reasons, the actual costs in the tables in Annex 1 are shown as zero.

25 Provincial and district disaster management bodies were previously ad hoc appointments of local government officials drawn from various sectors at the time of a declared disaster, and they returned to their home agencies once the emergency was past. The new permanent agencies are intended to professionalize the services and maintain continuity while incorporating principles of community-based disaster risk management (CDRM).

Table A2.4: Incomplete projects at closing date

Mentawai Sub-district

No. projects

average completion

level Pagai Selatan 9 84% Pagai Utara 2 72% Sikakap 1 98% Total 12 83%

Intermediate results indicators. The targets for all but one of the intermediate results indicators were achieved, the exception being the minimum 50% participation rate for women and the poor in planning and decision-making meetings. The project exceeded all other IRIs, including O&M arrangements in place as well as audits carried out, both of which further underscore improvements in governance.

IRI Achievement Comments 1. Minimum 50% participation rate of women and poorest community members in planning and decision-making meetings

Below target. Women’s partici-pation in planning and decision making meetings was 42.4%, while participation of the poor was 28.5%.

• PNPM Rural III only established a 45% target value for the participation rate of women and poorest community members in planning and decision-making meetings. In addition, while the participation rates of the poor in the planning and decision-making meetings appear to be significantly lower than the target, PMD reported that 44.7% of project beneficiaries came from poor households.

2. Number and type of infrastructure works, economic, and education and health sub-projects/activities completed in selected disaster areas.

No target set in the Results Framework. However, a baseline of 781 projects was planned from the outset of the project. 769 out of 781sub-projects (98%) have been completed and are operational.

• While no target value was established, using a similar comparator in PNPM Rural III where a target value of >85% of work plans were to be completed each year, 98% is a good result. See Annex 2 for sub-project details.

3. Minimum 70% of infrastructure works are evaluated as of high quality.

Target exceeded. Evaluations range from 86-91%.

• PMD/NMC evaluation of infrastructure quality in Central Java and D.I. Yogyakarta in 2013 indicated 91% of projects were high quality.26

• Technical evaluation as part of the EIRR analysis more than a year later in January 2015 indicated that 86% of

26 Penilaian Kualitas Prasarana di Provinsi Jawa Tengah dan D.I. Yogjakarta, National Management Consultant, 2013.

IRI Achievement Comments projects were considered good or very good quality.27

4. O&M arrangements are in place and functioning for minimum 50% of infrastructure works.

Target exceeded. Three measurements showed consistent results, in the 86-87% range.

• Technical evaluation in December 2014 included aspects of O&M in the analysis of project quality and in the evaluation of management quality. 86% of projects had good or very good technical quality, while 86% of management quality was good or very good.

• 2013 NMC analysis (noted above) reported all infrastructure projects were operational, and that 87% were well-maintained.

5. Minimum 70% of agreed additional consultants are recruited and trained; 90% of disaster-affected sub-districts get additional facilitators and training for disaster assessment.

Target exceeded. 100% of additional consultants hired and trained; 100% of affected sub-districts had additional facilitators.

In addition, one Disaster Rehabilitation Specialist was recruited and hired by NMC to support local teams.

6. Audits carried out in 25% of disaster affected sub-districts.

Target exceeded. External audits in 7/18 sub-districts (39%), and internal NMC audits in an additional 4 (+22%).

External audits for the additional four sub-districts planned for 2015.

27 Laporan Akhir Studi Kecil Dampak PNPM Mandiri Perdesaan Pola Khusus Pasca Bencana Wilaya Jawa Tengah, draft Januray 2015.

Annex 3. Economic, Financial and Technical Analysis Introduction. In 2014, PSF commissioned a study28 to analyze the economic benefits of small-scale infrastructure financed by the DMS project, and to survey beneficiaries on their level of satisfaction with the infrastructure sub-projects. This annex summarizes the final report of the consultant, which contains an analysis of a sample of 29 small-scale infrastructure projects distributed over nine sub-districts from three of the five districts included in the project. Mentawai was not included in the survey because most of the sub-projects were still in the final stages of implementation at the time of the survey; Sleman district in Yogyakarta was not included because activities took place in only one sub-district. The three sample districts cover 64% of the project’s total block grants, 85% of all sub-projects, and 88% of all infrastructure sub-projects. The analysis mirrors the previous surveys of economic benefits for PNPM Rural, and covers water supply, roads and bridges, and irrigation, which constitute 62% of the infrastructure subprojects undertaken in the DMS project. The report presented the results of three types of analysis: (i) economic internal rate of return (EIRR); (ii) beneficiary satisfaction; and (iii) technical and management quality. The results of the beneficiary survey are presented in Annex 5. The study followed the methodology that was developed for similar studies conducted in 200529 and 2012.30 Economic internal rate of return. The World Bank’s Handbook on Economic Analysis of Investment Operations requires the real economic rate of return (EIRR) of Bank-financed sub-projects to be equal or higher than the opportunity cost of capital of the country (which the Bank estimates at 12% per annum for developing country members, including Indonesia). After adjusting for outliers showing returns above 100%31, the average unweighted EIRR of the 29 projects analyzed by the consultant was approximately 60%, above the minimum required rate, and also above the target for the project (30%). The average unweighted EIRRs were highest for roads and bridges (63%), followed by irrigation (58%) and clean water systems (56%). The list of EIRRs by sub-projects and by district can be seen in Table A3.1 The average EIRR of sub-projects financed by PNPM Rural III—DMS is likely to be lower than the EIRRs presented in the study for the following reasons:

• The study did not take into account sub-projects that were no longer functional. These included two clean water projects in the sample selection, and if they are included in the calculations as representative, the average clean water EIRR drops to 40.38%, and the overall project average falls to 52.00%.

28 Laporan Akhir Studi Kecil Dampak PNPM Mandiri Perdesaan Pola Khusus Pasca Bencana Wilayah Jawa Tengah , January 2015. 29 Economic Impact Analysis of Kecamatan Development Program Infrastructure Projects, Anthony Torrens, January 2005. 30 Laporan Akhir Study Skala Kecil Analisis Manfaat Economi Projek Infrastructure PNPM Mandiri Perdesaan, PNPM Support Facility, July 2012. 31 The consultant identified one outlier for clean water systems which documented an EIRR of 340.5%, and one irrigation system with a calculated EIRR of 262.3%. While the calculations were judged to be correct based on the information available, a most conservative estimate strategy was used which assigned the sub-project an EIRR equal to the average EIRR for the other similar projects in that district.

• The study assumes that benefits generated by a subproject remains constant over

time until the end of its service live, even though benefits will in practice gradually decline because of inadequate maintenance. Assuming that benefits decrease by 5% per year, the EIRR drops from 52.00% to 40.30%. After these adjustments, it is likely that the average EIRR of most project-financed sub-projects will remain comfortably above both the 12% economic opportunity cost of capital and the project target of 30%.

Table A3.1: Types and Locations of Sub-projects, and their Average EIRRs

Number EIRR Clean Water Systems Klaten 2 32.80% Boyolali 3 61.40% Magelang 3 24.40% Subtotal 8 40.38% Roads & Bridges Klaten 4 64.70% Boyolali 2 48.20% Magelang 6 68.50% Subtotal 12 63.85% Irrigation Klaten 3 71.90% Boyolali 1 78.20% Magelang 5 25.00% Subtotal 9 46.54% Total 29 52.00%

Analysis of Technical and Management Quality. Two additional analyses were completed of the 29 sub-projects to determine the (i) the technical quality of infrastructure works, and (ii) an evaluation of the sub-projects’ management quality. The quality categories for both analyses included very good, good, average and poor/bad, with only small differences noted between very good and good. The results are presented in Table A3.2, and indicate that 86% of the infrastructure projects were considered technically good or very good.

Table A3.2: Technical Quality of Small-scale Infrastructure Works

number very good good average poor/bad

Clean Water Systems 8 2 3 1 2 Roads & Bridges 12 2 9 1 0 Irrigation 9 5 4 0 0 Total 29 9 16 2 2 Percentage 100% 31% 55% 7% 7%

Overall, 86% of all infrastructure sub-projects in the sample were evaluated as good or very good quality. This is consistent with a similar assessment by the NMC in 201332 which estimated 91% of projects were good or very good quality. Both of these exceed the minimum 50% target in the intermediate outcomes. A detailed assessment of management quality was also conducted, based on 28 aspects of implementation covering 5 phases: (i) sub-project preparation, (ii) implementation of works, (iii) evaluation of the physical works, (iv) maintenance, and (v) post project evaluation. The results of the analysis in Table A3.3 indicate that 86% of the sub-projects were managed either well or very well by the community implementation team, and only one sub-project from the sample was considered poorly managed. Table A3.3: Implementation Management Quality for Small-scale Infrastructure Works

number very good good average poor/bad

Clean Water Systems 8 4 1 2 1 Roads & Bridges 12 5 6 1 0 Irrigation 9 4 5 0 0 Total 29 13 12 3 1 Percentage 100% 44.8% 41.4% 10.3% 3.4%

32 Hasil Penilaian Kualitas Prasarana di Provinsi Jawa Tengah dan D.I. Yougyakarta, April-May 2013, results as reported in the PCR. No information is available on the methodology used for that assessment.

Annex 4. Loan Preparation and Implementation Support/Supervision Processes

(a) Task Team members

Names Title Unit Responsibility/ Specialty

Lending

Kevin A. Tomlinson Senior Operations Officer GSURR Task Team Leader (01/17/11-08/18/11)

John Victor Bottini Senior Social Development Specialist GSURR Operations

Sentot Surya Satria Senior Social Development Specialist GSURR Operations

Bisma Husen Procurement Specialist GGODR Procurement Zulfi Novriandi Procurement Analyst GSURR Procurement Indira Dharmapatni Senior Operations Officer GSURR Environmental Safeguards Virza Sasmitawidjaja Consultant, Safeguards GSURR Environmental Safeguards Yogana Prasta Operation Adviser EACIF FM and Disbursement

Unggul Suprayitno Senior. Financial Management Specialist GGODR Financial Management

Melinda Good Senior Counsel LEGES Legal Alexander B. Setiadji Social Development Specialist GSURR Governance Supervision /ICR

Jan Weetjens Lead Social Development Specialist GSURR PSF Manager

Sentot Surya Satria Senior Social Development Specialist GSURR

Task Team Leader (08/19/11-04/27/14)

Sonya Woo Senior Operations Officer GSURR Task Team Leader (since 04/28/14-11/30/14)

Susanne Holste Lead Social Development Specialist GSURR Operations

Yogana Prasta Operation Adviser EACIF Operations

Unggul Suprayitno Senior Financial Management Specialist GGODR Financial Management

Mariangeles Sabella Senior Counsel LEGES Legal Ahsan Ali Lead Procurement Specialist GGODR Procurement

Yash Gupta Senior procurement Specialist GGODR Procurement Achmad Zacky Wasaraka Procurement Analyst GGODR Procurement Zulfi Novriandi Procurement Analyst GSURR Procurement Franciscus Prahastanto Operations Analyst GSURR Operations Griya Rufianne Operations Analyst GSURR Operations

Robert Anders Anderson Operations Officer GSURR Operations Dennie Mamonto Consultant, Safeguards GSURR Environmental Safeguards Flávio Theodoro Chaves Consultant, Safeguards GENDR Environmental Safeguards Sulistiowati Nainggolan Consultant, Safeguards GEEDR Social Safeguards Ardiani Chandra Dewi Operations Analyst GSURR Operations Hanggar Irawan Operations Analyst GSURR FM & Disbursement Festina Lavida Consultant, Fiduciary GSURR Financial Management Nia Yuniarti Program Assistant GSURR Operations

(b) Staff Time and Cost

Stage of Project Cycle

Staff Time and Cost (Bank Budget Only)

No. of staff weeks US$ Thousands

(including travel and consultant costs)

Preparation FY11 16.97 52.97

Total: 16.97 52.97 Supervision/ICR

FY12 25.59 41.33 FY13 37.25 46.75 FY14 22.08 45.38 FY15 11.91 27.96

Total: 96.83 161.43

Annex 5. Beneficiary Survey A beneficiary survey was conducted in December 201433 of 416 respondents (209 F, 207 M) residing nearby the infrastructure sub-projects in the three districts revealed that 71.63% of all respondents were either very satisfied (13.94%) or satisfied (57.69%) with the infrastructure projects constructed in their areas, below the 80% target. Women were slightly more positive toward the projects than men: 74.16% were either very satisfied or satisfied. Only 15.3% of respondents were not satisfied with the completed infrastructure in their areas. The respondents’ levels of satisfaction is shown in Table A5.1, while the variation in satisfaction by type of project and by gender is shown in Table A5.2. Table A5.1: Levels of Satisfaction with Construction Quality/Usefulness of Infrastructure Sub-projects

Level of Satisfaction Total Respondents Female Respondents

Number Percentage Number Percentage Very Satisfied 58 13.94% 23 11.00% Satisfied 240 57.69% 132 63.16% Neither Satisfied nor Dissatisfied 54 12.98% 28 13.40% Dissatisfied 61 14.66% 25 11.96% Very Dissatisfied 3 0.72% 1 0.48% Total 416 100% 209 100% Total Satisfied and Very Satisfied 298 71.63% 155 74.16%

Table A5.2: Satisfied Respondents by Gender and Type of Infrastructure

Type of Infrastructure: Men and Women Women Clean Water Systems 50.9% 60.7% Roads & Bridges 90.0% 94.9% Irrigation 66.9% 61.6% Total 71.6% 74.2%

Roads were the most popular investment with a 90% general satisfaction level, rising to 95% for women. Roads were also the most common investment in disaster-affected areas, 34 making up 73% of the three types of projects selected for the sample, with irrigation projects making up 23% and clean water the remaining 4%. If the product-specific satisfaction level is applied to the proportions of infrastructure projects actually built (communities built 333 roads & bridges compared to just 19 clean water systems), the overall level of satisfaction rises to 83% overall (86% for women), meeting the project target.

33 Laporan Akhir Studi Kecil Dampak PNPM Mandiri Perdesaan Pola Khusus Pasca Bencana Wilayah Jawa Tengah , January 2015. 34 These types of projects were 69.4% of total infrastructure sub-projects: clean water systems (2.9%), roads and bridges (51%), and irrigation (15.1%).

Annex 6. Stakeholder Workshop Report and Results On April 30, 2015 PSF organized a workshop in Jakarta to collect comments on the initial findings of the project evaluation presented in this ICR, and to discuss lessons learned. Owing to the PNPM transition from MOHA to the Ministry of Villages (MoV) and the current processes to develop more comprehensive village level planning and investment mechanisms under the Village Law, stakeholders were invited from MOHA, MoV, and the National Board for Disaster Management (BNPB). Besides a review of implementation, a main objective was for all stakeholders to consider the lessons learned from the DMS project and how these might be usefully integrated into local government functions and processes now being developed. In addition to the World Bank team, representatives from MOHA/PMD, MoV and BNPB participated in the workshop. Discussions and conclusions can be summarized as follows: • There are important differences between emergency and reconstruction phases of

disasters. The World Bank is not generally well-suited to deal with immediate emergency responses because of the time frames required for design, appraisal and approvals, and the flexibility often required for effective emergency responses typically can’t be incorporated into Bank projects. On the other hand, the DMS project was both effective and was thought by stakeholders to be efficient in its support to communities for assessing the impacts, prioritizing and rebuilding in the reconstruction phase. Economic recovery should also be included in the medium-term recovery process, and DMS allowed these options. Providing additional grant funding to an existing project was a useful and relatively fast mechanism, but managers should not underestimate the extent of the additional responsibilities placed on them.

• It was agreed that aid in response to disasters requires treatment and/or implementation

guidelines tailored to the local environment and context. The conditions in Java and Mentawai were very, very different in all respects (geography, remoteness, infrastructure, tradition & culture, etc.) and in hind sight, it probably would have been better to divide the grant into two separate projects. This would allow some variation in implementation guidelines, and more importantly, in the implementation timeline. However, participants agreed that the GoI’s annual budget cycle was still a binding constraint for activities in remote areas because they took longer to organize, longer to procure materials, and overall were more expensive and required different allowances.

• All stakeholders acknowledged that facilitation in remote areas is a problem and can

have a major impact on all project implementation – not just disaster recovery. MOHA agreed that additional experienced facilitators were needed in post-disaster environments. BNPB had similar experience with the facilitators they trained and dispatched to Mentawai – turnover was very high, and few opted to stay at their posts. Stakeholders agreed that special steps should be taken to hire and train local facilitators in remote areas, as they are more likely to understand the needs of the local communities, their culture, and will not have the same difficulty living in the community. BNPB budgeted Rp. 325 trillion (US$25 million) to reconstruction in Mentawai, but between the difficulty in retaining facilitators and delays from resolving

land use issues, they ended up building 1,000 homes and returning Rp 300 trillion (92%) back to the central account.

• The participants put together a lengthy agenda of key issues for integrating disaster risk

management into the village planning process as part of the Village Law: - What mechanisms can best be used to integrate disaster risk management into the

annual village plan? - Will training modules have to be adapted or developed? - How will processes across MOHA, MoV and BNPB be coordinated and

reconciled? Under BNPB guidelines, local government is required to make contingency plans for disaster preparedness, yet these are not yet included in MoV planning processes. The BNPB also has a community resilience evaluation process with 20 indicators, including community participation in the annual village planning meeting.

- How will villages plan and manage cross-village investments for disaster management? For example, building and maintaining shelters along the evacuation route – how will ownership and O&M be managed under the new law? How will villages invest in equipment and maintain cross-village communications networks?

- All stakeholders agreed that the Ministry of Public Works needs to be involved in this discussion as well, since they hold significant responsibilities in urban disasters (especially flooding) and infrastructure rebuilding.

Annex 7. Borrower's Completion Report In December 2014, the Borrower submitted a project completion report to the Bank. A summary of this report has been prepared and presented here. Final Report for PNPM Post-Disaster Recovery Prepared by the National Management Consultant on behalf of MOHA/PMD Background, objectives, design, and implementation In late 2010, Indonesia suffered a series of natural disasters, including an earthquake (7.7 on the Richter Scale) and tsunami in the Mentawai Islands off the coast of West Sumatra, and the eruption of Mt. Merapi in Central Java. Between these two events, 451 people lost their lives and an additional 252 went missing, and the communities in 18 sub-districts suffered from physical damages and displacement. The objective of the project is to build on the underlying objectives of the PNPM Rural program – improve rural welfare by providing opportunities and employment for poor households, and use an empowerment approach to strengthen participatory development – in disaster-affected locations. The Government of Indonesia (GoI) responded with assistance under a two-stage recovery process for affected areas: (i) an initial preparatory phase focusing on employment in general public works and clean-up programs organized by communities to facilitate the post-emergency return to home villages, followed by (ii) a reconstruction phase when communities assess damages and needs, prioritize projects, and organize recovery activities through supplemental PNPM block grants. A modified, post-disaster operations manual (PTO) developed as part of previous PNPM disaster projects was used in these areas. Project implementation follows familiar PNPM Rural procedures involving the community members, facilitators, and local government. The first phase took place during 2011 in the target districts of Central Java and Yogyakarta, and consisted of cash-for-work programs for groups of roughly 10-20 persons at the neighborhood level. Villages received Rp. 100-500 million depending on the degree of damage suffered. The second phase reconstruction projects were funded in 2012 under the same PTO and covered three provinces: Central Java, Yogyakarta and West Sumatra. The PTO instructed communities to prioritize the use of block grants for reconstruction and rehabilitation, disaster risk reduction, supporting infrastructure for relocation, and economic and social recovery. Disbursements The project disbursed IDR 30 billion for cash-for-work activities during 2011, followed by IDR 91 billion in reconstruction block grants in 2012 (totaling approximately US$13.4 million). The GoI requested that all funds remaining in the trust fund in 2014 be allocated to subsequent community recovery activities in twelve sub-districts in North Sumatra and East Java in the wake of volcanic eruptions there.

Outcomes against objectives Villages organized 2,709 cash-for-work activities which generated 715,000 work days during 2011. The central core of the project – the recovery projects funded in 2012 – consisted of a total of 781 projects (of which 723 were physical infrastructure) with 491,699 counted beneficiaries (50.4% women; 44.7% from poor households). The remainder of funds used to reimburse community activities in East Java and North Sumatra in 2014 supported an additional 155 projects with 99,496 beneficiaries (49.6% women; 38.7% from poor households). The detailed outcomes are those presented in section 3.2 in the main body of the report. Lessons Learned The role of Government • Bureaucratic delays in revising government budget allocations (revised DIPA) for the

DMS project were a problem. These resulted in delays to the DMS-funded projects, and because these are often linked to regular PNPM or other projects, the delays sometimes had an impact beyond DMS activities. The usefulness of the community planning process was reduced, and mentoring support was less effective.

• Weak support from local government, for example poor coordination by the district and sub-district coordination teams (Satker) was compounded by the lack of budget allocations that would have enabled local government workers to oversee rehabilitation projects.

Working in a post-disaster environment • Materials delivery was disrupted due to damaged transportation infrastructure. • Potential problems need to be anticipated. For example, much material resulting from

the eruption remains on the slopes above the villages, and there is the thread of cold ash floods when the rainy season begins. For communities that live alongside riverbeds, this could destroy rebuilt infrastructure, especially weirs and irrigation canals.

Working in remote areas • The quantity and quality of facilitation services was not adequate in Mentawai,

resulting from high turnover, frequent vacancies and extreme geographic conditions that require long travel times between villages and high transport costs.

• Extreme geographic conditions in the villages, and lack of working infrastructure. For example, broken access roads, inadequate transportation, fuel shortages, and inadequate-or-no communications infrastructure.

• Changeable weather conditions resulted in irregular delivery of construction materials when boats ceased to operate between the provincial capital (Padang) and the islands.

• Limited supply and high demand for all processed goods (including construction materials).

Annex 8. Comments of Co-financiers No comments received.

Annex 9. List of Supporting Documents • Project Appraisal Document. World Bank. June 2011. • Aide Memoires (various). World Bank. 2011-2014. • Implementation Status Reports (various). World Bank. 2011-2014. • Laporan Akhir PNPM Pasca Bencana, MOHA PMD and the National Management

Consultant, December 2014 • Laporan Akhir Studi Kecil Dampak PNPM Mandiri Perdesaan Pola Khusus Pasca

Bencana Wilayah Jawa Tengah, January 2015 • Hasil Penilaian Kualitas Prasarana di Provinsi Jawa Tengah dan D.I. Yougyakarta,

April-May 2013 • Economic Impact Analysis of Kecamatan Development Program Infrastructure

Projects, Anthony Torrens, January 2005. • Laporan Akhir Study Skala Kecil Analisis Manfaat Economi Projek Infrastructure

PNPM Mandiri Perdesaan, PNPM Support Facility, July 2012. • Economic Impact Analysis of Kecamatan Development Program Infrastructure

Projects. Anthony Torrens. January 2005. • Monthly Report December 2014, National Management Consultant • Country Partnership Strategy for Indonesia FY2009-2012. World Bank. August 2008. • Country Partnership Strategy for Indonesia 2013-2015. World Bank. December 2012. • PNPM-Rural Impact Evaluation. John Voss. PSF. April 2012. • Technical Evaluation of Infrastructure – PNPM-Rural & Other Funding Sources.

PNPM Support Facility. July 2012.

Annex 10. Photographs