document of the world bankdocuments.worldbank.org/curated/en/621301484078214331/... ·...

Document of The World Bank

Report No: ICR00003877

IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA-48530)

ON A

CREDIT

IN THE AMOUNT OF SDR 3.2 MILLION (US$ 5.0 MILLION EQUIVALENT)

TO THE

INDEPENDENT STATE OF PAPUA NEW GUINEA

FOR A

FLEXIBLE AND OPEN DISTANCE EDUCATION PROJECT

December 21, 2016

Education Global Practice East Asia Pacific Region

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

ii

CURRENCY EQUIVALENTS

(Exchange Rate Effective December 18, 2016)

Currency Unit = PNG Kina 1.00 = US$0.32

US$1.00 = Kina 3.18

FISCAL YEAR January 1 – December 31

ABBREVIATIONS AND ACRONYMS

AAC Academic Advisory Committee AusAID Australian Agency for International Development CAS Country Assistance Strategy CODE College of Distance Education COL Commonwealth of Learning CPS Country Partnership Strategy DA Designated Account DO Development Objective DoE Department of Education FM Financial Management FODE Flexible and Open Distance Education Project GC Governing Council GoPNG Government of Papua New Guinea GST Goods and Services Tax ICR Implementation Completion and Results Report ICT Information and Communication Technology IDA International Development Association IFR Interim Financial Report ISM Implementation Support Mission ISR Implementation Status Report IT Information Technology M&E Monitoring and Evaluation MTR Mid-term Review PAD Project Appraisal Document PD Personal Development PDO Project Development Objective PNG Papua New Guinea PSC Project Steering Committee READ Reading Education Project SRC Subject Review Committee SWOT Strength, Weakness, Opportunity and Threat

iii

UBE Universal Basic Education Plan (2010-2019) UNICEF United Nations Children’s Fund WB World Bank XDR Special Drawing Rights

Senior Global Practice Director: Amit Dar

Sector Manager: Harry Anthony Patrinos

Project Team Leader: Binh Thanh Vu

ICR Team Leader: Binh Thanh Vu

iv

PAPUA NEW GUINEA Flexible and Open Distance Education

CONTENTS

A. Basic Information ........................................................................................................... v

B. Key Dates ....................................................................................................................... v

C. Ratings Summary ........................................................................................................... v

D. Sector and Theme Codes............................................................................................... vi

E. Bank Staff ...................................................................................................................... vi

F. Results Framework Analysis ......................................................................................... vi

G. Ratings of Project Performance in ISRs ....................................................................... ix

H. Restructuring (if any) ..................................................................................................... x

I. Disbursement Profile ..................................................................................................... xi

1. Project Context, Development Objectives and Design ................................................... 1

2. Key Factors Affecting Implementation and Outcomes .................................................. 7

3. Assessment of Outcomes .............................................................................................. 17

4. Assessment of Risk to Development Outcome ............................................................. 23

5. Assessment of Bank and Borrower Performance ......................................................... 23

6. Lessons Learned............................................................................................................ 25

7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners ............... 26

Annex 1. Project Costs and Financing .............................................................................. 27

Annex 2. Outputs by Component...................................................................................... 28

Annex 3. Economic and Financial Analysis ..................................................................... 31

Annex 4. Bank Lending and Implementation Support/Supervision Processes ................. 34

Annex 5. Beneficiary Survey Results ............................................................................... 36

Annex 6. Stakeholder Workshop Report and Results ....................................................... 37

Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ......................... 38

Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ........................... 39

Annex 9. List of Supporting Documents .......................................................................... 40

MAP .................................................................................................................................. 41

v

A. Basic Information

Country: Papua New Guinea Project Name: Flexible and Open Distance Education Project

Project ID: P116521 L/C/TF Number(s): IDA-48530

ICR Date: 06/03/2016 ICR Type: Core ICR

Lending Instrument: SIL Borrower: GOVERNMENT OF PAPUA NEW GUINEA

Original Total Commitment:

XDR 3.20M Disbursed Amount: XDR 3.18M

Revised Amount: XDR 3.20M

Environmental Category: C

Implementing Agencies: Flexible and Open Distance Education

Cofinanciers and Other External Partners: B. Key Dates

Process Date Process Original Date Revised / Actual

Date(s)

Concept Review: 06/03/2009 Effectiveness: 11/03/2011 11/03/2011

Appraisal: 03/16/2010 Restructuring(s): 03/06/2015

Approval: 01/11/2011 Mid-term Review: 04/07/2014 04/07/2014

Closing: 06/30/2016 06/30/2016 C. Ratings Summary C.1 Performance Rating by ICR

Outcomes: Moderately Unsatisfactory

Risk to Development Outcome: High

Bank Performance: Moderately Unsatisfactory

Borrower Performance: Moderately Unsatisfactory

C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings

Quality at Entry: Moderately Unsatisfactory

Government: Moderately Unsatisfactory

Quality of Supervision: Satisfactory Implementing Agency/Agencies:

Moderately Satisfactory

Overall Bank Performance:

Moderately Unsatisfactory

Overall Borrower Performance:


vi

C.3 Quality at Entry and Implementation Performance IndicatorsImplementation

Performance Indicators

QAG Assessments (if any)

Rating

Potential Problem Project at any time (Yes/No):

Yes Quality at Entry (QEA):

None

Problem Project at any time (Yes/No):

Yes Quality of Supervision (QSA):

None

DO rating before Closing/Inactive status:


D. Sector and Theme Codes

Original Actual

Sector Code (as % of total Bank financing)

Adult literacy/non-formal education 55 65

Information technology 11 15

Public administration- Education 19 10

Secondary education 15 10

Theme Code (as % of total Bank financing)

Education for all 100 100 E. Bank Staff

Positions At ICR At Approval

Vice President: Victoria Kwakwa James W. Adams

Country Director: Michel Kerf Ferid Belhaj

Practice Manager/Manager:

Harry Patrinos Eduardo Velez Bustillo

Project Team Leader: Binh Thanh Vu James A. Stephens

ICR Team Leader: Binh Thanh Vu

ICR Primary Author: Sandra Beemer, Kris McDonall F. Results Framework Analysis

Project Development Objectives (from Project Appraisal Document)

The development objective of this project is to increase the number of out-of-school youth who complete recognized secondary diploma and certificate equivalency programs.

vii

Revised Project Development Objectives (as approved by original approving authority) The revised project development objectives are: (i) to align the distance education curriculum with the national curriculum; and (ii) to strengthen FODE’s information management system. (a) PDO Indicator(s)

Indicator Baseline Value

Original Target Values (from

approval documents)

Formally Revised Target Values

Actual Value Achieved at

Completion or Target Years

Indicator 1: Aligned subject material for Grades 7-12 approved by Governing Council Value quantitative or qualitative)

0 All subjects of

grades 7-12

N/A 26

Date achieved 11/20/2010 6/30/2016 6/30/2016 Comments (incl. % achievement)

Target met. A total of 26 subjects were approved by the FODE governing council. Indicator added during the 2015 restructuring. Source: FODE

Indicator 2: Data available online and used for planning Value quantitative or qualitative)

No Yes N/A Yes

Date achieved 11/20/2010 6/30/2016 6/30/2016

Comments (incl. % achievement)

Target partially met. A new FODE internet-based data system was uploaded and provincial FODE centers training completed prior to project closing but too late to determine if used for planning. Indicator added during the 2015 restructuring. Source: FODE

Indicator 3: The absolute numbers of full time correspondence students who complete certificates (Grades 8, 10 and 12).

Value quantitative or qualitative)

Grade 8: 9 Grade 10: 226 Grade 12: 1

Grade 8: 150 Grade 10: 2,220 Grade 12: 180

N/A N/A


This indicator was dropped during the 2015 restructuring because data was not available.

viii

(b) Intermediate Outcome Indicator(s)

Indicator Baseline Value

Original Target Values (from

approval documents)

Formally Revised

Target Values

Actual Value Achieved at

Completion or Target Years

Indicator 1: New learning materials for all subjects for Grades 7-10 trialed and ready for printing

Value (quantitative or qualitative)

0 28 subjects for

G7-10 (7 subjects per grade)

N/A 26


Target met. The original target of 28 was a miscalculation because two grade 11 and 12 mathematics modules were counted as two subjects. Indicator added during the 2015 restructuring. Source: FODE

Indicator 2: Module writing for Grades 11-12 subjects completed and ready for trialing Value (quantitative or qualitative)

0 26 subjects for

G11-12 N/A 26


Target met. All grade 11 and 12 subjects were completed and approved by the governing council. Indicator added during the 2015 restructuring. Source: FODE

Indicator 3: Roll-out strategy for aligned distance learning curriculum developed and approved


No Final strategy

approved N/A

Final strategy approved by the

Governing Council

Date achieved 2/12/2015 6/30/2016 6/30/2016


Target met. Strategy was written, reviewed and approved by Bank and Governing Council and was presented nationally at November 2016 FODE New Curriculum Launch workshop. Indicator added during the 2015 restructuring. Source: FODE data

Indicator 4: Network connectivity established between FODE Headquarters and at least 10 provincial centers


0 10 N/A 21


Target exceeded by 110 percent increase. Indicator added during the 2015 restructuring. Source: FODE data

Indicator 5: FODE management information system upgraded and rendered functional Value (quantitative or qualitative)

No Yes N/A Yes

Date achieved 12/6/2010 6/30/2016 6/30/2016

ix


Target met. The system was modified and has been uploaded on the internet. Indicator was added at the time of the 2015 restructuring. Source: FODE data

Indicator 6: Harmonized annual project plans developed and submitted for Bank approval no later than January 15 of each year


No Yes N/A Yes


Target met. FODE management provided annual project plans as required. Indicator added during the 2015 restructuring. Source: FODE

Indicator 7: Completion Rates Value (quantitative or qualitative)

16 28 N/A N/A


Indicator was dropped during the 2015 restructuring because data was not available.

Indicator 8: Share of annual grades 8, 10, and 12 full time correspondence certificate completers compared to new enrollments for these certificate programs



Indicator was dropped during the 2015 restructuring because FODE paper based data system could not produce the data.

Indicator 9: Share of full correspondence students on annual subject level exams for grades 8, 10, and 12 Year 2009



Indicator was dropped during the 2015 restructuring because FODE paper based data system could not produce the data.

G. Ratings of Project Performance in ISRs

No. Date ISR Archived

DO IP Actual

Disbursements (USD millions)

1 06/03/2011 Satisfactory Satisfactory 0.00 2 01/24/2012 Satisfactory Satisfactory 0.70

3 07/18/2012 Moderately

Unsatisfactory Moderately

Unsatisfactory 0.70

x



Unsatisfactory 0.70


Unsatisfactory Moderately Satisfactory 1.22











11 07/07/2015 Moderately Satisfactory Moderately Satisfactory 2.86 12 12/31/2015 Moderately Satisfactory Moderately Satisfactory 3.80



Unsatisfactory 4.46

H. Restructuring (if any)

Restructuring Date(s)

Board Approved

PDO Change

ISR Ratings at Restructuring

Amount Disbursed at

Restructuring in USD millions

Reason for Restructuring & Key Changes Made

DO IP

03/06/2015 02/15/2015 MU MS 2.52

PDO simplification to better reflect intended outcome of project; indicators revised to be better aligned with the modified PDO. Simplification of components to align with modified PDO.

xi

I. Disbursement Profile

1

1. Project Context, Development Objectives and Design

1.1 Context at Appraisal 1. Country Context. Papua New Guinea (PNG) is an ethnically and socially diverse nation of seven million people speaking over 840 languages and is spread across some of the world’s most difficult terrain. PNG was, and continues to be, rich in natural resources, but subject to extreme weather events and natural disasters with many remote and difficult to reach areas. An estimated 88 percent of the population lives in rural areas, often isolated with poor communications. The agriculture and oil/minerals sectors dominated the economy, which led to markedly improved economic fortunes since the mid-2000s. The global commodity price boom and stronger macroeconomic management helped the country avoid the declining real incomes and macroeconomic volatility that the country had experienced in the past. The global downturn of 2009 had only a mild impact on PNG’s economy, which expanded by 5.5 percent, outperforming its regional peers and most other developing economies. This followed several years of 6 to 7 percent annual Gross Domestic Product (GDP) growth. While sharply higher global commodity prices strongly supported incomes (average export prices doubled between 2005 and 2008, and were still about 50 percent higher in mid-2012), the non-minerals sector led the resurgence, benefiting from private investment in future minerals production capacity as well as in the newly liberalized telecommunications and aviation sectors. However, even after several years’ of broader growth, PNG’s economy remained dominated by two sectors: (i) the agricultural sector, which engages most of the labor force; and (ii) minerals and energy extraction, which earns the largest share of export receipts. It should be noted that this is still the case in PNG. In 2014, the mining and petroleum sectors were estimated to account for about a quarter of GDP, 83 percent of exports and around 7 percent of total employment. In contrast, 87 percent of Papua New Guineans live in rural communities, engaging in traditional subsistence and semi-subsistence agriculture in the informal sector. At the time of appraisal, growth in demand was outpacing that of productive capacity, and inflationary pressures built and became increasingly domestically focused. PNG’s consumers were buffeted by the global energy and food price rises of 2007 and 2008, which lifted year-on-year urban average inflation above 13 percent and urban food price inflation to above 20 percent. The subsequent retreat in global prices slowed local price growth, but only to around 5 percent year on year. As domestic demand pressures built through 2011, reported inflation approached 10 percent. 2. Sector Context. The Government of PNG (GoPNG)’s National Education Plan 2005-2014 (NEP) appropriately gave priority to increasing access, retention and quality in basic education. This strategic document was complemented by the development of a Universal Basic Education (UBE) Plan, which was adopted by GoPNG in late 2009. The Department of Education’s (DoE) efforts to address the country’s needs for improving basic education were being supported by donors. The NEP also identified flexible and open distance education as a priority element of development. Second chance education opportunities and alternative pathways to developing general education skills were considered a priority in PNG because of problems with access and retention in the formal

2

system. At appraisal, up to 50 percent of school age children in PNG were not in school, increasing to as much as 90 percent when looking solely at the secondary level. The majority of out-of-school youth almost certainly lacked the publically-recognized certification that would have allowed them to qualify for higher levels of education or to meet the minimum educational requirements of some employers. The country’s capacity to afford and deliver formal secondary education was and continues to be limited, and GoPNG only plans to slowly expand access. 3. The College for Flexible and Open Distance Education (FODE). FODE has operated as a semi-autonomous agency under the DoE, enrolling out-of-school youth in self-study programs and certifying successful candidates for grade 8 and 10 certificates. Although the DoE is the designated government authority over FODE, it is coordinated through a long established Governing Council (GC). The GC has a broad range of representation from stakeholders who have an interest in education. The GC was originally established as a governing authority over the College of Distance Education (CODE), before it was transformed into the new FODE in 2009. Since that time, the structure has been maintained and it represents an important element in the development and implementation of policy and procedures reform for the organization. The principal and his management team serve as the secretariat for the GC. At the time of appraisal, FODE had national responsibilities for the provision of flexible learning programs and distance education at the upper primary (Grades 7 and 8) and secondary (Grades 9 and 10) levels, and was working on expanding a small matriculation program for upper secondary students (Grades 11 and 12). It provided courses in four subject areas, English, Mathematics, Science and Social Science, primarily delivered through correspondence-based distance learning, supported occasionally by on-site tutors and teachers. As the only public distance learning institution serving PNG’s upper primary and secondary school leavers, FODE oversaw 22 provincial centers that coordinated services provided within their respective provinces, as well as about 63 affiliated study centers and 48 affiliated correspondence centers (2008). Its 2009 budget was about PGK3.8 million (US$1.5 million), about 40 percent of which was covered by registration fees. 4. Although the institution had significant potential to support a relatively large, under-served and growing segment of out-of-school youth, it had suffered for many years from inadequate resources and technical support. At the same time, demand for FODE’s programs continued to rise. In 2009, GoPNG increased FODE’s operational subsidy by about PGK1 million – roughly doubling FODE’s non-salary operating budget – allowing FODE to greatly expand its programs. In 2008, 5,484 new students enrolled in 11,165 courses (the average student enrolled in slightly more than two subject courses). Success rates, however, remained low with 684 course completion certificates issued in 2009. 5. While many external donors in PNG were supporting formal education programs, PNG was benefiting from few donor investments aimed at building capacity to provide second chances for the large stock of out-of-school youth who had been failed by the formal system. FODE was PNG’s largest provider of second chance education opportunities. Funding for FODE’s programs was assessed by the Bank to be inadequate in two ways. First, FODE lacked the operational funding to keep up with the expanding demand for its

3

programs and to ensure quality service delivery. Second, FODE lacked the development funding to invest in better quality curriculum, instructional materials and tutor skills. 6. The Bank was giving priority throughout the Pacific sub-region to programs aimed at out-of-school youth, consistent with the World Bank’s Education Sector Strategy 2020. The PNG Country Assistance Strategy (CAS) 2008-2011 was designed to be delivered on two platforms, supporting policy and institutional development at the national level while working to improve planning and service delivery at the local level. One of the mainstreamed focuses of the CAS was given to addressing problems related to youth underemployment and exclusion. The FODE project would provide International Development Association (IDA) funding for FODE in the amount of SDR 3.2 million or US$5 million equivalent over five years (2011-2015), allowing FODE to expand its coverage of out-of-school youth and improve the quality of its programs and service delivery. 1.2 Original Project Development Objectives (PDO) and Key Indicators 7. The original project development objective was to increase the number of out-of-school youth who complete recognized secondary diploma and certificate equivalency programs. The principal performance indicator was the absolute numbers of youth who acquire a secondary equivalency certificate through FODE. Intermediate outcome indicators were measured using various enrolment and completion figures across different subjects and grades.

1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 8. As a part of the February 12, 2015 project restructuring, the PDO was revised to: (i) align the distance education curriculum with the national curriculum; and (ii) strengthen FODE's information management system. Due to the data collection issues and weak capacity faced by FODE, and to better reflect the revised PDO, both outcome and intermediate indicators were dropped and a new set of indicators was developed as follows: Table 1: History of Project Indicators

Original PDO Indicator Adjustments at the time of February 2015 Restructuring

The absolute numbers of full time correspondence students who complete certificates (Grades 8, 10 and 12).

Dropped during restructuring.

Aligned subject material for Grades 7-12 approved by Governing Council. Added at the time of restructuring.

Data available online and used for planning and budgeting purposes. Added at the time of restructuring.

4

Intermediate Indicators

Share of annual grade 8, 10 and 12 full time correspondence certificate completers compared to new enrollments for these certificate programs.

Dropped during restructuring

Share of full correspondence students on annual Subject-level exams for Grades 8, 10 and 12, Year 2009.

Dropped during restructuring

New learning materials for all subjects for Grades 7 - 10 trialed and ready for printing.

Module writing for Grades 11 – 12 subjects completed and ready for trialing.

Roll-out strategy for aligned distance learning curriculum developed and approved.

Network connectivity established between FODE HQ & at least 10 PCs.

FODE management information system upgraded and rendered functional.

Harmonized annual project plans developed and submitted for Bank approval no later than January 15 of each year.

1.4 Main Beneficiaries, 9. The primary beneficiaries for the FODE project were the 13,7541 students that access FODE courses across the country in 2015. With available data indicating enrolments are on the rise and the completion of the new equivalency curriculum in place, it is estimated that new enrolments will increase over the next five years, meaning many more of the 90 percent of PNG’s youth without access to secondary schools will benefit from the FODE project. Additional beneficiaries of the project were the 85 curriculum content writers and editors who have had increased capacity from their involvement in the development of the new curricula. In addition, the DoE Information Technology (IT) Division and FODE centers benefited from access to a stronger internet connection in all 22 provinces than previously existed, and provincial FODE staff have learned how to use IT to enter information and use the upgraded database.

1.5 Original Components 10. Component 1 – Updating of Curriculum and Instructional Materials (US$1.5 million equivalent-IDA). This component aimed to improve the quality of distance learning curricula and the quality and availability of instructional materials. Upgrading of the FODE curricula was made a pre-requisite by DoE for recognition of equivalency for FODE graduates. The project intended to support the development of new and/or upgrading of existing curricula for grades 7 through 12 using FODE’s standardized distance learning

1 The PAD originally indicated that 48,328 students had enrolled in FODE in 2008 in 95,049 subjects; these figures were based on minutes from FODE’s governing council. In 2015, based on numbers from the new data management system, enrollment was 13,754 students in 34,699 subjects.

5

format. This component also aimed at upgrading FODE’s printing and distribution system. The project intended to fund implementation of improvements with a view toward: (a) improving the accessibility of instructional materials; (b) improving the quality of documents produced; and (c) lowering of production and distribution costs. 11. Component 2 - Improving the Quality of Service Delivery (US$0.7 million equivalent IDA—US$1.0 million-counterpart). Component 2 aimed to improve the overall quality of services provided by FODE. Within the 22 FODE provincial centers, interventions were to be focused on improving performance through staff training, simplifications of forms and procedures, establishing and implementing an IT plan, and selecting and engaging contracted subject tutors in the provincial centers. Other subcomponents were to support improvements of services at regional study centers. The project also included the renovation of the FODE central headquarters, as well as renovation and expansion of provincial centers. All works in provincial centers were to be financed by GoPNG’s contribution to the project. 12. Component 3 – Grade 11-12 Matriculation Program Expansion (US$1.8 million equivalent). This component aimed to expand FODE’s existing grade 11-12 matriculation program into at least one secondary school in each of PNG’s 22 provinces. The component was to build on lessons learned in demonstration pilots that had previously been carried out, and demonstrate the viability of providing grade 11-12 equivalency programs in existing secondary schools. Project support would reimburse secondary schools for the use of their facilities, provide classrooms with small equipment and instructional materials, and cover the cost of after-hours teaching by existing subject teachers in each school. Further expansion of the program, potentially supported by provincial resources, would depend on matriculation success rates. 13. Component 4 – Improving Information Technology, Monitoring and Evaluation (US$ 1.0 million equivalent). Component 4 aimed to provide FODE with appropriate IT resources and train FODE staff to effectively operate that technology. It was to provide funding for an IT needs assessment, establishment of an IT plan for FODE, procurement of IT equipment, and implementation of the plan to fully integrate the use of IT into the administration, curriculum development, materials production, and teaching/learning activities conducted by the organization. The component also planned to support several pilots and studies aimed at demonstrating potential low-cost technologies to improve service delivery, as well as evaluating the effectiveness of project innovations. If proven successful, FODE could move to partially or fully implement new technologies within its pedagogical framework. The component would also prepare and implement studies that would be used to determine the effectiveness of project outcomes during and at the end of the project. These evaluations included a: (i) pilot on the use of ‘radio station in a box’; (ii) pilot on the use of one-way educational TV for delivery of program services to provincial centers; (iii) pilot in two provincial centers for IT-based instruction; (iv) baseline study based on the socio-economic status of FODE students; and (v) tracer studies on program leavers and program graduates to be updated at project mid-term and project completion. The project planned to provide about 6 months of international consultant support for these studies.

6

1.6 Revised Components 14. During the February 2015 restructuring, several revisions were made to the project components recognizing the limited capacity constraints of FODE to carry out the project activities as originally designed. The revised Component 1, "Curriculum Development", would support: (i) curriculum writing/module development for all Grades 7 – 12 subjects, including printing of new material and acquisition of curriculum development reference materials; (ii) the alignment of the revised curriculum with the national curriculum and the establishment of revised assessment rubrics for scoring and marking; (iii) trialing of the new curriculum; and (iv) the development of a roll-out strategy. The revised Component 2, "Information Management Infrastructure", would support: (i) the establishment of network connectivity between provincial FODE centers and headquarters, and aligning FODE information and communication technology (ICT) with the wider DoE ICT policy; and (ii) the remodeling/upgrading of FODE's existing Student Information System and provision of related training in using internet, data collection and data entry. 15. The original Component 3 and Component 4 were dropped during the restructuring because: (i) universities and colleges in PNG no longer recognize matriculation program certificates, rendering original Component 3 activities obsolete; and (ii) most IT activities under the original Component 4 were either included under the revised Component 2 or were no longer appropriate given FODE’s monitoring and reporting capacity and overall project implementation delays. A new Component 3, "Project management, monitoring and evaluation", was developed to support: (i) project management and day-to-day operations, including activity planning and budgeting, implementation monitoring and progress reporting; (ii) financial management and procurement activities, including reporting and audits; and (iii) monitoring and evaluation activities including gathering information and updating the results framework. 1.7 Other significant changes 16. The Office of the Minister for Treasury sent a letter on August 12, 2014 to the Bank requesting a restructuring of the project based on findings and recommendations from the April 2014 implementation support mission (ISM). Between April 2014 and March 6, 2015 and prior to Bank Board approval, the project unit had to ensure that ineligible taxes (GST) expenditures were refunded to the project Designated Account (DA) in accordance with GoPNG law and as per the financing agreement. DoE budget delays slowed the availability of funds to FODE which, in turn, limited FODE’s ability to refund the GST. This created an eleven month delay in the overall project restructuring. The Bank Board approved the restructuring on March 6, 2015. The Level 1 restructuring of the project included: (i) a revision of the PDO; (ii) simplified overall project design, reducing components from four to three and focusing on activities that support FODE management's priority areas; and (iii) modifications of the results framework to support measurement of the new PDO and establish a clear link between project activities and the results framework. The restructuring also modified the implementing agency risk assessment from "Modest" to "Substantial" for

7

FODE staff capacity and reporting compliance. This was based on challenges experienced during implementation and reflected the aim of the restructuring proposal for a simplified project design and additional technical assistance. The revision of components under the project restructuring meant a revision of funding allocations across components was also required, without changing the overall budget envelope or project timeline. Table 2: Revised project component costs

Original Component Name

Revised Component Name

Original Cost (US$M)

Revised Cost (US$M)

Action

Updating of Curriculum and Instructional Materials

Curriculum Development

1.5 1.80 Revised

Improving the Quality of Service Delivery

Information Management Infrastructure

0.7 2.52 Revised

Grade 11-12 Matriculation Program Expansion

1.8 0.00 Marked for Deletion

Improving Information Technology, Monitoring and Evaluation

1.0 0.00 Incorporated into component 2.

Project management, monitoring and evaluation

0.68 New

Total: 5.00 5.00

2. Key Factors Affecting Implementation and Outcomes

2.1 Project Preparation, Design and Quality at Entry 17. Project Preparation. In 2010, the FODE Project was prepared as a Specific Investment Loan (SIL) given the targeted nature of program interventions. The FODE project also marked a period of the Bank’s re-engagement with PNG during which the GoPNG, for the first time in more than a decade, borrowed for an investment credit in education. The project was consistent with pillar 2 of the Bank’s CAS (2008-2011) for PNG that aimed to enhance rural livelihoods, and improve local service delivery as well as the UBE focus on the need to address the problem of out-of-school youth. The preparation team included project activities based primarily on the March 2006 extensive review and analysis of FODE that was supported by the Australian Department of Foreign Affairs and

8

Trade (known as AusAID at the time).2 The preparation team also took lessons from other countries,3 which had implemented similar types of projects, which contributed to assisting the team in preparing and appraising the project. The Bank preparation and appraisal teams consisted of technical experts that were appropriate for the development of the program. The team consulted with peer reviewers that had knowledge and experience in developing and implementing second chance education and equivalency programs and provided guidance on the project design. Overall project preparation was completed as originally scheduled. The project was approved by the Bank Board of Executive Directors on January 11, 2011. 18. Project Design and Quality at Entry. The project design was aligned with the government’s UBE Plan (2010-2019), which provided the platform for sectoral planning and financing, as well as the CAS (2008-2011). The project design included a quality review by peer reviewers at the concept and appraisal stage with comments that included: (i) endorsement of the ‘simple’ project design, PDO, and PDO indicator; (ii) ensuring that baselines for performance indicators were included during appraisal; (iii) recognition that the project was supporting general education as opposed to technical and vocational programs and therefore project success would not be determined by employment outcomes; (iv) use of local systems for implementation except in the case of procurement and financial management which would use Bank approved systems; (v) strengthening fiduciary risk mitigation measures; and (vi) provision of project resources to support a well-designed tracer survey that would include a comparator group. 19. The original project design included four components with 12 sub-components. These components and sub-components were focused on: (i) improving the quality of distance learning materials, that had not been updated since the 1980s; (ii) improving the quality of service delivery at the FODE central campus and 20 provincial centers through staff training on curriculum and instructional materials; (iii) including a grade 11-12 matriculation program in at least one secondary school in each of PNG’s 20 provinces because FODE did not have the necessary qualified teachers or equipment to provide a grade 11-12 program; and (iv) improving the IT infrastructure/systems within the central and provincial FODE centers to fully integrate the use of IT into the administration, curriculum development, material production, and teaching and learning activities conducted by FODE. The design also included five pilots and/or studies. These original activities were recognized as necessary for improving the FODE delivery system as well as the overall goal of the project which was to increase the number of out-of-school youth who completed secondary diploma and equivalency programs. The team used the re-

2 The study was conducted by a research team that conducted field visits to provinces and held workshops. Data was collected from FODE coordinators, provincial education authorities, and students using several survey forms. The issues raised by the study showed that: (i) statistics and record keeping was unreliable; (ii) there were operating grant shortfalls; (iii) there was inadequate income from student fees; (iv) there was poor student achievement; and (v) there was an inadequate supply of teaching materials. 3 The project lessons were drawn from similar projects which included, but were not limited to, second chance and equivalency programs in Argentina, Chile, Mexico, the Philippines, and the Dominican Republic.

9

engagement period to mainstream project implementation within FODE so that capacity could be developed thereby sustaining the investments made in curriculum development and IT improvements. The design also included use of a project steering committee (PSC) to provide overall policy guidance and oversee project planning, organization administration, implementation, and monitoring of the components. The design identified risks associated with: (i) overall FODE institutional capacity; (ii) limited ability to produce information in a timely manner; (iii) limited experience and knowledge of Bank procurement procedures; and (iv) limited financial management experience that could delay project accounting and audits. The mitigation measures included use of national and international consultants to support FODE thereby reducing the overall operational project risk to moderate. 20. The original PDO was chosen (in part) based on an assumption that project activities, especially the new curriculum, would begin quickly and produce early project results that could be measured. However, the design did not properly anticipate the time needed or the capacity requirements to: (i) align the FODE curriculum with the formal system and to produce a complete equivalency curriculum with self-paced materials for six grades (grades 7-12) with a combined total of 63 subjects; and (ii) ensure the necessary connectivity to deliver course materials to more students. These two areas accounted for approximately 64 percent of overall project costs. The underestimation of time also meant that there was an underestimation of the funds needed to complete the curriculum development. The capacity risks were identified with mitigation measures that included the use of consultants to support the FODE teams. However, the pool of local consultants with the needed skills was limited and underestimated. The original PDO indicator which was to measure an increase in the absolute number of the secondary equivalency students could not be collected due to: (i) a cumbersome paper data collection process for the FODE information system; (ii) lack of an actual data point related to the indicator; and (iii) a database system that could not adequately extract information. In addition, FODE project staff lacked capacity to: (i) provide implementation updates; (ii) produce financial statements; (iii) conduct procurement activities based on Bank procurement systems; or (iv) begin all the project studies. Consultants that were hired early by the project were not able to help FODE staff jump start project implementation effectively so there were considerable implementation delays. Risks were identified correctly but the team underestimated the needed mitigation measures and thereby underestimated the overall risk level for the project. The underestimation of the risks associated with country capacity led to implementation delays in curriculum development, procurement and financial management. While the Bank preparation team consisted of experienced technical experts, in retrospect, the project preparation would have benefited from a thorough risk and capacity analysis considering this was the first Bank-supported operation in the education sector in PNG in more than a decade. Moreover, the ‘simple’ design, as agreed by reviewers, underestimated the time required to update learning materials particularly in light of the limited in-country capacity. 21. The 2015 Level I project restructuring, which was initiated in 2014, provided the opportunity to enhance the potential effectiveness of the project design by revising the PDO to ensure that the objectives could be measured and achieved. At this time, a

10

concerted effort was made to ensure that: (i) the revised indicators supported measurement of PDO achievement; (ii) there was a clear link between project activities and the results framework; (iii) relevant data existed or could be collected by FODE to assess and inform project performance; and (iv) project resources were reallocated for the completion of the curriculum and information management infrastructure. The intention during restructuring was that improved data gathering and reporting against the revised results framework would also be facilitated by focusing the interventions and project activities mainly on the completion of the equivalency curriculum and ensuring connectivity for the ultimate delivery of the new curriculum, which, as indicated above, accounted for approximately 64 percent of overall project costs and are detailed in sections 1.6 and 1.7. These modifications reduced the number of activities to a level that was commensurate with the local capacity and the risk level was increased to a substantial level, which was more appropriate considering the overall capacity constraints. These modifications are judged highly appropriate and far more realistic for achieving the revised project goals and design. 2.2 Implementation 22. Exogenous factors impacting implementation. There were exogenous factors that impacted project implementation, which were: (i) security concerns due to unemployment and lawlessness throughout the country that limited mobility of FODE teams to provide internet connectivity for FODE centers within the country and contributed to increases in operational costs due to security teams needed when traveling; (ii) inflationary factors between 2010 and 2013 related to increases in global commodity prices that impacted the project restructuring because government budget was not available for the re-payment of the taxes (GST); and (iii) a 2014-2016 downturn in country macroeconomic conditions caused by lower global commodity prices, which will impact the sustainability of the project. By 2015, some of these factors had led to PNG being defined as a fragile state4 with serious national budget challenges for continuing support for services throughout the country. 23. Implementation details. The FODE project got off to a slow start due to a seven month delay between the signing of the credit and effectiveness. The project was formally launched on April 19, 2011 but the credit did not become effective until November 3, 2011 because the project implementation unit had difficulty hiring a financial management specialist and an accountant, which were conditions of effectiveness. Although not yet

4 The 2015 OECD State of Fragility Report uses a new take on defining fragility, building off the sustainable development goal (SDG) 16, which is to promote peaceful and inclusive societies for sustainable development, provide access to justice for all and build effective, accountable and inclusive institutions at all levels. The model used for defining fragility uses five dimensions of fragility: violence, resilience, justice, economic foundations and institutions. Three of these indicators are related to targets of SDG 16 and two are from the wider SDG framework: violence, access to justice, accountable and inclusive institutions, economic inclusion and stability, and capacities to prevent and adapt to social, economic and environmental shocks and disasters. This useful model breaks down the drivers of fragility for each country and reveals different patterns of vulnerability, suggesting that fragility is not restricted to a few countries. Countries vulnerable across all five dimensions of fragility are most likely to be identified on fragile states lists.

11

effective, the implementation agency had begun the selection of approximately 12 national and five international consultants during 2011. In addition, GoPNG agreed to increase the counterpart funding for FODE so that all civil works for both the FODE central offices and provincial offices would be covered by government counterpart rather than the IDA credit. It was agreed at this time, that the US$500,000 of project funds that were earmarked for works would be re-allocated to other project activities. By January 2012, it was already obvious that FODE management was having difficulty managing the project and was slow to identify needed TA support. Moreover, it was increasingly evident that the fiduciary arrangements were problematic for FODE because procurement and financial management consultants were being shared by FODE and the READ PNG projects, and were not being managed effectively. 24. In March 2012, four months after effectiveness, the project development objective and implementation progress ratings were downgraded from satisfactory to moderately unsatisfactory (MU). This was due to the fact that the FODE unit had not been able to provide data on progress of the PDO indicator. In addition, there had been long delays in recruiting needed consultants for all aspects of implementation, which resulted in lack of supervision over the project and no implementation progress. The extension and refurbishment of the FODE office space needed to accommodate all the consultants to be hired was not moving forward because the DoE budget section had not transferred the PGK1 million counterpart funding, released by the Department of Finance and Department Treasury in 2011, from the main DoE account to the FODE project account.

25. By November 2012 there was positive movement with project implementation as follows: (i) five national curriculum content advisors and course editor consultants in science, English, mathematics, and social science had been hired; (ii) workshops on curriculum development had taken place; (iii) agreements had been reached on completing the revision of all subjects for grades 7 and 8 by early 2013; (iv) a training needs assessment had been completed; and (v) guidelines agreed on for selection of pilot schools for the matriculation program. Financial management had improved somewhat because the financial management manual had been approved by the Bank and the accounting software was scheduled for installation. However, the original international financial advisor had been replaced due to unsatisfactory performance and a new advisor was expected to be hired by December 2012. It was at this time that the Bank agreed that the FODE and Reading Education (READ PNG) 5 projects would have their own full-time national financial management and procurement consultants and they would share an international consultant in each area. 26. Between March 2012 and November 2013, the Bank team conducted approximately eight technical support missions to provide additional support to the FODE team as they began to implement the project, as well as three Implementation Support Missions (ISM) that reported on implementation progress. During this period the implementation progress (IP) rating improved from moderately unsatisfactory (MU) to

5 The READ PNG project and FODE project were prepared at the same time and it was originally agreed that the two projects would share a national financial management consultant and procurement consultant.

12

moderately satisfactory (MS) because: (i) several national key consultant positions had been filled: project management advisory, distance learning and pedagogical advisor, information technology advisor, and curriculum content advisor; (ii) international consultants to support project management and financial management were being finalized; (iii) counterpart funds had been allocated in the amount of PGK4 million for rehabilitation of FODE training centers; (iv) national curriculum content advisors and course editors had been hired and had been able to complete the development of Grades 7 and 8 materials, received6 GC approval, and were ready for printing; (v) development of Grade 9 materials had begun; (vi) a review of the current IT practices had been conducted which included a Strength, Weakness, Opportunity and Threat (SWOT) analysis with recommendations for improvement covering all aspects of FODE programs and services; and (vii) tutor manuals for the matriculation program had been drafted and many schools with grades 11 and 12 had expressed interest in participating in this program. Financial management was upgraded at this time from MU to MS only to be downgraded to MU in November 2013 as a result of: (i) large values of outstanding and unaccounted advances; (ii) large values of unaccounted funds; (iii) paying Goods and Services Tax (GST) with project funds (financing was exclusive of taxes); (iv) lack of reliable approved annual budget plans; (v) an inactive designated account (DA); and (vi) overdue audit reports. 27. The November 2013 ISM was the first time the Bank and FODE teams began discussing a project restructuring. It was agreed that a restructuring was needed because: (i) curriculum development had been delayed due to the fact that DoE had moved to an outcomes based curriculum, which led to uncertainty for FODE as to how to develop and align the 40 year old curriculum with the new outcomes based formal curriculum; (ii) FODE curriculum officers had very limited experience in developing curriculum for the FODE delivery mode since there had been no changes in 40 years; and (iii) there were logistical challenges for coordinating the large number of consultants needed to develop the materials, as well as the long review and approval process by the Subject Review Committee (SRC), Academic Advisory Committee (AAC) and GC. The DoE was also requesting that Component 3: Matriculation Program Expansion be canceled because: (i) the matriculation program was no longer being recognized by colleges and universities; (ii) FODE was to deliver grades 11 and 12 thereby eliminating the need for a matriculation program; and (iii) FODE could not sustain the program. Given these implementation realities the teams developed: (i) the revised PDO; (ii) a revised results framework to align with the new PDO; and (iii) a restructured project design taking the above constraints into consideration. The development of the restructured design was facilitated by the Bank through the organization of a 2-day workshop given by the Commonwealth of Learning (COL)7 that focused on FODE’s needs and a forward-looking strategy for FODE. This was a timely and highly relevant workshop.

6 The GC is the quality control body within the DoE that reviews and approves all new subject material prior to the printing and distribution of the materials. 7 COL is an intergovernmental organization of the Commonwealth of Nations, headquartered in Vancouver, British Columbia, Canada. The COL's mandate is to promote and develop the use of open learning and distance education knowledge, resources and technologies. COL has evolved into one of the world's leading distance education providers and leads the international development in distance education along with

13

28. The April 2014 mid-term review (MTR) confirmed that the FODE project remained highly relevant with its support for the development of an equivalency program for out-of-school youth, but that a restructuring was needed to ensure the project could reach the desired development objectives. During the MTR, the Bank team continued to work with FODE on the restructuring and by August 12, 2014 the government sent the official letter to the Bank requesting the project restructuring. However, the restructuring could not be sent formally to Bank management until February 2015 because of outstanding financial management issues. In February 2015, the required financial management actions needed for restructuring had been completed. This included refunding ineligible expenditures related to taxes (GST) that could not be paid from the Designated Account (DA) in accordance with GoPNG law and as per the financing agreement. The restructuring was formalized in March 2015. However, the FODE and Bank teams were already monitoring implementation based on the restructured project design agreed in the November 2013 and April 2014 MTR missions. At this point, approximately 46 percent of the project funds had been disbursed. It was also at this time that financial management was upgraded from MU to MS. Additional upgrades in the project were: (i) project management was upgraded from MU to MS due to improved responsiveness and proactivity; and (ii) counterpart funding was upgraded from MU to satisfactory (S) because counterpart funds had been received in the project DA. 29. The post-restructuring period, March 2015 until project closing on June 30, 2016, experienced a marked improvement in implementation based on the restructured design. It should be noted that, while there was considerable improvement in implementation, the DO rating was downgraded from MS to MU during this period because of concerns that FODE would not be able to finalize all subjects for all grades and the draft roll-out strategy for the newly aligned distance learning curriculum and have it approved by the GC. However, by project closing, FODE had finalized the strategy, the GC had approved the strategy, and FODE had prepared a national launch workshop for the first new FODE curriculum in 40 years, which is expected to take place in February 2017 before the 2017 academic year begins. In addition, by project closing, the FODE project, with considerable support and achievement by the FODE curriculum development team, had been able to: (i) develop new equivalency program subject material for all required subjects 8 in the curriculum for grades 7-12; (ii) trial the agreed sample of materials; (iii) get the approval of course materials for all grades by the GC as required by the DoE; (iv) provide internet connectivity for 21 (out of 22) provincial FODE learning centers; (v) purchase and deliver 10 high speed printers to provincial FODE centers so they could print their own materials, thereby reducing national distribution costs for materials; and (vi) complete the

UNESCO. COL is the only intergovernmental organization solely concerned with the development of distance education. 8 The number of subjects per grade are: grade 7-six subjects; grade 8-six subjects; grade 9-seven subjects; grade 10-seven subjects; grade 11-twelve subjects and grade 12-twelve subjects. For grades 7-10 there were a total of 111 books, 111 assessments, and 34 projects/supplementary readers produced. For grades 11-12 there were 102 modules and 102 assessment produced.

14

development of the new FODE monitoring and evaluation (M&E) database, train provincial FODE staff in the use of the system, and upload the system onto the internet, thereby eliminating the need for a paper based M&E system. (See Annex 2 for more details.) These are all substantial implementation achievements considering the project supported the development of the entire equivalency program for PNG that required a large volume of new materials and consultants and curriculum specialists to produce them. Moreover, it is a considerable achievement that the project was able to provide internet connectivity for all but one province considering the ongoing security concerns in PNG. 30. There were also challenges faced in financial management of the project. There were ongoing challenges related to timely submission of audited financial statements and IFRs, timely submission of withdrawal applications, and repayment of ineligible expenditures (namely the taxes, GST, which were not financed), which required DoE counterpart funding for the repayment. The DoE along with other government departments were experiencing budget cuts due to the overall national budget crisis. At the time of preparing the ICR, while all financial covenants are currently complied with, there are a number that need to be completed but due in the future, including: i) submission of the final audited financial statements no later than 30 June 2017, ii) corrected final IFRs, and iii) refund of the outstanding balance of the advance to the Designated Account no later than 31 December 2016. It is likely that the outstanding advance will not be refunded by the due date and will become a lapsed loan. The Task Team and CMU is constantly following this up, and will continue to do so until fully refunded. 31. There were 13 Bank ISMs and 8 technical missions during the project implementation period. These missions presented opportunities for the Bank team to work closely with the FODE team and to support capacity development for FODE given they had limited capacity in project management and implementation. The makeup of the Bank teams was consistently appropriate and well balanced. Each team composition reflected the needs of the respective mission and included specialists from the areas of education, financial management, procurement, social, and M&E specialists as necessary. In addition to ISMs, the Bank team helped organize substantive workshops with organizations like the COL as mentioned above. The Bank also hired COL staff to provide training on curriculum development in an effort to support the FODE curriculum officers. During implementation, the Bank task team leader was based in Sydney, which allowed for regular interaction between the Bank, donor community, and DoE. This also provided continuity for implementation of the program and enhanced the Bank’s ability to quickly respond to issues/challenges as they arose, further contributing to the project’s implementation. By project closing, the project had disbursed XDR3.184 million (US$4.67 million equivalent, which is less than the US$5 million equivalent at signing due to exchange rate changes) or 99.5 percent of the credit. However, at the end of the disbursement grace period there was a PGK 255,458 outstanding balance in the DA. The PGK amount is equivalent to XDR 76,053 (US$101.714 equivalent), which, once refunded, will amount to a total project disbursement of US$4.57 million, or 97.6 percent of the credit.

15

2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 32. Design. The original M&E design included several activities. On the monitoring side, FODE was to: (i) report quarterly on students registered for FODE programs; (ii) calculate annual completion rates for its programs; (iii) undertake annual socioeconomic surveys of students enrolling in FODE programs and students who sit for exams, which was to constitute a database of student characteristics for better targeting of services; and (iv) report on project indicators bi-annually to the FODE GC, project steering committee and the Bank. On the evaluation side, FODE was to: (i) produce a study during the first year of the project on the characteristics of its student body and the motivations of its beneficiaries for participating in the programs to help the FODE better target its programs, as well as the GC with potential policy changes related to better targeting; (ii) launch a tracer study of students leaving its program (both completers and non-completers); and (iii) undertake a separate study on youth outside its own beneficiary group so as to compare its own results with other surveys, such as the 2009/10 household survey. The original design proved to be overly ambitious for the limited capacity within FODE. In addition, the FODE IT center could not provide the needed data for reporting on the results framework because of the limitation of the FODE database design and the paper based reporting system. As stated above, both the Bank and FODE recognized these limitations during the April 2014 MTR and then confirmed them during the March 2015 restructuring by modifying the overall design of the project M&E. The restructuring appropriately modified the M&E by: (i) revising and simplifying the PDO and results framework with new PDO and intermediate indicators; (ii) eliminating the studies; and (iii) focusing on developing a database system that captured appropriate data for the FODE programs that was internet-based rather than paper-based. The re-design also focused on making internet connectivity available in at least 10 of the 22 PNG provinces so that the data could be provided on-line. All were appropriate and needed modifications to the M&E design. 33. Implementation and Utilization. As indicated, the pre-restructuring implementation period showed limited progress with the implementation of M&E activities. The restructured M&E design was simple and allowed FODE to: (i) collect and report data for the results framework; and (ii) develop a newly designed internet-based FODE database system that could provide FODE with the needed student data as well as useful reports for decision making. This achievement laid the all-important and necessary foundation for the FODE M&E system that had not existed prior to the project investments. 2.4 Safeguard and Fiduciary Compliance 34. Safeguards. The project was rated a Category “C” operation. The environmental safeguard, OP/BP 4.01, was triggered because the IDA credit was to fund renovations of the FODE headquarters and GoPNG would fund renovations of about 20 provincial training centers. The renovation of FODE’s headquarters were to include only works. The renovations of provincial centers were envisioned to entail some extension of classroom facilities, however, all works were to be carried out at existing facilities and there were therefore no land issues. Due to the minor nature of works, an environmental assessment was not required. In order to safeguard against negative environmental impacts, simple and clear criteria as outlined in the Project Appraisal Document (PAD) were to be included in

16

the bidding documents and in contract documentation, which the Bank would ensure compliance of during approval of such documents. During project restructuring, construction and rehabilitation activities were removed from the project; however, as Bank requirements mandate that a safeguards policy cannot be "un-triggered", the project remained a Category "C". Project safeguards were consistently rated satisfactory. 35. Financial Management. Financial management (FM) ratings fluctuated through the implementation period between moderately satisfactory and moderately unsatisfactory. This was primarily due to capacity constraints of FODE’s FM staff and the limited knowledge of Bank requirements stemming in part from the limited involvement of the Bank in the education sector during the preceding decade. This led to challenges in the timely submission of withdrawal applications, timely submission of the unaudited IFRs, and timely submission of audited financial statements. In addition, this led to challenges in ensuring compliance with the Financing Agreement in respect of eligible expenditure (financing was exclusive of taxes) in that the taxes (GST) were paid from the IDA finances in Designated Account (DA) contrary to GoPNG law and contrary to the financing agreement, leading to ineligible expenditures. The last FM supervision mission in August 2016 rated the FM performance as MU due to overdue audited financial statements, While at the time of preparing the ICR, this audit has been submitted and all financial covenants are currently complied with, there are a number that need to be completed but due in the future as mentioned in paragraph 30, A final FM supervision will be done 2017 to ensure all financial covenants are complied with.

36. Procurement. Procurement was rated moderately unsatisfactory for much of the life of the project, based primarily on delays in carrying out procurement activities due to the weak capacity of FODE in this area. The hiring of a part time International Procurement Advisor in April 2014 who continued support until project closing led to a rapid improvement in planning and execution of procurement activities, elevating the rating to moderately satisfactory where it stayed for the remainder of the project.

2.5 Post-completion Operation/Next Phase 37. The DoE is committed to continued support of FODE, as highlighted in the NEP 2015-2019. With the development of a new curriculum for the first time in 40 years that is aligned with the formal school system, FODE has enhanced its relevance as an education provider, with a target audience representing the 90 percent of youth in PNG without access to formal secondary education. Through the development of the curriculum for Grades 7 to 12 carried out by a team of writers, content editors, and language editors, FODE has benefitted from increased capacity in managing curriculum development and quality assurance, which includes: (i) examination for FODE students at entry level; (ii) defining standards for each subject by unit and credit to ensure equivalency with the regular education system; (iii) planning and monitoring the implementation schedules of each step of curriculum development, by subject and grade, while mobilizing the necessary technical skills to complete each step; and (iv) putting in place the necessary quality control points including SRC, AAC and GC, as well as curriculum trialing and sampling. Through the development and approval of a curriculum rollout strategy, connectivity of all provinces to FODE Headquarters (HQ), and the addition of 10 high-end printers to a number of the

17

provinces, FODE has the necessary foundation in place to ensure the new curriculum is rolled out across the country. 38. Through the rollout of internet connectivity to all FODE centers across the country with the technical support of DoE’s IT division, FODE has the hardware in place to improve communications between provincial centers and HQ, providing access to DoE’s intranet (EdNet) for the first time, and allowing fast transfer of resources in soft copy to permit provinces to handle distribution to students in a more effective and timely fashion. It also allows for instant transmission of student data to HQ, allowing FODE to make decisions based on more accurate and up-to-date data. The FODE student information database that was upgraded under the project facilitates better transmission of student data, and the training and user manuals provided to senior clerks from all provincial FODE centers helps ensure the database will be used going forward. DoE has committed to continuing the monthly internet service charge payments, which benefits the wider education department as it has provided a greater capacity to the internet hardware previously in use. However, given the current national budget crisis and DoE budget cuts, providing the needed resources for maintaining connectivity may be a challenge for DoE.

3. Assessment of Outcomes

3.1 Relevance of Objectives, Design and Implementation 39. Relevance of Objectives and Design. The original objective of increasing the out-of-school youth who completed a recognized secondary diploma and certificate equivalency program was substantially relevant to the country’s sectoral needs when the project was developed. The objectives were also consistent with pillar 2 of the Bank’s CAS (2008-2011) that aimed to enhance rural livelihoods, and improve local service delivery as well as the UBE focus on the need to address the problem of out-of-school youth. Moreover, it continues to fit into the goals of the UBE (2010-2019) and second pillar of the Bank’s 2013-2016 Country Partnership Strategy (CPS) to PNG, and continues to be a highly important delivery modality for reaching the out-of-school youth in PNG. However, the original design was overly ambitious with most project activities dependent on the development, completion and implementation of a new equivalency curriculum. The original design underestimated the time required to develop all the materials needed for every subject for all grades 7-12 so that additional project activities could be implemented and progress toward achievement of the PDO could be measured. In addition, data required for monitoring progress of the PDO could not be gathered because the design team did not fully appreciate the limitation of the paper-based data collection process. Therefore the relevance of the original design is judged to be modest. The 2015 restructuring that restated the project objective and narrowed the design scope of the project is judged to be of substantial relevance. This restructuring accommodated the realities of the limited capacity and focused the project objectives on the completion of the equivalency curriculum recognizing that this achievement would lay a solid foundation for FODE to build on going forward. The 2015 restructuring also modified the project design by: (i) reducing the number of activities and ensuring that the remaining activities focused specifically on completion of the full equivalency program; and (ii) ensuring internet

18

connectivity for the 22 provincial FODE centers to facilitate faster communication between the central FODE center and provinces so as to provide better services for the provinces and students. All-in-all, the adjustments made at the time of the 2015 restructuring were substantially relevant. 40. Relevance of Implementation. As indicated above, there were challenges throughout implementation related to FODE management capacity, procurement and financial management that ultimately led to the project restructuring. Despite the challenges, and after the restructuring, FODE was able to: (i) develop new equivalency program subject material for all required subjects in the curriculum for grades 7-12; (ii) provide internet connectivity for provincial FODE learning centers; (iii) complete the development of the new FODE M&E database, train provincial FODE staff in the use of the system, and upload the system onto the internet thereby eliminating the need for a paper based M&E system; and (iv) finalize all procurement according to Bank guidelines. The achievements came with strong support from technical assistance related to curriculum development and procurement, as well as the Bank team during technical missions and ISMs. However, financial management remained challenging even after completion of the project. 3.2 Achievement of Project Development Objectives 41. This section evaluates the outcomes against the project development objective and the results framework that was adjusted and approved by the Board on March 6 2015 formal restructuring. The development objective was: (i) to align the distance education curriculum with the national curriculum; and (ii) to strengthen FODE’s information management system. More details on project outputs can be found in Annex 2. 42. Alignment of the distance education curriculum with the national curriculum was to be measured by the PDO indicator aligned subject material for grades 7-12 approved by GC. The target was for six subjects per grade in grades 7, 8 and 10, seven subjects for grade 9, and 13 subjects per grade for grades 11 and 12.9 The targets for all grades 7-12 were met. In addition to actual targets being achieved, by the end of the project, all subjects for grades 7-12, received approval by the GC, thereby the PDO target met. The lasting impact of this achievement will be that all future graduates of the FODE program will have a degree that is equivalent to those graduates of the formal system.

43. Strengthening of FODE’s information management system was to be measured by the PDO indicator data available online and used for planning and budgeting purposes. By the end of the project, the information management system database had been remodeled and debugged and uploaded onto the internet so that provincial centers could enter data online. In addition, the FODE center had developed a new user’s manual and the provincial FODE center senior clerks had been trained on the new system. However, this indicator

9 It was identified in the final ISM that the actual number of subjects for grades 7-9 included PD, and the actual number of subjects in grades 11 and 12 was only 12. The reason for this discrepancy was due to two different mathematics modules incorrectly counted as two subjects.

19

was only partially met because by project closing the new data system had just been launched and FODE had not yet been able to use the data for planning and budgeting.

44. The intermediate indicators that supported alignment of the distance education curriculum with the national curriculum were: (i) new learning materials for all subjects for grades 7-10 trialed and ready for printing10; (ii) module writing for grades 11 – 12 subjects completed and ready for trialing; and (iii) roll-out strategy for aligned distance learning curriculum developed and approved. First, new learning materials were developed for 26 subjects for grades 7-10. A select number of subjects in all grades were trialed and materials updated based on the feedback. All 26 subjects were approved by the GC, thereby rendering them ready for printing. As mentioned earlier, the results framework target for this indicator was 28 subjects; however, this was due to two mathematics modules being counted as two subjects. Second, module writing for grades 11-12 was completed thereby meeting the target. In addition to the completion of the module writing, all 24 subjects for grades 11-12 were approved by the GC, exceeding the target for this indicator. Now that the GC has approved the modules they can be printed and used by all FODE center students. The connectivity provided through the project allows for course materials to be delivered via email and printed locally per registered subject rather than FODE making an estimation and delivering them, which caused delays and was a factor in high dropout rates or no-shows.This was a considerable achievement of the project. Finally, a roll-out strategy for the aligned distance learning curriculum was developed by FODE and approved by the GC, meeting the target for this indicator. As indicated, the roll-out strategy will be presented formally to the country during a national February 2017 curriculum launch. 45. The intermediate indicators that supported strengthening of FODE’s information management system were: (i) network connectivity established between FODE HQ for at least 10 provincial centers; and (ii) FODE management information system upgraded and rendered functional. By the project closing date, network connectivity had been established between FODE HQ and 21 provincial centers, exceeding the target. In the last of the 22 provinces, Hela, the connectivity hardware has been set up and is ready to be configured and tested once security concerns in the province improve enough for DoE IT and FODE staff to safely travel and complete the work. The FODE management information system was upgraded and rendered functional, as verified by DoE IT, meeting the target. In addition, as mentioned above, the system has also been uploaded onto the internet and available for use. In addition, a user manual was developed for the system, and training was carried out for senior clerks from all FODE provincial centers in the use of the system. 3.3 Efficiency 46. FODE itself is an efficient investment; under certain modeling assumptions, completing an additional year in FODE yields an internal rate of return between 5 and 6

10 To be considered “ready for printing”, the materials are required to have received GC approval.

20

percent11. Using earnings as a measure of the economic benefits of attending FODE, the increase in earnings from attending an additional year of FODE is modeled as the association between years of schooling and earnings at the secondary level nationally. This provides a measure of the benefit of attending FODE. The cost of attending FODE include both public expenditure and tuition fees per student, based on data presented in the original PAD and adjusted for inflation. If costs (including foregone earnings) and benefits for completing one additional year of FODE are compared, the internal rate of return ranges between 5.4 and 6.0 percent depending on the grade. In other words, there is always a positive net economic benefit for an individual to continue attending and completing FODE, based on the model.

47. While it is not possible to quantify the impact of the curriculum alignment component, the new data management system will enhance M&E of future investments in FODE. Aligning FODE’s curriculum to the national curriculum is crucial for second chance education, but it is not possible to quantify the impact of curriculum alignment on education outcomes, and subsequently human capital. This is because the curriculum alignment was completed at the very end of the project, and FODE does not have the capacity to monitor the outcomes (including completion and employment rates) that the original design of the project had envisioned. In addition, there is very little international evidence about the impact of second chance education curricula and labour market outcomes. The new data management system, however, provides FODE with the capacity to monitor and evaluate future investments to improve the programme. 3.4 Justification of Overall Outcome Rating Rating: Moderately Unsatisfactory 48. The overall outcome rating of the project is moderately unsatisfactory. The relevance of the objective pre-restructuring is rated substantial. The original project objective of increasing the numbers of out-of-school youth completing a certified equivalency program was and continues to be an important long-term goal for PNG considering that the net enrollment rate for formal secondary school is approximately 8 percent (NEP 2015-2019 data). Although the overall project design was ambitious and underestimated the time required to complete the project activities for measuring achievement of the PDO, the larger objective continues to be highly relevant for PNG. However, the pre-restructuring design is rated modest because it was overly ambitious. By 2013, the Bank implementation team recognized the design challenges and worked with government to restructure the project with objectives that could be achieved within the remaining implementation period. The new objectives, albeit more narrowly focused, supported the original objective by ensuring: (i) the new equivalency curriculum was completed and (ii) there was connectivity with the provinces for improved delivery of the curriculum. These were all important components for laying the necessary foundations for

11 The calculation of internal rate of return is based on students in the formal school system in the absence of data on FODE graduates.

21

the rollout of the new curriculum across the country. Based on this, the 2015 restructuring objective is rated substantial and design is also considered substantially relevant. The design, as indicated, was also fully aligned with the CAS/CPS and UBE for both pre- and post-restructuring. Although implementation faced many challenges, the restructured project activities were fully implemented, and there has been considerable capacity strengthening for all of the FODE staff that participated in the curriculum development as well as the development of the new FODE information system. Overall efficacy is rated moderately unsatisfactory. During the pre-restructuring period efficacy is rated moderately unsatisfactory. This is primarily due to the fact that: (i) data needed to measure progress could not be gathered because the curriculum was not completed; and (ii) the FODE information system was poorly designed and could not produce the needed data. Post-restructuring, the outcome indicators could be measured and, as indicated above, were achieved. Efficacy post restructuring is rated moderately satisfactory. Finally, as indicated above, the overall project efficiency rating is modest. Table 3: Project Rating Original Project–11/3/2011–3/06/ 2015–54.00 percent disbursement-net project funds1

Project Relevance Achievement of PDO (Efficacy) Efficiency Overall Rating Modest Moderately Unsatisfactory Low Moderately

Unsatisfactory

Restructured Project–3/7/2015–6/30/2016–46.00 percent disbursement-net project funds1

Project Relevance Achievement of PDO (Efficacy) Efficiency Overall Rating Substantial Moderately Satisfactory Low Moderately

Satisfactory

Overall Project Ratings – 97.6 percent disbursement of total grant as of 12/31/2016

Project Relevance Achievement of PDO (Efficacy) Efficiency Overall Rating2 Modest Moderately Unsatisfactory Low Moderately

Unsatisfactory2

1The net amount for the total project is US$4.57 million 2 Overall rating = (3)(.5400) +(4)(.4600) = 3.46 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 49. The current enrolment represents 45 percent female students. Although it is difficult to quantify the project impact on poverty reduction, gender and social development, it is believed that the new curriculum will greatly benefit future FODE students who will have access to more relevant quality education and will have equal opportunities for further education and jobs as those studying in the formal school system. According to the Government’s 2015 information, the enrolment rate of students in secondary schools is less than 10 percent. One of the major issues contributing to the low secondary school enrolment rates in PNG is the lack of supply of schools and teachers, with demand surpassing available school places. FODE’s relevant and equivalent curriculum could therefore benefit up to 90 percent of secondary school age students, the country’s potential FODE learners. As the analysis shows (paragraph 46), there is a positive net economic benefit for an individual to continue attending and completing FODE, and by completing

22

one additional year of FODE, the internal rate of return gains are between 5.4 and 6.0 percent depending on the grade. (b) Institutional Change/Strengthening 50. The project resulted in strengthening the capacity of the FODE curriculum development team to the extent that they are now able to: (i) develop work plans; (ii) manage their time to meet deadlines; (iii) work with teams; (iv) manage and monitor large teams of content specialists; and (v) format and design books. These are significant achievements for a team that had never taken on such a large scale revision of the FODE equivalency program with the formal system curriculum. (c) Other Unintended Outcomes and Impacts (positive or negative) Not applicable 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops Not applicable

23

4. Assessment of Risk to Development Outcome Rating: High 51. Going forward, the risks associated with curriculum updating and modifications should be moderate. Through the project, FODE has been able to substantially improve the capacity of the curriculum development staff which increases their ability to continuously update and improve the curriculum. In addition, FODE now has staff that can manage the new on-line information system and produce reports that are more useful for FODE policy makers. However, for any future operations the risk related to financial management and procurement should be substantial. While there was improvement in these areas, given the overall low capacity levels at the beginning of the project, it will take more time to develop this capacity. Finally, the risk for sustainability of overall project investments is rated high. This is due to the macroeconomic conditions that include a budget deficit in 2015 that increased to 5 percent of GDP compared with an estimated 4.5 percent in the 2015 Budget. This has led to a decline in public expenditure, which is affecting all government departments, including education. Moreover, in 2016, the country fell into ‘fragile state’ status because of the worsening macroeconomic situation in PNG.

5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Unsatisfactory 52. The project preparation team ensured that the project design was closely aligned with the UBE (2010-2019) and the Bank’s CAS (2008-2011). The original design took into consideration the recommendations from the peer reviewers and lessons learned from similar projects in other countries. However, the design was overly ambitious and underestimated the time required to develop and update a 40 year old curriculum for grades 7-12. In addition, the Bank team did not fully assess the capacity of the FODE human resources to manage the project or the information system capabilities prior to developing project indicators. The original project preparation team did identify the appropriate risks, however, the risks ratings were underestimated. The overall quality at entry for the original FODE project design is judged to be moderately unsatisfactory. (b) Quality of Supervision Rating: Satisfactory 53. The Bank conducted 21 technical, review, and implementation support missions, which included a mid-term review. The project team was actively engaged in supporting the government in its efforts to implement the project. Whenever implementation challenges arose, the Bank team worked with government to find solutions. The Bank team recognized early in the implementation period that a restructuring would be necessary and provided constant and consistent support to ensure that the restructuring was completed despite the various institutional processes and procedures associated with a restructuring. Furthermore, as the GST ineligible expenditures were later refunded to the project DA, the

24

formal restructuring process was not started until August 2014. The 2015 restructuring modified the design to ensure that the curriculum could be fully updated and FODE provincial centers were connected to the internet, all of which were achieved and have provided a strong foundation for future delivery of the equivalency program. All of the 2015 modifications were highly appropriate. The 2015 restructuring correctly reassessed the risks and upgraded the risk ratings, an appropriate modification to the design. The Bank team also provided strong support for the curriculum development by connecting FODE with the COL, participating in the workshops that FODE and COL organized, and hiring COL curriculum staff to provide training to FODE. The Bank teams consistently reported on financial management and procurement progress and worked with the FODE teams to build their capacity in these areas. The team also systematically documented project progress in aide-memoires, back-to-office reports, and Implementation Status Reports (ISR), all of which kept Bank management informed of progress and provided the foundation for the ICR analysis. (c) Justification of Rating for Overall Bank Performance Rating: Moderately Unsatisfactory 54. Based on the analysis above, more weight is given to the Bank team’s performance during implementation which was critical in achieving the restructured outputs. However, given the project implementation, one PDO indicator is only partially achieved. As a result, the overall Bank performance is rated moderately unsatisfactory. 5.2 Borrower Performance (a) Government Performance Rating: Moderately Unsatisfactory 55. At the time of preparation, the government was fully committed to improving and aligning the FODE equivalency program with the formal school curriculum. The FODE project was considered to be an important component in the government’s efforts to address the problem of out-of-school youth in PNG. The government actively supported the preparation of the project by ensuring the required government processing steps were completed in a timely fashion, participated in the project launch workshop, and remained committed to the overall goals of the project. However, during implementation there were ongoing challenges associated with provision of government budget to cover the counterpart funds particularly related to the construction of the new national FODE center facilities as well as to cover taxes (GST) which were not eligible for financing from the IDA. In addition, the government delayed decision making related to the restructuring and approving the need paperwork for implementation. These delays slowed implementation and impacted the financial management ratings during the project period. Based on this, the overall rating of the government performance is rated moderately unsatisfactory. (b) Implementing Agency or Agencies Performance Rating: Moderately Unsatisfactory 56. The FODE management was responsible for managing all aspects of implementation given the fact that project implementation was mainstreamed within the

25

FODE structure. As indicated, the FODE project was the first education project in more than a decade in PNG and as a result, there was limited project management capacity. The capacity constraints slowed curriculum development, procurement, financial management, reporting and overall implementation. After the project was simplified during the 2015 restructuring, FODE was able to successfully complete the curriculum development aspects of the project and, in the process, build strong capacity within the curriculum development department. This was a significant achievement. However, challenges remained in the areas of financial management, procurement, and overall project management which delayed aspects of implementation. Based on this, the implementation agency performance is rated moderately unsatisfactory. (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Unsatisfactory 57. Based on the analysis and ratings above, the overall Borrower performance is rated moderately unsatisfactory. 6. Lessons Learned 58. Lesson 1. Rigorous capacity assessments should be required at the time of project preparation when the Bank is re-engaging in a sector after an extended absence. The decade long absence of Bank involvement in the education sector led to limited knowledge of FODE capacity and in-country capacity in the sector. A thorough understanding of the overall capacity constraints could have played an important role in developing a simplified project design much like the 2015 restructured design. Moreover, it would have provided a better understanding of the available pool of in-country technical assistance for the project, which was heavily dependent on technical assistance support. 59. Lesson 2. Thorough knowledge of information systems capabilities is important for establishing project indicators. The FODE project indicators were established without full knowledge of what the FODE information system and data gathering system could produce. The paper-based system of data collection was slow and cumbersome. In addition, it became clear that the information system could not produce the needed reports to measure the PDO or some intermediate indicators. The restructured design helped to focus on developing an internet-based system that could produce the reports that were useful to FODE management in a timely manner. The design of the system and reports was based on an IT needs assessment carried out under the project.

60. Lesson 3. Time and knowledge requirements for curriculum development must be factored into project design. The original project design based most of the project activities on the completion of the grades 7-12 curriculum in the first couple of years of project implementation. This was too ambitious considering the curriculum had not been updated for 40 years. Moreover, the FODE curriculum team had never taken on an extensive task such as the realignment of the entire FODE curriculum, and therefore had limited knowledge of how to develop a distance education curriculum. Again, the restructured project factored these complexities into the design by focusing project

26

activities on completion of the curriculum, thereby ensuring a strong foundation for the national roll-out. 61. Lesson 4. Effective project management is essential for timely implementation. FODE management struggled with implementation throughout the project period due to capacity constraints and limited experience with project management. The capacity constraints slowed curriculum development and procurement. It was particularly problematic for financial management because the original payment of the ineligible taxes (GST) on contracts led to the slow restructuring of the project.

7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies No comments were received. (b) Cofinanciers (c) Other partners and stakeholders (e.g. NGOs/private sector/civil society)

27

Annex 1. Project Costs and Financing

(a) Project Cost by Component (in USD Million equivalent)

Components Appraisal

Estimate (USD millions)

Restructuring Estimate (USD

millions

Actual/Latest Estimate (USD

millions)

Percentage based on

Restructured Estimates1

Component 1 (revised): Curriculum Development

1.50 1.80 1.64 91.1

Component 2 (new): Information Management Infrastructure

2.52 1.96 77.8

Component 2 (old): Improving Quality of Service Delivery

0.70 0.00 0.00

Component 3 (new): Project Management, Monitoring and Evaluation

0.68 0.97 157.0

Component 3 (old): Grade 11-12 Matriculation Program Expansion

1.80 0.00 0.00

Component 4 (old): Improving Information Technology, Monitoring and Evaluation

1.00 0.00 0.00

Total Baseline Cost 5.00 5.00 4.572 97.62 Physical Contingencies 0.00 0.00 0.00 Price Contingencies 0.00 0.00 0.00

Total Project Costs 0.00 5.00 4.57 97.6 Total Financing Required 5.00 5.00 4.57 97.6

1 The percentages are have been calculated based on the new component allocations of the IDA credit during the 2015 restructuring. 2 By project closing, the project had disbursed XDR 3.184 million (US$4.67 million equivalent due to exchange rate losses) or 99.5 percent of the credit. However, at the end of the disbursement grace period there was PGK 255,458 outstanding balance in the DA. This is equivalent to XDR 76,053 (US$101.714), which, once refunded, will amount to a total project disbursement of US$4.57 million, or 97.6 percent of the credit. (b) Financing

Source of Funds Type of

Cofinancing

Appraisal Estimate

(USD millions)

Actual/Latest Estimate

(USD millions)

Percentage of Appraisal

Borrower 1.00 1.361 136.1 International Development Association (IDA)

5.00 4.57 97.6

28

Annex 2. Outputs by Component 62. Initial progress towards achieving project outcome and meeting output targets was measured by one PDO indicator and three intermediate indicators. Following the restructuring in 2015, the revised results framework included two PDO indicators and a revised six intermediate indicators. This annex is only focused on the project components and activities that were included in the 2015 restructuring. This is because there was very little progress in the activities described in the original project components, and any progress made is reported under the revised components. The Annex also uses the indicators as listed in the revised results framework to assess output achievements. Component 1: Curriculum Development (US$1.8 million revised IDA allocation after the 2015 restructuring) 63. Under the February 2015 restructuring, this component aimed to support: (i) curriculum writing/module development for all Grades 7 – 12 subjects, including printing of new material and acquisition of curriculum development reference materials; (ii) the alignment of the revised curriculum with the national curriculum and the establishment of revised assessment rubrics for scoring and marking; (iii) trialing of the new curriculum; and (iv) the development of a roll-out strategy. 64. The intermediate indicators that were used to measure achievement of this component were: (i) new learning materials for all subjects for Grades 7 - 10 trialed and ready for printing; (ii) module writing for Grades 11 – 12 subjects completed and ready for trialing; and (iii) roll-out strategy for aligned distance learning curriculum developed and approved. By the end of the project, new learning materials were developed for all six subjects of Grades 7 and 8, seven subjects of Grades 9 and 10, and 12 subjects of Grades 11 and 12, leading to the development and approval of 213 subject books, 213 assessment books, and 34 projects or supplementary books, bringing the total of new materials developed to 460 books. A curriculum roll-out strategy was developed and approved by the Governing Council (GC). There were 92 consultants hired to develop the new curriculum, including five curriculum content advisors, eight subject content, eight course language editors, 69 module course writers/assessment and markers guide writers, a course trialing coordinator, and a distance learning pedagogy, technologies and curriculum specialist. In addition, there were 400 reference and text books procured for the FODE curriculum library. Prior to restructuring, along with much of the writing of curriculum materials for Grades 7 to 12, component 1 supported: (i) staff capacity building through knowledge-sharing on distance education by the distance education pedagogy consultant and through curriculum writing workshops; (ii) an analysis of resource requirements, planning for consultation, review and approval of learning materials, and the development of realistic timelines to carry out these activities; and (iii) technical assistance from the Commonwealth of Learning (COL) given its expertise in distance learning delivery and curriculum development. Several workshops, a study tour to Namibia and other exchanges took place between October 2013 and June 2014, which helped ensure all curriculum-related activities could be completed within the life of the project and with the necessary quality tools in place (i.e., planning and sequencing of curriculum writing, terms of

29

reference revisions etc.). Under the original component 3, "Grade 11-12 Matriculation Program Expansion", which was dropped during restructuring, tutor manuals for all subjects were developed. 65. This is the first time the FODE curriculum has been updated in almost 40 years. It’s important to highlight that this activity has brought an increased capacity to manage a whole new process of curriculum development and quality assurance, which had not existed in PNG prior to the project’s support. This includes: (i) examination for FODE students at entry level; (ii) defining standards for each subject by unit and credit to ensure equivalency with the regular education system; (iii) planning and monitoring the implementation schedules of each step of curriculum development, by subject and grade, while mobilizing the necessary technical skills (content writers, content editors and language editors) to complete each step; and (iv) putting in place the necessary quality control points including SRC, AAC and GC, as well as curriculum trialing and sampling. Component 2: Information Management Infrastructure (US$2.52 million revised IDA allocation after the 2015 restructuring) 66. Under the 2015 project restructuring, the revised Component 2 was designed to support: (i) the establishment of network connectivity between provincial FODE centers and headquarters, and aligning FODE information and communication technology (ICT) with the wider, DoE ICT policy; and (ii) the remodeling/upgrading of FODE's existing Student Information System and provision of related training in using internet, data collection and data entry. The purpose of these activities was to assist FODE to improve its service delivery to all students (including those in remote/isolated areas) in a timely fashion, minimize communication and freight costs for materials distribution, and strengthen its monitoring and evaluation capacity. 67. The intermediate indicators that were used to measure achievement of this component were: (i) network connectivity established between FODE HQ and at least 10 provincial centers; and (ii) FODE management information system upgraded and rendered functional. By the project closing date, network connectivity was established between FODE HQ and 21 of the 22 provincial centers12 , exceeding the target. The Student Information System was also upgraded, tested, rendered functional, and training on the use of the system was provided to senior clerk staff from all provincial centers, which included provision of database user manuals. The project also purchased 10 high-end printers so that a select number of provinces could print the new learning materials. A standby generator was purchased and installed at FODE HQ to assist with the regular power losses the office was experiencing. Prior to restructuring, component 4 activities, some of which were merged into the restructured component 2, supported a FODE Information Technology (IT) needs assessment and the distribution of laptops and printers to all provincial FODE centers. Basic training in using e-mail and Microsoft Windows was provided to all provincial

12 The last of the 22 provinces to be connected, Hela, has had cable and equipment installed and ready for configuration once security concerns improve.

30

FODE coordinators. COL also supported the review of the existing Student Information System, and organized a workshop for relevant stakeholders to clarify their information needs and determine the remodeled system set-up. 68. With FODE provincial centers now connected with internet configured, they can access the DoE’s intranet (EdNet) and communicate via e-mail with FODE HQ for the first time. Initial observations indicate improved and strengthened communication and coordination between FODE HQ and provinces. Together with the purchase and installation of 10 high-end printers in the select provincial centers, FODE can now think more strategically about how to distribute the new learning materials in a more efficient way. Component 3: Project management, monitoring and evaluation (US$0.68 million revised IDA allocation after the 2015 restructuring) 69. This component was added during the 2015 project restructuring recognizing the limited capacity and resources of the FODE institution and project, aimed at providing targeted support in the areas of activity planning, project monitoring, management, coordination, staff proactivity, financial management and procurement. It specifically aimed to support: (i) project management and day-to-day operations, including activity planning and budgeting, implementation monitoring and progress reporting; (ii) financial management and procurement activities, including reporting and audits; and (iii) monitoring and evaluation activities including gathering information and updating the results framework. 70. The single intermediate indicator that was used to measure achievement of this component was harmonized annual project plans developed and submitted for Bank approval no later than January 15 of each year, which included the project’s implementation, procurement, budget and training plans. These plans showed signs of improvement later in project implementation, showing strengthened capacity of project management, procurement and financial management of FODE. The project provided the following consultants to assist with project implementation and build FODE capacity to carry out activities beyond the project closing date: (i) project management; (ii) procurement; (iii) financial management; (iv) IT engineer; and (v) database consultant. However, reporting and project milestones continued to be missed, with Bank support required beyond the project closing date to ensure activities were completed and all project expenditures accounted for and audited.

31

Annex 3. Economic and Financial Analysis 71. This project aligned FODE’s curriculum with the national curriculum and strengthened its data collection and management system. Curriculum alignment included updating the curriculum for six subjects in grades 7, 8 and 10; seven subjects in grade 9, and 13 subjects for grades 11 and 12. All subjects received approval by the governing council by the end of the project. In addition, the information management system has become operative allowing provincial centers to enter data online. 21 out of 22 provincial centers have been provided internet connectivity. The FODE M&E database has been activated and FODE have been trained on the system. 72. Attending FODE itself is a worthwhile investment if benefits of secondary education for FODE attendees are equal to the regular student population; under certain modeling assumptions, completing an additional year in FODE yields an internal rate of return of just below 6 percent. Using earnings as a measure of the economic benefits of attending FODE, the increase in earnings from attending an additional year of FODE is modeled as the association between years of schooling and earnings at the secondary level, nationally. Note that estimating the returns to attending FODE itself is not possible as there is no data on FODE graduates themselves. The association between years of schooling and earnings is estimated with the 2010 PNG Household Income and Expenditure Survey (HIES) using a Mincerian earnings function and allowing the association between earnings and schooling to differ in secondary school from primary school. This provides a measure of the benefit of attending FODE. The cost of attending FODE include both public expenditure and tuition fees per student, based on data presented in the original PAD and adjusted for inflation. For a student who has completed a certain number of years of schooling, he or she may begin work and receive an earnings stream resulting from his or her current years of schooling, or he or she may attend FODE, incur the public and private costs, and then begin working the following year. The difference in the two earnings streams represents the net benefit of attending FODE for one more year. The internal rate of returns of the net benefits range 5.35 percent to 6 percent (Figure 3.1). For each year, there is always a positive incentive for an individual to complete one more year of FODE. 73. While curriculum alignment is essential to FODE’s quality, the impact of this alignment on enrolment and completion rates as well as students’ cognitive outcomes is not measured. The original project design included monitoring of participation and completion rates as well as labor market outcomes for graduates; however, the restructuring of the project including the PDO reflected the limited capacity of FODE to collect this data. In addition, curriculum has been completed at the end of the project which leaves little time to observe any impact on outcomes if they had been measured. Finally, there is little international evidence on the impact of curriculum alignment in a distance education; international literature tends to be descriptive and focuses on higher education. 74. Going forward, the new data management system provides better data for identifying FODE’s investment needs and monitoring and evaluating these investments. The new data management system provides a much different picture of

32

student and subject enrolment (Table 3.1). For example, in 2008, the project’s PAD reported that 48,328 students had enrolled in FODE in 2008 in 95,049 subjects; these figures were based on minutes from FODE’s governing council. In 2014, based on numbers from the new data management system, enrolment was 11,893 students in 30,893 subjects. It is not clear whether this drop in the number of students enrolled is a result of an actual decline in enrolment or through better measurement; it is believed to be the latter. The new data management system also reveals that there are more males than females in FODE (Figure 3.2). This is especially true at the MS level (grades 11 and 12) where female enrolment represents 42.1 percent of total enrolment. For all FODE enrollees, females represent 45.6 percent. 75. To benefit fully from curriculum alignment, further investment is needed to resolve other bottlenecks; for example subject completion rates are low, at 60 percent in 2015. Based on data from the new data management system, 59.8 percent of all subjects attempted are completed in FODE (Table 3.2). At the 8th grade level, the completion rate was only 10.6 percent. While these figures are much higher than what was presented in the project’s PAD in 2008 (where only 335 subject completion certificates were issued for 95,049 subjects enrolled), it is unclear whether the PAD’s figures are comparable. A low completion rate hinders the economic benefit of FODE. It also hinders the economic benefit of the project’s investment in curriculum alignment. However, the new data management system’s accurate measure of enrolment and completion more easily justifies further investment in FODE as these outcomes can now be accurately monitored and investments evaluated.

5.35%

6.01% 6.00% 5.96% 5.90% 5.81%

0%

2%

4%

6%

8%

6 7 8 9 10 11

Highest grade completed prior to taking an additional year of FODE

Figure 3.1. Modeled internal rate of return of completing one more year of FODE by highest grade completed

33

Table 3.1 Student and subject enrolment by year

2006 2007 2008 2014 2015

Total students enrolled 39,893 42,844 48,328 11,893 13,754Total subjects enrolled 77,264 83,884 95,049 30,083 34,699Subjects completion certificates issued 713 286 335 n.a. n.a.

Source: FODE PAD Table A1 for 2006-08 data; Government of PNG for 2014-15 data

Table 3.2 Subject completion rate Grade 8, 10, and MSP 2015

Subjects enrolled

Subjects completed

Comple-tion (%)

8th grade 2,048 218 10.6 10th grade 16,412 10,137 61.8 MSP 8,932 6,018 67.4

Total 27,392 16,373 59.8

Source: Government of PNG

156361

1231

2993

1518

175 398

1423

3409

2090

0

500

1000

1500

2000

2500

3000

3500

4000

7 8 9 10 MS(11/12)Grade

Figure 3.2 FODE enrolment 2015 by grade and gender

female

male

34

Annex 4. Bank Lending and Implementation Support/Supervision Processes

(a) Task Team members

Names Title Unit Responsibility/

Specialty Lending James A. Stevens Senior Operations Officer GED02 Task Team Leader Stephen David Close Education Specialist GED02 Education Stephen Paul Hartung Financial Management Specialist GGO20 Financial Mgmt

Robert L. McGough Senior Education Specialist EASHD -

HIS Distance Education

Ellen Kulumbu Human Development Officer EACGF Country Operations Sheila Braka Musiime Chief Counsel LEGES Legal Counsel Evelyn Villatoro Senior Procurement Specialist GGO08 Procurement

Supervision/ICR Binh Thanh Vu Senior Education Specialist GED02 Task Team Leader Cristiano Nunes Senior Procurement Specialist GGODR Procurement David Whitehead Financial Management Specialist GGODR Financial Mgmt Franco Russo Senior Operations Officer GED02 Operations Stephen Close Education Specialist GED02 Education Myrna Machuca-Sierra Education Specialist GED02 Education Ellen Kulumbu Human Development Officer EACGF Country OperationsJohn Kalu Human Development Officer GHNDR Country OperationsKerry Pagau Senior Human Devt Specialist GHNDR Country OperationsDodi Doiwa Operations Analyst EACGF Procurement Joy Sagati Operations Analyst EACGF Financial Tasha Sinai Program Assistant EACGF Program Support Summerville Kave Team Assistant EACGF Admin Support Jane Sprouster Implementation Support Specialist EACNF Operations SupportSandra Beemer Consultant GEDDR ICR Author Kevin Macdonald Economist GEDDR ICR Econ. AnalysisKris McDonall Program Assistant EACNF ICR Co-Author (b) Staff Time and Cost

Stage of Project Cycle Staff Time and Cost (Bank Budget Only*)

No. of staff weeks USD Thousands (including travel and consultant costs)

Lending FY09 4.31 26.41 FY10 0.77 12.65 FY11 11.15 59.94 FY12 0.00 10.88

Total: 109.78

35

Supervision/ICR FY11 4.6 22.93 FY12 27.87 148.21 FY13 55.59 152.06 FY14 20.28 127.07 FY15 61.54 201.78 FY16 37.39 125.14 FY17 3.55 34.13

Total: 800.55 * Trust fund resources were also used to support lending (US$82,986.36) and supervision (US$25,254.64).

36

Annex 5. Beneficiary Survey Results Not Applicable

37

Annex 6. Stakeholder Workshop Report and Results Not Applicable

38

Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR Submission of the borrower portion not received.

39

Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Not Applicable

40

Annex 9. List of Supporting Documents Project Appraisal Document, World Bank, 2010 Aides-memoire and Implementation Status Reports 2010 – 2016 World Bank Country Assistance Strategy 2008-2011 Project Paper for a proposed restructuring, World Bank, 2015 World Bank Country Partnership Strategy 2013-2016 States of Fragility 2015: Meeting Post-2015 Ambitions, OECD Publishing, Paris

41

MAP