document of the world bank · 2016-07-08 · document of the world bank report no: icr00003570...

Document of

The World Bank

Report No: ICR00003570

IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA-45650)

ON A

CREDIT

IN THE AMOUNT OF XDR 17.0 MILLION (US$25 MILLION EQUIVALENT)

TO THE

REPUBLIC OF ARMENIA

FOR A

SECOND EDUCATION QUALITY AND RELEVANCE PROJECT (APL 2)

April 28, 2016

Education Global Practice Europe and Central Asia Region

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

CURRENCY EQUIVALENTS

(Exchange Rate Effective January 13, 2016)

Currency Unit = Armenian Dram AMD 1.00 = US$0.0021 US$1.00 = AMD 483.67

FISCAL YEAR January 1–December 31

ABBREVIATIONS AND ACRONYMS

ADS Armenia Development Strategy ANQA Armenian National Quality Assurance Agency APL Adaptable Program Loan ATC Assessment and Testing Center CEP Center for Education Projects CIF Competitive Innovation Fund CPS Country Partnership Strategy EDI Early Development Index EIP Education Improvement Project EMIS Educational Management Information System EQA External Quality Assurance EQRP Education Quality and Relevance Project HEI Higher Education Institution ICT Information and Communication Technology ICR Implementation and Completion Results Report ISR Implementation Status and Results Report IQA Internal Quality Assurance KG Kindergarten M&E Monitoring and Evaluation MoES Ministry of Education and Science MTEF Mid-Term Expenditure Framework MTR Midterm Review NaCET National Center for Education Technology NIE National Institute of Education NVAO Netherlands and Flandria Accreditation Agency PAD Project Appraisal Document PDO Project Development Objective PIU Project Implementation Unit QA Quality Assurance RAU Russian-Armenian Slavonic University SDP Sustainable Development Program TA Technical Assistance SMIS School Management Information System TEMIS Tertiary Education Management Information System

TIMSS Trends in International Mathematics and Science Study

ToR Terms of Reference UEE Unified Entrance Examination UNICEF United Nations Children’s Fund

Senior Global Practice Director: Claudia Costin Practice Manager: Cristian Aedo

Project Team Leader: Dandan Chen ICR Team Leader: Dandan Chen

ICR Primary Author: Julia Liberman

ARMENIA Second Education Quality and Relevance Project

CONTENTS

Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph

1. Project Context, Development Objectives and Design ............................................... 1 2. Key Factors Affecting Implementation and Outcomes ............................................... 7 3. Assessment of Outcomes .......................................................................................... 12 4. Assessment of Risk to Development Outcome ......................................................... 19 5. Assessment of Bank and Borrower Performance ...................................................... 19 6. Lessons Learned ........................................................................................................ 22 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners ........... 22 Annex 1. Project Costs and Financing .......................................................................... 23 Annex 2. Outputs by Component .................................................................................. 24 Annex 3. Economic and Financial Analysis ................................................................. 32 Annex 4. Bank Lending and Implementation Support/Supervision Processes ............. 34 Annex 5. Beneficiary Survey Results ........................................................................... 36 Annex 6. Stakeholder Workshop Report and Results ................................................... 37 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ..................... 40 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ....................... 49 Annex 9. List of Supporting Documents ....................................................................... 50 Annex 10. Linkages between APL 1 and APL 2 .......................................................... 54 Annex 11. Education Sector Reform Program Triggers ............................................... 55 Annex 12. Details on the Project’s Main Beneficiaries ................................................ 58 Annex 13. Reallocation of Financing ............................................................................ 60 Annex 14. Summary of Armenia’s Ratings in Bologna Process Stocking-taking Reports, 2007-2015 ............................................................................ 61 Annex 15. Linkages between PDOs and Outcomes ...................................................... 62 Annex 16. Discussion on Lessons Learned ................................................................... 66

MAP

i

DATA SHEET A. Basic Information

Country: Armenia Project Name: Second Education Quality and Relevance (APL 2)

Project ID: P107772 L/C/TF Number(s): IDA-45650

ICR Date: 02/22/2016 ICR Type: Core ICR

Lending Instrument: APL Borrower: REPUBLIC OF ARMENIA

Original Total Commitment:

XDR 17.00M Disbursed Amount: XDR 16.37M

Revised Amount: XDR 16.37M

Environmental Category: C

Implementing Agencies: Ministry of Education and Science

Cofinanciers and Other External Partners:

B. Key Dates

Process Date Process Original Date Revised / Actual

Date(s)

Concept Review: 07/17/2008 Effectiveness: 10/02/2009 10/02/2009

Appraisal: 02/09/2009 Restructuring(s): 07/07/2012 10/28/2014

Approval: 05/12/2009 Mid-term Review: 11/12/2012 12/14/2012

Closing: 11/30/2014 11/30/2015

C. Ratings Summary C.1 Performance Rating by ICR

Outcomes: Moderately Satisfactory

Risk to Development Outcome: Moderate

Bank Performance: Moderately Satisfactory

Borrower Performance: Satisfactory

C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings

Quality at Entry: Moderately Satisfactory Government: Satisfactory

Quality of Supervision: Moderately SatisfactoryImplementing Agency/Agencies:

Satisfactory

Overall Bank Performance:

Moderately SatisfactoryOverall Borrower Performance:

Satisfactory

C.3 Quality at Entry and Implementation Performance IndicatorsImplementation

Performance Indicators

QAG Assessments (if any)

Rating

Potential Problem Project at any time (Yes/No):

No Quality at Entry (QEA):

None

ii

Problem Project at any time (Yes/No):

No Quality of Supervision (QSA):

None

DO rating before closing/inactive status:

Moderately Satisfactory

D. Sector and Theme Codes Original Actual

Sector Code (as % of total Bank financing)

General education sector 47 55

Pre-primary education 10 12

Public administration- Education 13 7

Tertiary education 30 26

Theme Code (as % of total Bank financing)

Education for all 28 30

Education for the knowledge economy 72 70

E. Bank Staff Positions At ICR At Approval

Vice President: Cyril E Muller Shigeo Katsu

Country Director: Mercy Miyang Tembon Asad Alam

Practice Manager/Manager:

Mario Cristian Aedo Inostroza Tamar Manuelyan Atinc

Project Team Leader: Dandan Chen Juan Manuel Moreno Olmedilla

ICR Team Leader: Dandan Chen

ICR Primary Author: Julia Liberman F. Results Framework Analysis

Project Development Objectives (from Project Appraisal Document)

The proposed Project Development Objectives (PDOs) are to (i) enhance school learning in general education and improve the school readiness of children entering primary education; and (ii) support the integration of the tertiary education system into the European Higher Education Area. Revised Project Development Objectives (as approved by original approving authority) (a) PDO Indicator(s)

Indicator Baseline Value

Original Target Values (from

approval documents)

Formally Revised

Target Values

Actual Value Achieved at

Completion or Target Years

Indicator 1 : Improved EDI scores of students in KG and schools receiving the grants with regard to students in the control group

iii

Value quantitative or Qualitative)

No difference between control and treatment group in EDI score

EDI scores are significantly better in treatment group

EDI scores are higher in treatment group as opposed to control group and this difference is statistically significant

EDI score on average 19% higher in treatment group

Date achieved 02/02/2009 06/05/2009 02/05/2013 11/30/2015 Comments (incl. % achievement)

ACHIEVED. The aspect of the PDO that is demonstrated by this indicator is the extent to which school readiness of children entering primary education is improved (through expanded access to and quality of preschool education).

Indicator 2 : Improved examination and tests scores in project-targeted schools (TIMSS 2011 and Unified Entrance Examination)


TIMSS 2007: 488 Science; 499 Mathematics2008 UEE results

5% improvement in TIMSS (2012 Target Value) 5% improvement in UEE average scores (Armenian) (2014 Target Value)

Indicator dropped

Date achieved 06/05/2009 06/05/2009 10/28/2014 Comments (incl. % achievement)

DROPPED. This indicator was dropped as part of the second project restructuring and replaced with PDO Indicators 4 and 5.

Indicator 3 : Standings of Armenia in the Bologna Scorecard improving from Orange and Yellow to Green


Quality Assurance: 1 red, 2 yellows, 1 light green. Recognition: 1 yellow, 1 orange, 1 light green

75% of 1 ratings in QA, recognition and degree system are green

5 out of 9 (56%) indicators are green


NOT ACHIEVED. This indicator was to track the implementation of the Bologna Process using indicators that carry value judgment expressed through dark green (excellent performance), light green, yellow, orange, and red (little progress).

Indicator 4 : Share of test-takers passing the 9th grade graduation exam Value quantitative or Qualitative)

96.67% 99% 99.8%


ACHIEVED. The aspect of the PDO that is demonstrated by this indicator is the extent to which school learning in general education is enhanced. This indicator was introduced as part of the second project restructuring.

Indicator 5 : Gross graduation rate in upper secondary education Value quantitative or Qualitative)

85.95% 90% 90%

iv


ACHIEVED. The aspect of the PDO that is demonstrated by this indicator is the extent to which school learning in general education is enhanced. This indicator was introduced as part of the second project restructuring.

(b) Intermediate Outcome Indicator(s)

Indicator Baseline Value

Original Target Values (from

approval documents)

Formally Revised

Target Values

Actual Value Achieved at

Completion or Target Years

Indicator 1 : % of teachers intensively using ICT and other new materials in the classroom Value (quantitative or Qualitative)

45% Increased proportion

60% 74%


ACHIEVED. Baseline data was not available at appraisal and was subsequently added from a 2008 study on the ICT use in schools during the first project restructuring.

Indicator 2 : % of teachers taking part in pedagogical development programs Value (quantitative or Qualitative)

No baseline Increased proportion

Increased proportion

100%


ACHIEVED

Indicator 3 : % of teachers still using exclusively the presentation/recitation mode of teaching Value (quantitative or Qualitative)

43% Reduced proportion

Reduced proportion

32%


ACHIEVED

Indicator 4 : % of school principals trained in leadership and management with a specific focus on designing and implementing school development plans

Value (quantitative or Qualitative)

0 25% 25%


ACHIEVED. This indicator was revised as part of the first project restructuring. Original indicator: % of schools with school improvement plans approved and under implementation.

Indicator 5 : % of high school students who visited the resource center (laboratory and library) Value (quantitative or Qualitative)

No baseline 30% 100%

Date achieved 06/05/2009 02/05/2013 11/30/2015 Comments (incl. %

ACHIEVED. Indicator was revised under the first project restructuring. Original indicator: % of high school students who visited the resource center (laboratory and

v

achievement) library) in the previous month; % of daily lessons which take place at the resource center.

Indicator 6 : No. of public and private universities evaluated against the newly developed QA standards, criteria, procedures and guidelines


0 7 public and 7 private universities

10 public and 5 private universities


PARTIALLY ACHIEVED. Three more public universities than planned were evaluated; however, two fewer private universities than planned were evaluated.

Indicator 7 : No. of public and private universities that have operational internal QA Units in place according to the new QA standards


0 All public and 25% of private universities

All public and 5 private universities



ACHIEVED. Other universities, such as three inter-governmental universities that have QA standards corresponding to their home countries, as well as the police academy that had security considerations, did not participate in APL 2.

Indicator 8 : TEMIS will be used by MOES to develop annual reports on higher education and to prepare MTEF and annual budget for higher education


0 Annual Report produced by TEMIS

TEMIS on-going


NOT ACHIEVED. This indicator was revised as part of the first project restructuring. Original indicator: The MoES updates the Higher Education Strategy using information/data collected through TEMIS.

Indicator 9 : Universities produce/update a strategic plan and self-assessment reports using the information/data collected through TEMIS


0 All public and 25% of private universities

5 public universities and 2 private universities

On-going

Date achieved 06/05/2009 06/05/2009 02/05/2013 11/30/2015

Comments (incl. % achievement)

NOT ACHIEVED. Original indicator before the first project restructuring: Universities produce a strategic plan and self-assessment reports using the information/data collected through TEMIS. Institutions started data entry; entered data is incomplete.

Indicator 10 : Identification of funding sources for the student loan scheme Value (quantitative or Qualitative)

0 1 (in 2012) Indicator dropped

Date achieved 06/05/2009 06/05/2009 02/05/2013

Comments (incl. % achievement)

DROPPED. This indicator was dropped as part of the first project restructuring. The activity was shifted to other government funding under which the government introduced the student loan program. Fourteen banks have been identified to provide student loans.

vi

Indicator 11 : Number of teacher education curricula comparable to a typical Bologna program


0

6 teacher training curricula comparable to a typical Bologna program taught in Bachelors and Masters teacher education programs

Teacher training curricula approved

All teacher training curricula revised and approved


ACHIEVED. All teacher training curricula are now comparable to a typical Bologna program.

Indicator 12 : No. of students studying in the revised curriculum Value (quantitative or Qualitative)

0 80% Indicator dropped


DROPPED. This indicator was dropped as part of the first project restructuring.

Indicator 13 : One round of CIF proposal selection completed and initial disbursements made Value (quantitative or Qualitative)

No Yes Yes


ACHIEVED. This indicator was introduced as part of the first project restructuring.

Indicator 14 : Number of universities that completed implementation of CIF grants


0 At least 2 public and 1 private universities

At least 4 public and 1 private universities

8 public, 1 private, 1 inter-governmental universities


ACHIEVED. This indicator was introduced as part of the first project restructuring.

G. Ratings of Project Performance in ISRs

No. Date ISR Archived

DO IP Actual

Disbursements (USD millions)

1 06/19/2009 Satisfactory Satisfactory 0.00 2 09/15/2009 Satisfactory Satisfactory 0.00 3 06/28/2010 Satisfactory Satisfactory 0.59 4 01/11/2011 Satisfactory Satisfactory 1.99 5 11/30/2011 Satisfactory Satisfactory 5.88

vii

6 06/26/2012 Moderately Satisfactory Satisfactory 9.80 7 12/26/2012 Moderately Satisfactory Satisfactory 12.91 8 06/23/2013 Moderately Satisfactory Satisfactory 14.25 9 01/15/2014 Moderately Satisfactory Moderately Satisfactory 16.03

10 11/17/2014 Moderately Satisfactory Moderately Satisfactory 20.19 11 04/18/2015 Moderately Satisfactory Moderately Satisfactory 21.44 12 10/10/2015 Moderately Satisfactory Satisfactory 22.72 13 11/18/2015 Moderately Satisfactory Moderately Satisfactory 23.21

H. Restructuring (if any)

Restructuring Date(s)

Board Approved

PDO Change

ISR Ratings at Restructuring

Amount Disbursed at

Restructuring in USD millions

Reason for Restructuring & Key Changes Made

DO IP

07/07/2012 N MS S 10.30

The restructuring was based on the government-requested changes to the project. Key changes included: (a) activities under one of the project components were revised; (b) financing was reallocated across project activities; and (c) the project Results Framework was adjusted to reflect the proposed changes.

10/28/2014 N MS MS 19.90

The restructuring was based on the government-requested changes to the project. Key changes included: (a) modifying the Results Framework to better measure project outcomes, including replacing one outcome indicator; and (b) extending the project closing date from November 30, 2014 to November 30, 2015.

I. Disbursement Profile

1

1. Project Context, Development Objectives and Design

1.1 Context at Appraisal

Country and Sector Background

1. At appraisal, Armenia had increased its public expenditure on education, and Armenia’s second Poverty Reduction Strategy Paper, prepared in 2008, indicated the aim of the government to raise education spending to 4.5 percent of the gross domestic product by 2021. Despite its strong commitment to education, it was expected that the government may be forced to reduce education spending in response to the global economic crisis, which resulted in Armenia’s real gross domestic product falling by 14.1 percent in 2009 and its poverty rate increasing to 34.1 percent (from 27.6 percent in 2008).

2. At the time of appraisal, Armenia had high average enrollment rates at all levels of education, except at the preschool level where the enrollment rate was at an all-time low of 22 percent. The enrollment rate and the quality of preschool education sharply declined after the funding and management responsibilities for early childhood education were transferred from the central government to local governments under the Decentralization Law of 1996. At the higher education level, although the government had not increased spending on higher education, enrollments had been rapidly growing due to the high reliance on private spending that mitigated the lack of government funding, which raised an equity concern as there was limited access to higher education among the poor. However, tertiary education was regarded as key to helping the country build a competitive knowledge economy, and in 2005, Armenia joined the Bologna Process, committing substantial resources in an effort to build a strong tertiary education sector in line with international standards. At project appraisal, the government had just approved the Strategic Program for Reforms in Preschool Education of the Republic of Armenia (2008–2015). The Program was supported by the first phase of the Adaptable Program Loan (APL 1) and it sought to increase preschool enrollment to 90 percent by 2015.1 The government also approved the Strategic Program for High School Reform in 2008 which aimed to improve the quality of education at the high school level by more efficiently allocating resources and facilities.

Rationale for World Bank Assistance

3. In 2003, the government set out an Education Sector Reform Program, with the objective of improving the quality and relevance of the Armenian school system to meet the challenges of the knowledge society.2 A Sustainable Development Program (SDP) covering 2009–2012 was prepared by the government in 2008. The SDP aligned with the Education Sector Reform Program, and specified improvement in the access and quality of education—from preschool to higher education—as key priories that would lead to accelerated poverty reduction and economic growth. Aligned with the SDP, the Country Partnership Strategy (CPS) for Armenia (FY2009–2012) highlighted the Bank’s support to Armenia’s education sector as a key priority, and placed particular emphasis on improving the quality and

1 As referenced in the Project Appraisal Document (PAD), the government initially targeted to achieve preschool enrollment of 90 percent by 2015, but primarily because of the economic crisis in 2009, the target was postponed to 2017. 2 Armenia Education Sector Development Policy. November 4, 2003.

2

accessibility of preschool education, improving the quality of high school education, and strengthening higher education through integration with the European Higher Education Area in line with the Bologna Process.

4. With Bank support, the first phase of the APL (APL 1) in support of the Education Sector Reform Program enhanced the efficiency of sector management, upgraded the quality and relevance of education, and improved the evaluation of sector performance. Many of the activities supported under the APL 1 were either carried over from or formed the basis of activities for the second phased of the APL (APL 2). A summary of the activities under the APL 1 that carried over into the APL 2 are presented in annex 10. The second phase of the APL, the subject of this Implementation and Completion Results Report (ICR), continued to focus on the reforms of the general education system that were the primary focus of APL 1, and also addressed key issues in both preschool and higher education areas that are closely related to general education reforms, thus aligning with the SDP and CPS objectives and responding to the government’s demand for support of an expanded sector agenda. Following the implementation of the Education Financing and Management Reform project (P008281, closed on October 31, 2002) and APL 1 (P074503, closed on November 30, 2009) as well as the completion of the Public Expenditure Review of the Education Sector (2008) and the technical assistance (TA) provided in the areas of quality assurance (QA) and Higher Education Financing (2008), the Bank team was well positioned to continue to support the expanded reform agenda under APL 2.

1.2 Original Project Development Objectives (PDO) and Key Indicators (as approved)

5. The project was the second phase of the APL of a 10-year program divided into three phases (4, 3, and 3 years). The overall objective of the program was to improve the quality and relevance of the Armenian school system to meet the challenges of the knowledge society. While APL 1 and APL 2 had project specific outcome indicators, the overall program, according to a government document included in the Project Appraisal Document (PAD) of APL 13, only had possible outcome indicators for each of the phases of the program.

6. According to the Credit Agreement of APL 1, the objective of APL 1 was to support the first phase of the Borrower’s Program through: (a) improving and strengthening the efficiency of the education sector to meet the challenges of a knowledge economy; (b) upgrading the quality and relevance of the school education system; and (c) improving the evaluation of the education system’s performance.4 The single outcome indicator for this project was the same as the one listed by the government as a possible program outcome under phase one. Given the strong performance of the first phase of the APL, it was decided that

3 Project Appraisal Document 26266-AM. December 18, 2003. Annex 15.

4 Development Credit Agreement No. 3849-AM. The PDO specified in the APL 1 PAD differed from the project objective indicated in the APL 1 Credit Agreement. According to the PAD, the project’s development objective was the same as the Program’s overall development objective, which was to “improve the quality and relevance of the Armenian school system to meet the challenges of the knowledge society."

3

the second and third phases would be merged into a single project (referred to as the second phase of the APL, or APL 2), the subject of this ICR. Annex 11 presents more details.

7. The PDOs of APL 2 were to (i) enhance school learning in general education and improve the school readiness of children entering primary education; and (ii) support the integration of the tertiary education system into the European Higher Education Area.

8. Progress toward achieving the PDOs of APL 2 was to be measured using the following outcome indicators5:

Improved Early Development Index (EDI) scores of students in kindergarten (KG) and schools receiving the grants with regard to students in the control group.

Improved examination and test scores in project-targeted schools (TIMSS 2011 and Unified Entrance Examination).

Standings of Armenia in the Bologna Scorecard improving from Orange and Yellow to Green.

1.3. Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification

9. There were no changes to the PDO during implementation. However, during the restructuring that was signed by the Regional Director on September 10, 2014 (and countersigned by the government on October 28, 2014) the project’s PDO indicator –

5 The Project Description section of the Project Appraisal Document (PAD) (page 5, paragraph 19) states:

To measure the impact of the proposed operation, a series of outcome indicators have been identified: (i) increased preschool enrollments; (ii) improved school readiness among entering first graders in participating communities; (iii) improvement of learning outcomes of 8th graders as measured by TIMSS 2011 and of the 12th graders by the Armenian unified examination, using the 2007 and 2009 results as the baseline, respectively; (iv) in higher education, enhanced program and institution accreditation and quality assurance as measured by the reports produced by the new Armenian National Quality assurance Agency (ANQA) and eventually its acquiring the status of candidate member to the European Network of Quality Assurance Agencies; and (v) standings of Armenia in the Bologna Scorecard (both in the overall score and partial scores related to the quality assurance, recognition and program structure). These indicators will also be complemented with other indicators which are component-specific (See Annex 3).

When comparing the indicators listed in the Results Framework (Annex 3 of the PAD, p. 26-28), indicators (ii), (iii), and (v) were included as project outcome indicators. Indicators (i) and (iv) referenced in the aforementioned paragraph were not included in the Results Framework, however they were mostly achieved by project closing.

4

“Improved examination and test scores in project-targeted schools (TIMSS 2011 and Unified Entrance Examinations)” – was replaced by two indicators:

Share of test-takers passing the 9th grade graduation exam.

Gross graduation rate in upper secondary education.

10. There were two reasons for this adjustment. First, because preparatory activities for the Trends in International Mathematics and Science Study (TIMSS) 2011 were launched in January 2009 and the field tests were carried out in April 2010, only six months after the project became effective on October 2, 2009, the project could not have had an important impact on student learning that could have been reflected by the assessment. Second, the Unified Entrance Examinations (UEEs) were not designed to be comparable from year to year, and were therefore not appropriate indicators to use to measure and track project outcomes.

1.4 Main Beneficiaries

11. The project’s main beneficiaries included students, parents, and communities; schools, teachers, librarians, and school administrative staff; higher education institutions (HEIs); professional development providers; key sector institutions, including the National Institute of Education (NIE), the National Center of Education Technology (NaCET), and the Armenian National Quality Assurance Agency (ANQA); and government agencies, including the Ministry of Education and Science (MoES) and marz6 Education Units. Details on the beneficiaries are presented in annex 12.

1.5 Original Components (as approved)

Component 1: Enhancing the Quality of General Education (total estimated IDA financing US$15.73; actual IDA financing US$16.59 million)

Subcomponent 1.1: Promoting School Readiness and Equal Opportunities at the Start of General Education

12. The main objective of this subcomponent was to increase the level of school readiness among Armenian children, with a focus on the poorest and most vulnerable communities.

Subcomponent 1.2. Supporting Further Improvements in the Quality of Education through Improved Teacher In-service Training and Professional Development

13. The objective of this subcomponent was to improve the quality of teaching through (a) developing a national system for teacher and principal professional development, and (b) creating a school improvement network.

6‘Marz refers to a region in Armenia. “The territory of the Republic of Armenia is composed of ten marzes (regions) and Yerevan city which is governed by the law on local self-government in the city of Yerevan.” http://www.gov.am/en/regions/.

5

Subcomponent 1.3. Continuing Support to the Integration of ICT in Teaching and Learning Process

14. The objective of this subcomponent was to ensure that all schools in Armenia are equipped with up-to-date technologies, connected interactively through the Internet, and are ready to use all these technologies effectively to deliver the curriculum and to enhance and facilitate student learning.

Subcomponent 1.4. Supporting the Implementation of High School Reform

15. The objective of this subcomponent was to support teachers and schools in the implementation of the 12-year general education system and high school reform, including the curriculum and assessment reforms, so that school leavers will be equipped with knowledge and skills comparable to those of 18-year-olds in the Organisation for Economic Co-operation and Development (OECD) countries.

Component 2: Supporting Tertiary Education Reforms in the Context of the Bologna Agenda (total estimated IDA financing US$6.02 million; actual IDA financing US$6.48 million)

Subcomponent 2.1: Establishment and Strengthening of the National Quality Assurance System

16. The objective of this subcomponent was to establish and strengthen the capacity of the National Quality Assurance System for tertiary education.

Subcomponent 2.2: Developing a Tertiary Education Management Information System

17. The objective of this subcomponent was to support the creation of an integrated Tertiary Education Management Information System (TEMIS) as an effective tool to support policy planning, management, monitoring, and evaluation of performance of the sector both at the national and university levels.

Subcomponent 2.3: Strengthening the Capacity to Implement a Sustainable Financing System

18. The objective of this subcomponent was to support the government in defining and putting in place a sustainable financing system for tertiary education.

Subcomponent 2.4: Reforming Pre-service Teacher Education

19. The objective of this subcomponent was to support the development of initial teacher education in the spirit of the Bologna Process through (a) longer-term twinning arrangement between the Armenia State Pedagogical University (and the Consortium of Pedagogical Institutes, including Yerevan State University) and an international HEI, and (b) upgrading necessary teaching facilities and equipment in these institutes to support modernization of primary school teacher education.

6

Component 3: Project Management, Monitoring, and Evaluation (total estimated IDA financing US$3.25 million; actual IDA financing US$1.71 million)

20. The key objectives of this component were to (a) provide continued support of mainstreamed arrangements for the management, and administration and monitoring of project activities and (b) institutionalize key management functions in the MoES covering policy planning, and M&E.

1.6 Revised Components

21. Under the restructuring that was signed by the Regional Director on August 7, 2012 (and countersigned by the government on February 5, 2013) the following subcomponents were revised:

(a) The coverage of subcomponent 1.1 was expanded as a result of the high demand and commitment of communities to expand preschool education.

(b) Subcomponent 2.1 was expanded to include state universities not originally covered by the project, and to increase the individual grant sizes given that additional resources were required to build an Internal Quality Assurance (IQA) System in large institutions.

(c) Subcomponent 2.3 was revised to drop the design of the student loan scheme (the government considered that the introduction of a student loan scheme should be implemented under better socioeconomic conditions) and to expand the scope of ‘developing innovative financing mechanism through the Competitive Innovation Fund (CIF) in support of quality improvements and innovations in tertiary education institutions.’ While the original project design envisaged only the development of the CIF with implementation to take place at a later stage, following the design of the CIF, the government requested the Bank’s support to pilot the CIF under this project.

22. Consistent with the above changes, the project’s monitoring arrangements and Results Framework were adjusted, and US$1.5 million originally allocated to subcomponent 2.3 were reallocated across other activities as detailed in annex 13.

1.7 Other significant changes

23. In addition to the changes described above, the following significant changes took place.

24. Under the second restructuring that took place in 2014, the project closing date was extended by 12 months to November 30, 2015 to allow for sufficient time to (a) pilot the CIF and carry out grant audits at completion, and (b) direct existing preschool micro-project savings toward the expansion of preschool education micro-projects and high school refurbishment.

25. A number of project activities originally to be financed by the project were financed by non-project sources, as decided by the Prime Minister’s Office. These

7

activities, primarily trainings, workshops and consultancy assignments, were implemented using national and local resources. The shifting of activities to be covered under other national and local sources freed up funding for project-initiated and supported activities that were not covered by other sources.

2. Key Factors Affecting Implementation and Outcomes

2.1 Project Preparation, Design, and Quality at Entry

26. Adequacy of government commitment. The project addressed the government’s priorities and counted upon its continued commitment and ownership at the central, marz, and community levels. Armenia’s SDP covering 2009–2012 set the improvement of the education sector as one of the key development priorities. The Strategic Program for Reforms in Preschool Education of the Republic of Armenia (2008–2015) was further committed to strengthening the quality of primary and secondary education through improving the school readiness of children entering primary education. Project design was also aligned with the State Program for Establishment of the High School System, approved in 2008, that aimed to create a more diversified upper secondary education. Armenia’s participation in the Bologna Process reflected the country’s commitment to reform its tertiary education in alignment with international standards.

27. Soundness of the knowledge base and implementation experience. Project design benefitted from the 2008 World Bank Public Expenditure Review for the Education Sector with particular focus on preschool and high school education, and other analytical studies and reports. 7 Lessons from the APL 1 and from international experience informed project preparation and design, including lessons from the piloting of the preschool micro-projects and high school reform, the student loan scheme, and CIFs.

28. Coherency within the APL framework. As discussed in section 1.1, the APL 2 was part of the Bank’s longer-term support to the government’s Education Sector Reform Program 2003–2013. The project was responsive to the reform needs in the sector and closely aligned with government and CPS priorities. The project built upon the achievements of the first phase of the APL and its objectives were consistent with the program’s objectives as were outcome indicators.

29. Adequacy of the preparation process. Project preparation was highly participatory and benefitted from extensive consultations with key stakeholders (including other donors and the private sector) and direct beneficiaries. The Bank team comprised seasoned technical staff with expertise in key areas such as general education, early childhood development, and higher education. A Quality Enhancement Review was conducted and the advice of the Quality Enhancement Review Panel was integrated into project design, including the formulation of the PDO.

7Armenia: Choices in Development Policy 2008–2012 (2008); Armenia: Thematic Analyses 2008–2012 (2008); Early Childhood development: From Measurement to Action (2007); Quality Assurance in Higher Education in Republic of Armenia: Analyses and Recommendations (2008, draft paper); and Formulating the Financial Principles and Exploring Alternative Financing Mechanisms for Armenian Tertiary Education (2008, draft paper).

8

30. Project design, including the project’s organization, objectives and components, were realistic and in line with the objectives and goals of the program. The risk assessment assumptions made at appraisal reflected a good understanding of the education sector and appropriate mitigating measures were incorporated to address the risks identified at the preparation stage.

31. The quality of linkages between the PDO, PDO-level indicators, intermediate outcome indicators, and certain activities varied across indicators. The linkages related to the “Improving the school readiness of children entering primary education” and “Supporting the integration of the tertiary education system into the European higher education area” portions of the PDO were relevant and strong. However, while the corresponding indicator to measure school readiness was an appropriate one, the use of the Unified Entrance Examinations and TIMSS 2011 were not appropriate indicators, as discussion in Section 2.3 of this document.

32. A second shortcoming with regard to project preparation was the use of a disbursement condition affecting the entire credit that required all component and subcomponent heads to be in place at the Center for Education Projects (CEP). Absence of these key staff members may have been a signal that the project was not fully ready for implementation upon signing.8

2.2 Implementation

33. Project implementation initially got off to a slow start. The Credit was signed on June 5, 2009, and became effective on October 2, 2009. The disbursement condition mentioned in section 2.1 was met when the higher education component head was hired in January 2010. For almost two years, up to September 2011 (at which point 7 percent of the Credit had been disbursed), implementation was generally slow, being affected in part by the Bank’s failure to provide timely technical assistance and No Objections. Following the September 2011 implementation support mission, project activities ramped up and significant implementation progress was made.

34. The Midterm Review (MTR) in December 2012 took stock of the project’s status and highlighted key achievements and challenges. Project implementation progress was discussed with a broad range of stakeholders, and the Bank team evaluated the progress of project activities. The MTR assessed that the progress toward the achievement of the PDO was Moderately Satisfactory considering that the EDI of students in preschools and KG receiving grants relative to students in a control group, and Armenia’s standing on key indicators in the Bologna Scorecard improved, while the performance of Armenian students in TIMSS 2011 as compared to 2007 and 2003 decreased. Intermediate indicators were mostly on track indicating that achievement of the PDOs was likely. The MTR assessed overall implementation progress as Satisfactory considering that most of the core project activities were on track toward achieving the intended results. The MTR also identified project areas requiring fine-tuning through a restructuring, which was countersigned by the government on February 5, 2013. At the time of the MTR, the project had disbursed 49 percent of the Credit.

8 The project could not disburse until component and subcomponent heads were hired in the Center for Education Projects (CEP), which served as the Project Implementation Unit (PIU).

9

35. The first project restructuring that was signed by the Regional Director on August 7, 2012 (and countersigned by the government on February 5, 2013) clarified a number of indicators in the Results Framework and adjusted project activities to respond to changes in government priorities. As part of this restructuring, some project indicators were revised to accommodate the changes to Subcomponent 2.3 as discussed in section 1.6, to adjust the intermediate target values to reflect the implementation process, and to reword intermediate indicators to support appropriate measurement at project completion. Importantly, the restructuring allowed for the Credit to be used to address key government priorities that were in the beginning stages of development, such as the implementation of the preschool micro-projects and the pilot CIF, while at the same time providing the government with the opportunity to use other national and local resources to institutionalize activities for which the project had already built momentum.

36. The timeline and composition of project activities were adjusted during implementation. In response to the Prime Minister’s Office instructions, certain project activities were no longer supported under the project but were instead supported using other national resources, thus freeing up project funds to be used for other key activities.

37. A second project restructuring that was signed by the Regional Director on September 10, 2014 (and countersigned by the government on October 28, 2014) replaced one PDO indicator and extended the project closing date. The project was designed to be implemented over five years but benefited from a 12-month extension of its closing date as discussed in section 1.7, allowing for sufficient time to successfully complete activities, including piloting of the CIF, increasing the number of preschools benefitting from micro-projects from 140 as originally envisaged to 280, and expanding the number of high schools benefitting from refurbishment from 105 as originally planned to 195.

38. At project closing, most project outcomes and outputs had been achieved, with a few moderate shortcomings. The project was never at risk and at project closing, more than 95 percent of project funds had been disbursed. However, although TEMIS was developed and functionally available to use at project closing, official requirements had not been introduced to ensure that TEMIS would, in fact, be used (for example, there was no mandate for universities to enter data into the system or to create reports using the entered data; there were no deadline requirements for entering the data into TEMIS or for using the data). Therefore, although TEMIS was successfully developed under the project, at the time of this review it was not being used in a comprehensive or heterogeneous manner across universities or by MoES for the purpose of producing reports or planning. Discussions are underway to identify the best approach to ensure the use of TEMIS particularly in light of existing data management responsibilities at the universities. Additionally, the target of 7 public and 7 private universities evaluated against the newly developed QA standards, criteria, procedures, and guidelines was partially achieved as 10 public but only 5 private universities were in fact evaluated. It should be noted, however, that the five private universities that were evaluated under the project are considered to be the main private universities in the country.

2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization

39. M&E design. The PAD laid out the basic arrangements for data collection and results reporting. The indicators, including the PDO-level indicators in the M&E framework, were

10

largely relevant and adequate to measure the results chain, and implementation arrangements for M&E were identified in the PAD. At the same time, a number of issues were observed. Baselines and targets for most of the indicators were not set as of project effectiveness, and a provision for their identification was not included; by project closing, two baselines were still not set. Project documents prepared during implementation highlighted the need to prioritize the revision and consolidation of the M&E and the Results Framework, which was indeed done during both of the restructurings. As a results, two PDO indicators (“share of test-takers passing the 9th grade graduation exam” and “gross graduation rate in upper secondary education”) were introduced as part of the second project restructuring. Although the “share of test-takers passing the 9th grade graduation exam” and the “gross graduation rate in upper secondary education” were already relatively high prior to the implementation of project interventions, and were also likely influenced by factors outside of the scope of the project, the ICR team agrees that they were the best available measures that allowed to capture the improved relevance and delivery of education.

40. M&E implementation. Appropriate data was generally collected in a timely manner. Project indicators were regularly monitored during implementation, and reported on in the Implementation Status and Results Report (ISRs). The two restructurings mostly addressed the limitations of the M&E design, further defining indicators, baselines, and targets, and as discussed in section 2.2, making the Results Framework more appropriate for monitoring project outputs and outcomes. Adjustments were also introduced to better capture the PDO progress based on the Bologna Scorecard by identifying and monitoring the nine indicators that were consistent across the scorecards that were published in different years (annex 14). At the same time, the following issues remained. Indicators related to teacher training were not dropped as part of the first restructuring, although the associated activities were dropped. Also, the PDO indicator “Improved EDI scores of students in KG and schools receiving the grants with regard to students in the control group” was monitored using a series of studies that measured the development outcomes of children before and after their one-year participation in preschool and of a control group of children, but the development outcome of the control group was measured only once, at the time when the treatment group finished one year of preschool. This methodology presents limitations as it does not allow for the measurement of the differentiated gains between participants and non-participants between two time points. Finally, it is not clear why the PDO indicator “Improved examination and tests scores in project-targeted schools (TIMSS 2011 and Unified Entrance Examination)” that was ultimately dropped as part of the second restructuring was not addressed in the first restructuring.

41. M&E utilization. Timely data were collected and used to monitor and improve project performance and results on the ground. The M&E system strengthening went beyond the project itself. NaCET, which is responsible for the Educational Management Information System (EMIS), has substantially strengthened its capacity to collect, analyze, and report sector performance indicators. Its website acts as a useful data portal, and its annual statistical report provides the most up-to-date information on sector achievement and challenges. M&E arrangements are expected to be sustained at the CEP, which continues to carry out M&E activities of the Armenia Education Improvement Project (EIP) (P130182) that was approved by the Board on March 13, 2014.

2.4 Safeguard and Fiduciary Compliance

11

42. The project was classified as category C for its low impact on the environment. No safeguard policies were triggered.

43. Fiduciary Management. The financial management arrangements in place at the CEP including budgeting and planning, accounting, financial reporting, flow of funds, external audits, organization and staffing arrangements were overall satisfactory and acceptable to the Bank during the entire life of the project. The CEP maintained an adequate internal control system, while the few minor deficiencies in the internal control system observed during project implementation were effectively and successfully addressed by the CEP. The project’s fiduciary management was consistently rated satisfactory during the life of the project. The Interim Financial Reports of the project were submitted on time, reviewed, and found to be acceptable to the Bank. The auditor issued unmodified (clean) opinions on all financial statements, which were received on time, and no major issues were raised by the auditor in the management letters. Similarly, all CIF grants’ financial statements received unmodified (clean) opinions.

44. Procurement. The procurement process was generally considered moderately satisfactory for the majority of project implementation. Delays in the Bank’s no objection before September 2011 slowed the procurement process. Additionally, issues such as the lack of competition in procurement (which resulted in rebidding in some cases) affected procurement and contract implementation. At times, the MoES had difficulty filling key staff positions, and staff members had to take on responsibilities in addition to their primary job duties. With the support from the Bank team particularly following September 2011, the CEP addressed and resolved procurement issues in an efficient and effective manner.

2.5 Post-completion Operation/Next Phase

45. The transition arrangements for the operation of key project activities are adequate and comprehensive. A large number of activities have been incorporated into a follow-up operation, the Education Improvement Project (P130182) approved in March 2014, particularly those requiring long-term support to produce the expected outcomes, such as in the areas of preschool and upper secondary education, curriculum revisions, integration of information and communication technology (ICT), and the mainstreaming the CIF. Importantly, the government has mainstreamed key activities and initiatives that were started under the project: it took over the financial responsibilities for preschools (or communities in the case of KGs) following the project’s initial one-year support to preschool micro-projects; absorbed the oversight for training and capacity building of teachers; and supported the development and implementation of a student loan scheme that is benefitting students who would otherwise not be able to afford higher education. The government is also financing all of the work associated with the QA systems (at the ANQA and at universities) that were developed under the project. Thus, the government has effectively used project resources to initiate reforms, while putting in place the necessary institutional, capacity, and financial arrangements to sustainably implement these reforms beyond project closing.

3. Assessment of Outcomes

3.1 Relevance of Objectives, Design, and Implementation

12

46. The relevance of project’s objectives, design and implementation is rated as High. The project’s priorities were consistent with the government’s priorities and with the Bank’s CPS when it was prepared, and continue to be highly relevant and consistent with the country’s development priorities as specified in the Armenia Development Strategy (ADS) for 2014–2025 and the Bank’s 2013 CPS for the Republic of Armenia FY2014–2017. In the ADS, Armenia recognizes the importance of quality human resources for ensuring long-term economic development.9 Similarly, the Employment Strategy 2013–2018 of the Republic of Armenia aims to improve education and training to address the constraints to job creation and the bottlenecks for relevant skill formation. The latest five-year sector development strategy, the Education Development 2016–2025 State Programme of the Republic of Armenia, also continues to support education objectives at all levels of education, confirming the high relevance of project objectives and interventions. The project’s design was aligned with key policy priorities in the country and the overarching objectives of the government program that was supported by the APL. Throughout its implementation, the project supported the necessary capacity building of key stakeholders, ensuring that its activities were completed successfully during the project and were sustainable beyond project closing. Project supported activities were adequately adjusted in response to the country’s changing circumstances and priorities, and remained relevant throughout the project. The Bank’s implementation assistance was responsive to project needs and priorities as well. Thus, overall, project activities were relevant at the design stage and continued to be relevant during implementation.

3.2 Achievement of Project Development Objectives

47. The achievement of the PDO is rated as Substantial. Achievements of the aspects of the PDO are addressed below and detailed in annex 15.

48. Enhancing school learning in general education. The achievement of this objective is rated substantial as measured by the availability and use of educational inputs in schools and the impact of teacher training in the classrooms, leading to tangible improvements in student passing and graduation rates. At project closing, the associated PDO indicator “share of test-takers passing the 9th grade graduation exam” increased from 96.7 percent to 99.8 percent, a small but important change considering the increasing difficulty of reaching the last five percent of students. The original end-of-project target was 99 percent. Similarly, the “gross graduation rate in upper secondary education” reached its target of 90 percent at project closing, from 85.95 percent at baseline.10

9The relevant ADS goals to which the project aligns include improving the quality of the general secondary education; improving the quality and accessibility of preschool education; assuring the quality of general education and the implementation of the mission of upper secondary schools, including through refurbishment and support for educational materials; and improving the quality and streamlining professional education to ensure its alignment with the demands of the labor market and the country’s strategic educational and scientific development goals, including the provisions of Bologna and Turin Agreements. 10The PDO indicator “share of test takers passing the 9th grade graduation exam” is a measure of quality that informs this objective. The PDO indicator “gross graduation rate in upper secondary education” is also a measure of quality and systemic efficiency, and the interventions under Subcomponents 1.2, 1.3, and 1.4 improved the quality, relevance, and delivery of education to upper secondary school students, more of whom have graduated from upper secondary education that has been improved by the project.

13

49. Improved quality of teaching in general secondary school classrooms was achieved through support for the training of all teachers in professional development and innovative teaching methods. As a result of the training, the teaching style and behavior of teachers in the classroom changed. For example, by project closing, the majority of trained teachers were applying innovative teaching methodology in the classroom; and the percentage of teachers solely using the presentation/recitation mode of teaching decreased from 43 percent in 2011 to 32 percent in 2012.11 By 2015, 74 percent of teachers were intensively using computer and other digital materials, as compared to 25 percent in 2008. Twenty-five percent of principals were also trained by the Union of School Centers on designing and implementing school development plans. This training has been distributed to all schools and made publically available on the NaCET website. However, information was not collected to assess how principals have applied what they learned in their trainings, particularly with regard to school development plans. Importantly, project activities in support of this objective helped to develop Article 26 of the Law on General Education of Armenia regarding the provisions of teacher attestation in the country, as well as the Decree on Awarding Ranks to School Teachers and its subsequent revision.

50. With project support, resource centers (which include laboratory and library materials, as well as appropriate furniture and equipment) were established in all high schools, and all high school students visited them. Schools also benefitted from furniture, computer labs, and printers. Internet connectivity and computer resources supported through the project have made possible the introduction of an educational network, connecting all public schools and 22 vocational education institutions via the Intranet. An updated version of the School Management Information System (SMIS) has been installed in all public and some private schools, and NaCET has trained approximately 1,388 school representatives, mostly principals, on the use of SMIS software.

51. Improving the school readiness of children entering primary education. Achievement of this objective is rated as high. Specifically, the project provided grants to support one preschool in each of the 280 communities, including in poor and disadvantaged communities. Although during the project preparation it was envisaged that 140 communities and 8,750 children would benefit, the project exceeded its objective, and 280 communities and approximately 35,451 children benefitted as it was possible to complete some micro-projects with less financing, therefore freeing up resources to significantly expand the coverage of this intervention to other communities. As of project closing, all of the preschools continued to be fully operational. The Ministry of Education, United Nations Children’s Fund (UNICEF), and Step-by-Step Foundation provided the necessary training to preschool teachers and marz administrative staff.

52. In meetings with parents and marz authorities, they reported that the school readiness of their children after participating in the preschool micro-projects was high. Albeit the measurement limitations discussed in section 2.3, consistent with this anecdotal evidence are the findings of a survey of the students, which indicate that, on average, the EDI score was

11By 2013, the government mainstreamed the NIE’s responsibility for teacher training to maximize the efficiency of public funding, particularly because the NIE was already mandated with teacher training activities and it had full capacity supported by public funding to carry out those activities. As a result, the NIE became solely responsible for teacher training and benefitted from the teacher training materials developed under the project.

14

19 percent higher in the treatment than in the control group. Importantly, the project activities in support of this objective have been sustained through the Strategic Program for Reforms in Preschool Education of the Republic of Armenia (2008–2015), which includes per-student financing arrangements that ensure the sustainability of preschools beyond the project.

53. Supporting the integration of the tertiary education system into the European Higher Education area. The achievement of this objective is rated as modest strictly on the basis of the corresponding outcome indicator having only partially achieved its target (Armenia’s standing in the Bologna Scorecard according the published stocktaking reports of the Bologna Process). Despite important developments in the area of higher education in the country, 5 out of 9 indicators were green at project closing, compared to the target of 75 percent of indicators being green (i.e. at least 7 out of 9 indicators).

54. At the same time, the ICR team notes that the project’s contributions to Armenia’s tertiary education system with not much more than US$8 million were remarkable, including the accreditation of 15 HEIs, the establishment of Internal Quality Assurance Units in all 26 HEIs, the hosting of the Bologna Ministerial Conference in Yerevan for the first time outside of continental Europe, and the introduction of TEMIS in 183 higher and vocational education institutions. Other project achievements include support for higher education financing, revision of teacher education curricula, and not only the design but also the piloting of the CIF. Specifically, the project provided technical assistance for the development of a higher education financing strategy in 2011 and, in combination with the government’s support for student loan activities, by project closing 14 commercial banks had been selected to provide loans to students on various terms. Furthermore, as a result of the partnership between Oulu University and Armenian State Pedagogical University all pre-service teacher education curricula in Armenia have become comparable to a typical Bologna program. Importantly, the CIF projects allowed university students to gain key exposure to subjects and topics that were directly relevant to the labor market, and the teaching employed within the context of the CIF projects utilized the latest technologies and innovations, thus improving the quality and relevance of student learning. The 10 grantee universities successfully implemented their projects.

55. Furthermore, one of the original five triggers from the originally-envisaged APL 2 to APL 3, “New policy and implementation plan for cost-efficient reform of student social and support services agreed with the Bank and approved by the government,” that was not met at appraisal was met as part of the Higher Education Financing Strategy approved by the Government on June 30, 2011. The subsequent Criteria for Assessment of the Students’ Financial Capacity and Student Financial Support Program and associated regulations were approved by the Government in 2012. As a result, beginning in the 2013-2014 school year, students from socially vulnerable and poor families have benefited from financial support from the Government in pursuing higher education. Depending on the needs of students’ from vulnerable or poor families (as determined by the Ministry of Labour and Social Services), students receive 50 percent to 100 percent financial support from the Government in pursuing higher education. Twenty percent of students in higher education institutions benefit from such support and approximately 4,000 students have benefitted from the program since its inception. The Law on Higher Education of 2004 is still in force and mandates that higher education institutions provide scholarships (of up to 30 percent of tuition fees) to up to ten

15

percent of the student body that has financial limitations in pursuing higher education opportunities.

56. On the TEMIS, the ICR team notes the following shortcomings: The initial national report on higher education statistics generated through TEMIS was originally envisaged to be available at the end of the academic year 2015–2016; however, only some institutions have started data entry, and in many cases, the data entered is incomplete. It is not yet mandatory for universities to use TEMIS and the government is currently reviewing options to mandate its use. Thus, the intermediate indicator of 5 public and 2 private universities producing/updating a strategic plan and self-assessment reports using information collected through TEMIS was not met by project closing. The information provided by TEMIS at project closing was also not adequate for the MoES’ annual report on higher education for planning and budgeting purposes. Discussions are underway to identify the best approach to ensure the use of TEMIS particularly in light of existing data management responsibilities at the universities.

57. Overall, the project’s contributions to Armenia’s tertiary education system exceeded expectations but due to the fact that the related outcome indicator was not met, the ICR team feels bound to rate this portion of the PDO as modest.

3.3 Efficiency

58. Efficiency is rated as Modest.

59. The economic and financial analysis provided in the PAD included limited data and did not attempt to quantify the Economic Rate of Return, the Net Present Value, or conduct an expected or projected benefit/cost analysis for project components or subcomponents. Instead, the economic and financial analysis section in the PAD presented a general discussion on the expected (a) cost-effectiveness of investing in preschool education, (b) cost-effectiveness of investing in the ICT network and its integration in the teaching and learning process, and (c) cost-efficiency and quality improvement of high school reforms.

60. The project was expected to benefit 140 communities and 8,750 children in total over 5 years. In fact, 280 communities and 11,694 children benefitted from the first year of preschool that the project supported. The government and communities absorbed the costs of the preschools after the first year of project support. At project closing, all preschools and KGs that were supported by the project in their first year were still operating. Overall, a total of 35,451 children benefitted from the existence of a preschool or KG in their community over the life of the project. The expansion of coverage to more beneficiaries was due to the more efficient use of resources.

61. The ICR team conducted a benefit/cost analysis for the investment in preschool education.12 In general, preschool education is associated with better cognitive and non-cognitive skills that create better outcomes in the labor market and adult life indicators (such as better health, lower criminality, better-informed voters, and so on). The model used in this

12 The benefit/cost analysis was not conducted for other subcomponents as they were primarily capacity building in nature and a quantifiable analysis would have required making many arbitrary assumptions, undermining the accuracy of conclusions.

16

analysis assumes that the expansion and quality enhancement of pre-primary education would lead to more years of schooling (including higher education graduates). Children would also be more likely to finish high school and progress to higher education, which in turn would lead to higher productivity and income, and less unemployment. The benefit of this subcomponent comes from the higher expected income generated by a more qualified and educated workforce.

62. In quantifying these benefits, it is found that one year of preschool correlates with a half-year increase in schooling. 13 Using the 2013 (latest available) Integrated Living Conditions Survey data, the returns to one more year of schooling were found to be 2.8 percent for males and 3.6 percent for females. It is calculated that a half-year increase in schooling—the potential benefit of one year of preschool—results in a benefit/cost ratio of 5.04 for males and 6.83 for females. In general, this is an underestimate of the benefit given that these calculations reflect only private returns. Social returns to education can be more important in some cases. A more detailed analysis is presented in annex 3.

63. The benefit/cost ration estimates are close to the benefit/cost ratios of preschool interventions in other countries. For example, the benefit/cost ratio of similar early childhood development interventions in the Mexico Compensatory Education Project (P101369) was found to be 6.47. In the Indonesia Early Childhood Education and Development Project (P089479), the benefit/cost ratio was found to be 4.3 (these projects did not conduct benefit/cost analysis by gender). The benefit/cost ratios for the APL 2 are also consistent with those of the High/Scope Perry Preschool Study (Executive Office of the President of the United States 2014)14 which has a benefit/cost ratio of 6.2.15

64. The actual costs of various project activities are in general aligned with those estimated at appraisal, with the exception of activities related to the preschool micro-projects, where the costs per micro-project were lower than anticipated, allowing for significantly more communities and more children to benefit from the micro-projects as discussed in section 3.2. Additionally, project implementation was very efficient particularly given that the project was implemented with administrative costs that were almost half of the original projections.

65. Although not quantifiable at this stage, overall sector efficiency gain is likely a result of the two activities supported by the project: high school reform and tertiary education improvement. For high school reform, efficiency gains are likely to come from the economies of scale that have resulted from strategically grouping high school students and teachers, leading to more efficient use of resources and facilities, such as laboratories and libraries. At the tertiary level, structural changes particularly with regard to QA will lead to better institution management and accountability to public and private resources, and eventually

13 World Bank. Investing in Our Children’s Future, From Early Child Development to Human Development. http://elibrary.worldbank.org/doi/pdf/10.1596/0-8213-5050-1, p. 133. 14 “The High/Scope Perry Preschool Study provided preschool to low-income African-Americans at high risk of failing in school in Ypsilanti, Michigan during the 1960s. Perry is one of the most well-known preschool interventions in part because it was evaluated using a randomized trial yielding highly credible results, and also because data on its participants have been routinely collected longer than most other evaluations on any subject—until participants were 39 to 41 years old (Schweinhart, Barnes, and Weikart 1993.” 15 Heckman J. J., S. H. Moon, R. Pinto, P. A. Savelyeva, and A. Q. Yavitza. 2010. “The Rate of Return to the High/Scope Perry Preschool Program.” Journal of Public Economics 94, 114–128. doi:10.1016/j.jpubeco.2009.11.001.

17

more relevant and higher quality programs that can produce more qualified human resources in Armenia. The piloted CIF serves as an important instrument to stimulate innovation and quality enhancement in HEIs. It also contributes to the goal of diversifying the funding mechanism for better sector outcomes and improved efficiency.

66. At this stage, as the efficiency of a number of key project investments cannot be evaluated quantitatively, the efficiency for outcome achievement is rated as modest.

3.4 Justification of Overall Outcome Rating Rating: Moderately Satisfactory

67. The project’s overall outcome is rated Moderately Satisfactory. Project objectives were and continue to be highly relevant for the country context and are in line with the government’s development and education priorities, as well as with those laid out in the CPS. Project resources were used effectively and efficiently to initiate reforms and to introduce the necessary institutional, capacity, and financial arrangements to permanently and sustainably implement the reforms beyond project closing. At the same time, despite substantial achievements with regard to the project’s objectives, there were minor shortcomings in implementing planned activities and some targets were not achieved.

3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development

68. In Armenia, individuals who have a better education or are in the labor force are less likely to be poor or extremely poor. The project’s investment in improved access to and in the quality of education at all levels supported reforms that decreased the likelihood that project beneficiaries will experience poverty. Increased access to preschool for children in poor areas supported by the project has also likely benefitted the families who now have access to free childcare and the time to pursue employment or education opportunities. The preschool subcomponent also contributed to community development and set the stage for community development initiatives on a larger scale, such as those that are currently supported under the Social Investment and Local Development Project (P148836). Specifically, all communities with preschools/KG participating in the preschool micro-projects contributed 25 percent (cash or in-kind) of what the APL 2 provided to the micro-projects, helping communities strengthen their capacity in planning, implementing, and monitoring activities for further improvement of the preschools and of the communities to ensure sustainability.

69. Although the project did not directly target gender, it nevertheless directly benefited women: access to preschool has allowed mothers to pursue work or other personal or professional development activities, and teachers, most of whom are women, benefitted from teacher training opportunities under the project. Rates of return to education from the additional year of schooling were found to be higher for women than for men (3.6 percent for women versus 2.8 percent for men), therefore girls who attended preschool under the project are likely to receive a higher increase in earnings as compared to boys. A review of the gender aspects in select textbooks financed by the project also found that the representation of gender roles could be modernized, and provided recommendations for improving gender inclusiveness in the textbooks.

18

70. The project’s positive impact on institutional capacity and institutional development, particularly on the long-term development of Armenia’s capacity and institutions, has been substantial. The project supported the development of funding schemes, such as per capita financing, to ensure the sustainability of financing for preschools beyond the support of the project. The CIF funds flow mechanism has also introduced a culture of innovation and healthy competition in higher education, setting the stage for possible further diversification of financing mechanisms for higher education. It has strengthened the role of universities as drivers for change, initiating and implementing innovative solutions in areas in which they can use their expertise to affect change. The institutional capacity of ANQA, NaCET, the MoES, and universities has been strengthened through well-tailored training, and international TA and collaboration. Finally, the capacity of the CEP has been significantly enhanced over the course of the project through Bank and external training, as well as from on-the-job experience.

(b) Institutional Change/Strengthening

71. As discussed throughout section 3, the project resulted in the strengthening of key institutions, including of preschools, general, secondary, and tertiary education institutions, as well as of communities, marz and central education authorities, and key sector institutions including the NIE, NaCET, and ANQA.

(c) Other Unintended Outcomes and Impacts (positive or negative)

Not available

3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops

72. Stakeholder workshop and consultation sessions were carried out in September 2015 to learn the experiences and perceptions of project beneficiaries and to discuss lessons learned. The sessions highlighted that parents of preschool children value the preschools for the knowledge and skills that are taught, as well as for freeing up their time to pursue other livelihood opportunities. High school representatives (teachers, principals, students, and parents) noted that the computer and technology devices provided to schools have helped teaching and learning, however they also expressed that additional support is needed for the maintenance and training on the use of these technologies. Marz education authorities praised the project’s support to preschools, and highlighted a high demand for more preschool classrooms to accommodate all children in the community. This sentiment is supported by official statistics as over the course of the project, the number of children who attended pre-school institutions increased from approximately 53,900 in 2009 to approximately 72,700 in 2014 (latest available data), indicating an increase of approximately 35% in the number of children attending preschool institutions.16 University representatives praised the innovation centers created under the pilot CIF, noting the successful implementation of high quality initiatives, and noted the need for support for certain standard procedures, such as procurement. Teacher training providers expressed the need to apply a more tailored approach

16 National Statistical Service of the Republic of Armenia. 2015. Statistical Yearbook of Armenia (http://www.armstat.am/file/doc/99493613.pdf); National Statistical Service of the Republic of Armenia. 2012. Statistical Yearbook of Armenia (http://www.armstat.am/file/doc/99471448.pdf).

19

to teacher training, expanding the amount of time allocated for the training and introducing follow-up evaluations of teachers. A detailed discussion of the sessions is provided in annex 6.

4. Assessment of Risk to Development Outcome Rating: Moderate

73. The follow-up EIP is supporting the expansion of the original objectives and activities of APL 2, and the government and local communities have also absorbed the financial and implementation responsibilities for project activities. However, certain risks remain, including the following:

The macroeconomic risk in the region may affect the development outcome. Growth rate has slowed to 2.1 percent in 2015, remittances have decelerated in the second half of 2014, and the demand for imported consumer goods has declined. Thus, there may be a risk that funding for sustaining education initiatives that were started under the project will not continue within the context of a tighter fiscal envelope.

The sustainability of preschools is dependent on financing. Thus, for the preschools that do not receive per-student financing from the government, there is a risk that the communities will not be able to sustain them if there is an economic or financial crisis in the country or in a particular region.

There is a risk that universities may not provide updates to the TEMIS database on a regular basis. At this point, there are no mandates or incentives to use TEMIS.

With regard to the pilot CIF, although applicant universities have to specify the provision for financing to sustain the CIF projects upon graduation from project support, there is a risk that these universities may not be able to sufficiently invest and maintain the operation of the innovation projects after several years.

5. Assessment of Bank and Borrower Performance

5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Satisfactory 74. The overall objectives of the project were informed by extensive consultations with key stakeholders and aligned with the country priorities and the objectives and challenges in the sector. Project design drew on lessons learned from in-country experience, particularly regarding the piloting of the preschool micro-projects and of the high school reform under APL 1, as well as from international experience in the areas of preschool education, student loan schemes, and CIFs. The Bank team had an adequate skill mix that represented a wide range of sector and technical expertise, including specialized expertise in education

20

economics, early childhood education, higher education, and ICT. The project filled an important financing gap in the education sector.

75. Despite these positive features, there were some shortcomings which influenced quality at entry, particularly those related to M&E design that were discussed in section 2.3 as well as the inclusion of a disbursement condition (that was in effect the effectiveness condition) as discussed in sections 2.1 and 2.2. On the basis of the information discussed above, Bank Performance in Ensuring Quality at Entry is rated Moderately Satisfactory.

(b) Quality of Supervision Rating: Moderately Satisfactory

76. The Bank team comprised staff and consultants, both local and international, with the appropriate skills mix with regard to operations, the education sector, procurement, fiduciary management, and M&E. However, the borrower noted that during approximately the first two years following effectiveness, the Bank team was slow in responding to PIU correspondence, including to requests for No Objections, with delays of sometimes nearly two months.

77. The decisions to restructure twice showed responsive supervision. The restructurings allowed for the introduction of important improvements to project design, adjusting the scope of project components to address the government’s priories in a timely manner and foreseeing the need for additional time to properly implement key project activities. On the basis of the information discussed above, Quality of Supervision is rated Moderately Satisfactory.

(c) Justification of Rating for Overall Bank Performance Rating: Moderately Satisfactory

78. Based on the Bank performance in ensuring quality at entry and the quality of supervision, and the overall outcome rating, the rating for overall Bank performance is Moderately Satisfactory.

5.2 Borrower Performance (a) Government Performance Rating: Satisfactory

79. From the early stages of project preparation, the Government, particularly the MoES, demonstrated a high level of commitment to working with the Bank to address the issues of quality, access and equity in the education sector, and to ensure a coordinated approach among the various partners and activities. This was evidenced by the fact that the minister and the deputy ministers of the MoES guided the project effectively at all stages of the project cycle, and closely collaborated with the Bank. The minister personally participated in the launch of many project activities, ensuring the full engagement of stakeholders and of project visibility. Counterpart funding was consistent and timely, even during fiscal pressures. Furthermore, as discussed in section 1.7, the Prime Minister’s Office instructed that some project activities were to be funded by domestic resources to ensure the most effective use of the Credit, and the MoES proactively responded to the changes. Most of those project activities were shifted

21

to being supported through other government funding, and have been successfully implemented outside of the scope of the APL 2. The government has effectively used project resources to initiate reforms and to build institutional capacity, and is sustaining key reforms beyond the project as discussed in section 2.5. The Ministry of Finance was a collaborative partner, consistently and promptly supporting the implementation of project activities. On the basis of information discussed above, the government’s performance is rated Satisfactory.

(b) Implementing Agency or Agencies Performance Rating: Satisfactory

80. The MoES ensured the overall leadership for the reform process, and its CEP served as an effective PIU. The CEP was committed to carrying out project activities toward the achievement of the development objectives. The transition of CEP directors was seamless and the CEP teams continued to be fully engaged during transitions. The CEP implemented project activities and resolved implementation and procurement issues in a timely manner. The CEP was committed to its fiduciary responsibility, and handled several complaint cases effectively. All records were well kept and consistent with procurement policies and guidelines, and disbursement data was up-to-date. CEP specialists took part in training opportunities, including from the Bank and other organizations.

81. Despite challenges in finding appropriate staff in the country’s labor market, the CEP ensured that it had the necessary technical expertise and efficient administrative capacity to successfully perform its functions. It hired and trained the necessary staff as needed, and it completed its activities effectively and efficiently. The implementing agency performance is rated Satisfactory.

(c) Justification of Rating for Overall Borrower Performance Rating: Satisfactory

82. On the basis of government and implementing agency performance, borrower performance is rated as Satisfactory.

6. Lessons Learned

83. The project provided a number of lessons, with the key lessons summarized below and detailed in annex 16.

The Bank’s long-term engagement in the education sector provides a robust platform for well-performing projects, allowing for the broadening of the engagement in the sector.

Bank-supported project activities can catalyze key sector reforms through close engagement and effective dialogue in general, as well as during project preparation.

22

Within the context of supporting a long-term substantive sector reform, responsiveness to short-term sector challenges is key for sustaining project relevance.

Broad and complex projects can be successfully implemented in the context of timely and quality implementation support from the Bank and a client with high capacity.

The M&E and the Results Framework should incorporate only those indicators that can accurately measure the results achieved through the project.

Relevant, comprehensive and technically well-designed studies should be planned as part of project preparation to monitor and accurately measure project outcomes.

Successful implementation of the Competitive Innovations Fund (CIF) can be ensured through clearly defined roles and responsibilities of the CIF recipients and their timely and relevant capacity building.

Teacher professional development can significantly benefit from learning best practices in collaborative and hands-on settings, such as through international partnerships.

7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies

84. The borrower comments are summarized in annex 7.

(b) Cofinanciers

Not applicable

(c) Other partners and stakeholders Not applicable

23

Annex 1: Project Costs and Financing

The below project cost and financing information is provided as of March 31, 2016.

(a)Project Cost by Component(in US$, million equivalent)

Components

Appraisal Estimate

(US$, millions)

Restructuring Estimate

(US$, millions)

Actual/Latest Estimate

(US$, millions)

Percentage of

Appraisal

Component 1. Enhancing the Quality of General Education

19.66 19.91 20.74 105

1.1.Promoting School Readiness and Equal Opportunities at the Start of General Education

3.11 3.36 3.61 116

1.2.Supporting Further Improvements in the Quality of Education through Improved Teacher In-service Training and Professional Development

4.41 4.41 3.30 75

1.3.Continuing Support to the Integration of ICT in Teaching and Learning Process

4.65 4.65 4.00 86

1.4.Supporting the Implementation of High School Reform 7.49 7.49 9.83 131

Component 2. Supporting Tertiary Education Reforms in the Context of the Bologna Agenda

7.52 7.26 8.09 108

2.1.Establishment and Strengthening of the National Quality Assurance System 3.09 3.34 2.32 75

2.2.Developing a Tertiary Education Management Information System

0.49 0.49 0.26 53

2.3.Strengthening the Capacity to Implement a Sustainable Financing System

2.42 1.91 3.63 150

2.4.Reforming Pre-service Teacher Education

1.53 1.53 1.89 124

Component 3. Project Management, Monitoring, and Evaluation

4.06 4.06 2.14 53

Total Project Costs 31.25 31.25 30.97 99

(b) Financing

Source of Funds Type of

Cofinancing Appraisal Estimate

(US$, millions) Actual/Latest Estimate

(US$, millions)

Percentage of

Appraisal Borrower – 6.26 6.19 99 International Development Association (IDA)

– 25.00 24.78 99

24

Annex 2: Outputs by Component

1. Table 2.1 describes the planned activities, actual outputs, and the rationale behind the changes introduced under the project.

Table 2.1.Activities and Output (Original and Actual and Rationale for Change)

Planned Activities (from the PAD) Actual Outputs and Justification for any Change Component 1: Enhancing the Quality of General Education Subcomponent 1.1: Promoting School Readiness and Equal Opportunities at the Start of General Education 140 poor and disadvantaged communities benefit from targeted grants that will allow them to enhance the quality of preschool education (that is provided in an existing structure or through the establishment of a KG classroom in a local primary school if they do not have an operating preschool).

280 communities unable to create and sustain preschool institutions benefitted from the grants; the number is twice as high as it was planned as (a) some projects were possible to create with less money than initially budgeted and (b) savings under high school teacher training, after approval, were used under this subcomponent, based on the need.

140 poor and disadvantaged communities contribute to the financial sustainability and management of the enhanced preschools, including through in-kind contributions.

The preschool institutions based in schools (202 out of 280) are funded by a state budget based on the per capita financing scheme and stand-alone preschool institutions (78 out of 280) are funded by communities.

Approximately 8,750 children, ages 5–6, benefit from participation in a KG class, either through an enhanced existing preschool or through the establishment of a KG classroom in the local primary school.

At least 20,000 children in their last year of KG benefitted from participation in a KG class.

Training of preschool teachers and administrative staff (and invited Marz Education Department staff) directly involved in the delivery of preschool education.

Training took place for all teachers of established preschool institutions. Principals were not targeted in this training as in the majority of the micro-projects (202), the principal of the school, who received the training for principals, also headed the established preschool. Twenty-five percent of the principals of the schools that have preschools received trainings organized by the Union of School Centers (Subcomponent.1.2).

Training for preschool teachers and administrative staff provided in collaboration with the MoES, UNICEF, and Step-by-Step Foundation.

Ministry of Education, UNICEF, and Step-by-Step Foundation collaborated in the provision of the training.

Subcomponent 1.2: Supporting Further Improvements in the Quality of Education through Improved Teacher In-service Training and Professional Development Develop a teacher professional development system that is based on the teachers’ rights, obligation, and incentives to demand-driven professional development or training.

ABU, a German consulting company, was contracted to support the development of the teacher professional development system. It piloted and implemented the newly developed system.

Create a School Leadership Development System that provides basic knowledge and skills and continuous leadership training for school principals.

International Education Partners, a British private consulting company, was contracted to support the creation of a School Leadership Development System; it was integrated in the current school principals’ training program, which leads to the mandatory test intended for applicants for a school principal’s position.

Develop the National School Improvement Network that provides a supportive professional environment for professional development through school improvement.

Union of School Centers (http://dkm.am/), created under the project, provides a supportive, professional environment for professional development of teachers and principals as well as through school improvement

25

by enrichment of the learning environment. It is financially sustained by the government.

Create opportunities for diversified provision of professional development services for teachers and schools.

Various opportunities were provided to about 12,165 teachers in professional and pedagogical development, including in innovative teaching methods.

Subcomponent 1.3: Continuing Support to the Integration of ICT in Teaching and Learning Process Provide Internet connectivity to about 400 most-remote schools.

As part of APL2, Internet connectivity was provided to 459 schools. The most-remote schools (about 30) were connected through a General Packet Radio Service (GPRS) connection. As a result of efforts/contributions from APL1 and APL2, all 1,459 public schools throughout Armenia have been connected.

Provided schools with hardware for school management purposes.

1,400 schools received hardware for school management purposes.

Provide schools with electronic materials for all key school subjects.

Laboratories in 4 subjects for middle and high schools, including natural sciences and math, were developed and installed on the NaCET server, ready to be accessed and used by schools.

Train teachers in subject-specific use of ICT. Teachers have received subject-specific use of ICT as part of the mandatory pre-attestation training.

Support the National Education Portal as the host of electronic materials, virtual communities of teachers and learners, and dissemination tool for good practice.

According to the Prime Minister’s Instruction No. 099/13279-11 dated 06/27/2011, the activity was no longer funded under the project. The country is considered to be tech-savvy and ICTs are considered to be self-developing. Additionally, an education portal created under APL1 (http://lib.armedu.am/) has been maintained and sustained by NaCET.

TA on the use of ICTs for school management purposes

The SMIS is fully functional. NaCET installed an updated version of the SMIS in all public schools and some of the private schools.

Subcomponent 1.4: Supporting the Implementation of High School Reform Establishment of resource centers in 100 high schools (libraries plus computer labs)

105 high schools have received resource centers. Additionally, two sets of mobile computer stations with teacher management software controls + 11 student computers were procured for the general schools, which offer upper secondary education (grades 10 to 12).

Training of 150 high school librarians Trainings of all 105 high school librarians took place. In 2008, the country was ‘divided’ into educational clusters, where in each cluster there was going to be 1 high school. Thus, even though the intention was to create 150 high schools, 105 were created based on the population forecast and the nomination of schools by marz authorities (each marz was asked to nominate schools to convert into a high school). In rural areas, no high schools were created and instead, the decision was to maintain the upper secondary block of secondary schools there.

Training of 2,750 high school teachers. Approximately 1,400 teachers of 105 high schools received training. The training was no longer funded under the project and transferred to the NIE to merge with/become part of the teachers’ mandatory attestation training. In 2008, the country was ‘divided’ into educational clusters where in each cluster there was going to be 1

26

high school. Thus, even though the intention was to create 150, eventually 105 were created, based on the population forecast and the nomination of schools by marz authorities (each marz was asked to nominate schools to convert into a high school). In rural areas, no high schools were created and instead, the decision was to maintain the upper secondary block of secondary schools there.

Training of 150 principals of new high schools School Directors Training Manual was published and delivered to all schools in Armenia. The materials of the training of school directors were used during the school director’s training for receiving school director’s certification. However, according to the Prime Minister’s Instructions N 099/11334-12, dated 03/09/2012; N 02/23.12/4149-13, dated 03/27/2013, the training of school principals was no longer funded under the project.

Curriculum and syllabi design for each stream in the new high school, ensuring better articulation between high school and tertiary education

The activity took place in the academic year 2009–2010 and was later covered by the NIE from the state budget.

Capacity building for the staff of the MES and of Marz Education Departments who are directly in charge of the implementation of the high school reform.

Staff of the MES and Marz Education Departments, directly in charge of the implementation of the high school reform, were invited to public consultations on high school reform organized by the Ministry of Education.

TA to help design of counseling and guidance services at the new high schools

The Union of School Centers designed and conducted trainings on counseling and guidance services at the new high schools

Component 2: Supporting Tertiary Education Reforms in the Context of the Bologna Agenda Subcomponent 2.1: Establishment and Strengthening of the National Quality Assurance System Establishment of the External Quality Assurance (EQA) system, including the following:

Establish mutual understanding of quality concept, QA basic principles, and Bologna requirements on QA, with participation of stakeholders (ANQA, MoES, universities, students, and employers).

Fully done with the Netherlands and Flandria Accreditation Agency (NVAO), a Dutch company

Coordinate public awareness campaigns related to QA and Bologna Process through workshops, public talks, and mass media.

Public awareness campaigns related to QA and Bologna Process were held: Conference ‘Quality Assurance of Professional Education in Armenia: current issues and development prospects’ was held in September 2010; ANQA Forum/Workshop on Internal Quality Assurance of Tertiary Education Institutions was held in December 2011; a number of publications representing the main regulations and normative documents in the field of QA were made in different years during the project.

Develop capacities of EQA stakeholders (ANQA, MoES’s Higher Education and Licensing Departments, and university representatives).

Capacities of EQA stakeholders were developed and enhanced through conferences, workshops, and trainings.

Develop EQA policy, standards, criteria, procedures and guidelines, including consultation with stakeholders.

EQA policy, standards, criteria, procedures and guidelines, including consultation with stakeholders were developed as part of the ANQA responsibilities through collaboration with the NVAO.

27

Plan and implement orientation and briefing sessions for external evaluators.

Orientation and briefing sessions for external evaluators were implemented by the ANQA in collaboration with the NVAO.

Organize a study tour to learn activities of other EQA agencies.

Study tours to institutions in the Netherlands, Switzerland, and Belgium were conducted.

Pilot accreditation of selected universities and expand it to the remaining universities.

15 HEIs (10 public and 5 private) passed institutional accreditation process; 10 public and 3 private universities received institutional accreditation according to the decision of the Accreditation Committee.

Elaborate and disseminate outcomes and progresses of QA activities in Armenia and abroad.

The ANQA and NVAO conducted a stakeholder conference at the end of the NVAO contract.

Development of the IQA System, including the following:

Improve capacities of IQA stakeholders (university IQA Units, student representatives).

IQA stakeholders’ capacities were improved through trainings.

Develop approaches to IQA necessary policy, student assessment unified system, standards, procedures and guidelines and updating requirements for faculty qualifications.

Approaches to IQA necessary policy, student assessment unified system, standards, procedures and guidelines, and updated requirements for faculty qualifications were developed.

Provide grants to HEIs to develop IQA Units and improve the IQA System.

26 HEIs (21 public and 5 private) completed the implementation of the grant projects and established and strengthened IQA Units.

Specific grant scheme to promote good governance and transparency in HEIs.

The IQA System included promotion of good governance and transparency in HEIs.

Subcomponent 2.2: Developing a Tertiary Education Management Information System Provide TA to NaCET and universities to design, develop, and implement an integrated TEMIS.

TEMIS was designed, developed, and implemented in 183 higher and vocational education institutions. However, TEMIS is not yet fully operational as there is no mandate for universities to use it and all key individuals responsible for its regular use have been trained.

Provide TA to the MoES and universities to develop information-based planning and M&E capacities.

Information-based planning and M&E capacities were developed. However, because TEMIS is not yet fully operational, training has been more theoretical in nature. The new EIP allocation will be used to further develop information-based planning and M&E capacities.

NaCET to assess information needs for tertiary education and design TEMIS

TEMIS was designed and developed as planned.

Pilot TEMIS in a few public and private universities to observe and assess the operational feasibility and subsequently extend to all public and private universities.

Pilot of TEMIS was conducted.

Form committees of users (with representatives from universities and NaCET) to propose areas of priority to the software developers and facilitate the exchange of information and experiences among universities as the modules were prepared and gradually put in place in a number of institutions.17

No committees were formed, but a feedback mechanism/help desk was developed by NaCET. Few institutions have started data entry, and in many cases, the data entered is incomplete; therefore, a user guide is being developed to make the process easier.

17 Modules were to include (a) personnel management module (recruitment, legal and administrative decisions, salaries, pension, qualifications of lecturers, and so on); (b) academic management module (registration, curriculum,

28

A ministerial order for institutional mandate of management information provision is currently under discussion.

Invite those universities not represented on the committees of users to contribute comments and requests through email or any other communication means.

A help desk was developed by NaCET to allow users provide comments and requests through email and other communication means.

NaCET to organize intensive training programs to support the participating universities in the introduction and adaptation of the modules and strengthen their capacity to use the new computerized management system in an effective manner.

NaCET has provided intensive training, however additional support for capacity strengthening may be needed in selected institutions. The new EIP allocation will be used for NaCET’s capacity building to ensure the organization’s continued support this area.

Once TEMIS becomes operational, the MoES will update the Higher Education Strategy.

The Higher Education Strategy will be updated after TEMIS use becomes mandatory.

Each university will also produce an annual strategic plan and self-assessment reports using TEMIS data.

This was not accomplished because TEMIS is not yet fully operational. Annual strategic plans and self-assessment reports using TEMIS data will be produced by universities after TEMIS is fully operational.

Subcomponent 2.3: Strengthening the Capacity to Implement a Sustainable Financing System (as stated in the PAD) Setting up a student loan scheme (to provide all

needy students with financial aid to pay their tuition fees and living expenses)

The implementation of this subcomponent was underway and several of the activities had taken place when the Prime Minister Instruction N 02/14.1/15383-11, dated 09/19/2011, no longer funded those activities under the project. The remainder of the activities under this subcomponent was not financed in accordance with the first restructuring; however, student-loan-related activities continued under separate government funding and resulted in the introduction of student loans in the country.

Conduct a survey on the demand for student loans among university and college students and university leaders.

A survey on the demand for student loans among university and college students and university leaders was conducted. TA support was provided to the development of a higher education financing strategy in 2011. Targeted student financial aid has been provided.

Conduct a student tracer survey. A student tracer survey was conducted. Design a student loan scheme. A student loan scheme was designed. Develop a monitoring and reporting mechanism. A monitoring and reporting mechanism was planned to

be done after the pilot, which did not happen because this activity was no longer funded under the project.

Training of the government officials and tertiary education institutions

Training of the government officials and tertiary education institutions was planned to be done after the pilot, which did not happen because this activity was no longer funded under the project.

Provision of TA to design the loan scheme and organize a study tour on student loan schemes abroad

A TA to design the loan scheme was provided through an individual consultancy.

courses, research, examinations, classrooms, and so on); (c) student management module (student admissions, attendance, social affairs, continuing students and graduates, and so on); (d) financial management module (accounting, budgeting, payments, cash flow, cost analysis, and so on); and (e) infrastructure management module (buildings, laboratories, workshops, equipment, and so on).

29

Pilot the model with a small number of students. Piloting of the student loan scheme model did not happen because this activity was no longer funded under the project under the Prime Minister’s Instruction N 02/14.1/15383-11, dated 09/19/2011, and subsequent implementation beyond the Education Quality and Relevance Project(EQRP) scope.

Evaluate the model after implementing it for one academic year.

Evaluation of the SLS model did not happen because this activity was no longer funded under the project under the Prime Minister’s Instruction N 02/14.1/15383-11, dated 09/19/2011, and subsequent implementation beyond the EQRP scope.

Based on the evaluation of the pilot model, redesign the student loan scheme for a larger-scale expansion.

Redesigning of the SLS did not happen because this activity was no longer funded under the project under the Prime Minister’s Instruction N 02/14.1/15383-11, dated 19.09.2011, and subsequent implementation beyond the EQRP scope.

Establish a student loan oversight unit in the Ministry of Finance and/or the MoES (if it is decided to scale it up).

Beyond the EQRP scope, the government initiated a student loan program in February 2013. The respective project was approved by the government on March 14, 2013. In total, 14 commercial banks have been included in the student loan project providing loans to students on various terms (subsidies for 2 or 3 percent by the state depending on the achievement of students, and so on).

Designing a CIF (to assist the government in setting up priority areas and viable procedures for the operation of a future CIF in support of quality improvement and innovations in tertiary education):

Conduct a preliminary study on the models of the CIF.

A preliminary study on the models of the CIF was conducted through an individual consultancy.

Based on the study, define objectives of introducing a CIF in Armenia.

Objectives of introducing a CIF in Armenia were defined.

Prepare a terms of reference (ToR) and recruit international and local consultants to conduct a feasibility study and design a CIF

A ToR was developed and international and local consultants were recruited to conduct a feasibility study and design a CIF.

Conduct a feasibility study on a CIF. A feasibility study on a CIF was conducted. Design a CIF. CIF was designed. Pilot CIF by providing grants for HEIs selected

on a competitive basis under the CIF (activity added as part of the first restructuring).

10 universities were awarded grants under the pilot CIF and completed implementation by September 2015. These included 5 consortium projects and 5 individual projects.

Defining a sustainable funding strategy (review the existing resource allocation mechanisms for tertiary education, including current scholarship schemes, and provide evidence and alternatives for the government to design and implement a more sustainable, equitable, and efficient funding system):

A sustainable funding strategy was designed (the consultancy to support the design of the student loan scheme informed this strategy).

Prepare a TOR and recruit international and local consultants to design an overall funding strategy and prepare an implementation plan

A ToR was developed and international and local consultants were recruited to design an overall funding strategy and prepare an implementation plan.

Draft report on an overall funding strategy. A report on an overall funding strategy was drafted. Organize technical workshops to discuss the draft

report. Technical workshops were organized to discuss the draft report.

30

Formulate policies and measures to implement the recommendations of the study.

Policies and measures to implement the recommendations of the study were formulated; HEIs are currently implementing it.

Prepare an implementation plan to operationalize the funding strategy.

An implementation plan to operationalize the funding strategy was developed as part of the strategy that was approved by the government of Armenia on 06/30/2011.

Start implementing the strategy. The strategy is in place and being implemented.

Subcomponent 2.4: Reforming Preservice Teacher Education Develop long-term twinning arrangement between the Armenia State Pedagogical University (and the Consortium of Pedagogical Institutes, together with Yerevan State University) and an international HEI (to support development of initial teacher education in the spirit of the Bologna Process).

Long-term twinning arrangement between the Armenia State Pedagogical University and Oulu University in Finland was developed.

Upgrade the necessary teaching facilities and equipment in HEIs involved in the twinning arrangements to support modernization of primary school teacher education.

Teaching facilities and equipment in HEIs involved in the twinning arrangements were upgraded.

Support the implementation of TA delivered by a selected international university that focuses on upgrading degrees and curricula, enhancing methods of teaching and studying, strengthening educational research capacities, modernizing management and governance, setting the norms of ethics, and establishing networks of field schools for practical parts of new teacher education

TA focusing on upgrading degrees and curricula, enhancing methods of teaching and studying, strengthening educational research capacities, modernizing management and governance, setting the norms of ethics, and establishing networks of field schools for practical parts of educating new teachers, delivered by Oulu University in Finland were supported.

Provide TA, training activities, teaching and study facilities in laboratories and libraries, printing teaching and training materials, and relevant software for teaching and studying.

TA, training activities, teaching and study facilities in laboratories and libraries, printing teaching and training materials, and relevant software for teaching and studying were provided.

Provide targeted internal training and professional development to the consortium members by the staff (local training), provision of library materials and low-cost laboratory equipment to the primary teacher education institutions (or their faculties of primary teaching and methodology), and support to the design and pilot of the practical training in field schools.

Targeted internal training and professional development was provided to the consortium members by the staff; library materials and low-cost laboratory equipment were provided to the pedagogical universities and teacher education institutions; the design and pilot of the practical training in field schools were supported.

Component 3. Project Management, Monitoring, and Evaluation Finance the PIU with core staff hired as consultants. PIU staff were hired as consultants. Implement and manage the procurement process, disbursements, and financial management, project monitoring, and reporting.

Procurement process, disbursements, and financial management, project monitoring, and reporting were implemented.

Acquire minor additions/replacement of office furniture and equipment.

Additions/replacement of office furniture and equipment were acquired.

Provide financing associated with upgrading the PIU facilities.

The PIU facilities were upgraded.

Finance local travel, utilities and publications, translations, office supplies, small office repair, fuel, Internet service, bank commission charges, and vehicle maintenance and repair.

Local travel, utilities and publications, translations, office supplies, small office repair, fuel, Internet service, bank commission charges, and vehicle maintenance and repair were funded.

Provide for systematic maintenance of the M&E system with updates on key performance indicators, which will be carried out jointly between NaCET, the

Systematic maintenance of the M&E system with updates on key performance indicators, which will be carried out jointly between the NaCET, ATC, the entity

31

Assessment and Testing Center (ATC), the entity responsible for policy planning and M&E in the MoES, and the CEP.

responsible for policy planning and M&E in the MoES, and the CEP was conducted.

Provide for participation fees of Armenia for TIMSS 2011.

Participation fees of Armenia for TIMSS 2011 were covered.

Finance implementation and management of impact assessments and special studies planned.

Implementation and management of impact assessments and special studies were funded.

Facilitate coordination, communication flows, and dissemination of information with participating institutions.

Coordination, communication flows, and dissemination of information with participating institutions were facilitated.

Facilitate working group arrangements and the organization of seminars and workshops.

Working group arrangements and organization of seminars and workshops were facilitated.

Support NaCET in the usage and management of the EMIS.

NaCET was supported in the usage and management of the EMIS.

Provide training to staff in the CEP in the areas covering procurement, disbursements, information technology, project management, and other areas identified and proposed in the course of project implementation.

Training was provided to staff in the CEP in the areas covering procurement, disbursements, information technology, project management, and other areas identified and proposed in the course of project implementation.

32

Annex 3: Economic and Financial Analysis

Cost-effectiveness of Investing in Preschool Education

1. A competitive grant scheme was established through which a number of preschool micro-projects received grants to establish new or upgrade existing preschools. Such micro-projects were implemented in every school year between 2010 and 2015.18 While 208 micro-projects took place in existing primary schools, 72 micro-projects took place in independently-housed KGs. In the first year of the micro-projects, the Bank funded the micro-projects and all benefitting communities contributed at least 25 percent of what the Bank contributed, either financially or through in-kind contributions. Following the first year, the government supported the 208 micro-projects in existing primary schools through per capita financing. The communities supported the 72 micro-projects implemented in the KGs. Throughout the course of the project, communities made the necessary contributions during the first and subsequent years of the project.

2. The ICR conducted a benefit/cost analysis of the project using the number of children reached and the increase in educational attainment (0.5 years for children attending 1 year of preschool).19 Rates of return were calculated using the 2013 (latest available) Integrated Living Conditions Survey data20 and were found to be 2.8 percent for males and 3.6 percent for females.

3. The following assumptions were made as part of the benefit/cost analysis:

In accordance with the law, children start primary school at the age of 6.

The participation of children in the preschool micro-projects was their first exposure to preschool and children did not attend preschool before that year.

Once the preschool micro-projects are established, they continue to operate the following years.

The preschool institutions based in schools (202 out of 280) are funded by the state budget (following the first year of operation when support was provided by the project and local communities that contributed an amount equal to 25 percent of the amount of the Bank loan) based on per capita financing.

Since all stand-alone kindergartens have been operating past the first year of Bank support, it is assumed for the purposes of this analysis, that communities provide the same per capita contribution as the government following the initial year of Bank’s support.

18 Preschool micro-projects were implemented in the following school years: 2010–2011, 2011–2012, 2012–2013, 2013–2014, and 2014–2015. 19Investing in Our Children’s Future, From Early Child Development to Human Development. It has been found that “one additional year of preschool correlates with a half-year increase in schooling...” http://elibrary.worldbank.org/doi/pdf/10.1596/0-8213-5050-1 p. 133. 20 Household’s Integrated Living Conditions Survey Tools. http://www.armstat.am/en/?nid=378.

33

The number of male and female children participating in the preschool micro-projects is approximately equal.

11,694 children benefitted from the first year of each of the preschool micro-projects.

35,451 children benefitted from all years of the preschool micro-projects.

Since the retirement age is 63 and students graduate at the age of 17, children do not begin to realize the benefits of increased wages until age 18, and they work for 45 years.

One additional year of preschool correlates with a half-year increase in schooling.21

A 3 percent discount rate was used.

Exchange rate of AMD 474 for US$1.

Annualized wages for men are US$4,778 and for women, are US$3,150.22

4. For one year of preschool directly before enrollment in primary school, the benefit/cost ratio is 5.04 for males and 6.83 for females. For all children benefitting from the preschool micro-projects over the course of the project, the wage gains are approximately US$35 million, with a cost of approximately US$6 million, for a net benefit of approximately US$29 million. This is an underestimate of the benefit, given that the returns are only private returns as social returns have not been factored in. Thus, the benefit/cost estimates suggest that the project’s preschool micro-project subcomponent did better than break even.

Table 3.1. Five-year-old Cohort with 1 year of Preschool Education

Per Beneficiary (US$) Total (US$) Males Females Males Females Wage gains 842 1,142 14,931,177 20,242,937 Cost 167 167 2,962,286 2,962,286 Benefit/cost 675 975 11,968,891 17,280,651 Return for each US$ invested 5.04 6.83 5.04 6.83

21 Investing in Our Children’s Future, From Early Child Development to Human Development. It has been found that “one additional year of preschool correlates with a half-year increase in schooling...” http://elibrary.worldbank.org/doi/pdf/10.1596/0-8213-5050-1 p. 133. 22Women and Men in Armenia. 2015. Statistical Booklet. http://www.armstat.am/file/article/kanajq_ev_txamardik.pdf.

34

Annex 4: Bank Lending and Implementation Support

(a) Task Team Members

Names Title Unit Lending Junko Funahashi Lead Counsel LEGEN Aleksan Hovhannisyan Senior Operations Officer ECCAR Sachiko Kataoka Senior Economist GEDDR Carmen F. Laurente Senior Program Assistant GEDDR Caroline Mascarell Operations Adviser EAPDE Juan Manuel Moreno Olmedilla Team Leader GEDDR Karina Mostipan Senior Procurement Specialist GGODR Sophie Naudeau Senior Education Specialist GEDDR Pasi J. E. Sahlberg Senior Education Specialist ECSHD - HIS Jamil Salmi Consultant GED07 Arman Vatyan Senior Financial Management Specialist GGODR Supervision/ICR Dandan Chen Team Leader ECCU3 Cristian Aado Team Leader GEDDR Meskerem Mulatu Team Leader GEDDR Benedicta T. Oliveros Procurement Specialist GGODR Garik Sergeyan Financial Management Specialist GGODR Anush Shahverdyan Team Member GED03 Darejan Kapanadze Safeguards Specialist GENDR Julia Liberman Team Member GEDDR Nino Kutateladze Team Member GEDDR Junko Funahashi Lead Counsel LEGEN Maria E. Gracheva Senior Operations Officer GHNDR Aleksan Hovhannisyan Senior Operations Officer ECCAR Sachiko Kataoka Senior Economist GEDDR Carmen F. Laurente Senior Program Assistant GEDDR Anarkan Lilly Counsel LEGCF Caroline Mascarell Operations Adviser EAPDE Karina Mostipan Senior Procurement Specialist GGODR Sophie Naudeau Senior Education Specialist GEDDR Pasi J. E. Sahlberg Senior Education Specialist ECSHD - HIS Jamil Salmi Consultant GED07 Michael Trucano Senior Education Specialist GEDDR Arman Vatyan Senior Financial Management Specialist GGODR

(b) Staff Time and Cost

Stage of Project Cycle Staff Time and Cost (Bank Budget Only)

No. of Staff Weeks US, thousands (including

travel and consultant costs) Lending

FY08 11.36 29,897 FY09 52.57 154,032

35

Total: 63.93 183,929 Supervision/ICR

FY10 12.20 58,218 FY11 17.92 88,075 FY12 27.13 108,870 FY13 24.64 79,275 FY14 17.88 65,386 FY15 18.26 76,590 FY16 13.63 54,924

Total: 131.66 531,338

36

Annex 5: Beneficiary Survey Results

Not available

37

Annex 6: Stakeholder Workshop Report and Results

1. The ICR team visited to the project sites, and met with local authorities, parents, teachers, principals and students, and other direct and indirect beneficiaries. The key discussions from these meetings are summarized below.

University Representatives

2. The new innovation centers that have been created and the activities implemented under the CIF are important accomplishments that have led to the improvement in education quality, relevance, and social equity by adjusting education programs with labor market requirements, contributing to the modernization of education programs, and creating public-private partnerships. The ‘culture of quality’ has been introduced to universities throughout the country, and universities have improved their capacity in strengthening both internal and EQA systems. At times, however, procurement and other requirements with regard to the purchase of specific unique equipment were complex, and time limits for project implementation and the associated paperwork posed challenges.

Marz Education Authorities

3. The preschool institutions were established in all marzes throughout the country. They have become an attractive ‘children’s world’ that is quite different from the primary school and other institutions in the marz. The institutions provide a desired environment for free and dynamic personal development of children that promotes the growth of children’s abilities and preparedness for school. Creation of toilets with adjusted basins to suit the needs of young children is another important detail the project has secured. It is noteworthy that the training provided to the teachers in modern early education techniques is an important accomplishment of the project. In marzes such as Armavir and Aragatsotn the preschools have admitted children from families of national minorities. In other marzes such as Ararat, based on need, preschools have admitted children of neighboring communities as well, where there are no similar preschool institutions. The preschools have contributed to job creation, increase in birth rate, and parents’ employment, including engagement in agriculture. In some communities, for example Lori and Tavush, based on demand, there is a need to create more rooms for additional children. Additionally, there was a request to involve the capital city of Yerevan in the project as there is a high demand for preschool institutions there.

Parents of Preschool Children

4. The preschool institutions have provided an opportunity for many parents to work because their child was enrolled in the preschool. One of the parents, for example, started to work in a candy shop while another parent went to work in a factory. The only contribution that parents provide to the preschool is their children’s food during the school day. Everyone unanimously agreed that, because of the preschools, children learn and obtain knowledge and skills that are difficult or impossible for parents to teach.

38

Teacher Training Providers

5. In providing in-service teacher trainings, differences in competence levels of urban and rural teachers were obvious. It was necessary for the training providers to use a tailored approach to be able to engage teachers of different levels, especially in the use of ICT in teaching. The 12 hours allocated to the training on the use of ICT proved to be insufficient, given the different levels of computer/Internet mastery of the trainees. The poor level of ICT use is mostly due to teachers’ limited access to a computer.

6. There were differences in preference of the timing of the training—whether it should be held during an academic semester or school break. While the training providers accommodated the majority’s preference, several others complained that they were working overtime due to the trainings. The training served as a venue to come together, to get to know what is going on in other educational institutions, and to network with counterparts. Sometimes teachers preoccupied with daily work were cut off from ‘reality’ and from what is going on in other educational institutions, and the training became the place to catch up.

7. The main shortcoming of the training was that it contained no subsequent evaluation. Teachers completed through the training, which was not monitored or evaluated with regard to the use of the training and further application of newly gained knowledge and skills in the classroom. No feedback mechanism existed. Additionally, there is no link between a teaching performance and career progression in education or teaching. Even while mastering certain contemporary methodologies, teachers did not practice them because introducing and practicing something new requires extra effort, which in this case, does not lead to career progression and does not provide incentives for experimenting because all teachers are compensated at the same salary scale. Hopefully soon, the teacher ranking system will be fully implemented and teachers will have more incentives for improvement and better performance.

8. It is desirable to conduct the training at a time when teachers are free from their teaching obligations and can fully concentrate on the training.

High School Representatives (Teachers, Principals, Students, and Parents)

9. With an increasing number of computers and technology devices in schools, there is an increased responsibility for maintenance of these devices. More support from NaCET in providing trainings and securing a reliable Internet connection is desirable. Textbooks in high schools are overloaded with content and are written in an academic language; they are very theoretical. It was recommended that instead of grading students very frequently, teachers guide the learning process and provide qualitative insight into students’ understanding. The Bank’s support in providing up-to-date technologies and furniture is highly appreciated by students and teachers.

List of stakeholders met with during stakeholder workshops

University representatives: Mr. Gegham Gevorgyan, Deputy Rector, Yerevan State University, Ms. Nelli Hovhannisyan, CIF project coordinator, Yerevan State University; Ms. Naira Mangasaryan, Researcher, Russian-Armenian Slavonic University (RAU), Mr. Vladimir Gevorgyan, Deputy Rector, RAU; Mr. Aram Baykov, Head of Education Projects Dept.,

39

Yerevan State Medical University, Mr. Samvel Avetisyan, Deputy Rector, Yerevan State Medical University; Mr. David Muradyan, Rector, Yerevan State Institute of Cinematography, Mr. Varuzhan Avanesyan, Head of International Education and Lifelong Learning

Heads of Marz Education Departments: Ms. Gayane Sayadyan, Armavir Marz; Ms. Susanna Petrosyan, Aragatsotn Marz; Ms. Hasmik Gasparyan, Ararat Marz; Mr. Edik Matsakyan, Lori Marz; Mr. Hovik Alexanyan, Syunikmarz; Mr. Hambardzum Safaryan, Gegharkounik Marz; Mr. Arpiar Ghazaryan, Vayots Dzor Marz; Mr. Karen Nazaryan, Tavush Marz; Ms. Silva Achoyan, Yerevan municipality

Artak Poghosyan, Director, NaCET

In-service Teacher Training Providers: Mr. Armen Barseghyan, Head, ArmInfo LLC; Mr. Vagharshak Voskanyan and Mr. Nurijan Manukyan, National Institute of Education; Ms. Anzhela Nazaretyan, Physics-Mathematics Specialized School; Mr. Artashes Torosyan, Education Without Borders NGO; Ms. Silva Khrimyan, Textbook Revolving Fund; Ms. Meri Ghazaryan, Quantum College

UNICEF Armenia representatives: Mr. Tigran Tovmasyan, Disaster Risk Reduction Programme Officer; Mr. Radoslaw Rzehak, Deputy Representative; Ms. Armine Ter-Ghevondyan, Programme Assistant

Mr. Levon Sekhposyan, Principal of Gyumri Academic College; Mr. Karapet Nersisyan, Principal of Gyumri High School # 17; Ms. KarineKatayan, Head of Gyumri Branch of Textbook Revolving Fund; Mr. Gevorg Ghumashyan, Head of the Education Unit of Shirak Marz Education, Sports and Culture Department

Meeting with Principals of Preschools/General Education Schools: Ms. Mariam Hovhannisyan, Jrap Community Preschool Principal; Ms. Liana Vardanyan, Principal of Gyumri School # 10; Mr. Spartak Torosyan, Vahramaberd School Principal; Mr. Sargis Khachatryan, Arevshat secondary school; Mr. Hakob Gevorgyan, Principal of High School #23; Mr. Gagik Karapetyan, Principal of Gyumri school # 7; Mr. Hakob Nahapetyan, Principal of Beniamin School; Ms. Gayane Bichakhchyan, Principal of Gyumri School #29; and Ms. Narine Mesropyan, Principal of Gyumri School #19

Meeting with Gyumri High School #37 Principal Ms. Astghik Avetisyan, school students, teachers, librarians and parents, around 80 people; Mr. Suren Hovsepyan, Principal of the Gymnasium of Culture; Mr. Sargis Sekhposyan, Principal of Gyumri High School #26; Ms. Naira Tadevosyan, Principal of Gyumri High School # 1; Ms. Astghik Bardanyan, Principal of Gyumri High School # 45; Mr. Hovik Petrosyan, Balaton High School; Ms. Lianna Mardoyan, Gyumri High School # 42

40

Annex 7: Summary of Borrower’s ICR and/or Comments on Draft ICR

To ensure the implementation of the policies adopted by the Armenian government and continuation and completeness of the reforms in the education sector, the Armenia Education Quality and Relevance Project has taken into account the priorities of educational reforms and has also ensured the continuation and expansion of the activities implemented by the World Bank’s First Education Quality and Relevance Project (2003-2009). The Project consisted of two components. Each included four subcomponents.

Second Education Quality and

Relevance Project


Component 2. Supporting Tertiary Education Reforms in the Context of Bologna Agenda

Promoting School Readiness and Equal Opportunities at the Start of General Education

Establishment and Strengthening of the National Quality Assurance System

Supporting Further Improvements in the Quality of Education through Improved Teacher In‐service Training and Professional Development

Developing a Tertiary Education Management Information System (TEMIS)

Supporting Integration of ICT in General Education

Strengthening the Capacity to Implement a Sustainable Financing System

Supporting the Implementation of High School Reform Reforming Pre‐Service Teacher Education

During the Project implementation, another procedure was introduced prior to the execution of the planned activities – approval of procurement plan by the RA Prime Minister. Based on this, some of the planned activities under the Project that were not approved by the Prime Minister were suspended bringing changes in the total picture of the Project implementation. During the implementation of the Project, the planned trainings for teachers (partly) and school principals were suspended within the scope of Subcomponent 1.2. Teacher trainings were continued and financed from the state budget; and the trainings for school principals were implemented within the scope of the activities conducted by the school-centers. Within the component of higher education, Subcomponent 2.3 initially intended introducing sustainable financial system via developing higher education financing strategy, introducing student loan system and establishing Competitiveness Innovation Fund (CIF). In 2011, the introduction of the student loan scheme was suspended based on the government’s decision; and the intended funds were reallocated to the establishment of the CIF. Because of the changes mentioned above, project restructuring was initiated to reflect those changes and reallocate funds for other activities. The restructuring of the project required almost

41

half a year causing delays in the overall implementation of the project. Thus another amendment to the financing agreement was conducted to extend the deadline for the project completion for a year till November 30, 2015 instead of initially planned November 30, 2014.

Performance of the project outcome indicators The Projects had four outcome (Project development) indicators. Component 1 accounted for three outcome indicators, which have been achieved by the time of performing the current study. In the area of preschool education, the implemented micro-projects have contributed to the improvement of the Early Development Indicator (EDI) scores of the students in the involved preschools. The proportion of the test-takers passing the 9th grade graduation exam and the gross graduation rate in upper secondary education recorded significant improvements. The latter two indicators have grown by 3 and 11.5 percentage points respectively, exceeding the pre-set targets. Teacher trainings and professional development, ICT continuous integration in the learning process and high school reforms have had a significant input in these developments, ultimately contributing to the improvement of the quality of education. The Project outcome indicator for Component 2 has been measured with progress in the Bologna Process. Armenia’s ranking in rating system of the Bologna Process, which summarizes the standings of the countries involved in the process as of 2014, has improved and the share of the green-value indicators has grown from 44% to 70% since 2012. Improvement in one indicator is necessary to achieve the target value (75%). Taking into account that 15 universities in Armenia are officially accredited as of end of the Project, the 7th indicator – stage of development of external quality assurance system – will obviously improve its stance in the next report. Performance of the Project outcome indicators

Indicator Baseline value 2015


Improved EDI scores of students in preschools and schools receiving the grants

Target EDI scores of students in the participatory group are much higher than those of students in the control

group

Actual - EDI scores of students in the participatory group are on average 19 percent higher than those of students in

the control group

Share of test-takers passing the 9th grade graduation exam

Target 99% (2014)

Actual 96.67% (2009) 99.80% (2014)

Aggregate share of students who have completed the final stage of secondary education

Target 90% (2014)

Actual 85.95% (2010) 97.38% (2014)

Component 2. Supporting Tertiary Education Reforms in the Context of Bologna

Standings of Armenia in the Bologna Scorecard improving from Orange and Yellow to Green

42

Indicator Baseline value 2015

Target 75% of the indicators for quality assurance, recognition and degree system is green

Actual 1 red, 2 yellow, 1 light green in the group of quality assurance indicator;, 1 orange, 1 yellow, 1 light green in the group of recognition indicators

70% of the indicators for quality assurance, recognition and degree system is green

Note

Implemented

Implemented partly

Not implemented

The activities planned by Component 1 have mainly been implemented; and four out of the five intermediate outcome indicators have been achieved. Three out of the four subcomponents in Component 2 - establishment and strengthening of the national quality assurance, strengthening the capacity to implement a sustainable financing system and reforming pre-service teacher education - have successfully been carried out and the pre-set intermediate outcome indicators have been achieved. One of the subcomponents - developing tertiary education management information system, is still in the implementation phase and the target indicators of the subcomponent are still in process. Performance of the intermediate outcome indicators of the Project

Subcomponent Indicator Target, 2015 Actual, 2015

1.2. Supporting Further Improvements in the Quality of Education through Improved Teacher In-service Training and Professional Development

Percentage of teachers taking part in pedagogical development programs

Increasing share

20%

Percentage of teachers still using exclusively the presentation/recitation mode of teaching

Decreasing share

A decrease of 11.3 percentage point

(32%, as of 2013)

Percentage of school principals trained in leadership and management with a specific focus on designing and implementing school development plans

25% *

1.3. Supporting Integration of ICT in General Education

Percentage of teachers intensively using ICT and other new materials in the classroom

60% 61.1%

1.4. Supporting the Implementation of High School Reform

Percentage of high school students who visited the resource center (laboratory and library)

30% 81.4%

2.1. Establishment and Strengthening of the National Quality Assurance System

No. of public and private universities evaluated against the newly developed QA standards, criteria, procedures and guidelines

10 public and 5 private

universities

11 public and 6 private universities **

No. of public and private universities that have operational internal QA Units in place according to the new QA standards


universities


2.2 Developing a Tertiary Education Management Information System

MoES updates the Higher Education Strategy using the information/data collected through TEMIS

Annual report using TEMIS -

Universities produce/update a strategic plan and self-assessment reports using the information/data collected through TEMIS


universities -

43

Subcomponent Indicator Target, 2015 Actual, 2015

2.3 Strengthening the Capacity to Implement a Sustainable Financing System

One round of CIF proposal selection completed and initial disbursements made***

Yes Yes

No. of universities that completed implementation of CIF grants***

10 10

2.4 Reforming Pre-Service Teacher Education

No. of teacher education curricula comparable to a typical Bologna program****

Curricula are approved

Curricula are approved

Note

Implemented

Implemented partly

Not implemented

* Preparatory works for the trainings designed for the school principals within the scope of the Project were fully completed; however, the launch of the trainings was suspended in 2013 by the instruction of the RA Prime Minister. The materials developed within the Project’s framework have subsequently been reflected in the mandatory training courses of the school principals and candidates implemented for attestation purpose. With the Project support, the Union of School-Centers has implemented activities aimed at professional development and training of the school principals in leadership and management. Throughout 2012-2014, 30% of school principals participated in the trainings organized by the Union of School Centers.

**The HEIs that exceeded the target indicator have passed the assessment on their own initiative outside the scope of the grant program of the subcomponent aimed at accreditation.

*** The indicators have been introduced in 2012 after the subcomponent review.

**** The indicator includes only the programs of the Armenian State Pedagogical University.

Project performance

Component 1

Subcomponent 1.1. ''Promoting School Readiness and Equal Opportunities at the Start of General Education''. Micro-projects in preschool education have been successfully developed and implemented. Over 36,000 children have been provided with preschool education over the period of 2010-2015. 280 micro projects – double of the initially planned 140 have been carried out. The micro-projects have been implemented mainly in the marzes; specifically, in communities that lacked operating kindergartens or the existing conditions were insufficient to involve more children. The preschool institutions included in the micro-projects have been renovated, the teachers have passed trainings followed by mentoring support and consultations for teachers provided by an internationally experienced teacher training organization, and educational materials and facilities have been provided. According to the implemented study, the children involved in the micro-projects have recorded higher EDI scores, as compared to those who did not attend preschool. Thus, EDI scores of students in the participatory group were on average 19 percent higher than those of students in the control group.It is noteworthy that Schools implementing preschool micro projects received state financing for the children involved in preschool institutions ensuring sustainability of the projects. Overall, the execution of the Subcomponent has considerably contributed to the implementation of the government’s “Strategic Program on Preschool Education Reforms and Pilot Projects 2008-2015”, has significantly increased the level of children’s readiness for school and has ensured equal opportunities at the start of general education, especially in the regions.

44

Subcomponent 1.2. “Supporting Further Improvements in the Quality of Education through Improved Teacher In-service Training and Professional Development”. Teachers' professional development scheme has been developed and introduced. The ABU consulting company provided CEP with long-term and comprehensive support by developing legal normative documents on the attestation process, procedures for receiving professional ranks, developing teacher training programs to be used during the pre-attestation trainings. Over 12,500 teachers have been trained during 2011-2012. This has been followed by the trainings organized by the National Institute of Education in 2013 with funds allocated from the state budget. By the time of developing the current report, all teachers subject to mandatory attestation have passed the necessary trainings. Teachers have started to increasingly employ innovative and efficient methods of teaching. The activities under the Subcomponent 1.2 have contributed to the establishment of the Union of School Centers, which organizes activities aimed at dissemination of innovation in general education, practice exchange and improvement of the quality of teaching and learning. Subcomponent 1.3. “Supporting Integration of Information and Communication Technologies in General Education”. Around 460 schools with problematic accessibility, namely, remote geographical location and lack of appropriate infrastructure, have been connected to the Armenian Educational Network, thus completing the interconnection of all schools in Armenia. ICT trainings have been conducted for teachers within the framework of pre-attestation trainings. Use of ICT by teachers in the teaching process has increased. The schools have been provided with management information system along with necessary computer equipment, including a computer, a UPS and a multifunctional printer per school for administrative purposes. Representatives of all schools namely 1,420 staff members took part in the trainings aimed at effective use of the delivered hardware and software for the administrative purposes. Currently, an electronic repository of educational resources is being developed for four subjects taught at high and primary schools. It was initially planned to create an open education platform which would serve as a mechanism promoting ICT use among teachers and creation of electronic materials by them. The implementation of this activity was suspended by the decision of the Prime Minister’s Office. Instead, the development of an existing education platform (www.armedu.am) was considered expedient. Therefore, it was decided to establish an e-learning resource repository instead of the open education platform. This repository was created by NACET with the financial assistance of Vivacell MTS, a telecommunications business company in Armenia. Based on the findings of the “Needs Assessment of Electronic Education Content and Materials” study conducted by NACET, electronic education resources were developed for Mathematics, Physics, Chemistry and Biology subjects. The developed educational electronic materials for four subjects are available at the URL: http://esource.armedu.am/. Subcomponent 1.4. “Supporting the Implementation of High School Reform”. The project continued the support of high schools establishment. Resource centers have been established in the high schools, equipped with computers and furniture, mobile computer laboratories and active boards which are intensively used in the teaching process. About 1,200 teachers and the librarians of all high schools took part in trainings (trainings of the high school teachers started within the framework of the First Loan Program). Software for school library administration was developed and introduced in all high schools and trainings were organized for all high schools librarians. All

45

high Schools' resource-centers have been provided with library book stocks in Armenian, Russian and English languages, in total about 270 titles of books. Component 2 Subcomponent 2.1. “Establishment and Strengthening of the National Quality Assurance System”. National system for higher education quality assurance has been formed in Armenia, which includes internal and external quality assurance. Based on the established system, institutional accreditation process of the HEIs has successfully launched. Independent accrediting body defining quality assurance standards In Armenia has been formed – the National Center for Professional Education Quality Assurance Foundation (ANQA), which is an affiliated member of the European Association for Quality Assurance in Higher education (ENQA), and aspires to become a full member in 2016. 26 HEIs have established internal quality control systems and 15 HEIs have received institutional accreditation in the framework of the Subcomponent. About 400 representatives of all HEIs in Armenia have been trained to improve their skills and knowledge in regard to quality assurance issues. Subcomponent 2.2. “Developing a Tertiary Education Management Information System (TEMIS)”. Tertiary Education Management Information System has been developed and introduced in Armenia. Currently, the system is installed in 183 educational institutions and in the central hub. The system is not fully used yet by the educational institutions due to lack of skills and certain technical issues. In addition, large educational institutions have their own operating systems in place and do not fully realize the opportunities provided by the new system. A regulatory framework of the system usage is being launched and a detailed content guide is being developed to enhance the system users’ skills and universities’ motivation. Subcomponent 2.3. “Strengthening the Capacity to Implement a Sustainable Financing System”. New strategy for financing Armenia’s higher education sector has been developed and adopted, followed by implementation of a number of resultant activities. The assistance scheme, benefitting around 4,000 socially vulnerable families from public universities, has been funded additionally from the state budget. A Competitive Innovation Fund for financing innovative projects within universities was established and successfully launched providing grants to 10 HEIs. Based on the success of the Subcomponent, CIF grants have been included in the WB Education Improvement Project. Subcomponent 2.4. “Reforming Pre-Service Teacher Education”. Large-scale educational reforms have been implemented in the Armenian State Pedagogical University after Kh. Abovyan in cooperation with a European HEI, which has involved all levels and key directions of the university activities including curricula review, developing quality assurance system, reviewing internship placement requirements in the schools, developing career center, and strengthening management, transparency and internal quality assurance. The reforms have created base for a new and modern culture at the ASPU matching international standards. Additionally, the infrastructure of state pedagogical universities of Armenia has been renewed; in particular, comprehensive resource centres and modern laboratories for Physics, Chemistry, Biology and Geography have been established in five state pedagogical universities of Armenia. The cooperation between the

46

University of Oulu and the ASPU has not been limited by the scope of the Project and is continued via mutual visits and student exchange programs. Summary and conclusions The micro-projects implemented throughout the general education system have significantly contributed to raising the level of school readiness for children in Armenia and ensuring equal education opportunities, especially in the regions. A mandatory attestation system for teachers has been introduced aimed at ensuring the continuity of regular trainings for teachers. The use of ICT in the educational process has increased and has reached to a new level. The connection to internet of over 460 schools with problematic accessibility, apart from educational use, has had an important social impact for the stakeholder communities since previously internet accessibility in those communities was very low. As a result of the expanded educational and technical infrastructures, enhanced teacher capacities and other developments of the high school system, the latter has transformed from formation to development phase. This will enable full realization of the main mission of the high school system, which is to prepare students to independent life and receive further professional education. As a result of the implemented activities by the Project in higher education, internal and external quality assurance system has been established and independent accrediting body defining quality assurance standards has been formed - ANQA. The compulsory institutional accreditation process helps HEIs to better understand their problems, address the gaps, formalize processes and improve management systems. A modern educational system for preparing teachers has been launched at the Armenian State Pedagogical University. The Competitiveness Innovation Fund has been established and successfully launched. The grants provided by the CIF have contributed to improvements in higher education quality, relevance and accessibility. The synergy between the implemented activities is an additional leverage for the Project’s effectiveness. Particularly, reforming pre-service teacher education (Subcomponent 2.4) is expected to improve the quality of general education. Challenges Overall, the main risks for the materialization and sustainability of the Project’s implemented activities and achieved results are external. While the activities planned by the Project and set goals have mainly been achieved, the systemic problems existing in general and higher education sectors of Armenia may limit the widespread impact of the Project. Addressing these challenges will contribute to efficiency of the Project’s activities and ensure continuity. In the area of general education, there is a need to improve and update educational infrastructures, facilities, technology and other educational resources of the schools. The current model of trainings needs to be improved to fully address the requirements for knowledge, skills and capacity building of the teachers. In preschool education, there still are accessibility gaps, especially in regions. In the area of higher education, the established quality assurance system is a new educational culture. All stakeholders need to demonstrate consistent efforts for the system to succeed and

47

become a comprehensive element of the higher education system. Similarly, the reforms implemented at the ASPU have been crucial for the university; however, their sustainability necessitates ensuring consistent implementation of the introduced processes and systems, and disseminating the newly established culture into the rest of the pedagogical HEIs in Armenia. Overall, the rapid and continuous developments in educational needs, priorities and technologies make it imperative to upgrade the quality of educational resources and educational environment aimed at improvement of the quality of education. For this purpose, a joint agenda and continuous harmonization of the state policy and programs of the World Bank and other stakeholder international organizations are necessary to lead to greater synergy and improved results. A part of the existing challenges in the area of education in Armenia has been taken into account with regard to developing the WB Improving Education Project. Overall, the implementation of the World Bank’s Second Education Quality and Relevance Project has recorded tangible results and has had a positive impact in the education sector of Armenia. The implemented programs have created good base for implementing the next phase of initiatives under the WB Improving Education project aimed at further development of general and higher education in Armenia. Evaluation of the borrower’s own performance during the preparation and implementation of the project, with special emphasis on lessons learned that may be helpful in the future During the preparation of the project, MOES together with the other relevant agencies in the education sector, provided significant input to this process assisting both in the preparation and implementation of certain components of the project and the achievement of the overall goals and objectives of APL2. The performance of both MOES and PIU was positively acknowledged by the Government and stakeholders of the education sector. The preparation and implementation of the project also faced certain challenges. In particular, preparatory works were conducted by PIU and MOES without additional financing during the implementation of the APL1, which caused double workload of the staff. Evaluation of the performance of the Bank, any co financiers, or of other partners during the preparation and implementation of the operation, including the effectiveness of their relationships, with special emphasis on lessons learned The performance of the Bank was very effective during the preparation and implementation of the operations. At the beginning of the implementation of the project, feedback from the Bank was received with some delay, which also caused delays in the overall implementation of the project. After the change of the TTL, the Bank was very operative in providing feedback for requests/queries by PIU. There was a change of the TTL at the final stage of the implementation, but that change was very smooth and did not affect the overall implementation of the Project. The role of the project should not be interpreted as that of a provision of ultimate solutions – they are just vehicles to enhance the situation. For the designated areas to succeed there is a strong need

48

of ownership and full contribution by all stakeholders, unconditional support from the government, political willingness and full consensus and partnership with the civil society.

49

Annex 8: Comments of Cofinanciers and Other Partners/Stakeholders

Not available

50

Annex 9: List of Supporting Documents

Aide Memoires, Management Letters, Statements of Mission Objectives (SMOs), and ISRs.

Armenian Education Sector Development Policy. November 4, 2013.

Baker Tilly Armenia. 2015. Research on Development Dynamics of Children Enrolled in Preschool Institutions and Observation of Factors Affecting Their Development. Final report.

CFEP progress reports, audit reports, and presentations.

Decree on Awarding Ranks to School Teachers, Government Decree No. 1745-N, (2011).

Education, Audiovisual and Culture Executive Agency, (EACEA P9 Eurydice). 2012.The European Higher Education Area in 2012: Bologna Process Implementation Report.

European Commission. 2007. Bologna Process Stocktaking. London.

European Commission/EACEA/Eurydice. 2015. The European Higher Education Area in 2015: Bologna Process Implementation Report. Luxembourg: Publications Office of the European Union.

Executive Office of the President of the United States. 2015.The Economics of Early Childhood Investments. Washington, DC: Executive Office of the President of the United States.

Heckman J. J., S.H. Moon, R. Pinto, P.A. Savelyeva, A.Q. Yavitz. 2009. “The Rate of Return to the High/Scope Perry Preschool Program.”NBER Working Paper No. 15471.Journal of Public Economics 94, 114–128. doi:10.1016/j.jpubeco.2009.11.001.

Higher Education Financing Strategy, June 30, 2011.http://www.edu.am/DownloadFile/4571arm-Razmavarutyun.pdf.

International Monetary Fund. 2008. Republic of Armenia: Second Poverty Reduction Strategy Paper. IMF Country Report No. 08/376.

Ministry of Labor and Social Issues, Republic of Armenia. 2012. Employment Strategy 2013–2018 of the Republic of Armenia. Yerevan.

National Statistical Service of the Republic of Armenia. 2013. Household’s Integrated Living Conditions Survey Tools.

National Statistical Service of the Republic of Armenia. 2015. Statistical Yearbook of Armenia.

PCN documents, including review and minutes.

Project Financial Reports (FMRs) and summaries.

Project documents, including the PAD, Financing Agreement, Restructuring Papers, and related documents.

51

Rauhvargers, Andrejs, Cynthia Deane, and Wilfried Pauwels. 2009. Bologna Process Stocktaking Report 2009. Report from working groups appointed by the Bologna Follow-up Group to the Ministerial Conference in Leuven/Louvain-la-Neuve.

Republic of Armenia. 2003. Poverty Reduction Strategy Paper. Yerevan.

Republic of Armenia. 2008. Strategic Program on the Creation of the System of High Schools.

Republic of Armenia. 2008. Sustainable Development Program. Annex, RA Government Decree N 1207-N October 30, 2008. Yerevan.

Republic of Armenia. 2008. The Strategic Program for Reforms in Preschool Education of the Republic of Armenia (2008–2015). Government Decree, N 10, March 13, 2008.

Republic of Armenia. 2011. Education Development 2016–2025State Programme of the Republic of Armenia.

Republic of Armenia Development Strategy for 2014–2025. Annex to RA Government Decree No. 442 ‐ N, March 27, 2014.

The Law on General Education of Armenia Article 26, No. HO-160-N, (2009).

World Bank. Project Appraisal Document. Armenia Social Investment Fund (ASIF) (P094225).

World Bank. Project Appraisal Document. Education Financing and Management Reform Project (P008281).

World Bank. Project Appraisal Document. Education Quality and Relevance Project in Support of the First Phase of the Education Sector Reform Program. (P074503).

World Bank. Project Appraisal Document. E-Society and Innovation for Competitiveness Project (P115647).

World Bank. Project Appraisal Document. Indonesia Early Childhood Education and Development Project (P089479).

World Bank. Project Appraisal Document. Mexico Compensatory Project (P101369)

World Bank. Armenia. Education Quality and Relevance (APL2) Project (P107772 - IDA 45650) Fiduciary Review.

World Bank. Armenia: Second Education Quality and Relevance Project (P107772). Alternative Outcome Indicators Considered.

World Bank. Russia Education Aid for Development (READ) Trust Fund Program. http://www.worldbank.org/en/programs/read.

World Bank. 2000. Mustard, J. Fraser; van der Gaag, Jacques; Willms, J. Douglas; Love, John M.; Schochet, Peter Z.; Meckstroth, Alicia L.; Evans, Judith L.; Tarullo, Louisa B.; Scott-McDonald,

52

Kerida; Myers, Robert G.; Kirpal, Simone; Iglesias, Enrique V.; Shalala, Donna E; Doryan, Eduardo A.; Gautam, Kul C.; Foege, William H.. 2002. “Investing in Our Children’s Future, From Early Child Development to Human Development”. Mary Eming Young, Editor. Proceedings of a World Bank Conference on Investing in Our Children’s Future. Washington, DC, April 10–11.

World Bank. 2003. Project Appraisal Document. Second Education Quality and Relevance Project (APL 2) in Support of the Education Quality and Relevance (APL) Program. (P107772).

World Bank. 2007. Country Assistance Strategy Progress Report for Armenia for the Period FY05–08, Report No.: 38585–AM, 2007.

World Bank/IFC/MIGA. 2008.Quality Enhancement Review Panel Report. Armenia Second Education Quality and Relevance Project (APL2).

World Bank. 2009. Conformed Copy, Credit No. 4565-AM, Financing Agreement (Second Education Quality and Relevance Project) between Republic of Armenia and International Development Association.

World Bank. 2009. Country Partnership Strategy for the Republic of Armenia for the Period of FY09–FY12.

World Bank. 2009. Supplemental Letter No. 2. Regarding Credit No. 4565-AM (Second Education Quality and Relevance Project) Performance Monitoring Indicators.

World Bank. 2012. Armenia SABER-Early Childhood Development Report.

World Bank. 2012. Restructuring Paper on a Proposed Project Restructuring of Second Education Quality and Relevance (APL 2) Project (Approved On May 12, 2009), Credit No. 4565-Am, to the Republic Of Armenia.

World Bank. 2013. Armenia - Country Partnership Strategy for the Period FY14–FY17. Washington, DC: World Bank.

World Bank. 2013. Armenia: Credit No. 4565-AM (Second Education Quality and Relevance Project). Amendment to the Financial Agreement and Supplemental Letter No. 2.

World Bank. 2014.Restructuring Paper on a Proposed Project Restructuring of Second Education Quality and Relevance (APL 2) Project (Approved On May 12, 2009), Credit No. 4565-Am, to the Republic Of Armenia.

World Bank. 2014. Second Education Quality and Relevance Project (Credit No. 4565-AM) Amendment to the Financing Agreement and Project Outcome Indictors.

World Bank. 2014. The Inspection Panel Report and Recommendation. Armenia: Second Education Quality and Relevance Project (P107772), and Education Improvement Project (P130182). Report No. 89836.

53

World Bank. 2015. Country Program Snapshot. World Bank Group - Armenia Partnership. http://www.worldbank.org/content/dam/Worldbank/document/Armenia-Snapshot.pdf

World Development Indicators, The World Bank.

54

Annex 10: Linkages between APL 1 and APL 2

Education Level or Function

APL 2 APL 1

Preschool Education

Subcomponent 1.1: Grants to communities for improved access

and quality

n.a.

Primary and Lower Secondary Education

Subcomponents 1.2 and 2.4: Teacher training and professional

development Reforming preservice teacher training

Components 1 and 2: Development of national curriculum and subject

syllabi Teacher training/professional development

Upper Secondary Education

Subcomponent 1.4: Resource Centers Training of teachers Training of principals Training of librarians Introduction of reforms in secondary

education

Component 1: Development of national curriculum and subject

syllabi

Vocational Education and Training

EU assistance programs for reduction of poverty through reforms in vocational education and training

EU assistance programs for reduction of poverty through reforms in vocational education and training

Higher Education

Subcomponents 2.1 and 2.3: Implementing QA at national and university

level Pilot CIF

Monitoring and Evaluation

Subcomponents 1.3 and 2.2 : Development and implementation of EMIS

for secondary education Development of TEMIS Participation in TIMSS

Component 1: Establishment of the Assessment and Testing

Center

ICT in Education

Subcomponent 1.3: Educational technologies in schools Establishment of Internet and intranet

connection for all schools Development of digital content Teacher training in ICT Staff management in ICT

Component 2: Educational technologies in schools Internet connection Development of digital content Strengthening of NaCET Teacher training in ICT

Financial Reforms

Subcomponent 2.3: Supporting higher education financial

reforms

Component 4: Grade rationalization and school network

optimization Design of the per capita financing mechanism

55

Annex 11: Education Sector Reform Program Triggers

The APL supported Armenia’s Education Sector Reform Program. Its development objective was to improve the quality and relevance of the Armenian school system to meet the challenges of the knowledge society. As included in annex 15 of the Project Appraisal Document (PAD) of the first phase of the APL, the government provided the Bank with a “List of possible indicators and triggers (conditions) for a ten-year APL” that could be used to monitor each of the phases of the program. That document suggests that at the completion of APL 3 the following two outcome indicators would have been achieved: “schools are organizations in which pedagogical, financial, and administrative aspects are focused on increasing student learning;” and “student learning outcomes have improved.” This is the only document identified by the ICR team suggesting possible outcome indicators for the entire APL program, and there are no specific targets or further information on how they would be measured.

The APL 1 had as its objective to support the first phase of the Borrower’s Program through: (a) improving and strengthening the efficiency of the education sector to meet the challenges of a knowledge economy; (b) upgrading the quality and relevance of the school education system; and (c) improving the evaluation of the education system’s performance. 23 Its outcome and intermediate outcomes were the same as those included in the “List of possible indicators and triggers (conditions) for a ten-year APL” mentioned above; the single original outcome indicator was that “70 percent of teachers are engaged in activities likely to develop the necessary knowledge and competencies in their students.”

The APL 2, approved approximately five years after APL 1, had as its development objective: to (i) enhance school learning in general education and improve the school readiness of children entering primary education; and (ii) support the integration of the tertiary education system into the European Higher Education Area. Similarly, the APL 2 original outcome indicators differed from the single indicator included in annex 15 of the PAD for the first APL (“90 percent of teachers are engaged in activities likely to develop the necessary knowledge and competencies in their students”); the same is true for intermediate indicators. Additionally, the APL 2 included in both its development objective and indicators targets related to strengthening the higher education system, something that was not explicit in the original program design.

The triggers agreed for each phase of the APL remained unchanged and were all met during its implementation, as reported below. For that reason, as well as the lack of clarity regarding the program’s outcome indicators, the ICR team found them to be an important indication of the APL’s contribution toward improving the quality and relevance of the Armenian school system to meet the challenges of the knowledge society, its development objective.

23 As previously mentioned, the PDO included in the PAD differed from the project objective specified in the APL 1 Development Credit Agreement.

56

Triggers from APL 1 to APL 2 Status as of November 30, 2015

Policy for addressing the needs of special education students agreed with the Bank and approved by the Government

The Law on the Education for Children with Special Education Needs has been adopted by the National Assembly on May 25, 2005.

National plan for educational assessment and evaluation agreed with the Bank and approved by the Government

The Assessment Policy for Student has been adopted by the Government of Armenia, (Decree n. 14, April 14, 2005).

Minimum standards for school libraries approved by the Government, assessment of the current situation in schools with respect to these standards completed, and plan for libraries agreed with the Bank and approved by the Government

The “School Library Development principles, management and methodological guidelines for general education institutions of Armenia” was approved by the Ministerial Decree N 698-AIQ, dated September 9, 2008.

Textbook Revolving Fund cash flow projection for years 2008-2010 demonstrate its self-sustainability

For 2008, the cash flow projection was US$14.7 million, and increased subsequently.

Comprehensive teacher education and professional development strategy is approved by the Government

The Ministry of Education has established a working group with representatives from MoES, pre-service teacher institutions, NIE, CEP and other stakeholders. A draft document has been completed and, after internal discussion, has been formally approved as a Ministerial Decree on January 23rd, 2009.

Education Chapter of PRSP revised to reflect developments during phase 1

The Education Chapter of the draft PRSP reflects developments of phase 1 and was approved in January 2008

New policy and implementation plan for school management and inspection service agreed with the Bank and approved by the Government

The Law on inspection was passed by the National Assembly. (Adopted by RA National Assembly on November 15, 2005). The new State Inspectorate was established by the Government degree n. 497-N, March 16, 2006.

Triggers from APL 2 to APL 3 Status as of November 30, 2015

Evaluation of the Phase I curriculum and assessment reform completed and revisions made accordingly

Evaluation Report of the Curriculum and assessment reform was completed in September 2008 and is being used in revising the curriculum for the High School reform and the unified examination system

Textbook Revolving Fund cash flow projections for years covered by next phase demonstrate its sustainability

For 2008, the cash flow projection is US$14.7 million, and this will go up to US$16.0 million in 2010.

Comprehensive teacher education and Although the Strategy was only recently approved, there is a clear sense of direction in the teacher professional

57

professional development plan implementation progressing satisfactorily

development policy and all the envisaged reforms are progressing satisfactorily.

New policy and implementation plan for cost-efficient reform of student social and support services agreed with the Bank and approved by the Government

The Higher Education Financing Strategy that received support from the Bank was approved by the Government on June 30, 2011. The subsequent Criteria for Assessment of the Students’ Financial Capacity and Student Financial Support Program and associated regulations were approved by the Government in 2012.

Education chapter of PRSP revised to reflect developments during phase 2 of the APL

Chapter revised reflecting developments of the one-year extension of APL 1.

58

Annex 12: Details on the Project’s Main Beneficiaries

1. Based on the PAD and project papers, the project’s main beneficiaries belong to the following categories:

Students, parents, and communities. Preschool students were to benefit from participation in enhanced existing preschools or newly established KG classrooms in a local primary school. High school students were to benefit from new technologies and laboratories. University student representatives were to benefit from support to develop QA units. Higher education students were to benefit from the design of a student loan scheme. University students were to benefit from the implementation of the pilot CIF (under Subcomponent 2.3 according to the Restructuring Paper of August 2, 2012). Parents of preschool children were to benefit by having the opportunity to engage in educational or livelihood activities while their children were in school. Parents and families were also to benefit from the expected improved quality of education in new high schools and a decreased reliance on (and financial outlay for) private tutoring. Communities were to benefit from targeted grants either to enhance existing preschools or to establish a KG classroom in a local primary school.

Schools, teachers, librarians, and school administrative staff. Around 400 remote schools were to benefit from Internet connectivity, hardware, and software. High schools were to benefit from technologies and laboratories. Preschool teachers and administrative staff were to benefit from training. Teachers and principals were to benefit from improved in-service and preservice training, professional development, and training on the use of ICT. Teachers and librarians were to benefit from professional development on the use of new technologies and laboratories.

Higher education institutions. HEIs were to benefit from an improved high school education by being able to make more informed admissions decisions. Universities were to benefit from support in developing QA units. Universities were to also benefit from the data collected through TEMIS and from TA to develop information-based planning, M&E capacities. Universities were to benefit from the implementation of the pilot CIF (Project Paper, August 2, 2012). Researchers were to benefit from opportunities to further their training as part of reformed preservice teacher education. Armenia State Pedagogical University, Yerevan State University, and members of the Inter-institutional Consortium and its staff were to benefit from reformed preservice teacher education, including internal training, professional development, and provision of library materials and laboratory equipment.

Professional development providers. Professional development providers were to benefit by expanding their services to schools and teachers.

Key sector institutions. The NIE, the authority ensuring the quality of the training and professional development, was to benefit from TA. NaCET was to benefit from capacity development in ICT use in education and establish an integrated TEMIS.

59

ANQA was to benefit from capacity-building support in implementing the national QA system.

Government agencies. Marz education units and marz education officials were to benefit from training on preschool management and capacity building for implementing high school reforms. The MoES was to receive capacity-building support in implementing high school reforms. It was also to benefit from the data collected through TEMIS and from the TA to use TEMIS. The MoES was to benefit from a study to define a sustainable financing strategy for the development of the higher education system. It was also to benefit from the CIF experience and results of the CIF pilot (Project Paper, August 2, 2012).

60

Annex 13: Reallocation of Financing

1. The allocation of US$1.5 million for Subcomponent 2.3 (Strengthening the Capacity to Implement a Sustainable Financing System) was reallocated across the project’s components and subcomponents as follows:

US$250,000 for the expansion of preschool micro-projects under Subcomponent 1.1 (Promoting School Readiness and Equal Opportunities at the Start of General Education).

US$250,000 for the expansion of higher education grants, both with regard to size and coverage, for the implementation and enhancement of the IQA System under Subcomponent 2.1 (Establishment and Strengthening of the National Quality Assurance System).

US$1 million for the CIF in support of quality improvements and innovations in tertiary education institutions envisaged under Subcomponent 2.3 (Strengthening the Capacity to Implement a Sustainable Financing System).

Project Costs (US$, millions)

Components/Subcomponents Per the

PADFollowing

Restructuring %Change

Component 1. Enhancing the Quality of General Education 15.73 15.98 1.591.1.Promoting School Readiness and Equal Opportunities at the Start of General Education

2.49 2.74 10.04

1.2.Supporting Further Improvements in the Quality of Education through Improved Teacher In-service Training and Professional Development

3.53 3.53 0.00

1.3.Continuing Support to the Integration of ICT in Teaching and Learning Process

3.72 3.72 0.00

1.4.Supporting the Implementation of High School Reform 5.99 5.99 0.00 Component 2. Supporting Tertiary Education Reforms in the Context of the Bologna Agenda

6.02 5.76 −4.16

2.1.Establishment and Strengthening of the National Quality Assurance System

2.47 2.72 10.12

2.2.Developinga Tertiary Education Management Information System 0.39 0.39 0.00 2.3.Strengtheningthe Capacity to Implement a Sustainable Financing System

1.93 1.43 −25.91

2.4.ReformingPre-serviceTeacherEducation 1.22 1.22 0.00 Component 3. Project Management, Monitoring, and Evaluation 3.25 3.25 0.00Total 25.00 25.00 0.00

61

Annex 14: Summary of Armenia’s Ratings in Bologna Process Stock-taking Reports, 2007–2015

Indicator 2007 2009 2012 2015

Stage of implementation of the 1st and 2nd cycle

Green24 Green Green Green

Access to the next cycle Green Green Green Green

Implementation of national Qualification Framework

Orange Yellow Yellow Yellow

Implementation of external quality assurance

Light Green Yellow Yellow Yellow

Student participation in quality assurance Yellow Green Green Green

International participation in quality assurance

Red Orange Yellow Orange

Stage of implementation of diploma supplement

Yellow Yellow Light Green Light green

Stage of implementation of the European credit transfer and accumulation system

Orange Yellow Yellow Light green

Recognition of prior learning Orange Yellow Orange Red

Measures to support the participation of disadvantaged students

n.a. n.a. n.a. Light green

Portability of public grants and publicly subsidized loans

n.a. n.a. n.a. Orange

Financial mobility support to disadvantaged students

n.a. n.a. n.a. Red

Level of openness to cross-border QA of the European Quality Assurance Register registered activities

n.a. n.a. n.a. Green

Implementation of Lisbon Recognition Convention principles

Light Green Green Dropped Dropped

National implementation of Standards and Guidelines for QA in the European Higher Education Area

Yellow Dropped Dropped Dropped

Stage of implementation of diploma supplement

Yellow Dropped Dropped Dropped

Establishment and recognition of joint degrees

Light Green Dropped Dropped Dropped

24 The colors indicate the extent of progress on indicators. Green represents excellent performance, light green represents very good performance, yellow represents good performance, orange represents that some progress has been made, and red represents that little progress has been made yet.

62

Annex 15: Linkages between PDOs and Outcomes

PDO Enhancing School Learning in General Education

Formally Revised Target Value

Actual Value Achieved at Project Closing

Associated PDO Indicators

Share of test-takers passing the 9th grade graduation exam

99% 99.8%

Gross graduation rate in upper secondary education 90% 90%

Subcomponents 1.2: Supporting Further Improvements in the Quality of Education through Improved Teacher In-service Training and Professional Development 1.3:Continuing Support to the Integration of ICT in Teaching and Learning Process 1.4:Supporting the Implementation of High School Reform

Intermediate Outcome Indicators

Outcomes at Project Closing

% of teachers intensively using ICT and other new materials in the classroom

The school learning environment in general education was enhanced through the distribution of furniture and mobile computer labs to 195 schools25 and through the provision of Internet, computers, and printers for 459 schools that were yet to be connected to the Internet. By 2015, 74 percent of teachers were intensively using the computer and other digital materials, surpassing the target of 60 percent, as compared to 45 percent in 2008.

% of teachers taking part in pedagogical development programs

The project supported training of 12,165 teachers in professional development and innovative teaching methods by 13 training organizations between 2011 and 2012. By 2013, the government mainstreamed the NIE’s responsibility for teacher training to maximize the efficiency of public funding, particularly because the NIE was already mandated with teacher training activities and it had full capacity supported by public funding to carry out those activities. As a result, the NIE became solely responsible for teacher training and benefitted from the teacher training materials developed under the project. By 2015 approximately 27,000 additional teachers had participated in the mandated professional development programs. At project closing, 100 percent of teachers had taken part in pedagogical development programs at project closing.

% of teachers still using exclusively the presentation/recitation mode of teaching

The majority of trained teachers were applying innovative teaching methodologies in the classroom, and the percentage of teachers solely using the presentation/recitation mode of teaching decreased from 43 percent in 2011 to 32 percent in 2012.

% of school principals trained in leadership and management with a specific focus on designing and

School learning in general education was enhanced by the development of comprehensive training modules for school principals that have been distributed to all schools and made publically accessible on the NaCET website. The Union of School Centers (a network of 108 schools that support school-level activities in adjacent schools) also carried out training of school principals on designing and implementing school development plans, and by 2014, 25 percent of school principals had been trained

25The project supported the procurement of furniture for high schools. Based on a needs assessment and because of the substantial cost savings under this component, given the lower bid prices than originally anticipated, the ministry expanded the number of beneficiary schools to also include general schools, which offer upper secondary education (grades 10 to 12).

63

implementing school development plans

on this. However, no further data was collected after completion of the contract with the Union of School Centers in November 2014.

% of high school students who visited the resource center (laboratory and library)

Internet connectivity and computer resources supported through the project have made possible the introduction of the educational network, connecting all public schools and 22 vocational education institutions via intranet. An updated version of the SMIS has been installed in all public and some private schools, and NaCET has trained approximately 1,388 school representatives, mostly principals, on the use of the SMIS software. As a result of project activities in support of this objective, all high schools had fully operating resource centers and 100 percent of high school students visited them, exceeding the target of 30 percent.

PDO Improving the school readiness of children entering primary education

Formally Revised Target Value


Associated PDO indicator

Improved EDI scores of students in KG and schools receiving the grants with regard to students in the control group

EDI scores are higher in treatment group as opposed to control group and this difference is statistically significant.

EDI score on average 19% higher in treatment group

Subcomponent 1.1: Promoting School Readiness and Equal Opportunities at the Start of General Education


Although during project preparation, it was envisaged that 140 communities and 8,750 children would benefit, approximately 35,451 children have benefitted as it was possible to complete some micro-projects with less financing, therefore freeing up resources to be used in other communities. As of project closing, all of the preschools continued to be fully operational. The Ministry of Education, UNICEF, and Step-by-Step Foundation provided the necessary training to preschool teachers and marz administrative staff. Discussions with parents and marz authorities were positive, and stakeholders felt that the school readiness of their children after participating in the preschool micro-projects was high. PDO Supporting the integration of the Armenian Tertiary Education system into the

European Higher Education Area Formally

Revised Target Value


Associated PDO indicator

Armenia’s standing in the Bologna Scorecard according the published stocktaking reports of the Bologna Process

75% of indicators are green

55.6% of indicators are green

64

Component 2: Supporting Tertiary Education Reforms in the Context of the Bologna Agenda

Intermediate outcome indicators


Number of public and private universities evaluated against the newly developed QA standards, criteria, procedures, and guidelines

The project supported tertiary education reforms in the context of the Bologna Agenda, including the institutional accreditation process of 10 public and 5 private HEIs. While 11 of these received full accreditation and 2 received conditional accreditation, 2 universities were rejected by the Accreditation Commission (an independent body comprising representatives from ministries, universities, employers, and students). However, the target of 7 public and 7 private universities was not met.

Number of public and private universities that have operational IQA Units in place according to the new QA standards

All 26 HEIs that received grants for establishing the IQA Units established them. The capacity of the ANQA and of all HEIs in the area of QA has also been strengthened through successful collaboration with the NVAO.

TEMIS will be used by the MoES to develop annual reports on higher education and to prepare Medium-Term Expenditure Framework (MTEF) and annual budget for higher education

Although TEMIS was developed and installed/is functionally available to use, at project closing, there was no mandate or official requirements to actually use it (for example, there was no mandate for universities to enter data into the system or to create reports using the entered data, there are no requirements for deadlines for entering the data or using the data). Therefore, although TEMIS was developed, the MoES does not have accurate and comprehensive data available to produce reports or plans using that data.

Universities produced/updated a strategic plan and self-assessment reports using the information/data collected through TEMIS

Although TEMIS was developed and installed/is functionally available to use, at project closing, there was no mandate or official requirements to actually use it (for example, there was no mandate for universities to enter data into the system or to create reports using the entered data, there are no requirements for deadlines for entering the data or using the data). Therefore, although TEMIS was developed, universities do not have accurate and comprehensive data available to produce reports or plans using that data.

Identification of funding sources for the student loan scheme

By project closing, 14 commercial banks have been selected to provide loans to students on various terms.

Number of teacher education curricula comparable to a typical Bologna program

The project supported the partnership (twinning) between Oulu University and Armenian State Pedagogical University, the purpose of which was to reform preservice teacher education in Armenia in the spirit of the Bologna Process. As a result of this collaboration, all teacher education curricula have become comparable to a typical Bologna program. Among other activities, this partnership supported the Armenian State Pedagogical University in the revision of Bachelor and Master degree programs, strengthening of the IQA System, creation of a career center and improvement in the practical training system, development of good governance and transparency policy, and enhanced professional development opportunities.

One round of CIF proposal selection completed and initial disbursements made

The CIF was successfully piloted in 10 universities in Armenia.

65

Number of universities that completed implementation of the CIF grants

8 public, 1 private, and 1 inter-governmental university participated in the pilot CIF, exceeding the target of at least 4 public and 1 private universities participating in the pilot.

66

Annex 16: Discussion on Lessons Learned

1. The Bank’s long-term engagement in the education sector provides a robust platform for well-performing projects, allowing for the broadening of the engagement in the sector, including through the introduction of innovative programs and sound policy advice. The Bank’s continued presence in the sector and long-term support through the APL series helped deepen the understanding of education sector issues and to accumulate implementation support experience. The Bank has been an active development partner in the sector with its support aligned with the country’s and sector’s development objectives.

2. Bank-supported project activities can catalyze key sector reforms through close engagement and effective dialogue in general, as well as during project preparation. The successful implementation of several innovative programs under the project have been mainstreamed or scaled up by the government through domestic financing, such as for the preschool programs and the ANQA operations.

3. Within the context of supporting a long-term substantive sector reform, responsiveness to short-term sector achievements and challenges is key for sustaining project relevance. The two restructurings during the project cycle adjusted project activities and resource allocations. The restructurings ensured that a balance of government- and Bank-financed activities was maintained and that those programs that were prepared for piloting or scaling up, as was the case with the CIF and preschool programs, could in fact do so.

4. Broad and complex projects can be successfully implemented in the context of timely and quality implementation support from the Bank and a client with high capacity. Although the project got off to a slow start, proactive and effective implementation support from the Bank helped ensure completion of project activities. Importantly, successful implementation was possible due to a client that had extensive experience working with the Bank and that was dedicated to achieving the project outcomes.

5. The M&E and the Results Framework should incorporate only those indicators that can accurately measure the results achieved through the project. The indicators should be SMART (Specific, Measurable, Attributable, Realistic, and Time-bound). For example, if results of TIMSS are to be used, it is important to determine when the data will be collected to ensure that the assessment results can be attributed to the project. Additionally, if UEE results are to be used to measure progress over time, it is important to confirm that the assessment is designed in a way that will allow for such comparison.

6. Relevant, comprehensive, and technically well-designed studies should be planned as part of project preparation to monitor and accurately measure project outcomes. The project incorporated an evaluation of the impact of the project’s preschool interventions on the Early Development Index of children. The study was designed by a local firm under the supervision of the PIU; however, the study was not as rigorous as it should have been. Project implementation support should include support for ensuring the quality of study design, the quality of data collected, and the analysis that is prepared on the basis of that data. Rigorous studies will not only benefit the project M&E but will also contribute to the global knowledge in this area.

67

7. Successful implementation of the CIF can be ensured through clearly defined roles and responsibilities of the CIF recipients and their timely and relevant capacity building. Specifically, the key lessons include the following: (a) strengthening the procurement and financial management capabilities of the CIF recipient is crucial; (b) there is a need to establish clear mechanisms for problem solving; (c) there is a need to develop templates and procedures to clearly define responsibilities and M&E arrangements; (d) stable staffing of recipient projects is a key factor for success and should be a criteria for project implementation; and (e) a sustainability plan should be required as part of the CIF proposal.

8. Teacher professional development can significantly benefit from learning best practices in collaborative and hands-on settings, such as through international partnerships. The close partnership between Oulu University and Armenian State Pedagogical University ensured that all preservice teacher education curricula in Armenia are now comparable to a typical Bologna program. The quality of this teacher training will be sustained through the strengthened IQA System, enhanced professional development opportunities, as well as the newly developed good governance and transparency policy, all of which were introduced as a result of the international partnership.