vanuatu education support program (vesp) monitoring and ... 1 monitoring and...this document sets...

1

Education Support Program (Vanuatu)

is managed by Coffey on behalf of the

Australian Department of Foreign

Affairs and Trade

Vanuatu Education Support Program (VESP)

Monitoring and Evaluation Plan

Revised and endorsed by VESP Steering

Committee 29 August 2016

2

Table of contents

1 Introduction ............................................................................................................................... 5

1.1 Program Overview .......................................................................................................... 5

1.2 Overview of the Monitoring and Evaluation Plan .......................................................... 6

1.2.1 Purpose of the Monitoring and Evaluation Plan ................................................... 6

1.2.2 Information needs of program stakeholders ......................................................... 6

1.2.3 Scope of the MEP ...................................................................................................... 8

1.2.4 Summary of information sources ........................................................................... 8

2 Program Evaluability Assessment .......................................................................................... 8

3 VESP Program Theory ............................................................................................................ 10

3.1 Theory of Change .......................................................................................................... 10

3.2 VESP Results Framework ............................................................................................ 11

3.3 Program Monitoring ...................................................................................................... 12

3.4 Activity Monitoring ........................................................................................................ 12

3.5 Output Monitoring ......................................................................................................... 13

3.6 Outcome Monitoring ..................................................................................................... 15

4 Evaluation Questions .............................................................................................................. 16

5 Monitoring and Evaluation Approach ................................................................................... 17

5.1 Sources of information ................................................................................................. 18

5.1.1 Open VEMIS system ................................................................................................ 18

5.1.2 Program monitoring and reports........................................................................... 18

5.1.3 Proposed Monitoring Studies ................................................................................ 19

5.1.4 Independent mid-term and end program evaluation .......................................... 20

5.2 Relevant aspects of the context and key M&E risks ................................................. 20

5.3 Data collation, analysis and reporting ........................................................................ 20

6 Communication and Dissemination ...................................................................................... 21

6.1 External communication .............................................................................................. 21

6.2 Internal communication ................................................................................................ 21

7 Roles related to VESP M&E ..................................................................................................... 24

7.1 VESP Monitoring and Evaluation Adviser .................................................................. 24

7.2 VESP Program Implementation Team ......................................................................... 24

7.3 VESP Secretariat ........................................................................................................... 24

8 Workplan .................................................................................................................................. 24

Executive Summary .................................................................................................................. 1

1. Introduction ..................................................................................................................... 1

2. Overall VESP Progress ................................................................................................... 1

3. Progress towards end of program outcomes ........................................................................ 1

4. Lesons learned ................................................................................................................ 1

.5. Risk management .......................................................................................................... 1

3

7. Cross-cutting issues ....................................................................................................... 1

8. Program Management .................................................................................................... 1

This section includes: cumulative expenditure performance, expenditure performance for

the six month period, forecast expenditure for next six months, adequacy of

progress against budget, and fraud detection, investigation and reporting ............ 1

9. Conclusion ....................................................................................................................... 1

Tables

Table 1: Overview of program stakeholders’ information needs............................................................. 7

Table 2. Performance Indicators for Program Activities ....................................................................... 12

Table 3: Schedule of Activity Progress and Financial Reporting .......................................................... 13

Table 4: VESP Monthly Report Template ............................................................................................. 13

Table 5. Performance Indicators for Program Outputs ......................................................................... 13

Table 6: Schedule of Program Outputs Progress Reporting ................................................................ 14

Table 7. Performance Indicators for Program Outcomes .................................................................... 15

Table 8: Schedule of Program Outcomes Progress Reporting ............................................................. 16

Table 9: Key evaluation questions (KEQ) ............................................................................................. 17

Table 10: Vanuatu Public Service Annual Planning - Budget - Reporting Cycle .................................. 22

Table 11: Overview of external communication .................................................................................... 23

Table 12: Estimated budget for M&E activities ....................................... Error! Bookmark not defined.

Table 13: VESP M&E Workplan ............................................................................................................. 1

Figures

Figure 1: Overview of VESP Theory of Change ..................................................................................... 9

Annexes

Annex 1 Program Logic Model

Annex 2 Results Framework

Annex 3 VESP Monthly Progress Report Template

Annex 4 VESP Six-Monthly Progress Report Template

4

Abbreviations CB Capacity Building

CDU Curriculum Development Unit (MoET)

DFAT (Australia) Department for Foreign Affairs and Trade

EAU Examinations and Assessment Unit (MoET)

EBPP Evidence-Based Program and Planning

ECCE Early Child Care and Education

EoP Evaluation End of Program Evaluation

EoP Outcome End of Program Outcome

FacU Facilities Unit (MoET)

FinU Finance Unit (MoET)

IE Impact Evaluation

IO Intermediate Outcome

IP Implementing Partners

IS Implementation Strategy

ISU In-Service Training Unit (MoET)

JCSEE Joint Commission of Standards for Educational Evaluation Program

Evaluation standards KEQ Key Evaluation Question

M&E Monitoring and Evaluation

MEP Monitoring and Evaluation Plan

MFAT (New Zealand) Ministry of Foreign Affairs and Trade

MoET Ministry of Education and Training

MOV Means of Verification (data source)

MTR Mid Term Review

NER Net Enrolment Rate

OECD-DAC Organization for Economic Co-Operation and Development -

Development Assistance Committee PE Performance Evaluation

PPU Division of Policy and Planning Unit (MoET)

QED Quasi-Experimental Research Design

QTLSG Quality Teaching and Learning Strategy Group

RF Results Framework

SBM School Based Management (MoET)

ToC Theory of Change

ToR Terms of Reference

VEMIS Vanuatu Education Monitoring Information System (MoET)

VESP Vanuatu Education Support Program

VITE Vanuatu Institute of Teacher Education (MoET)

VSC VESP Steering Committee

5

1 Introduction

1.1 Program Overview

The Vanuatu Education Support Program (VESP) provides targeted support to the Ministry of Education

and Training (MoET) Corporate Plan with the long-term goals of improving education quality, providing

more equitable access to education and ensuring a well-managed education system.

Australia and New Zealand aid programs are providing funds for the program through a joint

partnership arrangement with program oversight provided through the VESP Steering Committee

(VSC). Coffey International Development is contracted to manage the bulk of the funds of behalf of

the donors. The donors are also providing technical assistance and funds directly to the Ministry for

School Grants and Early Childhood Care and Education (ECCE).

VESP commenced in September 2013 building on the earlier program of support provided to the

Vanuatu Education Road Map (VERM). The initial implementation period of VESP is until June 2017,

with a current budget of AUD $25.9m. A further one-year extension may be granted depending on

assessed progress towards the achievement of program goals and outcomes.

The focus of VESP support is on literacy and numeracy in schooling years 1-3 in the recognition that

strong early years’ literacy and numeracy programs form the bedrock of an education system with

higher retention rates and greater success in later years of schooling.

The program is being implemented through five interrelated strategies: 1) training and supporting

teachers to implement the new curriculum: 2) strengthening early childhood education; 3) engaging

communities through school based management; 4) efficient delivery of facilities; and, 5) improving

the capacity of MoET to manage an effective, de-concentrated education system.

Key outputs of the program and focus of support to MoET activities through VESP are summarized

as follows.

Key Output Focus of VESP support to MoET

New curriculum implemented for Years 1-3

Teacher guides and learning materials for the new curriculum

Professional development of teachers, principals and provincial support officers in the new curriculum.

Training (pre-service and in-service) programs for primary teachers updated to meet quality standards

Teacher Training and Professional Development Framework based on the approved teaching service standards.

Revised Diploma of Primary Education with new programs for pre-service, in-service and ECCE.

Trialling of flexible delivery mechanisms for upgrading qualifications of existing teachers.

ECCE strengthened to support early years learning.

Pilot approaches to strengthening ECCE in 60 Kindergartens (Implementing Partner - World Vision)

Review of national ECCE policy based on findings from the Pilot and other relevant sources.

Community engagement to support improved quality, access and participation in schooling.

Case studies of school community based initiatives to improve access of children with special needs.

Community-based advocacy campaign for enrolment of children in Year 1 at the right age.

Community contribution to new construction works on Tanna.

6

Key Output Focus of VESP support to MoET

School leadership to support improved learning in primary classrooms

Awareness raising and baseline for the Minimum Quality Standards (MQS) for primary schools

School improvement planning support for provincial education authorities, principals and school committees.

Instructional leadership training for principals.

Infrastructure planning to support quality facilities and improved access to primary schooling.

Asset survey of primary schools as basis for long-term asset management strategy

Capital works plans for provinces.

New classrooms for 18 schools on Tanna affected by Cyclone Pam.

Provincial management to support service delivery at school level

Provincial and school level planning and budgeting.

School registration and monitoring.

New teacher management processes.

School grant disbursement and acquittal processes.

M&E and data management system to support evidenced-based planning at central, provincial and school level.

The VESP Steering Committee, chaired by the Director-General of Education and comprising

representatives from the program partners, provides program oversight.

1.2 Overview of the Monitoring and Evaluation Plan

This document presents the revised VESP Monitoring and Evaluation Plan (MEP) that builds

upon earlier versions and is grounded in a series of consultations with all team members in April

2016. The result of these consultations has been the articulation of a revised Theory of Change

(ToC) and a revised results framework.

1.2.1 Purpose of the Monitoring and Evaluation Plan

The purpose of the MEP is to:

Articulate the revised program theory of change which underpins and guides VESP activities

and measures of success;

Establish a revised and updated result based performance framework for VESP;

Establish and outline two proposed evaluation studies to complement existing monitoring

arrangements.

Establish a plan for assessing the effects of (Australia) Department for Foreign Affairs and

Trade (DFAT) and (New Zealand) Ministry of Foreign Affairs and Trade (MFAT) funding on

the MoET and lives of children and young people in Vanuatu; and,

Establish a plan for learning to facilitate MoET, and VESP insight into what works and why.

This information will be used to enhance program delivery (both in relation to the current

program and future designs) and contribute to public knowledge.

1.2.2 Information needs of program stakeholders

7

The design of the MEP is based on the information needs of program stakeholders including the

GoV, VESP Steering Committee (VSC)1, the program team, and the MoET. Program monitoring

and evaluation activities, coupled with independent evaluation commissioned by VESP will

provide information for the purposes of:

Learning and improvement: to ensure that VESP and its partners deliver activities as

effectively as possible.

Management and accountability: to equip MoET, DFAT, MFAT and VESP to effectively

manage resources and account for the use of funding.

Enhancing public knowledge: to inform future program designs and decision making within

DFAT, MFAT and the education sector more broadly.

The focus of resources (including the level of rigour applied to evaluation and assessment

processes), the timing of evaluation events and the presentation of evaluation products have

been shaped by the intended use of the information produced. Table 1 provides a summary of the

information needs of these core stakeholder groups and the intended use of the information.

Table 1: Overview of program stakeholders’ information needs

Information required and M&E

Method Use of information Relevant stakeholder

Quality, reach and coverage of

VESP deliverables

(Performance Evaluation and

Program Monitoring)

Account for funding

Inform the delivery of the program

VSC

Program Team

DFAT/MFAT

Effect of Early Child Care and

Education (ECCE) initiative and

its effect on Kindergarten

quality, and child learning in the

home and school (Impact

Evaluation and project

monitoring)

Support MoET emphasis on

evidence-based program planning

(EBPP) and the use of Impact

Evaluation (IE) to evaluate new

initiatives prior to taking to scale

Improved project design

Communicate to education/ECCE

sector about what works in ECCE in

Vanuatu

MoET

Program Team

VSC

DFAT/MFAT

Education sector

in Vanuatu

Effect of New National

Curriculum on student learning

and achievement (Impact

Evaluation and project

monitoring)

Support MoET evidence-based

program planning and the use of IE

to evaluate new initiatives


Public communication about what

works in in Vanuatu

MoET

Program Team

VSC

DFAT/MFAT

Public in

Vanuatu

Effect of Community

Engagement through School

Based Management (SBM)

initiative (Impact Evaluation and

project monitoring) on school

performance in terms of student

access and learning outcomes

Support MoET emphasis on EBPP

and the use of Impact Evaluation

(IE) to evaluate new initiatives prior

to taking to scale


MoET

Program Team

VSC

DFAT/MFAT

Education sector

in Vanuatu

1 The VSC members include: Australian aid, MoET General Director and directors, New Zealand aid, VESP Secretariat.

8

Information required and M&E

Method Use of information Relevant stakeholder

Communication to education/ECCE

sector about what works in ECCE in

Vanuatu

Assessment of the program

Theory of Change Inform future program design

DFAT/MFAT

VESP

1.2.3 Scope of the MEP

This document sets out the strategy for monitoring and evaluation in relation to activities

undertaken by VESP from program inception in September 2013 to program completion in June

2018.

VESP will provide funding to a number MoET initiatives and will summarise and report on the

effects of this funding at the program level. VESP will support MoET and Implementing Partners

(IP) to strengthen their M&E capacity and will review IP M&E plans. VESP is not, however,

responsible for the monitoring and evaluation of IP M&E activities and performance indicators,

which are not included in this document.2

1.2.4 Summary of information sources

There are four principle sources of information that will inform VESP performance and reporting

against the evaluation questions (Refer to Section 6.1 below):

VESP program monitoring: Management reports will be used to monitor the

implementation of the program, the reach, coverage, and to the extent possible, quality of

program outputs. The progress towards program outcomes will be monitored through the use

of sample surveys and case studies to measure the extent to which outcomes are likely to be

achieved.

VESP monitoring studies: VESP will support thematic monitoring studies and an impact

evaluation of the ECCE Pilot, School Based Management, and National Curriculum initiatives.

The monitoring studies and evaluations will be used to: 1) supplement VESP and IP reporting on

project outputs and outcomes for the End of Program (EoP) Evaluation, 2) MoET EBPP; and 3)

information to the education sector on what works.

Independent program mid-term review: DFAT and MFAT will commission and manage an

independent mid-term review (MTR) of the program to assess its performance and impact, as

deemed appropriate. VESP will provide the MTR Team full cooperation in the execution of

their ToR such as access to program document and reports, interviews/group discussions

with staff from VESP, MoET, and IP, review and comment on MTR draft report and

recommendations as well as logistic support as appropriate.

Independent End of Program Evaluation: DFAT and MFAT will commission and manage

an independent EoP Evaluation of the program to evaluate VESP performance and impact.

VESP will afford the EoP Evaluation Team full cooperation in the execution of their ToR as

will be done for the Independent program mid-term review.

2 Program Evaluability Assessment

2 Notwithstanding this, VESP will support the conducting of a thematic evaluation on the ECCE initiative. Further detail is provided in

Section 6.1.2.

9

An informal assessment of the VESP program evaluability was taken based on a review of key

documentation, design and approval of the VESP Results Framework (RF), articulation of the

thematic monitoring studies, and consultation with DFAT/MFAT, VESP and MoET staff. The key

findings of the assessment are summarised below.

Articulation VESP program model

An important part of the evaluability assessment is clarifying the most appropriate way to articulate

the VESP program model. The VESP program model was updated during the development of the

VESP RF and Theory of Change (ToC). As a result of combining the activities of program updating

and RF, the evaluability of the program remained feasible and practical, although modified to

respond to the program adjustments. The changes to the program, the RF and its outcome

statements, indicators, baselines, targets, and means of verification (MOV) have been reviewed by

DFAT/MFAT, MoET managers and senior managers, VESP Secretariat and Advisor, and the MoET

M&E Unit.

End of program outcomes, intermediate outcomes, and means of verification

The design document sets out the program EoP Outcomes and Immediate Outcome (IO). These

align with the MoET EoP Outcomes and IO. VESP will contribute to these EoP Outcomes, IO and

achievement of the targets. An important part of the evaluability assessment was to determine

the: 1) the availability of reliable and verifiable baseline and EoP data and 2) likelihood that VESP

could achieve the targets within the lifetime of the program.

Baseline data exists or can be collected for each EoP Outcome. Refer to Annexes 1 & 2: Program

logic model and results framework for a complete list of outcome states, indicators, and existing

and proposed baselines and EoP data. Attention was given to work with the MoET M&E Unit to

ensure VESP indicators relied on MoET Vanuatu Education Monitoring Information System

(VEMIS) to the greatest extent possible. In the cases, when a non-MoET indicator was selected,

VESP worked with M&E Unit to ensure there was high probability of collecting the information

within a reasonable timeframe.

Impact Evaluation and Monitoring Studies

In case of the three program initiatives (ECCE pilot, new national curriculum rollout, and

community engagement through School Based Management) the data requirements exceeded

the design limits of the VEMIS. VESP and the MoET agreed to conduct two monitoring studies,

including sample surveys and case studies, and two independent evaluations to gather additional

information on the impact of the program on key intermediate outcomes The monitoring studies

are:

National Curriculum: Monitoring of New Curriculum Implementation: Initially a monitoring study

will be conducted using sample survey and case study methods to gather evidence of progress

towards the EOP outcomes by investigating the extent to which the new curriculum has

contributed to the improvement of literacy and numeracy skills of Year 1-3 students. Towards

the end of the program an impact evaluation will examine the effectiveness, efficiency and

sustainability of the initiative and determine the extent to which the EOP outcomes have been

achieved.

School Based Management: School Leadership Monitoring Study: This will use qualitative

research methods, such as focus groups and interviews to obtain information on the extent to

which schools have engaged with their communities to ensure that all children are attending

school. A monitoring study will be conducted to investigate the effectiveness of the community

advocacy plan in achieving program outcomes of increased community engagement and

equitable access to education.Impact evaluations are:

10

Independent Evaluation of VESP: DFAT conducts independent evaluations of the Program as

required to provide DFAT with independent technical and other advice on any aspects of the

Program and/or assist DFAT to assess the performance of the Services”

Definition of the scope of the program

The evaluability assessment sought clarification on the scope of the VESP program. Discussions

focused on defining the scope of VESP program as stated outcome statements, related

indicators, targets and Means of Verification (MOV).

Articulation of the program theory

Based on consultation and workshops conducted during the development of the program results

framework, the preliminary program theory of change was articulated and provisional agreed

upon by stakeholders. Following further feedback from MFAT/DFAT and IP the theory of change

and program logic have been revised to clarify and make explicit the main drivers of the expected

changes as they relate to the program outcomes. The program logic has been presented to the

VSC and MoET for comment and review. The revised program theory is summarised in Section 3.

Baseline data

The revised MEP has been prepared 18 months into program delivery with a large proportion of

activities underway. Most of the required baseline data has been collected through the annual

school survey and reported in the VEMIS. The monitoring of the national rollout of the new

curriculum and school leadership strengthening initiatives will use the baseline data to measure

improvements in educational outcomes over the life span of the program.

3 VESP Program Theory

The VESP program theory is based on the program design, lessons learned during the inception

phase, and consultation with DFAT and the VESP program team. In April 2016 a new M&E Adviser

commenced with the program and undertook in initial assessment of the existing M&E Plan and

Framework. An important component of this assessment was a series of consultations with MoET

counterparts and VESP advisers to discuss a revised ToC that reflected current program priorities

and activities. The revised ToC represented a streamlined approach to program implementation

and management and importantly aligned all VESP support for MOET activities to existing Minimum

Quality Standards (MQS) The revised program logic model encapsulating the program theory is

given in Annex 1 and Annex 2 provides the revised results framework. The program theory forms

the basis for the MEP.

3.1 Theory of Change

A basic theory of change outline for the VESP program is summarized in the diagram below and

is based on the VESP design document3. The theory of change presents the linking of program

strategies, at the school level, and how these lead to incremental changes in institutional

arrangements. In particular, the devolved management of education services leads to improved

school management which results in better school facilities and resources; strong school

leadership leads to improved engagement with the community which results in better access to

schooling, and the implementation of a new curriculum leads to improved teaching and learning

practices which results in better learning outcomes. The diagram also shows how the three main

program outcomes are linked, and contribute to the programs long-term strategic goals.

3 Vanuatu Education Support Program Design Document (AusAID, 2012)

11

Figure 1. Theory of Change for VESP program

As part of the revision process the basic ToC was tested and revised. A more strategically aligned

ToC was prepared that presented a subtle shift in overall strategy, implementation and program

arrangements. A revised ToC is included as Attachment 1.

Key changes included:

The shifting of MoET Program Goal statements and VESP End of Program Outcomes to

beyond the life of the current phase of the program to reflect the on-going nature of the

program as a potential 15-year investment.

An overall reduction in the number and revision of outcomes into immediate and

intermediate statements to better reflect causal linkages between current program

implementation and expected results.

The alignment of two realistic monitoring studies linked to key outcomes.

The structuring of key outputs into three broad program strategies - numeracy and literacy,

access and service delivery.

A revision in key indicators to better reflect existing and available data sources to

demonstrate more practical progress towards defined outputs and outcomes.

The alignment of all VESP M&E activities to MoET's Minimum Quality Standards (MQS) for

primary schools.

The last point mentioned above is a significant strategic shift and highlights the importance of VESP's

interventions in assisting MoET meet its own development outcomes. Importantly it also embeds VESP's

program activities within MoET strategy so as not to implement parallel programming. Annex 5 is an

attempt to map the linkages between the MQS, VESP activities and the proposed indicators and linkages

to the proposed monitoring studies. This mapping exercise also acts like a baseline in terms of the current

level of compliance and also the desired state the program seeks to reach in terms of alignment and

relevance.

3.2 VESP Results Framework

VESP activities are ultimately designed to improve numeracy and literacy, access and overall service

delivery. Revised program outputs and outcomes have been put into a results framework format (refer

annex 2).

Decentralised Education Services

Improved School

Management

Better Facilities & Resources

Better Access to Schooling

Improved Community

Engagement

School Leadership Program

New Curriculum

Improved Teaching &

Learning

Better Learning

Outcomes

12

3.3 Program Monitoring

This section of the MEP discusses the monitoring of program activities, outputs and outcomes. The section provides a statement of the purpose of monitoring activities, outputs and outcomes; a detailed description of the performance indicators used to measure the different program logic elements; a schedule of the monitoring activities planned to verify the results; an outline of the mechanisms used to quality assure the results; an explanation of the use of the results in policy development and planning; and the accountability mechanisms for producing and publishing the results.

3.4 Activity Monitoring

The monitoring of program activities is an important management function to ensure that the program activities are delivered on time and within budget for each of the five VESP strategies. The key strategies are: train and support teachers to implement the new curriculum; strengthen ECCE delivery; engage the community through school-based management; efficient delivery of school facilities; and improved MoET capacity to manage the education system. For each activity there are two process indicators that need to be regularly monitored and results reported on a monthly basis4. These are: progress against annual plan, and percentage of expenditure variation against budget.

Table 2. Performance Indicators for Program Activities

Program Activity Process Indicator

1.1 Curriculum Implementation

Progress against annual plan (on target/ on schedule,

on target/ delayed,

off target,

not started)

% expenditure variation against budget

1.2 Language Policy Implementation

1.5 Assessment, Monitoring & Reporting

1.3 Training to Support New Curriculum

1.4 VITE Institutional Strengthening

2.1 ECCE Policy Implementation

3.1 School Leaders Program (Special Needs)

3.1 School Leaders Program (Community Engagement)

3.1 School Leaders Program (Enrolment at Right Age)

4.1 School Construction Program*

4.2 School Maintenance Plan

5.1 Decentralisation of Education Services

5.2 Evidenced-based Policy and Planning

The monitoring of program activities is conducted by MOET Activity Managers, supported by VESP TAs, who are responsible for ensuring that the program activities are delivered on time and within budget. Progress with program activities is reported monthly to all VESP partners by the VESP Secretariat and financial reporting is provided to DFAT by the VESP Program Manager on a monthly basis. The costs of the monitoring activity are included within the respective roles and responsibilities.

4 DFAT IET and Pacific Branches. Evaluation Capacity Building Program: Guidance for Implementation Partners on DFAT Progress Reporting (2014 update)

13

Table 3: Schedule of Activity Progress and Financial Reporting

Monitoring

Task

Data Collection

Method

Timeframe Role &

Responsibility

Deliverable &

Reporting

Indicative

Costs

Activity

Progress

reporting

Monthly Report

template

End of

month

Activity Manager

& VESP TA

Monthly report

(15th of next mth)

$Nil

Activity

Financial

reporting

Financial System

& Budget

End of

month

Activity Manager

& VESP TA

Monthly report

(15th of next mth)

$Nil

The VESP Secretariat Manager who is responsible for the implementation of the program as specified in the program plan assures the quality of reporting activity progress. The internal process for quality assurance includes the monthly monitoring of the activities VESP TAs and reports of Activity Managers; and the financial management of program funds aligned with the program budget. The monthly reports provide details of the key inputs, outputs and challenges/risks to the activity. The external process for quality assurance includes the program mid-term review, and the financial audit of program finances. The monthly reports from activity managers are used to assess the extent to which the program is being delivered according to the annual plan and the budget (refer template below). That is, any delays in the implementation schedule will be accounted for, especially where the delay is likely to result in the activity not being completed within the planned timeframe. The monthly report also described and explains any remedial action taken in response to unexpected delays. Also, an assessment of the sufficiency of inputs to meet the end-of-program outcomes is provided, and if insufficient, an explanation of the causes and an assessment of the implications to the program are communicated to stakeholders. The monitoring of program activities provides evidence to account for the implementation of activities as planned for in the annual plan. The VESP Secretariat and MoET Management informs stakeholders of the extent to which the activities are on schedule and within budget. Stakeholders need this information to decide whether further inputs or resources are required to ensure end-of-program outcomes are achieved. The results are provided to VESP development partners on a monthly basis.

Table 4: VESP Monthly Activity Monitoring Template

Activity Progress Indicators Status Issues/risks

Activity output

specified in

approved plan

Target set in approved

plan

G - On schedule

A - Behind schedule – low risk

R - Behind schedule – high risk

G - No issue/risk

A – Issue/Risk under

control

R - Major issue/risk

3.5 Output Monitoring The monitoring of program outputs is important for assessing whether program deliverables are being delivered as planned. In the VESP program the key outputs are aligned to the three major streams of program implementation - literacy and numeracy, access and service delivery. For each stream, there are key outputs that need to be regularly monitored and results reported on a six-monthly basis. A key performance indicator measures each activity output. The output indicators are disaggregated by province.

Table 5. Performance Indicators for Program Outputs

Program Output Key Performance Indicator

Literacy and Numeracy

14

Output 1.1 New curriculum implemented in Years 1-3

# of schools receiving new curriculum (for Years 1-3 as rolled out)

Output 1.2 Training (pre-service and in-service) programs for primary teachers updated to meet quality standards

# of schools with teachers trained in new curriculum (for Years 1-3 as rolled out)

Output 1.3 ECCE strengthened to support early years learning

# Kindergartens implement ECCE policy of school readiness

Access

Output 2.1 Community engagement to support access and participation in early years of schooling.

# of schools that engage with communities to enrol children at the right age into year 1.

# of schools that engage community support for accessible facilities in primary schools

Service Delivery

Output 3.1 : Infrastructure planning to support quality facilities and improved access to primary schooling.

# of primary schools completing an asset survey

#of schools with new classrooms built and occupied

# of schools implementing maintenance plans in line with MoET guidelines.

Output 3.2 School leadership to support improved learning in classrooms

# schools with school improvement plans (SIPS)

# of schools with head teachers trained in new curriculum

Output 3.3 Provincial management to support service delivery at school level

# of eligible primary schools receive and acquit school grants on time

# provinces with capacity for devolved responsibilities for the management of teachers and schools.

The monitoring of program outputs is oversighted by the VESP M&E Adviser and prepared by Activity Managers with support from VESP TAs who are responsible for ensuring the program deliverables are delivered as planned. The reach, coverage and quality of key outputs are closely monitored and reported in Activity progress reports (refer template attachment)..

Table 6: Schedule of Program Outputs Progress Reporting

Monitoring

Task

Data Collection

Method

Timeframe Role &

Responsibility

Deliverable &

Reporting

Indicative

Costs

Outputs

progress

reporting

New Curriculum

Distribution and

Training Reports

On-going CDU, VITE (ISU)

& VESP TA

6 month

Progress Report

$Nil

School Leadership

Program Reports

On-going Education Services

(SBMU), HRDU

& VESP TA

6 month

Progress Report

$Nil

SECCE Pilot

Monitoring Reports

Six-monthly ECCEU & WVV

6 month

Progress Report

$Nil

Outputs are assessed against activity targets set out in the VESP annual plan, and by qualitative feedback from stakeholders. The feedback in relation to training sessions and workshops are evaluated to ascertain whether the training is relevant, meets the needs of the participants, and of a quality standard in terms of content, materials, and facilitation. The reported results are used jointly by MoET Management and VESP Secretariat to monitor the progress and assess the quality of implementation to date. The results also inform the extent to which the implementation activities are reaching the right people, covering the right schools, and providing the right kind of training. It is therefore important that each training report includes details of the

15

training (type, dates, location) as well as details about the participants (gender, age, experience, qualifications). The output results provide a justification of the level of expenditure on specific activities, and inform the stakeholders of the cost-efficiency of their investment in the overall program. The results also enable a joint assessment of the mutual accountability mechanisms established by the program. The results are presented to the VESP Steering Group on a half-yearly basis.

3.6 Outcome Monitoring The monitoring of program outcomes is tracking the progress of the program towards the sustainable achievement of the end of program outcomes. In the VESP program the main intermediate outcomes are linked to schools implementing the new school curriculum, children commencing school at the correct age and transitioning accordingly and provincial officers providing the necessary support to school leaders. In the Theory of Change a number of outcomes are highlighted. This is to show the causal linkages between outputs and the three streams of implementation. Not all outcomes will be assessed as part of VESP as they have less strategic value in linking to the overall end of program outcomes, however they are important context setters for the intermediate outcomes that will be assessed. The three intermediate outcomes (marked in red boxes in ToC) and contained in the table below will be measured and are linked to proposed monitoring studies. Table 7: Performance Indicators for Program Outcomes

Performance Indicator

EOP 1 Improved literacy and numeracy of

Year 1-3 students (female and male)

% of students meeting literacy standards in Year 4

% of students meeting numeracy standards in Year 4

Intermediate Outcome: % increase in number

of schools with teachers implementing the new

curriculum for Years 1-3 using effective

teaching and learning methodologies.

% of schools using national curriculum for Years 1-3.

EOP 2 Children (girls and boys), including

with special needs, have access to

kindergarten and Years 1-3

Enrolment Rate (NER & ASER) for kindergarten and Years 1 - 3 (girls and boys).

% of children (girls and boys) attending kindergarten and Years 1 - 3 with special needs

Intermediate Outcome: : Increase in number of

schools and kindergartens implementing

inclusion initiatives to improve access for

children to Kindergarten and Years 1-3.

% of children (boys and girls) age 6 entering year 1 who attended kindergarten

Retention (survival rates) for Year 1 students to Year 4)

EOP 3 Effective education service delivery for

kindergarten and years 1-3 at central provincial

and school level.

% of primary schools that can demonstrate improved achievement of Quality Minimum Standards (from 2013)

Intermediate Outcome 3: Provincial officers

providing necessary support to school leaders

% of schools that demonstrate effective management of school grants

% schools that demonstrate improved school management

The monitoring of program outcomes is undertaken by monitoring officers (VESP & MoET M&E officers) who have overall responsibility for tracking the progress of the program towards the achievement of the end of program outcomes. Activity managers play a vital role in ensuring that quality data is collected and reported for each monitoring task supported by the VESP TA for the activity. There are three critical data sources that are used to monitor program outcomes: VEMIS, MQS and VANSTA. The costs of these activities are included in the respective roles and responsibilities.

16

Table 8: Schedule of Program Outcomes Progress Reporting

Monitoring Task Data Collection

Method

Time-

frame

Role &

Responsibility

Deliverable &

Reporting

Indicative

Costs

Program

Outcomes

Progress

Reporting

VEMIS survey

Annual

(30 March)

PPU

& VESP TA

Statistics Digest

(annual)

$NIL

MQS standards

Yet to be

determined

(rolling)

SBMU

& VESP TA

Monitoring report

(annual)

$NIL

VANSTA

assessment

3 yearly

(30 Oct?)

CDU & VESP TA Assessment

report

(Baseline for

VANSTA set

2016, revised

VANSTA testing

starts 2017 )

$NIL

The quality of the outcome indicators are assured through the validation and verification mechanisms established by the MoET data management processes. Specifically, the VEMIS system is quality assured through processes for data entry, editing, analysis and reporting, the VMQS standards are verified through evidenced means of verification and independent inspection, and the VANSTA assessment is conducted using expert technical assistance and international assessment standards. The reported results will be used by MOET Management and VESP Secretariat to monitor progress towards the strategic goals As explained in the DFAT guidance for progress reporting5, it is important to ensure that the end-of-program outcomes are agreed with all stakeholders. In this case, both the VESP and the MoET end of program outcomes are aligned with the goals of the Vanuatu Education Road Map. The results of the outcome monitoring provide evidence to stakeholders on the success of the VESP program and its contribution to the achievement of the road map goals. The intermediate outcomes and progress toward the end-of-program outcomes are described in the annual report and the results presented to the VESP Steering Group. The community strategy ensures that results are communicated to all stakeholders in a timely and consistent manner.

4 Evaluation Questions

Evaluation questions have been developed to evaluate aspects of the program theory and inform

data collection, analysis and reporting (see Table 9 below). Responses to the evaluation questions

will elaborate on the adequacy of VESP performance to meet program EoP Outcomes. The EoP

Evaluation will be guided by the following specific objectives:

To determine the extent to which VESP results were achieved, and the factors that facilitated

or hampered achievements; and

To compile lessons learned and recommendations to inform and guide MoET and the

development partners towards the next phase of the program or other programs.

Evaluation questions are based on the OECD-DAC criteria of relevance, effectiveness, efficiency,

potential sustainability and impact6.

5 DFAT IET and Pacific Branches. Evaluation Capacity Building Program: Guidance for Implementation Partners on DFAT Progress Reporting (2014 update) 6 OECD-DAC criteria are a set of internationally recognized and applied set of evaluation principles for development assistance.

17

Table 9: Key evaluation questions (KEQ)

Key Evaluation Questions

1. Has the program been implemented as planned?

2. Does the program remain relevant to meeting the education needs of children in

Vanuatu?

3. Did the program achieve the expected results?

4. How efficiently were outputs achieved from available resources?

5. To what extent are program benefits likely to be sustained after funding has finished?

6. What contribution did VESP make to gender, disability, social inclusion, and child

protection?

5 Monitoring and Evaluation Approach

The design of the MEP is based on the priorities of the VESP program and on the information

needs of DFAT, MFAT and other key stakeholders. The particular features and principles of the

MEP are:

Responsive: VESP is designed to be able to respond to changes in the operating context in

Vanuatu and within the MoET. The MEP is designed to be flexible to adapt to changes in

program focus or approach. Higher-level goals and End of Program Outcomes are sufficiently

broad to accommodate a range of different types of activity.

Proportionate: Proposed monitoring and evaluation is based on the resources available and

also takes into account the level of investment made by DFAT/MFAT in the program.

Emphasis on generating robust evidence will be proportionate to the size of the funding.

Practical: The MEP is designed on good practices in results based management in program

monitoring, evaluation, evidence based planning, and reporting. The MEP is designed to

utilise existing MoET M&E system as well as develop new areas of EBPP, M&E, and

reporting requested by the MoET. The VESP and MoET RF, indicators and annual

benchmarks, and targets have been harmonised. (The MoET RF also aligns to key

stakeholders such as the UNESCO Education for All, and United Nations Millennium

Development Goals on education). The monitoring studies needed for the MoET as part of

their evaluation of pilot initiatives and EBPP requested the VESP EoP Evaluation. The MEP

provided opportunities to act upon the VESP activities related to capacity development of

MoET M&E staff.

Learning-focused: On-going assessment and reporting of VESP/MoET activities will be used

as opportunities to build the capacity of MoET to learn from their own practice and the

practice of others. MoET has committed to evidence-based planning and development for its

corporate plan, annual work plan, and taking to scale pilot programs such as the ECCE

initiative.

Useful: The information needs of DFAT, MFAT, VESP and MoET have shaped the focus of

the MEP: the structure of the MEP is in line with DFAT’s reporting requirements and the

OECD-DAC criteria for evaluating development assistance; MoET will benefit from M&E

capacity building support at an organisational level and individual; and opportunities for

18

reflection and knowledge sharing within the sector are provided for. Further detail of the

approach to communication and dissemination of evidence is provided in the VESP

Communication Strategy.

International evaluation principles. VESP and the MoET have agreed to adopt international

M&E principles, good practices as articulated in the 2014 Department of Foreign Affairs and

Trade (DFAT) and Pacific Branches. Evaluation Capacity Building Program Monitoring and

Evaluation Standards; Joint Commission of Standards for Educational Evaluation Program

Evaluation standards (JCSEE); OECD-DAC evaluation criteria for development assistance,

United Nations Results Based Management, Evidence Based Program and MoET Management

Response to Evaluation Recommendations. Methodology

5.1 Sources of information

There are a number of key sources of information that will be drawn on to assess the

performance of VESP against the program expected results and key evaluation questions. These

include: Open VEMIS system; program management reports; activity monitoring studies;

performance evaluation; independent program mid-term review; and independent End of Program

Evaluation

5.1.1 Open VEMIS system

Over the next few years MoET will introduce a web based school management system called

Open VEMIS to all schools. The Open VEMIS system will manage all school information including

student records, attendance, finance, pay, teacher registration, facilities, assets, and some exam

information. Schools, regional offices, the Ministry and some community members will access

different areas of Open VEMIS. The Open VEMIS system is expected to be available in mid-2016

for use by VESP and MoET staff, and will be used to provide quality time-series data for

monitoring program outcomes. The following records will be accessible in the Open VEMIS

system:

School Records: A register of registered schools and profile information will be kept on the

Open VEMIS system. This includes details of registration, finance (including school grants),

facilities and infrastructure, and school minimum quality standards. PEOs maintain

information on school registration directly on Open VEMIS.

Student Records: The new survey form is designed to save time, improve accuracy and also

mark the beginning of the transition to Open VEMIS. The first step of the transition is to collect

individual student records. A completed student record form is submitted with the rest of the

annual school survey to the PEO. Regional Offices enter individual student records in Open

VEMIS.

Teacher Records: The ministry no longer requests all teacher information as part of the

annual school survey. The ministry provides school principals with known details of teachers

who then audits and correct this information, indicating necessary corrections for the ministry.

Schools return audited teacher records with other survey forms to PEOs. PEOs will update

ministry teacher data. In future, Open VEMIS will give on line access to all teacher pay

information.

5.1.2 Program monitoring and reports

Monthly staff report: VESP staff members interact with MoET and the sector more broadly on

a daily basis. All VESP team members submit monthly reports, which provide insights into

changes in the operating environment and to the performance of MoET. Information from

these staff reports contributes to the program level performance assessment. Field visit

reporting is part of this monthly staff report.

Six monthly and annual progress reporting.

19

Twice annual VESP to the VSC: The VESP Secretariat will facilitate meetings for VSC

(coincide with the VESP Six Monthly Reporting) to discuss program achievements against

targets, lessons learned, challenges and risks, and key activities in the next six months.

Program implementation will be monitored through monthly and six monthly progress

reporting to assess the efficiency and effectiveness of delivery and use of available resources

(see Annex 3 and 4 for reporting templates).

Performance assessment will be based on the quality of outputs and training, timeliness, risk

management, quality of reporting, resource mobilisation, use of M&E information for process

improvement and evidence-based decision making, and dissemination of lessons learned

5.1.3 Proposed Monitoring Studies

VESP will conduct two monitoring studies to provide robust information about the effects of VESP

funding and to test key elements of the program theory. The focus of the monitoring studies on

assessing the extent of the impact of program activities on program outcomes has been agreed with

DFAT, MFAT and MoET. In each case, the monitoring studies will be used for the EoP Evaluations

as well as by the MoET as part of its EBPP of pilot initiatives.

The monitoring studies will focus on assessing the impact of VESP funding in key areas of the new

national curriculum, school based management, and MoET capacity development. The studies focus

on the performance of the initiative according to their relevance, effectiveness, efficiency,

sustainability and impact. The studies will be outsourced and separate ToR's and evaluation scopes

have been prepared will be advertised in July 2016.

An informal reference group will be established for each monitoring study to provide oversight which

includes ensuring: relevance to MoET/VESP plans, technical quality of the monitoring, compliance

with JCSEE evaluation ethics and DFAT standards, and issues of independence and conflict of

interest, and development of the Management Response to address recommendations.

Details of each monitoring study is provided below

Monitoring Study 1: Curriculum Implementation Monitoring Study

The aim of the monitoring study is to gather evidence about the extent to which the intermediate

program outcome, schools are implementing the new curriculum in Years 1-3 (literacy and numeracy)

is being achieved. Achievement of this intermediate outcome is fundamental for realization of the end

program outcome (strategic goal 1). The purpose of the study is to:

1. Determine the extent to which implementation of the new curriculum in Years 1-3 is occurring as intended, and

2. To understand why implementation might be successful or less successful in particular contexts to inform future practice.

This formative approach to monitoring will provide the Ministry with valuable information to review and

then revise aspects of curriculum implementation that are not showing to be as effective as they

should.

Monitoring Study 2: School leadership and management

The monitoring of the MQS standards will provide valuable information of the effectiveness of School

Based Management (SBM) initiatives to improve school management and leadership. The School

Leadership Program (SLP) is coordinated by SBMU and comprises three components with

responsibilities for implementation shared across different units in the Ministry. The three components

are (i) Leading for Learning, (ii) Leading for Quality Improvement and (iii) Leading for Improved

Access. A mapping process of current work against relevant MQS has been completed in May 2016.

The purpose of this study is to:

1. Determine the extent to which implementation arrangements have facilitated school leadership, and

20

2. Assess leadership approaches taken to improve school performance through strategic planning

and application of grants.

The study will follow a case study methodology where a purposeful sample of schools would be

selected.

5.1.4 Independent mid-term and end program evaluation

DFAT/MFAT will commission and manage an independent mid-term review in 2015/6. This review will

provide an independent assessment of the relevance, effectiveness, efficiency, impact and

sustainability of the program and will provide recommendations for how the design or delivery of the

program might be improved moving forward. DFAT/MFAT will commission and manage an

independent end of program evaluation in 2017/18 (to be advised). This evaluation will provide an

independent evaluation of the relevance, effectiveness, efficiency, impact and sustainability of the

program and will provide recommendations for how to improve the design or delivery of the next

program.

5.2 Relevant aspects of the context and key M&E risks

VESP built its RF and indicators using data from the MoET Open VEMIS information system to the

greatest extent possible. All new information requirements were discussed and agreed to with the

MoET M&E Unit and the Open VEMIS Team. There is a potential risk if information required for M&E

is not available from the Open VEMIS system due to delays in implementation. This risk will need to

be monitored and if necessary mitigated by ensuring the operation of the existing VEMIS system.

There is a potential risk to obtain the baseline and information required for the monitoring studies and

performance evaluation related to support and funding. The risk to support for the monitoring studies

was lowered by discussing and agreeing, in principle, to the monitoring with key stakeholders in the

MoET, VESP Secretariat, VESP Team, MFAT and DFAT. The issue of funding will be monitored. The

scale of the monitoring can and will be adjusted should funding become an issue.

The final potential threat to the baseline and EoP information relates to the collecting of Literacy and

Numeracy rates. VESP is assisting the MoET to design the assessment protocol. This is highly

technical, potentially expensive and labour intensive endeavour. To mitigate much of the risk, VESP

and MoET established a technical working group that will oversee the initiative. The senior most level

of the MoET supports the initiative.

5.3 Data collation, analysis and reporting

The process of data collation and collection will be ongoing throughout the program – data collection

tools will capture information according to the key evaluation questions and outcome areas, while

allowing space for reporting against emergent themes. The VESP M&E Adviser will have

responsibility for collating information provided by VESP and MoET program focus points used for

VESP six monthly reporting, annual reporting, VESP MTR, and EoP Evaluation.

Key findings against program targets will be presented to VESP management and the VSC in the

VESP six-monthly and annual reports as well as VSC Meeting. Findings from the monitoring studies,

MTR, and EoP Evaluations will be presented in formal presentation meeting. Data analysis and

reporting will take into account the quality of the data provided and will focus on:

Foremost program information requirements (type of data, timing, reporting modality and format)

are articulated by the key stakeholders (DFAT, MFAT, MoET, VESP, VSC, and MoET Activity

Managers).

Focus reports on program achievements against expected results including reach, coverage,

quality, lessons learned, challenges and mitigation strategies, and key activities in the next

reporting cycle.

21

Synthesising data to identify trends of patterns that require further analysis or will be useful in

informing future practice such as the effectiveness of particular delivery methodologies.

Placing an emphasis on the quality of data by improving VESP data collection, the MoET

information system including instituting a Data Quality Assurance mechanism. Triangulating of

sources of evidence will be done for all monitoring studies, MTR, and the EoP evaluation.

Data Utilisation. VESP will work with the MoET to ensure data is used in program planning and

monitoring. VESP and MoET have agreed that Annual Work Plans apply EBPP principles; and

that recommendation from MoET commissioned evaluations uses the Management Response

approach to ensure VESP/MoET accountability in responding to recommendations.

Monitoring key program risks

Key program risks will be monitored as part of the M&E reporting. Existing and emerging risks will be

reported at monthly meetings and summarised in six monthly progress reports together with their

assessment and management responses.

The VESP Risk Management matrix is the primary tool used for monitoring risk and minimising

impacts. This is updated regularly and included in six monthly and annual progress reports.

While risks can affect all aspects of delivery and implementation they are particularly relevant to

understanding factors that:

Have enabled or inhibited the performance of the program (KEQ 5)

Are impacting on the achievement of sustained or equitable outcomes for different groups

(KEQ 9).

6 Communication and Dissemination

In order to meet the information requirements of the stakeholders (shown in Table 1) it is important

that program performance information and evaluation findings are communicated clearly and in a

timely manner. The planning, budgeting and reporting cycle of the GoV is shown in Table 3 aligned

with the planning, budgeting and reporting cycle of VESP. The VESP Annual Implementation Plan

(AIP) is aligned with the MoET’s budget planning and annual program implementation. Regular

monthly and six monthly progress reports to the Director General will assist the MoET in identifying

progress towards targets and outcomes and making adjustments to forward plans.

6.1 External communication

External communication is largely for the purpose of program accountability and to enhance the

visibility of the program and the achievements of VESP working with the MoET. The VESP

Communication Strategy being developed will identify key messages to be communicated, target

audiences and a range of methods of communications. The MEP communication and dissemination

section will be updated once the VESP Communication Strategy is finalised. Table 4 summarises the

key tools for communicating findings generated through program monitoring and evaluation. The

VESP Communications Strategy will provide further detail about general program outreach and

communication.

6.2 Internal communication

It is important that VESP use performance information in order to inform and continuously improve its

activities. Monthly meetings are held for all staff and these meetings are an opportunity to summarise

and share the data from the staff reports. Monitoring visit reports will also be shared amongst all staff

to promote lessons learned and cross learning between teams and projects. In addition, all staff are

requested to submit a written monthly report highlighting key achievements and provide updated data

for the key performance indicators.

22

Table 10: Vanuatu Public Service Annual Planning - Budget - Reporting Cycle

Jan Feb Mar Apr May Jun July Aug Sept Oct Nov Dec

Ministry Planning

Prepare / Update Ministry Corporate Plans

Corporate Plans

considered by Central Agencies

Corporate Plans

submitted to Council

of Ministers

Final Review of Plans Post Budget, if necessary

National Budgeting

Budget Ceiling

Determined

Budget Figure / Narrative

SDTs Submitted to MFEM

Ministry / Agency Budget

Presentations

MBC Considers

Budget Estimates

Appropriation Bill

distribution to MPs

National Budget

considered by

Parliament

National Budget

Published

GoV International Performance Monitoring

Monthly Performance Report to Director General as Part of Monthly Meetings (Public service Instrument Instruction No. 1-1999)

GoV Annual Reporting

Prepare Annual Report Table Annual Report

VESP Planning and Reporting

Cycle

Prepare draft AIP

with MoET Activity

Managers

Confirm proposed

VESP support for MoET plan

with DG

VESP Annual

Implementation Plan (AIP) to DFAT

AIP to Steering

Committee for final

endorsement

VESP Monthly Progress Report to Director General

Prepare Six Monthly Report with MoET Activity Managers and present to

VESP Steering Committee

Six Monthly Progress Report to

DFAT

Prepare Six Monthly Report with MoET Activity Managers and present to

VESP Steering Committee

Six Monthly Progress Report to

DFAT

23

Table 11: Overview of external communication

Medium of

communication Content Purpose of communication

Primary

audience

Responsibilities for production &

communication

Annual Implementation

Plan

Assessment of program

performance, the adequacy of

performance and analysis of key

contextual factors influencing

program performance

Management and Accountability:

Account for funding and enable

strategic decision making in relation

to program management and

funding

VESP, VSC,

DFAT

Senior Program Manager will be

responsible for report writing and

submission with significant support

from VESP team and M&E Adviser

6 month progress report Summary of key results against

expected results and

identification of serious issues or

challenges faced by the program

VESP: accountability to results and

raise issues requiring management

attention

VESP, VSC,

DFAT

Senior Program Manager will be

responsible for report writing and

submission with significant support

from VESP team and M&E Adviser

Monitoring reports Monitoring of key aspects of the

program, and assessment of the

effects of VESP funding in target

sectors

VESP accountability to results and

inform future program design

MoET EBPP and taking to scale

successful pilot initiatives

Learning: education section

knowledge about what works

VESP, VSC,

DFAT, civil

society

M&E Adviser will be responsible for

the monitoring in terms of ensuring:

independence, ethics, quality,

dissemination, management response

VSC Meeting: VSC

review of VESP six

monthly results

Summary of VESP six monthly

results, lessons learned from

program implementation and

proposed actions for program

improvement to the VSC

VESP: accountability to results

VSC: learning from program

delivery

VESP, VSC,

DFAT,

VESP Secretariat Manager with

support from the M&E Advisor, VESP

Team, and MoET Activity Mangers

24

7 Roles related to VESP M&E

An overview of the roles and responsibilities of staff involved in monitoring and evaluation is provided

below.

7.1 VESP Monitoring and Evaluation Adviser

Coordinate and finalise the MEP, and oversee revisions to the MEP and reporting templates as

required.

Design project-level and activity-level performance assessments with MoET M&E and VESP

program team to assess the performance of VESP and report against end of program outcomes.

This includes providing on-going capacity develop for MoET staff in M&E and report.

Design monitoring studies with the MoET M&E Unit and VESP program team and provide technical

leadership in the implementation, reporting and utilisation of the studies. This includes providing on-

going capacity develop for MoET staff in M&E and report.

Provide high-level technical support to the program in participatory approaches to M&E, learning,

and other approaches as required (The M&E Advisor in collaboration with VESP Program Team is

to define evaluation/assessment methods, key questions, data collection tools, data analysis,

reporting and dissemination).

Provide quality oversight and technical input in compiling and analysing data for 6-monthly and

annual reports for DFAT, including developing baseline datasets for all indicators and managing the

collation and collection of data to provide evidence of progress towards achieving outcomes.

Contribute to maintaining the harmonisation of the VESP MEP to the MoET MEP.

Work with VESP Secretariat, VESP program team, and MoET to disseminate findings, lessons

learned.

7.2 VESP Program Implementation Team

The VESP program implementation team comprises the technical advisers and specialists (national

and international) engaged through VESP to support MoET Activity Managers their implementing

teams in each of the program activities. The role of the Program Implementation Team is to:

Work with MoET counterparts and M&E Adviser to articulate a clear program theory and expected

results; and in collaboration with the M&E Advisor define evaluation/assessment methods, key

questions, data collection tools, data analysis, reporting and dissemination.

Document the effects of VESP supported activities; and report achievements against expected

results and targets.

Undertake field visits, as appropriate, to document the effects of VESP supported activities

implemented.

Facilitate support to MoET counterparts and /or IP from the M&E Advisers as required.

7.3 VESP Secretariat

Oversee and coordinate VESP reporting and approve monitoring reports prior to dissemination.

8 Workplan

The cost of the monitoring studies is indicative and depends on the level of support able to be

provided by the M&E Adviser, and the final designs of the studies. It is anticipated that some of the

work involved in the monitoring studies will be subcontracted due to the intensive nature of the

research and analysis. Independent evaluations will be undertaken towards the end of the program

Table 12: VESP M&E Workplan

Tasks 2015 2016 2017

1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12

Finalisation of the MEP

Finalise the MEP with VSC approval

Project-level monitoring

Support MoET/VESP to develop M&E plans

Undertake Monitoring Studies (CDU, SBM)

Deliver ongoing technical support to MoET,

VESP in the implementation of M&E plan

Ongoing capacity building M&E

Capacity building activities for MoET and

M&E unit

Data Quality Assurance

Institute quality assurance

Reporting

Monthly staff progress report

VESP six-monthly report

VSC Meeting: VESP six-monthly report

VESP annual report

Information utilization for program planning and budget

Annual Workplan Development

(MoET and VESP)

Management Response

Annex 1

VESP Program Theory of Change

Annex 2 VESP Results Framework (revised July 2016)

Goal 1: Improve the quality of education in Vanuatu

Result Verifiable Indicator

Baseline

Achievement

Data Source

Responsibility

End-of-Program Outcome 1 Improved literacy and numeracy of Year 1-3 students (male and female)

% of students meeting literacy standards in Year 4

Year 4 Total: 25% at L3

Year 4 Total: VANSTA

2009 EAU (CDU)

Year 4 Total: 54% at critical level

Year 4 Total

2010 (VANEGRA) Comprehension: 42% (English), 32% (French) achieved average score. Reading Fluency: 35 (English), 32 (French),) average corrected words per minute Non-readers: 10.2% (English), 16.3% (French),

2015 sample testing: Comprehension: 50% (English), 52% (French), 57% (Bislama) achieved average score. Reading Fluency: 71 (English), 56 (French), 53 (Bislama) average corrected words per minute Non-readers: 5.5% 9English), 6.3% (French), and 18.9% (Bislama)

VANEGRA 2010

VESP 2015

Baseline

EAU (CDU)

% of students meeting numeracy standards in Year 4

VANSTA Numbers: 66% at L3 or better Operations: 34% at L3 or better

VESP Baseline Numbers 93% at L3 or better Operations 81.4% at L3 or better. This appears to show an improvement however it should be noted that the baseline test was delivered orally in Bislama and VANTSA was a written test in English and French.

VANTSA 2009 VESP

Baseline

EAU (CDU)

Intermediate Outcome 1: % increase in number of schools with teachers implementing the new curriculum for Years 1-3 using effective teaching and learning methodologies.

% of schools using new national

curriculum for Years 1-3. Total: 0% (2013) Total:

Monitoring Study

SBMU (PEO)

Output 1.1: New curriculum implemented in Years 1-3

# of schools receiving new curriculum (for Years 1-3 as rolled out)

Total: 0 (2013)

391 (100%) schools have received:

6 x teacher guides for new Year 1 curriculum.

1x literacy and numeracy kit for use in Year 1-2

1x year 1 reading kit (graded readers, posters and Big Books)

Provincial Training reports

ISU (VITE)

Output 1.2: Training (pre-service and in-service) programs for primary teachers updated to meet quality standards

# of schools with teachers trained in new curriculum (for Years 1-3 as rolled out)

Total: 0% (2013)

A total of 1023 (75%) of primary teachers have attended one or more training sessions for the new curriculum:

Effective Teaching and Learning (700 participants)

Year 1 teacher guide for Mathematics (475 participants)

Language and Communication - Round 1 (1023 participants).

Provincial Training reports

ISU (VITE)

Output 1.3: ECCE strengthened to support early years learning

#Kindergartens implement ECCE policy of school readiness

Goal 2: Improve the equitable access to education for children in Vanuatu

Result Verifiable Indicator

Baseline

Achievement

Data Source

Responsibility

End-of-Program Outcome 2: Children (girls and boys) including those with special needs have access to kindergarten and years 1-3

Enrolment Rate (NER & ASER) for kindergarten and Years 1 - 3 (girls and boys)

Kindy: Year 1-3: Year 1: 37.5% Year 2: 24.2% Year 3: 21.8%

Kindy: Year 1-3: Year 1: 50% Year 2: 40% Year 3: 35%

VEMIS enrolments

(EFA)

PPU (Statistics)

% of children (girls and boys) attending kindergarten and Years 1 - 3 with special needs

Kindy: 2.5% Year 1-3: 5.7% Year 1: Year 2: Year 3:

Kindy: 10% Year 1-3: 10% Year 1: Year 2: Year 3:

VEMIS enrolments

PPU (Statistics)

Intermediate Outcome 2: Increase in number of schools and kindergartens implementing inclusion initiatives to improve access for children to Kindergarten and Years 1-3.

% of children (boys and girls) age 6 entering year 1 who attended kindergarten

K-Year 1: 53% K-Year 1: VMQS Std 2.

SBMU (PEO)

Retention (survival rates) for Year 1 students to Year 4)

Year 4 Total 82.2% (girls 81.1% and boys 83.5%)

VEMIS SBMU

Output 2.1: Community engagement to

support access and participation in early years of schooling.

# of schools that engage with communities to enrol children at the right age into year 1.

Total: 0 (2013)

Total: Needs to be monitored - campaign just launched Case study established in 4 schools


SBMU

# of schools that engage community support for accessible facilities in primary schools

Liaison work

Goal 3: Improve the management of the education system in Vanuatu

End-of-Program Outcome 3: Effective

education service delivery for kindergarten

and years 1-3 at central provincial and school

level.

% of primary schools that can demonstrate improved achievement of Quality Minimum Standards (from 2013)

Total 0% (2013) Total: 20% of schools include standard 13 in SIP (2015)

Intermediate Outcome 3: Provincial officers

providing necessary support to school leaders

% of schools that demonstrate effective management of school grants

Total: 66% (271 of 408) of eligible schools submitted financial returns (tranche 2 2014)

Total: 81% (331 of 408) of eligible schools submitted financial returns for (tranche 1 2016)

School Improvement

Annual Report

(Finance)

SBMU

% schools that demonstrate improved school management

Total: 2% of schools with a SIP (2014)

Total: 25% (2016)

School Improvement

Annual Report (MQS

assessment)

SBMU

Output 3.1: Infrastructure planning to support quality facilities and improved access to primary schooling.

# of primary schools completing an asset survey

Total: 0 (2013)

Total: 341 out of 433 have an asset survey completed. 276 schools have an outline and indicative only capital works plan


SBMU

#of schools with new classrooms built and occupied

Total: 0 (2013)

Total: Construction commenced on 42 new classrooms in 18 schools.


SBMU

# of schools implementing maintenance plans in line with MoET guidelines.

Total: Total: School

Leadership Program

SBMU

Output 3.2 School leadership to support

improved learning in classrooms

# schools with school improvement

plans (SIPS) Total: 2% (2014)

Total: 147 schools in 2 provinces (61 Penama and 86 Malampa) trained in SIP.

103 schools (41 Penama, 63 Malampa) have submitted SIPs to provincial education offices.

2 provincial offices are

implementing monitoring and

support for SIP implementation

# of schools with head teachers trained in new curriculum

Total: 0 (2013)

308 schools have received at least one “leadership for learning “ support visits by ZCAs (principals package for new curriculum).

School number breakdown by province:Samna – 68, Malampa – 70, Penama – 47, Tafea – 47, Shefa – 76

Torba - Visit not conducted

Output 3.3 Provincial management to support service delivery at school level

# of eligible primary schools receive and acquit school grants on time

Total: 68% (271 of 408)

Total: 392 of 408 eligible Primary Schools (97.5%) received primary school grants in 2015 benefitting 43,214 students.


SBMU

# provinces with capacity for devolved responsibilities for the management of teachers and schools.

Total: Provincial:

Total: Provincial:


SBMU

Annex 3

VESP Monthly Progress Report Template

1. Overall status of Annual Implementation Plan (AIP) activities

2. Key outputs

3. Key inputs

Personnel

Activity expenditure

4. Major issues/risks requiring attention

Activity Schedule Budget Issues Overall Risks

1.1 Curriculum Implementation

1.2 Language Policy Implementation

1.3 Training/PD for new curriculum

1.4 VITE Strengthening

1.5 Assessment, reporting & Monitoring

2.1 ECCE Strengthening

3.1 School Leadership Program

4.1 Primary School Construction

4.2 School maintenance plan

5.1 De-concentration of education services

5.2 Evidence-based policy and planning

Annex 4

VESP Six-Monthly Progress Report Template

Executive Summary

A summary of the progress made during the reporting period.

1. Introduction

This section includes a program overview and summary of the impacts of significant events on

the management and operations of the program, as well as any major changes to program

monitoring and evaluation, and impacts on the reporting of results.

2. Overall VESP Progress

This section reports on progress during the reporting period towards the planned outputs and outcomes for the program. Reporting is provided at two levels:

Progress towards achievement of key outputs for the program in terms of reach, coverage and quality

Progress towards the end of program outcomes and an associated update on proposed evaluation studies for the next 12 months.

3. Progress towards end of program outcomes

This section outlines the current thinking for both end of program and intermediate outcomes

and associated indicators as well as an update on relevant baselines.

4. Lesons learned

5. Risk management

This section provides an update in relation to the VESP Risk Management Matrix (RMM) with

discussion of any revised and updated risks along with and other potential risks to successful

program implementation.

6. Technical Assistance

This section includes a summary of technical assistance provided by VESP technical advisers

7. Cross-cutting issues

This section includes: mainstreaming gender, disability and social inclusion and

communication/advocacy.

8. Program Management

This section includes: cumulative expenditure performance, expenditure performance for

the six month period, forecast expenditure for next six months, adequacy of progress

against budget, and fraud detection, investigation and reporting

9. Conclusion

Annex 5: MQS Landscape Review Goal Standards Performance Indicators

Contribution of VESP activities Performance measures/Indicators

AC

CE

SS

1. Every child between 6

and 12 years old is

enrolled in primary school

and completes Year 6

successfully regardless of

gender, ability or

background.

1.No. of children in year 1 at the

age of 6

3.1.2 - SIP training includes raising

awareness of principals and school

committees of need to ensure eligible

children are in school and of policy of

automatic progression

% of children aged 6 in Year 1 MQS Monitoring Study

Schools are aware of MOET enrolment, progression and repetition policies

Schools actively encourage enrolment into Year 1 of all eligible students in their catchment areas

School communities demonstrate improved awareness of the right age to enrol children in Year 1 (ABC-ID is looking after the monitoring of the campaign – you’ll need to talk to them)

3.1.4 Right Age campaign – community

awareness of value of starting at right

age

2.No. of children who complete year

6 at the age of 12

2. All teachers identify

students with special

needs and take

appropriate action to

ensure their presence and

full participation in class

and out-of-school school

activities.

1.No. of students with special needs

are identified

3.1.3 Case Study involving 4 schools

implementing local initiatives to improve

enrolment of children with special

No of children with special needs in Year 1 participating in learning programs. (In case study schools only)

MQS Monitoring

(Sherol will need some help to set this case study up so that there

is a good monitoring framework for it – have the initiatives

improved access and particularly participation of these children)

2.No. of students with special needs

who are present regularly and

participate actively in

class

QU

AL

ITY

3. All teachers must be

certified and receive

regular feedback on their

performance from Zone

Curriculum Advisers and

School Heads.

1.No. of the teachers who are

certified

1.4 Establishment of in-service

pathways for teachers to upgrade

through VITE

No of primary teachers who have received PD for new curriculum

No and % of primary teachers certified

% of courses revised for the new Primary Diploma (pre-service and in-service)

Curriculum Monitoring study

Principals can identify specific changes that have occurred in Year 1 classrooms

2.No. of teachers who participate

actively in on-going professional

development

1.3 PD for teachers in Year 1-3

curriculum

Goal Standards Performance Indicators


3.No. of teachers who receive

constructive feedback on their

performance

1.3 Training of principals and ZCAs in

instructional leadership

Teacher educators and provincial support staff describe increased confidence to support schools implement the new curriculum

Teachers report feeling supported in their efforts to implement the new curriculum

MQS Monitoring Study

Principals are able to articulate their role in improving learning outcomes for students in their schools.

Principals are monitoring Year 1-2 teaching programs to ensure they are in line with the curriculum

Principals feel supported in their role as “leaders for learning” by the provincial support personnel (Eg ZCAs)

Principals can articulate their role in improving learning outcomes for students in their schools.

3.1 Support visits by ZCAs to schools

following training

5.1 Establishment of performance

management system, devolved to

provinces and schools, for teacher

performance management

4 All teachers use various

child-centered

methodologies in their

lessons that are relevant

to the Vanuatu as well as

the local context and

culture

1.No. of teachers who use child-

centered method in teaching

1.1 Implementation of new curriculum

in Year 1-3

Curriculum Monitoring Study

Teachers feel confident to implement the new curriculum

Teachers can demonstrate how they have changed their teaching practice in response to training.

Teachers are using the new materials as intended.

Teachers, principals and parents describe increased student engagement with learning

1.2 Implementation of first language

learning in Year 1-3

1.3 Training in Effective Learning and

Teaching (ELT) and PD for new

curriculum

2.No. of teachers who teach

lessons that are relevant to Vanuatu

as well as the local context

5 All teachers use a variety

of assessment approaches

to monitor and evaluate

1.No. of teachers who use a variety

of methods of assessment

1.5 Assessment toolkit designed to

support teachers to implement class

based assessment in Year 1


Year 1 teachers are using contemporary assessment and reporting practices to monitor student progress towards the curriculum outcomes. 1.1 Teacher Guides include

assessment advice and activities



the academic progress of

students.

2.No. of teachers who monitor and

evaluate the learning progress of

students

1.5 Establishment of literacy and

numeracy standards for Year 4 and

Year 6 (using baseline study)

School Leadership Monitoring Study

Principals use available data on student performance in their improvement planning

1.5 VANSTA for Year 4 and 6 revised

in line with literacy and numeracy

standards

QU

AL

ITY

6 All teachers have a

teacher guide for each

subject as well as teaching

and learning materials and

these educational

materials are used by the

teacher during their

lessons

1.No. of teachers who use current

syllabus and teacher guides in

developing their lesson plans

1.1 Development and distribution of

teacher guides and learning support

materials for Year 1-3

No of primary schools that have received learning materials packages

for Year 1-3 curriculum implementation.


Teachers design teaching and learning programs using the new syllabus and teacher guides

Teachers use Bislama or the vernacular as the language of instruction in Year 1-2

School Leadership monitoring study

Schools have policies and procedures in place to store, maintain and make effective use of the new curriculum materials

2.No. of teachers who create and

use teaching and learning materials

7 All students have a text

book and stationery for

each subject

1.No. of students who have access

to text books for all subjects

2. No. of students who have access

to stationery for all subjects

8 All teachers and

students maintain good

personal hygiene and

mechanism are in place to

allow teachers and

1.No. of students who maintain

good personal hygiene

4.1 Infrastructure standards set

requirements for toilets

4.1 Primary school infrastructure asset

survey collects this data

2.No. of mechanisms are in place to

support teachers and students

medical attention



students to receive

medical attention

3.No. of working toilets available to

students

4.1 18 schools with new classrooms on Tanna to have adequate WASH facilities

QU

AL

ITY

9 School buildings meet

the MoE infrastructure

standards and School

Heads conducts monthly

safety audits of buildings

4.1.No. of school buildings that

meet MoET facilities infrastructure

guidelines

4.1 MoET Facilities minimum

infrastructure standards established

and asset survey of all primary schools

against the standards

4.1 42 new classrooms on Tanna that

meet the standard

No of schools meeting minimum infrastructure standards

No of schools with completed asset survey

No of schools with an indicative capital works plan

Number of new classrooms built to standard

2.No. of annual safety audits of

school buildings by school head

3.No. of school buildings that

received maintenance in last 12

months

3.1 SIP training on management of

resources and facilities

10 All teachers and

students have access to at

least two litres of potable

water every day

1.Total No. of litres of potable water

available to teachers and students

Not relevant to VESP

11 School policies have

been developed and are

implemented to protect

school staff and students

1.No. of school policies developed

to protect staff and students


2.No. of school policies

implemented to protect staff and

students


12 All teachers ensure that

both boys and girls are

able to equally participate

in school activities

1.No. of activities in which boys and

girls participate equally

Not relevant to VESP N/A



13 School Head and

Teachers encourage all

Community Members to

actively participate in

school events and

teaching activities

1.No. of events and teaching

activities in which community

members participated

3.1 SIP training and follow up MQS Monitoring

Schools are actively engaging their community in their plans for improving the quality of the education being delivered.

Man

agem

ent

14 The school has an

active and gender-

balanced School

Committee which has

developed a School

Improvement Plan which is

communicated to all the

local stakeholders in

education

1.No. of males and females in the

School Committee

3.1 MQS baseline

Number of primary schools implementing a SIP

Number of women on School Committees MQS Monitoring

The Provincial Education Office is using the results of the MQS baseline and schools SIPs to prioritise support for schools and develop provincial work plans.

The provincial education office provides feedback to schools on their SIPs with follow up support to those who need it.

The School Committee actively leads the SIP development, monitoring and reporting process

The school’s SIP is linked to the MQS and prioritises use of the school grant.

School improvement Plan (including use of the school grant) is communicated to local stakeholders

2.No.of School Improvement Plan

activities identified in the Annual

Work Plan

3.1 SIP training

5.1 Strengthening of provincial support

services to schools

3.No. of times the SIP is

communicated to local stakeholders

including School Grant

expenditure

15 The School Head

manages the daily school

administration by filing

and updating school

records

1.No. of required administration

records regularly updated

5.1 Capacity development and support

for principals from province level to

improve administration

5.1 Improving financial management

skills through school grant processes

5.2 Open VEMIS – tool to improve use

of data about the school, teachers and

students for planning, monitoring and

reporting.

Number of schools submitting annual enrolment returns on time

Number of schools acquitting school grants on time

Number of schools receiving school grants on time MQS Monitoring

Schools are using Open VEMIS to manage data

Schools maintain an accurate enrolment register

Schools maintain an accurate record of attendance by students

Schools maintain an accurate record of attendance by teachers

Schools acquit school grants on time

Schools maintain an asset register

vanuatu education support program (vesp) monitoring and ... 1 monitoring and...this document sets...

Documents