Presented by Dr Dragana Sretenov, Learning and Impact Assessment Adviser, Save the Children UK

April 2009

Education in Conflict Affected Fragile States:

Global Evaluation 2008-2010

Nepal

Presentation Outline

1. Global Terms of Reference

2. Methodology3. Findings in 4 Countries4. Data Use Plan

Note: Dr.Ruth Naylor is the consultant - Global Researcher leading this research.

Angola

Global Terms of ReferenceResearch Questions & Country Focus

1)How have Save the Children’s project level interventions contributed to quality primary education for children affected by conflict?

2)Which project level interventions have had what impact on the education quality of children affected by conflict?

Angola: Teacher Professionalism

Southern Sudan: Teacher Training

Afghanistan: Child & Parent Participation

Nepal: Schools as Zones of Peace

Global Terms of Reference

Objectives– Contribute to technical knowledge about what works and why– Contribute to internal and external documentation used in

advocacy, communications, and fundraising– Contribute lessons learned into the internal organizational and

operational decision making process

Audience– Donors*– External Communities of Practice*– Governments & International Bodies– Headquarters policy advisors and country office programmers

Timing– 2008 Midterm (formative) 2010 Final (summative)

Global Terms of ReferenceIn Country Methodology

• Design: in-country participatory evaluation design and implementation. Country specific

• Key Informants: students, child clubs, teachers, PTAs, partner organizations, parents

• Data Collection: focus groups, lesson observations, interviews, learning assessment, secondary data

• Sample: 6 schools per district, 3-5 districts per country

Global Terms of ReferenceParticipatory Design Agenda

1. Identify Research Team2. Identify Advisory Committee3. Narrow scope: effective, unique,

innovative4. Select intervention package5. Map intervention program theory6. Identify indicators for program theoretical

framework7. Identify informants8. Develop tools9. Pilot & Capacity Building

Global Terms of ReferenceTools and Sample Data Collection Plan (Angola)

Name of tool

description Respondent group Zaire Uige Kwanza Sul Total

School data collection

Objective data about the school based on observation and school records

Headmaster can assist,/ check documents/ school observation

7 schools

5 schools

6 ALP 18

Lesson observation

Observation checklist One for a lower grade (1-3) lesson one for a higher grade (4-6)

14 10 6 30

Learning Assessment

Oral questions to one child at a time

3 girls and 3 boys chosen at random from grade 3 per school

42 30 36 102

Student exercise

Group discussions on what students like and dislike about their teachers

4 girls and 4 boys from oldest class available

2 2 2 6

Director interview

Structured interview School director 7 5 Not applicable 12

Teacher interview

Structured section followed by in depth questions

2 teachers per school and one in depth

14 10 6-12 36

Supervisor interview

Structured interview supervisors Approx4

Approx 4

3 8

CPEE focus group

Focus group/ interview Focus group during longer school visits

3 2 Not applicable 5

CCPC focus group

Focus group/ interview Focus group during longer school visits

3 1 (Not applicable?)

Not applicable 4

Angola

Southern Sudan

Afghanistan

NepalQuestions?

Global Terms of Reference

Nepal: Schools as Zones of Peace

Program Theory: Schools as Zones of Peace

Child Club

School Management Committee

Village Child Protection Committee

Save the Children Partners

• Bring together community members and political groups

• Develop, agree on and display codes of conduct

Items on codes of conduct: no armed activities in school; no weapons in the school; no use of children in political activities; no political meetings or graffiti in school; no arrest, abduction and torture of children and teachers; no school closure for meetings; no corporal punishment; no cattle grazing; no alcohol or smoking in school.

Nepal: Successes of SZOP

• Increased sense of security in schools “Though some fear is still there with ongoing conflict, we feel safer than earlier and also do not feel fear while sending our children to school.” Parent, Mahottari

• In 2007 project schools on average were open 12 days more than comparison schools (of 220 days)

• Informants at over half of project schools reported improved student and teacher attendance

• Evidence of beating was observed in one project school only

“Instead of beating children, teachers now support and counsel them.” Parent, Palpa

• 11 out of 16 project schools had child representatives in the SMC

• 12 out of 16 project schools had developed thier own codes of conduct

Nepal: Challenges

• Monitoring systems for SZOP not well established

• Difficult to establish SZOP in contexts with limited experience of community participation in school development

• Ensuring that SZOP is accepted and respected by political parties and armed groups where conflict is ongoing

Methodological Challenges Nepal

• Partners implementing at the school level need to be involved in the design workshop

• Sampling plans must adapt to local conditions (weather, migration, and conflict)

• Time for staff capacity building should be built into piloting.

• More emphasis needed during design on participatory research methods for working with children

• School level data from districts to be collected prior to researcher arrival in country and made available at workshop

• Checklist sent to country offices giving requirements in terms of what needs to be done before, during and after the visit, who should be involved and for how long

Afghanistan: Child and Adult Participation

Afghanistan: Child and Adult Participation for Quality Education

Children’s participation in school

• Student Councils• Student representation in PTA• Student led education surveyChildren’s participation in

learning• Child-to-child health education• Training teachers in participatory

methodology• Resource centres, mobile

librariesParents’ participation in

school• PTA training and support

Afghanistan: Successes

Student group and PTA success stories• Follow up on absent students• Recruitment of new teachers• Construction/maintenance of

classrooms, boundary walls, latrines and school water supplies

• Reduction of corporal punishment

• Children learn about health through teaching it

Training provided by Save the Children was more effective when it was followed up by support to carry out a concrete action

Afghanistan: Challenges

•In two of the mixed project schools visited there were no females in the Student Councils

•Only five of the 12 project schools visited had mothers in their PTAs, and three of these were girls’ schools

•12% of teachers in formal schools are female

Participation of girls and women

PTA training: spot the mothers

Methodological Challenges Afghanistan• Translation of reading assessment to local

language/dialect is critical in cases where instruction is done in local language (e.g., Paghman ALC). Also, be careful about key words on assessment.

• Difficult to record all information. Need to have group discussion immediately following visit.

• Random selection of children for assessment can be difficult to orchestrate.

• Security situation may cause last minute challenges / changes in plans

• Full course of data collection is very time consuming and can be logistically challenging, with short hours and unpredictable constraints.

• Very difficult to avoid the bias of the most remote schools being underrepresented for practical reasons.

Global Evaluation: Angola

1)How have Save the Children’s project level interventions contributed to quality primary education for children affected by conflict?

2)Which project level interventions have had what impact on the education quality of children affected by conflict?

Angola: Teacher Professionalism

Southern Sudan: Teacher Training

Afghanistan: Child & Parent Participation

Nepal: Schools as Zones of Peace

Angola: Supporting Teacher Professionalism

Putting our knowledge to

use

Angola: Teacher Professionalism

School directors

Teachers

Supervisors

Other teachers

Parents Students

Save the Children

Provincial Department of Education (DPE) – main partner

Angola: Successes

• 93% of the teachers in schools visited had some training.

• Most teachers had been observed by supervisors and school directors

• Half of teachers observed had a lesson plan

• Respondents reported Improved teacher conduct (reduction in drunkenness and exchanging grades for money)

Angola: Challenges

• Only ¼ of grade 3 students tested were able to read well enough for simple comprehension

• There was no significant difference found between the teaching of trained teachers and that of untrained teachers

• 7 cases of teacher misconduct identified (in 14 schools + 6 ALCs)

• 12 out of 16 student focus groups reported physical and humiliating punishment in schools

Angola: Student Learning Findings

• Only ¼ of grade 3 students tested were able to read well enough for simple comprehension

• Over 50% of students tested were unable to read a single word

• Most children tested could solve simple addition and subtraction problems, including word problems

Angola: Teaching and Learning

THANK YOU!

Methodological Challenges Angola

• Context of very rapid economic and social change including major changes in education (new school buildings, distribution of books, mass teacher training programmes).

• School level data on dropout and changes in enrolment is highly influenced by migration (urbanisation) and construction of new schools, and hence does not reflect quality (or access gains).

• Evaluators measurement of teaching quality is restricted by low exposure to good teaching.

Southern Sudan: Teacher Training

Southern Sudan: Teacher Training

• Phase training• In-service training• Women into

Teaching • Intensive English

Course• Accelerated

Learning Programme teacher training

Southern Sudan: Successes

According to respondents, teachers trained by Save the Children:

•know how to structure lessons •adapt lessons to the learner•have increased confidence •have improved English skills•use less corporal punishment

Teachers with training from Save the Children scored significantly higher in lesson observations than those without.

Southern Sudan: Challenges

• No formal follow up of the training

• 60% of enrolled students absent from lessons observed.

• Only 10% of teachers in the evaluation counties were female

• Only 27% of students in the evaluation schools were female.

• Dropout rate for girls was 16%, and for boys 5%

Southern Sudan: Learning findings

• 19% of grade 3 students tested could not read a single word

• Boys score higher then girls in reading, comprehension and mathematics

• Multivariate analyses of the learning assessment results indicated no significant difference between the reading and comprehension scores of students from project and comparison schools

• In mathematics students from comparison schools scored significantly less

• Teacher training level did not appear to make a difference in the learning outcomes

Methodological Challenges Southern Sudan• There is a need for a higher level of involvement from the country

office team during the evaluation design training.• Due to geographical and funding reasons the design workshop had to

be separated from the training workshop for data collection, and the workshops were organised in 2 distant part of the country, which limited the possibility of the head office staff to take part in the data collection workshop. 

• The difficulty in undertaking the education research in Southern Sudan is also related to inaccessibility of schools in some parts of the country during the rainy session when schools are fully running. Project schools in each county were to be selected at random but flooding in two of the counties restricted the number of accessible project schools, so there was a bias towards the more accessible schools. 

• Comparison schools were chosen to match the project schools in terms of their distance from the nearest town as there was little other data available to base the selection on. 

• The enumerators had limited knowledge of teaching processes and the classroom observation tool had to be modified and simplified. The originally developed classroom observation tool was used by the international and local researcher only.

• There is a need to enable greater child participation in the data review process, which proved to be very productive in other countries.

Moving towards data use

• From our evaluation we know:In Angola, around half of the children tested

(project and comparison) could not read a word in a simple Portuguese text

In Nepal, a quarter of the children tested could not read a single word in a simple Nepali text

In Southern Sudan, of children tested, girls’ reading accuracy was 30% lower than boys (43% compared to 73%)

In Afghanistan, 40% of children tested in SC supported schools could read and comprehend a simple text, whereas 1/32 students tested in comparison schools couldSo what do we do now?

Evaluation Findings Data Use Plan

• Global Evaluation Executive Summary (4pgs)• Global Evaluation Midterm Report (20 pgs)• External sharing of lessons learned,

e.g. GMR, AEA, Education in Emergency Conference

• Follow up action plan at country office & headquarters

• Linking with other evaluations on related topics

Education in Conflict Affected Fragile States:

Global Evaluation 2008-2010

Nepal

Questions?

Education in CAFS: Global Evaluation 2008-2010

1)How have Save the Children’s project level interventions contributed to quality primary education for children affected by conflict?

2)Which project level interventions have had what impact on the education quality of children affected by conflict?

Angola

Southern Sudan

Afghanistan

NepalFor more information:

Angola – SC Denmark Gretehe Markuson GM@redbarnet.dk

Afganistán – SC US Rita Oswald Cristiano rchristiano@savethechildren.org

Nepal – SC Norway Inguun Nakkim

ingunn.nakkim@reddbarna.no

Southern Sudan – SC UK Dragana Sretenov d.sretenov@savethechildren.uk

Cynthia Koons, M&E Manager, Inernational Save the Children Alliance cynthia@save-children-alliance.org