i ssues in the evaluation of a program to promote educational achievements of ethiopian-israeli high...
TRANSCRIPT
ISSUES IN THE EVALUATION OF A PROGRAM TO PROMOTE EDUCATIONAL ACHIEVEMENTS OF ETHIOPIAN-ISRAELI
HIGH SCHOOL STUDENTS
"Improving Education through Accountability and Evaluation: Lessons from Around the World”
Rome, Italy, October 2012
DALIA BEN-RABI, VIACHESLAV KONSTANTINOV, RUTH BARUJ-KOVARSKY, MIRIAM COHEN-NAVOT
Engelberg Center for Children and YouthEngelberg Center for Children and YouthMyers-JDC-Brookdale InstituteMyers-JDC-Brookdale Institute
Since 1984 Most parents are illiterate In 2012 – most children were born in IsraelIn spite of efforts of government, local
authorities, and voluntary organizations large gaps in many areas, including academic achievements
2
IMMIGRATION FROM ETHIOPIA TO ISRAEL
3
ACHIEVEMENTS ON NATIONAL EXAMS OF ETHIOPIAN-ISRAELI 8TH GRADERS, COMPARED TO THOSE OF ALL STUDENTS IN HEBREW SPEAKING SCHOOLS IN ISRAEL, 2004/5 (AVERAGE SCORE, ON A SCALE OF 0-100)
4
ACHIEVEMENTS ON NATIONAL MATRICULATION EXAMS OF ETHIOPIAN-ISRAELI 12TH GRADERS, COMPARED TO THOSE OF ALL STUDENTS IN HEBREW SPEAKING SCHOOLS IN ISRAEL, 2004/5 (%)
Was first implemented in 2006Aimed at Ethiopian-Israeli students in middle
schools and high schoolsPlaces special emphasis on matriculation exams Instruction in small groups for about four hours
weekly, in different school subjectsSocial activities and personal coaching At its peak in 2008 reached over a third of all
Ethiopian-Israeli students in grades 7-12 nationwide
5
THE PROGRAM
Providing regular feedback each year to support ongoing program improvement: The implementation of the program
Satisfaction of the school principals and
the participating students
The impact of the program on the students’ scholastic achievements
6
EVALUATION
Eligibility for partial matriculation certificates (enables admission to advanced professional courses)
Eligibility for matriculation certificates (a prerequisite for many jobs and some colleges)
Eligibility for matriculation certificates with a high-level of English (a prerequisite for universities)
7
MEASURES OF ACHIEVEMENT
Are the achievements of program participants better than those of non-participants?
Is the difference in achievements growing over years, as the students are exposed to the program more years before taking the exams?
Are the gaps between Ethiopian and non-Ethiopian Israeli students narrowed?
8
QUESTIONS OF IMPACT: CAN WE ATTRIBUTE THE DIFFERENCES TO THE PROGRAM?
10
MATRICULATION ACHIEVEMENTS OF 12TH GRADERS, 2010 (%)
Partial matriculation
matriculation
University matriculation
11
Boys
Girls
GIRLS DO MUCH BETTER THAN BOYS: % OF BOYS AND GIRLS WITH MATRICULATION CERTIFICATES, 2010
12
GIRLS DO MUCH BETTER THAN BOYS: % OF BOYS AND GIRLS WITH MATRICULATION CERTIFICATES THAT MEET UNIVERSITY ADMISSION REQUIREMENTS, 2010
Boys
Girls
Schools and students were not randomly assigned to the program
Schools in the program are stronger than the average, and weakest students are not included
Schools came in and out of the program over the years
14
THE PROBLEM: NO CONTROL GROUP
Comparison groups:
1.Ethiopian-Israeli students from similar non-participating schools and with similar socio-demographic characteristics:
Student characteristics: country of birth (Israel/Ethiopia), genderSchool characteristics:
State or State Religious school The percentage of students eligible for matriculation certificates among
non-Ethiopian school students, in the year of the analysis The percentage of students eligible for matriculation certificates among
Ethiopian-Israeli school students in 2005, prior to the implementation of the program.
2. Non-Ethiopian Israeli students, from participating schools
15
THE SOLUTION: MATCHING STUDENTS BASED ON ADMINISTRATIVE DATA
As students have had the opportunity for more years of exposure to the program:
Differences in achievements between participants and non-participants should grow
Gaps between participants and non-Ethiopian students should be narrowed
The comparison is of the added value of the program as opposed to whatever form of other assistance may exist in program or comparison schools
16
ANALYSIS 1: CHANGES OVER TIME IN 28 SCHOOLS THAT PARTICIPATED IN THE PROGRAM FOR AT LEST 3 YEARS
17
ACHIEVEMENTS OF PARTICIPATING STUDENTS, ALL ETHIOPIAN-ISRAELI STUDENTS IN PARTICIPATING SCHOOLS AND STUDENTS IN COMPARISON GROUP, BY YEARS OF EXPOSURE TO THE PROGRAM (% OF STUDENTS ELIGIBLE FOR MATRICULATION CERTIFICATES)
Impact (difference between program participants and comparison group, divided by % in comparison group)
27%*
Years in the Program (28 schools) *p<0.01
*49% not significant
18
ACHIEVEMENTS OF PARTICIPATING STUDENTS, ALL ETHIOPIAN-ISRAELI STUDENTS IN PARTICIPATING SCHOOLS AND STUDENTS IN COMPARISON GROUP, BY YEARS OF EXPOSURE TO THE PROGRAM (% OF STUDENTS ELIGIBLE FOR MATRICULATION
CERTIFICATES MEETING UNIVERSITY ADMISSION REQUIREMENTS)
20%*
Years in the Program (28 schools) *p<0.05 **p<0.01
**63%
Impact (difference between program participants and comparison group, divided by % in comparison group)
not significant
19
THE ACHIEVEMENTS OF ETHIOPIAN AND THE NON-ETHIOPIAN ISRAELI STUDENTS IN THE SAME SCHOOLS, BY YEAR OF EXPOSURE TO THE PROGRAM (% OF STUDENTS ELIGIBLE FOR MATRICULATION CERTIFICATES)
Years in the Program
20
THE ACHIEVEMENTS OF ETHIOPIAN AND THE NON-ETHIOPIAN ISRAELI STUDENTS IN THE SAME SCHOOLS, BY YEAR OF EXPOSURE TO THE PROGRAM (% OF STUDENTS ELIGIBLE FOR MATRICULATION CERTIFICATES MEETING UNIVERSITY ADMISSION
REQUIREMENTS)
Years in the Program
21
IMPACT OF THE PROGRAM ON THE ACHIEVEMENTS OF STUDENTS WHO PARTICIPATED IN 2008-2010, TAKING ACCOUNT OF THEIR ACHIEVEMENTS IN EIGHTH GRADE
22
MATRICULATION EXAM ACHIEVEMENTS OF PARTICIPATING STUDENTS AND STUDENTS IN THE COMPARISON GROUP WITH THE LOWEST PREVIOUS ACHIEVEMENTS, AND THE IMPACT OF THE PROGRAM (2008-10, IN PERCENTAGES)
*p<0.05 **p<0.01 282 participating students
23**p<0.01 292 participating students
MATRICULATION EXAM ACHIEVEMENTS OF PARTICIPATING STUDENTS AND STUDENTS IN THE COMPARISON GROUP WITH MEDIUM PREVIOUS ACHIEVEMENTS, AND THE IMPACT OF THE PROGRAM (2008-10, IN PERCENTAGES)
24*p<0.05 167 participating students
MATRICULATION EXAM ACHIEVEMENTS OF PARTICIPATING STUDENTS AND STUDENTS IN THE COMPARISON GROUP WITH THE HIGHEST PREVIOUS ACHIEVEMENTS (2008-10), AND THE IMPACT OF THE PROGRAM (2008-10, IN PERCENTAGES)
Estimate the impact of a comprehensive scholastic assistance program for Ethiopian-Israeli high school students, while trying to overcome the obstacles resulting from the lack of random assignment of schools and students to the program by creating a comparison group of matched students, based on administrative data.
25
THE STUDY AIMED TO
Better achievements compared with non-participating students from similar schools and with similar socio-demographic characteristics
The impact of the program was also reflected in the narrowing of the gap between Ethiopian-Israeli and non-Ethiopian students
26
THE STUDY SHOWED A SIGNIFICANT PROGRAM IMPACT ON THE MATRICULATION RESULTS OF PARTICIPATING STUDENTS:
The impact has increased as students have had the opportunity for more years of exposure to the program
The most significant impact was on students with the lowest previous achievements
Nonetheless, considerable gaps still remain, among boys, and especially as regards English exams. This poses a barrier to obtaining a matriculation certificate that meets university admission requirements.
27
Cost-effectiveness – all students or weakest students?
Boys and girls – do they require different forms of assistance?
English as a third language The continuum between elementary school
assistance and high-school assistance What are realistic expectations for attainable student
achievements, given both the large gaps between them and other students and the scope of assistance that the program provides? 28
KEY ISSUES: