conducting and using impact evaluations · 10/03/2015 · • designing the intervention •...
TRANSCRIPT
Conducting and Using Impact Evaluations: A large-scale Experiment in Ghana
Annie Duflo & Jessica Kiessel March 10, 2015
• The problem and existing lessons • Designing the intervention • Designing the evaluation • Analyzing and understanding results • Conclusion
THE TEACHER COMMUNITY ASSISTANT INITIATIVE
LARGE LEARNING GAPS English Test Scores
24
57
98
11 20 31
0
20
40
60
80
100
P1 P2 P3Grade Level
The context: Heterogeneous levels
HETEROGENEOUS LEVELS
WHAT HAVE WE LEARNED?
Kenya: Adding extra teachers most
effective when class split by ability
India: Pull-out remedial classes for
low-performing students by
community tutors (Balsakhi)
India: Community-based volunteers running remedial classes in
communities (Read India Program)
FOCUSING INSTRUCTION AT THE R IGHT LEVEL IS KEY
• The problem and existing lessons • Designing the intervention • Designing the evaluation • Analyzing and understanding results • Conclusion
THE TEACHER COMMUNITY ASSISTANT INITIATIVE
FROM INDIA TO GHANA
• Key concept
• Context
• Political acceptance
• Sustainability
PROGRAM DESIGN AND SUPPORT
G OVERNMENT’ S QUESTIONS ABOUT MOST EFFECTIVE DESIGN
• The problem and existing lessons • Designing the intervention • Designing the evaluation • Analyzing and understanding results • Conclusion
THE TEACHER COMMUNITY ASSISTANT INITIATIVE
C Assistant-led
remedial classes after
school
D Extra assistant: Randomly split revision group
E Teacher-led
Targeted instruction
Randomly allocate to :
A
Control group
B Assistant-led
remedial classes during
school
500 schools across all 10 regions, 42 Districts
RANDOMIZED EVALUATION DESIGN
RANDOMIZED EVALUATION DESIGN
N ATIONALLY REPRESENTATIVE SAMPLE OF 42 ,000 GRADE 1 -3 PUPILS
Targeted Instruction
Extra materials
Remedial pedagogy & materials
Extra Assistant
Smaller class size
Extra school hours
1. Rem Ed during school hours
X
X
X
X
X
2. Rem Ed after schools
X
X
X
X
X
3. Extra Assistant X
X
X
4.Targeted instruction by teachers
X
X
X
5. Control
INTERVENTION KEY FEATURES
THE FRAMEWORK
2.
Testing Proof of concept
3.
Refining the concept’s
applications across contexts
4.
Testing Scalability
EVALUATION OBJECTIVES
Mobilizing and supporting decision-makers to use evidence
Randomized Controlled Trials
1.
Identifying problems and
potential solutions
• Final outcomes (oral and
written testing tools) • Intermediate variables
(midline observation surveys) • Supporting characteristics
(baseline student, school and community surveys)
13
DATA COLLECTION DATA COLLECTION
• The problem and existing lessons • Designing the intervention • Designing the evaluation • Analyzing and understanding results • Conclusion
THE TEACHER COMMUNITY ASSISTANT INITIATIVE
0.083* 0.075+ 0.047 0.036
0
5
10
15
20
25
30
35
40
45
Control EL2 Treat1 Treat2 Treat3 Treat4
Overall Test Scores (SD)
POSITIVE AND LASTING IMPACTS
0.14** 0.13** 0.11** 0.08*
0
10
20
30
40
50
60
Control Treat1 Treat2 Treat3 Treat4
Overall Test Scores - P3 & P4 (Standard Deviations)
Control Remedial Classes During School
Remedial Classes After
School
Extra Assistants
Teacher-led Targeted
Instruction
POSITIVE AND LASTING IMPACTS
0.18** 0.17* 0.06
0.16**
0
5
10
15
20
25
30
Control EL2 Treat1 Treat2 Treat3 Treat4
Local Lang Reading Skills (P3-4) (Standard Deviations)
Control Remedial Classes During School
Remedial Classes
After School
Extra Assistants
Teacher-led Targeted
Instruction
HIGHEST FOR TARGETED SECTIONS
0%
10%
20%
30%
40%
50%
60%
70%
80%
TCA present Teacherpresent
TCA teachingclass
Teacherteaching class
Attendance and Time on Task: TCAs and Teachers
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
ML 1 ML 2 ML 3 ML 4 ML 5 ML 6
Attendance of TCAs over time
During school
After school
DESPITE IMPLEMENTATION ISSUES
Effects on similar grades after similar amount of time in EL1 and EL2
T1 T2 T3 T4
P2 at EL1 (13 weeks) 0.084 0.091 0.016 0.035
P2 at EL2 (38 weeks) 0.041 0.068 0.053 0.038
IMPLEMENTATION OVER TIME
More than 3
times as effective
No effect
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
111113115117122124131133141143145151161163165172182191193201203
TCA Attendance Teacher Attendance
TCA vs. Teacher Attendance by District
LARGE REGIONAL VARIATIONS LINKED TO IMPLEMENTATION VARIATIONS
All English Loc Lang Math
During school 0.30* 0.26* 0.43* 0.34*
After School 0.21* 0.17* 0.26* 0.17*
All English Loc Lang Math
During school 0.12** 0.13** 0.11 0.1+
After School 0.14** 0.16** 0.14+ 0.1*
• Mostly depends on the school environment
- Likelihood of class taking place?
- Multigrade teaching • After school works better
for girls than boys - From 0.05 to 0.13 SD - Significant for girls in P1-2
Schools with multigrade teaching
Schools with no unused classroom at baseline
DURING OF AFTER SCHOOL?
WHAT’ S NEXT?
Nov 2009
• Shared results
• Funding • Planning • MoU
May 2011- Sept 2013
Evaluation
2014
• Analyze • Share • Next
steps
2014-2015
• Follow up research
• Policy outreach
2005-2011
Results from India and Kenya
2014-2015
Lessons integrated into system