photo credit: varkey foundation effectiveness of ... of interactive distance instruction: ... , each...
TRANSCRIPT
![Page 1: Photo Credit: Varkey Foundation Effectiveness of ... of Interactive Distance Instruction: ... , each broadcasting to 12 classes at a time ... Early Grade Mathematics Assessment (EGMA)](https://reader031.vdocuments.us/reader031/viewer/2022030510/5abae5ee7f8b9a8f058bdfd3/html5/thumbnails/1.jpg)
EffectivenessofInteractiveDistanceInstruction:ExperimentalEvidencefromGhanaianPrimarySchools
JAM I E J O HN S TON – S TAN FO RD UN I V E R S I T Y
CHR I S TO PH E R K SO L L – MATH EMAT I C A P O L I C Y R E S E A RCH
GhanaEducationEvidenceSummit2017
PhotoCredit:VarkeyFoundation
![Page 2: Photo Credit: Varkey Foundation Effectiveness of ... of Interactive Distance Instruction: ... , each broadcasting to 12 classes at a time ... Early Grade Mathematics Assessment (EGMA)](https://reader031.vdocuments.us/reader031/viewer/2022030510/5abae5ee7f8b9a8f058bdfd3/html5/thumbnails/2.jpg)
Howcanwedeliverhigh-qualityteachingtoruralstudents?
• Studentsinruralareassignificantlyunderperform(OECD2013,GES2012,WorldBank2012)
• Inremoteruralareas,shortageoftrainedprimaryteachers(WorldBank2012)
• Teachingqualitymattersforstudentlearning(Glewwe etal.2011,McEwan2015)
• Structuredpedagogyinterventionsappeartohavelargest,mostconsistentpositiveeffectsonlearning(Ganimian &Murnane 2016,Snilstveitetal2015,Kremeretal.,2013)
• Littleknownabouteffectivenessofinteractivedistanceinstructionmodels
![Page 3: Photo Credit: Varkey Foundation Effectiveness of ... of Interactive Distance Instruction: ... , each broadcasting to 12 classes at a time ... Early Grade Mathematics Assessment (EGMA)](https://reader031.vdocuments.us/reader031/viewer/2022030510/5abae5ee7f8b9a8f058bdfd3/html5/thumbnails/3.jpg)
VideoCredit:VarkeyFoundation
![Page 4: Photo Credit: Varkey Foundation Effectiveness of ... of Interactive Distance Instruction: ... , each broadcasting to 12 classes at a time ... Early Grade Mathematics Assessment (EGMA)](https://reader031.vdocuments.us/reader031/viewer/2022030510/5abae5ee7f8b9a8f058bdfd3/html5/thumbnails/4.jpg)
MGCubedSatelliteClassProgram• Varkey FoundationMGCubedProgram
• Equipped72schoolswithtechnologypackage
• Broadcastslivelessonsdailytoremoteclassrooms§ 6trainedstudioteachers,eachbroadcastingto12classesatatime§ 1hourmathand1hourEnglish,5daysaweek
• In-personteacherstrainedas“facilitators”
• Targetsmostmarginalizedstudentswithinschools§ Grades2-5§ Girlsandboys§ Identifiedaccordingtocriteria:distancetoschool,numberofsiblings,truancy,age
PhotoCredit:VarkeyFoundation
![Page 5: Photo Credit: Varkey Foundation Effectiveness of ... of Interactive Distance Instruction: ... , each broadcasting to 12 classes at a time ... Early Grade Mathematics Assessment (EGMA)](https://reader031.vdocuments.us/reader031/viewer/2022030510/5abae5ee7f8b9a8f058bdfd3/html5/thumbnails/5.jpg)
Girls’AfterSchoolProgram• Providesafter-schoolsessionstoin-schoolgirlsandout-of-schoolgirlsincommunity
(whohaveleftschool)
• 1 hourperday,4daysaweek
• Engagegirlsintopicsincluding§ Earlypregnancy§ Earlymarriage§ Girls’rights§ Financialliteracy
• Providesaccesstoadultfemalerolemodels
• Boys’monthlyafter-schoolprogramstartedinresponsetodemandforsimilarprogramPhotoCredit:VarkeyFoundation
![Page 6: Photo Credit: Varkey Foundation Effectiveness of ... of Interactive Distance Instruction: ... , each broadcasting to 12 classes at a time ... Early Grade Mathematics Assessment (EGMA)](https://reader031.vdocuments.us/reader031/viewer/2022030510/5abae5ee7f8b9a8f058bdfd3/html5/thumbnails/6.jpg)
TheoryofChangeImprovequantityandqualityofprimaryschooleducation
Increaseinstructionaltime-on-task:§ Reduceabsenteeism§ Increaseclassroomtime-on-task§ “Double”instructorpresence
Improveteachingpedagogy:§ Leveragestudioteachercapacity§ Improvingfacilitatorcapacity
Improvecurricularcontent:§ Moreengaging§ Moreinteractivity
In-Schoo
l
Gainsinnumeracyandliteracy
ResearchQuestions:
1) Istheprogrameffectiveatincreasingskills?
2) Whatarepotentialmechanismsofchange?
![Page 7: Photo Credit: Varkey Foundation Effectiveness of ... of Interactive Distance Instruction: ... , each broadcasting to 12 classes at a time ... Early Grade Mathematics Assessment (EGMA)](https://reader031.vdocuments.us/reader031/viewer/2022030510/5abae5ee7f8b9a8f058bdfd3/html5/thumbnails/7.jpg)
EvaluationApproach
cohortreceivesintervention
ClusterRandomizedControlledTrial(RCT)§ 70treatmentand77controlschools,assignedatbaseline§ Followedcohortofstudents(grades2-4atbaseline)fortwofullyears
Interventionbegins Year1 Year2
Jan2014baseline
Jun-Jul2014monitoring1
Year0
Feb-Mar2015monitoring2
Jun-Jul2016endline
Jan-Feb2017postendline
Jun-Jul2015midline
Mar-Apr2016monitoring3
Cohort:Grade2 Grade3 Grade4
Cohort:Grade3 Grade4 Grade5
Cohort:Grade4 Grade5
Cohort:Grade5(excludedfromRCT)
datacollections
2yearimpact
Grade6
1yearimpact
cohortnotreceivingintervention
![Page 8: Photo Credit: Varkey Foundation Effectiveness of ... of Interactive Distance Instruction: ... , each broadcasting to 12 classes at a time ... Early Grade Mathematics Assessment (EGMA)](https://reader031.vdocuments.us/reader031/viewer/2022030510/5abae5ee7f8b9a8f058bdfd3/html5/thumbnails/8.jpg)
Data1) Skillsassessments
§ Baseline,midline,andendline§ Literacy:EarlyGradeReadingAssessment(EGRA)§ Numeracy:EarlyGradeMathematicsAssessment(EGMA)
2) Studentandteacherdata§ Studentsurveysatbaseline,midlineandendline§ TeachersurveyinYear2§ Attendancemeasures(recordsandspotchecks)inYears1and2§ Qualitativeinterviewsandfocusgroupdiscussionswithstudents,teachersandadministrators
3) Classroomobservations§ Year2§ AdaptedStallingsObservationProtocol§ Snapshotsyieldingproportionofclasstimeon(1)activeinstruction,(2)passiveinstruction,(3)classroom
management,and(4)offtaskactivities
![Page 9: Photo Credit: Varkey Foundation Effectiveness of ... of Interactive Distance Instruction: ... , each broadcasting to 12 classes at a time ... Early Grade Mathematics Assessment (EGMA)](https://reader031.vdocuments.us/reader031/viewer/2022030510/5abae5ee7f8b9a8f058bdfd3/html5/thumbnails/9.jpg)
MeasuringLearning:EGRA&EGMA• EarlyGradeReadingAssessment(EGRA)
§ Oraltestmeasuringsubtasks:§ Lettersperminute§ Wordsperminute(WPM)<- focalmeasure§ Inventedwordsperminute§ Listeningcomprehension§ Oralvocabulary§ Readingcomprehension
§ BaselineWPM:11.5§ Baselinestandarddeviation:21.2
• EarlyGradeMathematicsAssessment(EGMA)§ Oraltest,measuringsubtasks:
1. Numberidentification(20%)2. Quantitydiscrimination(20%)3. Missingnumberpatterns(20%)4. Addition/subtraction/wordproblems(40%)
§ Baselinemeanscore:46.0percentcorrect§ Baselinestandarddeviation:20.1
focalmeasure:combinedtotal
SOURCE:RTIInternational,2009
EGRAWPMExamplePassage:
![Page 10: Photo Credit: Varkey Foundation Effectiveness of ... of Interactive Distance Instruction: ... , each broadcasting to 12 classes at a time ... Early Grade Mathematics Assessment (EGMA)](https://reader031.vdocuments.us/reader031/viewer/2022030510/5abae5ee7f8b9a8f058bdfd3/html5/thumbnails/10.jpg)
• N=5485atbaseline- 60%girls- Averageage=10.9- ~8householdmembers- 50%mothersattendedschool- 60%fathersattendedschool
• Treatmentandcontrolbalancechecks- Balanceonobservablesatbaseline,midline,andendline- Midlineattrition=13%- Endlineattrition=16%- Nodifferenceinattritionbetweentreatmentandcontrolgroups
SampleCharacteristics
![Page 11: Photo Credit: Varkey Foundation Effectiveness of ... of Interactive Distance Instruction: ... , each broadcasting to 12 classes at a time ... Early Grade Mathematics Assessment (EGMA)](https://reader031.vdocuments.us/reader031/viewer/2022030510/5abae5ee7f8b9a8f058bdfd3/html5/thumbnails/11.jpg)
MainResultsWhatistheeffectofsatelliteprogramonliteracyandnumeracyskills?
![Page 12: Photo Credit: Varkey Foundation Effectiveness of ... of Interactive Distance Instruction: ... , each broadcasting to 12 classes at a time ... Early Grade Mathematics Assessment (EGMA)](https://reader031.vdocuments.us/reader031/viewer/2022030510/5abae5ee7f8b9a8f058bdfd3/html5/thumbnails/12.jpg)
◦Classessignificantlyimprovednumeracy◦ 0.22standarddeviationimprovementinyear1◦ 0.26standarddeviationimprovementinyear2
◦Gainsinmathscoreslasted oneyearaftertheprogramended◦ 0.23standarddeviationgainsforstudentstestedoneyearafterprogramended
◦Aftertwoyears,marginallysignificantimprovementinliteracy◦ 0.07standarddeviationgainsaftertwoyears
GainsinNumeracyandLiteracy
![Page 13: Photo Credit: Varkey Foundation Effectiveness of ... of Interactive Distance Instruction: ... , each broadcasting to 12 classes at a time ... Early Grade Mathematics Assessment (EGMA)](https://reader031.vdocuments.us/reader031/viewer/2022030510/5abae5ee7f8b9a8f058bdfd3/html5/thumbnails/13.jpg)
◦ After2yearsofclasses,largeandsignificantimprovementinfoundationalskills◦ 0.82standarddeviationgaininlettersperminute◦ 0.17standarddeviationgainininventedwordsperminute◦ 0.17standarddeviationgaininoralvocabulary
LiteracySubtasks
0.07
0.82
0.17 0.17
0.04 0.000
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Words per minute Letters per minute Invented words per minute
Oral vocabulary Listening comprehension
Reading comprehension
Stan
dard
dev
iatio
n ga
ins
Two years of classes***p<0.01,**p<0.05,*p<0.1
***
****
**
![Page 14: Photo Credit: Varkey Foundation Effectiveness of ... of Interactive Distance Instruction: ... , each broadcasting to 12 classes at a time ... Early Grade Mathematics Assessment (EGMA)](https://reader031.vdocuments.us/reader031/viewer/2022030510/5abae5ee7f8b9a8f058bdfd3/html5/thumbnails/14.jpg)
◦ Afteroneyearofclasses,significantimprovementinallnumeracysubtasks◦ Aftertwoyearsofclasses,greatergainsinnumberidentityand missingnumbertasks
NumeracySubtasks
0.2
0.150.17
0.19
0.34
0.12
0.250.21
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
Number identification Quantity discrimination Missing number Addition/subtraction
Stan
dard
dev
iatio
n ga
ins
One year of classes Two years of classes***p<0.01,**p<0.05,*p<0.1
***
******
***
***
**
******
*** *
![Page 15: Photo Credit: Varkey Foundation Effectiveness of ... of Interactive Distance Instruction: ... , each broadcasting to 12 classes at a time ... Early Grade Mathematics Assessment (EGMA)](https://reader031.vdocuments.us/reader031/viewer/2022030510/5abae5ee7f8b9a8f058bdfd3/html5/thumbnails/15.jpg)
MechanismsWhatarethepotentialdriversoflearninggains?
![Page 16: Photo Credit: Varkey Foundation Effectiveness of ... of Interactive Distance Instruction: ... , each broadcasting to 12 classes at a time ... Early Grade Mathematics Assessment (EGMA)](https://reader031.vdocuments.us/reader031/viewer/2022030510/5abae5ee7f8b9a8f058bdfd3/html5/thumbnails/16.jpg)
TheoryofChangeImprovequantityandqualityofprimaryschooleducation
Increaseinstructionaltime-on-task:§ Reduceabsenteeism§ Increaseclassroomtime-on-task§ “Double”instructorpresence
Improveteachingpedagogy:§ Leveragestudioteachercapacity§ Improvingfacilitatorcapacity
Improvecurricularcontent:§ Moreengaging§ Moreinteractivity
In-Schoo
l
Gainsinnumeracyandliteracy
§ ->Noevidenceofeffectsonattendance§ ->Noevidenceofeffectsontime-on-task§ ->BUT:In-classroomfacilitatorsengagedinon-task
activities(includingactiveinstruction)morethanhalfofthetimethatthevideowasbeingshown,essentiallyincreasingclassrooms’teacher-studentratio
![Page 17: Photo Credit: Varkey Foundation Effectiveness of ... of Interactive Distance Instruction: ... , each broadcasting to 12 classes at a time ... Early Grade Mathematics Assessment (EGMA)](https://reader031.vdocuments.us/reader031/viewer/2022030510/5abae5ee7f8b9a8f058bdfd3/html5/thumbnails/17.jpg)
TheoryofChangeImprovequantityandqualityofprimaryschooleducation
Increaseinstructionaltime-on-task:§ Reduceabsenteeism§ Increaseclassroomtime-on-task§ “Double”instructorpresence
Improveteachingpedagogy:§ Leveragestudioteachercapacity§ Improvingfacilitatorcapacity
Improvecurricularcontent:§ Moreengaging§ Moreinteractivity
In-Schoo
l
Gainsinnumeracyandliteracy
§ ->Noquantitativeevidenceofimprovedteachingmethods§ ->Qualitativeevidencesuggestsimprovements
![Page 18: Photo Credit: Varkey Foundation Effectiveness of ... of Interactive Distance Instruction: ... , each broadcasting to 12 classes at a time ... Early Grade Mathematics Assessment (EGMA)](https://reader031.vdocuments.us/reader031/viewer/2022030510/5abae5ee7f8b9a8f058bdfd3/html5/thumbnails/18.jpg)
Mechanisms:TeachingPedagogy• Studentfocusgroups/interviewssuggestimprovementsinlearningexperience
“Sometimesteachersleaveyoubehindifyoudon’tunderstandandevencaneus.MGCubedteacherstrytheirbesttomakeusunderstandlessons.”
• Facilitatorsseenaseffectiveteachingintermediaries“ThefacilitatorintheMGCubedclassexplainswhatthestudioteacherteachesifwearefindingitdifficulttounderstandthestudioteacher.”
• Newteachingmethodsandapproacheslearnedaspartoffacilitatorrole“Andtheytaughtuswayswecanusetoteachthem...Thoughweweretaught‘thinkpairshare’atthetrainingcollegeIwasnotusingitsowhenwehadtheprogrammerecentlyIwentbackandamnowusingit.”
![Page 19: Photo Credit: Varkey Foundation Effectiveness of ... of Interactive Distance Instruction: ... , each broadcasting to 12 classes at a time ... Early Grade Mathematics Assessment (EGMA)](https://reader031.vdocuments.us/reader031/viewer/2022030510/5abae5ee7f8b9a8f058bdfd3/html5/thumbnails/19.jpg)
TheoryofChangeImprovequantityandqualityofprimaryschooleducation
Increaseinstructionaltime-on-task:§ Reduceabsenteeism§ Increaseclassroomtime-on-task§ “Double”instructorpresence
Improveteachingpedagogy:§ Leveragestudioteachercapacity§ Improvingfacilitatorcapacity
Improvecurricularcontent:§ Moreengaging§ Moreinteractivity
In-Schoo
l
Gainsinnumeracyandliteracy
§ ->Stillinvestigating
![Page 20: Photo Credit: Varkey Foundation Effectiveness of ... of Interactive Distance Instruction: ... , each broadcasting to 12 classes at a time ... Early Grade Mathematics Assessment (EGMA)](https://reader031.vdocuments.us/reader031/viewer/2022030510/5abae5ee7f8b9a8f058bdfd3/html5/thumbnails/20.jpg)
PolicyImplicationsandNextSteps• Interactivedistancemodeleffectiveatincreasingliteracyandnumeracyskills
• Increasedinstructorpresenceinformofdistanceand in-personinstructoranimportantlevertoexplore– aretherewaystomakerolesmorecomplementary?
• Forthcomingisacosteffectivenessanalysistodeterminehowinterventioncomparestootherheavier-handedstructuredpedagogyprograms
• Relevanceofinteractivemodelincreasingascostsofinternetanddistanceinstructiondecreasing andinstitutionsincreaseuseofonline/blendedinstruction
![Page 21: Photo Credit: Varkey Foundation Effectiveness of ... of Interactive Distance Instruction: ... , each broadcasting to 12 classes at a time ... Early Grade Mathematics Assessment (EGMA)](https://reader031.vdocuments.us/reader031/viewer/2022030510/5abae5ee7f8b9a8f058bdfd3/html5/thumbnails/21.jpg)
LinktoWorkingPapercepa.stanford.edu/sites/default/files/wp17-08-v201708.pdf