Impact Evaluation of MathCloud in Sri Lanka
South Asia Regional Symposium on ICT for EducationShangriLa Hotel, Colombo. February 28, 2018
Bilesha Weeraratne, PhD.
Institute of Policy Studies of Sri Lanka.
7
MC in Sri Lanka
1 Pilot
Treatment
Tests
Implementation
Methodology
Results
Bilesha Weeraratne, PhD.
MC Pilot Project
I 1383 Grade 8 Students
I MathCloud (MC) ran for 7 months - May to Nov. 2014.
I 3 Rounds of testing and surveying of students.
I Pre test and survey in May 2014
I Post Test 1 and survey Oct. 2014
I Post Test 2 and survey Nov. 2014
7
MC in Sri Lanka
Pilot
2 Treatment
Tests
Implementation
Methodology
Results
Bilesha Weeraratne, PhD.
MC Treatment
Control (C) Group
I 674 Students
I 5 days of the week
I All 5 days regular classroom environmentwith a mathematicsteacher
I Faced all 3 tests andsurveys
Treatment (T) Group
I 709 Students
I 5 days of the week
I 2-3 days : computerbased instructions by themath teacher using MCin a computer lab
I 2-3 days : regular classroom environment with amathematics teacher
I Faced all 3 tests and surveys
7
MC in Sri Lanka
Pilot
2 Treatment
Tests
Implementation
Methodology
Results
Bilesha Weeraratne, PhD.
MC Treatment
Control (C) Group
I 674 Students
I 5 days of the weekI All 5 days regular class
room environmentwith a mathematicsteacher
I Faced all 3 tests andsurveys
Treatment (T) Group
I 709 Students
I 5 days of the week
I 2-3 days : computerbased instructions by themath teacher using MCin a computer lab
I 2-3 days : regular classroom environment with amathematics teacher
I Faced all 3 tests and surveys
7
MC in Sri Lanka
Pilot
2 Treatment
Tests
Implementation
Methodology
Results
Bilesha Weeraratne, PhD.
MC Treatment
Control (C) Group
I 674 Students
I 5 days of the weekI All 5 days regular class
room environmentwith a mathematicsteacher
I Faced all 3 tests andsurveys
Treatment (T) Group
I 709 Students
I 5 days of the week
I 2-3 days : computerbased instructions by themath teacher using MCin a computer lab
I 2-3 days : regular classroom environment with amathematics teacher
I Faced all 3 tests and surveys
7
MC in Sri Lanka
Pilot
Treatment
3 Tests
Implementation
Methodology
Results
Bilesha Weeraratne, PhD.
Testing
I 30 Test items in each test
I Pre Test: both 2nd and 3rd academic syllabus
I Post-Test 1 : only 2nd academic content
I Post-Test 2 : both 2nd and 3rd academic content
7
MC in Sri Lanka
Pilot
Treatment
Tests
4 Implementation
Methodology
Results
Bilesha Weeraratne, PhD.
Implementation Experience
I SamplingI Planned for randomization but ended up being purposive
sampling.I Duration
I Planned for 1 years ended up with 7 months.I During intervention -
I shared computers, used MC outside their scheduled MChours.
I MethodologyI Planned for Difference in difference model - but couldn’t
collect all required previous test scores.
7
MC in Sri Lanka
Pilot
Treatment
Tests
Implementation
5 Methodology
Results
Bilesha Weeraratne, PhD.
Impact Evaluation Methodology
Propensity Score Matching (PSM)
I PSM controls for possible selection bias
I Makes T/C more comparable based on observable char.
I Similarity between subjects is based on estimated treatmentprobabilities = propensity scores.
3 versions of Test Scores
I Raw scores
I Standardized scores
I Item Response Theory adjusted Scaled Scores
7
MC in Sri Lanka
Pilot
Treatment
Tests
Implementation
Methodology
6 Results
Bilesha Weeraratne, PhD.
Impact Estimates of Change in Test Scores
Period Pre-Test to Post Test-1 Pre-Test to Post Test-2Outcome Raw Std Scaled Raw Std Scaled
MainATE 1.45*** 0.27*** 1.10*** 0.58* 0.11 0.65***ATT 1.46*** 0.27*** 1.13*** 0.47 0.09 0.59**
Rob 1ATE 0.97** 0.18** 0.35 0.49 0.09 0.37ATT 0.87** 0.16** 0.18 0.36 0.04 0.23
Rob 2ATE 1.38*** 0.25*** 1.08** -0.27 -0.05 0.55ATT 1.62*** 0.30*** 1.09** -0.04 -0.01 0.58
Notes :
Standard errors in parentheses
* p<0.10 ** p<0.05 *** p<0.01
Main Models : Matching based on students’ characteristics
Rob 1: Matching based on students’, teachers’ and school characteristics
Rob 2: Matching based on students’, teachers’ and school characteristics +
Sample restricted to common mathematics teachers for T/C
7
MC in Sri Lanka
Pilot
Treatment
Tests
Implementation
Methodology
6 Results
Bilesha Weeraratne, PhD.
Impact Estimates of Change in Test Scores
Period Pre-Test to Post Test-1 Pre-Test to Post Test-2Outcome Raw Std Scaled Raw Std Scaled
MainATE 1.45*** 0.27*** 1.10*** 0.58* 0.11 0.65***ATT 1.46*** 0.27*** 1.13*** 0.47 0.09 0.59**
Rob 1ATE 0.97** 0.18** 0.35 0.49 0.09 0.37ATT 0.87** 0.16** 0.18 0.36 0.04 0.23
Rob 2ATE 1.38*** 0.25*** 1.08** -0.27 -0.05 0.55ATT 1.62*** 0.30*** 1.09** -0.04 -0.01 0.58
Notes :
Standard errors in parentheses
* p<0.10 ** p<0.05 *** p<0.01
Main Models : Matching based on students’ characteristics
Rob 1: Matching based on students’, teachers’ and school characteristics
Rob 2: Matching based on students’, teachers’ and school characteristics +
Sample restricted to common mathematics teachers for T/C
7
MC in Sri Lanka
Pilot
Treatment
Tests
Implementation
Methodology
6 Results
Bilesha Weeraratne, PhD.
Impact Estimates of Change in Test Scores
Period Pre-Test to Post Test-1 Pre-Test to Post Test-2Outcome Raw Std Scaled Raw Std Scaled
MainATE 1.45*** 0.27*** 1.10*** 0.58* 0.11 0.65***ATT 1.46*** 0.27*** 1.13*** 0.47 0.09 0.59**
Rob 1ATE 0.97** 0.18** 0.35 0.49 0.09 0.37ATT 0.87** 0.16** 0.18 0.36 0.04 0.23
Rob 2ATE 1.38*** 0.25*** 1.08** -0.27 -0.05 0.55ATT 1.62*** 0.30*** 1.09** -0.04 -0.01 0.58
Notes :
Standard errors in parentheses
* p<0.10 ** p<0.05 *** p<0.01
Main Models : Matching based on students’ characteristics
Rob 1: Matching based on students’, teachers’ and school characteristics
Rob 2: Matching based on students’, teachers’ and school characteristics +
Sample restricted to common mathematics teachers for T/C
7
MC in Sri Lanka
Pilot
Treatment
Tests
Implementation
Methodology
6 Results
Bilesha Weeraratne, PhD.
Impact Estimates of Change in Test Scores
Period Pre-Test to Post Test-1 Pre-Test to Post Test-2Outcome Raw Std Scaled Raw Std Scaled
MainATE 1.45*** 0.27*** 1.10*** 0.58* 0.11 0.65***ATT 1.46*** 0.27*** 1.13*** 0.47 0.09 0.59**
Rob 1ATE 0.97** 0.18** 0.35 0.49 0.09 0.37ATT 0.87** 0.16** 0.18 0.36 0.04 0.23
Rob 2ATE 1.38*** 0.25*** 1.08** -0.27 -0.05 0.55ATT 1.62*** 0.30*** 1.09** -0.04 -0.01 0.58
Notes :
Standard errors in parentheses
* p<0.10 ** p<0.05 *** p<0.01
Main Models : Matching based on students’ characteristics
Rob 1: Matching based on students’, teachers’ and school characteristics
Rob 2: Matching based on students’, teachers’ and school characteristics +
Sample restricted to common mathematics teachers for T/C
7
MC in Sri Lanka
Pilot
Treatment
Tests
Implementation
Methodology
7 Results
Bilesha Weeraratne, PhD.
Takeaway Messages
I MC ⇑ scaled scores by approx 3.5 percentage points afterPost Test 1
I MC ⇑ scaled scores by approx 2 percentage points after PostTest 2