Impact Evaluation of MathCloud in Sri Lanka

South Asia Regional Symposium on ICT for EducationShangriLa Hotel, Colombo. February 28, 2018

Bilesha Weeraratne, PhD.

Institute of Policy Studies of Sri Lanka.

7

MC in Sri Lanka

1 Pilot

Treatment

Tests

Implementation

Methodology

Results

Bilesha Weeraratne, PhD.

MC Pilot Project

I 1383 Grade 8 Students

I MathCloud (MC) ran for 7 months - May to Nov. 2014.

I 3 Rounds of testing and surveying of students.

I Pre test and survey in May 2014

I Post Test 1 and survey Oct. 2014

I Post Test 2 and survey Nov. 2014

7

MC in Sri Lanka

Pilot

2 Treatment

Tests

Implementation

Methodology

Results

Bilesha Weeraratne, PhD.

MC Treatment

Control (C) Group

I 674 Students

I 5 days of the week

I All 5 days regular classroom environmentwith a mathematicsteacher

I Faced all 3 tests andsurveys

Treatment (T) Group

I 709 Students

I 5 days of the week

I 2-3 days : computerbased instructions by themath teacher using MCin a computer lab

I 2-3 days : regular classroom environment with amathematics teacher

I Faced all 3 tests and surveys

7

MC in Sri Lanka

Pilot

2 Treatment

Tests

Implementation

Methodology

Results

Bilesha Weeraratne, PhD.

MC Treatment

Control (C) Group

I 674 Students

I 5 days of the weekI All 5 days regular class

room environmentwith a mathematicsteacher

I Faced all 3 tests andsurveys

Treatment (T) Group

I 709 Students

I 5 days of the week

I 2-3 days : computerbased instructions by themath teacher using MCin a computer lab

I 2-3 days : regular classroom environment with amathematics teacher

I Faced all 3 tests and surveys

7

MC in Sri Lanka

Pilot

2 Treatment

Tests

Implementation

Methodology

Results

Bilesha Weeraratne, PhD.

MC Treatment

Control (C) Group

I 674 Students

I 5 days of the weekI All 5 days regular class

room environmentwith a mathematicsteacher

I Faced all 3 tests andsurveys

Treatment (T) Group

I 709 Students

I 5 days of the week

I 2-3 days : computerbased instructions by themath teacher using MCin a computer lab

I 2-3 days : regular classroom environment with amathematics teacher

I Faced all 3 tests and surveys

7

MC in Sri Lanka

Pilot

Treatment

3 Tests

Implementation

Methodology

Results

Bilesha Weeraratne, PhD.

Testing

I 30 Test items in each test

I Pre Test: both 2nd and 3rd academic syllabus

I Post-Test 1 : only 2nd academic content

I Post-Test 2 : both 2nd and 3rd academic content

7

MC in Sri Lanka

Pilot

Treatment

Tests

4 Implementation

Methodology

Results

Bilesha Weeraratne, PhD.

Implementation Experience

I SamplingI Planned for randomization but ended up being purposive

sampling.I Duration

I Planned for 1 years ended up with 7 months.I During intervention -

I shared computers, used MC outside their scheduled MChours.

I MethodologyI Planned for Difference in difference model - but couldn’t

collect all required previous test scores.

7

MC in Sri Lanka

Pilot

Treatment

Tests

Implementation

5 Methodology

Results

Bilesha Weeraratne, PhD.

Impact Evaluation Methodology

Propensity Score Matching (PSM)

I PSM controls for possible selection bias

I Makes T/C more comparable based on observable char.

I Similarity between subjects is based on estimated treatmentprobabilities = propensity scores.

3 versions of Test Scores

I Raw scores

I Standardized scores

I Item Response Theory adjusted Scaled Scores

7

MC in Sri Lanka

Pilot

Treatment

Tests

Implementation

Methodology

6 Results

Bilesha Weeraratne, PhD.

Impact Estimates of Change in Test Scores

Period Pre-Test to Post Test-1 Pre-Test to Post Test-2Outcome Raw Std Scaled Raw Std Scaled

MainATE 1.45*** 0.27*** 1.10*** 0.58* 0.11 0.65***ATT 1.46*** 0.27*** 1.13*** 0.47 0.09 0.59**

Rob 1ATE 0.97** 0.18** 0.35 0.49 0.09 0.37ATT 0.87** 0.16** 0.18 0.36 0.04 0.23

Rob 2ATE 1.38*** 0.25*** 1.08** -0.27 -0.05 0.55ATT 1.62*** 0.30*** 1.09** -0.04 -0.01 0.58

Notes :

Standard errors in parentheses

* p<0.10 ** p<0.05 *** p<0.01

Main Models : Matching based on students’ characteristics

Rob 1: Matching based on students’, teachers’ and school characteristics

Rob 2: Matching based on students’, teachers’ and school characteristics +

Sample restricted to common mathematics teachers for T/C

7

MC in Sri Lanka

Pilot

Treatment

Tests

Implementation

Methodology

6 Results

Bilesha Weeraratne, PhD.

Impact Estimates of Change in Test Scores

Period Pre-Test to Post Test-1 Pre-Test to Post Test-2Outcome Raw Std Scaled Raw Std Scaled

MainATE 1.45*** 0.27*** 1.10*** 0.58* 0.11 0.65***ATT 1.46*** 0.27*** 1.13*** 0.47 0.09 0.59**

Rob 1ATE 0.97** 0.18** 0.35 0.49 0.09 0.37ATT 0.87** 0.16** 0.18 0.36 0.04 0.23

Rob 2ATE 1.38*** 0.25*** 1.08** -0.27 -0.05 0.55ATT 1.62*** 0.30*** 1.09** -0.04 -0.01 0.58

Notes :

Standard errors in parentheses

* p<0.10 ** p<0.05 *** p<0.01

Main Models : Matching based on students’ characteristics

Rob 1: Matching based on students’, teachers’ and school characteristics

Rob 2: Matching based on students’, teachers’ and school characteristics +

Sample restricted to common mathematics teachers for T/C

7

MC in Sri Lanka

Pilot

Treatment

Tests

Implementation

Methodology

6 Results

Bilesha Weeraratne, PhD.

Impact Estimates of Change in Test Scores

Period Pre-Test to Post Test-1 Pre-Test to Post Test-2Outcome Raw Std Scaled Raw Std Scaled

MainATE 1.45*** 0.27*** 1.10*** 0.58* 0.11 0.65***ATT 1.46*** 0.27*** 1.13*** 0.47 0.09 0.59**

Rob 1ATE 0.97** 0.18** 0.35 0.49 0.09 0.37ATT 0.87** 0.16** 0.18 0.36 0.04 0.23

Rob 2ATE 1.38*** 0.25*** 1.08** -0.27 -0.05 0.55ATT 1.62*** 0.30*** 1.09** -0.04 -0.01 0.58

Notes :

Standard errors in parentheses

* p<0.10 ** p<0.05 *** p<0.01

Main Models : Matching based on students’ characteristics

Rob 1: Matching based on students’, teachers’ and school characteristics

Rob 2: Matching based on students’, teachers’ and school characteristics +

Sample restricted to common mathematics teachers for T/C

7

MC in Sri Lanka

Pilot

Treatment

Tests

Implementation

Methodology

6 Results

Bilesha Weeraratne, PhD.

Impact Estimates of Change in Test Scores

Period Pre-Test to Post Test-1 Pre-Test to Post Test-2Outcome Raw Std Scaled Raw Std Scaled

MainATE 1.45*** 0.27*** 1.10*** 0.58* 0.11 0.65***ATT 1.46*** 0.27*** 1.13*** 0.47 0.09 0.59**

Rob 1ATE 0.97** 0.18** 0.35 0.49 0.09 0.37ATT 0.87** 0.16** 0.18 0.36 0.04 0.23

Rob 2ATE 1.38*** 0.25*** 1.08** -0.27 -0.05 0.55ATT 1.62*** 0.30*** 1.09** -0.04 -0.01 0.58

Notes :

Standard errors in parentheses

* p<0.10 ** p<0.05 *** p<0.01

Main Models : Matching based on students’ characteristics

Rob 1: Matching based on students’, teachers’ and school characteristics

Rob 2: Matching based on students’, teachers’ and school characteristics +

Sample restricted to common mathematics teachers for T/C

7

MC in Sri Lanka

Pilot

Treatment

Tests

Implementation

Methodology

7 Results

Bilesha Weeraratne, PhD.

Takeaway Messages

I MC ⇑ scaled scores by approx 3.5 percentage points afterPost Test 1

I MC ⇑ scaled scores by approx 2 percentage points after PostTest 2

THANK YOU***

Bilesha Weeraratne, PhD.Institute of Policy Studies of Sri Lanka

100/20 Independence AvenueColombo 7, Sri LankaT: +94-112-143-100

www.ips.lk