teacher professional development and its effects on students: evidence from the m@t.abel program in...

Teacher Professional Development and its Effects on Students:

Evidence from the M@t.abel Program in Italy

Gianluca Argentin Aline Pennisi

Daniele VidoniGiovanni AbbiatiAndrea Caputo

The problem

Weakness of Italian students in international assessments on mathematics and science

(i.e. IEA, TIMSS and OECD PISA).

North South gap achievement.

North/South gap increasing over grades(difference from national average in the percentage of correct answers in grades 2,5,6 and 8 – source: SNV)

-5,0

-4,0

-3,0

-2,0

-1,0

0,0

1,0

2,0

3,0

4,0

5,0

2 5 6 8 2 5 6 8 2 5 6 8

North Centre South

difference from national average

The opportunity

Thanks to the EU funding, there has been a boost in initiatives

to help schools and teachers to improve student achievements.

(PON Istruzione 2007-2013I-3-FSE-2009-2)

Target regions of PONEU funding.

One of the solutions

A professional development program, called M@t.abel, was offered to

tenured math teachers in lower/upper secondary schools.

Enrolment on voluntary basis at school/teacher levels.

The questions

Does M@t.abel work?

Does it increase students’ math achievment?

Does it change teachers’ way of teaching?

We are tryng to get an answer using a rigorous method,

a Randomized Control Trial.

To our knwoledge, this is an absolute first in the Italian school system

How did we make randomization acceptable to Italian school

authorities?

Delaying by one year the PD of the control group,

instead of excluding it completely.

The M@t.abel program in the Italian context

The M@t.abel program:• math applied to daily life problems

• is based on a mixture of formal lectures and on-line mentoring;

• offers a huge repository ofscripts for math lessons;

• it lasts one entire school year;• it requires to be implemented in

classes (at least 4 units); • it promotes teacher community.

M@t.abel seems promising, according to the literature on

professional development[Garet et al 2001; Desimone et al 2002]:

• content focused;• extended in duration;

• active learning processes; • implemented directly in classes;

• based on peer collaboration.

M@t.abel seems promising also looking at the Italian teachers:

• the oldest worldwide [OECD 2007];• the majority did not have any

specific training in teaching;• the majority of math teachers

did not graduate in math/physics.

Moreover, during their career, the Italian teachers:• rarely attend PD;

• are not assessed at all;• do not have any salary differentiation

based on merit;• even do not have feedbacks

about their job [Talis 2008].

The Randomized Control Trial

The RCT – 1st year

1. Teachers applied for the PD through their school

2. The school must send at least 2 teachers

3. Schools were randomly assigned to treatment in the current year or to

delayed treatment next year

The estimate of the effect of the PD is the difference between T & C

Effect = T - C

For the first year, we have a classical experiment

Randomization

September 2009

Sent to the treatment

Control group

Schools 125 50Teachers 473 193

We checked the equivalence on an unusually wide set of characteristics at schools/teachers/students levels.

The equivalence is guaranteed.We found only minor differences,

anyway controlled in our estimates.

The outcomes

Students: • math test scores on the

standardized National Assessment• attitude toward math

Teachers: •self reported teaching behaviours•attitudes toward math teaching

Data collectionNovember 2009/January 2010

CATI survey pre-intervention on teachers’ attitudes

May 2010

Standardized math tests + questionnaires on students (background & attitude vs math)

+ data from teachers’ logs

December 2010CATI survey post-intervention on teachers’ attitudes and evaluation of the experience.

Intervention

November 2009 – May 2010

TEACHERS Sent to the treatment

(473)

Control group(193)

Lost – no data available

79 27

Compliers 156 166Non compliers 238 0

STUDENTS 7.692 3.372

Only 39% of teachers are compliers ….

Drop out Attended only lectures Attended courseAttended course + classroom activities with students

Effects estimations

ITTComparison of sent to treatment

and control group

ATTComparison of actually treated

and control group (Instrumental Variable regression)

The short term effects

1st year for studentsbeginning of 2nd year for teachers

Effects on students math achievment

No effect.

Math achievment scaled to an average of 500 and standard deviation of 100 for the 7th grade.

Descriptive statistics

Effect estimates and

standard errors

(OLS and IV regression)

Treatment Control

ITT ATT

Math score (mean) 493 496 1,8 4,8 (4,8) (12,3)

Effects on students attitudes

Some slight effects:• more interested in maths (1 item out of 4);

• feeling more time costraints;• less frequently attributing academic failure to

chance or to bad luck;• more anxious during the assessment;

• more frequently skipping at least one item during the assessment.

Development of a perfectionist attitude?

Estimated effects on teachersIncrease:

• more exchange with colleagues• more frequent lessons based

on group activities• production of didactic material

Decrease:• use of the school textbook

• mnemonic approach to math learning• perceived self efficacy in

making students work in groups

Further data collection on teachers

May-June 2012

Good reponse rate: 90%Teachers who answered all the surveys: 85%

Survey on•Attitudes

•Instructional practices•Use of Matabel 2 years after of the experiment

Further data collection on students

May 2011-May 2012

National Students Assessment

Measure of math performance

All previous classes (6/7th grade, now 7/8th on 2011, 6th grades now 8th grades on 2012),

but not all students (dropping out, failures)

Further step

Merging data at student individual level,obtaining a panel and assessing the

PON M@t.abel effect on the increase in math achievement across school years

Long term effects estimation on teachers

Concluding...

• it is possible to run a RCT evaluation also in the Italian school system;

• first year results do not show any effect of M@t.abel on the main outcome...

• ... but we found promising effects on teachers and students attitudes

• second and third year estimation will be crucial to evaluate the effectiveness of the

program

Thank you!

Further steps

…in Year 2 many (56%) of the former controls are treated

Computing the difference TY1&2 - TY2

we will get the effect of one additional year of exposure to the treatment.

If we want the effect of 2 years of exposure to the new method

We have to compute the difference TY1&2 - CY1

The non-testable assumption is that there is no cohort effect.

Additional analysis

1. Effects on the score distribution(quantile regression)

2. Subgroup analyses(effects’ heterogeneity)

Eterogeneous effects by teachers age

But too much uncertainty...

Probability of being full complier

Compliance is associated mostly with individual factors, such as:o Age (50-55 year old, - 16 perc.

points; over-55, - 22 perc. points)o Previous training experience (+11 perc.

points)o Personal motivation to enroll (-25 perc.

points if forced to enroll)

(Binary logistic regression models)

2. Data collection (3rd year)

3. Data collection on a brand new cohort of teachers

Dimensions Value

controls ITT ATT

Attitudes towards math

5 items factor (std score) -0,05 +0,05 +0,12

4 items factor (std score) -0,05 +0,05 +0,12

In math I’m good (1-4 points scale)

+2,78 +0,05** +0,12**

Curriculum pace

We proceeded even if some classmates did not understand the topic (1-4 points scale)

+1,55 +0,07*** +0,17**

Causal Attributions

Attribution of failures to bad luck (0-6 points scale factor)

+0,19 -0,04*** -0,09***

Test Anxiety

4 items factor (std score) -0,04 +0,05* +0,13*

I was so nervous I could not find the answers (1-4 points scale)

+1,89 +0,06*** +0,16***

Effects on students’ attitudes