Behavioral Responses to Teacher Transfer Incentives: Results from a Randomized Experiment

INVALSI Conference on Improving Education through Accountability and Evaluation: Lessons from Around the World

Rome, Italy

October 4, 2012

Steven Glazerman Ali Protik Bing-ru Teh Julie Bruch Neil Seftor

Best teachers may not be working with the students who need them the most

Shift focus from improving productivity of the teacher workforce to composition

Big gaps in knowledge– Weak documentation of the policy problem– Lack of data on teacher transfer behavior– Lack of data on whether skills transfer– Controversy about teacher quality measures

(value added)

Policy Problem

2

Policy Response: Talent Transfer Initiative

$20,000 transfer incentive

Identify highest-performing (HP) teachers– Use value-added analysis, three years of data– Three pools: elementary, MS math, MS language arts– Top 20% are “highest performing”

Identify potential “receiving schools”

Recruit transfer candidates, arrange interviews

Support transfer teachers, issue payments

HP teachers already in potential receiving schools get retention stipend of $10,000

3

1. How do HP teachers respond to a monetary transfer incentive?

2. How do hard-to-staff schools respond to the opportunity to hire a HP teacher?

3. What impact do transfer teachers have in their new settings?– Did their skills transfer, i.e. were they portable?– Was “value added” the right metric?

Research Questions

4

Summary of Findings to Date

Implementation– Filling vacancies was feasible

– Large pool of candidates needed

– Meaningful contrast achieved

Intermediate impacts– Increased experience and credentials slightly

– No significant impact on climate or collegiality

– No change in how students assigned to teachers

– TTI transfers used less & provided more mentoring

Impact on test scores and retention– Will be public in the final report (2013)

5

Study Design

6

Identify potential receiving schools with a vacancy in a targeted grade/subject

Unit of randomization = teacher team– Team types can be:

• Elementary self-contained math and reading• Middle school math• Middle school English/language arts (ELA)

Experimental Design

7

School A School B

Study Design, Illustration

8

Randomization Block

School A School BFocal Teachers

Randomly assign teacher teams (grade within school) to treatment or control

Study Design, Illustration

9

Data

10

Ten Large, Diverse Districts in the Study

11

Cohort 1: seven districts in five states Cohort 2: three districts in two more states

Primary Data Collection: Surveys– Candidates– Receiving school teachers in study grades– Receiving school principals

Secondary Data– District-provided test scores and demographics– School-provided teacher rosters

Data

12

7 districts – Large, diverse– 5 county, 2 city

1,012 transfer candidates– 63 transfers from 51 sending schools

86 receiving schools– 124 teams randomized

15,266 students– Below average prior achievement– 6% white, 48% African American, 72% free lunch

Sample (Cohort 1)

13

Behavioral Response to Incentives: Implementation Findings

14

Low takeup rates, most candidates do not apply

Not too low to fill positions (90% filled)

Hard to predict who transfers

Findings on Response to Incentives

15

Response to Incentives in 7 Districts: Takeup Rates

16

Types of Transfers by Change in School Achievement Ranks Before and After Transfer

17N = 63

Types of Transfers by Change in School Poverty Ranks Before and After Transfer

18N = 63

Who Filled the Vacancies?

Characteristic Treatment Focal

Control Focal

Difference

First Year Teaching (%) 0 21 -21*

Experience, Years in Teaching

13 8 5*

Has Master’s Degree or Doctorate (%)

48 21 27*

Has National Board Certification (%)

23 12 11

Transferred via TTI (%) 95 0 95*

Sample Size 63 41

19

Behavioral Response Within the Receiving Schools:

Intermediate Impacts

20

Survey questions on degree of collaboration, mutual trust, or sharing ideas: no evidence of impact

Differential assignment of students to teachers: mixed evidence of impact

Mentoring and leadership: treatment led to more mentoring provided, less mentoring used

Findings on Impacts on School Dynamics

21

Mentoring Received and Provided to Others

22

Receives Mentoring Mentors Others

Summary of Findings to Date

Implementation– Filling vacancies was feasible

– Large pool of candidates needed

– Meaningful contrast achieved

Intermediate impacts– Increased experience and credentials slightly

– No significant impact on climate or collegiality

– No change in how students assigned to teachers

– TTI transfers used less & provided more mentoring

Impact on test scores and retention– Will be public in the final report (2013)

23

Impacts on test scores and retention

Cost-benefit– Shadow price of raising test scores using CSR– Retention adjusted impacts, extrapolate into future?

Spatial analysis of mobility decisions

Related policies

Future Work

24

Transfer groups of teachers (e.g. through reconstitution)

Additional screening criteria for HP teachers

Bonus conditional on performance in new school

Policy that spans district boundaries (e.g. statewide)

Related Policies

25

THE END(extra slides follow)

26

Districts vary

Elementary and middle school differ

Overall pattern suggests:– Unequal access at middle school level– Less evidence for unequal access at elementary

level

Summary of Prevalence Findings

27

Prevalence of HP Teachers: Do Low-Income Students Have Equal

Access?

28

Prevalence of Highest-PerformingMiddle School Math Teachers*

29

Quintiles Based on Poverty

* Statistically significant

Prevalence of Highest-PerformingMiddle School Language Arts Teachers*

30

Quintiles Based on Poverty

* Statistically significant

Prevalence of Highest-PerformingElementary Teachers

31

Quintiles Based on Poverty

Results for Individual Districts Results, Five Districts at a Time

32

Prevalence of Highest-Performing Middle School Math Teachers (Districts A-E)

33

Quintiles Based on Poverty

Prevalence of Highest-Performing Middle School Math Teachers (Districts F-J)

34

Quintiles Based on Poverty

Overall, but Using Achievement to Divide Schools

35

Prevalence of Highest-PerformingMiddle School Math Teachers*

36

Quintiles Based on Achievement

* Statistically significant

Prevalence of Highest-PerformingMiddle School Language Arts Teachers*

37

Quintiles Based on Achievement

* Statistically significant

Prevalence of Highest-PerformingElementary Teachers*

38

Quintiles Based on Achievement

* Statistically significant

Decompose value added estimate

Components of Estimated Teacher Performance

39

ˆ jt j j jt jtQ X P Total Performance

Persistent Teacher Ability

Returns to Specialization

Transitory Performance

Noise, Luck, Measurement Error

Prevalence of Highest-Performing Middle School ELA Teachers (Districts A-E)

40

Quintiles Based on Poverty

Prevalence of Highest-PerformingMiddle School ELA Teachers (Districts F-J)

41

Quintiles Based on Poverty

Prevalence of Highest-Performing Elementary Teachers (Districts A-E)

42

Quintiles Based on Poverty

Prevalence of Highest-Performing Elementary Teachers (Districts F-J)

43

Quintiles Based on Poverty

School A School B

School pair with matching vacancies in two grades.

Study Design, Crossover Case

44

Randomization Block

School A School B

Study Design, Crossover Case (cont’d.)

45

Teacher Team Dynamics

46

Team-level– Impact estimate has intent-to-treat (ITT) interpretation

Under zero resource allocation effect:

Focal teacher comparison– Impact estimate denotes the direct impact

Nonfocal teacher comparison– Impact estimate denotes the indirect impact

Team and Focal Teacher Analysis

47

Dilution of direct effect

Non-compliers (unfilled vacancies)

Block-defined subgroups– High contrast transfers– High value added transfers– Complier blocks

Interpretation/Analysis Issues

48

Self-Reported Reasons For Not Applying

Factor (selected reasons)Cited as a Reason

Most Important Reason

Happy at old school 91 32Child care or family-related issues 33 15

Commuting issues 53 6Concerns about receiving school neighborhood/safety 36 2

Not confident about effectiveness in new school 29 3

Concerns about being unwelcome, unsupported in new school 43 4

Did not like principals at the receiving schools 6 2

Students in receiving schools too challenging 25 6

Stipend not big enough 25 3

49Percentages, N = 680

How Are Students Assigned to Classrooms?Principal Report (N=57 Treatment, 54 Control)

50

None of the differences are statistically significant.

Who Filled the Vacancies? (Controls)

Final Status of the Vacancy Number Percentage

Positions Filled

New to teaching 9 13.4

New hire (new to district or teaching) 6 9.0

Transfer from another school 13 19.4

Transfer from another grade/subject 18 26.9

Unknown origin/uncertain 6 9.0

Position Lost, Transfer Cancelled, or Layoff Rescinded 7 10.4

Unknown Status 8 11.9

All Vacancies 67 100.0

51

Who Filled the Vacancies? (Treatment)

Final Status of the Vacancy Number Percentage

Filled with TTI Candidate 63 90.0

Filled Outside TTI 4 5.7

Position Lost or Transfer Cancelled 3 4.3

All Vacancies 70 100.0

52

Measure (on a 5-point scale)Treatment

MeanControl

Mean Difference

Current levels

Level of collaboration 3.7 3.6 0.2

Degree of trust and mutual respect 3.9 3.8 0.1

Teaches seek ideas from one another 3.8 3.9 -0.1

Change from Prior Year

Level of collaboration 1.1 0.6 0.5

Degree of trust and mutual respect 1.1 0.6 0.4

Teaches seek ideas from one another 1.0 0.8 0.1

Principal Reports of Collaboration

53

None of the differences are statistically significant.

How Are Students Assigned to Classrooms? Teacher Report (N=169 Treatment, 155 Control)

54

“Compared to my colleagues’ students’ academic ability, my students’ academic ability is…”

None of the differences are statistically significant.

How Are Students Assigned to Classrooms? Teacher Report (N=169 Treatment, 155 Control)

55

“Compared to my colleagues’ students’ behavior, my students’ behavior is…”

None of the differences are statistically significant.