portability of teacher effectiveness across school settings
DESCRIPTION
Bill & Melinda Gates Foundation Evaluation of the Intensive Partnership Sites initiative. Portability of Teacher Effectiveness across School Settings. Zeyu Xu, Umut Ozek, Matthew Corritore. Motivation. › Introduction. › Data and Samples. › Methodology. › Findings. - PowerPoint PPT PresentationTRANSCRIPT
Portability of Teacher Effectiveness across School Settings
Zeyu Xu, Umut Ozek, Matthew Corritore
April 21, 2023
Bill & Melinda Gates FoundationEvaluation of the Intensive Partnership Sites initiative
Motivation Redistributing effective teachers at the center of several education
policy initiatives Teacher is the most important school input related to student learning The distribution of effective teachers is uneven (recruiting, who moves, and
to where)
Key assumption: Teachers effectiveness is portable Students face different challenges in learning School culture, environment and working conditions may affect teacher
learning, practices, efforts, burnout, etc.
Literature Jackson (2010), Jackson & Bruegmann (2009), Goldhaber & Hansen (2010) Sanders, Wright & Langevin (2009)
› Introduction› Data and Samples› Methodology› Findings› Summary and Discussion
April 21, 2023
Research Questions Do teachers retain their effectiveness across schools
On average Across schools with similar settings Across schools with different settings (by the direction of the change)
Teacher effectiveness measured by Value-added
Settings defined by School performance levels School poverty levels
Conditional on teachers switching schools
› Introduction› Data and Samples› Methodology› Findings› Summary and Discussion
April 21, 2023
Preview of Findings Among teachers who changed schools, on average their VA was
unchanged or slightly improved
The same conclusion holds regardless of the similarity/difference between the sending and receiving schools or the direction of the move
High-performing teachers’ VA dropped and low-performing teachers’ VA gained in post-move years
This pattern is mostly driven by regression to the within-teacher mean and has little to do with school moves
Despite this pattern, high VA teachers still performed at a higher level than low VA teachers in post-move years
› Introduction› Data and Samples› Methodology› Findings› Summary and Discussion
April 21, 2023
Organization Data and samples
Methodology
Findings
Summary and discussion
› Introduction› Data and Samples› Methodology› Findings› Summary and Discussion
April 21, 2023
Data North Carolina 1998-99 through 2008-09
Elementary level (4th and 5th grade math and reading teachers, self-contained classrooms)
Secondary level (algebra I and English I teachers, “Algebra I”, “Algebra I-B”, “Integrated Math II”, “English I” classrooms)
Florida 2002-03 through 2008-09 Elementary level (4th and 5th grade math and reading teachers, “core
courses” in a given subject) Secondary level (9th and 10th grade math and reading teachers, “core
courses” in a given subject)
› Introduction› Data and Samples› Methodology› Findings› Summary and Discussion
April 21, 2023
Sample restrictions Remove charter schools
Remove students and teachers who changed schools during a school year (about 2-4% of obs)
Remove students with missing values on covariates
Keep classrooms with 10~40 students
Remove classrooms with >50% special education students
› Introduction› Data and Samples› Methodology› Findings› Summary and Discussion
April 21, 2023
Sample sizes
North Carolina Florida
Elementary Secondary Elementary Secondary
Math 21,119 4,999 29,989 9,101
Reading 21,119 3,775 29,354 9,681
› Introduction› Data and Samples› Methodology› Findings› Summary and Discussion
April 21, 2023
Number of Unique Teachers in the Analytic Samples
Two-Stage Analysis
Estimate teacher-year value-added
Difference-in-differences analysis
› Introduction› Data and Samples› Methodology› Findings› Summary and Discussion
April 21, 2023
Estimate Teacher VA
Test scores standardized by year, grade and subject (mean=0, sd=1) (X) Covariates include:
1) grade repetition, 2) FRPL, 3) sex, 4) race/ethnicity, 5) gifted, 6) special education, 7) student school mobility and 8) grade level.
Bias (no school FE) Noise (EB adjustment) Alternative model specifications (achievement levels model)
› Introduction› Data and Samples› Methodology› Findings› Summary and Discussion
April 21, 2023
𝐴𝑖𝑡 −𝐴𝑖𝑡−1 = 𝑇𝑖𝑡𝛽+𝑋𝑖𝑡𝛾+𝜀𝑖𝑡
DiD
Three groups: non-movers, movers to a similar school setting, movers to a different school setting
FGLS, se clustered at the teacher level (Y) Year and (T) teacher FEs (X) Teacher experience (0-2, 3-5, 6-12, 13 or more years of exp) (S) School quality (average peer VA) (C) Classroom characteristics (FRL %, mean pretest score, sd of
pretest score) (Post) Post-move years indicator (DP, DN) Indicators for school setting differences
› Introduction› Data and Samples› Methodology› Findings› Summary and Discussion
April 21, 2023
𝛽መ𝑗𝑡 = 𝑌𝑡 +𝑇𝑖 +𝑋𝑗𝑡𝑣1 +𝑆𝑗𝑣2 +𝐶𝑗𝑡𝑣3 +𝑃𝑜𝑠𝑡𝑗𝑡𝑣4 +𝑃𝑜𝑠𝑡𝑗𝑡𝐷𝑃𝑗𝑣5 +𝑃𝑜𝑠𝑡𝑗𝑡𝐷𝑁𝑗𝑣6 +𝜀𝑗𝑡
Define School Settings School performance
NC: % students performing at or above grade level FL: School performance scores based on both levels and growth Standardized by year and aggregated across all years
School poverty % FRPL Aggregated across all years in which a teacher taught in that school
Change in school setting measures ∆ = Receiving school – Sending school Similar setting = within half a SD around the mean of the ∆ distribution DP = 1 if ∆ > 0.25 (performance) or ∆ > 0.15 (poverty) DN = 1 if ∆ < -0.25 (performance) or ∆ < -0.15 (poverty)
› Introduction› Data and Samples› Methodology› Findings› Summary and Discussion
April 21, 2023
Alternative DiD Specs Last pre-move year and first post-move year
Between- vs. within-district moves
Replace the post-move indicator with individual year dummies (It-1, It-2, It-3…; It+1, It+2, It+3)
› Introduction› Data and Samples› Methodology› Findings› Summary and Discussion
April 21, 2023
Distribution of Movers› Introduction› Data and Samples› Methodology› Findings› Summary and Discussion
April 21, 2023
By school performance setting change
Distribution of Movers› Introduction› Data and Samples› Methodology› Findings› Summary and Discussion
April 21, 2023
By school poverty setting change
Mover Characteristics› Introduction› Data and Samples› Methodology› Findings› Summary and Discussion
April 21, 2023
NC elementary school teachers, by mobility status
Pre-Post Change in VA (elem)› Introduction› Data and Samples› Methodology› Findings› Summary and Discussion
April 21, 2023
North Carolina Florida
Math Reading Math Reading
All 0.004 0.005 -0.001 0.002
By school perf.
Higher to lower 0.019 0.011 0.002 0.002
Similar 0.004 0.004 0.007 -0.001
Lower to higher -0.002 0.003 -0.005 0.004
By school poverty
Higher to lower -0.005 0.002 -0.004 0.002
Similar 0.005 0.004 0.000 0.002
Lower to higher 0.020 0.017 0.009 0.009
Pre-Post Change in VA (sec)› Introduction› Data and Samples› Methodology› Findings› Summary and Discussion
April 21, 2023
North Carolina Florida
Math Reading Math Reading
All 0.056 0.003 0.003 0.005
By school perf.
Higher to lower 0.067 -0.011 0.003 0.013
Similar 0.085 0.014 0.006 0.008
Lower to higher 0.030 0.005 0.002 0.000
By school poverty
Higher to lower 0.111 0.002 0.002 -0.006
Similar 0.057 0.010 0.004 0.006
Lower to higher -0.010 -0.020 -0.003 0.019
By Pre-Move VA› Introduction› Data and Samples› Methodology› Findings› Summary and Discussion
April 21, 2023
Actual year of move “Pseudo” move
By Pre-Move VA› Introduction› Data and Samples› Methodology› Findings› Summary and Discussion
April 21, 2023
Elementary math teachers Elementary math teachers (pseudo move)
By Pre-Move VA› Introduction› Data and Samples› Methodology› Findings› Summary and Discussion
April 21, 2023
Elementary reading teachers Elementary reading teachers (pseudo move)
By Pre-Move VA› Introduction› Data and Samples› Methodology› Findings› Summary and Discussion
April 21, 2023
Secondary math teachers Secondary math teachers (pseudo move)
By Pre-Move VA› Introduction› Data and Samples› Methodology› Findings› Summary and Discussion
April 21, 2023
Secondary reading teachers Secondary reading teachers (pseudo move)
Adjacent Year Correlations› Introduction› Data and Samples› Methodology› Findings› Summary and Discussion
April 21, 2023
Correlation
North Carolina Florida
Math Reading Math Reading
Yt-2, Yt-1 0.483 0.298 0.380 0.187
(0.426, 0.535) (0.232, 0.362) (0.314, 0.443) (0.111, 0.260)
Yt-1, Yt+1 0.341 0.270 0.302 0.138
(0.256, 0.420) (0.182, 0.354) (0.231, 0.369) (0.061, 0.213)
Yt-+1 Yt-+2 0.463 0.269 0.427 0.191
(0.381, 0.537) (0.175, 0.358) (0.363, 0.487) (0.115, 0.264)
Pre-Post Comparisons of VA› Introduction› Data and Samples› Methodology› Findings› Summary and Discussion
April 21, 2023
North Carolina
Pre-Post Comparisons of VA› Introduction› Data and Samples› Methodology› Findings› Summary and Discussion
April 21, 2023
Florida
Summary Among teachers who changed schools, on average their VA was
unchanged or slightly improved
The same conclusion holds regardless of the similarity/difference between the sending and receiving schools or the direction of the move
High-performing teachers’ VA dropped and low-performing teachers’ VA gained in post-move years
This pattern is mostly driven by regression to the within-teacher mean and has little to do with school moves
Despite this pattern, high VA teachers still performed at a higher level than low VA teachers in post-move years
› Introduction› Data and Samples› Methodology› Findings› Summary and Discussion
April 21, 2023
Discussion Teacher effectiveness does not appear to be hurt by moving to
schools with different settings.
Multiple years of VA estimates can be used with other teacher evaluation data to identify effective teachers, capturing persistent teacher performance better and reducing post-move year shrinkage.
All results take teacher school changes as given.
› Introduction› Data and Samples› Methodology› Findings› Summary and Discussion
April 21, 2023