informing policy: state longitudinal data systems jane hannaway, director the urban institute calder
TRANSCRIPT
Informing Policy: Informing Policy: State Longitudinal Data State Longitudinal Data SystemsSystems
Jane Hannaway, DirectorJane Hannaway, DirectorThe Urban InstituteThe Urban Institute CALDERCALDERwww.caldercenter.org
State of U.S. EducationState of U.S. Education
½ of minority students graduate ½ of minority students graduate from high schoolfrom high school
4 grade level gap between white 4 grade level gap between white and minority students by 12and minority students by 12thth grade grade
15% of minorities earn BAs w/in 9 15% of minorities earn BAs w/in 9 years of 9years of 9thth grade grade
The The WILLWILL and the and the WAYWAY
The The WILLWILL– Left, Right, CenterLeft, Right, Center– Agreement on education crisisAgreement on education crisis– Strange bedfellowsStrange bedfellows
The The WAYWAY– Few, but growing, guidepostsFew, but growing, guideposts
Finding the Finding the WAYWAY with Evidence with Evidence-A New Day--A New Day-
Who has the evidence?Who has the evidence?– States have the makings of the evidenceStates have the makings of the evidence
Where are the makings?Where are the makings?– State administrative data systemsState administrative data systems
Why do states have it?Why do states have it?– Important effect of NCLBImportant effect of NCLB
Why important?Why important?– Address questions never before possibleAddress questions never before possible
Research Background: Research Background: What We KnowWhat We Know
Teachers matter- single most important Teachers matter- single most important schooling contributor to student outcomesschooling contributor to student outcomes
Wide variation in teacher effectiveness. Wide variation in teacher effectiveness. Some teachers are simply much better than Some teachers are simply much better than othersothers
Standard measures of teacher quality not Standard measures of teacher quality not much related to effectiveness, but directly much related to effectiveness, but directly related to spending.related to spending.
Research Background:Research Background:What We What We Don’tDon’t Know Know
What is it about What is it about teachers that matters?teachers that matters?
3 Research Probes3 Research Probes
Teacher MaldistributionTeacher Maldistribution
Teacher SelectionTeacher Selection
Teacher MobilityTeacher Mobility
Teacher Teacher Maldistribution 1Maldistribution 1 Comparison of VA of teachers in Comparison of VA of teachers in
high/ low poverty schoolshigh/ low poverty schools North Carolina and FloridaNorth Carolina and Florida FindingsFindings
– Low poverty - higher va, but not Low poverty - higher va, but not muchmuch
– High poverty – larger variation in High poverty – larger variation in schoolschool
Teacher Value-Added at Teacher Value-Added at Percentiles by School Poverty Percentiles by School Poverty Levels (North Carolina-Math)Levels (North Carolina-Math)
North Carolina- Elementary Math
-0.4000
-0.3000
-0.2000
-0.1000
0.0000
0.1000
0.2000
0.3000
10 25 50 75 90
Percentile
Teacher Performance Percentile
Val
ue-
add
ed s
core
0-70% FRL
70-100% FRL
Teacher Value-Added at Teacher Value-Added at Percentiles by School Poverty Percentiles by School Poverty Levels (Florida- Math)Levels (Florida- Math)
Florida- Elementary Math
-0.4000
-0.3000
-0.2000
-0.1000
0.0000
0.1000
0.2000
0.3000
10 25 50 75 90
Percentile
Teacher Performance Percentile
Val
ue-
add
ed s
core
0-70% FRL
70-100% FRL
Novice teachers are less effective Novice teachers are less effective than experienced teachers.than experienced teachers.
Returns to experience taper off 3-Returns to experience taper off 3-5 years.5 years.
Distribution of Value-Added of Distribution of Value-Added of Elementary Math Teachers in Elementary Math Teachers in High Poverty SchoolsHigh Poverty Schools
0.5
11
.52
Den
sity
-1 -.5 0 .5 1 1.5Value-added score
kernel = epanechnikov, bandwidth = 0.0885
>=70% Poverty SchoolsDistribution of Value-Add of Elementary Math Teachers
Solid line: Novice teachersDash line: Teachers with 1-2 years of experienceDotted line: Teachers with 3-5 years of experience
Distribution of Value-Added of Distribution of Value-Added of Elementary Math Teachers in Elementary Math Teachers in Lower Poverty SchoolsLower Poverty Schools
0.5
11
.52
Den
sity
-1 -.5 0 .5 1Value-added score
kernel = epanechnikov, bandwidth = 0.0698
0-70% Poverty SchoolsDistribution of Value-Add of Elementary Math Teachers
Solid line: Novice teachersDash line: Teachers with 1-2 years of experienceDotted line: Teachers with 3-5 years of experience
Teacher Teacher Maldistribution 2Maldistribution 2 New York CityNew York City
– Phasing out of emergency Phasing out of emergency certificationcertification
– Introduction of alternative route Introduction of alternative route teachersteachers
LAST Exam Failure Rate of LAST Exam Failure Rate of Elementary Teachers by Elementary Teachers by Poverty Quartile, 2000-2005Poverty Quartile, 2000-2005
10%
15%
20%
25%
30%
35%
40%
2000 2001 2002 2003 2004 2005
Per
cen
t o
f te
ach
ers
Low est quartile 2nd quartile 3rd quartile Highest quartile
LAST Exam Failure Rate of LAST Exam Failure Rate of New Teachers by Poverty New Teachers by Poverty Quartile, 2000-2005Quartile, 2000-2005
10%
15%
20%
25%
30%
35%
2000 2001 2002 2003 2004 2005
% o
f n
ew
te
ac
he
rs
Lowest quartile 2nd quartile 3rd quartile Highest quartile
Predicted Effectiveness For Predicted Effectiveness For Highest and Lowest Highest and Lowest Poverty SchoolsPoverty Schools
-0.06
-0.04
-0.02
0
0.02
0.04
0.06
0.08
0.1
Most Affluent Decile Poorest Decile Gap
2001
2005
Narrows by 25%
Can change predicted Can change predicted effectiveness by selection effectiveness by selection up-frontup-front
Mean VA Mean VA by by
QuintileQuintile(poor (poor
schools)schools)Passed Passed ExamExam
Not Not CertifieCertifie
dd
MatMath h SATSAT
VerbVerbal al
SATSAT
college college competitivenesscompetitiveness
MosMostt
SomSomee
LesLesss
NoNott
-0.068-0.068 0.460.46 0.730.73 355355 440440 0.040.04 0.070.07 0.550.550.30.3
55
-0.032-0.032 0.660.66 0.140.14 414414 467467 0.050.05 0.070.07 0.540.540.30.3
44
-0.010-0.010 0.780.78 0.080.08 423423 462462 0.090.09 0.130.13 0.440.440.30.3
44
0.0100.010 0.850.85 0.030.03 450450 470470 0.160.16 0.200.20 0.370.370.20.2
77
0.0450.045 0.910.91 0.010.01 512512 474474 0.250.25 0.250.25 0.350.350.10.1
55
Meaningful difference based only on attributes, though monitoring, development and selective retention also needed
Teacher SelectionTeacher Selection
Teach for AmericaTeach for America– North CarolinaNorth Carolina
– Secondary schoolSecondary school
– Mainly math and scienceMainly math and science
TFA Findings – high TFA Findings – high schoolschool
Student FE, Math subjects
TFA v. all TFA v. all othersothers
TFA v. in-TFA v. in-field non-field non-
TFATFA
TFA v. TFA v. traditional traditional
tracktrack
TFATFA 0.110.11 0.100.10 0.080.08
ExperienceExperience
3-5 yrs3-5 yrs 0.050.05 0.060.06 0.030.03
6-10 yrs6-10 yrs 0.050.05 0.060.06 0.020.02
> 10 yrs> 10 yrs 0.050.05 0.050.05 0.030.03
All TFA coefficients are significant at the .05 level.
Teacher MobilityTeacher Mobility
Mobility highest at most Mobility highest at most challenging schoolschallenging schools
The worst teachers are the first to The worst teachers are the first to leaveleave
General tendency to move to General tendency to move to more affluent schoolsmore affluent schools
Topic of the Day: Topic of the Day: Performance Performance
IncentivesIncentives Objective??Objective??
– Recruitment/ selectionRecruitment/ selection– Retention/ deselectionRetention/ deselection– Increase performance thru effortIncrease performance thru effort
IssuesIssues
How good are the measures?How good are the measures?
Individual vs school rewards?Individual vs school rewards?
Teachers without test scores?Teachers without test scores?
VA MeasuresVA Measures
ProblemsProblems– Year to year variabilityYear to year variability– Measurement errorMeasurement error– SortingSorting
How serious?How serious?– Less serious for policy researchLess serious for policy research– More serious for individual stakesMore serious for individual stakes
Predicting Predicting PerformancePerformance Using first 2 yrs of performance – top Using first 2 yrs of performance – top
to top/ bottom to bottom quintileto top/ bottom to bottom quintile
– Goldhaber and Hansen (NC): 46%/ 44%Goldhaber and Hansen (NC): 46%/ 44%– Koedel and Betts (SanDiego): 29%/ 35%Koedel and Betts (SanDiego): 29%/ 35%– Sass (Florida): 22-32%/ 24-24%Sass (Florida): 22-32%/ 24-24%
Policy ImplicationsPolicy Implications
Use VA freely for researchUse VA freely for research
Use VA carefully for individual teacher Use VA carefully for individual teacher judgmentsjudgments– Important informationImportant information– CorrorborationCorrorboration
More years are betterMore years are better– Move tenure decision out!Move tenure decision out!
Research QuestionsResearch Questions
Are teachers in high poverty schools more/ Are teachers in high poverty schools more/ less effective (value-added) than teachers in less effective (value-added) than teachers in lower poverty schools?lower poverty schools?
Do Do school factorsschool factors affect differences in the affect differences in the value-added of high poverty and lower value-added of high poverty and lower poverty teachers?poverty teachers?
Do Do teacher qualificationsteacher qualifications affect differences in affect differences in the value-added of high poverty and lower the value-added of high poverty and lower poverty teachers? poverty teachers?
DataData
Florida (2000/01- 2004/05)Florida (2000/01- 2004/05)– ElementaryElementary– Student achievement – FCAT-SSSStudent achievement – FCAT-SSS
Grades 3-10Grades 3-10– Teacher linksTeacher links
Assignment, certification, experience, educationAssignment, certification, experience, education
North Carolina (2000/1-2004/5)North Carolina (2000/1-2004/5)– ElementaryElementary– Student achievementStudent achievement
EOG – grades 3-8EOG – grades 3-8 EOC – secondary subjectsEOC – secondary subjects
– Teacher linked through proctor and verificationTeacher linked through proctor and verification Assignment, certification, experience, educationAssignment, certification, experience, education
DefinitionsDefinitions
High povertyHigh poverty elementary elementary schools (>70% FRL students) schools (>70% FRL students)
Lower povertyLower poverty elementary elementary schools (<70% FRL students) schools (<70% FRL students)
Very low povertyVery low poverty schools schools (<30% FRL students). (<30% FRL students).
NC Student-Teacher LinkNC Student-Teacher Link
EOC student-level records
Aggregate to EOC test classrooms by school,
year, subject, proctor id
Instructional Classroom records including school, year,
subject, a teacher id.
Decision Rules
Match if teacher and proctor id identical and fit statistic < 1.5.
Sample RestrictionsSample Restrictions
Exclude charter schoolsExclude charter schools
Exclude schools that switch high poverty Exclude schools that switch high poverty to lower poverty statusto lower poverty status
Only classrooms w/ 10-40 studentsOnly classrooms w/ 10-40 students
Only self-contained elementary Only self-contained elementary classroomsclassrooms
Analytic SampleAnalytic Sample
0-30% 0-30% FRLFRL
30-70% 30-70% FRLFRL
70-100% 70-100% FRLFRL TotalTotal
FloridaFlorida(Elementary School (Elementary School Level)Level)
3, 0843, 084 6, 9756, 975 5,2325,232 14, 05214, 052
North CarolinaNorth Carolina(Elementary School (Elementary School Level)Level)
2,2072,207 5, 9455, 945 2, 3162, 316 9,2129,212
Note: We focus on elementary schools, grades 3-5 where poverty information is most reliable. We exclude teachers from charter schools and we exclude classrooms with <10 students or >40 students in our samples.
Methodological ChallengesMethodological Challenges
Non-random sorting of teachers and Non-random sorting of teachers and studentsstudents
Distinguishing teacher and school effectsDistinguishing teacher and school effects
Precision in Teacher Effects EstimatesPrecision in Teacher Effects Estimates
Sources of Teacher Effectiveness Sources of Teacher Effectiveness DifferentialsDifferentials
Descriptive Findings:Descriptive Findings:Elementary Student DemographicsElementary Student Demographics
Descriptive Findings: Descriptive Findings: Student PerformanceStudent Performance
FloridaFlorida North CarolinaNorth Carolina
0-30% 0-30% FRLFRL
30-70% 30-70% FRLFRL
70-100% 70-100% FRLFRL
0-30% 0-30% FRLFRL
30-70% 30-70% FRLFRL
70-100% 70-100% FRLFRL
Level Scores:Level Scores:
MathMath 0.49**0.49** 0.25**0.25** -0.16-0.16 0.43**0.43** 0.15**0.15** -0.32-0.32
ReadingReading 0.50**0.50** 0.26**0.26** -0.18-0.18 0.39**0.39** 0.14**0.14** -0.34-0.34
Gain Scores:Gain Scores:
MathMath -0.02**-0.02** -0.01**-0.01** 0.020.02 0.02*0.02* 0.010.01 0.020.02
ReadingReading -0.01-0.01 -0.01*-0.01* -0.01-0.01 0.02**0.02** 0.01**0.01** 0.000.00
* Differences between the given estimate and the corresponding estimates for schools with 70-100% FRL students significant at ≤ 5% and ** differences significant at ≤ 1%.
Descriptive Findings:Descriptive Findings:Teacher ExperienceTeacher Experience
Descriptive Findings:Descriptive Findings:Teacher QualificationsTeacher Qualifications
Distribution of Value-Added of Distribution of Value-Added of Elementary Reading Teachers in Elementary Reading Teachers in Lower Poverty SchoolsLower Poverty Schools
0.5
11
.52
2.5
Den
sity
-1 -.5 0 .5 1 1.5Value-added score
kernel = epanechnikov, bandwidth = 0.0517
0-70% Poverty SchoolsDistribution of Value-Add of Elementary Reading Teachers
Solid line: Novice teachersDash line: Teachers with 1-2 years of experienceDotted line: Teachers with 3-5 years of experience
Distribution of Value-Added of Distribution of Value-Added of Elementary Reading Teachers in Elementary Reading Teachers in High Poverty SchoolsHigh Poverty Schools
0.5
11
.52
2.5
Den
sity
-1 -.5 0 .5 1Value-added score
kernel = epanechnikov, bandwidth = 0.0709
>=70% Poverty SchoolsDistribution of Value-Add of Elementary Reading Teachers
Solid line: Novice teachersDash line: Teachers with 1-2 years of experienceDotted line: Teachers with 3-5 years of experience
FloridaFlorida North CarolinaNorth Carolina
Difference Difference DifferenceDifference
Math:Math:
FE Estimates FE Estimates -- ++
Student Covariate EstimatesStudent Covariate Estimates -- ++
Reading:Reading:
FE Estimates FE Estimates ++ ++
Student Covariate EstimatesStudent Covariate Estimates ++ ++
Differences in Estimates of Differences in Estimates of Teacher Value-AddedTeacher Value-Added
Magnitude of Differences in Magnitude of Differences in Value Added EstimatesValue Added Estimates
Differences in Standard Differences in Standard Deviations of Value-AddedDeviations of Value-Added
FloridaFlorida North CarolinaNorth Carolina
Difference Difference DifferenceDifference
Math:Math:
FE Estimates FE Estimates -- --
Student Covariate EstimatesStudent Covariate Estimates -- --
Reading:Reading:
FE Estimates FE Estimates -- --
Student Covariate EstimatesStudent Covariate Estimates -- --
Differences between Lower- and Differences between Lower- and High-Poverty by Percentile of High-Poverty by Percentile of Teacher Value AddedTeacher Value Added
Teacher Value-Added at Teacher Value-Added at Percentiles by School Poverty Percentiles by School Poverty Levels (North Carolina- Reading)Levels (North Carolina- Reading)
North Carolina- Elementary Reading
-0.4000
-0.3000
-0.2000
-0.1000
0.0000
0.1000
0.2000
0.3000
10 25 50 75 90
Percentile
Teacher Performance Percentile
Val
ue-
add
ed s
core
0-70% FRL
70-100% FRL
Teacher Value-Added at Teacher Value-Added at Percentiles by School Poverty Percentiles by School Poverty Levels (Florida- Reading)Levels (Florida- Reading)
Florida- Elementary Reading
-0.4000
-0.3000
-0.2000
-0.1000
0.0000
0.1000
0.2000
0.3000
10 25 50 75 90
Percentile
Teacher Performance Percentile
Val
ue-
add
ed s
core
0-70% FRL
70-100% FRL
Correlation of Teacher Correlation of Teacher Qualifications and Value-Qualifications and Value-AddedAdded
-0.0200
0.0000
0.0200
0.0400
0.0600
0.0800
0.1000
1-2
yrs
3-5
yrs
6-12
13-2
0
21-2
7
28+
yrs
Gra
d.
Reg
.
1-2
yrs
3-5
yrs
6-12
13-2
0
21-2
7
28+
yrs
Gra
d.
Reg
.
1-2
yrs
3-5
yrs
6-12
13-2
0
21-2
7
28+
yrs
Gra
d.
Reg
.
1-2
yrs
3-5
yrs
6-12
13-2
0
21-2
7
28+
yrs
Gra
d.
Reg
.
Florida North Carolina Florida North Carolina
Math Reading
Co
effi
cien
t
0-70% FRL
70-100 % FRL
Sources of Difference in Teacher Value-Sources of Difference in Teacher Value-Added Between High-Poverty and Added Between High-Poverty and Lower-Poverty Elementary SchoolsLower-Poverty Elementary Schools
-0.005
0.005
0.015
0.025
0.035
Florida NorthCarolina
Florida NorthCarolina
Math Reading
Dif
fere
nc
e in
Te
ac
he
r V
alu
e A
dd
ed
Difference due to differencesin characteristics
Difference due to differencesin marginal effect ofcharacteristics
Difference due to interactionof characteristics andmarginal effects
Sensitivity AnalysisSensitivity Analysis
School EffectSchool Effect
Empirical Bayes AdjustmentEmpirical Bayes Adjustment
ConclusionsConclusions
Teachers in high poverty schools, on Teachers in high poverty schools, on average, are less effective than average, are less effective than teachers in lower poverty schools.teachers in lower poverty schools.– Changing schools (high poverty/lower Changing schools (high poverty/lower
poverty) does not affect teacher poverty) does not affect teacher effectivenesseffectiveness
There is greater teacher variation There is greater teacher variation within high poverty schools than within high poverty schools than within lower poverty schools.within lower poverty schools.
Conclusions (con’t)Conclusions (con’t)
Differences in teachers in High Differences in teachers in High Poverty and Lower Poverty Poverty and Lower Poverty schools:schools:– only weakly related to teacher qualificationsonly weakly related to teacher qualifications
– more strongly related to marginal effect of more strongly related to marginal effect of qualifications (experience)qualifications (experience)
– not explained by school poverty levelnot explained by school poverty level
Study LimitationsStudy Limitations
Issues with value-added measuresIssues with value-added measures– separating current teacher contributions separating current teacher contributions
from other current contributionsfrom other current contributions E.g., current family circumstancesE.g., current family circumstances
- - dynamic sortingdynamic sorting sorting on time variant characteristicssorting on time variant characteristics
– Instability of measuresInstability of measures E.g., measurement error, motivationE.g., measurement error, motivation