attracting, preparing and retaining attracting, preparing ...data – programs, spring & summer...

Attracting, Preparing and Retaining Effective Teachers

Hamp LankfordNCSL Seminar on Linking Research and

Policy to Support Effective Teaching

Attracting, Preparing and Retaining Effective Teachers

Research Partners:Don Boyd and Jim Wyckoff, University at AlbanyPam Grossman and Susanna Loeb, Stanford UniversityJonah Rockoff, Columbia University

www.teacherpolicyresearch.org

Overview of Presentation

How can we identify effective teachers?

What is the role of licensure?

How can we prepare teachers to be more effective?

Examining pathways into teachingTeacher preparation and student achievement

Teacher Sorting in NYS Elementary Schools (Math 2002)

Percent of Students at Level 1 Percent of teachers Highest Quartile Lowest Quartile

0-1 years prior 21.3 12.3teaching experience

Failed general knowledge 31.0 10.1or LAST exam

BA from a least 24.0 11.4competitive college

Improving Teaching, Improving Student Outcomes

Student Outcomes

School

teacher quality

leaders

student ability &motivation

facilities

class size

District policies

Teacher preparationpathway

Student and Environment

family

Staterequirements

learningenvironment

neighborhood

teacher certification

peers

Prospectiveteachers

other teachingpolicies

teacher education

Teacher Workforce

salaryhiring

program structure

subject specificfield experiences

preparation fordiverse learners

professionaldevelopment

academic abilityprior experience

teaching

mentoringinduction

retention

Policy Challenge

How do public schools:

Attract potentially excellent teachers to teaching, especially in traditionally difficult-to-staff schools?

Provide the skills and experiences that help potentially excellent teachers develop into excellent teachers in those schools?

Retain strong teachers, especially in traditionally difficult-to-staff schools?

Teacher Qualifications and Student OutcomesIn general, only limited evidence that specific

teacher qualifications make a difference

For example, advanced degrees do not predict greater student achievement,

Qualifications such as certification exam scores have modest effects on student achievement and

Teachers from alternative certification routes perform about as well as those from traditional preparation programs.

However, some qualifications are important:Teacher experience over the first 3-5 years

0

0.04

0.08

0.12

0.16

1 2 3 4 5 6-10 11-15 16-20 > 20Years of NYC Experience

Pro

port

ion

of S

tand

ard

Devi

atio

n

Grade 4-5

Grade 6-8

In addition …

Teachers being certified appears to improve achievement (difficult to measure its effect as we don’t observe a system that does not require certification).

Taken as a group, observable teacher qualifications do appear relevant in distinguishing among more and less effective teachers

Teacher Qualifications and Student Achievement

Research question: How has the gap in teacher qualifications between poor and non-poor schools in NYC changed between 2000 and 2005 and has that affected the gap in student achievement?

Teacher qualifications: certification exam scores, SAT scores, undergraduate college ranking, experience, certification

LAST Exam Failure Rate of Elementary Teachers by Poverty Quartile of School’s Students,

2000-2005

10%

15%

20%

25%

30%

35%

40%

2000 2001 2002 2003 2004 2005

Perc

ent o

f tea

cher

s

Low est quartile 2nd quartile 3rd quartile Highest quartile

LAST Exam Failure Rate of New Teachers by Poverty Quartile of School’s Students, 2000-2005

10%

15%

20%

25%

30%

35%

2000 2001 2002 2003 2004 2005

% o

f new

teac

hers

Lowest quartile 2nd quartile 3rd quartile Highest quartile

Number of New Teachers by Pathway, 2000-2005

0

1,000

2,000

3,000

4,000

5,000

6,000

2000 2001 2002 2003 2004 2005

Num

ber o

f new

teac

hers

College recommeded and other Teaching Fellows and TFA Temporary license

Average Certification Exam Scores on First Taking (2004; Passing= 220,SD=~30)

246

247

267

276

242

200 210 220 230 240 250 260 270 280 290 300

CollegeRecommended

IndividualEvaluation

Teaching Fellow

Teach For America

Temp license

Policies Contributing to Change

In 2000 the NYS Regents created alternative certification routes

In 2000 the NYC Department of Education created its first cohort of Teaching Fellows

Effective September 2003, NYS Regents eliminated temporary licenses for uncertified teachers with only very limited exceptions

Between 2000 and 2003 starting salaries in NYC increased from $33,186 to $39,000

Using Student Achievement Tests to Examine Policies

Individual Student gainsAit - Ai(t-1)= student, class, teacher, school

Typically include about .5 to 1.5 million student tests linked to their teachers Statistical controls to make sure we isolate effects of teacherExamine lots of different estimates to make sure conclusions are consistent

05

1015

Pro

porti

on o

f Tea

cher

s

-.15 -.1 -.05 0 .05 .1 .15 .2Average Impact on Students in Standard Deviations

Rich 2001 Poor 2001Rich 2005 Poor 2005

Changes in Grades 4 & 5 Math Attributable to Teacher Qualifications, Rich and Poor Deciles 2001 & 2005

Effective and Ineffective Teachers in High Poverty Schools

Student VA AchievementLow High

Characteristics of Effective and Ineffective Teachers in High Poverty Schools (bottom quartile, math in grades 4 & 5)

VA Quintile Mean VA

Years Experience

Not Certified

Cert Exam Score

Math SAT

Verbal SAT

1 -0.11 1.78 0.66 236 421 4732 -0.04 4.67 0.33 241 434 4673 -0.01 6.85 0.06 243 434 4684 0.02 6.65 0.02 247 439 4615 0.06 5.87 0.01 252 488 462

Range 0.17 4.09 -0.66 16 67 -11

The Role of Teacher Preparation and Licensure

Preparation and licensure can improve teaching effectiveness if:They distinguish among more and less able teachersThey improve the knowledge and skills of potential teachersThey do not adversely affect the supply of potential teachers, and ifLocal hiring authorities are motivated to hire the most effective teachers

The Role of Teacher Licensure

Several points:Licensure vs. credentialsNo certification vs. out-of-field teachingDifficult to evaluate how licensure affects student outcomes as we do not observe a system without licensure.

The Role of Teacher Preparation

Emerging evidence on whether teacher preparation improves student outcomes

Substantial literature on lack of effect of graduate educationAlternative vs. traditional preparationPreparation coursework linked to student value addedParticular aspects of preparation linked to student value added

Examining Pathways Into Teaching

DEFINING PATHWAY - The route teachers follow into their first teaching job.

College RecommendedIndividual Evaluation (transcript review)NYC Teaching FellowsTeach for America Temporary License (uncertified)Other (i.e., reciprocity, other Transitional B programs)

Effects on Student Achievement by Pathway by Experience, Math, Grades 4-5 (Coll. Rec. 1 yr =0;)

-0.08

-0.04

0

0.04

0.08

0.12

1 2 3

Teacher Experience

Achi

evem

ent s

td. d

ev.

Coll. Rec.FellowsTFATemp. Lic.

*Caution – small sample size for third-year TFA makes interpretation of results tricky

Effects on Student Achievement by Pathway by Experience, Math, Grades 6-8 (Coll. Rec. 1 yr =0;)

*Caution – small sample size for third-year TFA makes interpretation of results tricky

-0.02

0

0.02

0.04

0.06

0.08

0.1

0.12

1 2 3

T eacher Experience

Ach

ieve

men

t Std

Dev

.

Coll RecFellowsTFATem p Lic

Grades 4-8 Teacher Attrition Following the 1st

Four Years of Teaching, 1999-2003 Cohorts

0%10%20%30%40%50%60%70%80%90%

CollegeRec.

Indiv.Eval.

Fellows TFA Temp

1st yr2nd yr3rd yr4th yr

Adjusted for grade School and Year

Teacher Preparation and Student Achievement – Background

Large literature on Teacher Education

Very little empirical research exploring the link between teacher preparation and student learning

The following is a step in that directionFocusing on childhood educationIn one large urban district, New York City

Questions – For Childhood Education…

What is the distribution of the average value-added of teachers from different preparation programs?

How are features of preparation programs associated with the value teachers add to student achievement gains in math and ELA?

How are teachers’ reported experiences in their preparation associated their value added?

Methods – 3 analysesPrograms /InstitutionsAijst = β0+β1Aijs(t-1)+Xitβ2+Cijstβ3+Tjstβ4+πj+ωs+εijst

Achievement as a function of program/institution (π), plus : prior achievement, student characteristicsclassroom characteristics teacher characteristics (in some specifications)school fixed-effects (in most specifications)

Program Featuressubstitute Pjstβ5 for πj cluster at program level

Reports of Experiencessubstitute Rjstβ6 for πj cluster at teacher level

Data – AdministrativeStudent data

demographic fileexam file: grades 3-8, math and ELAlinked to teachers through homeroom (elementary) or section (middle)

Teacher dataTeacher experience: transaction-level data from the NYCDOE Division of Human ResourcesTeacher demographics: range of data filesTest performance: NYS Teacher Certification Exam History File (EHF). Pathway: teacher certification applications plus separate data files for individuals who participated in Teach For America or the Teaching Fellows ProgramProgram for College-Recommended: NYSED’sprogram completers data files

Data – Programs, spring & summer of 200418 institutions: 26 traditional programs + 4 TF + 1 TFADocument collection, interviews with director and director of field experiences, surveys of math and ELA methods facultyMany measures but here focus mainly on link to practice. Program data are not ideal for this.

whether or not the program required a capstone project(50%)a composite measure of the extent that the program maintains oversight over student teaching experiences

requires a minimum number of years of teaching experience for its cooperating teachers (32%)program picks the cooperating teacher (42%) program supervisor observes their participants a minimum of five times during student teaching (27%)

for comparison math and English content courserequirements (# courses) and the proportion tenure-line faculty (.45)

Data – Survey Data, spring of 2005

All first year teachers in NYC, response rate 70%Survey available at www.teacherpolicyresearch.orgVariables

extent to which preparation included links to practice;opportunities to listen to an individual child read aloud for the purpose of assessing his/her reading achievement; …Plan a guided reading lesson, and …Study or analyze student math work (each 5-point scale).

opportunities to study the New York City curriculum;whether or not the teacher had student teaching experiencesunder supervision of the teacher-of-record in the classroom; the congruence between their student-teaching placement and their current job assignment in terms of subject matter or grade level;

Data – Survey Data, spring of 2005

Additional measures of preparation as controls and comparisons

opportunities to learn about teaching math; …to learn about teaching ELA; …to learn about handling student misbehavior; and …to learn about teaching English language learners.

Programs: Institution Effects, Math (x-axis) and ELA (y-axis), First-Year Teachers 2001-2006

(40 or more teachers with value-added estimates)

-0.08

-0.06

-0.04

-0.02

0.00

0.02

0.04

0.06

0.08

0.10

-0.06 -0.04 -0.02 0.00 0.02 0.04 0.06 0.08 0.10

Program Features – Practice Consistently Positive for First Year Teachers (2005-2006)

Capstone projectMath: 0.12*ELA: 0.10*

Oversight of student teachingMath: 0.12**ELA: 0.10**

Neither significant in year 2* Significant at 0.05; ** at 0.01

Program Features – Other Measures Less Consistent, Effect of Content Requirements small

Math Content CoursesMath: 0.02**(01-06), 0.01(05&06), 0.02*(2nd year)ELA: 0.00 for all samples

ELA Content Courses Math: 0.00 (01-06), -0.03**(05&06), 0.01(2nd year) ELA: -0.01*(01-06), 0.00(05&06), 0.01*(2nd year)

Proportion Tenure-lineMath: 0.12*(01-06), .06 (05&06), .09 (2nd year)

Survey Results – Practice for MathMath:

Opportunities in Practice Variable: (positive for all samples)

0.07** - 0.11** (std dev. 0.51)

NY Curriculum:0.05** - 0.07** (std dev 0.85)

Congruence of Field Experience and JobFirst year: .02** - .04** (std dev 0.80)Nothing second year

Survey Results –Practice not as strong for ELA

ELA:Opportunities in Practice Variable: nothing

NY Curriculum: only significant in college recommended sample

0.04 - 0.07 (std dev 0.85)

Congruence of Student Teaching & Job–not significant except in college recommended

-0.07 to - 0.12 in full and elementary

Survey Results – other variables

Very little elseOpportunities to learn teaching methods in Math or ELA not significantly related and very small point estimatesOpportunities to learn about handling student misbehavior is positive and significant (0.03) in the full model for ELA but no other modelsOpportunities to learn about teaching English learners never significantEmphasis on basic skills negative, sometimes significant for ELA

ConclusionsVariation across programs

Opportunities linked to practice appear to help teachers in their first year

Caveats – only a first stepBig district but small number of programs and teachers within schoolsMay be missing important features, or not measuring them well; developing instrumentsValue-added analysis may eliminate the most important variation (e.g. between schools or very weak preparation)Tests themselves may not proxy well for full range of learning that we care about

Summary: Improving TeachingRecruiting effective teachers

Increasing the supply of potentially effective teachersProviding evidenced-based preparation Improved screening by hiring authorities

Improving the skills of in-service teachersRetain effective teachersRemove ineffective teachers

Portfolio of evaluation methods

Papers, policy briefs, surveys, and other documents available at:

www.teacherpolicyresearch.org

Teacher Labor MarketsDemand for effective teachers

student demographicspolicy initiatives costs of hiring teachersteacher retention

Supply of effective teachersrelative working conditionsrelative compensationcosts of entering teachingpreparation and licensure requirements

Research, Evidence and PolicyEvidence of teacher effectiveness can come in a variety of forms, but ultimately should be linked to outcomes for studentsResearch that links characteristics of teachers to outcomes of their students is in the very early stages.Employing student achievement test results tied to teachers for teacher, administrator or school accountability places enormous pressure on assessments, data systems and statistical methods.States should approach the use of research or accountability systems built on value-added results cautiously as doing so could well have unintended consequences.

Entering NYC Teachers by Pathway: Teaching Fellows Replace Temp License

01,000

2,0003,0004,0005,000

6,0007,0008,000

9,00010,000

1997 1998 1999 2000 2001 2002 2003 2004 2005

Num

ber o

f Tea

cher

s

College Recommended Individual Evaluation Teaching FellowTeach for America Temporary License Other

Average Certification Exam Scores First Taking: (2004; Passing= 220,SD=~30)

246

247

267

276

242

200 210 220 230 240 250 260 270 280 290 300

CollegeRecommended

Individual Evaluation

Teaching Fellow

Teach For America

Temp license

Programs: Effects in Math with no controls (x-axis) and controls (y-axis) for teacher characteristics (y-axis)

-0.10

-0.05

0.00

0.05

0.10

0.15

-0.10 -0.05 0.00 0.05 0.10 0.15

Surveys and related papers:www.teacherpolicyresearch.org

Program Analysis Implications

Variation across institutions.08 > 1st year vs. 2nd year teacher

Variation approximately the same in math and ELA

Positive correlation between math and ELA average value-added (0.65)

Not much difference in effects with and without basic teacher controls

age, gender, race, certification exam scores

Additional Slides

Proportion of Teachers with Fewer than Three Years Experience, NYC Elementary Schools, by

Poverty Quartile of School’s Students, 2000-2005

15%

20%

25%

2000 2001 2002 2003 2004 2005

Perc

ent o

f tea

cher

s

Lowest quartile 2nd quartile 3rd quartile Highest quartile

Research Questions & Samples

Institution and program value-added2004-05 & 2005-06 1st year teachers2000-01 through 2005-06 1st year teachers2000-01 through 2005-06 2nd year teachers

Program features: same as aboveSelf-reported preparation experiences

2004-05 cohort in 1st & 2nd yearSome runs with college recommended, TF and TFA

and some only college recommended; Include random effects rather than school fixed

effects

Program Features: Math, First-Year 2001-06

0.54630.54780.54810.54450.5448R-squared2523224486289803159431594Observations

(0.0075)0.0324***Oversight

(0.0503)0.1184**Percent Tenure

(0.0159)0.0410**Capstone project

(0.0050)-0.0026ELA courses

(0.0062)0.0239***Math courses

Program Features: Math, First-Year 2005&06


(0.0345)0.1240***Oversight

(0.1242)0.0614Percent Tenure

(0.0545)0.1216**Capstone project

(0.0085)-0.0272***ELA courses

(0.0174)0.0098Math courses

Program Features: Math Second-Year 2001-06


(0.0125)-0.0145Oversight


(0.0221)-0.0077Capstone project

(0.0056)0.0087ELA courses

(0.0091)0.0225**Math courses

Program Features: ELA for First-Year 2001-2006


(0.0073)0.0122Oversight


(0.0112)0.0496***Capstone project

(0.0039)-0.0091**ELA courses

(0.0084)-0.0034Math courses

Program Features: ELA for First-Year 2005&2006


(0.0387)0.1038**Oversight

(0.0874)-0.0478Percent Tenure

(0.0501)0.1019*Capstone project

(0.0096)-0.0060ELA courses


Program Features: ELA for Second-Year 2001-2006


(0.0138)0.0022Oversight


(0.0178)-0.0271Capstone project

(0.0051)0.0113**ELA courses


Survey - Math

12446Observations0.5442R-squared-0.044No Student Teach x 20060.008misbehavior x 2006-0.02No Student Teaching0.011Exp misbehavior-0.051*FExp: Coherence x 20060.002Basic Skills x 20060.037FExp: Coherence with job-0.032Basic Skills-0.034Math x 2006-0.03NY Curriculum x 2006-0.002Math0.053***NY Curriculum-0.042Exp to teach ELs x 2006-0.058Practice x 20060.011Exp to teach ELs0.106***Practice

Survey - ELA

12612 Observations0.4923 R-squared0.062 No Student Teach x 2006-0.025 misbehavior x 2006

-0.106 **No Student Teaching0.034*Exp misbehavior0.033 FExp: Coherence x 2006-0.046 Basic Skills x 2006-0.012 FExp: Coherence with job-0.047 Basic Skills-0.003 ELA x 2006-0.008 NY Curriculum x 2006-0.004 ELA0.005 NY Curriculum-0.016 Exp to teach ELs x 2006-0.042 Practice x 2006-0.010Exp to teach ELs0.033 Practice

Opportunities to Learn Math Methods

alpha = 0.97In your teacher preparation program, prior to September 2004, how much opportunity did you have to do the following (5-point scale)?

learn typical difficulties students have with place value; learn typical difficulties students have with fractions; use representations (e.g., geometric representation, graphs, number lines) to show explicitly why a procedure works; prove that a solution is valid or that a method works for all similar cases; study, critique, or adapt math curriculum materials; study or analyze student math work; design math lessons; learn how to facilitate math learning for students in small groups; adapt math lessons for students with diverse needs and learning styles; And practice what you learned about teaching math in your teacher preparation program in your field experience.

Opportunities to Learn ELA Methods

alpha = 0.96In your teacher preparation program, prior to September 2004, how much opportunity did you have to do the following (5-point scale)?

learn about characteristics of emergent readers; learn ways to teach students meta-cognitive strategies for monitoring comprehension; learn ways to teach decoding skills; learn ways to encourage phonemic awareness; learn ways to build student interest and motivation to read; learn how to help students make predictions to improve comprehension; learn how to support older students who are learning to read; learn ways to organize classrooms for students of different reading ability; study, critique, or adapt student curriculum materials; learn how to activate students’ prior knowledge; listen to an individual child read aloud for the purpose of assessing his/her reading achievement; plan a guided reading lesson; discuss methods for using student reading assessment results to improve your teaching; and practice what you learned about teaching reading in your field experiences.

attracting, preparing and retaining attracting, preparing ...data – programs, spring & summer...

Documents