Striving to Link Teacher and Student Outcomes: Results from an Analysis of

Whole-school Interventions

Kelly Feighan, Elena Kirtcheva, and Eric Kucharik

Research for Better Schools, Philadelphia, PA

American Evaluation Association Annual Meeting,

November 12, 2009 in Orlando, Florida

1

Study Purpose

Investigate which variables best explain student reading outcomes following teacher professional development

Explore the contextual reasons that help explain why no intervention “impact” was detected

Inform educational policy and improve rigor of educational research

2

Project Background

Federal Striving Readers program aimed at improving pedagogy and student achievementSchools were matched in pairs and then randomly

assigned to the treatment or control condition

Professional Development: four-semester course, onsite literacy coaching, leadership seminar, and curricular material

Developer’s hypothesis: integrating literacy strategies in content areas will yield student gains

3

Factors Affecting Student Learning

Student-level: SES, socio-demographic variables, family background, early development (Barton & Coley, 2009)

Teacher/classroom-level: expectations, preparation, experience, class size (Cohen, McCabe, Mitchelli, and Pickeral, 2009)

School-level: school climate - safety, student- adult and peer relationships, curriculum rigor (Cohen, McCabe, Mitchelli, and Pickeral, 2009)

4

Study Participants

• 30 ELA teachers taught at eight schools– 16 taught at intervention schools

– 14 taught at comparison schools

• 2,114 students linked to these teachers– state assessment reading scores (N = 2,064)

– ITBS scale reading scores (N = 1741)

5

Methodology

Quantitative data sources:RBS teacher surveySchool district school climate surveyDepartment of Education teacher HQT

statistics and student discipline dataStudents’ scores on state assessment and

ITBS

6

Methodology

Qualitative data sources:Observations

56 classrooms (Year 1)48 classrooms (fall of Year 2)10 paired observations (spring of Year 2)

Interviews 8 principals and 19 school improvement team

members in Years 1 and 2Focus groups: seven groups with 62 teachers

7

Research Hypotheses

Exposure to professional development participants will yield gains in reading achievement

Including contextual variables in impact analysis will increase explanatory power of results

8

Quantitative Analysis

Used Hierarchical Linear Modeling (HLM) to predict student performance based on student-, teacher-, and school-level characteristics

Fully unconditional model represents how variation in an outcome measure is allocated across the three different levels

9

Variables Included in the HLM

10

Outcome Variables: Reading ScoresRandom Effect Variance

Componentdf Chi-

squareP Value Variance

Decomposition (% by level)

State Test

Students 823.47 81.18

Teachers 178.82 22 322.32 0.000 17.63

Schools 12.14 7 9.50 0.218 1.19

ITBS

Students 519.68 77.8

Teachers 147.58 22 361.18 0.000 22.0

Schools 1.08 7 7.48 0.380 0.2

11

Student-Level Variation

Across multiple model specifications, the only predictors with statistical significance were the student’s

Pre-test scoreGender ELL status

Modeling teacher-level factors produced no significant results

12

Classroom Observation Results

No baseline differences in levels of engagement & cognitive demand, or in instructional strategies

Cognitive demand level of lessons was low in Year 2, irrespective of research condition

Intervention teachers tended to use more literacy strategies than comparison teachers in Year 238.5% of intervention teachers used multiple literacy

strategies vs 18.2% of comparison teachers

13

Why We May Not Find Impact

Low cognitive demand of lessons

Counterfactual situations may “water down” the treatment’s effect

Low implementation fidelity

Limitations in outcomes measures (just say measurement error)

14

Implications for Further Research

Better understanding of the relationship between a school-level intervention and its

potential to affect student achievement

Correlates of student achievement

Why an intervention that did not show impact may nevertheless be of value

15