Evaluating a Literacy Curriculum for Adolescents:

Results from Three Sites of the First Year of Striving Readers

Eastern Evaluation Research Society ConferenceApril 14, 2008 • Galloway, New Jersey

Striving Readers—Overview

Goals of the Striving Readers program Raise student achievement in middle and high

schools by improving the literacy skills of struggling adolescent readers

Help build a strong, scientific research base around specific strategies that improve adolescent literacy skills.

Eight sites around the country Three sites discussed in this presentation

5-Year grant period (2006–2011), assuming continuation funding

READ 180 Classroom Model

READ 180 Logic Model

The Memphis Striving Readers Project:Year 1 READ 180 Findings

Deb Coffey

Research for Better Schools

Motivation behind MSRP

Memphis is one of the cities with the highest educational need in the U.S.

Motivation behind MSRP

MCS middle schools compared with schools nationwide

Motivation behind MSRP Memphis City Schools is 21st largest K–12

district in US (>116,000 students)

Over 95% of MCS’ 196 schools are Title I schools 71% of MCS students qualify for free or reduced-

price lunch 87% of MCS students are African American; 9%

are white; 4% are “other” 71.5% of students in grades 6–8 scored below the

50th percentile on the Reading/Language Arts portion of the Tenn. state assessment (TCAP)

MSRP Overview

Targeted intervention: READ 180 Focus of this presentation Participants (students) randomly selected from pool of

eligible students, i.e., struggling readers

Schoolwide intervention: Memphis Content Literacy Academy Four schools (of eight) randomly selected in matched-

pairs design Teachers participate in intensive professional

development program

Overall MSRP Goals — to determine

1. The effects of MCLA on core subject teachers’ knowledge and use of scientifically based reading strategies/methods

2. The separate and combined effects of MCLA and READ 180 on students’ reading achievement levels, especially students who are identified as struggling readers

3. The separate and combined effects of MCLA and READ 180 on students’ achievement in core subjects, especially students who are identified as struggling readers

Study Design

Evaluate student outcomes using an experimental design based on randomly assigning eligible students to treatment and control conditions within participating schools

Student outcomes include reading achievement (ITBS) and state assessment (TCAP) results in core content areas

Analytic Approach

Cross-sectional ITT analyses of reading and core content area achievement

Two-level HLM using spring ITBS and TCAP scores as a function of student and school variables

Students Enrolled in READ 180

Experiences of READ 180 students and control condition students

Variables included in impact analysis Independent

READ 180 Participation

Dependent Spring 2007 ITBS: Total Reading

ComprehensionVocabulary

Spring 2007 TCAP:Reading/LAMathematicsScienceSocial Studies

Covariates included in impact analysis

Fall 06 ITBS: Total Reading

ComprehensionVocabulary

Fall 06 TCAP: Reading/LAMathematicsScienceSocial Studies

Free or Reduced Price Lunch

Grade

Gender

African-American / Hispanic

English Language Learner

Percentage Female

Percentage African-American

Percentage Special Ed

Percentage FRL

Percentage ELL

School Enrollment

READ 180 Impacts on Students (Y1)

Conclusions

No significant Year One student impact Late startup (Most) students will receive two years of intervention

Planned Future Analyses: Exploratory analyses of relationships between amount

of READ 180 instruction and effects on student outcomes

Implementation: Changes

First day of teacher training divided into two groups (novices and experienced)

Follow-up teacher training was 2 days (instead of 1)

Year One training for administrators and implementation support were not provided

Classroom substitution of “boom boxes” for personal CD players

Implementation: Cross-Site Variations

Differential school-level participation in professional development

Special education students not assigned READ 180 at some schools (legal concerns)

Adherence to READ 180 model

Implementation: Barriers

Equipment delays DOE prohibition on evaluator sharing

implementation findings with MCS Teacher contracts preventing MCS from

requiring attendance of meetings held after hours

Secondary/Exploratory Analyses

What did students actually experience? Was class “on model”? Were students present?

Sources of data Classroom observations

6 annually: 3 by RBS, 2 by MCS, 1 by developer

Data produced inside READ 180 program District records Teacher surveys (RBS) Student surveys (MCS—element of READ 180

Secondary/Exploratory Analyses:Possible inclusion in addl. HLM Did classes follow the overall READ 180 model?

Number of students in class Length of class “Rotations” Use of READ 180 materials

How much READ 180 did students experience? Student absences Student engagement Time on task READ 180 quizzes

Secondary/Exploratory Analyses: Background information What was level of teacher professional

development for READ 180? Did teachers attend PD sessions? What was teachers’ level of satisfaction with PD?

How do students describe their experiences with READ 180?

Did teachers report systemic challenges that prevented adherence to READ 180 model?

Clarifying Questions?

END

READ 180 Training Responsibilities

Provided by Developer 2 full days of teacher and

coach PD 7 two-hour network

meetings (with PD) for teachers & coaches

PD for principals & technology coordinator

On-site technical support, as needed

Online “Red” Course (teachers & coaches)

Provided by MCS Classroom support, as

requested by teachers

Implementation Roles & Responsibilities

MCS Purchase equipment &

supplies Hire/assign READ 180

teachers Arrange schedule (90

minute) Roster randomly

assigned students

Developer Technical implementation

check (within 4 weeks) One year unlimited

(limited) support Review MCS data &

provide recommendations