does the mathematics recovery program improve students' mathematics learning? thomas smith,...

Does the Mathematics Recovery Program Improve Students'

Mathematics Learning?

Thomas Smith, Paul Cobb, Dale Farran, David Cordray and Charles

MunterVanderbilt University

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grants R305B070554 and R305B040110 to Vanderbilt

University. The opinions expressed are those of the authors and do not necessarily represent views of the U.S. Department of Education.

Framing the Problem• Children enter school with a wide range of math abilities

– Baroody, 1987; Dowker, 1995; Gray, 1997; Griffin & Case, 1999; Housasart, 2001; Wright, 1991, 1994a; Young-Loverage, 1989

• Children who are less ready for school typically come from low SES families, are of racial/ethnic minority backgrounds, have parents who do not speak English– Alexander & Entwistle, 1998; Berends et al, 2005; Crosnoe, 2005

• Pre-K math ability is highly predictive of achievement at the end of first grade and even at the end of 5th grade– Duncan, Claessens, and Engel, 2004; Princiotta, Flanagan, and Germino

Hauskens, 2006

• In the absence of intervention, the initial gap in mathematics achievement continues to widen– Aunola, Leskinen, and Lerkkanen, 2004

An Early Mathematics Intervention• Math Recovery (MR) is a early arithmetic intervention

that involves diagnostic, one-to-one tutoring.

• MR was first implemented in the United States in 1999 and has served over 3,000 students in 19 states.

• How does it work?– Tutors are typically selected from among elementary teacher

in a school implementing the program (½ time release )– Tutor Training– Pullout, one-on-one tutoring– Up to 55 half-hour tutoring sessions every day over an 11

week period– Each session is videotaped; tutors review to plan next session– Each tutor works with 9-12 students per year

MR Program Logic Model

Students develop increasingly sophisticated

strategies for solving number problems

Students catch up to their peers and are

successful in regular math classes

Tutors pose tasks within student’s zone

of proximal development

Initial diagnostic

assessment

MR Instructional Framework used to

identify appropriate types of instructional tasks

Student profile assigned using MR Learning

Framework

Tutor training in Learning and Instructional Frameworks

What does MR tutoring look like?

Research Questions

• Does participation in MR raise the mathematics achievement of low performing first-grade students?

• MR claim— “students performing below the 25th percentile will benefit most from Math Recovery”

• Are any gains made from participation in MR maintained through the end of second grade?

• We also test the extent to which tutor knowledge and fidelity of implementation influence the effectiveness of MR.

Research Setting and Participants• Setting

– 20 elementary schools (five urban, ten suburban and five rural), representing five districts in two states

– Each school was a ‘fresh site’

• Participants– Students were selected for participation at the start of

first grade based on kindergarten teacher recommendations and their performance on MR’s screening interview and follow-up assessment interview

• Students with IEPs, speech/language support, ESL support, or excessive absences were excluded

– The number of students eligible for tutoring ranged from 17 to 36 across each of the the 20 schools.

Multisite randomized field trial• From the pool of students identified as eligible

for MR within each school, students were randomly assigned to a tutoring cohort of 3 students each with a different start date– Cohort A—September– Cohort B—December– Cohort C—March

• The remaining students assigned to the “waiting list”– selected to join an MR tutoring cohort if an assigned

participant left their school or were deemed “ineligible” due to a special education placement

• Process repeated in year 2

Tutor selection• District coordinators recruited 18 teachers

to receive training and participate as MR tutors from the participating districts—all of whom had at least two years of classroom teaching experience.

• 16 of the tutors received half-time teaching releases to serve one school each; 2 of the tutors received full-time teaching releases to serve two schools each.

• All tutoring positions were underwritten by their respective school districts.

Assessments Administered• MR1.1 Screening Assessment

– Designed to identify where students are in the MR frameworks and assess eligibility for the intervention

– Assessment tasks parallel tasks used in MR tutoring

• MR proximal (MRP)-- timed assessment of math fluency, designed in consultation with the program developers

• Woodcock Johnson III Achievement tests (WJ III) subtests– Fluency (MF) speed of performing simple calculations

for 3 minutes– Applied Problems (AP)--oral, math "word problems,”– Quantitative Concepts (QC)-- oral questions about

mathematical factual information, operations signs, etc

Figure 1:MR Treatment and Assessment Cycle for Each School

MR1.1 = Full Math Recovery Assessment (not given by external assessor)MRp = Proximal Math Recovery Assessment (newly developed)WJ full = Full Woodcock Johnson III Achievement Assessment (Applied Problems, Math Fluency, and Quantitative Concepts)WJ MF = Math Fluency subtest of the Woodcock Johnson III Achievement Assessment only

Sample characteristics of entering 1st graders

Data Analysis• 2-level hierarchical linear model (HLM) with

students (i) nested within tutors (j)

Outcomeij=β0j+ β1j(treatment)ij+ β2j(pretest)ij + β3j(female)ij + β4j(lep)ij + β3j(frpl)ij + β4j(age)ij + β5j(Site 1)j + β6j(Site 3)j + β6j(Year2)ij + uj+ rij

OutcomesMR1.1 (Diagnostic Assessment)MR ProximalWJIII Math FluencyWJIII Applied ProblemsWJIII Quantitative ConceptsWJIII Math Reasoning (AP + QC)

Independent variablesTreatmentPretest (1st Principal Component of MRP, WJMF, WJAP, WJQC in fall of 1st grade)FemaleLimited English Proficient (LEP)Free or Reduced Price Lunch Status (frlp)Age at pretest (months)Site1; Site2; Site3 (Site 2 suppressed)Year2 (2007=0; 2008=1)

Model for assessing the effectiveness of MR at the end of 1st grade

AssessmentEffect size Sig N



MR1.1 (Diagnostic Assessment) 0.85 *** 759



MR1.1 (Diagnostic Assessment) 0.85 *** 759 MR Proximal 0.26 *** 775



MR1.1 (Diagnostic Assessment) 0.85 *** 759 MR Proximal 0.26 *** 775 WJIII Math Fluency 0.14 * 775



MR1.1 (Diagnostic Assessment) 0.85 *** 759 MR Proximal 0.26 *** 775 WJIII Math Fluency 0.14 * 775 WJIII Applied Problems 0.25 *** 775 WJIII Quantitative Concepts 0.21 *** 775 WJIII Math Reasoning (AP + QC) 0.26 *** 775

Interactions tested• No treatment X year effect

– Justifies pooling data across years– Implies tutors are not more effective in their second year

• No treatment by site effect– No evidence that tutors in some sites were more

effective at increasing student learning than in other sites

• Treatment X pretest effect for some assessments– Low performing 1st grade entrants appear to benefit

more from math recovery than their higher performing counterparts in mathematical reasoning

How much do the effect sizes change if the sample is limited to students

performing below the 25th percentile at the start of 1st grade?

Full Sample

Below 25th%ile on WJIII MR Pretest

Assessment ES ES % differenceMR1.1 (Screening) 0.85***

MR Proximal 0.26***

WJIII Math Fluency 0.14*

WJIII Applied Problems 0.25***

WJIII Quantitative Concepts 0.21***

WJIII Math Reasoning (AP + QC) 0.26***

* p<0.05, ** p<0.01, *** p<0.001 (two tailed tests)

How much do the effect sizes change if the sample is limited to students

performing below the 25th percentile at the start of 1st grade?

Full Sample


Assessment ES ES % differenceMR1.1 (Screening) 0.85*** 0.94*** 11%MR Proximal 0.26*** 0.29** 12%WJIII Math Fluency 0.14* 0.29** 107%WJIII Applied Problems 0.25*** 0.33** 32%WJIII Quantitative Concepts 0.21*** 0.27* 29%WJIII Math Reasoning (AP + QC) 0.26*** 0.35*** 35%* p<0.05, ** p<0.01, *** p<0.001 (two tailed tests)

Do the effects of participation in MR last through the end of 2nd grade?

Full Sample


Assessment ES Sig N ES N

MR Proximal (2nd grade)

WJIII Math Fluency

WJIII Applied Problems

WJIII Quantitative Concepts

WJIII Math Reasoning (AP + QC)

† P<= .10 (one tailed-test)


Full Sample



MR Proximal (2nd grade) 0.06 NS 320

WJIII Math Fluency -0.02 NS 320

WJIII Applied Problems 0.08 NS 320

WJIII Quantitative Concepts 0.00 NS 320

WJIII Math Reasoning (AP + QC) 0.05 NS 320



Full Sample



MR Proximal (2nd grade) 0.06 NS 320 0.13 NS 105

WJIII Math Fluency -0.02 NS 320 0.30 † 105

WJIII Applied Problems 0.08 NS 320 0.23 † 105

WJIII Quantitative Concepts 0.00 NS 320 0.00 NS 105

WJIII Math Reasoning (AP + QC) 0.05 NS 320 0.14 NS 105


WJIII Applied Problems Predicted Values—start of 1st to the end of 2nd grade

Summary• Positive causal effects on proximal and distal

outcome measures (WJIII) from participating in MR at the end of first grade– Some evidence MR is more effective for lowest performing

students

• Positive effects fade by the end of second grade– Some evidence of long term increases in math fluency and

applied problems among participants who started 1st grade performing below the 25% percentile

• We cannot attribute the lack of sustained gains through the second year solely to poor implementation– Process components: fairly faithful to the MR model– Structural components: less faithful to the MR model

Tutor Fidelity: Exposure/Duration

0.2

.4.6

.81

Tutor Fidelity: Adherence

Assessment 1.1 Assessment 2.1 Profile Assigned

Perfect Fidelity

No Fidelity

0.2

.4.6

.81

Tutor Fidelity: ProcessPerfect Fidelity

No Fidelity

Ongoing Assessment

ZPD Nature of Instruction

Positive Infidelity

“Positive Infidelity” (Cordray & Hulleman, 2009)

Includes instructional practices identified in mathematics education research:

• Re-voicing student thinking to highlight particular mathematical ideas, to introduce mathematics vocabulary or to position students in relation to each other and their arguments (Franke, Kazemi, & Battey, 2007; O’Connor & Michaels, 1993)

• Asking the student to solve a task (s)he has just solved in a different way, so that the student has an opportunity to approach the same problem from a potentially different mathematical perspective or to represent the mathematics in a different way to find methods that enable progress (Carpenter & Lehrer, 1999: NCTM, 2000)

• Asking the student to compare alternative strategies and why they work, to encourage the student to be reflective and to provide opportunities to make connections between various strategies and mathematical ideas (Carpenter & Lehrer, 1999; Rittle-Johnson & Star; 2007).

What did we learn from studying Math Recovery?• First evaluation of the causal effects of MR

– Tutoring meets goals immediately following treatment, but the effects fade

• Why?– Tutor knowledge?

• Although tutor knowledge improved among those that started low, no evidence that this improved outcomes

– Fidelity of implementation?• Tutors were faithful when doing engaging in strategy based

activities• They did not do this very much

• Highlights the importance of attending to the developer’s Theory of Action in designing an evaluation

MR intervention components

Initial AssessmentsAdministered correctly

Profile assigned

Instruction

12-15 weeks, 25-30 min

Ongoing assessment

Instruction targets ZPD

Socratic nature of instruction

“STRUCTURE”•Exposure/duration•Adherence

“PROCESS”•Quality of Delivery•Participant responsiveness•Program differentiation

“Positive infidelity”

0.2

.4.6

.81

Tutor Fidelity: ProcessPerfect Fidelity

No Fidelity

Ongoing Assessment*

ZPD Nature of Instruction

Positive Infidelity

*Significant (p<.05) differences between tutors and experts

= expert mean (n=15)

Model for assessing the Influence of fidelity to exposure/duration of MR model (end of grade 1)

Outcomeij=β0j+ β1j(pretest)ij+ β2j(lowses)ij + β3j(District 1)ij + β4j(District 2)ij + β5j(Year2)ij + +

β6j(# of sessions)ij + β7j(avg session time)ij + β8j(avg strategy time)ij + rij

β / sd(DV)***p < .001**p < .01

*p < .05

does the mathematics recovery program improve students' mathematics learning? thomas smith,...

Documents