project more 2001-2008 independent evaluation completed by the center for evaluation services...

Project MORE 2001-2008 Independent

Evaluation

Completed by

The Center for Evaluation Services

Bowling Green State University Updated 11/12

Project MORE Evaluation Results 2001-2008

• Independent Evaluation conducted from 2001-to 2008 by Center for Evaluation Services at Bowling Green State University

• Consistent Causal Comparative Results at the Statistically Significant Level (.05) over six years.

• Evaluation Results to be published in Education and Training in Developmental Disabilities (December, 2007) to qualify as Scientifically-Based Research.

Project MORE Results (2001-2002)

1st Grade Title 1 Comparison

MORE students made more reading progress gains than control students for Wordlist level and Reading Errors

2nd Grade Title 1 Comparison

MORE students made more reading progress gains than control students for Wordlist level

3rd Grade Title 1 Comparison

MORE students made more reading progress gains than control students for Wordlist level, Scaled Score, Instructional Reading Level, Grade Equivalent, and Normal Curve Equivalent

3rd Grade SLD Comparison MORE students made more reading progress gains than control students for for Scaled Score, Instructional Reading Level, and Normal Curve Equivalent

4th Grade SLD Comparison MORE students made more reading progress gains than control students for for Reading Fluency

• First, CES looked at Project MORE reading gains with BGSU Informal Reading Assessment and the STAR Reading Assessment

• Results: Students with specific learning disabilities, cognitive disabilities, emotional disturbances, and Title 1 students had statistically significant gains, and in many cases there were month-for-month reading gains.

• Project MORE students with disabilities and students receiving Title 1 services were compared to similar students based on the State of Ohio’s Similar District Software. Both the Project MORE students and the control students were receiving either Title 1 or special education prescribed services. However, Project MORE students received HOSTS as or in addition to their prescribed Title 1 or special education services. Results are presented below.

Project MORE Results (2002-2003)• First, CES looked at Project MORE reading gains with BGSU Informal Reading

Assessment, the STAR Reading Assessment, and the Woodcock-Johnson Reading Achievement Assessment

• Results: Students with specific learning disabilities, cognitive disabilities, emotional disturbances, and Title 1 students had statistically significant gains, and in many cases there were month-for-month reading gains on the STAR.

• Project MORE students with disabilities and students receiving Title 1 services were compared to similar students based on the State of Ohio’s Similar District Software. Both the Project MORE students and the control students were receiving either Title 1 or special education prescribed services. However, Project MORE students received HOSTS as or in addition to their prescribed Title 1 or special education services. The Woodcock-Johnson Reading Achievement battery was piloted with 2 groups for the 2002-2003 school year. Results are presented below.

2nd Grade Title 1 Comparison

MORE students made more reading progress gains than control students for BGSU Reading Fluency and Comprehension, STAR Scaled Score, Grade Equivalent, Instructional Reading Level, Normal Curve Equivalent, Woodcock-Johnson Word and Letter Identification

4th Grade LD Comparison MORE students made more reading progress gains than control students for for Word Attack


BGSU: Wordlist Level 3rd grade Title 1 students, 2nd grade students with cognitive disabilities

BGSU: Reading Fluency 3rd grade Title 1 students, 3rd grade students with specific learning disabilities4th grade students with specific learning disabilities

BGSU: Reading Comprehension 3rd grade students with specific learning disabilities

Woodcock Johnson: Broad Reading

2nd, 3rd grade Title 1 students, 3rd grade students with specific learning disabilities

Woodcock Johnson: Basic Reading

3rd grade Title 1 students, 3rd grade students with specific learning disabilities4th grade students with specific learning disabilities

Woodcock Johnson: Letter/Word Identification

3rd grade Title 1 students, 3rd grade students with specific learning disabilities

Woodcock Johnson: Word Attack (Decoding)

3rd grade Title 1 students4th grade students with specific learning disabilities

• First, CES looked at Project MORE reading gains with BGSU Informal Reading Assessment and the Woodcock-Johnson Reading Achievement Assessment

• Results: Students with specific learning disabilities, cognitive disabilities, and Title 1 students had statistically significant gains, and in many cases there were month-for-month reading gains.

• Project MORE students with disabilities and students receiving Title 1 services were compared to similar students based on the State of Ohio’s Similar District Software. Both the Project MORE students and the control students were receiving either Title 1 or special education prescribed services. However, Project MORE students received HOSTS as or in addition to their prescribed Title 1 or special education services. Results are presented below.


DIBELS 2nd grade students with specific learning disabilities, cognitive disabilities, and students receiving Title 1


2nd grade students receiving Title 1


2nd grade students with specific learning disabilities and students receiving Title 13rd grade students with specific learning disabilities

Woodcock Johnson: Letter/Word Identification

2nd grade students with specific learning disabilities and students receiving Title 13rd grade students students receiving Title 1

Woodcock Johnson: Reading Fluency

2nd grade students with cognitive disabilities and students receiving Title 1

Woodcock Johnson: Reading Comprehension

2nd grade students with cognitive disabilities


2nd grade students with specific learning disabilities 3rd grade students with specific learning disabilities

• Examined two Project MORE mentoring programs: HOSTS (established) v. Reading-tutors (pilot) v. control students

• Students in HOSTS and Reading-tutors had similar reading gains although as expected the established program had a few more significant gains

• Results: Students with specific learning disabilities, cognitive disabilities, and Title 1 students had statistically significant gains, and in many cases there were month-for-month reading gains.

• Project MORE students with disabilities and students receiving Title 1 services were compared to similar students based on the State of Ohio’s Similar District Software. Both the Project MORE students and the control students were receiving either Title 1 or special education prescribed services. However, Project MORE students received HOSTS or Readng-tutors as or in addition to their prescribed Title 1 or special education services. Results are presented below.


DIBELS 3rd grade students with specific learning disabilities and students receiving Title 1


3rd grade students receiving Title 1

Woodcock Johnson: Reading Fluency


Woodcock Johnson: Reading Comprehension


• Examined two Project MORE mentoring programs: HOSTS v. Reading-tutors v. control students in only 3 groups of students 2nd and 3rd grade students with specific learning disabilities and 3rd grade students receiving Title 1 services.

• Students in HOSTS and Reading-tutors had similar reading gains although Reading-tutors program had a few more significant gains

• Results: Students with specific learning disabilities and Title 1 students had statistically significant gains, and in many cases there were month-for-month reading gains.

• Project MORE students with disabilities and students receiving Title 1 services were compared to similar students based on the State of Ohio’s Similar District Software. Both the Project MORE students and the control students were receiving either Title 1 or special education prescribed services. However, Project MORE students received HOSTS or Reading-tutors as or in addition to their prescribed Title 1 or special education services. Results are presented below.


DIBELS 2nd, 3rd, and 4th grade students with specific learning disabilities


3rd and 4th grade students with specific learning disabilities



Woodcock Johnson: Letter-Word Identification

4th grade students with specific learning disabilities


3rd grade students with specific learning disabilities

• Examined two Project MORE mentoring programs: Reading-tutors v. control students in only 3 groups of students 2nd, 3rd, and 4th grade students with specific learning disabilities.

• Results: Students with specific learning disabilities had statistically significant gains, and in many cases there were month-for-month reading gains.

• Project MORE students with disabilities were compared to similar students based on the State of Ohio’s Similar District Software. Both the Project MORE students and the control students were receiving special education prescribed services. However, Project MORE students received Reading-tutors as or in addition to their prescribed special education services. Results are presented below.


DIBELS 3rd grade students with specific learning disabilities



Woodcock Johnson: Letter-Word Identification

3rd grade students with specific learning disabilities



• Examined two Project MORE mentoring programs: Reading-tutors v. control students in only 2 groups of students 3rd and 4th grade students with specific learning disabilities.

• Results: Students with specific learning disabilities had statistically significant gains, and in many cases there were month-for-month reading gains.

• Project MORE students with disabilities were compared to similar students based on the State of Ohio’s Similar District Software. Both the Project MORE students and the control students were receiving special education prescribed services. However, Project MORE students received Reading-tutors as or in addition to their prescribed special education services. Results are presented below.


Regression Discontinuity: In 2007-2008 CES utilized a Regression Discontinuity Design to evaluate the effects of Project MORE on students’ reading fluency gains.The Oral Reading Fluency section of DIBELS was given to all 2nd, 3rd, and 4th graders at grade level for the 11 Project MORE schools (not all schools used Project MORE at all three grade levels). The Fall Benchmark was the pretest, and the Spring Benchmark was the posttest. Each grade level had a different cut-off score to determine whether students were in the MORE group (intervention) or the control group. However, the cut-off score differed depending on school. For the overall analysis, we chose the highest Project MORE words-per-minute score at Fall Benchmark as the cut-off score. The control group students were all students at Project MORE schools who had higher words-per-minute scores on the Fall Benchmark, but did not receive the Project MORE intervention. Thus, the Project MORE group (intervention) were students with disabilities or at-risk for reading failure. The control group were general education students. The intervention group (Project MORE students), which has the lower reading pretest scores, must have significantly higher reading gains (in this case, words-per-minute) than their fellow students with higher pretest scores (control group). It is very difficult for an intervention to produce this type of result at a statistically significant level.

RESULTS: The results indicated that the DIBELS posttest scores increased at a statistically significantly higher rate for Project MORE students (with disabilities and at-risk) than for their classmates (without disabilities) in 2nd, 3rd, and 4th grades during the 2007-2008 school year.

2nd Grade Regression Discontinuity Results and Summary

2nd grade results: Fall-to-spring reading gains on DIBELS Oral Reading Fluency for Project MORE students (with disabilities and at-risk) were compared to their classmates (without disabilities) to measure reading growth rate due to the Project MORE intervention at 10 schools. The results indicated that the DIBELS posttest scores increased at a statistically significantly higher rate for Project MORE students (with disabilities and at-risk) than for their classmates without disabilities (T = 7.55 p < .00).

160140120100806040200

200

150

100

50

0

predorf

po s

tdo r

f

48

CONTORLINTRVNT

GROUP

3rd Grade Regression Discontinuity Results and Summary

3rd grade results: Fall-to-spring reading gains on DIBELS Oral Reading Fluency for Project MORE students (with disabilities and at-risk) were compared to their classmates (without disabilities) to measure reading growth rate due to the Project MORE intervention at 9 schools. The results indicated that the DIBELS posttest scores increased at a statistically significantly higher rate for Project MORE students (with disabilities and at-risk) than for their classmates without disabilities (T = 4.20 p < .00).

200150100500

200

150

100

50

0

predorf

post

dorf

71

01

GROUP

Scatterplot of postdorf vs predorf

4th Grade Regression Discontinuity Results and Summary

4th grade results: Fall-to-spring reading gains on DIBELS Oral Reading Fluency for Project MORE students (with disabilities and at-risk) were compared to their classmates (without disabilities) to measure reading growth rate due to the Project MORE intervention at 6 schools. The results indicated that the DIBELS posttest scores increased at a statistically significantly higher rate for Project MORE students (with disabilities and at-risk) than for their classmates without disabilities (T = 2.25 p < .03).

200150100500

200

150

100

50

0

predorf

post

dorf

79

01

GROUP

Scatterplot of postdorf vs predorf

Overall Project MORE Results and New Directions

• Over the 7 year period, the evaluation has established Project MORE students have significant reading gains that many times are month-for-month.

• Overall, Project MORE students outperformed the comparison students in the numerous comparisons on both informal and standardized assessments. Over the course of seven years, there were no analyses in which the control group outperformed Project MORE students at a statistically significant level.

• Piloting an Implementation Integrity Checklist for Reading-tutors program in was completed in the Spring of 2008-2009.

• Utilizing a Regression Discontinuity Design, the results indicated that the DIBELS posttest scores increased at a statistically significantly higher rate for Project MORE students (with disabilities and at-risk) than for their classmates (without disabilities) in 2nd, 3rd, and 4th grades. These results will be submitted for publication in the Fall of 2009.

• Project MORE Evaluation was published in Education and Training in Developmental Disabilities (December, 2007)

project more 2001-2008 independent evaluation completed by the center for evaluation services...

Documents