evaluation of a seven-year program on fostering reading
Post on 20-Apr-2022
1 Views
Preview:
TRANSCRIPT
Evaluation of a Seven-year Program on Fostering
Reading Comprehension
Olaf Köller1, Gesa Ramm2, and Thomas-Riecke-Baulecke2
1Leibniz-Institute for Science and Mathematics Education, Kiel
2Institute for Quality Development in Schools, Schleswig-Holstein
Leibniz Institute for Science and Mathematics Education
Paper presented at 15th EARLI Biennial Conference, München 2013
• Look back in anger: Findings of PISA 2000 in Germany
• Reforms after PISA 2000
• The project Reading Makes Students Smarter (RMSS; Lesen macht stark)
• Evaluation of RMSS
– Study 1: An 18 months evaluation of RMSS
– Study 2: Effects of reading coaches
Overview
Leibniz Institute for Science and Mathematics Education
Leibniz Institute for Science and Mathematics Education
Look back in Anger: Findings in PISA 2000
Students at Proficiency Level 1 or below
0 10 20 30
Poland
Portugal
Germany
Belgium
Switzerland
USA
Norway
OECD
Austria
New Zealand
Canada
Finnland
Korea
Students in %
below Level 1
Level 1
Leibniz Institute for Science and Mathematics Education
Reforms after PISA
Federal States agreed on common educational standards for primary and secondary school
Many intervention programs on reading comprehension
Reading comprehension as part of all school subjects
Language learning classes for students with migration background
Intensive research on determinants of reading comprehension
Problem: Well evaluated large-scale intervention programs were not available
Leibniz Institute for Science and Mathematics Education
What we Already Knew at that Time
The good reader is bright, has a lot of prior knowledge, is highly motivated, has a high reading
self-concept and has effective reading strategies (self-regulation skills) plus meta-cognitive
reasoning (Schiefele, 1996; Artelt, 2001; Guthries, 2004 )
Groups at risk are low SES-students and students with migration background
Interventions are more successful in younger cohorts
Long-term interventions are more successful
Interventions should not only take place within the regular classes but need additional time
Leibniz Institute for Science and Mathematics Education
Large-scale Intervention Program „Reading Makes Students
Smarter“ (RMSS)
Program started in 2006
Target Group (at the beginning): Secondary schools with high proportions of poor-achieving
students
5 Modules
Increasing reading time
Increasing reading motivation (reading interest)
Fostering meta-cognitions
Fostering self-regulated reading
Self-assessment of progress
Additional reading classes, additional reading material
Coaches for principles
TPD measures for teachers
Leibniz Institute for Science and Mathematics Education
Underlying Model of RMSS
Coach
Principal
Teachers‘
Professional
Development
Modules
Quality and Quantity
of Teaching
Students‘
- Reading Motivation
- Reading Self-concept
- Reading Activities
Students‘ Reading
Competence
Leibniz Institute for Science and Mathematics Education
Evaluation of the Program: Shortcomings
and Challenges
Program is a universal program for all students of a class even if they are already proficient (we thus
expected only small effects)
Randomized field trial was not accepted by local authorities
We started with 50 intervention schools; only 13 control classes in grade 5
In the first year many control classes changed to intervention classes
However, we could use data from a representative student sample which was part of a research
project of the University of Kiel (LISA project; PI: Jens Möller) and worked on some common test
items
Program can only be evaluated as a whole, effects of each module cannot be separated (but see
Slavin, 2008; for a justification of this design)
Leibniz Institute for Science and Mathematics Education
Study 1: Method
Two measurement points were realized (beginning of grade 5 and middle of grade 6)
Data was available from only 50 intervention schools
Instruments were standardized reading tests that were sufficiently reliable (>. 75); items came form
national and international studies (e.g., PIRLS)
Tests in RMS- and LISA-schools had some linking items
IRT-scaling methods were used to build a common scale for both groups
Leibniz Institute for Science and Mathematics Education
Findings: Reading Comprehension in the Intervention Cohort and
in the LISA Sample
360
380
400
420
440
460
480
500
Grade 5 Grade 6
Read
ing
sco
re
Interv. Group
LISA sample
d = -0.07
d = 0.41**
Leibniz Institute for Science and Mathematics Education
Gains over 18 Months Broken Down to Schools of the
Intervention Group
-100
-50
0
50
100
150
200
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51
Gain
Sco
re
Schools
Leibniz Institute for Science and Mathematics Education
Summary of Study 1
A lot of methodological problems
Results, however, provide some evidence that the program has been successful
Analyses on school level shows large differences in students‘ benefits from the program
No systematics control of the process quality in the intervention schools
Teachers’ professional knowledge and interaction quality as potential mediators
Further studies needed that help to explain why schools differ so dramatically in their gain
scores
Study 2
Leibniz Institute for Science and Mathematics Education
• To control for interaction quality, 12 reading coaches (experienced RMSS teachers) received
additional training and a standardized curriculum for 16 units
• Reading coaches worked only with poor readers (students in the middle of grade 5) who were screened
by means of a standardized reading test (easy test; cut score: less than 28 out of 41 items were solved
correctly)
• Each group consisted of a maximum of 10 poor reading students
• Students from 8 different RMSS schools served as a control group; they conducted the “ordinary”
RMSS program; only students with less than 28 items solved correctly in the pretest were included
• Additional measurement points were at the end of grade 5 and at the end of grade 6
Design of Study 2
Leibniz Institute for Science and Mathematics Education
Pretest (Middle
of grade 5)
Intervention Posttest (End of
grade 5)
Follow-up test
(End of grade 6)
Intervention
Group w w w w
Control Group w w w
Results of Study 2
Leibniz Institute for Science and Mathematics Education
0
5
10
15
20
25
30
35
Pretest Posttest Follow-up
Item
s s
olv
ed
co
rrectl
y
Intervention group
Control group
d = .57
d = .25
d = .18
Results of Repeated Measurement ANOVA:
Group x Time: F2, 474
= 3,23, p < .05
Summary of Study 2
Leibniz Institute for Science and Mathematics Education
• Again positive effects of RMSS; large gains of poor readers in both groups
• Higher degree of standardization of RMSS intervention had a small but significant effect that was
also observable in the follow-up test
• However, remember that the more standardized intervention was restricted to the second half of
grade 5
• Overall both studies provided some evidence that the RMSS program can help to overcome some
of the problems PISA has found out
Leibniz Institute for Science and Mathematics Education
Thank you very much for your attention!
Contact: koeller@ipn.uni-kiel.de
Gesa.Ramm@iqsh.landsh.de
top related