less is more? an application of propensity score stratification to first-grade retention mieke goos,...
TRANSCRIPT
Less Is More? An Application of Propensity Score Stratification to First-Grade Retention
Mieke Goos, Jan Van Damme, Patrick Onghena and Katja Petry
SREE 2010
1. Introduction
• Starting point:
– Many young children struggle in elementary school
– Countries deal with these early problems in a different way
– Internationally frequently applied measure = grade retention
1. Introduction
• Grade retention in Flanders: – Relatively high rate
• For example: PISA 2003
1. Introduction
– Relatively high rate … especially in Grade 1 About 7% of Flemish children repeat Grade 1
– Socially approved by educators, policy makers and parents
→ being a grade retainee in Flanders has a different connotation than for example in the US (negative overtone)
– No formal rules regarding grade promotion (no national/state standardized test procedures)
→ retention decision = joint decision by teacher and parents
1. Introduction
• Research question:
– Is Grade 1 retention an effective practice or not?
• Focus of this study:
– children’s psychosocial growth throughout elementary school
2. Method
• Subjects: representative sample from the Flemish SiBO-project
– 3624 first-graders, of which 298 were retained
– 222 classes
– 121 schools
followed until Grade 6
2. Method
• Instruments
– Psychosocial growth: Teacher questionnaire
rated yearly by the teacher items on a 1 to 6 point Likert scale 7 subscales
Social skills• Popularity among classmates• Aggressive behavior • Hyperactive behavior• Asocial behavior
Dynamic-affective attitudes and skills • Independent participation• School well-being• Self-confidence
2. Method
• Instruments (continued)
– Propensity of repeating Grade 1
official records achievement tests Standard Progressive Matrices teacher questionnaire about the child parent questionnaire teacher questionnaire about teacher didactics school staff questionnaire
• 68 prior student characteristics• 59 prior class characteristics• 42 prior school characteristics
2. Method
• Analyses: 4-steps-procedure
– Step 1: identification of ‘true’ confounders of Grade 1 retention
prior student, class and school characteristics that are related to both treatment (i.e., Grade 1 retention)
and outcome (i.e., children’s individual psychosocial growth)
– Step 2: estimation of propensity scores based on these confounders
3-level logistic regression analysis (students – classes – schools)
2. Method
• Analyses: 4-steps-procedure (continued)
– Step 3: decile stratification
10 strata of equal size
– Step 4: estimation of average psychosocial effects
3-level curvilinear growth curve analyses (measurements – students – schools)
2. Method
• Analyses: 2 comparison strategies
– Same-grade approach = comparing retainees with their younger grade-mates
– Same-age approach = comparing retainees with their age-mates who were promoted to a higher grade
2. Method
A
Research year 1(age 7)
Research year 2(age 8)
Research year 3(age 9)
B
3rd grade
C2nd grade D
E
1st grade Grade retention
Grade retention
F
H
GGrade retention
Promotion
Promotion
Promotion
Promotion
Cohort 1
Cohort 2 (not in SiBO dataset)
A
Research year 1(age 7)
Research year 2(age 8)
Research year 3(age 9)
B
3rd grade
C2nd grade D
E
1st grade Grade retention
Grade retention
F
H
GGrade retention
Promotion
Promotion
Promotion
Promotion
Cohort 1
Cohort 2 (not in SiBO dataset)
2. Method
SAME-GRADE COMPARISON
A
Research year 1(age 7)
Research year 2(age 8)
Research year 3(age 9)
B
3rd grade
C2nd grade D
E
1st grade Grade retention
Grade retention
F
H
GGrade retention
Promotion
Promotion
Promotion
Promotion
Cohort 1
Cohort 2 (not in SiBO dataset)
2. Method
SAME-AGE COMPARISON
2. Method
• Analyses: 2 comparison strategies (continued)
– Why? ~ 2 different questions
How do Grade 1 repeaters, at the cost of one extra year of education, develop in comparison to younger children with whom they will eventually finish elementary school?
SAME-GRADE APPROACH
How would Grade 1 retainees have developed, had they been promoted to Grade 2 instead?
SAME-AGE APPROACH
3. Results
• Propensity scores
– based on 52 prior student characteristics
Promoted students
M = -4.47
Retained students
M = -0.12
3. Results
• Propensity score stratification
– Cut-offs for strata based on overlap
– Division into 10 strata of equal size
3. Results
– Within-stratum balance in propensity score
N M N M N M
1 233 -6,03 1 -6,32 234 -6,03
2 233 -5,57 1 -5,45 234 -5,57
3 233 -5,24 1 -5,23 234 -5,24
4 233 -4,93 1 -4,78 234 -4,93
5 232 -4,60 2 -4,70 234 -4,61
6 231 -4,30 3 -4,28 234 -4,30
7 234 -3,97 1 -3,81 235 -3,97
8 232 -3,58 3 -3,54 235 -3,58
9 230 -2,95 5 -2,90 235 -2,95
10 132 -1,17 103 0,40 235 -0,48
Total 2223 -4,37 121 -0,26 2344 -4,16
Stratum
Promoted students Retained students Total
=
≠
≠
3. Results
– Within-stratum balance in 97% of the observed pre-retention student, class and school characteristics
→ Retained and promoted children within a certain stratum are equivalent (within sampling fluctuations) in terms of risk factors preceding retention
3. Results
• Same-grade comparisons
– On average:
during their retention year, Grade 1 retainees show a similar psychosocial functioning in comparison to younger grade-mates who are at similar risk of being retained
but … over time they (mostly) grow significantly slower they end up showing more hyperactive behavior, feeling less well at school etc.
! One exception: popularity among classmates
3. Results
1
2
3
4
5
6
0 1 2 3 4 5
Time
Ag
gre
ssiv
e b
ehav
ior
Retained
Promoted
3. Results
1
2
3
4
5
6
0 1 2 3 4 5
Time
Hyp
erac
tive
beh
avio
r
Retained
Promoted
signsign
3. Results
1
2
3
4
5
6
0 1 2 3 4 5
Time
Aso
cial
beh
avio
r
Retained
Promoted
3. Results
1
2
3
4
5
6
0 1 2 3 4 5
Time
Ind
ep
en
de
nt
pa
rtic
ipa
tio
n
Retained
Promoted
signsign
3. Results
1
2
3
4
5
6
0 1 2 3 4 5
Time
Sc
ho
ol w
ell-
be
ing
Retained
Promoted
signsign
3. Results
1
2
3
4
5
6
0 1 2 3 4 5
Time
Se
lf-c
on
fid
en
ce
Retained
Promotedsign
3. Results
! One exception !
1
2
3
4
5
6
0 1 2 3 4 5
Time
Po
pu
lari
ty a
mo
ng
cla
ss
ma
tes
Retained
Promoted
signsign
3. Results
• Same-age comparisons
– On average:
Grade 1 repeaters would have developed a similar or even better psychosocial functioning, had they been promoted to Grade 2 instead, both in the short and long run
3. Results
1
2
3
4
5
6
0 1 2 3 4 5
Time
Po
pu
lari
ty a
mo
ng
cla
ss
ma
tes
Retained
Promoted
signsign
3. Results
1
2
3
4
5
6
0 1 2 3 4 5
Time
Ag
gre
ssiv
e b
ehav
ior
Retained
Promoted
3. Results
1
2
3
4
5
6
0 1 2 3 4 5
Time
Hyp
erac
tive
beh
avio
r
Retained
Promoted
sign
sign
3. Results
1
2
3
4
5
6
0 1 2 3 4 5
Time
Aso
cial
beh
avio
r
Retained
Promoted
3. Results
1
2
3
4
5
6
0 1 2 3 4 5
Time
Ind
ep
en
de
nt
pa
rtic
ipa
tio
n
Retained
Promotedsignsign
3. Results
1
2
3
4
5
6
0 1 2 3 4 5
Time
Sc
ho
ol w
ell-
be
ing
Retained
Promoted
sign
3. Results
1
2
3
4
5
6
0 1 2 3 4 5
Time
Se
lf-c
on
fid
en
ce
Retained
Promotedsign
4. Conclusions and discussion
• Overall, Grade 1 retainees do not seem to benefit much from their retention year
– over time they grow slower compared to grade-mates, making them end up feeling less well at school etc. at the end of elementary school
– while they would have developed a similar or even better psychosocial functioning, had they been promoted to Grade 2 instead
→ Practical implication: Our results call the practice of Grade 1 retention in Flanders into question
4. Conclusions and discussion
• Future research is needed
– Sensitivity analyses
– Other outcome: growth in math and reading skills
– Moderating effects: provision of additional support
--------------------------------------------------------------------------------------------------------------------
2ND BIENNIAL MEETING OF THE EARLI SPECIAL INTEREST GROUP 18
“Educational Effectiveness: Models, Methods and Applications”
Leuven, Belgium 25-27 August 2010
http://www.sigee2010.org
--------------------------------------------------------------------------------------------------------------------
Keynote lectures by Prof. Dr. Stephen Raudenbush, Prof. Dr. Robert E.Slavin, Prof. Dr. Jürgen Baumert and Prof. Dr. Jan-Eric Gustafsson
