What’s the Matter?

Methodological Challenges in School Effectiveness Research

2

Session Objectives

• Why is school effectiveness hard?– Causality, complex structure, change, measurement ...

• What models/methods are available?– Multi-level, structural equation, multiple membership, MCMC ...

• Conclusions– Evidence about school effectiveness, future direction

Methodological Challenges for School Effectiveness

3

Great Scientific Advances

The Double Helix (1944-1953) One Giant Step for

Mankind (1961-1969)

School Effectiveness

(1964- )

Coleman et al. ( 1966); (Strand 2007)

Methodological Challenges for School Effectiveness

What ? How ? Why ?

4

School Performance is Dynamic

• Parents want to know about future performance; school league tables record the past

• Time-School-Pupil: a 3 level model

• Effectiveness changes; so does the definition of “effective”

Leckie and Goldstein (2009)

Methodological Challenges for School Effectiveness

5

• The education system is dynamic

• 43% of primary pupils move between schools

• League tables credit achievement to last school

• Needs multiple membership and cross-classification

• Pro rata apportionment corrects school effects by 17%

Creemers, Kyriakides and Sammons (2010); Goldstein, Burgess and McConnell (2007); Leckie (2008)

School Pupils are Dynamic

Methodological Challenges for School Effectiveness

6

Attainment and Attenuation

• Prior attainment is hugely important in CVA

• Standard 2-level models omit measurement error

• Unaccounted measurement error attenuates the estimated slope at the pupil level

Raudenbush and Bryk (2002); Woodhouse, Yang, Goldstein and Rasbash (1996); Hutchison (2007)

Methodological Challenges for School Effectiveness

7

Phantom Contextual Effects

• The estimate at the school level is less attenuated

• The CVA model has a different parameterization

• Where

• Differential attenuation = Phantom contextual effect

Woodhouse, Yang, Goldstein and Rasbash (1996); Harker and Tymms (2004)

Methodological Challenges for School Effectiveness

8

Measurement and School Effects

• School effects are assessed using variance components

• Phantom effects carry through to residual variation

• We under-estimate VPC unless we omit the context

Methodological Challenges for School Effectiveness

9

Poverty of Data• Low socio-economic status (SES) pupils make slower

academic progress but SES is hard to collect

• So DCSF uses free school meals (FSM)

• FSM has a number of weaknesses– Only one component of SES– Dichotomous– Missing data– High mis-classification rates

Gorard (2009); Kounali, Goldstein and Lauder (2007)

Methodological Challenges for School Effectiveness

10

Measuring Dichotomously

Methodological Challenges for School Effectiveness

-2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

PovertyFSMError

11

Measuring Dichotomously

Methodological Challenges for School Effectiveness

... Badly

60%

40% 17.2%

82.8%

FSM=1

Actual Observed

16%

24%

1.2%

58.8%FSM=0

16/17.2

24/82.8

ExpectedValue

Kounali, Goldstein and Lauder (2007); Goldstein, Kounali, and Robinson (2007)

.64

12

FSM and School Effects• Unreliability due to dichotomous measurement and

mis-classification compound

• CVA omits a contextual term for FSM

• Only observing a fraction of FSM pupils may cause us to further overestimate school effects

Methodological Challenges for School Effectiveness

Goldstein, Kounali, and Robinson (2007)

13

Manifestations of the Phantom• The phantom effect manifests on compositional variables

• “... differences in the composition of schools’ intake achievements ... appears to be a major driver of between-school differences in GCSE scores.”

• “school level coefficients which show strong differential effects for prior achievement, FSM and SEN.”

Leckie and Goldstein (2009); Leckie (2008)

Methodological Challenges for School Effectiveness

14

Discriminating Among Schools• Discrimination between schools is (largely) a function of

school size and VPC

• Coleman (1966) estimated VPC at around 10%

• “ ... adjusting for these [compositional] variables halves the between-school variance; the VPC drops from 10.4% to 7.2%” Leckie and Goldstein (2009)

• To improve discrimination, we need to allow for measurement error

Methodological Challenges for School Effectiveness

15

Dealing with Unreliability• Methods to deal with unreliability need to :– Work in conjunction with complex, multi-level methods– Handle complex, multi-level error and categorical variables

• Some competing methodologies:– Regression discontinuity– Multiple measures in MLSEM– Item Response Theory (IRT)– Sensitivity analysis approach (MCMC)

Luyten (2006); Luyten and Tymms (2010 ); Marsh et al. (2009); Fox (2004 ); Ecob and Goldstein (1983); Goldstein, Kounali, and Robinson (2007)

Methodological Challenges for School Effectiveness

16

Summary / Conclusions

• Phantom effects are ubiquitous in school effectiveness research

• FSM is deeply flawed as a proxy for SES and makes interpretation of results very problematical

• Merely including cluster-level aggregates will tend to understate school-level residual variance

• SER needs complex ML methods in combination with sophisticated tools to model unreliability

Methodological Challenges for School Effectiveness

17

BibliographyEquality of Educational Opportunity (Coleman et al. 1966)Minority Ethnic Pupils in the Longitudinal Study of Young People in England (Strand 2007)The estimation of school effects (Raudenbush and Willms 1995)Methodological Advances in Educational Effectiveness Research (Creemers, Kyriakides and Sammons 2010)The limitations of using school league tables to inform school choice (Leckie and Goldstein 2009)Incorporating Student Mobility in Achievement Growth Modeling: A Cross-Classified Multiple Membership Growth Curve Model

(Grady and Beretvas 2010)Modelling The Effect Of Pupil Mobility On School Differences In Educational Achievement (Goldstein, Burgess and McConnell

2007)Effects of Pupil Mobility and Neighbourhood on School Differences in Attainment (Leckie 2008)School Value Added Measures in England (Ray 2006)The Effects of Student Composition on School Outcomes (Harker and Tymms 2004)Modelling Response Error in School Effectiveness Research (Fox 2004)Multilevel, Fully Latent-variable Models of Contextual Effects ... (Marsh et al., 2009)Instrumental variable methods for the estimation of test score reliability (Ecob and Goldstein 1983)Modelling measurement errors and category misclassifications in multilevel models (Goldstein, Kounali, and Robinson 2007)An empirical assessment of the absolute effect of schooling: regression discontinuity applied to TIMSS-95 (Luyten 2006)The probity of free school meals as a proxy measure for disadvantage (Kounali, Goldstein and Lauder 2007)Modelling measurement errors and category misclassifications in multilevel models (Goldstein, Kounali, and Robinson 2007)Serious doubts about school effectiveness (Gorard 2009)

Methodological Challenges for School Effectiveness