the mathematics skills of school children: how does england compare to the high performing east...
TRANSCRIPT
The mathematics skills of school children: How does England compare to the high
performing East Asian jurisdictions?
PISA maths – England versus East Asia
EnglandSwedenPolandAustriaFrance
Slovak RepublicNorway
SloveniaDenmark
IcelandEstonia
GermanyAustraliaBelgium
New ZealandMacao-China
NetherlandsCanada
JapanSwitzerland
LiechtensteinFinland
Chinese TaipeiKorea
Hong Kong-ChinaSingapore
Shanghai-China
0.00 0.50 1.00 1.50 2.00 2.50 3.00
Years of schooling ahead of England
By age 15, children in East Asian counties are approximately 1.5 years of schooling ahead of children in England
Huge policy interest in this….
These regions and nations – from Alberta to Singapore, Finland to Hong Kong, Harlem to South Korea - should be our inspiration’
(Secretary of State for Education in England, Michael Gove)
‘we must learn from high-performing nations like Japan’(Shadow Secretary of State for Education in England, Stephen Twigg)
‘We have to see this as a wake-up call….. We can quibble, or we can face the brutal truth that we’re being out-educated’
(US Secretary of Education, Arne Duncan)
WHY!?
....we don’t really know
Possible explanations• Biology / genetics
• Cultural differences- Commitment and highly value education (‘Tiger mums’)
• ‘Failing’ secondary schools
• Teaching methods
• Design of the curriculum
• Many more!!
Possible explanations• Biology / genetics - Can’t do anything about
• Cultural differences – Hard to change in short-run- Commitment and highly value education (‘Tiger mums’)
• ‘Failing’ secondary schools
• Teaching methods Things we can change
• Design of the curriculum
• Many more!!
...but we do not know the relative importance of each of these factors
Policy – hard to know what we need to do to catch up!
Current research
Lots of research looking at the ‘high-performing’ systems
(particularly East Asia)
• E.g. Review of the curriculum
https://www.education.gov.uk/publications/standard/publicationDetail/Page1/DFE-RR178
• E.g. Teaching observations in Shanghi
Pretty much all qualitative
This paper: Think what quantitative research can add to the debate
But there is a MAJOR problem for both qualitative and quantitative research
East Asian countries have a lot of East Asian children in them!
….Big confounding factor
Implications of this confoundingQualitative research
– We observe differences in curriculum– We observe differences in teaching methods– BUT we don’t know if this has causal impact– Really the reason why East Asian countries doing better?
Quantitative research– Can’t easily ‘control away’ cross-country confounders– Therefore also difficult to establish causal impacts– PISA can tell us how far we are behind……– …..But it can’t tell us why!
My paper…..What else can be done to inform the debate????
Starting point– How do we judge school performance?– Widely recognised it should not be on % 5 A*- C grades– Should be judged on ‘value-added’ (how much children
improve)
International application– Should think same way about countries / school systems– Don’t judge UK secondary schools by PISA test scores– Judge them by value-added – How much do UK children improve in secondary school
(relative to East Asian children)
We try to start thinking about this…..
(1) How do average test scores change between age 10, 14 and 16 in England compared to East Asia?
(2) How does the distribution of test scores change between 10, 14 and 16 in England and East Asia?
(3) How does the socio-economic gap in children’s test scores change over age in England compared to East Asia?
Data
Ideal data→ Longitudinal
→ Follow same kids over time
→ Test scores designed to be comparable over time and across countries
→ E.g. A longitudinal PISA
This type of very rich data does not exist
Therefore have to think about an alternative
Data we do have….→ TIMSS 2003 4th grade (age 9/10)
→ TIMSS 2007 8th grade (age 13/14)
→ PISA 2009 (age 15 year 3 months – 16 years 2 months)
– Think of this as ‘repeated cross-sectional data’
– Investigate change over time at the population level under certain assumptions:
• Samples drawn from same population at each time point• The same skill is being measured at each time point
Argument:
Do these assumptions hold for TIMSS / PISA data?
Argue they provide an OK approximation….
An aside – how similar are PISA / TIMSS?
Often argued that PISA / TIMSS measure ‘different skills’• PISA = ‘Functional ability’• TIMSS = ‘Curriculum’ based skills
Qualitative research:
Analysis of questions suggests there are differences..
…but unable to say of this has major impact
Other statistical differences
Item-response model used
Target populations etc
But really how different are PISA and TIMSS??
PISA 2009 vs TIMSS 2011 (maths)KR SGTW
HK
JP
RU
IL FIUSGB(E)HU AUSILTIT
NZKZ SENO
ROAE TRMY
GE THTNCL
QA JO
ID
350
400
450
500
550
600
TIM
SS
2011
350 400 450 500 550 600PISA 2009
Strong correlation between x-national results (r = 0.88)
Actually tell us pretty similar things about performance
PISA 2003 vs Key stage 3 test scores (England)
Correlation PISA maths with Key Stage 3 average scores = 0.83
Correlation PISA maths with Key Stage 3 math scores = 0.85
Previous graphs suggest that key assumptions of the method used may approximately hold
20
International z-scoresNevertheless ……..
• PISA and TIMSS raw test scores are not directly comparable – based on a different array of countries.
• Convert into international z-scores.
• Each country’s mean test score (for each wave of the survey) is adjusted by subtracting the mean score achieved amongst all children in the countries for that particular survey and dividing by the standard deviation
• Estimates refer to English pupils’ test performance relative to other countries
21
Countries included in this study• Include all countries that took part in TIMSS 4th grade (2003),
TIMSS 8th grade (2007) and PISA 2009
• Leaves 13 countries:- England (Focus)- Asian Tigers (Hong Kong, Japan, Singapore, Taiwan)- Other developed (e.g. Italy, Norway, US)
22
Measures‘Average’ = Mean z-score
‘Inequality’ = Standard deviation of test scores
‘Low achievers’ = 10th percentile of test distribution
‘High achievers’ = 90th percentile of test distribution
‘Low SES’ = 0 – 25 books in the home
‘High SES’ = More than 200 books in home
(i) Measuring construct of interest?
(ii) Measurement error?
23
Methods‘Average’ = OLS regression (allowing for clustering). Two-sample
t-test for statistical significance
‘Low achievers’ = Quantile regression at 10th percentile. Bootstrapped standard errors to allow for clustering.
‘High achievers’ = Quantile regression at 90th percentile. Bootstrapped standard errors to allow for clustering.
SES differences in achievement = OLS regression. Maths test scores as response. Books in home as key explanatory variable. Controls for gender and immigrant status.
24
Results
…. we are just as far behind at age 9/10 as at 15/16 (on average)
Gap between England and East Asia does not widen between 9/10 and 15/16
(in terms of our relative position)
Age 9/10 Age 13/14 Age 15/16
-0.2
0
0.2
0.4
0.6
0.8
1
Hong KongJapanSingaporeEnglandTaiwan
(Ave
rage
test
sco
re (
inte
rnat
iona
l z-s
core
s)
….(some evidence) low achievers catch-up (age 10 – age 16)
-0.50
-0.30
-0.10
0.10
0.30
Taiwan
Hong Kong
Japan
Singapore
England
Low achievers in England may improve relative to low achievers in East Asian countries between primary and secondary school
…. But the high achievers may lose some ground
High achievers in England may fall back relative to those in East Asian countries between primary and secondary school
-0.3
-0.1
0.1
0.3
0.5
0.7
Taiwan Hong Kong Japan
Singapore
England
Ch
an
ge
(in
tern
ati
on
al
sta
nd
ard
dev
iati
on
)
Age 9/10 Age 13/14 Age 15/160.2
0.4
0.6
0.8
1.0
1.2
EnglandSingaporeJapanTaiwanHong KongCross-country average
Soc
io-e
con
omic
gra
die
nt
in t
est
scor
esSES gap in math test scores
Big gap in England….…seems to increase between 10 and 16
….but this holds true in all countries
Caution neededDifficulties with measurement of SES (and with books in the home)
29
Conclusions
Suggests (but does not conclusively prove) that....
• ‘Poor’ PISA ranking not about failing secondary schools...- Just as far behind East Asian countries before children each secondary school (on average)
• .......though still possible some changes are needed- E.g. Curriculum / incentives to stretch the highest achievers more?
• Focus should perhaps be earlier years???- Primary school? Early childhood?
• .... but still can’t rule out ‘culture’ as the dominant force?
- Likely to take hold early in life.
- Hence consistent with the large cross-national differences at age 10.