1 the appropriate use of naplan data national symposium, 23 july, 2010 margaret wu university of...
TRANSCRIPT
1
The Appropriate Use of The Appropriate Use of NAPLAN DataNAPLAN Data
National Symposium, 23 July, 2010
Margaret WuUniversity of [email protected]
1
2
NAPLAN TestsNAPLAN Tests
Conducted once a yearAbout 40 test questions per subject
areaTest scores are used to infer
◦the achievement levels of studentsHow reliable can NAPLAN test scores
reflect◦Student achievement level?◦School performance?
2
3
Margin of error in measuring Margin of error in measuring student performancestudent performance
David - a Grade 5 student in 2008. Reading score was 25 out of 40.David’s reading test scores could vary
between 20 and 30, out of 40.◦if similar tests are administered (e.g.,
2009, 2010 tests )One test collects only a small sample
of performance.Variation in scores is called
Measurement Error.3
4
How big an error size is acceptable?How big an error size is acceptable?
The answer is◦It depends.
An example◦Effectiveness of a weight loss program◦Expect a loss of 0.5 kg after one week.◦Measurement scale is accurate to 1kg.◦Not good enough for measuring
individual change◦OK for a group change, if group size is
‘large’.
4
5
On the NAPLAN scale…On the NAPLAN scale…NAPLAN 2008 reading scores
200
300
400
500
600
700
800
grade 3 grade 5 grade 7 grade 9
2.5%tile
mean
97.5%tile
5
6
On the NAPLAN scale…On the NAPLAN scale…NAPLAN 2008 reading scores
200
300
400
500
600
700
800
grade 3 grade 5 grade 7 grade 9
2.5%tile
mean
97.5%tile
6
7
Measuring GrowthMeasuring GrowthNAPLAN 2008 reading scores
200
300
400
500
600
700
800
grade 3 grade 5 grade 7 grade 9
2.5%tile
mean
97.5%tile
Growth measure?
Margin of error of growth measure
± 76 points
Expected growth is 50 points
7
8
Class mean scoresClass mean scoresAverage score for a class
◦Effect of measurement error reducesNew source of error
◦Sampling errorCohort of students changes from year
to yearVariation in class mean score because
of the sample of students in a classClass mean ± 20 points
◦(1 year’s growth)
8
9
Teacher effectTeacher effectA high performing teacher can
raise student standards by one more year of growth as compared to a low performing teacher.
excellent teacher
average teacher
poor teacher
50 points
Margin of error of teacher effect based on two testing occasions: ± 20 points
NAPLAN 2008 reading scores
200
300
400
500
600
700
800
grade 3 grade 5 grade 7 grade 9
2.5%tile
mean
97.5%tile
9
10
MySchool WebsiteMySchool Website
It is a league table◦It compares and ranks schools
It is the worst kind of league table◦Because it is claimed that the red
bars reflect “underperforming schools”
◦Simple league tables do not have this claim.
11
Summary - 1Summary - 1NAPLAN results are NOT suitable
for measuringStudent achievement level
◦beyond a rough “lower”, “average”, “higher” groups
Student progress Teacher effect School performance
11
12
Summary - 2Summary - 2NAPLAN results are for the
systems, e.g.◦Compare girls and boys◦Compare rural and urban◦Trends, if equating design is improved
NAPLAN results should NEVER be published.
Parents/caregivers should not be encouraged to use the results to judge schools.
12
13
Finally…Finally…Conflicting advice from different
experts?An easy way to check out:Ask proponents of MySchool
website to publicly name one underperforming school.
13
14
ReferencesReferencesWu, M.L. (2010). Measurement,
sampling and equating errors in large-scale assessments. Educational Measurement: Issues and Practice, (In press: Volume 29 Number 4).
Nye, B., Konstantopoulos, S., & Hedges, L. (2004). How Large Are Teacher Effects? Educational Evaluation and Policy Analysis, Vol. 26, No. 3 (Autumn, 2004), pp. 237-257 .
15
Leigh, A. (2009). Estimating teacher effectiveness from two-year changes in students’ test scores. Economics of Education Review.
Byrne, Coventry, Olson, Wadsworth, Samuelsson, Petrill, Willcutt and Corley. (2009). Teacher Effects in Early Literacy Development: Evidence From a Study of Twins. Journal of Educational Psychology, 2009.