using cem data in practice: feedback from cem secondary assessments glasgow conference 13 th feb...
TRANSCRIPT
Using CEM Data in Practice:Feedback from CEM Secondary
Assessments
Glasgow Conference13th Feb 2013
Peter Hendry: CEM Consultant
[email protected]@cem.dur.ac.uk
Assessment for Excellence: 2 systems
S1/S2 Baseline Assessment (MidYIS):
A Test of Developed Ability at start of S1 or S2
• 4 sections: Vocabulary, Maths, Non-Verbal and Skills (Proof Reading,
and Perceptual speed and accuracy
• Computer adaptive* or paper-based
• Any time Term 1 + catch ups
• Value Added measures to SOSCA and to Scottish Qualifications
S2 Curriculum-based Assessments (SOSCA):A curriculum assessment of knowledge, skills and understanding at end of S2
• Assesses what has been taught in the classroom
• Covers, Maths, Science, Reading (English)
• Computer adaptive*
• Four week assessment window mid April to mid May + catch ups
• Value Added measures from PIPs, InCAS and MidYIS and to Scottish Qualifications
• School and students’ test performances are compared to a nationally representative sample
• Assessment data from the tests provides school and pupil data to inform teaching and learning
• ‘Predictive’ data indicates future potential performance to SQ, and will to National 4 and 5, to inform target setting and subsequent monitoring
• ‘Value-added data’ provides measures of relative performance at SQ (and will for National 4 and 5) and SOSCA from earlier baselines to inform self-evaluation
AfE: key elements
AfE Feedback: MidYIS and SOSCA
Assessment Feedback:
• Standardised score system: 50-150, mean 100, SD 15
• School data including component averages and band profiles (historical and current)
• Pupil data in the form of spreadsheets and Individual Pupil Records (IPRs)
AfE Feedback: MidYIS and SOSCA‘Predictions’ Feedback:
• Currently to S2 SOSCA from S1/S2 baseline (MidYIS) and to SQ from MidYIS and SOSCA
‘Value-Added’ Feedback:
• To SQ from curriculum baseline (SOSCA), and from S1/S2 baseline (MidYIS)
• To S2 curriculum assessment (SOSCA) from P7 baseline (PIPS or InCAS) and/or S1/S2 baseline (MidYIS)
www.cemcentre.org
Data for your school.
Detailed Information about the assessment. Well worth a look.
Bands, percentiles, standardised scores…
0
50 60 70 80 90 100 110 120 130 140 150
Standardised scores
DC B
A
105 20 301 40 90 958070 996050Percentiles:
25%
25% 25%
25%
98
Mean 100, Standard Deviation of 15
MidYIS Year 8/S1 Test Academic Year 2010/2011
Scores Standardised On A Nationally Representative Sample of Schools
Standardised Scores Sort Data By...
Mean Score: 100
Standard Deviation: 15
Vocabulary Maths Non Verbal Skills MidYIS Score
Average 93.3 89.9 90.9 97.3 90.7
St. error 1.2 1.2 1.2 1.2 1.2
Surname Forename CEM Id Form Sex Cust
JAMES M 81 D 69 D 75 D 68 D 72 D
COREY M 76 D 61 D 64 D 57 D 65 D
JOHN M 93 C 87 D 101 B 92 C 89 D
SAMUEL M 88 D 91 C 82 D 91 C 88 D
DANIELLE F 117 A 98 C 107 B 105 B 109 B
FRAZER M 98 C 89 D 98 C 117 A 93 C
Sort Data By...
SCHOOL A: S1 baseline dataMidYIS Year 8/S1 Test Academic Year 2010/2011
Standardised Scores Sort Data By...
Mean Score: 100
Vocabulary Maths Non Verbal Skills MidYIS Score
St.
Sco
re
Ban
d
St.
Sco
re
Ban
d
St.
Sco
re
Ban
d
St.
Sco
re
Ban
d
St.
Sco
re
Ban
d
Average 93.3 89.9 90.9 97.3 90.7
St. error 1.2 1.2 1.2 1.2 1.2
Surname Forename CEM Id Form Sex Cust
JAMES M 81 D 69 D 75 D 68 D 72 D
COREY M 76 D 61 D 64 D 57 D 65 D
JOHN M 93 C 87 D 101 B 92 C 89 D
SAMUEL M 88 D 91 C 82 D 91 C 88 D
DANIELLE F 117 A 98 C 107 B 105 B 109 B
FRAZER M 98 C 89 D 98 C 117 A 93 C
Sort Data By...
SCHOOL A: S1 baseline dataIntake profiles: overall and test component bands
A school with a ‘completely average’ intake would have 25% of pupils within each band.
MidYIS Year 8/S1 Historical Band ProfilesScores Standardised on a Nationally Representative Sample of Schools
Year Band A Band B Band C Band D
2013 5 12 17 65
2012 5 18 28 49
2011 6 18 30 46
2010 11 20 29 41
2009 8 18 38 36
Percentage of pupils falling into each MidYIS Band over time.(Band A = Top 25% of the sample, Band D = Bottom 25% of sample.)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2009 2010 2011 2012 2013
Pe
rce
nta
ge
MidYIS Baseline Testing Year
Historical Band Profiles: Overall Score(Nationally Representative Standardisation)
Band D
Band C
Band B
Band A
SCHOOL A: S1 Baseline
End Year2009 2013OverallTest Section Start Year
Show Band Percentage Values
AfE S2 Curriculum based Standardised Feedback 2012
Scores Standardised On AfE Curriculum based Sample 2012
Standardised Scores
Mean Score: 100
Standard Deviation: 15
Average94.2 98.1 97.2 95.9 92.5 92.6 92.6 92.1 96.9 96.1 97.6 96.9
St. error 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3
Surname Forename AfE Id Form Sex Cust
James M 94 C 77 D 87 D 84 D 88 D 78 D 89 D 84 D 84 D 104 B 91 C 92 C
Corey M 71 D 77 D 90 D 75 D 66 D 78 D 63 D 66 D 50 D 57 D 74 D 50 D
John M 76 D 101 B 87 D 86 D 50 D 50 D 53 D 50 D 78 D 111 A
Samuel M 81 D 94 C 78 D 81 D 50 D 66 D 84 D 73 D 90 D 79 D
Danielle F 115 A 119 A 110 B 117 A 105 B 100 B 90 C 98 C 137 A 139 A 121 A 138 A
Frazer M 85 D 106 B 90 D 92 C 96 C 91 C 100 B 95 C 107 B 96 C 95 C 99 C
Sort Data By...
SCHOOL A
AfE S2 Curriculum based Standardised Feedback 2012
Standardised Scores
Mean Score: 100
Standard Deviation: 15
St. S
core
Ban
d
St. S
core
Ban
d
St. S
core
Ban
d
St. S
core
Ban
d
St. S
core
Ban
d
St. S
core
Ban
d
St. S
core
Ban
d
St. S
core
Ban
d
St. S
core
Ban
d
St. S
core
Ban
d
St. S
core
Ban
d
St. S
core
Ban
d
Average94.2 98.1 97.2 95.9 92.5 92.6 92.6 92.1 96.9 96.1 97.6 96.9
St. error 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3
Surname Forename AfE Id Form Sex Cust
James M 94 C 77 D 87 D 84 D 88 D 78 D 89 D 84 D 84 D 104 B 91 C 92 C
Corey M 71 D 77 D 90 D 75 D 66 D 78 D 63 D 66 D 50 D 57 D 74 D 50 D
John M 76 D 101 B 87 D 86 D 50 D 50 D 53 D 50 D 78 D 111 A
Samuel M 81 D 94 C 78 D 81 D 50 D 66 D 84 D 73 D 90 D 79 D
Danielle F 115 A 119 A 110 B 117 A 105 B 100 B 90 C 98 C 137 A 139 A 121 A 138 A
Frazer M 85 D 106 B 90 D 92 C 96 C 91 C 100 B 95 C 107 B 96 C 95 C 99 C
Text
C
ompr
ehen
sion
Pas
sage
C
ompr
ehen
sion
Ove
rall
Rea
ding
Num
ber,
Mon
ey &
M
easu
rem
ent
Info
rmat
ion
Han
dlin
g
Reading Mathematics Science
Sha
pe, P
ositi
on &
M
ovem
ent
Ove
rall
Mat
hem
atic
s
Livi
ng T
hing
s &
th
e P
roce
sses
of
Life
Ear
th &
Spa
ce
Ene
rgy
& F
orce
s
Ove
rall
Sci
ence
Spe
ed R
eadi
ng
Sort Data By...
SCHOOL A: S2 curriculum (SOSCA) data
School no.1
School no.2
50 60 70 80 90 100 110 120 130 140 150
Standardised Test Score
Stanine 1
2 3 4 5 6 7 8 9
4% 4 % 7% 7% 12% 12% 17% 17% 20%
STANINES
Individual Pupil Record DANIELLE Date Of Birth: 22/02/99 Student Number:Sex: F Form:
Band Stanine PercentileStandard
ScoreVocabulary A 7 87 117
Maths C 5 45 98
Non-Verbal B 6 67 107
Skills B 6 63 105
MidYIS Score B 6 73 109
Band 25% of nationally representative sample is in each band (A = highest)
Stanine One of nine divisions of the Normal distribution (9 = highest)
Percentile Percentage of the nationally representative sample scoring less than this student
Standard Score Score standardised against a nationally representative sample
50
60
70
80
90
100
110
120
130
140
150
Vocabulary Maths Non-Verbal Skills MidYIS Score
Na
tio
na
lly
Sta
nd
ard
ise
d S
core
s
Nationally Standardised Scores with 95% Confidence Band
S1 baseline IPR
Individual Pupil Record DanielleDate Of Birth: 22/2/1999 Student Number:Sex: F Form:
AfE Band Stanine PercentileAfE Standardised
Score
Speed Reading A 7 84 115
Text Comprehension A 8 90 119
Passage Comprehension B 6 74 110
Overall Reading A 7 87 117
Number, Money & Measurement
B 6 62 105
Information Handling B 5 51 100
Shape, Position & Movement
C 4 26 90
Overall Mathematics C 5 46 98
Living Things & the Processes of Life
A 9 99.3 137
Earth & Space A 9 99.5 139
Energy & Forces A 8 92 121
Overall Science A 9 99.4 138
S2 curriculum (SOSCA) IPR
Individual Pupil Record DanielleDate Of Birth: 22/2/1999 Student Number:Sex: F Form:
S2 curriculum (SOSCA) IPR
• Currently available to Standard Grade from AfE baselines
• In the future to National 4 & 5 Qualifications from AfE baselines
• Also to AfE S2 curriculum (SOSCA) from S1 and/or S2 baseline (MidYIS)
• Includes Chances Graphs
Predictions Feedback
BASELINE SCORE
G
RA
DE
G
RA
DE
***********************
*******************************
*******************************************
*************************************
**************************************
********************************
*****************************
*****************
Trend line (regression line)
3
2
1
4
How is a ‘prediction’ generated?50% on or above the trend line
50% on or below the trend line
‘PREDICTION’ (expected grade)
Data for 2012 examinationsG
rade
Predictions & Chances Graphs to Standard Grade subjects
based on 2012 AfE scores
Please refer to the Information Sheet for details of
the method used to make these predictions
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
Forename Sex Form AfE Id Cust AfE
Rea
ding
Sco
re
AfE
Mat
hs S
core
AfE
Sci
ence
Sco
re
Ove
rall
AfE
Sco
re
Adm
inis
trat
ion
Art
& D
esig
n
Bio
logy
Bus
ines
s M
anag
emen
t
Che
mis
try
Com
putin
g S
tudi
es
Cra
ft &
Des
ign
Dra
ma
Eng
lish
Fre
nch
Geo
grap
hy
Ger
man
James M 84 84 92 85 3.8 3.1 4.3 3.3 3.8 3.8 3.3 3.0 3.4 4.0 4.2 4.2 3.7 3.9 3.1 4.5 3.7 2.5 2.8 4.1 3.7
Corey M 75 66 50 54 5.9 4.2 6.6 5.4 5.6 5.6 4.7 4.3 4.8 5.9 6.4 6.0 5.2 5.9 4.5 6.8 5.7 3.5 4.0 6.0 5.2
Kieran Roland M 68 60 72 63 5.3 3.9 5.9 4.8 5.1 5.1 4.3 3.9 4.4 5.4 5.8 5.5 4.8 5.4 4.1 6.2 5.2 3.2 3.7 5.5 4.8
Megan F 100 89 93 93 3.3 2.8 3.6 2.7 3.3 3.3 2.9 2.6 3.0 3.5 3.6 3.7 3.3 3.3 2.7 3.9 3.2 2.2 2.5 3.6 3.3
Danielle F 117 98 138 120 1.5 1.8 1.5 1.0 1.7 1.7 1.7 1.4 1.8 1.9 1.7 2.1 2.0 1.6 1.4 1.8 1.4 1.3 1.4 1.9 2.0
Frazer M 92 95 99 95 3.2 2.7 3.5 2.6 3.2 3.2 2.8 2.5 2.9 3.4 3.5 3.6 3.2 3.2 2.6 3.7 3.1 2.2 2.4 3.5 3.2
Teacher's Adjustment
(Levels)
Standard Grade Point Predictions Standard Grade Point PredictionsPoints To Grades Adjust
based on 2012 AfE scores
Please refer to the Information Sheet for details of
AfE S2 Curriculum based Assessments 2012
James M 84 84 92 85 4 3 4/5 3/4 4 4 3/4 3 3/4 4 4 4 3/4 4 3 4/5 3/4 2/3 3 4 3/4
Corey M 75 66 50 54 6 4 6/7 5/6 5/6 5/6 4/5 4/5 5 6 6/7 6 5 6 4/5 7 5/6 3/4 4 6 5
Kieran Roland M 68 60 72 63 5/6 4 6 5 5 5 4/5 4 4/5 5/6 6 5/6 5 5/6 4 6 5 3 3/4 5/6 5
Megan F 100 89 93 93 3/4 3 3/4 2/3 3/4 3/4 3 2/3 3 3/4 3/4 3/4 3/4 3/4 2/3 4 3 2 2/3 3/4 3/4
Danielle F 117 98 138 120 1/2 2 1/2 1 1/2 1/2 1/2 1/2 2 2 1/2 2 2 1/2 1/2 2 1/2 1/2 1/2 2 2
Frazer M 92 95 99 95 3 2/3 3/4 2/3 3 3 3 2/3 3 3/4 3/4 3/4 3 3 2/3 3/4 3 2 2/3 3/4 3
Predictions: Chances Graphs50% chance of a grade 2 – the most likely single grade. 50% chance of a different grade
Chances Graphs based on Pupil’s SOSCA Test Score
Point Prediction =1.8
Grade Prediction = 2
Not a label for life...just another piece of information
• The Chances graphs show that, from almost any baseline score, students come up with almost any grade - - -there are just different probabilities for each grade depending on the baseline score.
• In working with students these graphs are more useful than a single predicted or target grade
• Chances graphs show what can be achieved: – By students of similar ability– By students with lower baseline scores
Latest information as at 13 Feb 2102• Replacing SG with Nat 4 & 5 predictions:
CEM statisticians have put together a proposal.
• This has been looked at by colleagues in Scottish Authorities.
• The CEM Technical team have now published the new predictive data.
• This data is now available on the CEM secondary+ website alongside the existing SG predictive data
The equivalence we have assumed between CfE outcomes and Standard Grade outcomes isoutlined below:
Standard Grade
National Level 3National Level 2
N3N2
Where a subject is only available at National Level 4 (e.g. Business), only grades N4, N3, N2will be reported.
Credit Level 1Credit Level 2
General Level (3 and 4)Foundation Level (5 and 6)
Standard grade level 7
National Level 4D
N4
For each student in each CfE subject, a predicted grade is given. The range of predictedgrades is as follows
Grade
National LevelNational Level 5 A,BNational Level 5 C,D
National Level 4National Level 3National Level 2
For subjects only available at National Level 5 (e.g. Accounting), the full range of grades willbe reported. Whilst we recognise that grades N4, N3, N2 are not available we have includedthem in the chances graphs to aid the teacher in supporting the student in making realisticsubject choices.
National Level 5 ANational Level 5 BNational Level 5 CNational Level 5 D
CEM Predicted GradeABC
AfE
Rea
din
g S
core
AfE
Mat
hs
Sco
re
AfE
Sci
ence
Sco
re
Ove
rall
AfE
Sco
re
Acc
ou
nti
ng
(N
5)
Ad
min
istr
atio
n &
It
Art
& D
esig
n
Bio
log
y
Bu
sin
ess
(N4)
Bu
sin
ess
Man
agem
ent
(N5)
Car
e
Ch
emis
try
Co
mp
uti
ng
Sci
ence
Dan
ce (
N5)
Des
ign
& M
anu
fact
ure
Dra
ma
CfE Grade Predictions
New slide!!
95 104 99 99 D N4 N4/D N4/D N4 D N4/D N4/D N4/D N4/D N4/D N4/D
102 103 116 108 C D D D N4 C D D D D D D
115 112 123 119 B C D C N4 B C C C C D C
125 135 117 129 A B C B N4 A B B B B C B
Predictions & Chances Graphs for CfE National Qualifications
based on 2012 AfE scores
The grade 'predictions' and chances data presented here are simulated based on the relationship between AfE S2 Curriculum based,
Standard Grade and Curriculum for Excellence levels, in the absence of any actual CfE examination data being available.
They should therefore be treated with caution. The relationships will be updated once the first cohort are awarded CfE qualifications in 2014.
AfE S2 Curriculum based Assessments 2012
.
4 4.5 5 5.5 6 6.5 7 7.5 820
40
60
80
100
120
Subject X
Out
com
e
0 2 4 6 8
-ve VA+ve VA
Trend Line/Regression Line
Raw Residual
Measuring Value-Added – Terminology
Exam
gra
de
BASELINE SCORE
50
60
70
80
90
100
110
120
130
140
150
160
170
50 60 70 80 90 100 110 120 130 140 150 160 170
AfE
Sco
re
MidYIS Test Score (Standardised on a Nationally Representative Sample)
Year 8 MidYIS Score 2010/2011 Against AfE Overall Mathematics 2012
SCHOOL A: S2 curriculum (SOSCA) data
SCHOOL C:AfE data
Standardised Residual Bar Chart
99.7% confidence limit
95% confidence limit
ANY VALUE IN THE INNER SHADED AREA SHOWS THE EXPECTED RESULTS i.e. AVERAGE VALUE ADDED
SCHOOL C:AfE data
SCHOOL C:SQ data