advanced users – midyis, yellis & alis durham 2013
TRANSCRIPT
Advanced Users –
MidYIS, Yellis & ALIS
Durham 2013
Understanding the Students
Introduction to the Test Data
Underlying Principle
If we measure a student’s ability we can determine ‘typical progress’ for the individual and use this to inform likely outcomes and against which to measure performance of
individuals and groups.
Q. How does this work ?
Q. How do we measure and interpret ‘ability’ ?
Q. How do we interpret the data fairly and reliably ?
Measuring and Interpreting Ability
Options
1) Use Pre-existing qualification data Post-16 – Average GCSE
2) Use Baseline TestPost-16 and Pre-16 – Computer Adaptive Baseline Test
Note: Issues regarding use of CABT alongside Average GCSE at Post-16 will be examined later in the day with predictive information.
Adaptive approach
Low Average High
Baseline Test Standardisation
• Test scores are standardised; Mean = 100, SD = 15
Standardised Score
National Percentage
Comment
>130 Top 2.5% Traditional classification of ‘mentally gifted’
>120 Top 10%
>100 Top 50%
<80 Bottom 10%
<70 Bottom 2.5% Potential special educational needs ??
50 60 70 80 90 100 110 120 130 140 150
Standardised Test Score
Stanine 1
2 3 4 5 6 7 8 9
4% 7% 12% 17% 20% 17% 12% 7% 4%
Band A25%
Band D25%
Band C25%
Band B25%
105 20 301 40 90 958070 996050
Percentiles:
Average
Above AverageBelow Average
G & T ??SEN ??
Cohort Ability
Intake Profiles
Intake Profiles
Intake Profiles (Historical)
Student Ability
IPRs
Individual Pupil Record Sheets (IPRs)
Look for sections that are inconsistent
Two studentsSame AbilityDifferent Profiles
General IPR Patterns
• Pupils with high scores across all components• Pupils with low scores across all components• Pupils with significant differences between one or two components
• Vocab lower than others• Vocab higher than others• Maths higher than others• Maths lower than others• Non-Verbal higher than others• Non-Verbal lower than others• Low Skills• High Skills
www.cem.org/midyisiprbooklet
Vocab significantly lower than other sections
• English Second Language ?• Understanding language used in learning and assessment ?• Language enrichment ?
Vocab significantly higher than other sections
• Good Communicator ?• Work in class may not be to this standard
→ Weak Non-verbal→ Weak Maths→ Weak Skills (speed of working ?)
• Many benefit from verbal descriptors ?
Maths significantly higher than other sections
• Strong Maths ability• Not 100% curriculum free• May depend on prior teaching effectiveness• Far East influence ?
Maths significantly lower than other sections
• Implications not just for maths but other numerate or data-based subjects
• General poor numeracy ?• Remedial Maths ?
Non-Verbal significantly higher than other sections
• Good spatial and non-verbal ability• May have high specific skills• Low Vocab, Maths & Skills may indicate has difficulty
communicating• Frustration ?
Non-Verbal significantly lower than other sections
• Difficulty understanding diagrams or graphical instructions ?• Verbal explanation ?• Physical demonstration ?• Physical Models ?
Low Skills Scores
• Skills = Proof Reading and Perceptual Speed & Accuracy• Speed of Working• Work well in class / homework but underachieve in exams ?• Problems checking work or decoding questions ?• Low Skills + Low Vocab
→ Poor written work in class (unable to work quickly) → Dyslexia ? Further specialist assessment required
High Skills Scores
• Skills = Proof Reading and Perceptual Speed & Accuracy• Can work quickly and accurately• Difficulty communicating and expressing ideas ?• May perform poorly in areas using numeracy skills and subjects
needing 3D visualisation and spatial concepts ?• May struggle in most areas of curriculum
Working with Individual Pupil Records (IPRs)
• To gain understanding of interpreting IPRs • To share strategies for supporting individual pupils
Objectives
Strategy
• To look first at interpretation of MidYIS IPRs
• Exercises with MidYIS IPRs
• Use generic patterns to apply to exercises on IPRs with ALIS, Yellis and INSIGHT though there are slight differences
What does the MidYIS test measure?
VocabularyMost culturally linked. Affects all subjects but most important in English,
History and some Foreign Languages. Measures fluency rather than knowledge.
MathsThe Maths score is well correlated with most subjects but is particularly
important when predicting Maths, Statistics, ICT, Design & Technology and Economics.
Non-verbalTests 3D visualisation, spatial aptitude, pattern recognition and logical
thinking. Important when predicting Maths, Science, Design & Technology, Geography, Art and Drama.
SkillsTests proof reading skills (SPG) and perceptual speed and accuracy
(e.g. matching symbols under time pressure). Measures fluency and speed necessary in exams and in the work place. Relies on a pupil’s scanning and skimming skills.
Using MidYIS IPRs to Inform Teaching and Learning
The IPR on its own simply tells us about the relative performances of the pupil on the separate sections of the test, where the pupil
is strong, where performance has been significantly above or below national averages or where the pupil has significantly
outperformed in one section or another.
It is when the IPR is placed in the hands of a teacher who knows that pupil that it becomes a powerful tool.
It is what teachers know about individual pupils: what has happened in the past, how they respond to
given situations and how they work in the teacher’s specific subject that inform the interpretation of the IPR.
If the IPR data from MidYIS, the teacher’s personal and subject specific knowledge and experiences regarding
the pupil can be shared, then there becomes a much more powerful instrument for supporting pupils’ learning needs.
Examples of Individual Pupil Profiles
For each example look atthe information contained in the graphthe issues that may arise for this pupil in your
subjectstrategies you could employ to support that
pupil (either for the whole class or for that specific individual)
Student A
Strategies for Student A
Word banks for each topicPractise writing with words rather than symbols e.g. To find the common denominator, first of all you ...Discussion groups (although ensure pupils with low vocabulary scores do not all congregate)Wider readingVisits/trips etc. to enrich language and cultural experience
Student B
May struggle to understand diagrams – use spoken and written explanations, paired work or group work to interpretPhysical/practical/kinesthetic explanations may help (e.g. modelling solar system with clay/string or demonstrating distance between planets on football pitch etc.)Use drama/active methods to demonstrate difficult concepts
Strategies for Student B
Student C
Pupil may seem more able than is the case, e.g. ‘talks a good talk’Allow paired work or group discussion to communicate answers orallyDescribe maths problems Encourage leadership roles as well as debates/drama Support with scaffolding/writing frames etc.
Strategies for Student C
Student D
AnalysisA pupil like this may:struggle to proof read his work, therefore achieve a lower grade than he seems capable ofstruggle to interpret or understand exam questionseither work slowly with more accuracy OR work quickly with less accuracy – result is similar i.e. lower test score than expected
Strategies:allow extra timepractise timing e.g. clock on IWB use a range of question words to develop ability to understand instructionsdevelop proof reading technique e.g. spotting comon errorsconsider further testing for dyslexia
Strategies for Student D
2009Vocabulary Maths Non-Verbal Skills Overall
Score Band Score Band Score Band Score BandScor
eBan
d
Pupil 01 122 A 125 A 116 A 107 B 126 A
Pupil 02 105 B 110 A 127 A 95 C 108 B
Pupil 03 105 B 93 C 110 B 89 D 99 C
Pupil 04 91 C 116 A 130 A 115 A 103 B
Pupil 05 111 A 144 A 122 A 103 B 129 A
Pupil 06 107 B 112 A 85 D 97 C 109 B
Pupil 07 115 A 106 B 100 C 86 D 112 A
Pupil 08 141 A 137 A 132 A 135 A 143 A
Pupil 09 104 B 92 C 105 B 109 B 98 C
Pupil 10 99 C 119 A 114 A 99 C 109 B
Pupil 11 108 B 126 A 130 A 140 A 118 A
Pupil 12 106 B 123 A 120 A 105 B 116 A
Pupil 13 103 B 96 C 103 B 104 B 99 C
Pupil 14 108 B 110 B 112 A 108 B 110 A
Pupil 15 95 C 104 B 103 B 122 A 99 C
Some pupil data MIDYIS
A useful quick reference for staff
The class from Waterloo Road
A Selection Of MidYIS Scores For ‘Waterloo Road’
Vocabulary Maths Non Verbal Skills MidYIS Score
St. Score Band St. Score Band St. Score Band St. Score Band St. Score Band
Surname Sex
A F 81 D 110 B 108 B 112 A 94 C
B F 128 A 107 B 105 B 94 C 120 A
C M 106 B 121 A 103 B 90 D 114 A
D F 107 B 84 D 96 C 107 B 96 C
E M 96 C 90 D 130 A 91 C 92 C
F F 86 D 86 D 120 A 74 D 84 D
G F 100 B 115 A 80 D 103 B 108 B
H F 121 A 96 C 114 A 86 D 111 A
I M 92 C 100 C 96 C 123 A 95 C
J M 100 C 105 B 100 C 99 C 102 B
K M 128 A 132 A 114 A 131 A 133 A
L M 76 D 70 D 74 D 73 D 71 D
What do I need to know/do to teach this (difficult) class of twelve pupils?
Why would this be a very challenging class to teach?
These are real anonymous scores from a number of schools around the UK
• Vocabulary scores significantly lower than other component scores• Second language? Deprived areas? Difficulty accessing curriculum.? Targeted help does work. Seen in
nearly all schools. Worth further diagnosis. Could potentially affect performance in all subjects.
• Vocabulary scores significantly higher than other component scores• Good communicators. Get on. Put Maths problems in words?
• Mathematics significantly higher than other scores• From Far East? Done entrance tests? Primary experience?
• Mathematics significantly lower than other scores• Primary experience. Use words and diagrams? Sometimes difficult to change attitude… Difficulties with
logical thinking and skills such as sequencing.
• Low Mathematics scores with high Non-verbal Scores• Use diagrams. Confidence building often needed.
• Pupils with non-verbal scores different from others (High) Frustration? Behaviour problems? Don’t do as well as good communicators or numerate pupils? Good at 3D and 3D to 2D visualisation and spatial awareness. Good at extracting information from visual images.
• Pupils with non verbal scores different from others (Low) - Peak at GCSE? A level ?
• Pupils with low Skills scores - Exams a difficulty after good coursework? Suggests slow speed of processing.
• High Skills Scores - Do well in exams compared with classwork?
• The Average Pupil - They do exist!
• High scores throughout - Above a score of 130 puts the pupil in the top 2% nationally
• Low scores throughout - Below a score of 70 puts the pupil in the bottom 2% nationally
IPR Patterns – A Summary
Interpreting IPRs Exercises
Have a look at the IPRs on the following pages.These show examples for Yellis (Year 10) and ALIS (Year 12) as well as MidYIS.
What do the scores suggest about the students and how would you use this information to aid the teaching and learning process for each of them?
1 2
3 4
5
6
Proof-Reading 88PSA 108
7
8 Yellis
Name Overall Vocab Maths Non Verbal Average A Level subjects chosen
St.Score Band St.Score Band St.Score Band St.Score Band GCSE
A 78 D 49 D 99 B 92 C na Biology, Maths, Business, Art
B 94 C 115 A 85 D 104 B na Biology, Business, Psychology, English
C 88 D 97 C 85 D 104 B 5.6 History, Psychology, English, Media
D 101 B 107 B 97 C 80 D 5.9 Business, History, English, Drama
E 104 B 87 D 112 A 116 A na Biology, Physics, Maths, Business
F 81 D 47 D 103 B 111 B na Maths, Further Maths, Business
G 93 C 113 A 84 D 113 A na Biology, Business, French, Geography
H 97 C 111 A 89 D 99 C 7 Art, English, Psychology, Religious St.
I 87 D 68 D 100 B 109 B 5.4 Maths, Geography, French, Music
J 105 B 67 D 124 A 85 D 6.1Maths, Further Maths, Psychology, Economics
K 96 C 71 D 110 A 97 C na Biology, Maths, Art, English
L 92 C 60 D 111 A 97 C na Maths, History Religious St., English
You are given data relating to an institution where students completed the ALIS computer adaptive test. They are chosen because they show significant differences between the various parts of the test. Remember scores are standardised around 100.
a) Are there any apparent mismatches between the subjects being followed and this data?
b) What support can be given to those students who have weaknesses in Vocabulary or Mathematics?
c) How might predictions made for these students be tempered in the light of the inconsistencies in the test components and missing average GCSE points scores?
Case Study 1
Case Study 2
What are the strengths and weaknesses of this A/AS level student?
To use the IPR (Individual pupil record) familiarise yourself with the terms standard score, band, stanine, percentile and confidence band.
a) Which AS/A level subjects might be avoided?
b) This student chose English, Film Studies, Music Technology and Psychology. Is this a good choice?Do you foresee any problems?
INSIGHT Pupil IPR
Band Stanine PercentileStandard
ScoreKS3
Equivalent
Speed Reading B 6 69 107 6c
Text Comprehension B 5 60 104 5a
Passage Comprehension C 5 45 98 5b
Overall Reading B 5 59 103 5a
Number & Algebra D 3 19 87 4a
Handling Data B 6 61 104 6a
Space, Shape & Measures D 4 23 89 5c
Overall Mathematics C 4 31 93 5b
Biology A 7 82 114 6a
Chemistry A 9 96 127 7a
Physics A 8 89 118 7c
Overall Science A 8 93 122 7b
Vocabulary B 6 65 106
Non Verbal B 6 70 108
Skills A 8 92 121
Overall Ability C 4 36 94
Comments?
40
60
80
100
120
140
160
Sp
eed
Rea
din
g
Tex
t Co
mp
rehe
nsio
n
Pas
sag
e C
om
preh
ensi
on
Ove
rall
Rea
ding
Num
ber &
Alg
ebra
Han
dlin
g D
ata
Sp
ace,
Sh
ape
& M
easu
res
Ove
rall
Mat
hem
atic
s
Bio
logy
Ch
emis
try
Ph
ysic
s
Ove
rall
Sci
ence
Vo
cabu
lary
No
n V
erba
l
Ski
lls
Ove
rall
Abi
lity
Sta
nd
ard
ised
Sco
res
Standardised Scores With 95% Confidence Band
Mat
hs
Looking Forwards
Introduction to ‘Predictions’
Theory
Subject X
‘Ability’ (Baseline)
Res
ultSubject X
How CEM ‘Predictions’ are made…
A* / A
C
A*ABCDE
40
60
80
100
120
140
5 6 7 8
Average GCSE
Gra
de
Photography
Sociology
English Lit
Psychology
Maths
Physics
Latin
Some Subjects are More Equal than Others….
A-Levels
>1 grade
A*ABC
A
A*
B
C
D
E
Some Subjects are More Equal than Others …
Performance varies between subjects, thus analysing and predicting each subject individually
is essential.e.g. Student with Average GCSE = 6.0
Subject Choices Predicted Grades
Maths, Physics, Chemistry, Biology
C, C/D, C/D, C/D
Sociology, RS, Drama, Media
B, B/C, B/C, B/C
F
E
D
C
B
A
A*
Test Score
GC
SE
Gra
des
Art & DesignBiologyChemistryEconomicsEnglishFrenchGeographyGermanHistoryIctMathematicsMedia StudiesMusicPhysical EducationPhysicsReligious StudiesScience (Double)Spanish
Some Subjects are More Equal than Others …
GCSE
1 grade
Feedback
Predictions – MidYIS example
Similar spreadsheets available from Yellis, INSIGHT
5.0
6.0
4.4
4.8
3.9
C
B
C/D
C
D
Adjusting Predictions in MidYIS / Yellis / INSIGHT
0.5
6.3
6.9
5.8
6.1
5.5
2 3
10
23
33
23
5
1 00
5
10
15
20
25
30
35
40
U G F E D C B A A*
Per
cen
t
Grade
Individual Chances Graph for Student no.5 - GCSE EnglishMidYIS Score 82 MidYIS Band D
Most likely grade
Prediction/expected grade: 3.8 grade D
Chances Graphs
Post-16 : CABT vs Average GCSE
Average GCSE correlates very well to A-level / IB etc, but by itself is not sufficient….
• What is a GCSE ?
• Students without GCSE ?
• Years out between GCSE & A-level ?
• Reliability of GCSE ?
• Prior Value-Added ?
The Effect of Prior Value Added
Beyond Expectation
+ve Value-Added
In line with Expectation
0 Value-Added
Below Expectation
-ve Value-Added
Average GCSE = 6 Average GCSE = 6 Average GCSE = 6
Do these 3 students all have the same ability ?
• Do students with the same GCSE score from feeder schools with differing value-added have the same ability ?
• How can you tell if a student has underachieved at GCSE and thus can you maximise their potential ?
• Has a student got very good GCSE scores through the school effort rather than their ability alone ?
• How will this affect expectation of attainment in the Sixth Form ?
• Can you add value at every Key Stage ?
Baseline testing provides a measure of ability that (to a large extent) is independent of the effect of prior treatment.
Rationale for CABT in addition to GCSE
‘Predictions’
Probability of achievingeach grade
Expected Grade
Predictions Based on GCSE
Predictions Based on Baseline Test
Which predicted grades are the most appropriate for this student ?
Step 1
75th Percentile
PriorValue-Added
Adjusting Predictions in ALIS (Paris Software)
Working with ‘Predictions’(Average performance by similar pupils in previous years)
• To gain understanding of the interpretation of ‘predictions’
• Remembering that they are not really PREDICTIONS but part of a ‘chances scenario’
• Using chances to explore the setting of targets
• Discussion of monitoring performance against targets
Objectives
stud
ent n
o.
Sex
For
m
Mid
YIS
Sco
re
Mid
YIS
Ban
d
Art
& D
esig
n
Bio
logy
Eng
lish
Fre
nch
His
tory
Mat
hem
atic
s
Sci
ence
1 M 7E 131 A 6.8 7.2 7.1 6.8 7.2 7.5 7.0
2 F 7E 120 A 6.3 6.7 6.4 6.1 6.4 6.5 6.3
3 F 7C 110 B 5.8 6.2 5.7 5.4 5.5 5.6 5.5
4 F 7J 101 B 5.4 5.8 5.1 4.8 4.9 4.9 4.9
5 M 7D 82 D 4.5 4.9 3.8 3.5 3.4 3.2 3.5
1 M 7E 131 A A A A A A A*/A A
2 F 7E 120 A A/B A/B A/B B A/B A/B A/B
3 F 7C 110 B B B B/C B/C B/C B/C B/C
4 F 7J 101 B B/C B C C C C C
5 M 7D 82 D C/D C D D/E D/E E D/E
Point and grade ‘predictions’ to GCSE
WHY ARE THE SUBJECT PREDICTIONS DIFFERENT?
Concentrate on student 4
0 0 1
5
19
39
27
9
10
5
10
15
20
25
30
35
40
45
U G F E D C B A A*
Per
cen
t
Grade
Individual Chances Graph for Student 4- GCSE EnglishMidYIS Score 101 MidYIS Band B
Teacher's Adjustment : 0 grades / levels / points
Prediction/expected grade: 5.1 grade C
Most likely grade
What are the chances a) of getting a grade C or above ?
b) of not getting a C ?
Yellis predictive data: baseline score 103 (55%)
SubjectBusiness Studies 5.4 (B/C)English 5.7 (B/C)French 5.4 (B/C)Geography 5.6 (B/C)Mathematics 5.7 (B/C)Physical Education 5.7 (B/C)Science: GCSE 5.6 (B/C)Science: GCSE Additional 5.6 (B/C)SC Religious Studies 5.3 (B/C)
Weighted Average 5.6 (B/C)
Yellis Average
Predicted SubjectBusiness Studies 5.6 (B/C)*English 5.9 (B)*French 5.6 (B/C)*Geography 5.8 (B)*Mathematics 5.9 (B)*Physical Education 6.0 (B)*Science: GCSE 5.8 (B)*Science: GCSE Additional 5.8 (B)*SC Religious Studies 5.6 (B/C)*
Weighted Average 5.8 (B)
Yellis Average
Predicted
SubjectBusiness Studies 5.4 (B/C)English 5.7 (B/C)*French 5.4 (B/C)Geography 5.8 (B)*Mathematics 5.9 (B)*Physical Education 6.7 (A/B)*Science: GCSE 6.3 (A/B)*Science: GCSE Additional 5.6 (B/C)SC Religious Studies 5.7 (B/C)*
Weighted Average 5.8 (B)
Yellis Average
Predicted
Comment?
Chances graphs MidYIS and Yellis
SituationYou are a tutor to a Year 10 pupil and you wish to help him/her to set target grades. Here is a chances graph based on the pupil’s Year 7 MidYIS test (114) and one based on the Year 10 Yellis test (58%)
MidYIS Chances Graph Yellis Chances Graph
This graph is based on the pupil’s exact MidYIS score, adjusted to include the school’s previous value-added performance.
This graph is based on one ability band and has no value-added adjustment.
a) What do the graphs tell you about this pupil’s GCSE chances in this subject (Maths)?
b) What could account for the differences between the two graphs and are these important?
How was this information produced?The MidYIS graphs are produced using the predictions spreadsheet. Select the pupil(s) and subject(s) to display or print using the GCSE Pupil Summary 1 tab. Adjustments for value-added can be made for individual subjects on the GCSE Preds tab.The Yellis graphs for all GCSE subjects (showing all four ability bands) can be downloaded from the Secondary+ website.
IMPORTANT FOR STAFF AND STUDENTS TO UNDERSTAND THE DIFFERENCE
Fixed Mindset:[My intelligence is fixed and tests tell me how clever I am.]This graph tells me I’m going to get a B, but I thought I was going to get an A. I’m obviously not as clever as I hoped I was and so the A and A* grades I’ve got for my work so far can’t really be true.
Growth Mindset:[My intelligence can develop and tests tell me how far I have got.]This tells me that most people with the same MidYIS score as me achieved a B last year, but I think I have a good chance of an A and I know that my work has been about that level so far so I must be doing well. What do I need to do to be one of the 10% who gets an A*?
From MidYIS - The most likely grade is a B (35%) but remember there is a 65% (100-65) chance of getting a different grade but also a 75% (35+30+10) chance of the top three grades. From Yellis - The most likely grade appears to be a C but remember that the band has been decided over a range, not for the individual student and this pupils score is near the top of that range, 58 compared with 60.8. It has also not been adjusted for this school’s prior value added in the past.
In an interview with the student one has to use your professional judgement about that student, taking everything into account. Certainly the Yellis chart warns against complacency, but if the school has a strong value added history it is better to rely in this case on the MidYIS chart for negotiating a target. Grade A is a fair aspirational target for the student but accountability for a teacher cannot fairly be judged by not achieving this grade with this student. Even a very good teacher may only achieve B or C with this student.Can the aspirational target set for the student be the same as that used for staff accountability purposes? There is a trap here.
Commentary
ALISYou are the subject teacher and are discussing possible A2 target grades with individual students. You are about to talk to Jonathan who achieved an average GCSE score of 6.22. This gives a statistical prediction=28.35x6.22-99.57= 77 UCAS points using the regression formula at A2 for this subject (Grade C at A2). Assume that the computer adaptive baseline test confirms this prediction. Chances graphs for this subject are shown showing the percentage of students with similar profiles achieving the various grades.
Individual chances graph for Jonathan
(b) ‘Most candidates with Jonathan’s GCSE background score achieved a C in my subject last year so Jonathan’s target grade should be a C’. What are the weaknesses of this statement?
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
(c) What other factors should be taken into consideration apart from chances graph data, when determining a target grade?
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
a) Why are these two chances graphs different?
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
The difference in the chances graphs is that one of them provides for a range of GCSE scores whilst the other is linked to Jonathan’s individual average GCSE score of 6.22. The strength of the chances graph is that it shows more than a bald prediction.
True, most students starting from an average GCSE score like Jonathan did achieve a C grade at A2 in examinations for this subject. However the probability of a B grade is also high since his score was not at the bottom of this range. This might be reflected too if the department also has a history of high prior value added. The converse is also true with a D grade probability warning against complacency. Students are not robots who will always fit with statistics so it is dangerous to make sweeping statements based on one set of results.
As well as looking at the prediction you should use the chances graph as a starting point, with your professional judgement taking into account factors such as his and the department’s previous performance in the subject, his attitude to work, what he is likely to achieve based on your own experience. You might want to start with the most popular outcome grade C and use your judgement to decide how far up (or down!) to go. He may be a very committed student and if the department has achieved high value added in the past, an A/B grade may be more appropriate though A* looks unlikely. If you are using aspirational targets for psychological reasons with students then A may be appropriate even though it less probable than B/C.
Key Questions for Intelligent Target Setting
• What type of valid and reliable predictive data should be used to set the targets?
• Should students be involved as part of the process (ownership, empowerment etc.)?
• Should parents be informed of the process and outcome?
Key points to consider might include:
Where has the data come from?What (reliable and relevant) data should we use?Enabling colleagues to trust the data: Training (staff)Communication with parents and studentsChallenging, NOT Demoralising, students…….Storage and retrieval of dataConsistency of understanding what the data means and does not mean The process of setting targets is crucial…….
There is wide-ranging practice using CEM data to set student, department and institution targets.
Increasingly sophisticated methods are used by schools and colleges.
The simplest model is to use the student grade predictions. These then become the targets against which student progress and achievement can be monitored.
Theoretically, if these targets were to be met, residuals would be zero so overall progress would be average.
The school/college would be at the 50th percentile.
More challenging targets would be those based on the basis of history. For example. Where is the school/college now? Where is your subject now?
If your subject value added history shows that performance is in the upper quartile it may be sensible to adjust targets. This may have the effect of raising point predictions between 0.2-0.5 of a grade.
This would be a useful starting point, but it would not be advisable to use the predictions for below average subjects, which might lead to continuing under-achievement.
Yellis Predictions For Modelling
FOUR approaches
• YELLIS GCSE Predictions
• YELLIS GCSE Predictions + say 0.5 a grade
• Prior value added analysis based on 3 year VA per department
• 75th percentile (upper quartile) analysis
SubjectNumber of Students
Percentage of A* to C Grades
Percentage of A* to G
Grades
Art & Design 68 84 100 5.2 (C)Business Studies 64 48 100 4.3 (C/D)Design & Technology 103 63 100 4.7 (C/D)Drama 27 85 100 5.3 (B/C)English 181 64 100 4.8 (C)English Literature 15 60 100 4.6 (C/D)French 53 64 100 4.9 (C)Geography 84 63 100 4.8 (C)German 7 71 100 5.1 (C)History 49 67 100 5.1 (C)Home Economics 48 48 100 4.5 (C/D)ICT 71 68 100 4.9 (C)Maths 180 54 100 4.5 (C/D)Music 12 67 100 5.2 (C)Physical Education 72 65 100 4.9 (C)Religious Studies 37 70 100 5.2 (C)Double Science 180 52 100 4.4 (C/D)Welsh 177 72 100 5.1 (C)
4.7 (C/D)
106 58%141 77%181 99%181 99%
98 54%93 51%36 20%
5 or more A* to C Grades inc Maths and English: 2 or more A* to C Grades - Sciences: 1 or more A* to C Grades - Modern Foreign Language:
The underlying predictions summarised here are based on expectations for an average school achieving zero value added results. Appropriate care should be taken in interpreting them within your school.
Please note that the cut-off points for grade C and grade G have been set at 4.5 and 0.5 respectively. Due to the sensitive nature of the cut off points, predictions may vary for your school if the cut off points
could be altered.
1 or more A* to C Grades: 5 or more A* to G Grades: 1 or more A* to G Grades:
Average Grade
School Average GCSE score:
Counted Performance Statistics (Based on Subject Choice Predictions)5 or more A* to C Grades:
SubjectNumber of Students
Percentage of A* to C Grades
Percentage of A* to G
Grades
Art & Design 68 84 100 5.2 (C)Business Studies 64 48 100 4.3 (C/D)Design & Technology 103 87 100 5.3 (B/C)*Drama 27 100 100 6.0 (B)*English 181 69 100 4.9 (C)*English Literature 15 67 100 4.9 (C)*French 53 96 100 6.4 (A/B)*Geography 84 73 100 5.2 (C)*German 7 86 100 5.6 (B/C)*History 49 67 100 5.1 (C)Home Economics 48 79 100 5.2 (C)*ICT 71 96 100 5.7 (B/C)*Maths 180 57 100 4.6 (C/D)*Music 12 92 100 5.7 (B/C)*Physical Education 72 65 100 4.9 (C)Religious Studies 37 70 100 5.3 (B/C)*Double Science 180 59 100 4.7 (C/D)*Welsh 177 86 100 5.5 (B/C)*
5.1 (C)
125 69% *162 89% *181 99% *181 99% *
102 56% *106 58% *
54 30% *
(*Predictions Adjusted for Positive Prior Value-added Performance)
5 or more A* to C Grades inc Maths and English: 2 or more A* to C Grades - Sciences: 1 or more A* to C Grades - Modern Foreign Language:
1 or more A* to C Grades: 5 or more A* to G Grades: 1 or more A* to G Grades:
Average Grade
School Average GCSE score:
Counted Performance Statistics (Based on Subject Choice Predictions)5 or more A* to C Grades:
SubjectNumber of Students
Percentage of A* to C Grades
Percentage of A* to G
Grades
Art & Design 68 97 100 5.5 (B/C)*Business Studies 64 63 100 4.6 (C/D)*Design & Technology 103 73 100 5.0 (C)*Drama 27 96 100 5.5 (B/C)*English 181 70 100 5.0 (C)*English Literature 15 67 100 4.9 (C)*French 53 74 100 5.1 (C)*Geography 84 70 100 5.1 (C)*German 7 71 100 5.4 (B/C)*History 49 84 100 5.4 (B/C)*Home Economics 48 63 100 4.8 (C)*ICT 71 77 100 5.2 (C)*Maths 180 61 100 4.8 (C)*Music 12 83 100 5.5 (B/C)*Physical Education 72 72 100 5.2 (C)*Religious Studies 37 81 100 5.5 (B/C)*Double Science 180 59 100 4.7 (C/D)*Welsh 177 82 100 5.4 (B/C)*
5.0 (C)
123 68% *162 89% *181 99% *181 99% *
109 60% *106 58% *
41 23% *
(*Predictions Adjusted for 75th Percentile)
5 or more A* to C Grades inc Maths and English: 2 or more A* to C Grades - Sciences: 1 or more A* to C Grades - Modern Foreign Language:
1 or more A* to C Grades: 5 or more A* to G Grades: 1 or more A* to G Grades:
Average Grade
School Average GCSE score:
Counted Performance Statistics (Based on Subject Choice Predictions)5 or more A* to C Grades:
Here is the Individual Pupil Record from the ALIS computer adaptive test taken in Year 12 for a current Year 13 student.
This student had a high positive value added in every GCSE subject as measured using MidYIS as a baseline.
( Average GCSE score 7.44)
On the next page are her A level predictions and chances graphs.
Why are the predictions different?
Are the chances graphs useful here?
Case Study
Using PARIS software and tweaking the predictions for prior value added by these subjects, then from a GCSE baseline A*s are predicted in three of the four.
If we did the same for the adaptive test baseline solid Bs might be predicted in all three.
It is also worth looking at the value added at GCSE. See commentary
Predictions and chances graphs
The differences in prediction from the GCSE baseline and the computer adaptive test for some students are interesting and these can be in either direction. Here there has been a very large value added at GCSE which may or may not be sustainable at A level. This student’s history is shown below:
Drama English English Lit German Latin Maths MusicGCSE PREDICTIONS MIDYIS ALL 5.8 6 6 5.5 6.5 6 5.9 5.8 from year7 dataGSCE Grade predictions B- B B B/C A/B B B- B-GCSE ACHIEVED A* A* A A A* A* A AVALUE ADDED RAW 2.2 2 1 1.5 1.5 2 1.1 1.2
GCSE PREDICTIONS MIDYIS IND. 6.6 6.8 6.7 6.5 6.9 6.9 6.9 6.9 from year9 dataGSCE Grade predictions A/B A- A- A/B A- A- A- A-GCSE ACHIEVED A* A* A A A* A* A AVALUE ADDED RAW IND 1.4 1.2 0.3 0.5 1.1 1.1 0.1 0.1
Average GCSE score =7.44
Science
The value added here at GCSE is between 1 and 2 grades (for all institution data at year 7) and significantly positive for subjects (for the Independent school data from year 9).Actually if we measure this student’s value added from an average GCSE score of 7.44 next year, it does not tell the whole story. We need to look as well at the value added from the computer adaptive test too.The chances graphs should be used with extreme caution here and the growth mindset is vital if used with students.
Commentary
Case study : setting departmental targets
Uses valid and reliable data e.g. chances graphsInvolves sharing data with the studentsGives ownership of the learning to the studentEnables a shared responsibility between student, parent(s)/guardian, and the teacherEncourages professional judgementLeads to the teachers working smarter and not harder
Leads to students being challenged and not ‘over supported’, thus becoming independent learners…
DEPARTMENT:
GCSE ANALYSIS
yearno. of pupils raw resid.
av. Std. Resid
2006 66 0.8 0.62007 88 0.8 0.52008 92 1.1 0.82009 108 0.7 0.6
n.b. A raw residual of 1.0 is equivalent to one grade.
TARGETS FOR 2011, using CEM predictive data and dept's prior value-addedThe target grade has a prior value-added of 0.8
predictionpred
grade targettarget grade
dept adj grade
1 M 5.4 (B/C) 6.2 B A2 F 3.8 (D) 4.6 C C3 M 3.6 (D/E) 4.4 D D4 F 4.2 (D) 5.0 C D5 M 5.7 (B/C) 6.5 B B6 F 6.5 (A/B) 7.3 A A*7 M 7.0 (A) 7.8 A* A*8 M 3.8 (D) 4.6 C C9 F 4.2 (D) 5.0 C C10 M 5.9 (B) 6.7 A B12 M 3.8 (D) 4.6 C D
etc.
0 0 03
14
36
32
14
20
5
10
15
20
25
30
35
40
45
U G F E D C B A A*
Per
cen
t
Grade
Individual Chances Graph for student A- GCSE EnglishMidYIS Score 105 MidYIS Band B
Teacher's Adjustment : 0 grades / levels / points
Prediction/expected grade: 5.4 grade B/C
Most likely grade
Student no.1 GCSE Geography
0 0 0 04
20
36
32
9
0
5
10
15
20
25
30
35
40
45
U G F E D C B A A*
Per
cen
t
Grade
Individual Chances Graph for Student A- GCSE EnglishMidYIS Score 105 MidYIS Band B
Teacher's Adjustment : 0.8 grades / levels / points
Prediction/expected grade: 6.2 grade B
Most likely grade
Student no.1 GCSE Geography
Results13192321106
COMMENTS?
Monitoring Student Progress
Monitoring students’ work against target grades is established practice in schools and colleges, and there are many diverse monitoring systems in place
Simple monitoring systems can be very effective
Current student achievement compared to the target grade done at predetermined regular intervals to coincide with, for example internal assessments/examinations
Designated staff having an overview of each student’s achievements across subjects
All parents being informed of progress compared to targets
Review of progress between parents and staff
Subject progress being monitored by a member of the management team in conjunction with the head of subject/department
A tracking system to show progress over time for subjects and students
Schools and departments use various monitoring systemsfor comparing present progress with either the target grade or in some cases the minimum acceptable grade or basic suggested grade.
Six examples from schools are shown.
If you were Polly Bolton’s Form Teacher, how would you approach a discussion with her parents at a Parents’ Evening? Should parents be told the baseline scores?
Monitoring Progress:
Polly Bolton
AUTUMN SPRING SUMMER
Subject Teach
curr
ent
grad
e
targ
et
gra
de
is:
effo
rt
conc
ern
curr
ent
grad
e
targ
et
gra
de
is:
effo
rt
conc
ern
year
10
exam
targ
et
gra
de
is:
effo
rt
conc
ern
English CB C B LIKELY 5 ORG B B SECURE 5
English Literature CB B B SECURE 5 ORG C B LIKELY 5
Maths MC E B UNLIKELY 3 HW D B POSSIBLE 4
Science CPa E C UNLIKELY 4 ORG D C POSSIBLE 4 ORG
Science Additional CPa D C LIKELY 4 ORG B C LIKELY 4 ORG
French CK C B LIKELY 4 C B LIKELY 5
History KM A A SECURE 5 B A SECURE 5
RS CG D B POSSIBLE 4 C B POSSIBLE 4
KEY - target is: SECURE LIKELY POSSIBLE UNLIKELY
effort: 5 excellent - 4 good - 3 satisfactory - 2 poor - 1 very poor
concern: WW working well - ATT attitude - BEH behaviour TEN attendance - PUN punctuality - HW homework CON confidence - ORG organisation - EAL language
Subjects
Student: Peter Hendry Department: Geology 2006-8
test: Geol Time Scale
test essay: radiometric
dating test: datinghomework rock cycle
pract: rock textures
test: igneous
rocks
target grade 15/09/2006 22/09/2006 06/10/2006 20/10/2006 06/11/2006 21/11/2006
A 97% 84%
B 68%
C 57%54%
D 50%
E
U
Tracking at departmental level for one student
subject: BIOLOGY yr 12 07-08OCT DEC MAR
SURNAME FORENAME initi
al ta
rget
gra
de
nego
ciat
ed ta
rget
gr
ade
curr
ent l
evel
effo
rt
punc
tual
ity
mee
ting
dead
lines
curr
ent l
evel
effo
rt
punc
tual
ity
mee
ting
dead
lines
curr
ent l
evel
effo
rt
punc
tual
ity
mee
ting
dead
lines
Briggs Alice C C D 1 2 1 C 1 1 1Fletcher Kevin A B B 2 2 2 B 2 1 1Green Felicity C B A 1 1 2 B 2 2 2Havard Michael A A A 3 3 4 B 4 2 2
etc
Traditional mark book approach
0.59938 -7.07013Name MidYis Score Test Score
A 80 33 41.12001 49.34402 32.89601 Astronomy 7NB 96 63 50.17065 60.20478 40.13652C 95 80 49.87096 59.84515 39.89677D 119 80 64.1362 76.96344 51.30896E 111 73 59.46104 71.35324 47.56883F 84 45 43.33772 52.00526 34.67017g 67 45 33.02838 39.63406 26.42271h 88 63 45.85511 55.02614 36.68409I 118 50 63.83651 76.60381 51.06921J 91 60 47.47344 56.96813 37.97875K 120 50 64.79552 77.75462 51.83641L 108 35 57.60296 69.12355 46.08237M 115 35 62.09831 74.51797 49.67865N 87 58 45.31567 54.37881 36.25254O 117 83 62.99738 75.59685 50.3979P 105 45 55.80482 66.96578 44.64386Q 98 73 51.54922 61.85907 41.23938R 69 5 34.4669 41.36028 27.57352S 69 30 34.10727 40.92872 27.28581T 115 70 61.91849 74.30219 49.5348U 118 50 63.71663 76.45996 50.97331V 109 45 58.32222 69.98666 46.65777W 123 60 66.47378 79.76854 53.17903X 89 30 46.03493 55.24191 36.82794Y 115 65 61.55887 73.87064 49.24709Z 76 10 38.48274 46.17929 30.78619ZA 90 55 46.57437 55.88924 37.2595ZB 97 70 50.8899 61.06789 40.71192
-7.07013 -8.484156 -5.656104-7.07013 -8.484156 -5.656104-7.07013 -8.484156 -5.656104-7.07013 -8.484156 -5.656104-7.07013 -8.484156 -5.656104-7.07013 -8.484156 -5.656104
MidYis Test Review
0
10
20
30
40
50
60
70
80
90
100
60 70 80 90 100 110 120 130
MidYis Score
Tes
t S
core
Targets for learning…. reporting to pupils
Su
rnam
e F
ore
nam
e
Gen
der
% A
tten
dan
ce Y
10
LO
ND
ON
RE
AD
ING
Mid
YIS
Ove
rall
Ban
d Y
ear
Mid
YIS
Ove
rall
Sta
nd
ard
is
Mid
YIS
Mat
hs
Sta
nd
ard
ised
Mid
YIS
Vo
cab
ula
ry S
tan
dar
Mid
YIS
No
n V
erb
al S
tan
dar
Mid
YIS
Ski
lls S
tan
dar
dis
e
AR
TA
Art
Su
bje
ct W
a
DA
TA
Des
an
d T
ech
Su
b W
a
EN
TA
En
glis
h S
ub
ject
Wa
St.
res
. MID
YIS
- K
S3
En
GE
TA
Geo
gra
ph
y S
ub
Wa
HI T
A H
isto
ry S
ub
ject
Wa
IC T
A In
f T
ech
Su
b W
a
MA
TA
Mat
hs
Su
bje
ct W
a
St.
res
. MID
YIS
- K
S3
Ma
MF
TA
MF
L S
ub
ject
Wa
MU
TA
Mu
sic
Su
bje
ct W
a
PE
TA
Ph
ys E
d S
ub
ject
Wa
SC
TA
Sci
ence
Su
bje
ct W
a
St.
res
. MID
YIS
- K
S3
Sc
SO
SC
A (
ST
A.)
Rea
din
g
SO
SC
A S
t. S
core
Bio
log
y
SO
SC
A S
t. S
core
Ch
emis
try
SO
SC
A S
t. S
core
Ph
ysic
s
SO
SC
A S
t.S
core
H.D
AT
AM
ath
s
SO
SC
A S
t.S
core
NU
MB
ER
Mat
hs
SO
SC
A S
t.S
core
SP
AC
E.M
ath
s
SO
SC
A M
ath
s
MIDYIS ON ENTRY KEY STAGE 3 STATUTORY TEACHER ASSESSMENT SOSCA STANDARDISED SCORES
J M 97.3 101 A 132 131 127 105 94 5 4 5 -2.2 6 5 6 6 -2.5 5 6 6 5 -3 92 113 98 90 103 97 95 98
C F 71.8 99 B 101 83 116 94 86 6 4 5 -0.1 5 4 3 4 -2 5 5 5 4 -1.8 96 98 83 102 87 83 95 88
Not a label for life... just another piece of information
• The Chances graphs show that, from almost any baseline score, students come up with almost any grade - - - there are just different probabilities for each grade depending on the baseline score
• In working with students these graphs are more useful than a single predicted or target grade
• Chances graphs show what can be achieved: – By students of similar ability– By students with lower baseline scores
Student 1
Student 2
Student 3
Student 4
Student 4 - IPR
Performance Monitoring
Introduction to Value-Added
Theory
Ability(Baseline)
Res
ult
Subject X
-ve VA+ve VA
Residuals
VA
How CEM ‘Value-Added’ is calculated…
Burning Question :
What is my Value-Added Score ?
Better Question :
Is it Important ?
Key Value Added Charts
1) SPC (Statistical Process Control) chart
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
Year
Performance in line with expectation
VA Score
Performance below expectationProblem with Teaching & Learning ?
Performance above expectationGood Practice to Share ?
Additional A
pplied Science
Additional S
cience
Art &
Design
Biology
Business S
tudies
Chem
istry
Design &
Technology
Dram
a
English
English Literature
French
Geography
Germ
an
History
Mathem
atics
Music
Physical E
ducation
Physics
Religious S
tudies
Science
Spanish
Short C
ourse Religious S
tudies
-4
-3
-2
-1
0
1
2
3
4
0.00.8 0.5
-0.3
1.1
-0.4
1.00.2 0.4 0.1 0.1
0.0
0.0 0.1
0.0
0.0
-0.3
0.2 0.5
-0.3
0.70.2
-2.9
0.0
Average Standardised Residuals by Subject
Ave
rag
e S
tan
dar
dis
ed R
esid
ual
2) Subject Bar Chart
Religious Studies
3) Scatter Plot
A2 – English Literature
General Underachievement ?
Scatter Plot Example 1
A2 – English Literature
Too many U’s ?
Scatter Plot Example 2
Other things to look for…
Why did these students do so badly ?
Why did this student do so well ?
How did they do in their other subjects ?
Post-16 : Impact of Baseline Choice on Value-Added
Same School - Spot the Difference ?
GCSE as
Baseline
Test as
Baseline
Does the Type of School make a Difference ?
Comparison to all schools
Comparison to Independent Schools Only
Comparison to all schools
Comparison to FE Colleges Only
Questions:
→ How does the unit of comparison used affect the Value-Added data and what implications does this have on your understanding of performance ?
→ Does this have implications for Self Evaluation ?
Using Value-Added Data
Necessary knowledge base to use CEM systems to their potential
1. The forms of Value Added Data:• scatter graphs• raw and standardised residuals• SPC charts • tables of data• use of PARIS for further analyses (e.g. by gender, teaching group)
2. Predictive Data:• point and grade predictions• importance of chances graphs• availability of different predictive data
3. Baseline Data• band profile graphs• IPRs• average GCSE score• computer adaptive tests
4. Attitudinal Data
• Make curriculum changes
• Adjust staffing structure and cater for student needs
• Self-evaluation procedures including the analysis of examination results using value added data
• The target setting process
• School and department development plans…….
• Improve your monitoring and reporting procedures
• Provide information to governors
• Have conversations with feeder primary schools
• Etc. etc.
If you have the tools you can use them to do these:
0.2
-0.6 -0.5-0.2 -0.3
-1.3
0.1
-0.1
0.1
-1.1
0.1
-0.1-0.3
0.4
-0.5 -0.5-0.9
-0.5
0.3
-4.0
-3.0
-2.0
-1.0
0.0
1.0
2.0
3.0
4.0
Art
& D
esig
n
Des
ign
& T
echn
olog
y
Eng
lish
Eng
lish
Lite
ratu
re
Fre
nch
Geo
grap
hy
His
tory
Mat
hem
atic
s
Med
ia S
tudi
es
Mus
ic
Rel
igio
us S
tudi
es
Sci
ence
: G
CS
E
Sci
ence
: G
CS
E A
dditi
onal
SC
Art
& D
esig
n
SC
Geo
grap
hy
SC
His
tory
SC
Rel
igio
us S
tudi
es
Voc
App
lied
Sci
ence
Voc
Hea
lth &
Soc
ial C
are
Ave
rag
e S
tan
dar
dis
ed R
esid
ual
GCSE value added A challenging school
Any result within the outer shaded area decreases the probability that the value added result is down to chance. The probability here is about 1 in 20.
Outside the 99.7% confidence limit chance is less than 3 in a 1000.
Below are the value added charts from Yellis to GCSE for two contrasting institutions. Which subjects are outside the confidence limits in a ‘negative value added’ way? There must be questions to ask regarding teaching and learning?Which subjects are outside the confidence limits in a ‘positive value added’ way?
0.0 0.0
0.4
0.9
0.10.2
1.4
0.4 0.3
-0.1
0.5 0.6
0.2
0.8
-0.1 -0.2
0.3 0.4
0.9
-4.0
-3.0
-2.0
-1.0
0.0
1.0
2.0
3.0
4.0
Art
& D
esig
n
Bus
ines
s S
tudi
es
Des
ign
& T
echn
olog
y
Dra
ma
Eng
lish
Eng
lish
Lite
ratu
re
Fre
nch
Geo
grap
hy
Ger
man
His
tory
Hom
e E
cono
mic
s
ICT
Mat
hs
Mus
ic
Phy
sica
l Edu
catio
n
Rel
igio
us S
tudi
es
Dou
ble
Sci
ence
Wel
sh
SC
IC
T
Ave
rag
e S
tan
dar
dis
ed R
esid
ual
GCSE value added A successful school
Here is a value-added subject report from a recent examination session at a school
GCSE
score
MidYIS score
Write the equivalent GCSE grades next to the points scores.
Compare the value-added performance of candidates scoring A*, B, and D grades.
Which result would cause you to ask questions?
Sex Band Maths Vocab Pattern YELLIS Female B 46 61 53 54
Weight Subject YELLIS
Predicted Achieved
Grade Raw
Residual Standardised
Residual 1 Drama 5.6 (B/C) 8 (A*) 2.4 1.8 1 English 5.4 (B/C) 7 (A) 1.6 1.6 1 English Literature 5.4 (B/C) 7 (A) 1.6 1.4 1 French 4.9 (C) 8 (A*) 3.1 2.5 1 Geography 5.3 (B/C) 8 (A*) 2.7 2.2 1 Maths 5.2 (C) 7 (A) 1.8 1.7 1 Physical Education 5.4 (B/C) 6 (B) 0.6 0.4 2 Double Science 5.1 (C) 8 (A*) 2.9 2.6 1 Welsh 5.6 (B/C) 7 (A) 1.4 1.0
Weighted Average 5.3 (B/C) 7.4 (A*/A) 2.1 1.8
Student A
Sex Band Maths Vocab Pattern YELLIS Male A 55 71 76 63
Weight Subject YELLIS Predicted Achieved Grade Raw
Residual Standardised
Residual 1 Business Studies 5.7 (B/C) 4 (D) -1.7 -1.4 1 Drama 6.1 (B) 6 (B) -0.1 0.0 1 English 6.0 (B) 5 (C) -1.0 -1.0 1 English Literature 6.0 (B) 5 (C) -1.0 -0.9 1 Geography 6.0 (B) 3 (E) -3.0 -2.4 1 Maths 6.0 (B) 6 (B) 0.0 0.0 1 Physical Education 6.0 (B) 2 (F) -4.0 -3.1 2 Double Science 5.8 (B) 5 (C) -0.8 -0.7 1 Welsh 6.1 (B) 5 (C) -1.1 -0.8
Weighted Average 6.0 (B) 4.6 (C/D) -1.4 -1.1
Student B
Compare the data for Student A and Student B
Find students A and B on each of the scatter graphs English and Maths
0
1
2
3
4
5
6
7
8
0 10 20 30 40 50 60 70 80 90 100
YELLIS Test Score (%)
GC
SE
Gra
de
A*
A
B
C
D
E
F
G
U
Scatter graph English
0
1
2
3
4
5
6
7
8
0 10 20 30 40 50 60 70 80 90 100
YELLIS Test Score (%)
GC
SE
Gra
de
A*
A
B
C
D
E
F
G
U
Scatter graph Maths
Is there anything to learn from these scatter graphs?
Common Scenario…
Jane has completed her 6th form studies and a review has been received for her by the college after A level results. Choose one subject at a time and look carefully at what happened in that subject both from a baseline of average GCSE grades and from a baseline of the computer adaptive test.
This student has been placed in different bands, band B from average GCSE score and band C from the computer adaptive test. This sometimes happens. It may be that the student had an off day when she did the computer adaptive test or it may be that there could have been a lot of ‘spoon feeding’ at GCSE. Jane may do better at coursework! Even though we may not know the cause it can act as a warning when analysing results though the predictions are not wildly out.
Review: Final_Result Review Review Average AverageSubject Points Grade Points Grade Residual Std. Residual(A1) Health & Social Care30.00 D 39.64 C -9.64 -0.66(A2) Religious Studies 80.00 C 89.84 B/C -9.84 -0.52(A2) English Literature 80.00 C 87.29 B/C -7.29 -0.40(A2) Drama & Theatre St100.00 B 91.84 B/C 8.16 0.46
FINAL RESULTS PREDICTIONS
RAW
STANDARDISED
Profile Sheet: Jane (from Average GCSE)Year: 2007
DOB: 01/06/89(Average GCSE = 6.00 (Band B))
Chances Graphs - Band B from average GCSE
010203040506070
U E D C B A
perc
ent
010203040506070
(A1) Health & Social Care
3 718
30 2813
010203040506070
U E D C B A
perc
ent
010203040506070
(A2) Religious Studies
0 211
3140
15
010203040506070
U E D C B A
perc
ent
010203040506070
(A2) English Literature
0 215
36 34
13
010203040506070
U E D C B A
perc
ent
010203040506070
(A2) Drama And Theatre Studies
0 110
3142
17
Individual Chances Graphs for Jane from average GCSE score
010203040506070
U E D C B A
perc
ent
010203040506070
(A1) Health & Social Care
4 1020
29 2512
010203040506070
U E D C B A
perc
ent
010203040506070
(A2) Religious Studies
0 212
3037
19
010203040506070
U E D C B A
perc
ent
010203040506070
(A2) Drama And Theatre Studies
0 110
3240
17
010203040506070
U E D C B A
perc
ent
010203040506070
(A2) English Literature
0 217
33 32
16
Profile Sheet: Jane (from Computer adaptive test)Year: 2007
DOB: 01/06/89(Online adaptive test = 0.11 (Band C))
RAW
STANDARDISED
Review Average Average Subject Points Grade Points Grade Residual St (A1)Health & Social Care 30.00 D 25.97 D/E 4.03 0.24(A2) Religious Studies 80.00 C 79.01 C 0.99 0.04(A2) English Literature 80.00 C 74.3 C/D 5.7 0.26(A2) Drama & Theatre St 100.00 B 85.16 B/C 14.84 0.70
PREDICTIONS
FINAL RESULTS
a) Did Jane reach her potential in all subjects? --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
b) Jane had been set aspirational targets prior to AS and A level by her teachers as below:Health and Social Care CDrama BReligious Studies BEnglish B Were these reasonable target grades?----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
c) Why should these grades not be used for accountability of her teachers?
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Commentary
a) Jane certainly reached her potential in Drama and Theatre Studies with positive standardised residuals by both methods. On the basis of the computer adaptive test she broadly reached potential in all subjects. On the basis of average GCSE two A level subjects were broadly down about half a grade and she dropped a grade in the AS.
b) Hopefully you agree these were reasonable target grades. Remember we don’t know the student, but use the chances graphs and if these were aspirational grades for the student then accountability of a department’s staff on that basis is not appropriate, but it certainly is on the basis of a whole class’s average standardised residuals, particularly over a number of years.
YELLIS ATTITUDINALYou are looking at the attitudinal feedback from Year 10 over time:
Do you notice a pattern between the four charts? There was an initiative in the school that contributed, but was it sustainable?
Yellis Further Comparison charts for English and Maths
What concerns you about these charts? Can you suggest which is the stronger department?
Departmental analysis (based on average GCSE score baseline)
SUBJECT A (A LEVEL)
-2.0
-1.0
0.0
1.0
2.0
20
03
20
04
20
05
20
06
20
07
20
08
20
09
20
10
20
11
Exam Year
-2.0
-1.0
0.0
1.0
2.0Final_Result
Ave
rag
e S
tan
da
rdis
ed
Re
sid
ua
l
There may be lots of reasons for changes in performance but here one factor is known by the school. The subject teacher for subject A goes on long sick leave for one of the autumn terms. When do you think this happened? To help you the AS chart is shown below for this same subject. It may or may not be relevant.
-2.0
-1.0
0.0
1.0
2.0
20
03
20
04
20
05
20
06
20
07
20
08
20
09
20
10
20
11
Ave
rag
e S
tan
da
rdis
ed
Re
sid
ua
l
Exam Year
-2.0
-1.0
0.0
1.0
2.0Final_Result
A LEVEL AS LEVEL
This school has a 5-year development plan which includes as one of its goals:
“To help students prepare for university and the world of work by developing independent learning skills, the ability to reflect and to learn from others and to maximise the benefits to learning offered by emerging technologies.”
The graphs on the next page reflect students’ perceptions of the style of learning that has been adopted in their A-level classes in two broadly similar subjects.
a) If the students’ perceptions are an accurate reflection of what takes place in the classroom, which subject seems more on board with the school’s development plan?
b) How would these perceptions inform the Senior Management Team’s evaluation of progress with its 5-year plan if the subject achieving significantly better value-added results was
i) Subject 1? ii) Subject 2?
Subject 1 Subject 2
Case study A: ALIS value-added data
Many sets of VAD are available!
From average GCSE baseline:• all ALIS cohort • type of Institution• syllabus
Also the same from the baseline test
SPC Chart with confidence limits: WHOLE SCHOOL
All ALIS Cohort Institution
Syllabus
-0.3
-0.2
-0.1
-0.0
0.1
0.2
0.3
0.4
2004
2005
2006
2007
2008
2009
2010
Ave
rage
Sta
ndar
dise
d R
esid
ual
Exam Year
-0.3
-0.2
-0.1
-0.0
0.1
0.2
0.3
0.4Final_Result
Using PARIS software: Baseline Test
Whole School
From your perspective, which set of VAD would you use for the different user groups? (Governors, HoDs, Media, Parents, SMT/SLT...)
• USE ONE YEAR’S DATA WITH CAUTION!
• Better to use three years’ data, as patterns over time are more significant.
Using data to inform leadership decisions
1. Which data do I need AND which data do I not need? (e.g. MidYIS cohort or Independent Sector)
2. What does the data mean and what does the data not mean? (e.g. staff INSET and support)
3. Who is the data for?
4. Storage, retrieval and use of data (e.g. self-evaluation and preparing for Inspection)
Some key questions:
The use of this data needs to allow us to do our best to help every student to at least achieve if not exceed their potential.
It may challenge• The culture of ‘my’ school/college • Accountability policy• Expectations • Staff training in use of data and ability to cope with data
(data overload)• Integrating the data into school procedures, storage,
retrieval, distribution and access • Roles and Responsibilities
Who should data be shared with?
Colleagues
A. Subject Teachers
B. Heads of Department
C. Pastoral Staff
D. Managers
1. This will be interpreted as a personalised prediction
2. The data doesn’t work for this particular student
3. You’re raising false expectation – he’ll never get that result
4. You’re making us accountable for guaranteeing particular grades – when the pupils don’t get them we’ll get sacked and the school will get sued
Subject Teachers/HODs
Remind them that:
1. Baseline data can give useful information about a pupil’s strengths and weaknesses which can assist teaching and learning
2. “Predictions” are not a substitute for their professional judgement
Reassure them that:
3. It is not a “witch hunt”
4. Value added data is used to assess pupil performance not teacher performance!
Subject Teachers/HODs
Pupils
1. Make sure they know why they are taking the test
2. Make sure they take it seriously3. Make sure they don’t deliberately mess it up in
order to lower their BSGs!4. Be prepared to look for clear anomalies and re-
test if necessary5. Explain the chances graphs to them clearly
Parents
1. Make sure they know why the pupils are taking the test
2. Explain the results to them3. Explain lots of times that the chances graphs
and BSGs do NOT give personalised predictions
4. Ensure that they receive good quality feedback from staff when ambers or reds are awarded
5. Encourage them to ask lots of questions
YOUR QUESTIONS
