using cem data: target setting, monitoring & reporting
DESCRIPTION
Using CEM Data: Target Setting, Monitoring & Reporting. Belfast, March 6 th 2013 Neil Defty Business & Development Manager CEM [email protected]. Which Baseline to use?. Student 1 . Student 2. Student 3. Student 3 - IPR. Key Questions for Target Setting. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/1.jpg)
Using CEM Data:Target Setting, Monitoring &
Reporting
Belfast, March 6th 2013Neil Defty
Business & Development ManagerCEM
![Page 2: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/2.jpg)
Which Baseline to use?
![Page 3: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/3.jpg)
Student 1
![Page 4: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/4.jpg)
Student 2
![Page 5: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/5.jpg)
Student 3
![Page 6: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/6.jpg)
Student 3 - IPR
![Page 7: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/7.jpg)
Key Questions for Target Setting
• What type of valid and reliable predictive data should be used to set the targets?
• Should students be involved as part of the process (ownership, empowerment etc.)?
• Should parents be informed of the process and outcome?
![Page 8: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/8.jpg)
Key points to consider might include:
• Where has the data come from?• What (reliable and relevant) data should we
use?• Enabling colleagues to trust the data: Training
(staff)• Communication with parents and students• Challenging, NOT demoralising, students….• Storage and retrieval of data• Consistency of understanding what the data
means and does not mean
![Page 9: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/9.jpg)
Value Added: The theory and Stats bits…
![Page 10: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/10.jpg)
20
40
60
80
100
120
Subject X
Out
com
e
.
Trend Line/Regression Line
Measuring Value Added – Terminology
BASELINE SCORE
-ve VA+ve VA
Residuals
VA
Exa
m g
rade
![Page 11: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/11.jpg)
Measuring Value Added – An Example
Low Ability Average Ability
High Ability
Baseline Score
A*
U
B
C
D
E
F
G
Res
ult Aldwulf Beowulf
Cuthbert+ve (+ 2 grades)
-ve (- 2 grades)
National Trend
‘Average’ Student
The position of the national trend line is of critical importance
Subject A
Subject B
![Page 12: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/12.jpg)
5 6 7 840
60
80
100
120
140
PhotographySociologyEnglish LitPsychologyMathsPhysicsLatin
Average GCSE
Gra
de
Some Subjects are More Equal than Others….A-Level
>1 grade
A*ABC
A
A*
B
C
D
E
![Page 13: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/13.jpg)
Burning Question :
What is my Value Added Score ?
Better Question :
Is it Important ?
![Page 14: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/14.jpg)
Value Added ChartsPre 16
![Page 15: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/15.jpg)
Performance in line with expectation
VA Score
Performance below expectationProblem with Teaching & Learning?
Performance above expectationGood Practice to Share?
![Page 16: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/16.jpg)
Which Subjects Cause Most Concern?
Danger of Relying on Raw Residuals Without Confidence Limits
![Page 17: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/17.jpg)
Additional A
pplied Science
Additional S
cience
Art &
Design
Biology
Business S
tudies
Chem
istry
Design &
Technology
Dram
a
English
English Literature
French
Geography
Germ
an
History
Mathem
atics
Music
Physical E
ducation
Physics
Religious S
tudies
Science
Spanish
Short C
ourse Religious S
tudies
-4-3-2-101234
0.00.8 0.5
-0.3
1.1
-0.4
1.00.2 0.4 0.1 0.1
0.0
0.0 0.1
0.0
0.0
-0.3
0.2 0.5
-0.3
0.70.2
-2.9
0.0
Average Standardised Residuals by Subject
Aver
age
Stan
dard
ised
Res
idua
l
Which subjects now cause most concern ?
![Page 18: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/18.jpg)
Business Studies
![Page 19: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/19.jpg)
![Page 20: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/20.jpg)
Religious Studies
![Page 21: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/21.jpg)
Value Added ChartsPost 16
![Page 22: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/22.jpg)
SPC Chart
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
Year
Performance in line with expectation
VA Score
Performance below expectationProblem with Teaching & Learning?
Performance above expectationGood Practice to Share?
![Page 23: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/23.jpg)
Subject Summary - 3 Year Average
Subject Summary - Current Year
![Page 24: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/24.jpg)
-0.60-0.48-0.36-0.24-0.120.000.120.240.360.480.60
2002 2003 2004
Aver
age S
tanda
rdise
d Res
idual
Year
-0.60-0.48-0.36-0.24-0.120.000.120.240.360.480.60
A2-English Literature
Statistical Process Control (SPC) Chart
2008 2009 2010Year
![Page 25: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/25.jpg)
Student Level Residuals (SLR) Report
Scatter Plot
A2 – English Literature
General Underachievement?
![Page 26: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/26.jpg)
Student Level Residuals (SLR) Report
Scatter Plot
A2 – English Literature
Too many U’s?
![Page 27: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/27.jpg)
Other things to look for…
Why did these students do so badly?
Why did this student do so well?
How did they do in their other subjects?
![Page 28: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/28.jpg)
Summary of Process
• Examine Subject Summary• Determine ‘interesting’ (i.e. statistically significant) subjects• Look at 3 year average as well as single year if available• Look at trends in ‘Interesting Subjects’• Examine student data – Scatter graphs• Identify students over / under achieving• Any known issues?• Don’t forget to look at over achieving subjects as well as under
achieving
![Page 29: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/29.jpg)
Baseline Choice
![Page 30: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/30.jpg)
• Do students with the same GCSE score from feeder schools with differing value-added have the same ability?
• How can you tell if a student has underachieved at GCSE and thus can you maximise their potential?
• Has a student got very good GCSE scores through the school effort rather than their ability alone?
• Does school GCSE Value-Added limit the ability to add value at KS5?
• Can you add value at every Key Stage?
GCSE or Baseline Test?
![Page 31: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/31.jpg)
The Effect of Prior Value Added
Beyond Expectation+ve Value-Added
In line with Expectation0 Value-Added
Below Expectation-ve Value-Added
Average GCSE = 6 Average GCSE = 6 Average GCSE = 6
Do these 3 students all have the same ability?
![Page 32: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/32.jpg)
Same School - Spot the Difference ?
GCSE as Baseline
Test as Baseline
![Page 33: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/33.jpg)
National or School Type Specific?
![Page 34: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/34.jpg)
Comparison to all schools
Comparison to Independent Schools Only
![Page 35: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/35.jpg)
Comparison to all schools
Comparison to FE Colleges Only
![Page 36: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/36.jpg)
Questions:
→ How does the unit of comparison used affect the Value Added data and what implications does this have on your understanding of performance?
→ Does this have implications for Self Evaluation?
![Page 37: Using CEM Data: Target Setting, Monitoring & Reporting](https://reader030.vdocuments.us/reader030/viewer/2022013012/568168ae550346895ddf6526/html5/thumbnails/37.jpg)
Definitions:• Residual – difference between the points the student attains and
points attained on average by students from the CEM cohort with a similar ability
• Standardised Residual – the residual adjusted to remove differences between qualification points scales and for statistical purposes
• Average Standardised Residual – this is the ‘Value Added Score’ for any group of results
• Subject VA – average of standardised residuals for all students’ results in the particular subject
• School VA – average of standardised residuals for all students’ results in all subjects for a school / college
• Confidence Limit – area of statistical uncertainty within which any variation from 0 is deemed ‘acceptable’ and outside of which could be deemed ‘important’