introduction to cem secondary information systems dr robert clark alis project manager
TRANSCRIPT
![Page 1: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/1.jpg)
Introduction to CEM
Secondary Information Systems
Dr Robert Clark
ALIS Project Manager
![Page 2: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/2.jpg)
The Projects
![Page 3: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/3.jpg)
The Projects
Year 7Year 7
Year 8Year 8(+ additional)(+ additional)
Year 9Year 9
MidYISMidYIS Computer Adaptive Baseline Test(paper test also available)
Year 10Year 10
Year 11Year 11 YellisYellis Computer Adaptive Baseline Test(paper test also available)
Year 12Year 12
Year 13Year 13 AlisAlis GCSE Computer Adaptive Baseline Test(paper test also available)
GCSE
A / AS / Btec / IB etc
INSIGHTINSIGHT Combines curriculum tests with developed ability
![Page 4: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/4.jpg)
The baseline score is measured as early as possible and the information sent to the CEM Centre
The institution then receives feedback from the CEM Centre: baseline test results and predictions
The public examination results are sent to the CEM Centre when received by the institution
The value added data is available from September via the Institution’s website at the CEM Centre
Typical Timeline
Early Testing & Predictions
Testing is available from June at end of previous academic year i.e. Y6 / Y9 / Y11
![Page 5: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/5.jpg)
Predictions
![Page 6: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/6.jpg)
.
Subject X
020406080
100120140
4 5 6 7 8
Baseline
Out
com
e
0 2 4 6 8
Subject X
How CEM ‘Predictions’ are made…
A / B
C
Regression Line (…Trend Line, Line of Best Fit)
Outcome = gradient x baseline + intercept
Correlation Coefficient (~ 0.7)
![Page 7: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/7.jpg)
40
60
80
100
120
140
5 6 7 8
Average GCSE
Gra
de
Photography
Sociology
English Lit
Psychology
Maths
Physics
Latin
Some Subjects are More Equal than Others….
>1 grade
A*ABC
A
A*
B
C
D
E
![Page 8: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/8.jpg)
F
E
D
C
B
A
A*
Test Score
GC
SE
Gra
des
Art & DesignBiologyChemistryEconomicsEnglishFrenchGeographyGermanHistoryIctMathematicsMedia StudiesMusicPhysical EducationPhysicsReligious StudiesScience (Double)Spanish
Some Subjects are More Equal than Others….
GCSE (MidYIS or Yellis)
1 grade
![Page 9: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/9.jpg)
Baseline Measurement
![Page 10: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/10.jpg)
Problems with Key Stage Baselines
1. Not all students have KS baselines
No KS3
Foreign Students
Vocational Students
Adult Learners
2. KS exams do not always represent ‘Start of Course’ ability
Post-16 : Year(s) out or intermediate years
3. Prior Value-Added
Can you add value at every Key Stage ?
Under achievement leading to under expectation
One teacher’s output = another teacher’s input
4. Teaching to the test
Does performance represent ability, effort or exam technique ?
5. Aptitude & fluency vs Achievement & knowledge
6. Transfer of Data
![Page 11: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/11.jpg)
Although Key Stage baselines can be a very good indicator of potential attainment, by themselves they are not sufficient.
Key Stage baselines are confounded by the effects of prior treatment.
![Page 12: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/12.jpg)
The Computer Adaptive Test
• Test performed online – results automatically transmitted to CEM.
• Minimal installation / setup required - if any.• Adaptive – difficulty of questions changes in relation to ability of
student.• Efficient – no time wasted answering questions that are far too
easy or difficult.• Wider range of ability• Less stressful on students – more enjoyable experience than
paper test.• Less demanding invigilation.• Cheaper !
In 2010/ 2011 over 200,000 students across yrs 7-13 sat this test
Try it yourself at www.intuproject.org/demos
![Page 13: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/13.jpg)
Baseline Feedback
Reports, Graphs
& Predictions
![Page 14: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/14.jpg)
IPRs (Individual Pupil Record Sheets)
Look for sections that are
inconsistent
Also available based on MidYIS, Alis, SOSCA & INSIGHT scores
![Page 15: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/15.jpg)
Intake Profiles
Also available based on MidYIS, Yellis, SOSCA and INSIGHT scores
![Page 16: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/16.jpg)
Intake Profiles (Historical)
![Page 17: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/17.jpg)
Predictions – MidYIS example
Similar spreadsheets available from Yellis, SOSCA, INSIGHT
![Page 18: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/18.jpg)
Predictions - Alis example
![Page 19: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/19.jpg)
English Language - band D
2
10
23
3124
9
1 0 00
10
20
30
40
50
U G F E D C B A A*
Grade
Perc
ent
English Language - band C
1 26
20
3529
71 0
0
10
20
30
40
50
U G F E D C B A A*
Grade
Perc
ent
English Language - band B
0 0 27
24
40
21
50
0
10
20
30
40
50
U G F E D C B A A*
Grade
Perc
ent
English Language - band A
0 0 0 1
8
26
35
23
7
0
10
20
30
40
50
U G F E D C B A A*
Grade
Perc
ent
Chances Graphs
![Page 20: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/20.jpg)
Predictions
vs
Targets
![Page 21: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/21.jpg)
What is a ‘Prediction’ ?
NOT a forecast of the grade the student will get
An indication of the grade (points score) achieved on average by students of similar ability in the previous year
Targets ?• Minimum Targets – Round CEM prediction down ?
• Realistic Targets – Use nearest CEM grade ?
• Challenging Targets – 75th percentile ? Prior Value Added ? Arbitrary grade fraction ?
![Page 22: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/22.jpg)
75th Percentile Predictions• If all students attain 75th percentile predictions, School VA will be at top 25% • Provided by all Projects in addition to 50th percentile• Alis : Excel spreadsheet - ‘Predictions – Spreadsheet (75th Percentile)’• Pre-16 : Adjust button on predictions spreadsheet
Prior Value Added• Only where prior VA is positive ?• 1 year or 3 ?• Reasonable to use raw residual figures as this is an approximate measure and
raw residuals give grade fractions• Alis : Can be calculated using PARIS software• Pre-16 : Adjust button on predictions spreadsheet
Data used to inform, not replace, professional judgement
![Page 23: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/23.jpg)
75th Percentile
Prior Value Added
![Page 24: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/24.jpg)
Step 1
75th Percentile
PriorValue-Added
![Page 25: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/25.jpg)
Value-Added Feedback
Reports & Graphs
![Page 26: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/26.jpg)
Subject X
20
40
60
80
100
120
4 5 6 7 8
Baseline
Out
com
e
.
-ve VA+ve VA
Trend Line/Regression Line
Raw Residual
Measuring Value-Added – Terminology
Exa
m g
rade
BASELINE SCORE
![Page 27: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/27.jpg)
Measuring Value-Added – An Example
Low Ability Average Ability High Ability
Baseline Score
A*
U
B
C
D
E
F
G
Res
ult
Alf Bob
Chris+ve (+ 2 grades)
-ve (- 2 grades)
National Trend
‘Average’ Student
The position of the national trend line is of critical importance
Subject A
Subject B
![Page 28: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/28.jpg)
Standardisation of Residuals
• (Raw) Residuals can be used to examine an individual’s performance
• Standardised Residuals are used to compare performance of groups
• Standardised Residuals are independent of year or qualification type
• For a class, subject, department or whole institution the Average Standardised Residual is the ‘Value-Added Score’
• Standardised Residual = Residual / Standard Deviation (National Sample)
• When using Standardised Residuals then for an individual subject
• 95% Confidence Limit = 2.0 x Standard Error• 99% Confidence Limit = 2.6 x Standard Error• 99.7% Confidence Limit = 3.0 x Standard Error
N
1ErrorStandard where N = number of results in the
group
(for combinations of subjects consult the relevant project)
![Page 29: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/29.jpg)
Burning Question :
What is my Value-Added Score ?
Better Question :
Is it Important ?
![Page 30: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/30.jpg)
Value Added Feedback…
Statistical Process Control (SPC) Chart
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
Year
![Page 31: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/31.jpg)
-0.2 -0.2 -0.1
0.1
-0.6
0.1
-0.2
-0.7 -0.7-0.4 -0.4
-0.2
-2.2
-0.5
2.0
-0.7-0.5
0.10.4
-4.0
-3.0
-2.0
-1.0
0.0
1.0
2.0
3.0
4.0
Art
Bio
logy
Che
mis
try
Des
ign
and
Tec
hnol
ogy
Dra
ma
Eng
lish
Lang
uage
Eng
lish
Lite
ratu
re
Fre
nch
Geo
grap
hy
Ger
man
His
tory
Hom
e E
cono
mic
s
Info
rmat
ion
Tec
hnol
ogy
Mat
hs
Med
ia S
tudi
es
Mus
ic
Phy
sics
Rel
igio
us S
tudi
es
Spa
nish
Av
era
ge
Sta
nd
ard
ise
d R
es
idu
al
Subject Summary
Standardised Residual Graph
![Page 32: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/32.jpg)
The Scatter Plot
Baseline Score
Gra
de
Po
ints
Eq
uiv
alen
t
Look for Patterns…
General Underachievement / over achievement ?
Do any groups of students stand out ?
– high ability vs low ability ?
– male vs female ?
![Page 33: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/33.jpg)
Other things to look for…
Why did these students do so badly ?
Why did this student do so well ?
How did they do in their other subjects ?
![Page 34: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/34.jpg)
PARIS Software
![Page 35: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/35.jpg)
PARIS is …..•Software to install and use in school
•An interactive reporting tool
•Included free with Alis / Yellis / MidYIS
PARIS provides …..•Student level reports
•Subject Level reports
•Institution Level Reports
PARIS analyses …..•Potential Performance
•Intermediate Performance
•Actual Performance
![Page 36: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/36.jpg)
Attitudes
![Page 37: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/37.jpg)
There is more to school / college than exams….
• Student attitudes• Student Welfare & Safety• Non-academic activities• Support• Social and personal development• Parental Survey• Induction Survey
Attitudinal MidYIS INSIGHT Attitudinal Yellis Full ALIS
Self Evaluation (Every Child Matters)
Try it yourself at www.intuproject.org/demos
![Page 38: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/38.jpg)
Other Issues
![Page 39: Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager](https://reader036.vdocuments.us/reader036/viewer/2022062304/56649ddc5503460f94ad30e0/html5/thumbnails/39.jpg)
Points to mull over…
• Independence – no agendas
• Transparency of Analysis
• Self Evaluation
• Straightforward and standardised administration
• Prompt Feedback
• Full working hours phone / email support
• Student focus
• Replacement for KS3
• Innovative online adaptive testing available – student experience
• Longitudinal analysis with appropriate error backgrounds
• Non-curriculum embedded baselines available
• Attitudinal surveys available – Every Child Matters…