NEW USERS OF ALIS WORKSHOP

JUNE 2011 London Conference

Geoff Davies

FIVE EXERCISES

COVERAGE

Baseline data

Initial feedback

Use of Chances graphs

‘Predictions’

Use of Scatter Graphs

Setting ‘target’ grades

Interpreting Statistical process charts

Basic ALIS is concerned mainly with value added analysis using baseline data collected early in the sixth form

Full ALIS covers this as well as an in depth analysis of attitudes, teaching and learning processes, resources, extra curricular activities, and more

• Average GCSE score --- repeatedly been found to be the best single indicator of post 16 performance

• Computer adaptive test---CEM centre’s jewel in the crown across all spheres of education (TDA paper test still available for A level students)

TWO BASELINES

INITIAL FEEDBACK

Intake Profiles

0

10

20

30

D C B APe

rcen

tage

Band

0

10

20

30

19

24

28 29GCSE

(Alis)

0

10

20

30

D C B A

Perc

enta

ge

Band

0

10

20

3028 27

22 23

TDA / CABT

(Alis)

Predictions

Predictions are available in the following forms:

• Formal Reports

• Spreadsheets

• Paris Software

Average performance by similar students in past exams

Student ID Name Gender Date of Birth Year Group GCSE Score Subject LevelGCSE Based

Predictions

                Points Grade

Student Prediction Reports

The ‘prediction’ for each subject is based on the previous year’s ALIS ‘trend-line’ for that subject

Example:

Lisa Fry has GCSE grades 2A*, 3A, 4B and 1C

Total GCSE Points = 66 (i.e. 2x8 + 3x7 + 4x6 + 1x5)

AVGCSE = 66 / 10 = 6.6

Use ALIS subject formula (21.71 x 6.6) – 44.69 =99

(B at A level)

0

20

40

60

80

100

120

1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0U

CA

S P

oint

s

Average GCSE score

0

20

40

60

80

100

120

Predicted Grade = (21.71 * baseline Score) - 44.69

Chances Graphs

010

203040

506070

U E D C B A

perc

ent

010

203040

506070

Average GCSE between 4.90 and 5.63

2

15

36 34

12 2

010

203040

506070

U E D C B A

perc

ent

010

203040

506070

Average GCSE between 5.63 and 6.45

0 3

14

35 36

12

010

203040

506070

U E D C B A

perc

ent

010

203040

506070

Average GCSE greater then 6.45

0 0 1 9

30

60

0

20

40

60

80

100

120

1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0

UC

AS

Poi

nts

Average GCSE score

0

20

40

60

80

100

120

Predicted Grade = (21.71 * baseline Score) - 44.69

Based on 14069 students :Correlation coefficient = 0.72, Standard Deviation = 17.47Regression Graph

(A2) Geography 2008

The thick region indicates the interquartile range (i.e. the middle 50% of students).

010

203040

506070

U E D C B A

perc

ent

010

203040

506070

Average GCSE Less Than 4.90

9

31 36

19

4 2

Situation

You are the subject teacher and are discussing AS target grades with individual students in the Autumn term of Year 12. You are about to talk to Belinda who achieved an average GCSE score of 5.9.

This gives a statistical prediction=14.75x5.9-59.57 = 27.5 UCAS points using the regression formula at AS for this subject (Grade D/E at AS).

The computer adaptive test seems to confirm this prediction.

Exercise 1

PRACTICAL EXERCISES FOR NEW ALIS USERS

Chances graph showing % of students achieving each AS grade in previous year based on 24175 students

010

203040

506070

U E D C B A

perc

ent

010

203040

506070

Average GCSE between 5.63 and 6.45

19 22 24 19

12 3

AS GRADE

Belinda (5.9)0

10203040506070

U E D C B A

perc

ent

010203040506070

(A1) Chemistry

20 21 23 1912

5

Individual Chances graph example

Other examples Chemistry different cohorts

‘Most candidates with Belinda’s GCSE background score achieved a D in my subject last year so Belinda’s target grade should be a D’. What are the weaknesses of this statement?

a)

What other factors should be taken into consideration apart from chances graph data, when determining a target grade?

b)

Exercise 1The strength of the chances graph is that it shows more than a bald prediction.

True, most students starting from an average GCSE score like Belinda did achieve a D grade at AS in examinations for this subject. However the probability of a C grade is also high since her score was not at the bottom of this range. Students are not robots who will always fit with statistics so it is dangerous to make sweeping statements based on one set of results.

As well as looking at the prediction you should use the chances graph as a starting point with your professional judgement taking into account factors such as previous performance in the subject, her attitude to work and what she is likely to achieve based on your own experience. You might want to start with the most popular outcome grade D and use your judgement to decide how far up (or down!) to go. She may be a very committed student, so a B grade may be more appropriate though A looks unlikely. If you are using aspirational targets for psychological reasons with students then B may be appropriate

Exercise 2

SituationJane has completed her 6th form studies and a review has been received for her by the college after A level results. Choose one subject at a time and look carefully at what happened in that subject both from a baseline of average GCSE grades and from a baseline of the computer adaptive test.

Review: Final_Result Review Review Average AverageSubject Points Grade Points Grade Residual Std. Residual(A1) Health &Social Care 30.00 D 39.64 C -9.64 -0.66(A2) Religious Studies 80.00 C 89.84 B/C -9.84 -0.52(A2) English Literature 80.00 C 87.29 B/C -7.29 -0.40(A2) Drama &Theatre St.100.00 B 91.84 B/C 8.16 0.46

FINAL RESULTS PREDICTIONS

RAW

STANDARDISED

Profile Sheet: Jane (from Average GCSE)Year: 2007

DOB: 01/06/89(Average GCSE = 6.00 (Band B))

Chances Graphs - Band B from average GCSE

010203040506070

U E D C B A

perc

ent

010203040506070

(A1) Health & Social Care

3 718

30 2813

010203040506070

U E D C B A

perc

ent

010203040506070

(A2) Religious Studies

0 211

3140

15

010203040506070

U E D C B A

perc

ent

010203040506070

(A2) English Literature

0 215

36 34

13

010203040506070

U E D C B A

perc

ent

010203040506070

(A2) Drama And Theatre Studies

0 110

3142

17

Individual Chances Graphs from average GCSE score

010203040506070

U E D C B A

perc

ent

010203040506070

(A1) Health & Social Care

4 1020

29 2512

010203040506070

U E D C B A

perc

ent

010203040506070

(A2) Religious Studies

0 212

3037

19

010203040506070

U E D C B A

perc

ent

010203040506070

(A2) Drama And Theatre Studies

0 110

3240

17

010203040506070

U E D C B A

perc

ent

010203040506070

(A2) English Literature

0 217

33 32

16

Profile Sheet: Jane from Computer adaptive testYear: 2007

DOB: 01/06/89(Online adaptive test = 0.11 (Band C))

Review: Final_Result

RAW

STANDARDISED

Review Average Average Subject Points Grade Points Grade Residual St (A1)Health &Social Care 30.00 D 25.97 D/E 4.03 0.24(A2) Religious Studies 80.00 C 79.01 C 0.99 0.04(A2) English Literature 80.00 C 74.3 C/D 5.7 0.26(A2) Drama &Theatre St. 100.00 B 85.16 B/C 14.84 0.70

PREDICTIONS

a) Can you suggest possible reasons for the differences between the predictions (average grade) made by the two baselines? ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------b) Did Jane reach her potential in all subjects? ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------c) Jane had been set aspirational targets prior to AS and A level by her teachers as belowHealth and Social Care CDrama BReligious Studies BEnglish B Where these reasonable target grades?----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Why should these grades not be used for accountability of her teachers?-----------------------------------------------------------------------------------------------------------------------

Exercise 2a) This student has been placed in different bands, band B from average GCSE score and band C from the computer adaptive test. This sometimes happens. It may be that the student had an off day when she did the computer adaptive test or it may be that there could have been a lot of ‘spoon feeding’ at GCSE. Jane may do better at coursework! Even though we may not know the cause it can act as a warning when analysing results though the predictions are not wildly out.b) Jane certainly reached her potential in Drama and Theatre studies with positive standardised residuals by both methods. On the basis of the computer adaptive test she broadly reached potential in all subjects. On the basis of average GCSE two A level subjects were broadly down about half a grade and she dropped a grade in the ASc) I hope you agree these were reasonable target grades. Remember we don’t know the student, but use the chances graphs and if these were aspirational grades for the student then accountability of departments staff on that basis is not appropriate, but it certainly is on the basis of a whole classes average standardised residuals particularly over a number of years.

Exercise 3 Scatter Graphs (A2) Subject Report

A2 Subject Report Year: 2008Review: Final_Result Baseline: Average GCSE

0

20

40

60

80

100

120

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0

UC

AS

Poi

nts

Average GCSE Score

0

20

40

60

80

100

120

Predicted Grade = 23.16 * Average GCSE - 66.31

BC

Exercise 3 Hopefully this did not create too much difficulty The arrows are vertical and join the point to the trend line.

Not easy to judge from this. It might be worth plotting standardised residual against average GCSE score and look

for deviations from a horizontal line (more advanced)

Standardised residuals allow one subject to be compared with another because different subjects have varying distributions of grades. These are not dependent on the year either.

Exercise 4Change in teaching methods used had major effectSept 2007 probably a factor!

Exercise 5Some schools and colleges use average GCSE points together with chances graphs to tweak appropriate course choices for their students on the basis of probability of success. Perhaps more useful is alerting year 12 tutors to the strengths and weaknesses of students as shown by the computer adaptive test.

Using the chances graphs

Of the students with a point-score below

5.0 a total of 11 achieved an ‘A’

grade and a total of 562 achieved a ‘U’

grade!

Example: A/S Accounting (2005)

Using the chances graphs

Of the students with a point-score above

6.4 a total of 113 achieved an ‘A’

grade and a total of 27 achieved a ‘U’

grade!

Example: A/S Accounting (2005)

Comparison between subjects

AS Biology (2005) AS Film Studies (2005)

NEW USERS OF ALIS WORKSHOP

JUNE 2011 London Conference

Geoff Davies