Table of Contents

1. Introduction .................................................................................................................... 41.1 Comparable outcomes and GCSE predictions................................................................... 41.2 Existing research ............................................................................................................... 51.3 Aims of current project....................................................................................................... 71.4 Data................................................................................................................................... 8

1.4.1 Data provided by awarding organisations (AOs).......................................................... 81.4.2 Data from the National Pupil Database ....................................................................... 8

2. Review of current method of generating predictions..................................................... 102.1 Description of current method.......................................................................................... 102.2 Possible alternative measures of KS2 attainment ............................................................ 122.3 Correlations between different measures of KS2 attainment and GCSE grades .............. 162.4 Examining differences in KS2GCSE correlations across subjects .................................. 202.5 Predictive power of different KS2 measures across years ............................................... 242.6 Differences with predictions from screening (concurrent attainment) ............................... 282.7 Practical differences between predictions based on different measures .......................... 31

2.7.1 Further exploration of the effect of the KS2 grade inflation adjustment...................... 362.8 Summary ......................................................................................................................... 43

3. Review of tolerances for reporting outcomes that do not meet predictions ................... 443.1 Method and results .......................................................................................................... 453.2 Comparison with tolerances calculated using simple random sampling (SRS) methods .. 483.3 Quantifying tolerances as percentage rather than percentage point changes .................. 493.4 Expected difference with screening predictions ............................................................... 503.4 Summary ......................................................................................................................... 51

4. Review of differences between screening outcomes and predictions ........................... 524.1 Comparison of KS2 and screening predictions ................................................................ 524.2 Are screening predictions influenced by the combination of GCSE specifications

candidates have taken at GCSE? .................................................................................... 544.3 Possible solutions to the issue of underprediction of AO differences .............................. 56

4.3.1 Adjusting the KS2 method using ideas from equating ............................................... 564.3.2 Controlling for centrelevel attainment in predictions ................................................. 574.3.3 Using historical differences to adjust predictions ....................................................... 57

4.4 Final thoughts on the underprediction problem ............................................................... 644.5 Summary ......................................................................................................................... 65

5. Appropriate tolerances for predictions based on concurrent GCSE performance ......... 665.1 Comparison with tolerances calculated using simple random sampling (SRS) methods .. 675.2 Summary ......................................................................................................................... 68

6. Differences in the relationship between KS2 and GCSE achievement between yearsand AOs ...................................................................................................... 69

6.1 Differences between years .............................................................................................. 696.2 Differences between AOs ................................................................................................ 736.3 Summary ......................................................................................................................... 76

7 Further investigation of centre effects ........................................................................... 777.1 Using centre type in predictions....................................................................................... 77

7.1.1 Is the separate treatment of candidates from selective and independent schoolsjustified?............................................................................................................................. 797.1.2 Does accounting for centre type in the model give any benefit? ................................ 80

7.2 Controlling for mean centrelevel KS2 ............................................................................. 827.3 Summary ......................................................................................................................... 82

8. Further work and final thoughts.................................................................................... 838.1 Summary of results.......................................................................................................... 83

2

8.2 Other issues not explored................................................................................................ 848.3 Final note......................................................................................................................... 85

References................................................................................................................... 87

Appendix 1: GCSE predictions using mean Key Stage 2 Level as the measure of priorattainment.................................................................................................... 89

Appendix 2: Detailed description of methodology used to estimate tolerances for each AOand each subject ......................................................................................... 95

Further validation of the method ............................................................................................ 96

Appendix 3: A modified method for producing GCSE predictions based upon Key Stage 2.................................................................................................................... 99

Appendix 4: Examination of the relationship between KS2 match rate and agreement ofresults with screening outcomes................................................................ 100

3

1. Introduction To help ensure that GCSE and A level results are comparable with the standards of previous years, awarding organisations (AOs) use data on pupil attainment to predict the percentage of candidates expected to achieve the key grades (such as GCSE grades A*, A and C) in each subject overall. This is a key tool for guiding awarders when they set grade boundaries and for maintaining standards over time.

To predict the expected outcomes for any given years GCSE cohort, AOs look at the relationship between GCSE performance in a relevant reference year and that cohort's attainment at Key Stage 2 (KS2) (where available). This allows them to produce a model of the relationship they can use to produce expected outcomes for the given years GCSE cohort. A detailed description of the process used for the majority of predictions in 2013 is given in Appendix 1. A more general description of the process is provided within Section 2.

The aim of the research in this report is to provide a thorough technical evaluation of the relationship between GCSE results and prior attainment at KS2, including a consideration of whether predictions can be made more valid, and a review of the general approach in terms of using KS2 data to support the maintenance of standards. This report will also examine the continuing validity of using average KS2 attainment to produce predictions given that the last national KS2 Science tests took place in 2009, and hence no data from these tests will be available for the 16 year old GCSE cohort of 2015.

1.1 Comparable outcomes and GCSE predictions The use of GCSE predictions based on KS2 attainment to help define GCSE grade boundaries is part of Ofquals wider strategy known as comparable outcomes. This means that, under usual circumstances1, the aim is that roughly the same proportion of students will achieve each grade as in the previous year. (Ofqual, 2012, page 2)2.

The aim to achieve comparable outcomes is explicitly set against the aim for each grade to represent comparable performance over time. On the one hand this is argued for from the basis of avoiding a dip in grades whenever a new specification is introduced as teachers become used to the new material. However, it is also explicitly intended to combat grade inflation. That is, the focus on comparable outcomes is intended to reduce the extent to which there are increases in the percentage of students achieving the highest grades yearonyear.

Given the overarching aim to ensure that the overall grade distribution will be roughly equivalent between years, the next task is to decide upon how grades should be distributed across different subjects and (within subje