introduction to cem and the baseline tests

Introduction to CEM and the Baseline Tests

Robert ClarkNeil Defty

Nicola Forster

Who / What is CEM ?

Centre for Evaluation & Monitoring• Not for Profit Organisation• Part of Durham University• Close links to School of Education• Established 1985• Nursery / Reception Post-16• Monitoring Systems, Research and Evaluation Projects• UK & International• Over 1/3 Secondary & Post-16 Establishments• Informed by Evidence from Research

The Systems

The Systems

Year 7Year 8

Year 9MidYIS Computer Adaptive Baseline Test

Year 10Year 11 Yellis

Computer Adaptive Baseline Test

Year 12Year 13 Alis

GCSE Computer Adaptive Baseline Test

GCSE

A / AS / Btec / IB etc

INSIGHT Combines curriculum tests with developed ability

KS3

Typical TimelineMeasure BaselineSTART:

Autumn Term

Feedback

Reports

Collect and Send Results

Value-Added

Feedback

August

END:

September

The baseline score is measured as early as possible and the information sent to CEM

The institution then receives feedback from CEM: baseline test results and predictions

The public examination results are sent to CEM when received by the institution

The value added data is available from September via the Institution’s website at CEM

Typical Timeline

Early Testing & Predictions

Testing is available from June at end of previous academic year

e.g. Y6 / Y9 / Y11

Baseline Assessments of Developed Ability

Baseline Assessment

• Year 7 (8)• Year 9• Year 10 (11)• Year 12 (13+)

Computer Adaptive Baseline Test

Vocab Maths Non Verbal

What is an Adaptive test ?

• All questions allocated to groups of different difficulty• All students in a year group get common difficulty starting question• If answer correct, next question harder• If answer wrong, next question easier• Test looks at all answers in a section and homes in on questions of a

suitable difficulty for student

Try it yourself at www.intuproject.org/demos

X

Standardisation

• Test scores are standardised; Mean = 100, SD = 15

Standardised Score

National Percentage

Comment

>130 Top 2.5% Traditional classification of ‘mentally gifted’

>120 Top 10%

>100 Top 50%

<80 Bottom 10%

<70 Bottom 2.5% Potential special educational needs ??

The Computer Adaptive Baseline Test:

• Adaptive components common to MidYIS, Yellis, and Alis are:– Vocabulary– Maths

• Non- adaptive sections:– Non-verbal (MidYIS, Yellis, Alis)– Skills (MidYIS only)

• Assessments are ‘curriculum free’: i.e. students are not taught prior to the test

• Measure of acquired ability• There is no preparation for the test

Difficult vocabulary

Difficult maths

Non-Verbal: Cross-sections

Non-Verbal: Blocks

PicturesNon-Verbal: Pictures

Skills: Proof Reading

Skills: Perceptual Speed and Accuracy

Additionally for INSIGHT:Developed Ability

• Vocabulary

• Non verbal, Skills

• Attitudinal measures

Curriculum-based assessments covering:

Maths - Number & Algebra- Handling Data- Space, Shape & Measurement

Science - Biology, Chemistry, Physics, - Attitudes to Science

Reading - Speed Reading- Text Comprehension- Passage Comprehension

INSIGHT

INSIGHT - Maths

INSIGHT - Maths

Plus Attitude to Science: Perception of Science, Relevance to Self, Environmental Actions, Enjoyment of Science

INSIGHT - Science

INSIGHT – Reading(Text Comprehension)

INSIGHT – Reading(Passage Comprehension)

INSIGHT – Reading(Speed Reading)

INSIGHT – Developed Ability

Measuring AbilityKey Stage Data

or Baseline Test ?

Key Stage Baselines – KS2; GCSEStrengths WeaknessesRelated to curriculum Dependent on teaching effectiveness (prior Value-

Added) as well as student ability

Automatically available Open to manipulation

In Depth Not all students have KS data

Linked to student learning experience

Reliability of KS measure

Consistency of KS score (i.e. GCSE & Btec First etc)

Independent Baseline Test – CEM Adaptive Test

The Effect of Prior Value Added

Beyond Expectation+ve Value-Added

In line with Expectation0 Value-Added

Below Expectation-ve Value-Added

Average GCSE = 6 Average GCSE = 6 Average GCSE = 6

Do these 3 students all have the same ability ?

Baseline Feedback

Feedback from Baseline Test

• Intake Profiles• Individual Pupil Records (IPRs)• Predictions & Chances Graphs

Individual Pupil Record Sheets(IPRs)

Look for sections that are inconsistent

Available in all systems pre & post 16

Predictions

.

Subject X

020406080

100120140

4 5 6 7 8

Baseline

Out

com

e

0 2 4 6 8

Subject X

How CEM ‘Predictions’ are made…

A* / A

C

Regression Line (…Trend Line, Line of Best Fit)

Outcome = gradient x baseline + intercept

Correlation Coefficient (~ 0.7)

40

60

80

100

120

140

5 6 7 8

Average GCSE

Gra

de

PhotographySociologyEnglish LitPsychologyMathsPhysicsLatin

Some Subjects are More Equal than Others….

>1 grade

A*ABC

A

A*

B

C

D

E

F

E

D

C

B

A

A*

Test Score

GC

SE G

rade

s

Art & DesignBiologyChemistryEconomicsEnglishFrenchGeographyGermanHistoryIctMathematicsMedia StudiesMusicPhysical EducationPhysicsReligious StudiesScience (Double)Spanish

Some Subjects are More Equal than Others….GCSE (MidYIS or Yellis)

1 grade

Predictions – MidYIS example

Similar spreadsheets available from Yellis & INSIGHT

2 3

10

23

33

23

5

1 00

5

10

15

20

25

30

35

40

U G F E D C B A A*

Perc

ent

Grade

Individual Chances Graph for Student no.5 - GCSE EnglishMidYIS Score 82 MidYIS Band D

Most likely grade

Prediction/expected grade: 3.8 grade D

Predictions - Alis example

Why KS Baselines do not always tell the whole story

Student 1

Student 2

Student 2 - IPR

Student 3

Student 3 - IPR

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 330

5

10

15

20

25

30

Vocab Item Responses

Question

Resp

onse

Tim

e (s

)

Correct

Wrong

Thank YouRobert Clark – robert.clark@cem.dur.ac.uk

Neil Defty – neil.defty@cem.dur.ac.ukNicola Forster – nicola.forster@cem.dur.ac.uk