electronic marking of examinations pete thomas open university p.g.thomas@open.ac.uk

Electronic Marking of Examinations

Pete ThomasOpen University

p.g.thomas@open.ac.ukhttp://mcs.open.ac.uk/eap

The Context

1. Distance education – Open University, UK

2. On-line study

3. Improving assessment

4. Enhancing the student experience

5. Computing

Distance Education: OU Style

Open University(HQ)

Marker

Marker

ExamCentre

ExamCentre

ExamCentre

ExamCentre

ExamCentre

Typical Computing course has 2 – 4 thousand students

Motivation

“The gap between the environment in which students learn and how they are assessed is widening … there is a radical discontinuity in their educational experience.”

Race et al. (1999)

Electronic Assessment Project Aims for Examinations

• To put the whole examination process on-line.• To design an appropriate exam paper.• To define the client and server functions.• To investigate invigilation issues.• To provide automatic marking of free text.• To provide useful feedback on answers.• To provide induction to the process.• To provide support for examiners.

Marking Exam Scripts

• The objects in the examination system:– Questions and sub-questions– Specimen (sample) solutions– Mark schemes– Rubrics (rules)– Student answers

• Marking consists of – Comparing an answer with a specimen solution and

assigning a mark– Applying the rubric

Issues

• Problems with questions– Incomplete, erroneous or inconsistent data– Asking for an inappropriate conclusion– Ambiguity

• Checking used to remove defects

• Errors come to light:– When candidates read the question– When examiners attempt to grade answers

Issues

• Problems with answers– Understanding the question– Poorly expressed ideas– Lack of knowledge– Poor language skills (spelling)– Use of abbreviations of own devising– Lack of typing skills

• Actions to address the problems– Standardization between markers

Issues

• Problems with specimen solutions– Do not match the question (as perceived by

candidates)– Numerous acceptable alternatives– Incomplete – Incorrect

• Actions to address problems– Expect markers to use professional

judgement– Standardization between markers

Issues

• Problems with mark schemes– Incorrect– Inappropriate for the question as perceived by

candidates– Inappropriate for the difficulty of the question

• Actions to address problems– Revise in the light of experience– Standardization between markers

Electronic Marking

• A specimen solution can consist of more than one solution.

• Each solution can have a different mark.

• A solution can be split into parts with each part having its own marks.

• In general, an ‘electronic solution’ can be composed of a set of alternatives and

each alternative is composed of parts.

Solution RepresentationSuppose that the solution to a (trivial) question consists of the following three phrases:

range memory locations, bounds registers,address outside range exception

AND (3, 6.0)

bounds registersrange memory locations address outside range exception

storage protection keys

storage protection keysbounds registers

OR (2, 1, 3.0)

1.51.5

3.0 3.0

Synonyms?

Thesaurus

First Experiment

20 student scripts from a conventional written exam

3 independent human markers

Question 7 12 (a) 12 (b) 12 (c) 12 (d) 12 Total Over all

Marks allocated

4 8 2 8 2 20 24

Marker averages

1.7 6.1 1.1 4.8 1.13 13.0 14.6

Tool averages

1.75 5.5 0.98 4.8 0.8 12.4 14.2

Second ExperimentElectronic mock examination (3 hours)

Full exam scripts (10 Part 1 and 3 Part 2 questions)

11 students

3 independent human markers

Automatic Score Markers’ mean score

Mean 44.18 56.25

St.Dev. 8.28 10.09

Automatic score is lower and there is a smaller standard deviation

Deficiencies

Description Number

Number of significant spelling errors in answers 8

Number of deficient solution trees (out of 39) 15

Number of thesaurus deficiencies (synonyms) 12

Number of deficient specimen solutions 2

Number of lexical deficiencies (abbreviations) 1

Number of language parsing errors (software errors) 2

Number of deficient questions 2

Deficiency Reduction• Examined one script taken at random and

corrected/amended the solution trees and the thesaurus.

• Repeated the process with 3 more scripts.

Automatic Marker Human Marker Mean

0 1 2 3 4

Mean 44.2 46.0 50.5 52.8 53.2 55.85

Comparison

35

40

45

50

55

60

65

70

40 45 50 55 60 65 70 75 80

Mark awarded by markers

Mar

k aw

ard

ed b

y el

ectr

on

ic t

oo

l

Correlations

Pearson correlation Part 1 Part 2 Total

Tutor 1 & Tutor 2 0.8260 0.9680 0.9456

Tutor 1 & Tutor 3 0.9531 0.9350 0.9281

Tutor 2 & Tutor 3 0.9279 0.9170 0.9362

All tutors & electronic 0.9026 0.8244 0.8604

Examiner Support

Thesaurus maintenance

Exam Building System

Further Experiments

Marker comparisons by part (%) April 03

0

10

20

30

40

50

60

70

Total Part I Part II

Au

R

N

Ad

M

Revised algorithm

Marker comparisons by part (%) April 03

0

10

20

30

40

50

60

70

Total Part I Part II

Au

R

N

Ad

M

Marking Diagrams (April 03)

• What features should a drawing tool provide?• How familiar should students be with the tool

prior to the examination?• How should the tool be provided to students in

order to be used under examination conditions?

• How should a diagram be represented for transmission to the server?

• How should a diagram be represented for grading purposes?

• How to grade a diagram?

Exam Question

Use the drawing tool to draw a diagram that illustrates how the data hazard inherent in the execution of the pair of instructions

ADD R2, R3, R1SUB R1, R5, R4

by a 4-stage pipeline, can be overcome.

Specimen Solution &Drawing Tool

Associations & Constraint Multiset Grammars

a box a link a box

the start box containing text

the end box containing text some text

Association → Link, Weight where (exists Box1 Box2 attached (Link.start, Box1.area) &attached (Link.end, Box2.area)

) {Association.from = Box1 &Association.to = Box2 &Association.text = Link.text &Association.weight = Weight

}

CMG for a pipelinePipeline → Association1, Association2, Association3

where (Association1.to = Association2.from &Association2.to = Association3.from &Association1.from.text.string = “fetch ADD” &Association2.from.text.string = “decode” &Association3.from.text.string = “execute” &Association3.to.text.string = “write R1” &

) {Pipeline.assoc1 = Association1 &Pipeline.assoc2 = Association2 &Pipeline.assoc3 = Association3 &

}

Initial Experiment

Student Human Markers (average)

Diagram Marking Tool

1 2.0 2.0

2 3.0 4.0

3 4.0 3.0

4 2.5 4.0

5 2.0 2.5

Mean 2.7 3.1

St. Dev 0.837 0.894

Thank Youp.g.thomas@open.ac.uk

http://mcs.open.ac.uk/eap