education | elections | census controlling marking quality in e-marker ®

Education | Elections | Census

Controlling marking quality

in e-Marker®


Commercial in confidence 2012

Marking quality control

• Marking quality can be delivered by:– e-Standardisation– Seeding and qualification– Percentage double marking– S-Process



Seeding or percentage double marking?

• If the marking guidance is detailed and objective and you would expect complete agreement among markers – use seeding

• If the marking guidance is subjective – or gives guidance for the approach to marking, rather than what is a right or wrong answer – use percentage double marking

• You can use seeding and percentage double marking on the same component

• S-Process can be added to both quality methods for further quality control if desired



Seeding and qualification



Seeds

• Seeds provide an in-built quality check• The administrators define and maintain the levels and

frequency of the quality checks• Senior markers pre-mark a selection of responses - seeds• Quality control applies at individual question level• Quality is checked at the start of each day (qualification)

and while the markers mark (seeding)• Markers can see the seeds they have failed so that they can

improve their marking



Seeding

Qualify at start of day

Mark seeds

Mark batch of

questions

Pass FailStop

MarkingQuestion

Quality check: pass 9 out of 10

Mark batch of

questions

Start marking after discussions with Senior or mark other questions

Quality check: must not fail both seeds

Quality check: must not fail both seeds



Seed window

Quality check:7 out of 10 within tolerance



Seeding is set by senior markers

• Seeding needs to be done before the markers start marking• Ensure there are enough seeds – the minimum seed bank

size should be reached• If seeds are retired, they will need replacing• On a seeded question, the senior markers’ mark will be

awarded to the candidate



Seeding for real



What makes a good seed?

• Legible• Unambiguous and complete• Reflects the standards agreed in standardisation• No tricks!• Does not have to be the right answer• Don’t set “not attempted” responses as seeds



Percentage double marking




• Percentage double marking involves:– Comparing two marking opinions in real time at question

level;– Where differences in marking exceed a set tolerance,

automated quality checks are used to invoke adjudication by a senior marker or peer




1

2

The Senior Marker adjudicates. In this case he/she agrees with Marker 2, so Marker 1 receives a penalty

Marker 2 marks the same clip, not knowing it has been marked before. If Markers 1 and 2 agree, no further action is taken and the mark is awarded to the candidate. In this example however, it is marked outside tolerance of Marker 1, so the marking of both markers is forwarded for adjudication by the Senior Marker

Marker 1 receives a portion to mark. It is selected by e-Marker® as a quality control item (benchmark item) and is forwarded to the next available marker




• There are controls in e-Marker® administration to:– Ensure that markers mark at a similar pace;– Don’t collude in their marking;– Identify when discrepancies are forwarded to peer or

senior; and– Identify when adjudication goes against a marker



Pioneer cap

Too high?

• First marker could mark their entire quota without being checked against other markers

• Be particularly careful in small marker pools

Too low?

• Could slow down marking progress with markers being halted too often

• Could make the frequency of the quality checks overt to markers

The pioneer cap defines how many benchmarks the fastest marker can be ahead of the next fastest. The prevents one marker marking too much without any quality control



Partnering gap

Too high?

• Marking is likely to be slowed down as new markers are needed

• Could leave difficult, outstanding parts to mark at the end of marking

Too low?

• Could have markers colluding, or markers with similar opinions remarking each other’s work

The partnering gap defines how frequently a marker’s benchmarks can be marked by each other marker



Suspect cap

Too high?

• Markers could potentially carry on marking when their marking is eventually shown to be substandard

Too low?

• Markers could be stopped from marking too frequently and, potentially, unnecessarily

The suspect cap defines how many adjudications can be outstanding for a marker. This prevents markers from getting too far ahead without the senior checking their work



Penalty cap

Too high?

• A marker below the standards required will be allowed to continue marking

Too low?

• Markers stopped too often, and potentially, unnecessarily

• Additional work for senior markers in stopped marker management

The penalty cap defines how many adjudications can be delivered against a marker before the stopped marker management process is invoked



Marker quality review

• Marker review process is used during percentage double marking– Can be used if there are concerns about a particular

marker or about marking standards at a particular time– Senior markers can request that a greater proportion of

marking is checked– Can be done retrospectively, for example, work done in

the past can be checked



Marker review process

Marker 1

Marker 2Marker 3



Stopped markers



Stopped markers

• Senior markers can view quality failures of markers within stopped marker management. They can:– Permanently stop markers– Restart markers– For seeding and S-Process, can identify if quality check is

not of a good standard



Setting the parameters



Seeding parameters

• The function and application of seeding is governed by parameters set in the e-Marker® Administration system

• These parameters define:– The number of seeds in qualification– The frequency at which seeds are presented in marking– The acceptable number of seeds that can be failed

before a marker is stopped– The minimum seed bank level required



Percentage double marking parameters

• The function and application of percentage double marking is governed by parameters set in the e-Marker® Administration system

• These parameters define:– The frequency with which marking is benchmarked– The pioneer cap– The partnering gap– The suspect cap– The penalty cap



Parameter modelling


Controlling Marking Quality

in e-Marker®

education | elections | census controlling marking quality in e-marker ®

Documents

quality of markers

controlling marking

marking window

quality control questions

quality checkssenior

marking of exemplar

senior markers seeds

quality methods