education | elections | census controlling marking quality in e-marker ®

28
Education | Elections | Census Controlling marking quality in e-Marker ®

Upload: silvia-parks

Post on 30-Dec-2015

216 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Education | Elections | Census Controlling marking quality in e-Marker ®

Education | Elections | Census

Controlling marking quality

in e-Marker®

Page 2: Education | Elections | Census Controlling marking quality in e-Marker ®

Education | Elections | Census

Commercial in confidence 2012

Marking quality control

• Marking quality can be delivered by:– e-Standardisation– Seeding and qualification– Percentage double marking– S-Process

Page 3: Education | Elections | Census Controlling marking quality in e-Marker ®

Education | Elections | Census

Commercial in confidence 2012

Seeding or percentage double marking?

• If the marking guidance is detailed and objective and you would expect complete agreement among markers – use seeding

• If the marking guidance is subjective – or gives guidance for the approach to marking, rather than what is a right or wrong answer – use percentage double marking

• You can use seeding and percentage double marking on the same component

• S-Process can be added to both quality methods for further quality control if desired

Page 4: Education | Elections | Census Controlling marking quality in e-Marker ®

Education | Elections | Census

Commercial in confidence 2012

Seeding and qualification

Page 5: Education | Elections | Census Controlling marking quality in e-Marker ®

Education | Elections | Census

Commercial in confidence 2012

Seeds

• Seeds provide an in-built quality check• The administrators define and maintain the levels and

frequency of the quality checks• Senior markers pre-mark a selection of responses - seeds• Quality control applies at individual question level• Quality is checked at the start of each day (qualification)

and while the markers mark (seeding)• Markers can see the seeds they have failed so that they can

improve their marking

Page 6: Education | Elections | Census Controlling marking quality in e-Marker ®

Education | Elections | Census

Commercial in confidence 2012

Seeding

Qualify at start of day

Mark seeds

Mark batch of

questions

Pass FailStop

MarkingQuestion

Quality check: pass 9 out of 10

Mark batch of

questions

Start marking after discussions with Senior or mark other questions

Quality check: must not fail both seeds

Quality check: must not fail both seeds

Page 7: Education | Elections | Census Controlling marking quality in e-Marker ®

Education | Elections | Census

Commercial in confidence 2012

Seed window

Quality check:7 out of 10 within tolerance

Page 8: Education | Elections | Census Controlling marking quality in e-Marker ®

Education | Elections | Census

Commercial in confidence 2012

Seed window

Quality check:7 out of 10 within tolerance

Page 9: Education | Elections | Census Controlling marking quality in e-Marker ®

Education | Elections | Census

Commercial in confidence 2012

Seeding is set by senior markers

• Seeding needs to be done before the markers start marking• Ensure there are enough seeds – the minimum seed bank

size should be reached• If seeds are retired, they will need replacing• On a seeded question, the senior markers’ mark will be

awarded to the candidate

Page 10: Education | Elections | Census Controlling marking quality in e-Marker ®

Education | Elections | Census

Commercial in confidence 2012

Seeding for real

Page 11: Education | Elections | Census Controlling marking quality in e-Marker ®

Education | Elections | Census

Commercial in confidence 2012

What makes a good seed?

• Legible• Unambiguous and complete• Reflects the standards agreed in standardisation• No tricks!• Does not have to be the right answer• Don’t set “not attempted” responses as seeds

Page 12: Education | Elections | Census Controlling marking quality in e-Marker ®

Education | Elections | Census

Commercial in confidence 2012

Percentage double marking

Page 13: Education | Elections | Census Controlling marking quality in e-Marker ®

Education | Elections | Census

Commercial in confidence 2012

Percentage double marking

• Percentage double marking involves:– Comparing two marking opinions in real time at question

level;– Where differences in marking exceed a set tolerance,

automated quality checks are used to invoke adjudication by a senior marker or peer

Page 14: Education | Elections | Census Controlling marking quality in e-Marker ®

Education | Elections | Census

Commercial in confidence 2012

Percentage double marking

1

2

The Senior Marker adjudicates. In this case he/she agrees with Marker 2, so Marker 1 receives a penalty

Marker 2 marks the same clip, not knowing it has been marked before. If Markers 1 and 2 agree, no further action is taken and the mark is awarded to the candidate. In this example however, it is marked outside tolerance of Marker 1, so the marking of both markers is forwarded for adjudication by the Senior Marker

Marker 1 receives a portion to mark. It is selected by e-Marker® as a quality control item (benchmark item) and is forwarded to the next available marker

Page 15: Education | Elections | Census Controlling marking quality in e-Marker ®

Education | Elections | Census

Commercial in confidence 2012

Percentage double marking

• There are controls in e-Marker® administration to:– Ensure that markers mark at a similar pace;– Don’t collude in their marking;– Identify when discrepancies are forwarded to peer or

senior; and– Identify when adjudication goes against a marker

Page 16: Education | Elections | Census Controlling marking quality in e-Marker ®

Education | Elections | Census

Commercial in confidence 2012

Pioneer cap

Too high?

• First marker could mark their entire quota without being checked against other markers

• Be particularly careful in small marker pools

Too low?

• Could slow down marking progress with markers being halted too often

• Could make the frequency of the quality checks overt to markers

The pioneer cap defines how many benchmarks the fastest marker can be ahead of the next fastest. The prevents one marker marking too much without any quality control

Page 17: Education | Elections | Census Controlling marking quality in e-Marker ®

Education | Elections | Census

Commercial in confidence 2012

Partnering gap

Too high?

• Marking is likely to be slowed down as new markers are needed

• Could leave difficult, outstanding parts to mark at the end of marking

Too low?

• Could have markers colluding, or markers with similar opinions remarking each other’s work

The partnering gap defines how frequently a marker’s benchmarks can be marked by each other marker

Page 18: Education | Elections | Census Controlling marking quality in e-Marker ®

Education | Elections | Census

Commercial in confidence 2012

Suspect cap

Too high?

• Markers could potentially carry on marking when their marking is eventually shown to be substandard

Too low?

• Markers could be stopped from marking too frequently and, potentially, unnecessarily

The suspect cap defines how many adjudications can be outstanding for a marker. This prevents markers from getting too far ahead without the senior checking their work

Page 19: Education | Elections | Census Controlling marking quality in e-Marker ®

Education | Elections | Census

Commercial in confidence 2012

Penalty cap

Too high?

• A marker below the standards required will be allowed to continue marking

Too low?

• Markers stopped too often, and potentially, unnecessarily

• Additional work for senior markers in stopped marker management

The penalty cap defines how many adjudications can be delivered against a marker before the stopped marker management process is invoked

Page 20: Education | Elections | Census Controlling marking quality in e-Marker ®

Education | Elections | Census

Commercial in confidence 2012

Marker quality review

• Marker review process is used during percentage double marking– Can be used if there are concerns about a particular

marker or about marking standards at a particular time– Senior markers can request that a greater proportion of

marking is checked– Can be done retrospectively, for example, work done in

the past can be checked

Page 21: Education | Elections | Census Controlling marking quality in e-Marker ®

Education | Elections | Census

Commercial in confidence 2012

Marker review process

Marker 1

Marker 2Marker 3

Page 22: Education | Elections | Census Controlling marking quality in e-Marker ®

Education | Elections | Census

Commercial in confidence 2012

Stopped markers

Page 23: Education | Elections | Census Controlling marking quality in e-Marker ®

Education | Elections | Census

Commercial in confidence 2012

Stopped markers

• Senior markers can view quality failures of markers within stopped marker management. They can:– Permanently stop markers– Restart markers– For seeding and S-Process, can identify if quality check is

not of a good standard

Page 24: Education | Elections | Census Controlling marking quality in e-Marker ®

Education | Elections | Census

Commercial in confidence 2012

Setting the parameters

Page 25: Education | Elections | Census Controlling marking quality in e-Marker ®

Education | Elections | Census

Commercial in confidence 2012

Seeding parameters

• The function and application of seeding is governed by parameters set in the e-Marker® Administration system

• These parameters define:– The number of seeds in qualification– The frequency at which seeds are presented in marking– The acceptable number of seeds that can be failed

before a marker is stopped– The minimum seed bank level required

Page 26: Education | Elections | Census Controlling marking quality in e-Marker ®

Education | Elections | Census

Commercial in confidence 2012

Percentage double marking parameters

• The function and application of percentage double marking is governed by parameters set in the e-Marker® Administration system

• These parameters define:– The frequency with which marking is benchmarked– The pioneer cap– The partnering gap– The suspect cap– The penalty cap

Page 27: Education | Elections | Census Controlling marking quality in e-Marker ®

Education | Elections | Census

Commercial in confidence 2012

Parameter modelling

Page 28: Education | Elections | Census Controlling marking quality in e-Marker ®

Education | Elections | Census

Controlling Marking Quality

in e-Marker®