Moving from opaque to clear standards

Professor Tansy JessopTESTA Workshop

University College Dublin10 February 2017

What the literature says…

Marking is important. The grades we give students and the decisions we make about whether they pass or fail coursework and examinations are at the heart of our academic standards (Bloxham, Boyd and Orr 2011).

Grades matter (Sadler 2009).

What the papers say…

https://www.timeshighereducation.co.uk/news/examiners-give-hugely-different-marks/2019946.article

QAA: a paradigm of accountability

• Learning outcomes• Criteria-based learning• Meticulous specification• Written discourse• Generic discourse (Woolf 2004)• ‘validating practices’ (Shay 2004) • Transparent to staff and students• Intended to reduce the arbitrariness of staff

decisions (Sadler 2009).

TESTA data on internalising standards

• Consistently low scores on the AEQ for clear goals and standards

• Alienation from the tools, especially criteria and guidelines

• Symptoms: perceptions of marker variation, unfair standards and inconsistencies in practice

Hang on, this is puzzling

We’ve got two tutors- one marks completely differently to the other and it’s pot luck which one you get.

They have different criteria, they build up their own criteria.

It’s such a guessing game.... You don’t know what they expect from you.

They read the essay and then they get a general impression, then they pluck a mark from the air.

What’s going wrong here?

There are criteria, but I find them really strange. There’s “writing coherently, making sure the argument that you present is backed up with evidence”. Q: If you could change one thing to improve what would it be?A: More consistent marking, more consistency across everything and that they would talk to each other.

Take five

• Which quote strikes a chord for you?

• What is the central problem?

• Any ideas to solve it?

Is this quite ‘normal’?

Differences between markers are not ‘error’, but rather the inescapable outcome of the multiplicity of perspectives that assessors bring with them (Shay 2005, 665).

The tension between ‘the scientific aspirations of assessment technologies to represent an objective reality and the unavoidable subjectivities injected by the human focus of these technologies’ (Broadfoot 2002, 157).

ImplicitCriteria

Explicit WrittenI justify

Co-creation and

participation

Active engagemen

t by students

Is there a right way to mark….?

Having ‘an eye for a dog’

The Art and Science of Evaluation

Judging is both an art and a science: It is an art because the decisions with which a judge is constantly faced are very often based on considerations of an intangible nature that cannot be recognized intuitively. It is also a science because without a sound knowledge of a dog’s points and anatomy, a judge cannot make a proper assessment of it whether it is standing or in motion.

Take them round please: the art of judging dogs (Horner, T 1975).

Criteria problem 1: Analytic Vs Holistic

Analytic grading:Separate qualitative judgements on each of the criteria. After criterion by criterion judgements are made, they are combined, usually by way of a formula. The resulting aggregate is converted into a grade.

Holistic grading

The assessor progressively builds up a complex mental response to a student work. Attending to particular aspects and allowing an appreciation of the work as a whole to emerge.

Criteria Problem 2: How many criteria are enough?

Criteria problem 3: Vive la difference?

Different lecturers use different sets of criteria. Selecting some excludes others, is one more legitimate than another? What effect would applying different criteria have? What signals do using different sets of criteria send to students?• Does your department use the same criteria?• Does it matter?• Does it influence grades?

Criteria problem 4: Staff and students see criteria differently

Criteria problem 5: Staff do not always share the same understanding

What’s behind the marking mask?

• Values• Interpretation• Connoisseurship• Tacit understanding• Subjective readings• Privatisation• Exposing what we think

Marking as social practice

Situated in the discipline – not just ‘an eye for a dog’

Far from being mere personal opinion or an arbitrary ‘taste’ or ‘gut-feel’, this subjective reading is a socially constituted, practical mastery (Shay 2005).

Marking as social practice

Marking as social practice

The typical technologies of our assessment and moderation systems – marking memorandum, double-marking, external examiners – privilege reliability. These technologies are not in themselves problematic. The problem is our failing to use these technologies as opportunities for dialogue about what we really value as assessors, individually and as communities of practice

(Shay 2005).

Marking as social practice

Taking action: internalising goals and standards• Regular calibration exercises• Discussion and dialogue• Discipline specific criteria (no cut and paste)

Lecturers

• Rewrite/co-create criteria• Marking exercises • Discussing exemplars

Lecturers and

students

• Enter secret garden - peer review• Engage in drafting processes• Self-reflection

Students

ReferencesBloxham, S. , P. Boyd, and Orr S. (2011) Mark my words: the role of assessment criteria in UK higher education practices. Studies in Higher Education. 36.6. 655-670.O'Donovan, B , Price, M. and Rust, C(2008) 'Developing student understanding of assessment standards: a nested hierarchy of approaches', Teaching in Higher Education, 13: 2, 205 — 217http://dx.doi.org/10.1080/13562510801923344D. Royce Sadler (2009) Indeterminacy in the use of preset criteria for assessment and grading, Assessment & Evaluation in Higher Education, 34:2, 159-179. http://dx.doi.org/10.1080/02602930801956059Shay, S.B. 2005. The assessment of complex tasks: A double reading. Studies in Higher Education. 30: 663–79.Woolf, H. (2004) Assessment criteria: Reflections on current practices. Assessment and Evaluation in Higher Education 24:4 479-93.