ESMEA online

2017

+

To assure standards at graduation from

medical school or postgraduate certification

To award the MBBS to students who meet the

University’s standards (University interest)

To certify that graduates are suitable for provisional

registration (GMC interest)

To ensure graduates are fit to undertake residency

posts (NHS/employer interest)

To award PG certification to those who have met

required specialist standards

To distinguish between the competent and the

insufficiently competent (Public interest)

Standard setting – why bother?

+

Test

Result

Pass

Fail

Competent Incompetent

+Definition of Standards

A standard is a statement about whether an examination performance is good enough for a particular purpose

Standards are based on judgments about examinees’performances against a social or educational construct

TerminologyPassing standard

Pass mark

Cut score

A special score that serves as the boundarybetween passing and failing

+ Types of Standard Setting

Methods

Relative standards: norm referenced methods Candidates are compared to others in the group

A set proportion of candidates fails regardless of how well they perform

Competitive situation eg selection

Absolute standards: criterion referenced methods Candidates are compared to a set standard

Define the outcomes of course

Competence eg tests of knowledge/skills

Compromise methods

+ What does this mean in

practice?

‘Historical precedent’ cut scores are no longer credible for use in high stakes summative tests Even if your HE institution requires it

We are dealing with health professional competence

Setting passing standards should be determined by careful analysis and judgement of what acceptable performance is This may vary from test to test

Decisions around how to set cut scores should draw on the best available evidence Defensibility

+ Standards

Standards are based on judgments about

examinees’ performances against a social or

educational construct

Requires judgment of experts

Dependent on context

e.g. Competent practitioner or student ready for

graduation

+

Standard setting at graduation level

Pass/Fail: distinguish between

the competent and the not yet

sufficiently competent

Level: what a candidate knows/is able to do to

become a FY1 doctor/ Day 1vet /

Day 1 dentist/ graduate nurse

What is the purpose of standard

setting at Finals/graduation?

+

Pass/Fail: distinguish between

the competent and the not yet

sufficiently competent at the

specified level

Level: what a candidate knows/is able to do to

become a specialist/enter a speciality

training programme

What is the purpose of standard

setting at postgraduate

certification levels?

+

Standard Setting Methods

+Principles of standard setting

Method used should be:

Credible

Defensible

Feasible

Acceptable to all stakeholders

Choice of method less important than whether it:

Relies on informed expert judgement

Demonstrates due diligence

Produces standards consistent with purpose of the test

Is easy to explain and implement

Supported by body of evidence in the literature

Norcini, J. J. (2003). Setting standards on educational tests. Medical Education, 37, 464-469.

+Setting the pass mark

Who should decide on the standards?

Teachers of the students

Academics and clinicians

Familiar with the curriculum

Familiar with levels of student knowledge

Understand the purpose and level of examination

Familiar with ‘what is good enough’ for this particular test

+Defining the ‘Borderline

Candidate’

Group of judges should discuss:

Nature and purpose of examination

Nature and level of examinees

What is good enough for this particular test

Thinking about the JUST GOOD enough

candidate

Characteristics of their knowledge base

+ Criterion-referenced Methods

Panel Judgments About Individual

Test Items

Angoff’s method

Ebel’s method

+Angoff method

Each judge estimates the probability of the

‘just good enough candidate’ getting the

correct answer to each item

Between 0 and 100 % likelihood

Each judge allocates estimates to every item

Discussion among judges about differences

Adjustments

Final pass mark is summated

Angoff Method

Items Judge1 Judge2 Judge3 Judge4 Judge5 Mean

1 60 70 55 75 65 65

2 80 90 85 95 90 88

3 70 75 80 75 40 68

4 45 55 50 60 55 53

5 90 95 85 95 90 91

6 30 40 30 50 40 38

7 60 60 50 70 60 60

Total

463

62% 70% 74%62% 62%Pass mark 463/7 = 66%

Ebel method - 1

Easy Medium Hard

Essential

Important

Acceptable

RELEVANCE

DIFFICULTY

Judges make “Difficulty-Relevance” decisions

Ebel method - 2

Weighting Easy Medium Hard

Essential 90% 80% 70%

Important 60% 50% 40%

Acceptable 30% 20% 10%

Ebel method - 3

Weighting Easy Medium Hard

Essential 90% x 2 =

1.8

80% x 3 =

2.4

70% x 1 =

0.7

Important 60% x 3 =

1.8

50% x 3 =

1.5

40% x 1 =

0.4

Acceptable 30% x 1 =

0.3

20% x 1 =

0.2

10% x 1 =

0.1

16 items

{Sum/ no of items} x

100 = {9.2/16}x100 =

57.5%

Weighting x

number of items

+ Criterion referenced methods: Angoff

and Ebel

Advantages

They focus attention on item content

They are relatively easy to use

There is a considerable body of published work supporting their use

They are used frequently in high stakes testing

Disadvantages

The concept of a "borderline group" is sometimes difficult to define

Judges sometimes feel they are "pulling numbers out of the air” (Angoff)

The methods can be tedious & time-consuming

+Standard setting methods

for OSCEs

Methods specifically

developed for OSCEs

Borderline Group Method

Contrasting group method

Borderline Regression Method

Boulet, J. R.; De Champlain, A.F.; McKinley, D.W. (2003) Setting defensible performance

standards on OSCEs and standardized patient examinations. Medical Teacher, 3, 245-249.

+ The examiner’s role in

standard setting

Examiner’s clinical expertise is used to judge the candidate’s performance overall (separate from scoring rubric)

Examiner allocates a global judgement based on the candidate’s performance at that station

Remembering the level of the examination

FailBorderline

Pass

Checklist1. Hs shjs sjnhss sjhs sjs sj

2. Ksks sksmsiqopql qlqmq q q qkl

3. Lalka kdm ddkk dlkl dlld

4. Keyw dd e r rrmt tmk

5. Jfjfk dd

6. Hskl;s skj sls ska ak akl ald

7. Hdhhddh shs ahhakk as

TOTAL

Passing score

Borderline score distribution

Pass, Fail, Borderline

Test score distribution

Borderline Group Method

Checklist

1. Hs shjs sjnhss sjhs sjs sj

2. Ksks sksmsiqopql qlqmq q q qkl

3. Lalka kdm ddkk dlkl dlld

4. Keyw dd e r rrmt tmk

5. Jfjfk dd

6. Hskl;s skj sls ska ak akl ald

7. Hdhhddh shs ahhakk as

TOTAL

Overall rating 1 2 3 4 5

1 2 3 4 5

Checklist

Score

X

X = passing score

1 = Clear fail2 = Borderline3 = Clear pass4 = Very good5 = Outstanding

Clear Borderline Clear Very Outstanding

fail pass good

Borderline Group Method

+ Performance-based standard setting

Advantages

Utilises the expertise of

clinical examiners

Real-time observation

Large number of examiners

set a collective standard while

observing the candidates –

not just an academic exercise

Reliable: cut-off score based

on large sample of judgments

There is a considerable body

of published work supporting

their use

Credible and defensible:

based on expert judgment in

direct observation

Disadvantages

• Requires large cohort of candidates to achieve enough numbers in the‘borderline’group

• Passing score not known in advance

• Judgments not independent of checklist scoring

• Examiners require training

• Methods require expert processing of marks immediately after the exam

• Delay in producing results

+ Implementation Guidelines for

Setting Standards

There is no substitute for a considered assessment strategy, high quality items and careful test construction

Choose the method which suits your context best Consider resources: time, individuals

Select the judges

Assign an appropriate number (at least 6-8 for high stakes testing)

Select the characteristics the group should possess

Develop an efficient design for the process

+ The choices

Different standard setting methods lead to

different cut off scores – no gold standard exists

Robust method should be credible and acceptable

and produce realistic outcomes

Process is more important than method in overall

validity of your assessment – collect and analyse

your own data

Not using a recognised method is NOT a viable

choice for assessing the competence of this

century’s health professionals

+

Thank you!

Email:

katharine.boursicot@hpac.sg

Our website: http://hpac.sg