Work based assessmenth.davies@shef.ac.uk

Challenges and opportunities

Choosing assessments?

What do you want to assess - e.g. Professionalism Clinical reasoning Technical skills Clinical evaluation

Who do you want to assess Med student Resident Practising physician

Choosing assessments

Patient outcome gold standard but attribution a problem

Judgements largely based on process measures

Case mix and reliability an issue

Assessors

If sufficient subjective judgements are combined the

collated judgement about performance can be reliable

i.e objectivity and reliability are not the same thing

Assumes that assessor has both observed competence

in question and can make a judgement about its

quality

How many assessors is enough?

MSF in healthcare settings

Feedback about observable behaviors is provided by some or all of Physician colleagues (peers, referring MDs, referral MDs) Co-workers (e.g., nurses, pharmacists, dieticians) Patients Self

If trainee borderline or in difficulty sample more

0

1

2

3

4

5

6

0

1

2

3

4

5

6

Assessors Self

Insight

MSF

One to one feedback Problems may need further diagnostic work Need support mechanisms in place

Bias

Consider sources of bias such as Gender Working relationship Working environment Ethnic group University of graduation

Cons SASG

F1/F2

SHO

SpR

Nurse

F=524.1

p= <0.001

GP

AHP

Mini-CEX case complexity

Validity

Extent to which a test assesses what it purports to Content validity

Criterion validity

Construct validity

Face validity

Consequential validity

Recognised need for further validity data in relation to

MSF and other work based assessments1

Evans, R., G. Elwyn, and A. Edwards, Review of instruments for peer assessment of physicians. BMJ, 2004. 328(7450): p. 1240.

Validity

Content validity Blueprint assessments to the curriculum

Evidence from a range of sources supporting the hypothesis that the score really measures what it is meant to

Relationship with patient ratings

SHEFFPAT vs SPRAT r=0.12 NS

SHEFFPAT vs PATSCOREr=0.45 p<0.01

Predictive and consequential validity

Centralisation

Facilitates standardisation and robust QA Facilitates movement between locations Economies of scale Strategies to enhance local ownership

important

Practicalities

Is it feasible? Electronic vs paper Centralised vs localised Who assesses? How is training organised? Who will do the QA?

Acknowledge legitimate concerns

Healthy scepticism Resources - especially time Conflicting demands espy CLINICAL Need for sampling Lack of standardisation Training needs Loss of local ownership

Support doctors in difficulty

Be patient

Op-ti-mist n Somebody who tends to feel hopeful and positive

about future outcomes Recognise extent of cultural change Ensure work based assessment is done as

well as possible

“ Every problem is just an opportunity waiting to be made use of ”

Who assesses?

57.6%

14.8% 19.4%

7.1%

Consultant 1SASG 2SpR 3GP 4

The programme should be quality assured

RR VV EE CC AA

importanceimportance

100 %100 %

0 %0 %

aspectaspectFF

van der Vleuten C. The assessment of professional competence: developments, research and practical implications. Advances in Health Sciences Education. 1996;1:41-67.

Sampling

Content specificity Being good at one thing doesn’t mean you are

good at everything Must sample clinical content widely Map to curriculum - blueprint

Sources of variance Assessors significant source of variance Use lots of assessors

Classification Scheme for Work-Based Assessment

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Norcini BMJ 2003

Quality assurance

Quality assure assessment system

Modify assessment system in response to QA

Effective (work based) assessment

RESOURCES and CO-ORDINATION

TIME

FUNDING

0

1

2

3

4

5

6

7

RB SELF PRACT

Key messages

Sampling is crucialConsider sources of biasHave as many different clinical

problems and as many assessors as possible

Subjectivity ≠ unreliabilityTrain the raters

Holmboe, E. S., R. E. Hawkins, et al. (2004). "Effecs of training in direct observation of medical residents' clinical competence; a randomized trial." Ann Intern Med 140: 874-881.

PERFORMANCEOF

PHYSICIAN

Personal pressures: •Home•Health

Attributes of training:

•Post •Programme•Supervisor

Context of training:•Workload

•Colleagues•Patients

Attributes of physician: •Personality

•Background •Aptitude for specialty

MSF