standard setting for clinical assessments katharine boursicot, bsc, mbbs, mrcog, mahpe reader in...
TRANSCRIPT
![Page 1: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/1.jpg)
Standard setting for clinical assessments
Katharine Boursicot, BSc, MBBS, MRCOG, MAHPEReader in Medical EducationDeputy Head of the Centre for Medical and Healthcare EducationSt George’s, University of London
The Third International Conference on Medical Education in the Sudan
![Page 2: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/2.jpg)
WHAT are we testing in clinical assessments?
• Clinical competence
• What is it?
![Page 3: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/3.jpg)
A popular modern model: elements of competence
• Knowledgeo factualo applied: clinical reasoning
• Skillso communicationo clinical
• Attitudeso professional behaviour
Tomorrow’s Doctors, GMC 2003
![Page 4: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/4.jpg)
Knows
Knows how
Shows how
Behaviour~ skills/attitudes
Cognition~ knowledge
Another popular medical model of competence
Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S67.
Does
Pro
fess
ion
al
auth
enti
city
![Page 5: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/5.jpg)
Assessment of competence
• A review of developments over the last 40 years
![Page 6: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/6.jpg)
Knows
• 1960: National Board of Medical Examiners in the USA introduced the MCQ
• MCQs conquered the world
• Dissatisfaction due to limitation of MCQs
Knows
Knows how
Shows how
Does
![Page 7: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/7.jpg)
Knows how
• 1965: Introduction of PMPPatient Management Problem
Knows
Knows how
Shows how
Does
![Page 8: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/8.jpg)
Patient Management Problem
Clinical Scenario
Action Action Action
Action Action
Action Action Action
Action Action ActionAction Action Action
Action
![Page 9: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/9.jpg)
Knows how
• 1965: Introduction of PMPo Patient Management Problem
• Well constructed SBA format MCQs can test the application of knowledge very effectively
Knows
Knows how
Shows how
Does
![Page 10: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/10.jpg)
Shows how
• 1975: Introduction of Objective Structured Clinical Examination (OSCE)
OSCEs are conquering the world
Knows
Knows how
Shows how
Does
![Page 11: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/11.jpg)
Does
• > 2000: emerging new methods
• WBAs – Workplace-Based Assessmentso Mini Clinical Examination Exerciseo Direct Observation of Practical Procedureo OSATSo Masked standardized patientso Video assessmento Patient reportso Peer reportso Clinical work samples
o ………
Knows
Knows how
Shows how
Does
![Page 12: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/12.jpg)
Mini CEX (Norcini, 1995)
• Short observation (15-20 minutes) and evaluation of clinical performance in practice using generic evaluation forms completed by different examiners
(cf. http://www.abim.org/minicex/)
![Page 13: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/13.jpg)
Example of mini-CEX form
![Page 14: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/14.jpg)
DOPS – Direct Observation of Practical Procedure
![Page 15: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/15.jpg)
OSATS – Objective structured Assessment of Technical Skills
![Page 16: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/16.jpg)
WBAs – Workplace-Based Assessments
• All based on the principle of an assessor observing a student/trainee in a workplace or practice setting
![Page 17: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/17.jpg)
Past 40 years: climbing the pyramid.....
Knows
Shows how
Knows how
Does
Knows Factual tests: SBA-type MCQs…..
Knows how (Clinical) Context based tests:SBA, EMQ, MEQ…..
Shows how Performance assessment in vitro:OSCEs
DoesPerformance assessment in vivo:Mini-CEX, DOP, OSATS, …..
![Page 18: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/18.jpg)
Standard setting – why bother?
• To assure standards o At graduation from medical schoolo For licensingo For a postgraduate (membership) degreeo For progression from one grade to the nexto For recertification
![Page 19: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/19.jpg)
At graduation from medical school
• To award a medical degree to students who meet the University’s standards (University interest)
• To distinguish between the competent and the insufficiently competent (Public interest)
• To certify that graduates are suitable for provisional registration (Regulatory/licensing body interest)
• To ensure graduates are fit to undertake F1 posts (employer interest)
![Page 20: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/20.jpg)
Definition of Standards
• A standard is a statement about whether an examination performance is good enough for a particular purposeo a particular score that serves as the
boundary between passing and failingo the numerical answer to the question
“How much is enough?”
![Page 21: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/21.jpg)
Standard setting
All methods described in the literature are based on ways of translating
expert (clinical) judgement
into a score
![Page 22: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/22.jpg)
‘Classical’ standard setting methods
• For written test items:o Angoff’s methodo Ebel’s method
• For OSCEs:o Borderline group methodo Regressions based method
![Page 23: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/23.jpg)
Performance-based standard setting methods
• Borderline group method
• Contrasting group method
• Regression based standard method
Kramer A, Muijtjens A, Jansen K, Düsman H, Tan L, van der Vleuten C Comparison of a rational and an empirical standard setting procedure for an OSCE, Medical Education, 2003 Vol 37 Issue 2, Page 132
Kaufman DM, Mann KV, Muijtjens AMM, van der Vleuten CPM. A comparison of standard-setting procedures for an OSCE in undergraduate medical education. Acad Med 2000; 75:267-271.
![Page 24: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/24.jpg)
![Page 25: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/25.jpg)
The examiner’s role in standard setting
• Uses the examiner’s clinical expertise to judge the candidate’s performance
• Examiner allocates a global judgement based on the candidate’s performance at that station
• Remember the level of the examination Pass
BorderlineFail
![Page 26: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/26.jpg)
Borderline Group Method
Checklist1.
2.
3.
4.
5.
6.
7.
TOTAL
Passing score
Borderline score distribution
Pass, Fail, Borderline
Test score distribution
![Page 27: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/27.jpg)
Contrasting groups method
Checklist
1.
2.
3.
4.
5.
6.
7.
TOTAL
Pass, Fail, Borderline P/B/F
Test score distribution
Passing score
PassFail
![Page 28: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/28.jpg)
Regression based standard
Checklist
1.
2.
3.
4.
5.
6.
7.
TOTAL
Overall rating 1 2 3 4 5
1 2 3 4 5
ChecklistScore
X
X = passing score
1 = Clear fail2 = Borderline3 = Clear pass4 = Excellent5 = Outstanding
Clear Borderline Clear Excellent Outstanding fail pass
![Page 29: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/29.jpg)
Work Based Assessment tools
• No gold standard
standard setting method!
![Page 30: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/30.jpg)
Standard setting
• Standards are based on informed judgments about examinees’ performances against a social or educational construct
e.g.
• competent practitioner
• suitable level of specialist knowledge/skills
![Page 31: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/31.jpg)
Standard setting for Work Based Assessment tools
o Based on descriptors for a particular level of training
o Information gathering relying on descriptive and qualitative judgemental information
o Descriptors agreed by consensus/panel of clinical experts
o Purpose of WBA tools: formative rather than summative: feedback
![Page 32: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/32.jpg)
Feedback
• Giving feedback to enhance learning is some form of judgement by the feedback giver on the knowledge and performance of the recipient
• It is a very powerful tool!
![Page 33: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/33.jpg)
WBAs and feedback
• Underlying principle of WBA tools is FEEDBACK fromo Teacher/supervisoro Peers/team memberso Other professionalso Patients
![Page 34: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/34.jpg)
Conclusions
• It’s not easy to set standards for Work Based Assessments (in the ‘classic’ sense)
• Expert professional judgement is required• Wide sampling from different sources:
range of tools, contexts, cases and assessors
• Feedback to the trainee
![Page 35: Standard setting for clinical assessments Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical](https://reader036.vdocuments.us/reader036/viewer/2022081518/551b6c6d550346ae7a8b60b3/html5/thumbnails/35.jpg)