standard setting determining the pass mark - osces
Post on 21-Dec-2015
238 views
TRANSCRIPT
![Page 1: Standard setting Determining the pass mark - OSCEs](https://reader030.vdocuments.us/reader030/viewer/2022032521/56649d5e5503460f94a3de39/html5/thumbnails/1.jpg)
Standard setting Determining the pass mark
- OSCEs
![Page 2: Standard setting Determining the pass mark - OSCEs](https://reader030.vdocuments.us/reader030/viewer/2022032521/56649d5e5503460f94a3de39/html5/thumbnails/2.jpg)
The Old Way (1)
……..I think that ..I think that the pass mark the pass mark in this is exam in this is exam is probably is probably about here…..about here…..
there is a there is a natural break I natural break I can see can see dividing the dividing the bottom from bottom from the restthe rest
![Page 3: Standard setting Determining the pass mark - OSCEs](https://reader030.vdocuments.us/reader030/viewer/2022032521/56649d5e5503460f94a3de39/html5/thumbnails/3.jpg)
The Old Way (2)
……..The pass ..The pass mark is 60%.mark is 60%.
![Page 4: Standard setting Determining the pass mark - OSCEs](https://reader030.vdocuments.us/reader030/viewer/2022032521/56649d5e5503460f94a3de39/html5/thumbnails/4.jpg)
Assessments
![Page 5: Standard setting Determining the pass mark - OSCEs](https://reader030.vdocuments.us/reader030/viewer/2022032521/56649d5e5503460f94a3de39/html5/thumbnails/5.jpg)
Assessments
Match OutcomeObjectives
![Page 6: Standard setting Determining the pass mark - OSCEs](https://reader030.vdocuments.us/reader030/viewer/2022032521/56649d5e5503460f94a3de39/html5/thumbnails/6.jpg)
Assessments
Match OutcomeObjectives
Integrated
![Page 7: Standard setting Determining the pass mark - OSCEs](https://reader030.vdocuments.us/reader030/viewer/2022032521/56649d5e5503460f94a3de39/html5/thumbnails/7.jpg)
Assessments
Match OutcomeObjectives
IntegratedClinical Competenciestested throughout
![Page 8: Standard setting Determining the pass mark - OSCEs](https://reader030.vdocuments.us/reader030/viewer/2022032521/56649d5e5503460f94a3de39/html5/thumbnails/8.jpg)
Assessments
Match OutcomeObjectives
IntegratedClinical Competenciestested throughout
Progressivetesting
![Page 9: Standard setting Determining the pass mark - OSCEs](https://reader030.vdocuments.us/reader030/viewer/2022032521/56649d5e5503460f94a3de39/html5/thumbnails/9.jpg)
Assessments
Match OutcomeObjectives
IntegratedClinical Competenciestested throughout
Progressivetesting
Housestyle
![Page 10: Standard setting Determining the pass mark - OSCEs](https://reader030.vdocuments.us/reader030/viewer/2022032521/56649d5e5503460f94a3de39/html5/thumbnails/10.jpg)
Assessments
Match OutcomeObjectives
IntegratedClinical Competenciestested throughout
Progressivetesting
Housestyle
From approvedlist
![Page 11: Standard setting Determining the pass mark - OSCEs](https://reader030.vdocuments.us/reader030/viewer/2022032521/56649d5e5503460f94a3de39/html5/thumbnails/11.jpg)
Assessments
Match OutcomeObjectives
IntegratedClinical Competenciestested throughout
Progressivetesting
Housestyle
From approvedlist
Studentinformation
![Page 12: Standard setting Determining the pass mark - OSCEs](https://reader030.vdocuments.us/reader030/viewer/2022032521/56649d5e5503460f94a3de39/html5/thumbnails/12.jpg)
Assessments
Match OutcomeObjectives
IntegratedClinical Competenciestested throughout
Progressivetesting
Housestyle
From approvedlist
Studentinformation
External Examiners
![Page 13: Standard setting Determining the pass mark - OSCEs](https://reader030.vdocuments.us/reader030/viewer/2022032521/56649d5e5503460f94a3de39/html5/thumbnails/13.jpg)
Standard setting
Knowledge Skills
![Page 14: Standard setting Determining the pass mark - OSCEs](https://reader030.vdocuments.us/reader030/viewer/2022032521/56649d5e5503460f94a3de39/html5/thumbnails/14.jpg)
Angoff
Hofstee
Borderline method
Knowledge Skills
Standard setting
![Page 15: Standard setting Determining the pass mark - OSCEs](https://reader030.vdocuments.us/reader030/viewer/2022032521/56649d5e5503460f94a3de39/html5/thumbnails/15.jpg)
Borderline procedureBorderline procedure
Used for clinical (OSCE) examinations
![Page 16: Standard setting Determining the pass mark - OSCEs](https://reader030.vdocuments.us/reader030/viewer/2022032521/56649d5e5503460f94a3de39/html5/thumbnails/16.jpg)
![Page 17: Standard setting Determining the pass mark - OSCEs](https://reader030.vdocuments.us/reader030/viewer/2022032521/56649d5e5503460f94a3de39/html5/thumbnails/17.jpg)
Borderline procedureBorderline procedure
Used for clinical (OSCE) examinations
Examiners score the students’s performance at the station e.g 17/20
Examiners judge the overall performance
clear pass / borderline / clear fail
Mark sheets rated borderline identified and the scores of borderline students averaged
Process repeated for each station
Calculate the median borderline score for all stations
![Page 18: Standard setting Determining the pass mark - OSCEs](https://reader030.vdocuments.us/reader030/viewer/2022032521/56649d5e5503460f94a3de39/html5/thumbnails/18.jpg)
Item I Clinical Problem
Type of station/question
Max marks available
Median Borderline score
Station1 Cranial nerves Examination 40 28.75
Station2b Patient education
Communication 20 13.38
Station3 Breathlessness History 20 12.00 Station4 chest Examination 20 11.00
Overall borderline score
510 297.89
Overall borderline score %
100
58.41
Borderline Method
![Page 19: Standard setting Determining the pass mark - OSCEs](https://reader030.vdocuments.us/reader030/viewer/2022032521/56649d5e5503460f94a3de39/html5/thumbnails/19.jpg)
1 2 3
Pass/Fail
Station 15 (static) Oral Lesions
0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 10.00
Score
0
10
20
30
40
50
Co
un
t
![Page 20: Standard setting Determining the pass mark - OSCEs](https://reader030.vdocuments.us/reader030/viewer/2022032521/56649d5e5503460f94a3de39/html5/thumbnails/20.jpg)
Borderlinegroup
pass
fail
Median Borderline score 58.41
![Page 21: Standard setting Determining the pass mark - OSCEs](https://reader030.vdocuments.us/reader030/viewer/2022032521/56649d5e5503460f94a3de39/html5/thumbnails/21.jpg)
Borderlinegroup
pass
fail
Median Borderline score 58.41 How wide should
this band be?
![Page 22: Standard setting Determining the pass mark - OSCEs](https://reader030.vdocuments.us/reader030/viewer/2022032521/56649d5e5503460f94a3de39/html5/thumbnails/22.jpg)
Borderlinegroup
pass
fail
Median Borderline score 58.41 How wide should
this band be?
+/- 1 - standard deviation- standard error- or what?
![Page 23: Standard setting Determining the pass mark - OSCEs](https://reader030.vdocuments.us/reader030/viewer/2022032521/56649d5e5503460f94a3de39/html5/thumbnails/23.jpg)
The Standard Error of MeasurementThe Standard Error of Measurement
• depends on the reliability of the test (R)
• depends on the standard deviation of the test (SD)
• SEM = SD 1 - R• acts as a confidence interval in high stakes situations
![Page 24: Standard setting Determining the pass mark - OSCEs](https://reader030.vdocuments.us/reader030/viewer/2022032521/56649d5e5503460f94a3de39/html5/thumbnails/24.jpg)
The Standard Error of MeasurementThe Standard Error of Measurement
• depends on the reliability of the test (R)
• depends on the standard deviation of the test (SD)
• SEM = SD 1 - R• acts as a confidence interval in high stakes situations
R = Variance Variance x Error
![Page 25: Standard setting Determining the pass mark - OSCEs](https://reader030.vdocuments.us/reader030/viewer/2022032521/56649d5e5503460f94a3de39/html5/thumbnails/25.jpg)
Borderlinegroup
Pass
Fail
Median Borderline score 58.41
Pass Score60.93
} - 1 SEM
Fail score 55.88
} + 1 SEM
![Page 26: Standard setting Determining the pass mark - OSCEs](https://reader030.vdocuments.us/reader030/viewer/2022032521/56649d5e5503460f94a3de39/html5/thumbnails/26.jpg)
Number of students
Mean Score (%)
Overall Borderline Score (%)
Standard deviation
Standard Error of Measurement
Pass mark (%)
Reliability
04 195 73.25
58.28 5.50 2.40 60.68
0.81
03 215 74.4 58.41 5.77 2.52 60.93
0.81
02 214 73. 0 59. 81 4.95 2.98 62.79
0.64
01 195 71.85
58.90 5.86 3.21 62.11
0.70
![Page 27: Standard setting Determining the pass mark - OSCEs](https://reader030.vdocuments.us/reader030/viewer/2022032521/56649d5e5503460f94a3de39/html5/thumbnails/27.jpg)
Internal Reliability of the ExamCronbach’s Alpha
Cronbach’s alpha
Shows whether randomly split halves of the exam by item vary together
Split halfCorrelations
(SH)
Item / Stations
Population
(students)
![Page 28: Standard setting Determining the pass mark - OSCEs](https://reader030.vdocuments.us/reader030/viewer/2022032521/56649d5e5503460f94a3de39/html5/thumbnails/28.jpg)
Borderline
Grading System
Borderline
Fail
Pass
Good Pass
Top Mark
Borderline Mark
Distinction
Pass Mark
Pass
Fail
![Page 29: Standard setting Determining the pass mark - OSCEs](https://reader030.vdocuments.us/reader030/viewer/2022032521/56649d5e5503460f94a3de39/html5/thumbnails/29.jpg)
Reducing measurement errorReducing measurement error
1. Increase reliability of exam
2. Compromise with feasibility/cost
* Content specificity
* More items in the exam
* Blueprinting
* Quality assurance of item writing
* Training examiners/standardised patients
* Feedback from exam performance
![Page 30: Standard setting Determining the pass mark - OSCEs](https://reader030.vdocuments.us/reader030/viewer/2022032521/56649d5e5503460f94a3de39/html5/thumbnails/30.jpg)