qsapple2015 back to the future electronic marking of objective structured clinical examinations and...
TRANSCRIPT
![Page 1: Qsapple2015 back to the future electronic marking of objective structured clinical examinations and admission interviews for the higher education market](https://reader036.vdocuments.us/reader036/viewer/2022070520/58f1cbf21a28ab8a558b457f/html5/thumbnails/1.jpg)
“Back to the Future: Electronic Marking of Objective Structured Clinical Examinations and Admission Interviews for the higher education market”
Dr Thomas JB Kropmans
![Page 2: Qsapple2015 back to the future electronic marking of objective structured clinical examinations and admission interviews for the higher education market](https://reader036.vdocuments.us/reader036/viewer/2022070520/58f1cbf21a28ab8a558b457f/html5/thumbnails/2.jpg)
Objective Structured Clinical Examinations e.g. OSCE/MMI
Station 4
Station 1 Station 2 Station 3ExaminerPatient Student
Waiting Area – To be Examined
Student
Station 5Station 6Station 7Waiting Area - Examined
Secretary
Desk
StudentStudentStudent
Student
StudentStudentPatient Student Patient Student
Patient StudentPatient StudentPatient StudentPatient Student
Student Examiner Patient Secretary Flow of Students
Examiner Examiner
Examiner Examiner Examiner Examiner
Tablet/PDA
Web Server/Database
Web Server/Central Database
![Page 3: Qsapple2015 back to the future electronic marking of objective structured clinical examinations and admission interviews for the higher education market](https://reader036.vdocuments.us/reader036/viewer/2022070520/58f1cbf21a28ab8a558b457f/html5/thumbnails/3.jpg)
Admission interviews (Students)Selection interviews (HR)Objective Structured Clinical Examinations (Health Care, Law)Workplace Based Assessment
Where to be used
![Page 4: Qsapple2015 back to the future electronic marking of objective structured clinical examinations and admission interviews for the higher education market](https://reader036.vdocuments.us/reader036/viewer/2022070520/58f1cbf21a28ab8a558b457f/html5/thumbnails/4.jpg)
Only student ability to pass an examination/interview is being assessedDifficulty of the assessment is not taken into accountHeterogeneity of measurement instruments (stations) limits comparabilityComparison of student competence across institutions and assessment (OSCE/MMI) settings is requiredUniversal adoption of a standardized instrument is recommended (Clinical Teacher September 2014)
Without standard setting
![Page 5: Qsapple2015 back to the future electronic marking of objective structured clinical examinations and admission interviews for the higher education market](https://reader036.vdocuments.us/reader036/viewer/2022070520/58f1cbf21a28ab8a558b457f/html5/thumbnails/5.jpg)
Of 1000 forms 300 contains errorsLow Cronbach’s Alpha (< 0.8 poor)Pass mark of 50%?Introduction Online Marking Tool in December 2008
30% based upon error
![Page 6: Qsapple2015 back to the future electronic marking of objective structured clinical examinations and admission interviews for the higher education market](https://reader036.vdocuments.us/reader036/viewer/2022070520/58f1cbf21a28ab8a558b457f/html5/thumbnails/6.jpg)
Purpose of a (formative or summative) clinical skills exam?
Assess clinical skills or non-cognitive skillsDetermine between ‘good’ and ‘bad’ performanceTo provide adequate and timely feedbackReliable decision on ‘pass’ or ‘fail’
![Page 7: Qsapple2015 back to the future electronic marking of objective structured clinical examinations and admission interviews for the higher education market](https://reader036.vdocuments.us/reader036/viewer/2022070520/58f1cbf21a28ab8a558b457f/html5/thumbnails/7.jpg)
Real time solution
![Page 8: Qsapple2015 back to the future electronic marking of objective structured clinical examinations and admission interviews for the higher education market](https://reader036.vdocuments.us/reader036/viewer/2022070520/58f1cbf21a28ab8a558b457f/html5/thumbnails/8.jpg)
Internal Consistency Stations
![Page 9: Qsapple2015 back to the future electronic marking of objective structured clinical examinations and admission interviews for the higher education market](https://reader036.vdocuments.us/reader036/viewer/2022070520/58f1cbf21a28ab8a558b457f/html5/thumbnails/9.jpg)
Cronbach’s AlphaInternal consistency
![Page 10: Qsapple2015 back to the future electronic marking of objective structured clinical examinations and admission interviews for the higher education market](https://reader036.vdocuments.us/reader036/viewer/2022070520/58f1cbf21a28ab8a558b457f/html5/thumbnails/10.jpg)
Issues with examiners
![Page 11: Qsapple2015 back to the future electronic marking of objective structured clinical examinations and admission interviews for the higher education market](https://reader036.vdocuments.us/reader036/viewer/2022070520/58f1cbf21a28ab8a558b457f/html5/thumbnails/11.jpg)
Skew distribution• Median = 74%; min score = 4 (20%) and max 20 (100%)
Distribution of scores
74%4
100%
Nr o
f stu
dent
s
2020%
50%
![Page 12: Qsapple2015 back to the future electronic marking of objective structured clinical examinations and admission interviews for the higher education market](https://reader036.vdocuments.us/reader036/viewer/2022070520/58f1cbf21a28ab8a558b457f/html5/thumbnails/12.jpg)
Professional impression examiners
![Page 13: Qsapple2015 back to the future electronic marking of objective structured clinical examinations and admission interviews for the higher education market](https://reader036.vdocuments.us/reader036/viewer/2022070520/58f1cbf21a28ab8a558b457f/html5/thumbnails/13.jpg)
0 1 2 3 40.0
1.0
2.0
3.0
4.0
5.0
6.0
7.0
8.0
9.0
10.0
Method 1 Single borderline score
Method 1: Global score (Fail=0, Borderline=1, Clear Pass=2, Good Pass=3, Excellent=4)
Stat
ion
scor
es
Borderline Regression Analysis
![Page 14: Qsapple2015 back to the future electronic marking of objective structured clinical examinations and admission interviews for the higher education market](https://reader036.vdocuments.us/reader036/viewer/2022070520/58f1cbf21a28ab8a558b457f/html5/thumbnails/14.jpg)
Improved pass/fail decisions
![Page 15: Qsapple2015 back to the future electronic marking of objective structured clinical examinations and admission interviews for the higher education market](https://reader036.vdocuments.us/reader036/viewer/2022070520/58f1cbf21a28ab8a558b457f/html5/thumbnails/15.jpg)
19 prestigious universities>200 OSCE/MMIs successfully administeredStudent and Examiner results being analysedCost reduction of 70% Error reduction by 30%
19% MORE NUIG students fail after introducing BRAInternal consistency varies from 0.45 – 0.85Cut-off score varies between stations from 40 – 75%Generalisability Kappa between departments 0.4 – 0.9
Results