nailed it! doctor of pharmacy students’ self-awareness of ......nailed it! doctor of pharmacy...
TRANSCRIPT
Nailed it! Doctor of Pharmacy Students’ Self-Awareness of Performance on Objective
Structured Clinical Examinations (OSCE)
Kara Sermersheim, PharmD, BS
Kimberly K. Daugherty, PharmD, BCPS
Sarah E. Raake, PharmD, MSEd, BCACP, LDE
Sullivan University College of Pharmacy & Health Sciences
Louisville, KY
Disclosures: The speaker has no actual or potential conflicts of interest in relation to this presentation
Objectives
1. Discuss Doctor of Pharmacy (PharmD) students’ self-awareness related to their performance on Objective Structured Clinical Examinations (OSCE).
2. Identify the Doctor of Pharmacy students who are not self-aware based on study data.
3. Discuss how this data could be used to develop student self-awareness skills
Background: Sullivan Itself
− Sullivan University College of Pharmacy and Health Sciences− 3-year accelerated program
− Louisville, KY
− First graduating class in 2011
− Objective Structured Clinical Examinations (OSCEs) − Experiences that link didactic learning to simulated real-world cases
Why?Self-assessment of OSCE performance may help students develop self-awareness
in the practice setting
Self-awareness
Self-awareness
− Conscious knowledge of one’s own character, feelings, motives, and desires
− CAPE Outcome 4.1: students should display self-awareness defined as the ability to examine and reflect on personal knowledge, skills, abilities, beliefs, biases, motivation, and emotions that could enhance or limit personal and professional growth
− There is currently limited data relating OSCEs to student self-awareness
Oxford Online Dictionary. Definition of self-awareness. https://www.oxfordlearnersdictionaries.com/us/definition/english/self-awareness. (accessed 25 Mar 2020).Accreditation Council for Pharmacy Education. Accreditation standards and key elements for the professional program in pharmacy leading to the doctor of pharmacy degree. https://www.acpe-accredit.org/pharmd-program-accreditation/ (accessed 2019 Sept 16).
Objective Structured Clinical Examinations (OSCEs)
OSCEs as natural place to assess self-awareness
− OSCEs in pharmacy education are experiences that link didactic learning to simulated real-world cases
− Self-assessment of OSCE performance may help students develop self-awareness in the practice setting
Past examples of students lacking self-awareness in OSCEs
− Underconfident learner
− Overconfident leaner
Sample case (Directions to student)
Self-awareness survey
Grading checklist (SP completes)Pass Fail
Checklist item 1 X
Checklist item 2 X
Checklist item 3 X
Checklist item 4 X
Checklist item 5 X
Skills Section Score 80%
Checklist item 1 X
Checklist item 2 X
Checklist item 3 X
Checklist item 4 X
Communication Section 25%
Overall Score (80% skills + 20% communication) 69%
Return to the Pharmacist (RTP) No
Overall Student Performance
Sample SP checklist and comments
Why?We do these cases to assess patient care skills as well as soft skills such as,
communication, professionalism, empathy, and potentially self-awareness.
Pharmacist patient care process
Joint Commission of Pharmacy Practitioners. Pharmacists’ patient care process. https://www.pharmacist.com/sites/default/files/files/PatientCareProcess.pdf. (accessed 2020 Mar 23).
Case “buckets”
Collect Assess Plan Implement Follow-up
Communication/Law X X X
Patient Counseling X X
Patient Medication Recommendation
X X X X
Provider Recommendation
X X X X
Skills X X
Grading Checklist (SP completes)Pass Fail
Collect Checklist item 1 X
Assess Checklist item 2 X
Assess Checklist item 3 X
Plan Checklist item 4 X
Implement Checklist item 5 X
Skills Section Score 80%
Checklist item 1 X
Checklist item 2 X
Checklist item 3 X
Checklist item 4 X
Communication Section 25%
Return to the Pharmacist (RTP) No
Overall Score (80% skills + 20% communication) 69%
Overall Student Performance
The ProjectStudy Design & Statistics
Methods
– Retrospective review
– Single institution
– 3-year program
– Single cohort– 4 OSCEs completed
– 23 discrete cases
Joint Commission of Pharmacy Practitioners. Pharmacists’ patient care process. https://www.pharmacist.com/sites/default/files/files/PatientCareProcess.pdf. (accessed 2020 Feb 22).
• Reliability• Validity
Self-awareness survey
• Reliability• Validity
OSCE cases
Obtaining validity and reliability
– Microsoft Forms– Survey
– 23 individual OSCE cases
– Utilized 15 faculty members
Validity and reliability
– Lawshe method– Average CVR for all 23 cases was 0.74 (range 0.36-0.92)
– Sixteen cases were content valid (CVR ≥ 0.75)
– Seven cases had a CVR between 0-0.74
– Cronbach’s Alpha– Average Cronbach alpha for all 23 cases was 0.646 (range 0.193-0.887)
– Four cases had a Cronbach alpha of <0.5
– Nine cases were between 0.5-0.699
– Ten cases had a Cronbach alpha >0.7
Validity and reliability
– Spearman-Brown – The overall self-awareness survey (nail it/fail it) was found to be reliable
(0.829)
– Only the return to the pharmacist (0.743) and overall student performance (0.611) subscores were found to be reliable
Eisinga R, te Grotenhuis M, Pelzer B. The reliability of a two-item scale: pearson, Cronbach or spearman-brown. Int J Public Health. 2012;58(4), 637–642.
Data reviewed using Chi-squared
OSCE Grade
Failed (≤69.4%)
Passed (≥69.5%)
Survey Nail it (1)
Fail it (0)
Eisinga R, te Grotenhuis M, Pelzer B. The reliability of a two-item scale: pearson, Cronbach or spearman-brown. Int J Public Health. 2012;58(4), 637–642.
Identifying the disconnect
Overall Performance
OSCE Score
Failed
(≤69.4%)
Passed
(≥69.5%)
Survey
Nail it (1)
Fail it (0)
Return to the Pharmacist
SP Answer
No (0) Yes (1)
Survey
Yes (1)
No (0)
The ProjectResults & Limitations
Overall-Self Awareness
Case breakdown and odds Ratio
Limitations
– Single cohort of students
– Single institution
– Low stakes
ConclusionsAction Plan(s)
Action Plan
– Data-driven report card
– Moving to modular dashboard feedback mechanism
– Face-to-face meeting
Watch list
Last Name Overall Score Skills Communication RTP Collect Assess Plan Implement Follow-up/Monitor Notes Advisor
Learner 1 Smith
Learner 2 XX Jones
Learner 3 Dennis
Learner 4 Lawrence
Learner 5 Jennings
Future directions
– Student remediation– Develop activities
– Faculty advisor meetings
– Identify areas of self-improvement
– Learner and faculty development opportunities
– Review and improvement of OSCE cases not deemed valid
Final take-away
– Nail it/Fail it surveys are a useful tool to identify students who are not self-aware
Acknowledgements
– Katie F. Leslie, PhD, MS Director of The Center for Health and Wellness and Clinical and Administrative Sciences Associate Professor
Nailed it! Doctor of Pharmacy Students’ Self-Awareness of Performance on Objective
Structured Clinical Examinations (OSCE)
Kara Sermersheim, PharmD, BS
Kimberly K. Daugherty, PharmD, BCPS
Sarah E. Raake, PharmD, MSEd, BCACP, LDE
Sullivan University College of Pharmacy & Health Sciences
Louisville, KY
Disclosures: The speaker has no actual or potential conflicts of interest in relation to this presentation