resident self -assessment in the milestones era: a win-win … · 2015. 3. 28. · a win-win...
TRANSCRIPT
![Page 1: Resident Self -Assessment in the Milestones Era: A Win-Win … · 2015. 3. 28. · A Win-Win Approach to Resident Professional Development and Program Improvement APPD Meeting March](https://reader035.vdocuments.us/reader035/viewer/2022062507/5fc2be7aae2c6d097f2abff1/html5/thumbnails/1.jpg)
Resident Self-Assessment in the Milestones Era: A Win-Win Approach to Resident Professional
Development and Program Improvement
APPD Meeting March 28, 2015
![Page 2: Resident Self -Assessment in the Milestones Era: A Win-Win … · 2015. 3. 28. · A Win-Win Approach to Resident Professional Development and Program Improvement APPD Meeting March](https://reader035.vdocuments.us/reader035/viewer/2022062507/5fc2be7aae2c6d097f2abff1/html5/thumbnails/2.jpg)
Presenters Kim Gifford Children’s Hospital at Dartmouth Franklin Trimm University of Southern Alabama Bridget Oliveri Children’s Hospital at Dartmouth Susan Guralnick Winthrop University Hospital Ann Burke Wright State University No conflicts of interest to report
![Page 3: Resident Self -Assessment in the Milestones Era: A Win-Win … · 2015. 3. 28. · A Win-Win Approach to Resident Professional Development and Program Improvement APPD Meeting March](https://reader035.vdocuments.us/reader035/viewer/2022062507/5fc2be7aae2c6d097f2abff1/html5/thumbnails/3.jpg)
Objectives Participants will be able to:
• Prepare residents to perform milestones-based self-assessment
• Integrate resident self-assessment into the process of clinical competence assessment
• Use resident self-assessment and clinical competency committee assessments to promote learner development
• Use resident self-assessment and clinical competency committee assessment for program improvement
• Create a plan to utilize milestones-based assessments for learner development and/or program improvement for use at their home institution
![Page 4: Resident Self -Assessment in the Milestones Era: A Win-Win … · 2015. 3. 28. · A Win-Win Approach to Resident Professional Development and Program Improvement APPD Meeting March](https://reader035.vdocuments.us/reader035/viewer/2022062507/5fc2be7aae2c6d097f2abff1/html5/thumbnails/4.jpg)
Self-assessment Literature Review
Self-assessment does not correlate well with external absolute measures of performance (Davis et al 2006)
![Page 5: Resident Self -Assessment in the Milestones Era: A Win-Win … · 2015. 3. 28. · A Win-Win Approach to Resident Professional Development and Program Improvement APPD Meeting March](https://reader035.vdocuments.us/reader035/viewer/2022062507/5fc2be7aae2c6d097f2abff1/html5/thumbnails/5.jpg)
Slowing Down When You Should
Self-assessment of strengths and weaknesses may be more accurate (Lynn et al 2006)
Learners often “know when to look it up” (Eva & Regehr 2007)
![Page 6: Resident Self -Assessment in the Milestones Era: A Win-Win … · 2015. 3. 28. · A Win-Win Approach to Resident Professional Development and Program Improvement APPD Meeting March](https://reader035.vdocuments.us/reader035/viewer/2022062507/5fc2be7aae2c6d097f2abff1/html5/thumbnails/6.jpg)
Self-Assessment Can be Calibrated
Martin et al 1998 Ward et al 2003 Colthart et al 2008
![Page 7: Resident Self -Assessment in the Milestones Era: A Win-Win … · 2015. 3. 28. · A Win-Win Approach to Resident Professional Development and Program Improvement APPD Meeting March](https://reader035.vdocuments.us/reader035/viewer/2022062507/5fc2be7aae2c6d097f2abff1/html5/thumbnails/7.jpg)
Milestones as a Roadmap (Schumacher et al 2013)
![Page 8: Resident Self -Assessment in the Milestones Era: A Win-Win … · 2015. 3. 28. · A Win-Win Approach to Resident Professional Development and Program Improvement APPD Meeting March](https://reader035.vdocuments.us/reader035/viewer/2022062507/5fc2be7aae2c6d097f2abff1/html5/thumbnails/8.jpg)
Self-Assessment Exercise Assess your own level on the Milestones self-assessment sheet provided.
Discuss the self-assessment experience with your table
• Which competencies are easier/harder to self-assess?
• What external data could you seek to improve the accuracy of your own self-assessment?
• What guidance should (or do) you give your residents to complete a milestones-based self-assessment?
![Page 9: Resident Self -Assessment in the Milestones Era: A Win-Win … · 2015. 3. 28. · A Win-Win Approach to Resident Professional Development and Program Improvement APPD Meeting March](https://reader035.vdocuments.us/reader035/viewer/2022062507/5fc2be7aae2c6d097f2abff1/html5/thumbnails/9.jpg)
Collection and Synthesis of Data to Determine Clinical Competence
Synthesize into Milestone Levels
-Portfolios -OSCEs
-Direct Observation -Case Logs
-Global -3600
![Page 10: Resident Self -Assessment in the Milestones Era: A Win-Win … · 2015. 3. 28. · A Win-Win Approach to Resident Professional Development and Program Improvement APPD Meeting March](https://reader035.vdocuments.us/reader035/viewer/2022062507/5fc2be7aae2c6d097f2abff1/html5/thumbnails/10.jpg)
Triangulation improves clarity
Self-assessment recommended for inclusion in CCC review AHME Teleconference “Clinical Competency Committee: The ABC’s of CCC’s”- 10/7/14
![Page 11: Resident Self -Assessment in the Milestones Era: A Win-Win … · 2015. 3. 28. · A Win-Win Approach to Resident Professional Development and Program Improvement APPD Meeting March](https://reader035.vdocuments.us/reader035/viewer/2022062507/5fc2be7aae2c6d097f2abff1/html5/thumbnails/11.jpg)
Validity of Resident Self-Assessment Using the Pediatric Milestones:
How Well Does Pediatric Resident Self-Assessment Match External Assessment?
Su-Ting Li, MD, MPH; Kimberly Gifford, MD; Daniel Tancredi, PhD; Alan Schwartz, PhD; Ann Burke, MD; Ann Guillot, MD; Susan Guralnick, MD; John Mahan, MD;
R. Franklin Trimm, MD For APPD LEARN validity of resident self-assessment using
pediatric milestones group
![Page 12: Resident Self -Assessment in the Milestones Era: A Win-Win … · 2015. 3. 28. · A Win-Win Approach to Resident Professional Development and Program Improvement APPD Meeting March](https://reader035.vdocuments.us/reader035/viewer/2022062507/5fc2be7aae2c6d097f2abff1/html5/thumbnails/12.jpg)
Method Prospective multi-institutional study Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network (APPD-LEARN)
IRB approval through UC Davis, AAP, and each individual study site.
Comparison of: • Residents self-assessed milestones • CCCs determined resident milestones
![Page 13: Resident Self -Assessment in the Milestones Era: A Win-Win … · 2015. 3. 28. · A Win-Win Approach to Resident Professional Development and Program Improvement APPD Meeting March](https://reader035.vdocuments.us/reader035/viewer/2022062507/5fc2be7aae2c6d097f2abff1/html5/thumbnails/13.jpg)
Analysis
Univariate and bivariate analyses for mean milestones levels by year and competency Pearson correlation coefficients (PCC) calculated between self-assessment and CCC assessment scores for each learner
![Page 14: Resident Self -Assessment in the Milestones Era: A Win-Win … · 2015. 3. 28. · A Win-Win Approach to Resident Professional Development and Program Improvement APPD Meeting March](https://reader035.vdocuments.us/reader035/viewer/2022062507/5fc2be7aae2c6d097f2abff1/html5/thumbnails/14.jpg)
Results
47 pediatric residency programs (25% of all pediatric programs)
1741 residents
![Page 15: Resident Self -Assessment in the Milestones Era: A Win-Win … · 2015. 3. 28. · A Win-Win Approach to Resident Professional Development and Program Improvement APPD Meeting March](https://reader035.vdocuments.us/reader035/viewer/2022062507/5fc2be7aae2c6d097f2abff1/html5/thumbnails/15.jpg)
Resident and CCC Assessment of Competencies by Level of Training
![Page 16: Resident Self -Assessment in the Milestones Era: A Win-Win … · 2015. 3. 28. · A Win-Win Approach to Resident Professional Development and Program Improvement APPD Meeting March](https://reader035.vdocuments.us/reader035/viewer/2022062507/5fc2be7aae2c6d097f2abff1/html5/thumbnails/16.jpg)
![Page 17: Resident Self -Assessment in the Milestones Era: A Win-Win … · 2015. 3. 28. · A Win-Win Approach to Resident Professional Development and Program Improvement APPD Meeting March](https://reader035.vdocuments.us/reader035/viewer/2022062507/5fc2be7aae2c6d097f2abff1/html5/thumbnails/17.jpg)
Results • Highest PCC for patient care • Lowest PCC for professionalism and PBLI 4
PCC improved from 1st to 2nd observation period Higher # missing CCC-assessments for: • Practice-based learning and improvement • Professionalism • Systems-based practice
![Page 18: Resident Self -Assessment in the Milestones Era: A Win-Win … · 2015. 3. 28. · A Win-Win Approach to Resident Professional Development and Program Improvement APPD Meeting March](https://reader035.vdocuments.us/reader035/viewer/2022062507/5fc2be7aae2c6d097f2abff1/html5/thumbnails/18.jpg)
Individual Activity
Using the index cards on your table
On side 1 - How could you use learner self- assessment and CCC assessment to promote learner development ? On side 2– How you might use this data for program improvement?
![Page 19: Resident Self -Assessment in the Milestones Era: A Win-Win … · 2015. 3. 28. · A Win-Win Approach to Resident Professional Development and Program Improvement APPD Meeting March](https://reader035.vdocuments.us/reader035/viewer/2022062507/5fc2be7aae2c6d097f2abff1/html5/thumbnails/19.jpg)
Learner Development Discussion
Identify ways in which resident self-assessment, when compared with clinical competency committee assessments, can be used to promote learner development Each table has specialty-specific milestones profiles to help stimulate discussion.
![Page 20: Resident Self -Assessment in the Milestones Era: A Win-Win … · 2015. 3. 28. · A Win-Win Approach to Resident Professional Development and Program Improvement APPD Meeting March](https://reader035.vdocuments.us/reader035/viewer/2022062507/5fc2be7aae2c6d097f2abff1/html5/thumbnails/20.jpg)
Sharing of ideas
Using milestones-based self-assessment for learner development
![Page 21: Resident Self -Assessment in the Milestones Era: A Win-Win … · 2015. 3. 28. · A Win-Win Approach to Resident Professional Development and Program Improvement APPD Meeting March](https://reader035.vdocuments.us/reader035/viewer/2022062507/5fc2be7aae2c6d097f2abff1/html5/thumbnails/21.jpg)
From the resident perspective
![Page 22: Resident Self -Assessment in the Milestones Era: A Win-Win … · 2015. 3. 28. · A Win-Win Approach to Resident Professional Development and Program Improvement APPD Meeting March](https://reader035.vdocuments.us/reader035/viewer/2022062507/5fc2be7aae2c6d097f2abff1/html5/thumbnails/22.jpg)
Suggestions for Using Self-assessment for Learner Development
Inputs: Specific goals
Milestones Questions (ITE/quiz) Recalled experience
Recorded performance
Outputs: Career development
Learning needs Study plan
Goal setting Curricular choices
External comparison and/or coach facilitation Context: Didactics/retreats, coaching (PD/advisor/supervisor), remediation
![Page 23: Resident Self -Assessment in the Milestones Era: A Win-Win … · 2015. 3. 28. · A Win-Win Approach to Resident Professional Development and Program Improvement APPD Meeting March](https://reader035.vdocuments.us/reader035/viewer/2022062507/5fc2be7aae2c6d097f2abff1/html5/thumbnails/23.jpg)
Program Improvement Discussion
Identify ways in which resident self-assessment and clinical competency committee assessments can be used to target areas for Program Improvement. Each table has specialty-specific milestones summaries to help stimulate discussion.
![Page 24: Resident Self -Assessment in the Milestones Era: A Win-Win … · 2015. 3. 28. · A Win-Win Approach to Resident Professional Development and Program Improvement APPD Meeting March](https://reader035.vdocuments.us/reader035/viewer/2022062507/5fc2be7aae2c6d097f2abff1/html5/thumbnails/24.jpg)
Sharing of ideas
Using milestones-based self-assessment for program improvement
Other thoughts?
![Page 25: Resident Self -Assessment in the Milestones Era: A Win-Win … · 2015. 3. 28. · A Win-Win Approach to Resident Professional Development and Program Improvement APPD Meeting March](https://reader035.vdocuments.us/reader035/viewer/2022062507/5fc2be7aae2c6d097f2abff1/html5/thumbnails/25.jpg)
Suggestions for Using Self-assessment for Program Improvement
Program Components: Curricular quality
Gaps/redundancies Individual resident needs
Improvements: Assessments
Curriculum structure Rotation content
Faculty development
Program Evaluation Committee, retreats, housestaff meetings, rotation-specific
![Page 26: Resident Self -Assessment in the Milestones Era: A Win-Win … · 2015. 3. 28. · A Win-Win Approach to Resident Professional Development and Program Improvement APPD Meeting March](https://reader035.vdocuments.us/reader035/viewer/2022062507/5fc2be7aae2c6d097f2abff1/html5/thumbnails/26.jpg)
Overall Value of Self-assessment
Residents understand milestones
Teach/assess Practice-Based Learning and Improvement and Professionalism • Clarifies learner perspective and self-awareness
– Creates insight and motivation for learner – Allows resident to drive improvement (better goals)
• Guides coaching – Maximizes experiences by focusing efforts and
allowing for more ah-ha moments – Helps remediation planning
![Page 27: Resident Self -Assessment in the Milestones Era: A Win-Win … · 2015. 3. 28. · A Win-Win Approach to Resident Professional Development and Program Improvement APPD Meeting March](https://reader035.vdocuments.us/reader035/viewer/2022062507/5fc2be7aae2c6d097f2abff1/html5/thumbnails/27.jpg)
Commitment to Action
Given the ideas you heard today, how will you plan to utilize milestones-based self-assessment for learner development and/or program improvement? Write a note to yourself to commit to taking a first step toward your goal! Please leave your notecards on the table we will synthesize the suggestions and email to the full group