![Page 1: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/1.jpg)
1
Helping Your CCC Make Better Decisions:The Key to Incorporating Multisource
Assessments
Frederick M. Schiavone MD VP and DIO HCA WFD
1
![Page 2: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/2.jpg)
2
Disclosure:
No Conflicts to Disclose
Adapted from work by: Eric Holmboe, William Iobst,Andem Ekpenyong, Sandi Moutsios and Chandra Ross
2
![Page 3: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/3.jpg)
3
Discuss the value of Multisource Feedback (MSF) inprogrammatic assessment
Review current evidence for MSF
Discuss pros and cons of available MSF instruments for GME
Discuss how to use self-assessment more effectively
Objectives:
3
![Page 4: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/4.jpg)
4
New ICS HarmonizedSubcompetency
4
![Page 5: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/5.jpg)
5
“Exceptional health care in which patients, family members, and other care team members are able to consistently co-create care that is meaningful, safe, reliable, resilient, efficient, and exquisitely responsive to the hopes, needs, goals, and purposes of everyone involved.”
Definition of Interprofessional Collaborative Care
5
![Page 6: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/6.jpg)
6
Interprofessional Collaborative Education and Care
• Interest in promoting more team-based education for U.S. health professions is not new.
• At the first IOM Conference, “Interrelationships of Educational Programs for Health Professionals,” and in the related report “Educating for the Health Team” (IOM, 1972), 120 leaders from allied health, dentistry, medicine, nursing, and pharmacy considered key questions at the forefront of contemporary national discussions about interprofessional education (IPE)
6
![Page 7: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/7.jpg)
7
Core Competencies for Interprofessional Collaborative Practice • Interprofessional education: “When students from two or more professions learn about, from
and with each other to enable effective collaboration and improve health outcomes” (WHO, 2010) • Interprofessional collaborative practice: “When multiple health workers from different
professional backgrounds work together with patients, families, carers [sic], and communities to deliver the highest quality of care” (WHO, 2010)
• Interprofessional teamwork: The levels of cooperation, coordination and collaboration characterizing the relationships between professions in delivering patient-centered care
• Interprofessional team-based care: Care delivered by intentionally created, usually relatively small work groups in health care, who are recognized by others as well as by themselves as having a collective identity and shared responsibility for a patient or group of patients, e.g., rapid response team, palliative care team, primary care team, operating room team
• Professional competencies in health care: Integrated enactment of knowledge, skills, and values/attitudes that define the domains of work of a particular health profession applied in specific care contexts
• Interprofessional competencies in health care: Integrated enactment of knowledge, skills, and values/attitudes that define working together across the professions, with other health care workers, and with patients, along with families and communities, as appropriate to improve health outcomes in specific care contexts
Reprinted with permission from: World Health Organization (WHO). (2010). Framework for Action on Interprofessional Education & Collaborative Practice. Geneva: World Health Organization.
7
![Page 8: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/8.jpg)
8
Existing MSF Instruments Teamwork Effectiveness Assessment Module
(TEAM)
Open source module available from ACGME
IPA MSF tool
NBME Assessment of Professional Behaviors (MedEdPortal)
Adapted from: Eric Holmboe, William Iobst, Andem Ekpenyong, and Sandi Moutsios and ACGME :Multi-source Feedback (MSF): An Approach to Assess the “Other Competencies”
8
![Page 9: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/9.jpg)
9
Challenges to Implementation of Interprofessional Competences
• Institutional Level Challenges – There is a lack of top administrative leadership support for adequate resources to create an interprofessional component to health professions students’ education. In institutions that implement systematic programs of interprofessional education top leadership support has been critical.
• The Lack of Institutional Collaborators - Some schools interested in launching interprofessional learning have no other or limited professional schools in their institution to partner with, and some potential partners are unwilling to take on an interprofessional agenda.
• Practical Issues - Scheduling and finding time to bring students together across the professions remains an issue.
• Faculty Development Issues - Health professions faculty need training to become effective interprofessional educators. The content and process of interprofessional learning differ from other academic content they teach.
• Lack of Regulatory Expectations - Recognition by accrediting bodies of interprofessional competencies as vital to health professions educational programs reinforces the imperative to address it by faculty and institutional leaders.
9
![Page 10: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/10.jpg)
10
Interprofessional Professionalism Interprofessional Professionalism Collaborative
MSF tool to measure IPA Initial article just published (see below).
Website: http://www.interprofessionalprofessionalism.org/
Website contains three videos for faculty development and teaching
PDF of survey instrument now available
26 items with narrative comments
Frost JS, Hammer DP, Nunez LM, et.al. The intersection of professionalism andinterprofessional care: development and initial testing of the interprofessionalprofessionalism assessment (IPA). J Interprof Care. 2018 Sep 24:1-14.
10
![Page 11: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/11.jpg)
11
Interprofessional Teamwork and *IOM Core Competencies
• Communication • Cooperation• Coordination and
Collaboration in teamwork (Informatics)• Rely on evidence based practice • Team-based Care
• This requires;• Mutual Respect of Team Members• Emotional Intelligence • Commitment to teaming • Commitment to continuous improvement of the Team *Institute of Medicine
11
![Page 12: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/12.jpg)
Challenges• Challenges to infusing interprofessional content
across multiple discipline curriculums included:
• Time and scheduling challenges
• Faculty workload for team teaching
• Full disciplinary curricula
• Financial and physical resource constraints.
12
12
![Page 13: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/13.jpg)
13
Also referred to as 360o evaluation Definition
Assessment completed by multiple individuals, usually fromdifferent perspectives based upon observations in differentcontexts
Includes raters, “processes and instruments for information-gathering, appraisal and feedback...”
Multi-source Feedback (MSF)13
![Page 14: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/14.jpg)
14
Competence Assessments
• Clinical competence has been defined as “the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and community being served.”
How do we measure attitudes?
Behavior begets Behavior
14
![Page 15: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/15.jpg)
15
• To date, limited attention has been focused on optimizing PD effectiveness to assess
• There is a paucity of literature regarding assessments related to the PD role with MSF • PDs are required to implement a system of multi-source evaluation of residents without
participating in such systems themselves • This lack of emphasis on evaluation represents a missed opportunity to help PDs maximize their
positive impact
Miller’s Pyramid of Assessment15
![Page 16: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/16.jpg)
16
Assessment Methods Available16
![Page 17: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/17.jpg)
17
Clinical Competency Committee
Clinical Competency Committee
End-of-Rotation Evaluations
Peer Evaluations
SelfEvaluations
Case Logs
Student Evaluations
Patient/ Family Evaluations
Operative Performance Rating Scales
Nursing and Ancillary
Personnel Evaluations
Assessment of Milestones
Clinic Workplace Evaluations
Mock Orals
OSCE
ITE
SimLab
UnsolicitedComments
MedHub
17
![Page 18: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/18.jpg)
18
Value of MSF
• Whether a trainee (resident or fellow) is “doing well” clinically may differ from doing well in other contexts, and doing well in the eyes of peers may differ from doing well in the eyes of patients, managers, or other colleagues.
• Hence, doing well cannot be captured in a standardized single measure, or evaluated from one perspective or by one person.
• MSF, or 360° evaluation, is the combined evaluation of a person by multiple individuals that have different working relationships with the person, through use of questionnaires and a compiled feedback report.
• For trainees, reviewers generally include peers, supervisors, other health professionals, and patients/families.
18
![Page 19: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/19.jpg)
19
MSF: Potential RatersPatients
Faculty
Nurses
Peers RESIDENT SELF
Patient Families
Medical Asst. Social workers
ParamedicsOther health care
providers...
Consultants
Trainees
Standardized Patients
19
![Page 20: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/20.jpg)
20
Assessing interpersonal skills and communication, professionalism, and systems-based practice (harmonized Milestones): ICS Patient and family centered communication Interprofessional and team communication Communication within healthcare systems
Professionalism Professional behavior and ethical Principles Accountability/Conscientiousness
Systems-based practice System navigation for patient centered care
MSF Essential for…20
![Page 21: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/21.jpg)
21
Systematic Review 2014: Reviewed 43 articles MSF effective method in providing feedback to physicians
across range of specialties Conclusions In general, assessment of physician performance was based on the
completion of the MSF instruments by 8 medical colleagues, 8 coworkers, and 25 patients to achieve adequate reliability and generalizability coefficients of α ≥ 0.90 and Ep2 ≥ 0.80, respectively.
MSF Evidence
Donnon T, et. al. The Reliability, Validity, and Feasibility of Multisource Feedback PhysicianAssessment: A Systematic Review. Acad Med. 2014.
21
![Page 22: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/22.jpg)
22
Harvard associative study of 264 surgeons and malpractice claims
MSF and Malpractice
Lagoo J, et. al. Multisource Evaluation of Surgeon Behavior is Associated with Malpractice Claims. Ann Surg, 2018; online first. DOI: 10.1097/SLA.0000000000002742
22
![Page 23: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/23.jpg)
23
MSF: Peers
Peer assessment is clearly a valuable feedback approach Key issues in peer assessment
Assessment of task versus global ratingPerformance of specific actions versus “quality” of those actionsDoes the peer have the requisite experience and competency tomake the judgments asked of the tool? Ability to make distinctions Results not be shared –for PD’s eyes only
23
![Page 24: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/24.jpg)
24
MSF: Nurses
Nurses and other health professionals excellent source ofjudgment and feedback on competencies of interpersonalskills and communication, professionalism and teamwork
Older study by Butterfield found just 3-5 nursing evaluations could identify “outlier” (i.e. problematic physicians) 90% of the time.
24
![Page 25: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/25.jpg)
25
MSF: Nurses
Nurses and other health professionals ratings often do not correlatewith faculty ratings and other measures Ogunyemi, et. al. (2009) Residents assessed on communication with patients,
interactions with peers, and professionalism Weak correlations occurred between nursing and faculty evaluations
(r = 0.065-0.119, P < .001). Weak negative correlations between nursing evaluations and standard
medical examination scores (r = -0.08 to -0.10, P <.001).
25
![Page 26: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/26.jpg)
26
MSF: Patients
Surveys should target patient experience, not just satisfaction Level of sufficient reliability depends on purposeProvider-level CAHPS: Consumer Assessment of Healthcare Providers and Systems45 per physician for higher stakes decisions – measures patient experience
Other studies: reasonable reliability with 25 per physicianPatient experience does correlate with quality performancePatient experience surveys best used as a formative assessment tool in residency and fellowship
26
![Page 27: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/27.jpg)
27
Patient Experience Differs from Patient Satisfaction
• Patient satisfaction and patient experience are often used interchangeably, but they are not the same thing
• Patient Experience• Need to find out from patients whether something that should have happened
in a health care setting (such as clear communication with a provider) actually happened
Patient Satisfaction• Satisfaction is whether a patient’s expectations about a health encounter were
met. • Two people who receive the exact same care, but who have different
expectations for how that care is supposed to be delivered, can give different satisfaction ratings because of their different expectations.
Press Ganey has been implementing the CGCAHPS visit-specific survey on behalf of clients since 2010.
27
![Page 28: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/28.jpg)
28
Clinimetric Approach
Three simple questions at end of visit:
What did you like the most?
What did you like least?
What one thing would you like to see change?
1Concato J, Feinstein AR. Asking patients what they like: overlooked attributes of patientsatisfaction with primary care. Am J Med. 1997 102:399-406
28
![Page 29: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/29.jpg)
29
Link Between Communication and Outcomes
Teach PC* Communication
Skills
Immediate Outcomes
Improved PC communication behavior during
the patient encounter
Intermediate Outcomes
• Increased patient knowledge
• Increasedpatient self-efficacy• Better IDM•Increased adherence
• Improved self-care
Health Outcomes
• Improved biologic outcomes
• Improved QoL and well-being
• Improved survival• Reduce costs
•Reduced disparities
Levinson W, Lesser CS, Epstein RM. Developing Physician Communication Skills forPatient-centered care. Health Affairs. 2010; 29: 1310-18.
*PC = patient-centered
29
![Page 30: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/30.jpg)
30
Systematic review of effect of patient feedback
Feedback source Credibility and valence
Feedback specificity Feedback delivery Facilitation quality and use of reflection
Narrative comments Often perceived as most helpful
Limitation of study – lack of articles from health services field
30
![Page 31: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/31.jpg)
31
Where and how do you use multi-source feedback in your program?What members of the healthcare “team” are asked to assess thetrainee?If you do not currently involve patients and families, how mightyour assessment program include their voice?
What are your Action Plans
31
![Page 32: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/32.jpg)
32
Discussion of Existing MSF Tools
32
![Page 33: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/33.jpg)
33
Teamwork Effectiveness Assessment Module (TEAM)
Open source module available from ACGME
IPP MSF tool
Peer-NBME Assessment of Professional Behaviors (MedEdPortal)
Existing Multi-Source Feedback Instruments33
![Page 34: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/34.jpg)
34
New ICS Harmonized Subcompetency34
![Page 35: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/35.jpg)
35
TEAM: Teamwork Effectiveness Assessment Module
Self-assessment
Physician identifies
team members
Assessment by team
members
Reflect on results and
develop improvement
plan
Formative 360° assessment (not summative) For inpatient physicians and their teams
Unique features: Guided process to identify interprofessional team In-depth qualitative + quantitative feedback Guided reflection with “trusted peer(s)”
4-step process
35
![Page 36: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/36.jpg)
36
New: Open Access Assessment Tools
https://dl.acgme.org/pages/assessment
36
![Page 37: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/37.jpg)
37
Interprofessional Professionalism
Interprofessional Professionalism Collaborative
MSF tool to measure IPP Initial article just published (see below).
Website: http://www.interprofessionalprofessionalism.org/
Website contains three videos for faculty development and teaching
PDF of survey instrument now available
26 items with narrative comments
Frost JS, Hammer DP, Nunez LM, et.al. The intersection of professionalism and interprofessional care: development andinitial testing of the interprofessional professionalism assessment (IPA). J Interprof Care. 2018 Sep 24:1-14.
37
![Page 38: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/38.jpg)
38
Interprofessional Professionalism Assessment
38
![Page 39: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/39.jpg)
39
NBME Assessment ofProfessional Behaviors
Uses MSF approach to assess “professional behaviors” Piloted in a number of schools and residency programs Originally designed to be used as a “program” with training
modules Program discontinued in July 2013 Now available to all interested programs through the AAMC
MedEdPortal
39
![Page 40: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/40.jpg)
40
Assessment of Professional Behaviors
Downloaded from: https://www.mededportal.org/publication/9902/
40
![Page 41: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/41.jpg)
41
Pitfalls in Self-Assessment
41
![Page 42: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/42.jpg)
42
Self-Assessment Skills
Systematic review (Davis, JAMA, 2006)Accuracy of self-assessment compared to external observation
17 studies included; 20 total comparisons13 demonstrated little, no or inverse relationship
Worst accuracy of self-assessment amongleast skilled physicians
42
![Page 43: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/43.jpg)
43
Confidence versus Competence
MisinformedDangerous
MasteryRight action
UninformedParalyzed
DoubtHesitation
CONFIDENCE
H
L
L HCOMPETENCE
Bruno, Confidence-based Learning
43
![Page 44: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/44.jpg)
44
Unskilled and Unaware of It (1999)
“Unskilled individuals suffer a dualburden: not only do these people reacherroneous conclusions and make unfortunate choices, but theirincompetence robs them of themetacognitive ability to realize it”
Kruger-Dunning Effect
44
![Page 45: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/45.jpg)
45
Informed Self Assessment
• Literature clear• cannot perform effective or accurate self- assessment alone or in
isolation
• Eva and Regher
• “Self-directed assessment seeking”
• Sargeant
• Informed self-assessment essential but complex process thatrequires support and coaching of the learner
• Individualized learning plans (ILPs) important component
45
![Page 46: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/46.jpg)
46
New PBLI Harmonized Subcompetency46
![Page 47: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/47.jpg)
47
Milestones Learning Cycle
Compare/Contrast SA with CCC Milestone Judgments
Perform Self-assessment (SA) Prior to CCC
Gain/Refine Understanding and Apply Milestones in Practice
Complete/Implement Individualized Learning Plan
47
![Page 48: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/48.jpg)
48
Next Steps How would you implement, change or modify how you use self-assessment as part your evaluation system in yourprogram?
How could you use Milestones in an informed self-assessment process with individualized learning plans?
48
![Page 49: Helping Your CCC Make Better Decisions: The Key to](https://reader031.vdocuments.us/reader031/viewer/2022012509/6186d10f03c7106267263d4e/html5/thumbnails/49.jpg)
4949
Thank You
Questions ?
49