360 Degree Evaluation
Craig McClure, MDMay 15, 2003
Educational Outcomes Service Group
Description
• Use of rating forms to report frequency of observed behavior
• Multiple people in contact with resident act as evaluators
• Often survey type form• Ratings summarized by topic• Include goal-setting
Background
• Human resources in business• ACGME found no published reports
of use in GME
Use for “Soft” Areas
• More accurate for formative than summative feedback
• Interpersonal & communication• Professional behavior• Limited
– Patient care– Systems-based practice
Decision to Utilize
• Accepted and used by residents, faculty, staff?
• Develop or purchase?• Cost? • Who are the raters? • How will the tool be used?
Decision to Utilize (2)
• To whom is the information available?
• What core competencies will be evaluated with this tool?
• How nurture trust the process remains confidential?
• Platform of evaluation
Acceptance
• Will all potential evaluators fully participate?
• Will raters be fair & honest?• Will residents accept the feedback
from non-faculty?
Develop or Purchase
• Development permits tailoring• Development time may be
considerable• Purchasing gives a ready-made
product• Purchasing: computer based
Developing
• Expert in educational testing• Programming expertise• Pilot period
Purchase
• Items measured appropriate?• Does it perform as claimed?• Inter-rater reliability?• Degree of support and ability to
customize
Cost
• If purchasing, monetary cost• If developing, personnel support• Data management system• Personnel time to complete forms• Annual development plan
Cost (2)
• Addressing EEOC/grievance complaints
• Handling disputes over data• Divisive & counterproductive for
those resistant
Personnel Evaluation Time
• 5 to 10 nurse evaluators per resident to give reproducible results
• More for faculty• More for patients
Identify Raters
• Patients (how explain process)• Nursing staff• Clerical staff members• Physician faculty members• Non-physician faculty members• Residents
Identify Raters (2)
• Medical students• Allied Health Personnel• Self-assessment
Patients as Raters
• Literacy• Language• Culture (medical and otherwise)• Personality
Intended Utility
• Intervals: monthly, quarterly, yearly
• Summative versus formative• To support high stakes decisions?
Access to Information
• Resident• Advisor• Program Director
Confidentiality & Trust
• Raters require anonymity• Residents require confidentiality• Both need the process to be
positive & constructive• Prior history conditions
expectations• Education to process aids current
participation
Platform of Evaluation
• PDA• Paper• Computer
Challenges
• Securing appropriate instruments for variety of evaluators
• Managing data successfully
Advantages
• Electronic database for documentation
• Ease of access for raters• Rapid turnaround for feedback• “Gap” analysis (self perception
versus image of others)
Disadvantages
• Hardware/software costs• Lack of validation in GME• Potential information overload• Selection bias• Discoverability• Potential for invalid feedback
References
• Assessment of Communication and Interpersonal Skills Competencies, C.C. Hobgood, et.al. Academic Emergency Medicine 2002;9: 1257-69
• ACGME/ABMS Joint Initiative Toolbox of Assessment Methods, September 2000
References (2)
• 360-degree Feedback, K.G. Rodgers,et.al. Academic Emergency Medicine 2002;9:1300-1304
• Letter from ADFM listserv, Goldsmith to Kikano