360 degree evaluation craig mcclure, md may 15, 2003 educational outcomes service group

360 Degree Evaluation

Craig McClure, MDMay 15, 2003

Educational Outcomes Service Group

Description

• Use of rating forms to report frequency of observed behavior

• Multiple people in contact with resident act as evaluators

• Often survey type form• Ratings summarized by topic• Include goal-setting

Background

• Human resources in business• ACGME found no published reports

of use in GME

Use for “Soft” Areas

• More accurate for formative than summative feedback

• Interpersonal & communication• Professional behavior• Limited

– Patient care– Systems-based practice

Decision to Utilize

• Accepted and used by residents, faculty, staff?

• Develop or purchase?• Cost? • Who are the raters? • How will the tool be used?

Decision to Utilize (2)

• To whom is the information available?

• What core competencies will be evaluated with this tool?

• How nurture trust the process remains confidential?

• Platform of evaluation

Acceptance

• Will all potential evaluators fully participate?

• Will raters be fair & honest?• Will residents accept the feedback

from non-faculty?

Develop or Purchase

• Development permits tailoring• Development time may be

considerable• Purchasing gives a ready-made

product• Purchasing: computer based

Developing

• Expert in educational testing• Programming expertise• Pilot period

Purchase

• Items measured appropriate?• Does it perform as claimed?• Inter-rater reliability?• Degree of support and ability to

customize

Cost

• If purchasing, monetary cost• If developing, personnel support• Data management system• Personnel time to complete forms• Annual development plan

Cost (2)

• Addressing EEOC/grievance complaints

• Handling disputes over data• Divisive & counterproductive for

those resistant

Personnel Evaluation Time

• 5 to 10 nurse evaluators per resident to give reproducible results

• More for faculty• More for patients

Identify Raters

• Patients (how explain process)• Nursing staff• Clerical staff members• Physician faculty members• Non-physician faculty members• Residents

Identify Raters (2)

• Medical students• Allied Health Personnel• Self-assessment

Patients as Raters

• Literacy• Language• Culture (medical and otherwise)• Personality

Intended Utility

• Intervals: monthly, quarterly, yearly

• Summative versus formative• To support high stakes decisions?

Access to Information

• Resident• Advisor• Program Director

Confidentiality & Trust

• Raters require anonymity• Residents require confidentiality• Both need the process to be

positive & constructive• Prior history conditions

expectations• Education to process aids current

participation

Platform of Evaluation

• PDA• Paper• Computer

Challenges

• Securing appropriate instruments for variety of evaluators

• Managing data successfully

Advantages

• Electronic database for documentation

• Ease of access for raters• Rapid turnaround for feedback• “Gap” analysis (self perception

versus image of others)

Disadvantages

• Hardware/software costs• Lack of validation in GME• Potential information overload• Selection bias• Discoverability• Potential for invalid feedback

References

• Assessment of Communication and Interpersonal Skills Competencies, C.C. Hobgood, et.al. Academic Emergency Medicine 2002;9: 1257-69

• ACGME/ABMS Joint Initiative Toolbox of Assessment Methods, September 2000

References (2)

• 360-degree Feedback, K.G. Rodgers,et.al. Academic Emergency Medicine 2002;9:1300-1304

• Letter from ADFM listserv, Goldsmith to Kikano