Teaching and Assessing in Competency Based EducationFaculty Development Workshop

Susan B. Promes, M.D.Residency Program Director, Dept. of Emergency

MedicineOffice of Graduate Medical Education

Patricia S. O’Sullivan, Ed.D. Office of Research and Development in Medical

Education

Objectives Describe competency based education

Identify promising teaching and assessment practices and strategies for successful implementation

Describe the relationship between teaching and assessing competencies and overall program evaluation.

Schedule1:00 – 1:15 Welcome, Overview and Introductions

1:15 – 1:45 Competency Update

1:45 - 2:45 Teaching Strategies

2:45 - 3:00 Break

3:00 – 4:00 Assessment Tools

4:00 – 4:45 Implementation

4:45 – 5:00 Wrap Up and Evaluation

Competency Based Education

1. Explicit and clearly aligned with expected competencies;

2. Criteria-driven, focusing on accountability in reaching benchmarks and, ultimately, competence;

3. Grounded in “real-life” experiences;

4. Focused on fostering the learners’ ability to self-assess;

5. Individualized, providing more opportunities for independent study and achievement of milestones.

Name the Competencies used at UCSF

• Medical Knowledge• Patient Care• Practice Based Learning and

Improvement• Systems Based Practice• Professionalism• Interpersonal and Communication

Skills

Milestones

• Explicate the general competencies by describing a developmental progression of observable behaviors

• Provide specific feedback and evaluation to learners

• Ensure acquisition of necessary knowledge skills and attitudes for advancement

Milestone

EPA defined

• A core unit of work reflecting a responsibility that should only be entrusted upon someone with adequate competencies

Ole ten CateMedical Teacher 1010;32:669-675

CONTEXT is key!

Entrustable Professional Activities

(EPAs)

• Define important clinical activities• Link competencies / milestones• Include professional judgment of

competence by clinicians

2Viewpoint: Competency-Based Postgraduate Training: Can We Bridge the Gap between Theory and Clinical Practice?ten Cate, Olle; Scheele, Fedde Academic Medicine. 82(6):542-547, June 2007.

Dreyfus Model

Your program should document and demonstrate

• Learning opportunities in each competency domain

• Evidence of multiple assessment methods

• Use of aggregate data to improve the educational program

Curriculum Expectations

• Overall educational goals for a program distributed annually

• Competency-based goals and objectives for each educational experience and level of learner, distributed annually and reviewed with learners prior to educational experiences

• Who oversees this?

Learner Evaluation Expectations

• Formative– An assessment should occur for each learning experience

– Assessments of competency/milestones reflective of the 6 areas

– Multiple formats & evaluators (faculty, peers, patients, self, others)

– Progressive improvement by learner across the continuum

– Documented

• At least semi-annual evaluation for residents

• At end of course/clerkship/rotation

• Summative

– Competency Committee review

– Screening Committee review

Clinical Competency Committees

• Must have a written description of committee responsibilities– Review all resident evaluations by all

evaluators semi-annually– Prepare and assure the reporting of

Milestones evaluations of each resident semiannually to ACGME

– Make recommendations to program director for resident progress (promotion, remediation, dismissal)

Program Improvement• Formal systematic evaluation annually

• Must monitor and track:– Learner performance– Faculty development– Graduate performance– Program evaluations by learners and

faculty• Must use results to improve the program

– Written plan of action– Reviewed and approved by teaching faculty

UCSF Resources

• Glossary of Competencies and Assessments

http://medschool2.ucsf.edu/gme/curriculum/competencies.html

• Evaluation and Assessment for GME Programs

http://medschool2.ucsf.edu/gme/curriculum/Evaltools.html

• UCSF UME Competencieshttp://meded.ucsf.edu/ume/md-competencies

Teaching Strategies

1:45– 2:15 Group Activity – Madness to Methods2:15 – 2:45 Share Best Method to Objective Match__________________________________________________Each group nominate one or two “best method to objective match”

Playing the Exercise• Each player receives methods cards equal to the number of players in the

game, + one extra (5 players = 6 cards per player). • Dealer turns over the first objective card leaving it face up in the table. Each

player chooses one methods card from his/her hand that will achieve the teaching objective. A player holding a “wild card” can create a new or use a previously discussed method.

• Beginning clockwise from the dealer, all players place their chosen methods card face up near the objective card in the center of the table.

• Each player has one minute to persuade the other players that their teaching method will effectively, efficiently and appropriately achieve the objective.

• Once all players have presented their “method”, all players vote for the best method for achieving the teaching objective, but cannot vote for themselves.

• Players receive a point for each vote they receive.• The dealer tallies the votes by player and records on a score sheet. • The winner is the player with the most votes.

Simpson D, Fenzel J, Rehm J, Marcdante K. Enriching Educators' Repertoire of Appropriate Instructional Methods . MedEdPORTAL; 2010. Available from: www.mededportal.org/publication/7968

Why assess?

Whatever we measure, we tend to improve

David Leach, former CEO of ACGME

What makes a good assessment tool?

• Appropriate for what you are measuring

• User must know how to use it• Monitor tool use to make sure it is

working

What makes a good evaluation of a learner?

• Sufficient data• Sufficient contact with learner• Sufficient sources• Ability to render a judgment

MillerMiller’’s Assessment s Assessment TriangleTriangle

Does

Shows How

Knows How

Knows

Record ReviewPatient Survey360° Ratings

Undercover SPsObservationVideo logs

ChecklistsOSCE’s

SP’s

MCQ’s Checklists

Procedure/Case Logs

Clinical context-based testsOral exams

Essays

Factual tests Oral exams

MCQ’s Essays

Assessment Tools

3:00– 3:30Group Activity – Looking Glass3:30 – 4:00 Share Best Practices__________________________________________________For the competency your group selected:

Identify/share the methods your program is currently using to assess this competency (in addition to any global assessments you conduct after a rotation)What specific tools do you use, how frequently, and how you prepare the learners and evaluators for the assessment? Discuss the challenges you have faced and surmounted in implementing these assessments. Nominate one or two “best practices” from the group.

How to Implement

4:00 – 4:45 Getting the work done examples and group discussion

4:45 – 5:00 Wrap-up and Evaluation

________________________________________________________________________________________________________

• How does your program make sure that all stakeholders have and act on the results of competency assessments?

– Describe how your program uses learner competency results to make individual learner and program improvements.

– What are the efficient and effective steps to:•getting the data, getting it to the learner, •getting it to the program director, •getting it to the faculty for program review?

Neurology

Program Evaluation

Performance Assessment Teach for UCSF

Certificate• Follow the link provided in the

email

• All Participants:– Complete Action Plan– Complete Evaluation