Competency-Based Education
Milestone or Millstone?
Geoff Norman, PhDMcMaster University
Competency-Based Education
• Basic Tenets
– PG education should be directed at defined competencies, not defined time blocks
– PG trainees should be evaluated on achievement of these competencies (milestones) to varying levels, culminating in independent practice
The Basic Assumptions
• Achievement of a competency can be visualized as a smooth curve of increasing competency with time and practice.
• At some point the competency is sufficient to permit independent professional practice
– The Entrustable Professional Activity (EPA)
• Different individuals will achieve different competencies at different rates.
Entrustable Professional Activity (EPA)(Wijnen-Meier, 2012)
EPA• a critical unit of professional work to be
entrusted to a trainee once competence at a sufficient level has been reached.
• independently executable and should be observable and measurable in process and outcome.
“Adoption of OBE would better equip medical graduates to respond effectively and efficiently continue to expand the depth and breadth of the requisite competencies”
– Carnegie Foundation Report, 2010
“OBE is an intuitive approach that engages the range of stakeholders… it encourages a student-centered approach and at the same time supports the trend for greater accountability and quality assurance.” Harden, 2007
• Dissemination
– The Scottish Doctor (Scotland)
– ACGME Competencies (US)
– Tomorrow’s Doctor (UK)
–CanMEDS roles (Canada, Netherlands, 24
others)
The Scottish Doctor
• CAN DO– Clinical Skills– Practical procedures– Patient Investigation– Patient Management– Health Promotion– Communication– Medical Informatics
• APPROACH and PRACTICE– Basic, social and clinical sciences– Attitudes, ethics and legal– Decision making , reasoning, judgment
• PROFESSIONAL– Role within health service– Personal development
ACGME COMPETENCIES
• Patient Care
• Medical Knowledge
• Practice Based Learning and Improvement
• Systems Based Practice
• Professionalism
• Interpersonal Skills and Communication
• Source: ACGME Core Competencies | University of Maryland Medical Center http://umm.edu/professionals/gme/competencies
CanMEDS Competencies
CAnMEDS Roles
• Adopted by 26 countries worldwide
• Implemented (?) in all RCPS training programs
The ISSUES
• Problems with:
– Objective proliferation
– De-emphasis of “softer” competencies
– Mastery modelling
– Sampling and statistical precision
– Assessment – macro or micro
– Logistics and remediation
Objective Proliferation
• Each major goal decomposed into objective, sub-objectives
• Number of sub-objectives become unmanageable
• Even at sub-sub goal, it is still not sufficiently specific
CanMEDS Roles
Level 2 Level 3 • Expert 6 28• Communicator 5 17• Collaborator 2 16• Manager 4 13• Advocate 4 13• Scholar 4 25• Professional 3 14
• TOTAL 28 126
Subgoals proliferate
• CanMEDS 126
• Scottish Doctor 86
• ACGME (Internal Medicine) 142
however…. still not specific
CanMEDS sub – sub - goals
1.1.1 Effectively perform a consultation, including the presentation of welldocumented assessments and recommendations in written and/or verbal form response to a request from another health care professional
1.2.1 Apply knowledge of the clinical, socio-behavioural, and fundamental biomedical sciences relevant to the physician’s specialty
1.5.2 Demonstrate effective, appropriate, and timely performance of therapeutic procedures relevant to their practice
3.1.8 Describe the principles of team dynamics
4.1.4 Describe principles of healthcare financing, including physician remuneration, budgeting and organizational funding
4.2.4 Set priorities and manage time to balance patient care, practice requirements, outside activities and personal life
6.3.1 Describe principles of learning relevant to medical education
Scottish Doctor Goal and subgoal
GOAL
‘‘Take a history from patients, relatives and others”
Sub- GOAL
• All age groups; local/multi-ethnic factors; a wide range of different contexts; and a patient-centred, sensitive, structured and thorough approach with demonstration of principles of good communication
(And, by the way, how many kinds of history are there?)
ACGME Competencies
“Ability to perform and interpret common laboratory tests”
But there are 39 “common laboratory tests” in internal medicine (Wigton, 1989)
De-emphasis of soft competencies
“an outcome-based approach highlights areas in the curriculum which may be neglected whether they are traditional aspects of medical practice such as ethics and attitudes….” Harden, 2007
************************
“educational processes, which influenced the development of values, insight, and judgement, could not be subsumed into a curriculum model that focused strictly on behavioural objectives. When students learned affectively, socially, culturally, aesthetically, or ethically from experience, it was not possible to specify goals or assess them objectively but that did not mean such types of learning were unimportant” Morcke, 2012
“qualities, such as humanism, altruism, professionalism, and scholarship, are difficult to define in objective, measurable terms, and thus may receive short shrift in assessment” Norman, Bordage, Norcini, 2014
Mastery Modelling
• The identification of a single smooth curve crossing a threshold (the EPA):
– results in a formidable measurement problem
– is dependent on a precise specification of the EPA
– Is inconsistent with actual accumulation of skills / competencies
Growth of competence over time
training deliberate professional practice
Dreyfus & Dreyfus, 1986
proficient
expert
competent
advanced
novice
However…
• It’s not that easy to identify the EPA level
(Wijnen-Meier, 2012)
• 35 MD students• 28 FY1 residents• 23 FY2 residents …. and their supervisors
• Rate 10 clinical and 4 general activities1= Not yet able to do a task2= May act under full supervision3 = May act under moderate supervision4= May act independently5= May act as instructor
• 10 Clinical Activities– e.g. draw up an examination plan for a new
patient at the Outpatient department
– e.g. To treat a skin laceration of about 4 cm at one of the extremities.
• 4 General activities– e.g. write an evidence-based case-report for
colleagues with limited preparation time (three hours).
Can’t tell them apart
• “Principal component analysis on ‘clinical activities’ shows a one-factor solution that explains 53% of the variance”
• “Principal component analysis on the part ‘‘general activities’’, revealed a one-factor solution explaining 65% of the variance”
• “Reliability analysis showed high internal consistency (alpha = .80) between supervisors”
Level of Competence
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
MD GY1 PGY2
Clin Self
Clin Super
Gen Self
Gen Super
(Wijnen-Meier, 2012)
Level of Competence
MD GY1 PGY2
Clin Self
Clin Super
Gen Self
Gen Super
Mastery modelling
• Even simple behaviours, learned on controlled settings, with multiple measurements, do not follow a smooth master curve
The Learning Curve
Ward ST, Mohammed MA, et al. BMJ Open Resp Res, 2014
297 residents, 36,730 colonoscopies
“Cecal Intubation Rate” (CIR) at 90% achieved at 233 procedures
Based on 297 data points, 36,730 observations
234 290
The “Pursuit Rotor”
Learning on Pursuit Rotor (Adams, 1952)
– 39 subjects, 2 groups (massed, distributed
– Record time when pointer in contact with brass disc
– Recording every 10 sec.
Massed• 6 min./day x 5 days = 30 min.
Distributed• 36 x 10 sec. trials / day x 5 days with 40 sec. break = 30
min.
Average of 20 students
Average of 19 students
48 108
60 145
Sampling Issues
Because of content specificity, it is not feasible to collect sufficient data to provide sufficient precision to permit defensible identification of time to competency
McMaster Undergraduate Medical Program (classes 2005-2016)
McMaster Undergraduate Medical Program (class of 2014)
McMaster Undergraduate Medical Program
(Individual Students in Class of 2014)
McMaster Undergraduate Medical Program
(Individual Students in Class of 2014)
McMaster Undergraduate Medical Program
(Individual Students in Class of 2014)
Assessment Issues--Macro or Micro
“General competencies are what most governing bodies and educational leaders responsible for physician training like to stress.”
“It is not necessary to choose between competencies and activities. Rather, by acknowledging that both are relevant pieces of the training process and that each represents a different dimension of the same overall objective of professional training, we can reconcile the concepts of competence and clinical practice.”
Ten Cate and Scheele, 2008
Can we assess competencies reliably?
• MICRO level sub or sub-sub goals– Assessment of individual activities, like suturing,
venipuncture
• MACRO level – overall goals– Assessment of CanMEDS roles directly, using ongoing
assessment like mini-CEX (Sherbino, 2013) or summative assessment like ITER ( Tromp, 2012; Ginsburg, 2013)
Global Rating Checklist
N = 21 18 9 N = 21 18 9
MICRO Level (Dwyer et al, 2014)6 stations – Clinical Radiograph, Procedure39 Residents5 point global rating, specific checklist
• Some junior residents outperformed some fellows
• But despite same training, differences were present between groups
“Despite identical intensive teaching to a set curriculum within a CBME model, junior residents were not able to demonstrate or apply clinical skills as well as senior residents, suggesting that experience is a critical factor”
Simple mastery model may not be appropriate
MACRO level Summative
• Tromp, 2012
– Competency Assessment List COMPASS
• 7 CanMEDS roles; 19 sub-objectives
– Completed by trainer over successive 3 month periods
– 68 trainees
Results“Compared with T1, most trainees received higher ratings in T2. A relative small group of trainees with no increase in ratings was found in all roles: Medical Expertise (21.3%), Communication (31.3%), Management (22.5%) Collaboration (23.8%), Social Accountability (22.5%), Science and Education (22.5%) and Professionalism (16.3%).”
Change over 3 month about 1.0/6, ES = .6
“Scores of the Compass ... show[ed] excellent internal consistency ranging from .89 to .94.”
But that‘s too high. Does not differentiate roles.
• Ginsburg, Eva, Regehr
– ITER with 7 CanMEDS roles, 19 items and 5 point scale
– 63 residents, average 9 rotations over 2 years
“The reliability of a single score (end of rotation evaluation) was 0.11 for both the year one and year 2 data”
“The unrotated factor solution revealed two factors with the first accounting for 66% of the variance and every item loading more or less equally… the second factor accounted to 5.4% of the variance.”
- Reliability too low
- Unable to discriminate roles
Conclusions - MACRO summative
Summative assessments are incapable of differentiating mastery at different levels of different competencies.
– The EPA model fails
MACRO Ongoing
• Sherbino et al., 2013
– 4 week EM rotation
– 155 medical students
– 1819 evaluations of encounters
– 7 CanMEDS roles with descriptors and 10 point scale
– Evaluator assessed Medical Expert and 1 or 2 other roles
Proportion of completed ratings
0
20
40
60
80
100
120
Column1
Percent of variance in Scores
2
Student
Rater
Item
Error
67%
20%10%
3%
“a single rater’s rating had a reliability of 0.12. Averaging across a minimum of 17 raters was required for an acceptable reliability of 0.7”
“[Factor] analysis revealed a single factor ..accounting for 87% of the variance in scores… the next largest factor accounted for less than 4% of the variance….Loadings ranged from 0.88 to 0.97”
• Few assessments of roles other than Medical expert
• Unable to discriminate roles
• Reliability too low
Conclusions - MACRO ongoing
Ongoing assessments are incapable of differentiating mastery at different levels of different competencies.
– The EPA model fails
Conclusion
• We will never be able to assess competencies at the required level of specificity with the required level of precision to make defensible judgments of “time to attain the EPA level of proficiency” across the full range of competencies
• The notion that CBE will achieve efficiency through individually reduced training time is fundamentally flawed
Residual benefits
CBE does represent an important approach to identifying core competencies
– Useful for curriculum design to ensure uniform exposure
– Useful for formative assessment using sampling strategies
– Useful for summative assessment (licensure, certification) using sampling strategies
• Ferguson PC, et al. J Bone Joint Surg (2013)
Introduction of CanMEDS CBE increased number of assessments/ per resident
No of assessments Old / New
0
5
10
15
20
25
30
35
40
Old New
PGY1
PGY2
PGY3
“The difference between an optimist and a pessimist is that the optimist believes this is the best of all possible worlds….
and the pessimist is afraid he’s right.
J.R.Oppenheimer