the anatomy of a failure: can we predict which residents are going to pass/ fail

The Anatomy of a Failure: Can We Predict Which Residents are Going to Pass/ Fail the ABS Certifying Exam?

Marco Zahedi, BS, Ajay V. Maker, MD, Dana Villines, MA, Vijay K. Maker, MD

University of Illinois at Chicago, Metropolitan Group of Hospitals Residency in General Surgery

Purpose

To study the patterns of failure in the ABS Certifying examination and To determine if residents at risk of not passing the ABSCE can be identified a priori.

Methods

Control(n=13)

Test(n=13) p-value

Gender (male) 9 (69.2%) 9 (69.2%) 1.00

International Medical School Graduate 6 (46.2%) 3 (23.1%) 0.22

Non-native English Speaker 8 (61.5%) 9 (69.2%) 0.68

SAMPLE(2001-2010) Graduated 60 residents

Non Native Speakers

Total Sample

Test Group

Control group p-value

Language

(non-native)

13

(36%)

6

(35%)

7

(37%)0.92

• Knowledge• Communication• Spoken English

KNOWLEDGE SCORES COMMUNICATION SKILLS SCORES

SPOKEN ENGLISH

USMLE 1 IN-HOUSE MOCK ORALS (MONTHLY) *

NATIVE VS, NON-NATIVE

USMLE 2 CITY-WIDE MOCK ORALS(YEARLY) *

INTERNATIONAL VS.U.S. MEDICAL GRADUATE

ABSITE SCORES (YEARLY)

IN-HOUSE MOCK ORAL KNOWLEDGE (MONTHLY)

CITY-WIDE MOCK ORAL KNOWLEDGE (YEARLY)

Descriptive categories with variables of Interest

* Separate measuring tool

USMLE 1

…emphasis on principles and mechanisms underlying health, disease, and modes of therapy.

USMLE 2

…apply basic patient-centered skills that provide the foundation for the safe and competent practice of medicine.

ABSITE

Percentile scores of the 5 yrs. of residency

UIC/MGH RESIDENCY IN GENERAL SURGERY″MOCK ORALS″

Objective Evaluation of Clinical Scenarios4th Tuesday, 7:00 a.m. to 8:00 a.m.

Score

Criteria

0/1FAIL

2 4 5EXCELLENT

* Knowledge

Answers may harm the patient

Knows 75% of key points

Knows all key points

Communication"To the Point"

"Lost" Inspite of offering direction

Unable to answer all key points in 10 minutes

Answered keypoints in less than 10 minutes

Professional BehaviorDoesn't listen, Stuck on answers, rigid

Accepts queues and able to reconsider a diagnosis

Accepts queues and able to reconsider a diagnosisRemains calm, objective and humble

Professional AppearanceStreet clothes disheveled

Acceptable "but"Clean "white" coat -Well groomed

Evaluator Signature:Resident Signature:

Citywide Mock Orals

KNOWLEDGE SCORES COMMUNICATION SCORES

SPOKEN ENGLISH

USMLE 1 IN-HOUSE MOCK ORALS (MONTHLY) *

NATIVE VS, NON-NATIVE

USMLE 2 CITY-WIDE MOCK ORALS(YEARLY) *

INTERNATIONAL VS.U.S. MEDICAL GRADUATE

ABSITE SCORES (YEARLY)

IN-HOUSE MOCK ORAL KNOWLEDGE (MONTHLY)

CITY-WIDE MOCK ORAL KNOWLEDGE (YEARLY)

Descriptive categories with variables of Interest

* Separate measuring tool

UIC/MGH RESIDENCY IN GENERAL SURGERYCRITIQUE FOR MOCK ORAL EXAM COMMUNICATION SKILLS

4th Tuesday, 7:00 a.m. to 8:00 a.m.

Evaluator Signature:

Resident Signature:Guzman E, Babakhani A, Maker VK. Improving outcomes on the ABS certifying examination: can monthly mock orals do it? J Surg Educ. 2006;65:441-43.

1Unacceptable

2 3Appropriate

AttireProfessional appearanceCoats buttoned

Body LanguageOpen versus Defensive

PostureSits upright with hands at side

Motor BehaviorCalmNo repetitive movementsNo agitated motor behavior

Eye ContactMakes eye contact with examiner

Linguistic SkillsUse of full sentencesNo Run-on answersNo Um, UhAble to accept verbal cues

IntonationNot hostileNot defensiveNot argumentative

Overall Confidence LevelAppears ConfidentSelf Assured

Comments:

Non Verbal Communication

• Makes up two-thirds of all communication• and should match verbal communication• Less then 10 seconds to make a good impression• First sight observation makes 83% of the impact• Best practiced when persons are face to face

Pease B., Pease A. 2004“The Definitive Book of Body language”

SOLER: Non Verbal communication skills Egan G. 1975

Charles Darwin 1871 The Expressions the Emotions in Man and the Animals

Results

Total Sample TestGroup Control group p-value Partial η2

USMLE Step 1 205.58(16.04)

205.45(16.11)

205.69 (16.64) 0.97 .000

USMLE Step 2 208.08 (20.03)

200.33 (20.11)

215.23 (17.80) 0.06 .144

ABSITE scores 57.88(18.86)

51.77 (19.29)

64.00 (16.98) 0.10 .109

In-House MockOral Knowledge scores

55.81(7.39)

52.62(7.35)

57.77(7.00) 0.124 .120

City-wide Mock Oral Knowledge scores

4.48(1.66)

4.00(1.85)

4.77(1.54) 0.32 .053

Mean Knowledge Scores

Total Sample

TestGroup

Control group p-value Partial η2

In-House Mock Oral Professionalism scores

(mean with SD)

90.55(5.28)

87.22(5.78)

92.85(3.53) 0.01 .287

City-wide Mock Oral Professionalism scores (count of residents with positive communication

comments)

6(30.0%) 0 6

(42.6%) 0.03 --

Verbal and Non-verbal Communication Scores

Candidates with higher level of Communication proficiency have higher levels of Organization of their knowledge points

James E. Houston and Everett V. Smith Jr. UICEval Health Prof 2008 31:404

• Difficulty focusing• Does not listen• Needs to be more confident with answers• Difficulty listening to questions• Poor communicator x 4• Looking at side and floor. Poor posture• Slow to respond

Citywide Mock Orals “F”

231 / 248 / 80

Citywide Mock Orals “P”• Nervous, low confidence, needs prompting• logical, poised, confident• soft spoken, needs more confidence• improve rate of speech and focus• lacks confidence, poor posture/attn to details• complete approach, competent• sharp• very confident, relaxed• improve eye contact and rate of speech• avoid smirking, laughing, rolling eyes• good verbal skills• good eye contact and rate of speech• too wordy, improve confidence• stumbles at times• too informal, poor posture• nervous, improve eye contact• good eye contact and confidence• good structure. needs more confidence

243 / 245 / 95

Total Sample

TestGroup

Control group p-value

Non-native English Speaker

20 (69.0%)

8 (61.5%)

9 (69.2%) 0.68

International Medical Graduate

10 (34.5%)

6 (46.2%)

3 (23.1%) 0.22

Spoken English Scores

Native versus Non-native English Speaker

NativeSpeaker

Non-native Speaker

p-value

USMLE Step 1 200.78(13.12)

208.12(17.21)

0.28

USMLE Step 2210.25(17.28)

207.06(21.63)

0.72

ABSITE scores48.78

(17.89)62.71

(18.02)0.07

In-House MO Knowledge scores

53.43(6.66)

57.00(7.69)

0.31

City-wide MO Knowledge scores

4.00(2.31)

4.71(1.27)

0.37

In-House MO Professionalism scores

89.71(6.50)

90.93(4.81)

0.63

City-wide MO Professionalism scores

(count of residents with positive communication comments)

2(28.6%)

4(30.8%)

0.92

US versus International Medical School Graduate

US International p-value

USMLE Step 1 202.29(14.01)

211.78(18.59)

0.16

USMLE Step 2204.62(17.81)

214.22(23.30)

0.26

ABSITE scores50.53

(15.43)71.78

(17.43)0.004

In-House Mock Oral Knowledge scores

55.08(8.33)

57.00(5.88)

0.58

City-wide Mock Oral Knowledge scores

4.31(1.80)

4.75(1.48)

0.57

In-House Mock Oral Professionalism scores

90.50(5.33)

90.63(5.55)

0.96

City-wide Mock Oral Professionalism scores(count of residents with positive communication

comments)

3(25.0%)

3(37.5%)

0.55

1. Professional verbal and non-verbal communication was the most important factor in the first time pass rates in the certifying exam. of the American Board of Surgery (ABSCE).

2. USMLE Step2 scores are more aligned with ABSCE pass rates than Step 1 scores.

3. US or International Medical School Graduates as well as native or non-native English speakers did not differ for ABSCE pass rates.

CONCLUSIONS

Limitations

• Sample size• Make up of sample ( AMG 83% IMG 17%)• Communication Tools• Spoken English Tools

The Second Time Around

Pass: Second Time : 85% (11) Pass: Third Time : 15% (2)

Mark R. Raymond and Ulana A. Luciw-DubasNBMEEval Health Prof 2010 33:386

•Measurement error lacks reliability rater inconsistency•Construct irrelevance

• Chief Residents who fail ABSCE are lesser Surgeons ?

• Do they become better surgeons when they pass on the second attempt?

• Should they be given same operating privileges ?

• Should they get reimbursed the same amount of money as you ?

• Is second or third time “pass” a public information, if yes…legal implications?

Questions for APDS

The CIS subcomponent of Step 2 CS has been redesigned to assess a fuller range of competencies. The new approach divides communication skills into a series of functions. Beginning June 17, 2012, the Communication and Interpersonal Skills (CIS) scale will focus on five functions:

1. Fostering the relationship2. Gathering information3. Providing information4. Making decisions: basic5. Supporting emotions: basic

Several additional functions are still under development; these include making decisions: advanced; supporting emotions: advanced; and helping patients with behavior change.

Availability of USMLE step 2 CK and CS prior to interviews/ match date

Section Question:____ Question:____ Question:____ Question:____

Perform History & Physical Pass Fail Pass Fail Pass Fail Pass Fail

Select appropriate test Pass Fail Pass Fail Pass Fail Pass Fail

Interpretation of test results Pass Fail Pass Fail Pass Fail Pass Fail

Establish Differential Diagnosis Pass Fail Pass Fail Pass Fail Pass Fail

Discuss TX options/ Initiate TX Pass Fail Pass Fail Pass Fail Pass Fail

Discuss Surgical/ Medical Approach Pass Fail Pass Fail Pass Fail Pass Fail

Discuss post op/ Follow-up care Pass Fail Pass Fail Pass Fail Pass Fail

Manage Complications and Unexpected Findings Pass Fail Pass Fail Pass Fail Pass Fail

Describe Professional Behaviors Pass Fail Pass Fail Pass Fail Pass Fail

Specific Question Evaluation Pass Fail Pass Fail Pass Fail Pass Fail

Overall Session Evaluation: PASS FAIL

Comments:

Metro Chicago Chapter ACSMock Oral BoardsEVALUATION SHEET Name of Resident:____________________________

Year of Training: Please circle one: □ PGY IV □PGY V•Professional Appearance ___ Acceptable ___ Not Acceptable ___ Needs Improvement•Eye Contact ___ Yes ___ No •Rate of Speech __ Acceptable ___ Not Acceptable ___ Needs Improvement•Tone of Voice, Clarity __ Acceptable ___ Not Acceptable ___ Needs Improvement•Posture __ Acceptable ___ Not Acceptable ___ Needs Improvement•General Confidence __ Acceptable ___ Not Acceptable ___ Needs Improvement•Use of appropriate language __ Acceptable ___ Not Acceptable ___ Needs Improvement

SUMMARY OF APPLICATION FOR Ranking

UIC/MGH INTERVIEW DATE:NAME:PRELIMINARY CATEGORICAL

UNDERGRADUATE SCHOOL:MEDICAL SCHOOL:OTHER TRAINING:

RATING SCALE PART I

USMLE PART 1

200-210 (20-40 %ile) = 0.5

211-214 (40-50 %ile) = 0.75

215-225 (51-60 %ile) = 1.0

226-230 (61-70 %ile) = 1.25

231-237 (71-80 %ile) = 1.50

238-248 (81-90 %ile) = 1.75

249-261 (91-97 %ile) = 2.00 AOA

TRANSCRIPT – BASIC SCIENCE

“A” HONORS = 1.0

“A”/”B” = 0.5

≥ “C” = 0.25

ANY FAILURES = 0

RATING SCALE PART II

CLINICAL ROTATIONSOUTSTANDINGHONORS = 1.5

“A”/”B” = 1.0≥ “C” = 0.5ANY FAILURES = 0

LETTERS OF RECOMMENDATION

LETTERS BY KNOWN AUTHOR (WE WANT HIM/HER IN OUR PROGRAM) – OUTSTANDING

= 1.5

GOOD = 1.0AVERAGE = 0.5

INTERVIEW PART III

COMMUNICATION SKILLS, APPLIED MEDICAL KNOWLEDGE, RESEARCH, & OTHER PERSONAL TRAITS IN LINE WITH ACGME COMPETENCIES

TOP THIRD, WILL ACE BOARD EXAMS ( I and II)=3

MIDDLE THIRD, WILL PASS BOARD EXAMS ( I and II) =2

BOTTOM THIRD, ACCEPTABLE,WILL NEED TO BE OBSERVED CLOSELY =1

NOT SUITABLE FOR OUR PROGRAM REJECT

COMMENTS (COMPULSORY):

the anatomy of a failure: can we predict which residents are going to pass/ fail

Documents

citywide mock oralsyearly

inhouse mock orals monthly

knowledge communication

resident signature

native vs

groomed evaluator signature

key points communicationto

general surgery critique