Measuring Non-technical Aspects of Surgical Clinician Development in an

Otolaryngology Residency Training Program

Jennifer J. Shin, M.D. S.M.Michael J. Cunningham, M.D. Kevin G. Emerick, M.D.Stacey T. Gray, M.D.

Society for University OtolaryngologistsNovember 13, 2015

The Constitution of a Surgeon

It takes five years to learn how to operate and twenty years to learn when and when not to.

-Anonymous

The Constitution of a Surgeon

It takes five years to learn how to operate and twenty years to learn when and when not to.

-Anonymous

I would like to see the day when somebody would be appointed surgeon somewhere who had no hands, for the operative part is the least part of the work.

-Harvey Williams Cushing, Letter to Dr Henry Christian,

1911. Quoted in 'The Best Hope of All', Time

(3 May 1963).

Validated Instruments

…Clinical Outcomes…Evidence-based Practice

Validated Instruments

…Educational Outcomes…Evidence-based Residency

Novice Intermediate Competent

Instrument to measure the development of

clinical practice ability?

Instrument Design

Rapid/real time evaluation

Insight into each trainee’s thought process

Concrete feedback and data compilation

Benefits beyond the learning/measurement itself

Instrument AdministrationMock oral board exam Case criteria:

1) Single patient with a single main diagnostic issue.

2) Case culminates in an intervention, with one resulting complication

3) Enough detail available so that a full history, physical examination, and complement of diagnostic testing may be obtained -- actual radiologic images, lab values, audiometric studies, or other diagnostic test results are available

Instrument Development Iterative process:

22 successive drafts

311 candidate questions

Face validity evaluation: Residency program director

Division chairman

Professionalism director

Resident education curriculum supervisor

Epidemiologist/Instrument validation scientist

Validation phase: Instrument assessment

Inter-rater Reliability• Cohen Kappa 0.66 (SE 0.03)• 72.5% agreement

Internal Consistency• Cronbach alpha >0.87

Responsive to Change

OBJECTIVES

• To utilize the clinical practice instrument (CPI) to measure non-technical diagnostic and management skills during otolaryngology residency training

• To determine whether there is demonstrable change in these skills between PGY-2, 4, and 5 residents

• To evaluate whether results vary according to subspecialty topic or method of administration.

METHODS

• Prospective study of an otolaryngology residency training program

• Institutional review board approved

• n=248 evaluations of 45 otolaryngology resident trainees at regular intervals

• Analysis of variance with nesting and post-estimation pairwise comparisons to evaluate total and domain scores according to training level, subspecialty topic, and method of administration

• Examination preparation and security

PGY-4 & 5

RESULTS

Total scores were significantly different among PGY-levels of training, with lower scores seen in the PGY-2 level compared to the PGY-4 or PGY-5 level (p<0.0001).

PGYLevel

n MeanStandard Deviation

Minimum Maximum25th Percentile

75th Percentile

2 51 44.2 16.3 23 87 32 53

4 101 63.6 12.8 39 94 54 72

5 96 68.8 13.3 34 92 60 78.5

All 248 61.6 16.5 23 94 50 74

RESULTS

Residents scored higher in general otolaryngology than in the subspecialties (ANOVA, p<0.003).

There was a significant difference in between general otolaryngology and pediatric otolaryngology (p<0.0001), and between general otolaryngology and head and neck surgery (post-ANOVA pairwise comparison, p<0.0033)

RESULTS

Administering the examination with an electronic scoring system, rather than a paper-based scoring system, did not affect these results.

The calendar year of administration did not affect these results.

RESULTS

CONCLUSIONS

• Standardized interval evaluation with the CPI demonstrates improvement in qualitative diagnostic and management capabilities as PGY-levels advance.

• Administration of the CPI has been formally incorporated into the Harvard otolaryngology residency curriculum.

• The CPI can potentially be adapted for use in any otolaryngology training program, and is potentially applicable to any surgical specialty.

FUTURE DIRECTIONS

• Short-term educational missions…

• Milestones…

• Electronic case library…

• Program update follow up?

• Benchmarks?

• Other sites?

Thank you