national operative case log growth charts in ... · plotting a child’s growth against these...
TRANSCRIPT
Original Research—General Otolaryngology
National Operative Case Log GrowthCharts in Otolaryngology–Head and NeckSurgery Training
Otolaryngology–Head and Neck Surgery2015, Vol. 152(1) 73–79� American Academy ofOtolaryngology—Head and NeckSurgery Foundation 2014Reprints and permission:sagepub.com/journalsPermissions.navDOI: 10.1177/0194599814552400http://otojournal.org
Noel Jabbour, MD1, and Terance Tsue, MD2
Sponsorships or competing interests that may be relevant to content are dis-
closed at the end of this article.
Abstract
Objective. To report national standard case log growthcurves for operative procedures in otolaryngology andto describe a method by which program directors canchart surgical case numbers over resident training tolongitudinally assess sufficiency of cases and paritybetween residents.
Study Design. Data visualization and analysis.
Setting. American Council for Graduate Medical Education(ACGME) national case log data for otolaryngologyresidency.
Subjects. National data set; no individual subjects.
Methods. National statistical case log reports for otolaryngologywere obtained from the ACGME for each postgraduate year(PGY) level in 2009, 2010, and 2011. Estimated means and stan-dard deviations were calculated. The mean and increments ofstandard deviation were graphed against time to create case loggrowth charts, similar to pediatric growth charts.
Results. Case log growth charts were made for eachACGME Otolaryngology Residency Review Committee keyindicator procedure. Progress of an individual resident or ofa cohort of residents may be graphed against this growthchart background over their training time.
Conclusions. National operative case log growth charts allowresidents and program directors to graphically assess prog-ress in obtaining a sufficient variety and number of operativeprocedures over time throughout training. This can provideearly identification when residents begin to fall below thegrowth curve during training.
Keywords
operative log, case log, surgical education, surgical compe-tency, developmental growth charts
Received November 1, 2012; revised August 28, 2014; accepted
August 28, 2014.
There are many parallels between the development of
a surgical trainee and the growth of a child. So, too,
are there similarities between the methods used to
assess the progression of resident development and of child
development.
The program director’s semiannual review is the residency
equivalent to a well-child check. At a routine well-child visit,
a child’s neurocognitive, communication, and psychomotor
development is compared with that of his or her age-matched
peers. The counterpart to this in resident education is clinical
competency assessments, faculty evaluations of clinical per-
formance, and now, at a national level, the development of
the Accreditation Council for Graduate Medical Education
(ACGME) Milestones.1
In addition, at a well-child visit, a child’s anthropometric
measurements—height, weight, and head circumference—
are compared with those of his or her peers. This is made
possible by normative growth charts, which were developed
in the 1940s.2
Plotting a child’s growth against these standards has become
a routine method to identify deviations from normal growth so
that they may be addressed at the earliest possible time.
In surgical training, the operative case log is the best cur-
rent anthropometric measurement. However, there is no ana-
logue to the pediatric growth chart. Currently, operative
cases are most often viewed in isolation or at best with com-
parison to graduate resident median and mean numbers.
This method provides little insight into the adequacy of
training at various time points in the process and can lead to
late identification of training inadequacy.
Our goal is to develop and report national standard case
log growth curves for operative procedures in otolaryngol-
ogy and to describe a method by which program directors
1Children’s Hospital of Pittsburgh of UPMC, Department of Otolaryngo-
logy, University of Pittsburgh, Pittsburgh, Pennsylvania, USA2Department of Otolaryngology, University of Kansas Medical Center,
Kansas City, Kansas, USA
This article was presented at the 2012 AAO-HNSF Annual Meeting & OTO
EXPO; September 9-12, 2012; Washington, DC.
Corresponding Author:
Noel Jabbour, MD, Children’s Hospital of Pittsburgh of UPMC, Department
of Otolaryngology, University of Pittsburgh, 4401 Penn Avenue, Faculty
Pavilion, 7th Floor, Pittsburgh, PA 15224, USA.
Email: [email protected]
may use these curves to assess resident progress throughout
training.
Methods
Final mean operative case log numbers for ‘‘resident sur-
geon’’ classification, including standard deviation for 2011
otolaryngology graduates, were obtained from the ACGME.
In addition, median case numbers for each procedure cate-
gory and each key indicator procedure in otolaryngology
were obtained at each postgraduate year (PGY) level for the
academic years 2008-2009, 2009-2010, and 2010-2011.
Pooled national data sets were used, and no individual
data sets were accessed or obtained. As such, no institu-
tional review board approval was required for data visuali-
zation and analysis.
The median numbers for ‘‘resident surgeon’’ classification
at each PGY year level were averaged across the 3 years of
data to obtain average national median case numbers for each
procedure at each PGY level.
Estimation of the means and standard deviations (SD)
from the mean for each procedure at each PGY level was
made using 2 assumptions: (1) the ratio between the median
and the mean is relatively consistent throughout training,
and (2) the ratio between the mean and 1 standard deviation
above and below the mean is relatively consistent through-
out training.
Thus, for each procedure, estimated means at each PGY
level were calculated from the median numbers by using the
following formula:
MeanPGY�X ~5MedianPGY�X � MeanGrad=MedianGradð Þ:
Estimated standard deviations from the mean were calcu-
lated in a similar fashion at each PGY level by using the
following formula, where 1 SD equals 1 SD above the
mean:
1SDPGY�X ~5MeanPGY�X � 1SDGrad=MeanGradð Þ:
For each key indicator procedure, these values were
plotted on the y-axis against time in training on the x-axis.
Increments of 1 SD were similarly calculated and plotted. A
curve of best fit was drawn for each estimated mean and
increments of both 1 and 2 standard deviations above and
below the mean.
The shape of each growth curve was described based on
which portion of a sinusoidal curve it most resembled: the
growth phase of the curve, the transition phase of the curve,
or the plateau phase of the curve (Figure 1). Thus, the
shape of the curve at the conclusion of residency was
termed the end-point geometry; each curve was labeled as
fitting 1 of 3 categories: (1) growth phase (if the rate of
accumulation of cases was progressively increasing during
the final years of training), (2) transitional phase (if the rate
of accumulation of cases was just beginning to decrease in
the final years of training), or (3) plateau phase (if the
number of cases was relatively flat in the final year of train-
ing with peak growth in number of cases occurring earlier
in training).
Key indicator growth charts were created with collapsed
data for each of the 4 key indicator domains. The standard
deviation used for the combined data was estimated from
the square root of the sum of the variances of each individ-
ual procedure.
Results
Using the above methods, national case log growth charts
were developed for each of the 14 ACGME Otolaryngology
Residency Review Committee key indicator procedures.
For the Head & Neck Surgery key indicator domain, each
of the 4 key indicator procedures—glossectomy, parotidect-
omy, neck dissection, and thyroid/parathyroid surgery—had a
growth-phase end-point geometry (Figure 2). The same was
true for all 3 key indicator procedures in the Otologic
Surgery domain—mastoidectomy, tympanoplasty, and stape-
dectomy/ossiculoplasty (Figure 3). In the Facial Plastic
Surgery domain, mandible/midface fracture had transitional-
phase geometry, while both rhinoplasty and flaps had
growth-phase end-point geometry (Figure 4).
In the General/Pediatric key indicator domain, ethmoi-
dectomy had a growth-phase end-point geometry. Both the
Airway–Pediatric and Adult key indicator and Congenital
Neck Mass key indicator had a transitional-phase end-point
geometry, while Bronchoscopy had a plateau-phase geome-
try (Figure 5).
Figure 1. End-point geometry of growth curves. Examples of 3 categories: (A) growth phase, (B) transitional phase, and (C) plateau phase.
74 Otolaryngology–Head and Neck Surgery 152(1)
When viewing combined data across each domain, in the key
indicator domains of Head & Neck Surgery, Otologic Surgery,
and Facial Plastic Surgery (Figure 6), growth-phase end-point
geometry was found. The General/Pediatric key indicator domain
demonstrated transitional-phase end-point geometry.
Discussion
Program directors are charged by the ACGME to evaluate
the sufficiency of case exposure for each resident as well as
the parity of case exposure between residents.3 This should
be evaluated on a semiannual basis. Until recently, there has
been no described method for making these comparisons gra-
phically. In addition, there was no ability to compare cases
with national averages except for graduate case numbers.
Program-specific growth charts have been developed in
the Department of Otolaryngology at the University of
Minnesota that compare current resident cases with the
growth charts of historical averages of recent graduates of
the program. These have been termed Growth in Operative
Procedures against Historical Expected Results (GOPHER)
graphs. However, to our knowledge, national operative log
growth charts have not been described in any field.
In developing national case log growth charts, we sought to
obtain mean numbers of cases for every resident year in train-
ing. These data are theoretically available in the ACGME case
log system but are not easily extracted to obtain national
means except for graduate case numbers. However, median
numbers for each year as well as final mean numbers may be
more easily extracted. Thus, as described in the Methods, the
mean numbers presented are estimated means based on the
assumption that the ratio between the median and the mean
remains consistent throughout training.
There is often a significant discrepancy between mean
and median case numbers due to extreme highs in many
cases that inflate the mean, as has been demonstrated by
Rosenberg and Franzese.4 Creating a curve based on the
median numbers would be possible without this calculated
adjustment to estimate the mean but would display a trajec-
tory of a curve that would significantly undershoot the pub-
lished mean numbers. In addition, the standard deviation
must be displayed relative to the mean rather than the
median. However, it should be noted that a curve based on
the mean provides a high bar that most residents will not
attain.
Figure 2. Head & Neck Surgery key indicator domain: resident surgeon numbers vs postgraduate year (PGY) level of training.
Jabbour and Tsue 75
With an understanding of these limitations, program direc-
tors may use these growth curves as part of the semiannual
review for residents at every level of training. Each resident’s
case numbers may be plotted against the background of these
growth curves. This would be most helpful in the third through
fifth years of training, as key indicator cases tend to be
procedures that are performed in the latter years of training.
This can be demonstrated by the end-point geometry of the
key indicator growth curves.
Most procedures performed early in training, such as
tympanostomy tube placement and tonsillectomy, follow a
sigmoidal-shaped growth curve. That is, there is a growth
Figure 3. Otologic Surgery key indicator domain: resident surgeonnumbers vs postgraduate year (PGY) level of training.
Figure 4. Facial Plastic Surgery key indicator domain: resident sur-geon numbers vs postgraduate year (PGY) level of training.
76 Otolaryngology–Head and Neck Surgery 152(1)
phase that increases exponentially and then linearly, a tran-
sitional phase in which there is beginning to be a decreased
slope to the growth curve, and then a plateau phase in
which the curve nears an asymptote—when it is no longer
performed by senior residents with significant frequency.
Many of the key indicator cases have curves that resemble a
truncated version of this graph; we have thus categorized
them based on which portion of this graph the curve most
resembles.
Ten of the 14 key indicators had a growth-phase end-
point geometry, where the rate of accumulation of these
cases was still increasing in the final years of training. Four
of the key indicators had transitional-phase or plateau-phase
end-point geometries—mandible/midface fractures, airway–
pediatric and adult, congenital neck mass, and broncho-
scopy. For these cases, the peak time period for exposure to
these cases is before the final year of training. These key
indicators may be better suited for evaluation of middle-
level residents.
In addition, the program director may use these growth
charts as a background for graphically displaying overall
changes in residency case numbers compared with national
averages in a nearly real-time approach. This may enable a
more accurate assessment of the effect of programmatic
changes, such as changes to rotation schedules, affiliated
faculty, participating sites, or complement increases or
decreases.
In making these comparisons, it should be remembered
that programs vary in the timing of procedures based on
their rotation schedules; however, we hope that the ability
of an individual resident to plot his or her case log progress
against the background of a national case log growth chart
would enhance each resident’s ability to reflect on the
strength and weaknesses of his or her training. This would
provide further information to senior residents, who may be
charged with making case assignments in some institutions,
to identify the categories of cases in which they may still be
deficient.
At a national level, the ability to plot a resident case log
against a national average case log could provide valuable
information to the ACGME Residency Review Committee.
In addition to reaccreditation decisions, the residency
review committee is charged with the difficult task of
approving initial accreditation and approvals of complement
Figure 5. General/Pediatric key indicator domain: resident surgeon numbers vs postgraduate year (PGY) level of training.
Jabbour and Tsue 77
increases. In the current system, information about the addi-
tion of trainees in each of these situations—a new program
or a change in the number of residents in an existing
program—is not available to the ACGME Residency
Review Committee until 6 years after the decision has been
made, as only graduate resident case numbers are available
for review. This time period is often beyond the tenure of
those making the initial decision, and if a deficiency is ulti-
mately noted, there has already been a missed opportunity
for remediation for several years of resident graduates. It
would be possible to monitor such changes in nearly real
time to assess the progress of trainees in programs that have
undergone these significant changes.
Prior to their use for high-stakes decision making, signif-
icant improvements could be made for future iterations of
these growth charts. Continued efforts should be made to
obtain the true median numbers at each PGY level and stan-
dard deviations at each PGY level to avoid the use of esti-
mated means. Also, it would be possible to extract data at
time intervals shorter than 1 year, such as quarterly or
monthly. This would allow for improved accuracy of the
curves when used for comparison in a semiannual or real-
time fashion. In the future, it may be possible to develop
growth curves that are integrated into a case log report
system, such that they may be automatically generated.
Many have argued that there is too much emphasis
placed on case numbers and that it is operative performance
that should receive increased attention. We would argue that
both are crucial during surgical training. Revisiting the
well-child visit analogy, there is value to knowing both the
quantitative measures of physical growth and the more qua-
litative assessment of the child’s developmental milestones.
Similarly, the case logs serve as a numeric marker of the
case exposure that is a necessary prerequisite for meeting
the functional goals of operative performance.
Conclusion
We have developed national case log growth curves for
operative procedures in otolaryngology. These growth
curves may be used as a standard against which to assess
resident case exposure progress longitudinally throughout
training.
Figure 6. Collapsed key indicator procedure domains: resident surgeon numbers vs postgraduate year (PGY) level of training.
78 Otolaryngology–Head and Neck Surgery 152(1)
Author Contributions
Noel Jabbour, design, acquisition of data, analysis, drafting of
article, final review and approval; Terance Tsue, design, analysis,
critical revisions, final review and approval.
Disclosures
Competing interests: Noel Jabbour was a member of the ACGME
Otolaryngology RRC from 2010 to 2012 and has been a member
of the ACGME Otolaryngology Milestones Working Group from
2011 to present. Terance Tsue has been a member of the ACGME
Otolaryngology RRC from 2010 to present and a chairperson of
the ACGME Milestones Working Group from 2011 to present.
Views expressed in the article are those of the authors and not
those of the ACGME.
Sponsorships: None.
Funding source: None.
References
1. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME
accreditation system—rational and benefits. N Engl J Med.
2012;366:1051-1056.
2. Kuczmarski RJ, Ogden CL, Guo SS, et al. 2000 CDC growth
charts for the United States: methods and development.
National Center for Health Statistics. Vital Health Stat. 2002;
11(246):1-190.
3. ACGME program requirements for graduate medical education
in otolaryngology: effective July 1, 2014. http://www.acgme.org/
acgmeweb/Portals/0/PFAssets/ProgramRequirements/280_ otolar-
yngology_07012014.pdf. Accessed July 23, 2014.
4. Rosenberg TL, Franzese CB. Extremes in otolaryngology resi-
dent surgical case numbers. Otolaryngol Head Neck Surg. 2012;
147:261-270.
Jabbour and Tsue 79