O

ER(

C

DT

BoTcpetdf

Mrrmabtqf

Rie

CtfifiSA

Kt

CAk

nD

3

RIGINAL REPORTS

stablishing Learning Curves for Surgicalesidents Using Cumulative SummationCUSUM) Analysis

PT Amy Young, MC, USA,* LTC Joseph P. Miller, MC, USA,† and COL Kenneth Azarow, MC, USA*

epartments of *Surgery and †Anesthesia/Operative Services, Madigan Army Medical Center,

acoma, Washington

B

TppcABaeoo

esoiqttdascrenadtpra

etT

ACKGROUND: The assessment of technical proficiency isf paramount importance in the training of surgical residents.he fact that technical proficiency is underrepresented in the

ontext of the ACGME outcomes project is evidenced in thatroficiency skills comprise less than 5% of all assessments thatvaluate residents. In this study, we use Cumulative Summa-ion Analysis (CUSUM) as a visual objective analytic tool toetermine performance accuracy and establish learning curvesor PGY-1s in surgery.

ETHODS: From April 2001 to May 2002, 11 surgicalesidents completed a 1-month anesthesia rotation. Eachesident was asked to complete a preoperative airway assess-ent followed by endotracheal intubation with induction of

nesthesia. Airway assessment was performed independentlyy a resident and a licensed anesthesiologist or certified anes-hetist with the modified Mallampati Score. Data were se-uentially collected and plotted for summated successes andailures.

ESULTS: The average intern required approximately 19ntubation attempts to complete the learning curve experi-nce. There was no learning curve for airway assessment.

ONCLUSIONS: The CUSUM analysis is an effective objec-ive tool to define learning curves for technical skills. Vital in-ormation is provided for surgical programs that place residentsn positions to manage airways, and limitless potential for de-ning the learning curves for technical skills is provided. (Currurg 62:330-334. Published by Elsevier Inc. on behalf of thessociation of Program Directors in Surgery.)

EY WORDS: cumulative summation analysis, CUSUM, in-ubation, learning curve, GME, surgical education

orrespondence: Inquiries to Kenneth S. Azarow, MD, Department of Surgery, Madiganrmy Medical Center, MCHJ-SGY, Tacoma WA 98431; fax: (253) 968-0232; e-mail:enneth.azarow@us.army.mil

The opinions and assertions contained herein are the private views of the authors and are

bot to be construed as the official policy or position of the United States Government, theepartment of Defense, or the Department of the Army.

CURRENT SURGERY • Published by Elsevier Inc. on behalf of theAssociation of Program Directors in Surgery

30

ACKGROUND

he evaluation of technical proficiency is a difficult and com-lex task. Historically, evaluating surgical residents has beenrimarily based on subjective criteria. These criteria have beenhallenged by the ACGME, a challenge that resulted in theCGME’s outcomes project. Tests, such as the Americanoard of Surgery In Training and Basic Science Examinationlong with oral examinations commonly assess factual knowl-dge and reasoning ability, even within the structure of theutcomes-based project. However, for technical proficiency, nobjective measuring stick exists.

Control charts are statistical tools originally developed byngineers to test the efficiency of machinery and mechanicalystems. Cumulative Summation Analysis (CUSUM) is a typef control chart that has recently gained acceptance in the med-cal field.1 The basic premise of the analysis is to plot the se-uential difference of a set of measured values and to define aarget level for those values.2 Thus, the analysis can determinehe overall proficiency at achieving success of a given task. Byoing so, it is believed that tighter control over deviation fromstandard can be achieved. In addition, if we postulate that all

ubjects have the same baseline experience, we can use thehanging success rates as a learning curve until a steady successate is achieved. To achieve an assumption of equal baselinexperience, interns or first-year residents (PGY-1s) become aatural source of a standardized pool of participants. Finally, asll results are summated, the data from all participants canevelop a learning curve for the average of all participants. In-erpretations can be made for a population rather than for eacherson alone, which provides the potential application for cur-icula development, rotational experiences, mentor evaluations,nd credentialing experiences.

The aim of this study is to identify whether CUSUM canstablish a defined number of procedural attempts necessary forhe average resident to achieve proficiency of a particular task.he skills of (1) airway assessment and (2) endotracheal intu-

ation were chosen as the model for this analysis.

0149-7944/05/$30.00doi:10.1016/j.cursur.2004.09.016

M

FA1pamtppabitetss

rcwspplptgsclPs“cwotsetdH

tpapsdota

iswti

H

R

FaNcsams2giPbal(rt

D

OpppapsteudslalepgolTo

C

ETHODS

rom April 2001 to May 2002, 11 surgical PGY-1s at Madiganrmy Medical Center (Ft Lewis, Washington) completed a-month anesthesia rotation. Each resident received the samerestudy training, including ATLS and a didactic session bynesthesia on the first day of the rotation. Thus, this studyakes the assumption that all participating interns start with

he same baseline intubation experience. All PGY-1s had com-leted ACLS before beginning the academic year, had com-leted ATLS during the month before this anesthesia rotation,nd underwent a half-day of didactic instruction on intubationefore the initiation of their measured experience. To standard-ze the patient population, eligibility criteria of patients werehose aged 18 years and older and ASA class I or II, undergoinglective surgery under general anesthesia. In addition, any pa-ient whose airway assessment received a modified Mallampaticore of 3 or 4 was excluded from the intubation part of thetudy (see below).

For both tasks (assessment and intubation), target successates were set at 95%. These success rates were determined viaonsensus of the anesthesia department at our institution. Air-ay assessment was determined with the modified Mallampati

core (score corresponds to grade for this study).3 In a supineosition, with neck flexed and not extended or neutral, theatient protrudes his/her tongue while phonating. By simply

ooking in the pharynx, the PGY-1 determines the visibility ofharyngeal structures and assigns a score based on exposure ofhe glottis. Grade I � glottis could be fully exposed. Grade II �lottis could be partially exposed (anterior commissure not vi-ualized). Grade III � glottis could not be exposed (corniculateartilages only visible). Grade IV � glottis including cornicu-ate cartilages could not be exposed. The score assigned by theGY-1 was then recorded and compared with a staff (anesthe-iologist or chief nurse anesthetist) independent assessment. Asuccessful” attempt was one in which the PGY-1’s score wasoncordant with that of the anesthesia staff. As long as the scoreas graded as 1 or 2, the patient was then moved into theperating suite and placed under general anesthesia. The internhen made an attempt at endotracheal intubation under directupervision of a credentialed anesthesia provider. A “successful”ndotracheal intubation was defined as one in which the endo-racheal tube is in place with cuff inflated within 30 seconds ofirect laryngoscopy and confirmed by end tidal CO2 of 30 mmg for 3 breaths.The CUSUM equation is defined as � � Xi–Xo, where � is

he cumulative sum, Xi is an individual attempt, and Xo is theredetermined failure rate inherent for the procedure. Xi isssigned a score of 0 for a success and 1 for a failure. As aremise, every procedure has an inherent failure rate. The targetuccess rate for intubation was set at 95% by our anesthesiaepartment. Thus, the inherent failure rate was defined at 5%r 0.05. The score, after each attempt, was sequentially added tohe cumulative score and plotted on a graph. Graphs were then

nalyzed on the basis of their slope. A positive, up-going slope

URRENT SURGERY • Volume 62/Number 3 • May/June 2005

ndicates a series of failures, whereas a negative, down-goinglope indicates a series of successes.1,2 The portion of the curvehere the maximal change in slope begins to decrease signifies

he end of the initial learning process (ie, “getting off the learn-ng curve”).

This study was approved by the Institutional Review Board anduman Use Committees at Madigan army Medical Center.

ESULTS

igure 1 demonstrates individual intern successes plotted versusttempts. Thus, a CUSUM analysis is done for each resident.ote the wide variability in the shapes and slopes of these

urves. When this data are summated (ie, all first attempts areummed and then added to all second attempts, and subsequentttempts), Table 1 is formed. This table demonstrates the cu-ulative summation data for all residents at each of the tested

kills. The cumulative summated data are demonstrated in Figs.and 3. These curves represent the success or failure of the totalroup. Thus, the point at which the change of slope decreasesndicates the number of procedures necessary for the averageGY-1 to get off of the learning curve. For endotracheal intu-ation, this point was estimated at 19 attempts. For airwayssessment, the slope never changed; thus we deduce that noearning curve exists for this skill. Thus, despite a positive slopeie, a success rate that falls below the defined acceptable rate),esidents were as good as they were going to get at the outset forhis standardized patient.

ISCUSSION

ver the last 15 years, the CUSUM method has been used byhysicians in various medical disciplines to assess trends androficiencies. It has assessed antimicrobial treatment in neutro-enic patients by plotting temperature curves.4 Schlup et alnalyzed the technical proficiency of a single endoscopist inerforming ERCP. A 90% target success rate was achieved forelective cannulation after 100 procedures and 120 interven-ions.5 McCarter et al examined the learning curves for FASTxaminations performed by 5 trauma surgeons, as both individ-al persons and as an institution. Graphs were plotted for 3ifferent success rates, namely, 85%, 90%, and 95%. Theseubjects achieved accuracy from the outset, that is, without aearning curve.6 The following 2 studies are the only studiespplying CUSUM to assess surgical residents. Van Rij et alooked at 17 surgical residents. They documented that 25 op-rations were needed before acceptable speed was reached inerforming appendectomies, open cholecystectomies, and in-uinal hernia repairs.7 Molloy et al studied proficiency at intra-perative cholangiography performed during laparoscopic cho-ecystectomies. A 95% success rate was achieved after 46 cases.wenty-four and 16 cases were required to attain success ratesf 90% and 85%, respectively.8

Learning curves were constructed by de Oliveira and for

331

ae8rfp

seikpputhwirfer

ivfitiicd

gttctattdpaF

s and f

3

nesthesia residents in basic anesthesia procedures, includingndotracheal intubation.9 This study sets a lower success rate of0% but holds to a more stringent set of criteria by not allowingepeat attempts at intubation. In our study, we not only allowedor repeated attempts at intubation but also used a standardizedool of patients selected for less complex airways.Establishing a learning curve for residents to accomplish the

kill of endotracheal intubation has accomplished 3 importantndpoints. First, it has allowed us to define the minimum train-ng experience necessary to succeed at this skill. Armed with thenowledge that at least 19 attempts are necessary before nearingroficiency, we have increased the caseload volume to accom-lish this on a consistent basis. We feel this information is alsoseful when arranging the order of clinical rotations. As bothrauma and critical care rotations require intubation skills, weave scheduled them to follow the anesthesia month. Second,e now have an objective method for measuring technical abil-

ty. With the establishment of an average learning curve, futureesidents can be objectively evaluated based on their ability toollow this curve. Finally, this method will allow for potentialarly intervention when skills are lower than average. If a given

1

-1

1

3

5

1 3 5 7 9 11 13 15 17 19 21 23 25 27

# attempts

CUSUM

2

-1

1

3

5

1 3 5 7 9 11 13 15 17 19 21 23 25 27

attempts

CUSUM

3

-1

4

9

1 4 7 10 13 16 19 22 25 28 31 34 37

attempts

CUSUM

4

-1

1

3

5

7

1 3 5 7 9 11 13 15 17 19 21 23 25 27

attempts

cusum

5

0

1

2

3

4

1 3 5 7 9 11 13

attempts

Cusum

-2

0

2

4

6

1 3 5 7 9 11

a

CUSUM

-1

-0.5

0

0.5

1

1.5

2

1 3 5 7 9

CUSUM

0

5

10

1 4 7 10 13

attempts

CUSUM

FIGURE 1. Cumulative summation of successe

esident shows a persistently positively steep slope, and can be e

32

dentified early in the process, extra mentorship could be pro-ided. This point is demonstrated by intern #8 in Fig. 1. Thisrst-year resident was the only person not able to establish arend at which the learning process could be identified. Retrain-ng or a change in the training process for this task would bendicated for this person. The dynamic nature of this analysisan allow for early intervention and success, which avoids aeclaration of failure at the end of the training period.Regarding the skill of airway assessment, residents displayed

reat variability in individual proficiency and as a group failedo achieve the target success rate of 95%. Some residents mas-ered the skill at the first attempt, whereas others displayed aontinuous pattern of successes and failures. The conclusionhat we must draw is that this skill requires no learning toccomplish and that degree of success or failure is fixed ratherhan alterable based on experience. This conclusion is based onhe assumption that airway assessment is a measurable skill withefined parameters. An alternative conclusion for the observedattern is that the criteria necessary to perform a CUSUMnalysis were not met and thus the conclusion is not correct.actors that would make the CUSUM invalid would be inad-

21

17 19 21 23 25

17 19 21 23 25

9

-1

0

1

2

3

4

5

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39

Attempts

Cusum

10

0

2

4

6

8

10

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

Attempts

CUSUM

11

0

0.5

1

1.5

2

2.5

3

3.5

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

A t t e mpt s

ailures versus attempts for each intern enrolled.

15 17 19

6

13 15

ttempts

7

11 13 15

attempts

8

16 19 22

quate prestudy training such that the residents did not have

CURRENT SURGERY • Volume 62/Number 3 • May/June 2005

elac

duca

mgtdrm

R

1

2

3

4

5

6

7

TA

Ftw1

Ftr

C

qual skill levels at the onset. Perhaps a more extensive initialecture on the modified Mallampati Score would be helpful. Inddition, a standard quiz at completion of prestudy trainingould be administered to establish each resident’s cognitive un-

ABLE I. Cumulative Summation Scores for All Residents at Eachttempt at Intubation and Airway Assessment.

Intubation

Attempt

Assessment

SUM CUSUM SUM CUSUM

6.45 6.45 1 0.45 0.456.45 12.90 2 0.45 0.902.45 15.35 3 0.45 1.352.45 17.80 4 0.45 1.801.45 19.25 5 0.55 2.354.45 23.70 6 0.45 2.801.45 25.15 7 �0.55 2.252.45 27.60 8 0.45 2.701.45 29.05 9 0.45 3.104.45 33.50 10 2.55 5.70

�0.55 32.95 11 �0.45 5.251.45 34.4 12 �0.55 4.70

�0.55 33.85 13 �0.55 4.151.45 35.30 14 1.45 5.600.45 35.75 15 0.55 6.150.45 36.20 16 1.45 7.601.45 37.65 17 0.45 8.053.45 41.10 18 0.45 8.501.45 42.55 19 1.45 9.950.45 43.00 20 1.65 11.602.50 45.50 21 0.50 12.100.55 46.05 22 0.55 12.65

�0.45 45.60 23 �0.45 12.20�0.45 45.15 24 1.55 13.75

0.60 45.75 25 �0.30 13.45�0.40 45.35 26 0.60 14.05�0.30 45.05 27 �0.30 13.75

Intubation

-1

9

19

29

39

49

1 6 11 16 21 26

attempts

CUSUM

IGURE 2. Cumulative summation (y-axis) for all residents plotted againsthe number of attempts (x-axis) at endotracheal intubation. The point at

hich the average intern comes off of the learning curve is approximately9.

URRENT SURGERY • Volume 62/Number 3 • May/June 2005

erstanding of this skill. Another explanation is that this eval-ation is a subjective judgment rather than a measured techni-al skill. Thus, it would not be subject to the same assumptionsnd objective determinations of success and failure.

This study has established CUSUM analysis as an objectiveethod of evaluating the technical proficiency of first-year sur-

ical residents at our institution. We anticipate using it in mul-iple other procedures and the overall evaluation of our resi-ents at all levels. We have already begun to alter scheduleotations, completely change rotational curricula, and initiateentoring programs based on this evaluation tool.

EFERENCES

. Ravin L. The CUSUM score. A tool for evaluation of clinicalcompetence. Ugeskrift For Laeger. 2001;163:3644-3648.

. Goldsmith ODaP. Statistical Methods in Research and Pro-duction. London: Longman; 1976.

. Constantikes J. Predicting difficult tracheal intubation us-ing a modified Mallampati sign: a pilot study report. CRNA.1993;4:16-20.

. Kinsey SE Giles FJ, Holton J. Cusum plotting of tempera-ture curves for assessing antimicrobial treatment in neutro-penic patients. BMJ. 1989;299:775-776.

. Schlup MM, Williams SM, Barbezat GO. ERCP: a reviewof technical competency and workload in a small unit. Gas-trointest Endosc. 1997;46:48-52.

. McCarter F, Luchette FA, Molloy M, et al. Institutionallearning curves for focused abdominal untrasound fortrauma, cumulative summation analysis. Ann Surg. 2000;231:689-700.

. Van Rij AM, McDonald JR, Pettigrew R, Petterill M,Reddy C, Wright J. Cusum as an aid to early assessment of

Assessment

-1

3

7

11

15

19

23

1 5 9 13 17 21 25

attempts

CUSUM

IGURE 3. Cumulative summation (y-axis) for all residents plotted againsthe number of attempts (x-axis) and airway assessment. As the slope waselatively constant, no learning curve was demonstrated.

the surgical trainee. Brit J Surg. 1995;82:1500-1503.

333

8 9

3

. Molloy M, Bower RH, Hasselgren P, Dalton B. Cholan-giography during laparoscopic cholecystectomy. Cumula-tive summation analysis of an instituional learning curve. J

Gastrointest Surg. 1999;3:185-188.

34

. de Oliveira Filho GR. The construction of learning curvesfor basic skills in anesthetic procedures: an application forthe cumulative summation analysis. Anesth Analg. 2002;95:

411-416.

CURRENT SURGERY • Volume 62/Number 3 • May/June 2005