t2atelsrosfo

GEN

ERA

LT

HO

RA

CIC

Standardized Combined Outcome Index as anInstrument for Monitoring Performance AfterPulmonary ResectionAlessandro Brunelli, MD, Majed Refai, MD, Michele Salati, MD, Cecilia Pompili, MD,and Armando Sabbatini, MD

Division of Thoracic Surgery, Ospedali Riuniti Ancona, Ancona, Italy

Background. Modern healthcare systems demand moretransparent and accurate monitoring of clinical perfor-mance with the purpose to improve standards of care in acost-effective way. Outcomes, such as mortality, are still themost widely used quality indicators in our specialty. How-ever, previous studies have shown that mortality alone doesnot reflect performance accurately in our specialty. Ideally,multiple risk-adjusted outcomes should be used for a morecomprehensive assessment. The objective of this analysiswas to develop and use an index combining multiplerisk-adjusted outcomes to track down the performance ofour thoracic surgery unit over time.

Methods. In all, 511 major lung resections (465 lobec-omies, 46 pneumonectomies) performed from January005 through September 2010 were analyzed. Four risk-djusted outcomes were considered: 30 days or in-hospi-al mortality, cardiopulmonary morbidity, unplanned/mergency intensive care unit admission, and prolongedength of stay (more than 14 days, prolonged hospitaltay). Risk adjustment was performed using publishedegression models. Each indicator was converted into itspposite (ie, mortality rate to survival rate) so that highercores reflected better performance. Moreover, to accountor differences in measurement scales, the standardized

utcomes were rescaled according to their mean total

Ospedali Riuniti Ancona, Via Conca 71, Ancona 60124, Italy; e-mail:brunellialex@gmail.com.

© 2011 by The Society of Thoracic SurgeonsPublished by Elsevier Inc

standard deviations. Finally, the individual rescaled in-dicators of each year were summed to generate a com-bined outcome index.

Results. Mean cumulative observed mortality, morbid-ity, unplanned intensive care unit, and prolonged hospi-tal stay rates were 1.8%, 23%, 6.6%, and 7.4%, respec-tively. The combined outcome index scores showed aprogressive improvement of performance during thestudy period, progressing from �3.48 in 2005 to 2.87 in2009. The combined outcome index was also used pro-spectively in a variable life-adjusted display chart totrack down trends of practice variation in the last 6months.

Conclusions. The present analysis is proposed as amethodologic template for developing a risk-adjustedindex combining four different outcomes. It aims atovercoming inherent limitations of outcomes when usedindividually for performance assessment. This or similarcombined indexes may be effective instruments of inter-nal clinical audit and could be incorporated along withprocess indicators in composite performance scores tomore comprehensively evaluate the postoperative do-main of our practice.

(Ann Thorac Surg 2011;92:272–7)

© 2011 by The Society of Thoracic Surgeons

The modern healthcare system demands more mea-surement, more accountability, and more transpar-

ency, which translate in to a greater scrutiny of ourprofession. A continuous process of clinical practice im-provement should be realized systematically and withthe highest level of assurance in the selection of qualityindicators [1]. Quality of care can be defined as “thedegree to which health services for individuals andpopulations increase the likelihood of desired healthoutcomes and are consistent with current professionalknowledge” [2]. Although outcomes may not be the idealindicators, they represent the ultimate product of a

Accepted for publication March 11, 2011.

Presented at the Poster Session of the Forty-seventh Annual Meeting ofThe Society of Thoracic Surgeons, San Diego, CA, Jan 31–Feb 2, 2011.

Address correspondence to Dr Brunelli, Division of Thoracic Surgery,

process of care. For this reason, much of the literature onthis topic has focused on outcomes as quality endpoint.However, it is now recognized that most commonly usedoutcomes such as mortality, particularly when usedalone, do not always reliably reflect the level of careprovided to the patients [3, 4]. It is a fact that outcomemeasures are important on their own right and thatprocesses are of value only if they are linked to thedesired outcomes [5]. For this reason, outcomes such asmortality and morbidity have been included in compos-ite performance scores particularly to evaluate the post-operative temporal domain [6-8]. However, when theyare used to monitor performance, steps must be taken tominimize misleading information about quality of carederived from their interpretation. It has been recom-mended that multiple risk-adjusted outcome endpointsshould be used, as each of them may reflect different

aspects of performance [9]. In this regard, the objective of

0003-4975/$36.00doi:10.1016/j.athoracsur.2011.03.038

(

273Ann Thorac Surg BRUNELLI ET AL2011;92:272–7 COMBINED OUTCOME INDEX AND PULMONARY RESECTION

GEN

ERA

LT

HO

RA

CIC

this investigation was to develop and test a compositeoutcome index incorporating multiple risk-adjustedoutcomes.

Patients and Methods

This is an observational investigation performed on pro-spectively collected data in an electronic clinical databaseapproved by the local Institutional Review Board(#211098) after informed consent was obtained from allpatients to use their personal information and data forclinical and scientific use.

Five hundred eleven consecutive patients operated onfor major anatomic pulmonary resection (465 lobecto-mies, 46 pneumonectomies) from January 2005 throughSeptember 2010 were analyzed. All patients were oper-ated on by board-certified general thoracic surgeons andcared for after in a specialized general thoracic surgeryunit. As a rule, patients were extubated in the operatingroom and managed in a dedicated general thoracic sur-gery ward. Intensive care unit (ICU) admission wasreserved for major cardiopulmonary complications re-quiring assisted ventilation, cardiocirculatory inotropicsupport, and invasive monitoring.

The following outcome indicators were use to build thecomposite outcome index (COI): (1) 30-day or in-hospitalmortality; (2) 30-day or in-hospital cardiopulmonarymorbidity, defined according to the European Society ofThoracic Surgeons database definitions: respiratory fail-ure requiring assisted mechanical ventilation longer than24 hours, atelectasis requiring bronchoscopy, pneumonia(defined as meeting at least three of the followings: fevermore than 38°C, leukocytosis, chest roentgenograms withinfiltrates, positive culture from sputum), pulmonaryedema, pulmonary embolism confirmed by computedtomography scan, myocardial ischemia confirmed byelectrocardiogram and laboratory isoenzymes findings,cardiac failure, arrhythmia requiring medications or elec-trical cardioversion, cerebrovascular accident, acute renalinsufficiency confirmed by an increase of serum creati-nine level above 2 mg/dL or greater than twice thepreoperative level or new requirement of dialysis post-operatively; (3) emergency or unplanned admission toICU for major cardiopulmonary complications requiringactive life-supporting treatment [10]; and (4) prolongedlength of stay in hospital (PLS) defined as a stay longerthan 14 days according to published evidence (11).

All four indicators were risk-adjusted according topublished models:

(1) Mortality [12]: Logit: �6.97 � 0.095 � age �0.042 �ppoFEV1 (c-index: 0.77; Hosmer-Lemeshow statis-tic: p � 0.9)

(2) Cardiopulmonary morbidity [12]: Logit: �2.4 �0.03 � age �0.02 � ppoFEV1 � 0.6 � cardiaccomorbidity (coded as 1 and including CAD, anyprevious cardiac surgery, history and treatment forarrhythmia, congestive heart failure, hypertension[c-index: 0.65; Hosmer-Lemeshow statistic: p �

0.6])

3) Unplanned ICU admission [1]: Logit: �4.48 � 0.05 �age �0.03 � ppoFEV1% � 0.95 � pneumonectomy(coded as 1 for pneumonectomy and 0 for lobec-tomy [c-index: 0.72; Hosmer-Lemeshow statistic:p � 0.6])

(4) Prolonged hospital stay (recalibrated model fromWright and colleagues [11]): Logit: �3.776 � 0.038 �age �0.014 � ppoFEV1% � 0.433 � ECOG score[c-index: 0.71; Hosmer-Lemeshow statistic: p �0.3])

The first three models were originally developed byusing a population of more than 700 patients operated onin our unit and in another European unit from 2000through 2004 (no overlap with the population analyzed inthe present work). Logistic regression analysis and boot-strap resampling technique [13–15] were used to buildand validate those models. The last model is a recalibra-tion of the equation published by Wright and colleagues[11] predicting PLS. The equation was recalibrated in ourpopulation by using logistic regression and bootstrap toassess reliability of the predictors. Only significant andreliable factors (those resulting significant in more than50% of bootstrap samples) were included in the finalmodel used in the present study. The models were usedto build risk-adjusted outcome measures to track downthe performance of our unit during the last 6 years (2005to 2010).

Determination of Final Composite Outcome IndexRisk-adjusted morbidity, mortality, PLS, and unplannedICU admission rates were calculated for each year ofpractice by dividing the observed to the predicted out-come multiplied by the mean observed outcome rate inthe total population (2005 to 2010). Risk-adjusted out-come rates are regarded as the outcome rates a unitwould have in that period if its case mix would be similarto the average case mix in the entire population. The finalcomposite score combined the four risk-adjusted out-come measures into a single comprehensive qualityscore.

To assure consistent directionality (increasingly posi-tive values reflecting better performance), each outcomerate was converted in its counterpart (ie, mortality rateswere converted to survival rates: risk-adjusted survivalrate � 100 minus risk-adjusted mortality rate).

To account for differences in measurement scales, thescales of measurement were standardized by the recip-rocal of their standard deviations to obtain a rescaledscore (rescaled score � original score minus averagescore of the entire population divided by standard devi-ation of the entire population). This rescaling methodwas applied to all the outcome indicators before summa-rizing in the final composite score of each year.

The four risk-adjusted outcomes were used to con-struct variable life-adjusted display (VLAD) chart for theyear 2010. The VLAD is a form of risk-adjusted controlchart to monitor the trend in performance [16]. Patientsare ordered sequentially by the date of operation. Each

patient is assigned a score computed by the sum of the

Il

otramrItpopsm

274 BRUNELLI ET AL Ann Thorac SurgCOMBINED OUTCOME INDEX AND PULMONARY RESECTION 2011;92:272–7

GEN

ERA

LT

HO

RA

CIC

individual risk-adjusted outcomes (in case of absence ofnegative event) or risk-adjusted outcomes minus 1 (incase of presence of negative event). At each observation,the graph moves up or down of the total score for eachpatient. In this way, a positive trend is indicated by aconsistent upward direction of the curve.

All data were at least 95% complete. Missing datawere imputed by averaging the nonmissing values fornumerical variables and by taking the most frequentcategory for categorical variables. All the statisticaltests were two-tailed and a significance level of 0.05was accepted. The analysis was performed by using theSTATA 9.0 (Stata Corp, College Station, TX) statisticalsoftware.

Results

Table 1 shows the general characteristics of the patientsincluded in this study. The mean observed mortality,cardiopulmonary morbidity, unplanned admission, andprolonged hospital stay rates in the entire populationwere 1.8%, 23%, 6.6%, and 7.4%, respectively.

Table 1. Characteristics of Patients (n � 511) Included inStudy

CharacteristicMean (SD) or

n (%)

Age, years 67.4 (9.8)Male, n, % 398 (78)Body mass index 26.3 (4.5)FEV1% 84.4 (18.4)Dlco% 78.7 (19)American Society of Anesthesiologists score 2.1 (0.5)Eastern Cooperative Oncology Group score 0.6 (0.8)Coronary artery disease, n (%) 63 (12)Chronic obstructive pulmonary disease, n (%)a 135 (26)Smoking pack-years 42.5 (33.7)Predicted postoperative FEV1% 66.6 (15.3)Predicted postoperative Dlco% 62.4 (16.2)Induction chemotherapy, n (%) 63 (12)

Results are expressed as means (SD) unless otherwise indicated. a Forchronic obstructive pulmonary disease, FEV1 less than 80% and FEV1 toforced vital capacity ratio less than 0.7.

Dlco � diffusing capacity of lung for carbon monoxide; FEV1 �forced expiratory volume of air in 1 second.

Table 2. Average Risk-Adjusted Individual Outcomes by Year

PeriodsRisk-AdjustedSurvival (%)

Risk-Adjustedof Morbidit

2005 99.1 70.02006 99.1 70.12007 100.0 82.62008 98.9 77.32009 100.0 79.82010 99.4 80.9

ICU � intensive care unit.

Table 2 shows the risk-adjusted individual outcomemeasures for each year of activity. The average risk-adjusted mortality, cardiopulmonary morbidity, un-planned ICU admission, and PLS in the entire population(with their respective standard deviations) were 0.45%(0.6), 25% (6.9), 5.8% (4.2), and 2% (2.8). According to theirstandard deviations, a 1% increase in mortality wouldcorrespond to 11.5% increase in morbidity, 7% increasein unplanned ICU admission, and 4.7% increase in PLS.To account for these scale differences, the individualrisk-adjusted measures were rescaled according to theirstandard deviation before being summed in the COI.

The rescaled outcome scores with their respectivecomposite outcome indexes are reported in Table 3.Compared with the earlier period of activity, there wasan increased COI value in the more recent years, sug-gesting an improved outcome. The only two standard-ized indicators highly correlated (r � 0.82) were PLS andCU. All other indicators had a correlation coefficientower than 0.5.

Figure 1 depicts the VLAD chart of patients operatedn in 2010. The curve was generated using the cumula-ive difference between predicted and observed outcomeates and plotting the patients ordered by date of oper-tion. If the patient did not have the adverse event (ie,ortality), the patient is assigned a score equal to the

espective predicted outcome expressed in decimal form.n case of adverse event (ie, death), the score is equal tohe value of the predicted outcome minus 1. Thus, if aatient with a predicted mortality of 0.07 survived theperation, the mortality score would be 0.07, but if theatient died, the score would be �0.93. The Figure 1hows a consistent uninterrupted trend of good perfor-ance throughout the year 2010.

Comment

The use of composite measures of performance has beenrecently proposed in our field by both The Society ofThoracic Surgeons [6] and the European Society of Tho-racic Surgeons [7, 8] quality committees. The advantageof these composite indexes is to provide a single metricthat is easily interpretable and actionable by providers, inlight of the abstract nature of quality of care. In addition,composite indicators seem particularly useful for sum-marizing and comparing the quality of care delivered by

ctivity

nce Risk-AdjustedNo ICU (%)

Risk-Adjusted NoProlonged Stay (%)

93.1 93.191.2 96.090.4 96.493.1 98.097.2 100.098.0 95.0

of A

Absey (%)

tcrmito

vmbsppnmp

p

cbtlcfpemcaAt

dlaptcWtwrnoaarp

222222

275Ann Thorac Surg BRUNELLI ET AL2011;92:272–7 COMBINED OUTCOME INDEX AND PULMONARY RESECTION

GEN

ERA

LT

HO

RA

CIC

healthcare providers. That is particularly true in manysurgical areas where the small sample size and the lowrate of adverse events diminish the statistical utility ofindividual outcomes comparisons [17]. Thus, compositeindicators may provide a quantitative basis for clinicians,organizations, and planners aiming to achieve improve-ment in care and the process by which patient care isprovided.

One of the possible implementations of the compositescore proposed by The Society of Thoracic Surgeons taskforce for quality measurement in cardiac surgery [6] washe construction of separate composite scores for pro-esses and outcomes, with the aim to determine theirelative ability to discriminate overall performance. Sum-arizing the different outcomes in to a single standard-

zed indicator is appealing as it may represent a practicalool to assess with one reliable measure the entire post-perative domain.Hence, the objective of this investigation was to de-

elop a composite outcome index, including the fourost commonly studied outcomes in our setting (mor-

idity, mortality, ICU admission, and prolonged hospitaltay). The purpose was to provide a methodologic tem-late that could be reproduced in future analyses inde-endent of the selected outcomes. We also applied theewly developed score to evaluate the internal perfor-ance of our unit during a 5-year period to provide a

ractical example of its use.It is well accepted that to overcome one of the major

roblems of outcome measures (selection bias, different

Fig 1. Variable life-adjusted display (VLAD)of patients (pts) operated on in 2010.

Table 3. Rescaled Outcome Scores and Corresponding Combin

Periods Rescaled Survival (%)Rescaled Absenceof Morbidity (%)

005 �0.75 �0.72006 �0.75 �0.70007 0.75 1.11008 �1.08 0.34009 0.75 0.70010 �0.25 0.86

ICU � intensive care unit.

ase mix), they need to be risk adjusted before they cane used as instrument of audit. All endpoints included in

he present score were risk-adjusted according to pub-ished models. After summing all rescaled indicators, aomposite score was assigned to each year of activity. Weound an increase of the COI value in the most recenteriod, likely reflecting an improvement in the postop-rative domain level of care. The use of a COI appearsore comprehensive compared with using single out-

omes. For instance, by using risk-adjusted mortalitylone, 2007 would have resulted the best year of activity.ccording to the combined index, 2007 resulted instead

hird in the rank.We also used the four risk-adjusted outcomes to track

own graphically the internal performance during theast year of activity by using VLAD charts [16]. Thispproach provides a visual representation of trends inerformance when the patients are ordered according to

heir date of operation. The VLAD is a form of controlhart meant at monitoring selected clinical indicators.

e used the sum of the four standardized outcomes asarget indicator, assuming that this composite scoreould be more reliable and representative than single

isk-adjusted outcomes taken individually. The VLAD isot a test of statistical significance but a graphical displayf sequential outcome trends that has the theoreticaldvantage to be more reactive to changes than traditionalggregate statistical analysis, which requires longer pe-iods to aggregate the necessary number of events toerform reliable statistics [18, 19]. The control chart is

utcome Index by Year of Activity

escaled NoICU (%)

Rescaled NoProlonged Stay (%)

Combined OutcomeIndex

�0.26 �1.75 �3.48�0.71 �0.71 �2.87�0.90 �0.57 0.39�0.26 0.00 �1.00

0.71 0.71 2.870.90 �1.07 0.44

ed O

R

brtrFcm

smdiwaepm

migaaobrrd

276 BRUNELLI ET AL Ann Thorac SurgCOMBINED OUTCOME INDEX AND PULMONARY RESECTION 2011;92:272–7

GEN

ERA

LT

HO

RA

CIC

reported here only with the purpose to show a possibleuse of composite risk-adjusted endpoints, and the dis-cussion of advantages (ie, intuitiveness) and limitations(ie, difficulty in defining thresholds) of this method isbeyond the scope of this work.

Outcomes are the most frequently used quality indica-tors in our specialty. They have advantages and draw-backs [4]. Among the advantages is that they reflect allaspects of the entire process of care, including those thatare not measurable. Their use as quality indicators pro-mote “whole system” collaboration between differentparts of the healthcare system and nurture innovation, asclinicians are encouraged to pursue technologies that willenhance outcome. Outcome measures are usually ge-neric and less specific to the technology used thanprocess measures. As a consequence, once implementedthey are likely to require only periodic refinement ratherthan wholesale review. Some aspects of outcome mea-sures, most notably mortality, are relatively immune tomanipulation by providers.

Conversely, one of the major limitations of outcomemeasures is that they are not a direct measure of qualityand do not provide immediate feedback to providersabout how to improve what they are doing. Variations inoutcome may depend on multiple factors including dif-ferences in case mix, differences in ascertainment anddefinition of indicators and risk factors, chance, and truedifferences in quality of care. The development and useof a standardized COI may overcome some of theselimitations inherent to outcome measures when usedindividually. Each outcome may reflect a different aspectof performance [9]. As a consequence, whenever possi-

le, multiple outcomes should be used to have a moreeliable audit. The incorporation of multiple outcomes ino a single index provides an indicator, which in thisegard appears more reliable than individual outcomes.inally, a COI may be in turn included in to a moreomprehensive performance score including also processeasures [4, 6, 7].This study may have potential limitations. Although

ome outcomes are by their own nature objective (ie,ortality), some others may have inherent problems of

efinitions and recording (ie, morbidity). Even though thiss an observational analysis on prospectively collected data,hich have been defined a priori, variations in recording

nd subjective interpretation may always occur over timeven in the same unit. The data for this study have beenrospectively collected by a single trained physician dataanager to minimize this problem.Risk adjustment has been performed using publishedodels. It is well known that a model performs less well

n a population other than the one in which it has beenenerated [20]. Nevertheless, the mortality, morbidity,nd ICU models used for this study were developed frompopulation comprising in part patients operated on in

ur unit in a previous period and internally validated byootstrap resampling technique. This warrants a goodeproducibility of the model [13–15]. The PLS model wasecalibrated in the present series to assure satisfactory

iscrimination and calibration [20].

Although more reliable and comprehensive than indi-vidual outcome indicators, the COI assesses only thepostoperative domain. Other indicators, preferably pro-cesses of care, should be used to evaluate the preopera-tive and intraoperative phases [6]. The COI (like alloutcome endpoints) does not provide per se immediatelyactionable feedback to providers, but should trigger morein-depth analyses to ascertain the nature and origin ofthe problem. A careful review of pathways of care andcritical processes should be undertaken to verify thesource of underperformance and undertake correctivemeasures.

The feasibility and reliability of using the COI formulticenter comparative audit should be verified byindependent investigations. Future studies are needed toassess the degree of congruence between compositeprocesses and outcomes indicators and their relativeability to discriminate overall performance; to test new orrevised measures that provide important incrementalinformation about overall care; and to continuously as-sess the relationship between process measures andshort- or long-term outcomes, with potential eliminationof process measures that demonstrate limited clinicaleffectiveness.

In conclusion, we were able to develop and test astandardized COI that could be used for performancemonitoring and quality initiatives. The present analysis isproposed as a methodological template for developing arisk-adjusted index combining four different outcomes. Itaims at overcoming inherent limitations of outcomeswhen used individually for performance assessment.This or similar combined indexes may be effective instru-ments of internal clinical audit and could be incorporatedalong with process indicators in composite performancescores to more comprehensively evaluate the postopera-tive domain of our practice.

References

1. Brunelli A, Rocco G. The comparison of performance betweenthoracic surgical units. Thorac Surg Clin 2007;17:413–24.

2. Lohr KN, Schroeder SA. A strategy for quality assurance inMedicare. N Engl J Med 1990;322:1161–71.

3. Mant J, Hicks NR. Assessing quality of care: what are theimplications of the potential lack of sensitivity of outcomemeasures to differences in quality? J Eval Clin Pract 1996;2:243–8.

4. Brunelli A, Rocco G. Clinical and nonclinical indicators ofperformance in thoracic surgery. Thorac Surg Clin 2007;17:369–77.

5. Davies HT, Crombie IK. Assessing the quality of care. BrMed J 1995;311:766.

6. O’Brien SM, Shahian DM, DeLong ER, et al, for The Societyof Thoracic Surgeons Task Force. Report of the STS QualityMeasurement Task Force. Quality measurement in adultcardiac surgery: part 2. Statistical considerations in compos-ite measure scoring and provider rating. Ann Thorac Surg2007;83(Suppl):13–26.

7. Brunelli A, Berrisford RG, Rocco G, Varela G, for theEuropean Society of Thoracic Surgeons Database Commit-tee. The European Thoracic Database project: compositeperformance score to measure quality of care after major

lung resection. Eur J Cardiothorac Surg 2009;35:769–74.

277Ann Thorac Surg BRUNELLI ET AL2011;92:272–7 COMBINED OUTCOME INDEX AND PULMONARY RESECTION

ERA

LT

HO

RA

CIC

8. Brunelli A, Rocco G, Van Raemdonck D, Varela G, Dahan M.Lessons learned from the European thoracic surgery data-base: the composite performance score. Eur J Surg Oncol2010;36(Suppl 1):93–9.

9. Silber JH, Rosenbaum PR, Schwartz JS, Ross RN, WilliamsSV. Evaluation of the complication rate as a measure ofquality of care in coronary artery bypass graft surgery.JAMA 1995;274:317–23.

10. Zimmerman JE, Wagner DP, Knaus WA, Williams JF, Kola-kowski D, Draper EA. The use of risk predictions to identifycandidates for intermediate care units. Implications for in-tensive care utilization and cost. Chest 1995;108:490–9.

11. Wright CD, Gaissert HA, Grab JD, O’Brien SM, Peterson ED,Allen MS. Predictors of prolonged length of stay afterlobectomy for lung cancer: a Society of Thoracic Surgeonsgeneral thoracic surgery database risk-adjustment model.Ann Thorac Surg 2008;85:1857–65.

12. Brunelli A, Morgan-Hughes NJ, Refai M, Salati M, SabbatiniA, Rocco G. Risk-adjusted morbidity and mortality modelsto compare the performance of two units after major lungresections. J Thorac Cardiovasc Surg 2007;133:88–96.

13. Blackstone EH. Breaking down barriers: helpful break-

through statistical methods you need to understand better.J Thorac Cardiovasc Surg 2001;122:430–9.

to analyses wherein they and others study real world,

© 2011 by The Society of Thoracic SurgeonsPublished by Elsevier Inc

14. Grunkemeier GL, Wu YX. Bootstrap resampling method:something for nothing? Ann Thorac Surg 2004;77:1142–4.

15. Brunelli A, Rocco G. Internal validation of risk models inlung resection surgery: bootstrap versus training and testsampling. J Thorac Cardiovasc Surg 2006;131:1243–7.

16. Lovegrove J, Valencia O, Treasure T, et al. Monitoring theresults of cardiac surgery by variable life adjusted display.Lancet 1997;18:1128–30.

17. Dimick JB, Welch HG, Birkmeyer JD. Surgical mortality asan indicator of hospital quality: the problem with smallsample size. JAMA 2004;292:847–51.

18. Poloniecki J, Valencia O, Littlejohns P. Cumulative risk-adjusted mortality chart for detecting changes in death rate:observational study of heart surgery. Br Med J 1998;316:1697–700.

19. Rogers CA, Reeves BC, Caputo M, Ganesh JS, Bonser RS,Angelini GD. Control chart methods for monitoring cardiacsurgical performance and their interpretation. J Thorac Car-diovasc Surg 2004;128:811–9.

20. Ivanov J, Tu JV, Naylor CD. Ready-made, recalibrated orremodeled? Issues in the use of risk indexes for assessing

mortality after coronary artery bypass graft surgery. Circu-lation 1999;99:2098–104. G

EN

INVITED COMMENTARY

In their article Dr Brunelli and colleagues [1] propose amethodology to design procedure-specific quality evalu-ation for patients undergoing pulmonary resection. Theauthors are to be commended for a thorough and logicalderivation of their combined outcome index (COI) score.This measure involves risk-adjusted outcomes that areimportant and objective in pulmonary surgery, and therisk-adjustment methodology they have used reflectsthe applicability of the extrinsic risk models in theirparticular population. In short, in designing their COIscore, the authors have assembled what may ultimatelyrepresent an important quality measure in thoracicsurgery.

But what comes next? Should this measure be widelyadopted by thoracic surgery? Not just yet. Although theCOI seems well designed, the authors have yet to presentany evidence that the COI score is (1) directly related toquality or (2) any better than measuring any of theindividual components in isolation. In other words, thisnew combined outcome measure needs validation, bothinternally within their care setting and in other externalsettings.

Furthermore, if the COI score is to assume a meaning-ful role in the European Society of Thoracic Surgeons orThe Society of Thoracic Surgeons, one would want to besure it is worth all of the extra statistical “work.” Giventhe authors’ expertise in this area, we will look forward

widely representative datasets, and provide comparisonsof each of the individual outcomes, and the COI score,and other prominent quality measures that are measuredin the The Society of Thoracic Surgeons and EuropeanSociety of Thoracic Surgeons. Studies such as these willsupply objective evidence of the potential advantages ordisadvantages of this score, as compared with moresimple, univariate quality indicators, and this will help todetermine which quality measures, simple or complex,work best.

Philip Goodney, MD

Department of SurgeryThe Dartmouth Institute for Health Policy

and Clinical PracticeVA Outcomes Group (111B)VA Medical Center215 North Main StWhite River Junction, VT 05009e-mail: philip.goodney@hitchcock.org

Reference

1. Brunelli A, Refai M, Salati M, Pompili C, Sabbatini A. Stan-dardized combined outcome index as an instrument formonitoring performance after pulmonary resection. Ann

Thorac Surg 2011;92:272–7.

0003-4975/$36.00doi:10.1016/j.athoracsur.2011.03.106