Accepte

ReprintDepartmE. DeBaHouston

Differentiation of ileostomy fromcolostomy procedures: Assessingthe accuracy of current proceduralterminology codes and the utilityof natural language processingElaine Vo, BS,a Jessica A. Davila, PhD,b Jason Hou, MD,b,c Krystle Hodge, MPH,a Linda T. Li, MD,a

James W. Suliburk, MD,a,d Lillian S. Kao, MD, MS,e David H. Berger, MD, MHMC,a,b andMike K. Liang, MD,a Houston, TX

Background. Large databases provide a wealth of information for researchers, but identifying patientcohorts often relies on the use of current procedural terminology (CPT) codes. In particular, studies of stomasurgery have been limited by the accuracy of CPTcodes in identifying anddifferentiating ileostomy proceduresfrom colostomy procedures. It is important to make this distinction because the prevalence of complicationsassociated with stoma formation and reversal differ dramatically between types of stoma. Natural languageprocessing (NLP) is a process that allows text-based searching. The Automated Retrieval Console is an NLP-based software that allows investigators to design and performNLP-assisted document classification. In thisstudy, we evaluated the role of CPT codes and NLP in differentiating ileostomy from colostomy procedures.Methods. Using CPT codes, we conducted a retrospective study that identified all patients undergoing astoma-related procedure at a single institution between January 2005 and December 2011. All operativereports during this time were reviewed manually to abstract the following variables: formation or reversaland ileostomy or colostomy. Sensitivity and specificity for validation of the CPT codes against the masterysurgery schedule were calculated. Operative reports were evaluated by use of NLP to differentiate ileostomy-from colostomy-related procedures. Sensitivity and specificity for identifying patients with ileostomy orcolostomy procedures were calculated for CPT codes and NLP for the entire cohort.Results. CPT codes performed well in identifying stoma procedures (sensitivity 87.4%, specificity97.5%). A total of 664 stoma procedures were identified by CPTcodes between 2005 and 2011. The CPTcodes were adequate in identifying stoma formation (sensitivity 97.7%, specificity 72.4%) and stomareversal (sensitivity 74.1%, specificity 98.7%), but they were inadequate in identifying ileostomy(sensitivity 35.0%, specificity 88.1%) and colostomy (75.2% and 80.9%). NLP performed with greatersensitivity, specificity, and accuracy than CPT codes in identifying stoma procedures and stoma types.Major differences where NLP outperformed CPT included identifying ileostomy (specificity 95.8%,sensitivity 88.3%, and accuracy 91.5%) and colostomy (97.6%, 90.5%, and 92.8%, respectively).Conclusion. CPT codes can identify effectively patients who have had stoma procedures and are adequatein distinguishing between formation and reversal; however, CPT codes cannot differentiate ileostomy fromcolostomy. NLP can be used to differentiate between ileostomy- and colostomy-related procedures. The roleof NLP in conjunction with electronic medical records in data retrieval warrants further investigation.(Surgery 2013;154:411-7.)

From the Michael E. DeBakey Department of Surgery,a Michael E. DeBakey VA Medical Center, Baylor Collegeof Medicine, Houston, TX; the Houston VA Health Services Research & Development Center of Excellence,b

Michael E. DeBakey VA Medical Center, Baylor College of Medicine, Houston, TX; the Department ofGastroenterology,c Michael E. DeBakey VA Medical Center, Baylor College of Medicine, Houston, TX; theMichael E. DeBakey Department of Surgery,d Ben Taub General Hospital, Baylor College of Medicine,Houston, TX; and the Department of Surgery,e University of Texas Health Sciences Center at Houston,Houston, TX

d for publication May 10, 2013.

requests: Mike K. Liang, MD, Michael E. DeBakeyent of Surgery, Baylor College of Medicine, Michaelkey VA Medical Center, 2002 Holcombe Boulevard,, TX 77030. E-mail: ml3@bcm.edu.

0039-6060/$ - see front matter

Published by Mosby, Inc.

http://dx.doi.org/10.1016/j.surg.2013.05.022

SURGERY 411

SurgeryAugust 2013

412 Vo et al

STOMAS SERVE as an important tool to help divertstool from distal pathology, to relieve a distalobstruction, to protect a high-risk anastomosis, orto avoid creating a high-risk anastomosis. Often,the creation of a stoma and its subsequent reversalis considered to be relatively benign and toproduce few complications.1,2

Stoma formations and reversals are associatedwith substantial morbidity.3 Patients commonlysuffer psychologically, socially, and physicallyfrom having a stoma and are eager to undergoreversal. Stoma formation is associated with highoutput complications (dehydration, electrolyteimbalance, renal failure), dehiscence, prolapse,retraction, hernia formation, and obstruction. Inaddition, it is common that stomas intended tobe temporary are never reversed. Stoma reversalis associated with incisional surgical-site infection,anastomotic leak, need for another stoma, andhernia formation.1-3

The prevalence of complications differs dramat-ically by stoma type. Patients with ileostomies haveincreased risk for high-output stomas, whereaspatients with colostomies have an increased riskfor prolapse, retraction, and surgical site infec-tion.1-4 When undergoing reversal, patients withileostomies are more likely to have trans-stomalreversals, whereas patients with colostomies aremore likely to have laparoscopic or open rever-sals.4 Differentiating ileostomy from colostomyhas important implications for measuring and forimproving the quality of care, as well as forresearch purposes.

The vast majority of retrospective studies evalu-ating stomas have used a convenience sample anddo not specify the criteria defining the stomaprocedure or the stoma type used to obtain theirstudy samples. Studies utilizing large databasessuch as the American College of SurgeonsNational Surgical Quality Improvement Programuse current procedural terminology (CPT) codesto identify operative procedures, but they may havelimited accuracy.

There has been growing interest in the role ofnatural language processing (NLP) in research.NLP is a subdiscipline of computer science andlinguistics that focuses on developing methods toadd structure automatically to otherwise unstruc-tured free text.5 Unlike word searches, which cansimply identify the terms ileostomy or colostomy butare unable to differentiate between ‘‘no ileostomywas created’’ as opposed to ‘‘ileostomy wasreversed,’’ NLP may be able to distinguish betweenthe 2 situations. The development of automatedapproaches that extract specific medical concepts

from textual medical documents such as operativereports offers a powerful alternative to labor-intensive and expensive manual chart reviews.6

The purpose of this study was to evaluate theaccuracy, sensitivity, and specificity of NLP analysisand CPT coding of stoma procedure patients. Wehypothesized that NLP analysis would be superiorto CPT coding.

METHODS

This study was approved by the InstitutionalReview Board at Baylor College of Medicineand the Research Committee of the MichaelE. DeBakey Veterans Affairs Medical Center(MEDVAMC) in Houston, Texas.

All patients with stoma-related procedures atthe MEDVAMC between January 2005 andDecember 2011 were identified by CPT codesand confirmed by manual review of theirelectronic medical records (EMRs). Stoma-relatedprocedure was defined as an operation thatinvolved stoma formation, reversal, or revision inthe small or large bowel. This approach includedany ileostomy- or colostomy-related proceduresand excluded gastrostomy, jejunostomy, esopho-gostomy, ureterostomy, cystostomy, and ileal loopurostomy.

CPT codes correlating to stoma procedures wereidentified on the basis of the 2005 through 2010CPT standard editions. Two authors (EV, MKL)identified a comprehensive list of codes indepen-dently (Table I). Using the description given in theCPT guide, a prediction of the stoma procedure(formation, reversal, revision) and type of stoma(ileostomy, colostomy, or undesignated) wasmade (Table I). Predefined criteria were used toclassify stoma procedure and type of stoma. Stomasformed at a new site with bowel matured wereclassified as stoma formations. Stomas takendown with the site closed were classified as stomareversals. Stomas taken down when a new stomawas formed at the same site during the sameprocedure were classified as revisions. Stomaslocated in the small intestine were ileostomies,and stomas located in the large intestine werecolostomies. CPT codes with insufficient informa-tion to predict the stoma location were classifiedas undesignated. Each case was reviewed manually,and the following variables were recorded:associated CPT codes, date of operation, actualstoma procedure, and actual type of stoma.

A subgroup of patients undergoing generalsurgery procedures involving the gastrointestinaltract between May 2010 and December 2011 wereidentified by using a master surgery schedule.

Table I. Current procedural terminology codes used to identify stoma procedure and stoma type

CPT code Description of CPT codePredicted stoma

procedurePredictedstoma type

No. of CPTcodes identified

44125 Enterectomy, resection of small intestine; with enterostomy Formation Ileostomy 1644130 Enteroenterostomy, anastomosis of intestine, with or without

cutaneous enterostomy (separate procedure)Formation Ileostomy 9

44141 Colectomy, partial; with skin level cecostomy or colostomy Formation Colostomy 3244143 Colectomy, partial; with end colostomy and closure of distal

segment (Hartmann-type procedure)Formation Colostomy 68

44144 Colectomy, partial; with resection, with colostomy orileostomy and creation of mucofistula

Formation Undesignated 61

44146 Colectomy, partial; with coloproctostomy (low pelvicanastomosis), with colostomy

Formation Colostomy 16

44150 Colectomy, total, abdominal, without proctectomy; withileostomy or ileoproctostomy

Formation Ileostomy 22

44151 Colectomy, total, abdominal, without proctectomy; withcontinent ileostomy

Formation Ileostomy 3

44155 Colectomy, total, abdominal, with proctectomy; withileostomy

Formation Ileostomy 16

44156 Colectomy, total, abdominal, with proctectomy; withcontinent ileostomy

Formation Ileostomy 2

44157 Colectomy, total, abdominal, with proctectomy; withileoanala anastomosis, includes loop ileostomy, and rectalmucosectomy, when performed

Formation Ileostomy 3

44158 Colectomy, total, abdominal, with proctectomy; with ileoanalanastomosis, creation of ileal reservoir, includes loopileostomy, and rectal mucosectomy, when performed

Formation Ileostomy 5

44187 Laparoscopy, surgical; ileostomy or jejunostomy, non-tube Formation Ileostomy 1544188 Laparoscopy, surgical, colostomy or skin level cecostomy Formation Colostomy 2544206 Laparoscopy, surgical; colectomy, partial, with end colostomy

and closure of distal segment (Hartmann type procedure)Formation Colostomy 12

44208 Laparoscopy, surgical; colectomy, partial, with anastomosis,with coloproctostomy (low pelvic anastomosis) withcolostomy

Formation Colostomy 15

44210 Laparoscopy, surgical; colectomy, total, abdominal withoutproctectomy, with ileostomy or ileoproctostomy

Formation Ileostomy 1

44211 Laparoscopy, surgical; colectomy, total, abdominal, withproctectomy, with ileoanal, creation of ileal reservoir, withloop ileostomy, includes mucosectomy, when performed

Formation Ileostomy 7

44212 Laparoscopy, surgical; colectomy, total, abdominal, withproctectomy, with ileostomy

Formation Ileostomy 1

44227 Laparoscopy, surgical, closure of enterostomy, large or smallintestine, with resection and anastomosis

Reversal Undesignated 8

44310 Ileostomy or jejunostomy, non-tube Formation Ileostomy 3344312 Revision of ileostomy; simple (release of superficial scar)

(separate procedure)Revision Ileostomy 12

44314 Revision of ileostomy; complicated (reconstruction in-depth)(separate procedure)

Revision Ileostomy 12

44320 Colostomy or skin level cecostomy Formation Colostomy 5344322 Colostomy or skin level cecostomy; with multiple biopsies

(eg, for congenital megacolon) (separate procedure)Formation Colostomy 3

44340 Revision of colostomy; simple (release of superficial scar)(separate procedure)

Revision Colostomy 14

44345 Revision of colostomy; complicated (reconstruction in-depth) (separate procedure)

Revision Colostomy 10

44346 Revision of colostomy; with repair of paracolostomy hernia(separate procedure)

Revision Colostomy 6

(continued)

SurgeryVolume 154, Number 2

Vo et al 413

Table I. (continued)

CPT code Description of CPT codePredicted stoma

procedurePredictedstoma type

No. of CPTcodes identified

44605 Suture of large intestine (colorrhaphy) for perforated ulcer,diverticulum, wound, injury, or rupture (single or multipleperforations); with colostomy

Formation Colostomy 1

44620 Closure of enterostomy, large or small intestine Reversal Undesignated 4144625 Closure of enterostomy, large or small intestine; with

resection and anastomosis other than colorectalReversal Undesignated 71

44626 Closure of enterostomy, large or small intestine; withresection and colorectal anastomosis (eg, closure ofHartmann-type procedure)

Reversal Undesignated 38

45110 Proctectomy; complete, combined abdominoperineal, withcolostomy

Formation Colostomy 23

45113 Proctectomy, partial, with rectal mucosectomy, ileoanalanastomosis, creation of ileal reservoir, with or withoutloop ileostomy

Formation Ileostomy 3

45119 Proctectomy, combined abdominoperineal pull-throughprocedure (eg, colo-anal anastomosis), with creation ofcolonic reservoir (eg, J-pouch), with diverting enterostomywhen performed

Formation Undesignated 1

45123 Proctectomy, partial, without anastomosis, perineal approach Formation Colostomy 145126 Pelvic exenteration for colorectal malignancy, with

proctectomy (with or without colostomy), with removal ofbladder and ureteral transplantations, and/orhysterectomy, or cervicectomy, with or without removal oftube(s), with or without removal of ovary(s), or anycombination thereof

Formation Colostomy 1

45395 Laparoscopy, surgical; proctectomy, complete, combinedabdominoperineal, with colostomy

Formation Colostomy 19

45397 Laparoscopy, surgical; proctectomy, complete, combinedabdominoperineal, with enterostomy

Formation Undesignated 4

45805 Closure of rectovesical fistula; with colostomy Formation Colostomy 1

There were 21 procedures that had multiple codes.CPT, Current procedural terminology.

SurgeryAugust 2013

414 Vo et al

The master surgery schedule is a list in which allgeneral surgery operating room procedures arerecorded (including emergency and off-hourprocedures). All patients having stoma-relatedoperations in the defined study period wereincluded and were confirmed by manual reviewof the EMRs.

NLP classification algorithm. All available stoma-related operative reports were extracted from thecomputerized patient record system in text format.All operative reports were dictated by residents.Operative reports that were unavailable wereextracted as blank text files. A split validation studywas performed by dividing the available reportsrandomly into a 70% training set for algorithmderivation and a 30% testing set for validation. Thereports, which were all reviewed manually, wereimported into the Automated Retrieval Console(ARC), which was developed by the MassachusettsVeterans Epidemiology Research and Information

Center and classifies text documents using NLPpipelines to break down documents into structuredfragments of text based on parts of speech, negatedterms, and a library of medical and nonmedicalterms.7

Using the training set, ARC automaticallyattempts combinations of features for the specifiedclassification problem, calculating the perfor-mance of several models against the training setusing tenfold cross-validation.5 The best perform-ing algorithm, based on the F-measure, theharmonic mean of recall (estimate of sensitivity),and precision (estimate of positive predictivevalue [PPV]), is then applied to the testing cohort.The performance of the algorithm in terms ofrecall and precision was compared with manualclassifications of operative reports.

We performed analyses using all stoma-relatedoperative reports to classify formation ornonformation, reversal or nonreversal, ileostomy

Table II. Outcomes comparing CPT codes to NLP

Stoma procedures (n = 664)Stoma formations (n

= 461) Stoma reversals (n = 158)

Stoma formations Stoma reversals Ileostomy Colostomy Ileostomy Colostomy Ileostomy Colostomy

CPTSensitivity 97.7% 74.1% 35.0% 75.2% 52.3% 91.6% 0.0% 0.0%Specificity 72.4% 98.7% 88.1% 80.9% 86.8% 74.4% 100.0% 100.0%PPV 84.0% 96.2% 70.1% 76.2% 70.5% 77.6% NA NANPV 95.6% 89.5% 63.1% 80.0% 75.0% 90.2% 36.1% 67.1%Accuracy 87.5% 91.1% 64.6% 78.3% 73.8% 82.9% 36.1% 67.1%

NLP*Sensitivity 99.3% 92.9% 95.8% 95.9% 97.6% 98.6% 96.9% 100.0%Specificity 85.4% 96.7% 88.3% 90.5% 97.1% 96.9% 96.3% 96.1%PPV 91.1% 92.9% 85.8% 88.6% 95.3% 96.9% 97.9% 92.5%NPV 98.7% 96.7% 96.6% 96.6% 98.5% 98.6% 94.5% 100.0%Accuracy 93.7% 95.5% 91.5% 92.8% 97.3% 97.7% 96.7% 97.4%

*NLP utilized 70% of the cases as a training set for algorithm derivation and 30% as the testing set for validation.CPT, Current procedural terminology; NA, not applicable (calculation was unable to be performed); NLP, natural language processing; NPV, negativepredictive value; PPV, positive predictive value.

SurgeryVolume 154, Number 2

Vo et al 415

or nonileostomy, and colostomy or noncolostomy.We performed additional analyses using only CPTcode-selected stoma formation or stoma reversaloperative reports to classify ileostomy or nonileos-tomy and colostomy or noncolostomy.

Analyses. We calculated the sensitivity, speci-ficity, positive predictive value, negative predictivevalue, and accuracy of using the CPT codes and ofusing ARC (the NLP-based software) to predictstoma procedures and types of stoma. Accuracy wasdetermined by the proportion of true results in thepopulation and calculated by (number of truepositives + number of true negatives)/(number oftrue positives + false positives + false negatives +true negatives).

RESULTS

Selecting CPT codes. There were a total of 31stoma-formation CPT codes, of which 14 predictedileostomy, 14 predicted colostomy, and 3 wereundesignated. There were a total of 4 stoma-reversal CPT codes that were undesignated. Therewere 5 stoma revision CPT codes of which 2predicted ileostomy and 3 predicted colostomy(Table I).

CPT validation. Between 2010 and 2011, a totalof 734 cases involving the gastrointestinal tractwere identified from the master surgery schedule,including 206 (28%) stoma-related procedures.CPT codes were 87.4% sensitive and 97.5% specificin identifying stoma-related procedures that wereon the master surgery schedule, and they had aPPV of 93.3%, an NPV of 95.2%, and an accuracyof 94.7%. (Table II).

Manual classification and natural languageprocessing. Using the selected CPT codes andexcluding duplicate data, a total of 664 caseswere included in this study. There were 443 stomaformations, 151 stoma reversals, and 49 stomarevisions (Fig). There were 10 (1.5%) operativereports that were not found in the EMRs.

The NLP-based software ARC was able toidentify stoma formation, stoma reversal, and typesof stoma with greater sensitivities and specificitiesthan the CPT codes). Although ARC wasmore accurate than CPT codes in differentiatingstoma formations (93.7% vs 87.5%) from stomareversals (95.5% vs 91.1%), these differenceswere unimportant. The major circumstances inwhich ARC outperformed CPT codes includedaccuracy in identifying ileostomy procedures(91.5% vs 64.6%) and colostomy procedures(92.8% vs 78.3%). This trend proved to be truewhen evaluating only stoma formations and whenevaluating stoma reversals as well.

DISCUSSION

The primary goals of this study were to determinethe validity of CPTcodes in identifying patients withstoma procedures and in identifying whether CPTcodes or NLP-based software would be able toidentify the typesof stoma, ileostomy, andcolostomy.

We validated the accuracy of CPT codes toidentify stoma procedures in all operationsinvolving the gastrointestinal tract (sensitivity87.4%, specificity 97.5%, accuracy 94.7%).

We demonstrated that CPT codes were adequatein identifying stoma formation (sensitivity 72.4%,

Fig. Case distribution according to CPT code. *There were 21 cases with more than 1 associated CPT code.CPT, Current procedural terminology. (Color version of figure is available online.)

SurgeryAugust 2013

416 Vo et al

specificity 72.4%) and stoma reversals (sensitivity74.1%, specificity 98.7%). CPT codes werefar less sensitive in classifying the types ofstoma, ileostomy (sensitivity 35% [CPT], 95.8%[NLP]), and colostomy (sensitivity 75.2% (CPT),95.9% [NLP]) in comparison to the NLP-basedsoftware ARC.

It appears that although CPT codes may be ableto identify accurately stoma procedures and todistinguish stoma formations from reversals forstoma database research, the limitations of CPTcodes become apparent with more specific queriessuch as the type of stoma. In this study,the NLP-based software ARC demonstrated thatit could be a more accurate method thanadministrative codes for further identification ofileostomy and colostomy.

Administrative codes, such as the InternationalClassification of Diseases, 9th edition (ICD-9) andthe CPT codes, have been used traditionally indata-based research to identify cohorts of patientswith specific diseases. Although they are frequentlyused for research purposes, previous studies haverevealed the inconsistencies of the diagnostic andprocedure codes.8 Studies that have validated theadministrative codes generally used a combinationof ICD-9 and CPT codes to identify cohorts and

found this algorithm to be valid for various clinicalconditions, including acute myocardial infarction,hepatitis C, colorectal cancer liver metastasis, andknee and hip replacements.9-12 Few publishedstudies, however, have validated the use of CPTcodes alone, and none have validated administra-tive codes for stoma procedures.

NLP has emerged as an appealing approachthat could improve the accuracy of administrativedata-based research. An NLP-based approachoffers several potential advantages. Whereasadministrative codes, which were originallydesigned for billing, can restrict the questionbeing asked, NLP allows the customization ofunique data queries. Clearly, an NLP-based searchstrategy is far more practical than manualabstraction in large-scale studies. With its growingapplicability and popularity in health research,NLP tools have been employed in research forprescreening in clinical trials, case identification,adverse-event detection, and chart-review facilita-tion.13 Our findings of superiority of NLP overCPT codes mirror those of other studies thathave demonstrated improved identification ofpatient safety indicators, procedure utilization,and radiology characterization.5,6,13 To date, nopublished studies have investigated the use

SurgeryVolume 154, Number 2

Vo et al 417

of NLP tools for the identification of stomaprocedures or stoma types that we have suggestedor that have compared directly the performanceof NLP to CPT codes.

Our study has several limitations. First, the studyused administrative coding at a single institutionwithin the VA health care system. Therefore, thevariability and quality of coding may be limited orbiased on the basis of coding patterns specific toour institution and might not be generalizable toother VA or non-VA hospitals. Additional trainingand validation will have to be performed in alarger cohort to address this issue. Second, theperformance of NLP-based free-text search relieson the accuracy of the free text being processed.Because the MEDVAMC is a teaching facility, theoperative reports were not standardized, and thequality may have varied. In the methodology, wereported using only dictated operative reports;however, we found that 1.5% of reports were notavailable and were, therefore, left as blankdocuments for the NLP software to search. Thisapproach may have not represented the NLP’sperformance accurately.

In conclusion, CPT codes are adequate toidentify patients who have undergone stomaprocedures but are inadequate in distinguishingileostomy from colostomy procedures.

NLP-based classification algorithms can differ-entiate ileostomy and colostomy proceduresaccurately and can differentiate formation andreversal procedures with greater sensitivity andspecificity than can CPT codes alone. We havedemonstrated that NLP may be used to examinethis question in larger datasets for future studiesthat investigate the complications, reversibility, andquality of care in the various types of stoma.Further study and validation are warranted.

REFERENCES

1. Cottam J, Richards K, Hasted A, Blackman A. Results of anationwide prospective audit of stoma complications within3 weeks of surgery. Colorectal Dis 2007;9:834-8.

2. Robertson I, Leung E, Hughes D, Spiers M, Donnelly L,Mackenzie I, et al. Colorectal Dis 2005;7:279-85.

3. Martin ST, Vogel JD. Intestinal stomas: indications,management, and complications. Adv Surg 2012;46:19-49.

4. Liang MK, Li LT, Avellaneda A, Moffett JM, Hicks SC, AwadSS. Outcomes and predictors of incisional surgical siteinfection in stoma reversal. JAMA Surg (in press, 2013).

5. Hou JK, Chang M, Nguyen T, Kramer JR, Richardson P,Sansgiry S, et al. Automated identification of surveillancecolonoscopy in inflammatory bowel disease using naturallanguage processing. Dig Dis Sci 2012.

6. Murff HJ, FitzHenry F, Matheny ME, Gentry N, Kotter KL,Crimin K, et al. Automated identification of postoperativecomplications within an electronic medical record usingnatural language processing. JAMA 2011;306:848-55.

7. D’Avolio LW, Nguyen TM, Farwell WR, Chen Y, Fitzmeyer F,Harris OM, et al. Evaluation of a generalizable approach toclinical information using the automated retrieval console(ARC). J Am Med Inform Assoc 2010;17:375-82.

8. Lawson EH, Louie R, Zingmond DS, Brook RH, Hall BL,Han L, et al. A comparison of clinical registry versusadministrative claims data for reporting of 30-day surgicalcomplications. Ann Surg 2012;256:973-81.

9. Anaya DA, Becker NS, Richardson P, Abraham NS. Use ofadministrative data to identify colorectal liver metastasis.J Surg Res 2012;176:141-6.

10. Kramer JR, Davila JA, Miller ED, et al. The validity of viralhepatitis and chronic liver disease diagnoses in VeteransAffairs administrative databases. Aliment Pharmacol Ther2008;27:274.

11. Petersen LA,Wright S, Normand SL, et al. Positive predictivevalue of the diagnosis of acute myocardial infarction in anadministrative database. J Gen Intern Med 1999;14:555.

12. Singh JA, Ayub S. Accuracy of VA databases for diagnosesof knee replacement and hip replacement. OsteoarthitisCartilage 2010;18:1639-42.

13. Danforth KN, Early MI, Ngan S, Kosco AE, Zheng C, GouldMK. Automated identification of patients with pulmonarynodules in an integrated health system using administrativehealth plan data, radiology reports, and natural languageprocessing. J Thorac Oncol 2012;7:1257-62.