J O U R N A L O F T H E A M E R I C A N C O L L E G E O F C A R D I O L O G Y V O L . 6 5 , N O . 1 2 , 2 0 1 5

ª 2 0 1 5 B Y T H E A M E R I C A N C O L L E G E O F C A R D I O L O G Y F O U N D A T I O N I S S N 0 7 3 5 - 1 0 9 7 / $ 3 6 . 0 0

P U B L I S H E D B Y E L S E V I E R I N C . h t t p : / / d x . d o i . o r g / 1 0 . 1 0 1 6 / j . j a c c . 2 0 1 5 . 0 1 . 0 2 1

The Cardiovascular In-Training Examination

Development, Implementation, Results, and Future Directions

Jeffrey T. Kuvin, MD,* Amanda Soto, MSED, EDD,y Lauren Foster, MS,y John Dent, MD, MS, MHCM,zAndrew M. Kates, MD,x Donna M. Polk, MD,k Barry Rosenzweig, MD,{ Julia Indik, MD, PHD#

ABSTRACT

Fro

de

Wa

pit

Ne

the

of

Lis

Yo

Ma

BACKGROUND The American College of Cardiology (ACC), in collaboration with the National Board of Medical

Examiners (NBME), developed the first standardized in-training examination (ITE) for cardiovascular disease fellows-

in-training (FITs). In addition to testing knowledge, this examination uses the newly developed ACC Curricular Milestones

to provide specific, competency-based feedback to program directors and FITs. The ACC ITE has been administered

more than 5,000 times since 2011.

OBJECTIVES This analysis sought to report the initial experience with the ITE, including feasibility and reliability of test

development and implementation, as well as the ability of this process to provide useful feedback in key content areas.

METHODS The annual ACC ITE has been available to cardiovascular disease fellowship programs in the United States

since 2011. Questions for this Web-based, secure, multiple-choice examination were developed by a group of cardio-

vascular disease specialists and each question was analyzed by the NBME to ensure quality. Scores were equated and

standardized to allow for comparability. Trainees and program directors were provided detailed feedback, including a list

of the curricular competencies tested by those questions answered incorrectly.

RESULTS The ITE was administered 5,118 times. In 2013, the examination was taken by 1,969 fellows, representing 194

training programs. Among the 3 training years, there was consistency in the examination scores. Total test scores and

scores within each of the content areas increased with each FIT year (there was a statistically significant difference in each

cohort’s average scale score across administration years). There was also significant improvement in examination scores

across the fellowship years.

CONCLUSIONS The ACC ITE is a powerful tool available to all training programs to assess medical knowledge.

This examination also delivers robust and timely feedback addressing individual knowledge gaps, and thus, may serve

as a basis for improving training curricula. (J Am Coll Cardiol 2015;65:1218–28) © 2015 by the American College of

Cardiology Foundation.

A ssessment of medical knowledge is an essen-tial part of graduate medical education.An in-training examination (ITE) is a formal

method for the evaluation of medical knowledge,and many specialty and subspecialty post-graduate

m the *Cardiovascular Center, Tufts Medical Center, Boston, Massachu

lphia, Pennsylvania; zCardiovascular Division, University of Virginia,

shington University School of Medicine, St. Louis, Missouri; kCardiovascal, Boston, Massachusetts; {Leon H. Charney Division of Cardiology, N

w York; and the #Division of Cardiology, University of Arizona, Tucson,

National Board of Medical Examiners. All other authors have reported tha

this paper to disclose.

ten to this manuscript’s audio summary by JACC Editor-in-Chief Dr. Vale

u can also listen to this issue’s audio summary by JACC Editor-in-Chief D

nuscript received October 13, 2014; revised manuscript received January

training programs incorporate ITEs into their cur-ricula. In fact, formal validated assessment of medicalknowledge, such as an ITE, is a key component of theAccreditation Council for Graduate Medical Educa-tion (ACGME) program requirements (1,2). Internal

setts; yNational Board of Medical Examiners, Phila-

Charlottesville, Virginia; xCardiovascular Division,

ular Medicine Division, Brigham and Women’s Hos-

ew York University School of Medicine, New York,

Arizona. Ms. Foster and Ms. Soto are employees of

t they have no relationships relevant to the contents

ntin Fuster.

r. Valentin Fuster.

6, 2015, accepted January 14, 2015.

AB BR E V I A T I O N S

AND ACRONYM S

ABIM = American Board of

Internal Medicine

ACC = American College

of Cardiology

J A C C V O L . 6 5 , N O . 1 2 , 2 0 1 5 Kuvin et al.M A R C H 3 1 , 2 0 1 5 : 1 2 1 8 – 2 8 Cardiovascular In-Training Examination

1219

Medicine training programs have incorporated a na-tional ITE since 1988 (3). In recent years, medicinesubspecialties, such as nephrology (4) and oncology(5), have developed computer-based ITEs in associa-tion with the National Board of Medical Examiners(NBME).

TABLE 1 American College of Cardiology

In-Training Examination Blueprint

Medical Content Category Relative Percentage

Arrhythmias 12.0

Coronary artery disease 12.5

Acute coronary syndromes 12.0

Valvular disorders 12.0

Congenital disorders 7.0

Pericardial disease 3.0

Aorta/peripheral vascular disease 9.0

Hypertension/Pulmonary disorders 7.0

Pharmacology 5.0

Congestive heart failure 13.0

Physiology/biochemistry 6.0

Miscellaneous 1.5

SEE PAGE 1229ACGME = Accreditation

Council for Graduate

Medical Education

FIT = fellow-in-training

ITE = in-training examination

NBME = National Board of

Medical Examiners

In 2011, the American College of Cardiology (ACC)developed and implemented the first standardizedITE for cardiology fellows-in-training (FITs). Thissecure examination is executed by the NBME andserves as a benchmark for trainees as they progressthrough training and prepare for the AmericanBoard of Internal Medicine (ABIM) certification ex-amination. The goal of the cardiology ITE, like mostITEs, is to assess trainee medical knowledgethrough multiple-choice questions in the contentareas specified by the ABIM. Moreover, the ACC ITEincorporates a unique feedback process wherebyeach examination question is tagged to the recentlydeveloped, competency-based ACC Curricular Mile-stones. Thus, after examination completion, pro-gram directors and FITs receive information thatidentifies knowledge gaps and areas in need offurther curricular emphasis. During the past 3 years,the ITE has become the standard in-training exam-ination in adult cardiology fellowship programsacross the United States. This paper offers an anal-ysis of the data accrued thus far, and outlines futuredirections for the ITE examination.

METHODS

The cardiology ITE was developed by a test-writingcommittee consisting of a chair and 11 members(Online Appendix). Eight members of the test writingcommittee were fellowship training program di-rectors and all committee members underwentinstructional training for test question writing pro-vided by the NBME. Questions were developed spe-cifically for this examination and were reviewed inface-to-face committee meetings. Each question wasdesigned to begin with a clinical stem and incorporatea single-best-answer, multiple-choice format. Manyquestions were accompanied by high-resolution stillor video images. The NBME editors and psychome-tricians analyzed all questions for structure, content,and statistical performance.

The ITE was designed to assess the knowledge ofgeneral cardiovascular disease FITs. Each year’s ex-amination consisted of 150 questions, separated into5 equal sections of 30 questions apiece. FITs wereallotted 1 h per session, with breaks after each

session. The examination blueprint mirrorsthe ABIM certifying cardiovascular diseaseexamination in terms of the percent ofquestions dedicated to each content area(Table 1).

The ACC provides information to cardiol-ogy fellowship training program directors andcoordinators regarding the ITE via an internetlistserv and FITs are registered for the ex-amination online. The examinations arescheduled on 2 consecutive days in October,with a makeup date 1 week later. FITs takethe examinations online via a secure website

provided by the NBME. Neither test takers nor pro-gram directors are allowed access to the questionsafter the examination.

The goal of the ITE is to test medical knowledgeacross the six core ACGME competencies (1,2). Eachexamination question is linked to an ACC CurricularMilestone (6), which allows for direct feedback tofellows and program directors. Thus, after the ex-amination responses are scored, fellows receive adetailed assessment of scores along with a list ofcurricular milestones associated with core clinicaldiagnoses (Figure 1) for each question answeredincorrectly, with the objective of providing insightinto knowledge gaps. In addition, each FIT receivesan analysis of his or her own performance on thetotal test, represented by a score for the proportionof the 150 test questions answered correctly. FITsadditionally receive scores for the proportion ofitems answered correctly in each of the major con-tent areas, as well as the mean proportion answeredcorrectly in each area by fellows in each year oftraining.

FIT scores are equated using Item Response Theory(7) and are placed on a standardized score scale set

FIGURE 1 Examination Feedback Data Provided to Fellows and Program Directors

(A) Fellows receive a total score for the examination, including the percent of questions answered correctly, a percentile rank, and a scale score. In addition,

specific scores in American Board of Internal Medicine content areas are provided. (B) Individual scores for each section, along with mean and standard

deviation for fellowship year, are provided to fellows and training directors. (C) Fellows and program directors receive a list of American College of

Cardiology (ACC) Curricular Milestones and diagnoses for each question answered incorrectly, thereby identifying knowledge gaps. CS ¼ competency

statement; D ¼ diagnosis.

Kuvin et al. J A C C V O L . 6 5 , N O . 1 2 , 2 0 1 5

Cardiovascular In-Training Examination M A R C H 3 1 , 2 0 1 5 : 1 2 1 8 – 2 8

1220

FIGURE 1 Continued

J A C C V O L . 6 5 , N O . 1 2 , 2 0 1 5 Kuvin et al.M A R C H 3 1 , 2 0 1 5 : 1 2 1 8 – 2 8 Cardiovascular In-Training Examination

1221

to a mean of 500 points and a standard deviationof 100 points. Equating allows for comparabilityin scale scores across each fellow year (first, second,and third) within a testing year, and across testingyears. Thus, these processes allow for comparisonacross fellowship years and provide a measure

of improvement during training. FITs are able tocompare their individual performance with peersnationally by using the provided percentile ranks(for comparison within fellow year) and the equatedscale scores (for comparisons across administrationsof the test).

FIGURE 1 Continued

Kuvin et al. J A C C V O L . 6 5 , N O . 1 2 , 2 0 1 5

Cardiovascular In-Training Examination M A R C H 3 1 , 2 0 1 5 : 1 2 1 8 – 2 8

1222

TABLE 2 In-Training Examination Fellow Participants

Administration Year Fellowship Year Number of Exam Takers

2011 First 353

Second 453

Third 513

Total* 1,337

2012 First 482

Second 636

Third 614

Total* 1,751

2013 First 541

Second 742

Third 686

Total* 2,030

*Totals include individuals who reported a fellow year other than first, second,or third.

J A C C V O L . 6 5 , N O . 1 2 , 2 0 1 5 Kuvin et al.M A R C H 3 1 , 2 0 1 5 : 1 2 1 8 – 2 8 Cardiovascular In-Training Examination

1223

Program directors are provided with summaryreports outlining overall FIT performance within anindividual training program and across the countryin each content area (with the FITs divided byfellowship year). Each program is provided with ahistogram representing the distribution of meanpercent correct scores across all programs. Programdirectors are provided with a breakdown of the indi-vidual performances of each of their individual FITs,the same performance data provided to the fellowsthemselves, including scale score, an overall percentcorrect score, and scores within each of the majorcontent areas.

RESULTS

The ITE was administered to 3,388 FITs over a3-year period with a total of 5,118 examinationadministrations (Table 2). The ITE was given to1,319 FITs in 127 cardiology fellowship trainingprograms in the first examination year (2011), to1,732 FITs in 165 programs in the second year (2012),and to 1,969 FITs in 194 programs in the third year(2013). All training programs that participated in

TABLE 3 In-Training Examination Statistics

Year Reliability (a) SEM* Average Item Difficulty � SD†

2011 0.73 53 0.62 � 0.23

2012 0.81 51 0.65 � 0.20

2013 0.80 51 0.62 � 0.21

*Standard error of measurement (SEM) on the basis of the Reference Group andthe reported score scale. †Item difficulty with SD on the basis of p values foradministered items for the Reference Group.

the initial 2 examinations participated in the thirdexamination.

The ITE maintained comparable reliability duringthe 3 administration years, and had a reliabilitycoefficient (alpha) of between 0.73 and 0.81 each year(Table 3) (8). The standard error of measurement,indicating the degree of precision in scores, fluctu-ated between 51 and 53 points across administrations,thereby indicating that if an individual were to reteston the same content, his or her expected score wouldbe within approximately 53 scale score points of theiroriginal score.

A comparison of the means and standard de-viations in scale scores across administrations andfellow years is provided (Figure 2). There was con-sistency in the examination scores between the 3training years, with first-year fellows scoring simi-larly on each examination administration, with in-cremental improvement in average scores withadditional years of fellowship training. Summaryplots of training program and individual examineeperformance are shown (Figures 3 and 4).

FIT scores tended to increase on the basis of totaltest performance and performance within the majorcontent areas as trainees moved through the trainingyears. This is reflected in a statistically significantdifference in each cohort’s average scale score acrossadministration years (Table 4). For example, second-year fellows in 2012 scored higher, on average, thanfirst-year fellows did the year prior. Although thereis not complete fellow overlap across years (that is,not all second-year FITs took the examination dur-ing their first year), the trend among those whotook the ITE during consecutive training years istoward a higher score with more advanced training.There was an incremental and statistically signifi-cant improvement in examination scores acrossthe fellowship years, indicating that within eachadministration, fellows with more advanced trainingscored higher. Second-year and third-year FITperformance increased significantly across adminis-trations on the basis of 1-way analysis of variance(F[2,1316 ¼ 114.56], F[2,1729 ¼ 218.78], F[2,1966 ¼221.37], p < 0.05).

DISCUSSION

The ACC’s cardiology ITE has become an importantpart of fellowship training (Central Illustration).The ITE assists in assessing fellow knowledge andallows training programs to adjust curriculum toaddress specific content areas. As shown in thisanalysis, the examination is robust in its ability todiscern beginner knowledge compared with that of

FIGURE 2 In-Training Examination Scale Scores

800

700

600

500

400

300

2002011

First-Year Fellows Second-Year Fellows Third-Year Fellows

534

569

511

416

578

524

437

496

428

Administration Year

ACC In-Training Exam - Performance Across Administrations

Scal

e Sc

ore

2012 2013

Scoring data � SD divided by fellowship year (2011 to 2013).

Kuvin et al. J A C C V O L . 6 5 , N O . 1 2 , 2 0 1 5

Cardiovascular In-Training Examination M A R C H 3 1 , 2 0 1 5 : 1 2 1 8 – 2 8

1224

the advanced trainee. This type of knowledgeassessment is key to allowing the trainee to focus onspecific topics that are underdeveloped, and perhaps,under-recognized. Providing FITs with timely feed-back on knowledge deficits allows the developmentof remediation plans during training, instead ofdiscovering these knowledge deficits years later, andshould help in preparation for the ABIM certifyingexamination. An ACC survey of cardiology trainingprograms indicates that training directors are nowusing the results of the ITE to provide individualizedcounseling to fellows and to further develop trainingcurricula (personal communication from Julia

TABLE 4 Cohort Score Changes Across Fellowship Years

Groups Compared Number of Fellows

2011 first-year and 2012 second-year fellows 288

2011 second-year and 2012 third-year fellows 402

2012 second-year fellows and 2013 third-year fellows 536

2012 first-year fellows and 2013 second-year fellows 403

Mean differences are significant on the basis of the results from Fisher’s least significan

df ¼ degrees of freedom.

Bainbridge, American College of Cardiology, October1, 2014). Future data will help determine whetherscores on the ITE correlate with ABIM certifying ex-amination scores. In years past, some training pro-grams developed unique in-training modules toassess trainees. Now, there is a national, secure ex-amination that allows trainees and programs toassess an individual fellow’s knowledge and providecomparison with trainees from around the country.The high number of repeating examinees and pro-grams indicate that the ITE is a useful tool and isrelevant to today’s training needs.

The ACC ITE provides a unique assessment tool forcardiology trainees and programs. The examination islinked to the newly developed ACC Curricular Mile-stones and provides valuable information about spe-cific areas in need of further education and training.Each question is tagged to a specific competencystatement within the curricular milestones and isdesigned to direct the examinee to content areasrelated to questions answered incorrectly. In addi-tion, the examination was built on the ABIM cardio-vascular medicine examination blueprint, withspecial emphasis on the 6 core competency areas.Thus, in addition to assisting in preparation for theABIM certifying examination, the ITE provides fel-lows with the opportunity to identify areas ofstrength and weakness and to compare themselves totheir peers across the country. On a national level,data derived from the ITE will also engendercompetency-based training by providing a broad un-derstanding of the educational gaps of cardiologyfellows.

The 3-year data presented indicate that the testis reliable and robust, as demonstrated by the statis-tically significant improvement in scores as FITsprogress through training. This pattern is to be ex-pected on several fronts. First, feedback received byfirst-year fellows allows them to better prepare forthe second and third years of cardiovascular train-ing. Second, training experiences and curricularknowledge increase commensurate with fellowshiplevel.

Year 1(mean � SD)

Year 2(mean � SD) t (df) Significance

426.31 � 94.03 519.26 � 123.58 —18.27 (287) p < 0.01

493.88 � 100.66 576.62 � 126.541 —19.15 (401) p < 0.01

514.61 � 123.78 584.91 � 120.35 —19.54 (535) p < 0.01

418.28 � 112.10 534.98 � 117.99 —26.32 (401) p < 0.01

t difference test.

FIGURE 3 Training Program Mean Percent Correct Scores

A B22

20

18

16

14

12

10

Num

ber o

f Pro

gram

s

Mean Total Test Percent Correct Score Mean Total Test Percent Correct Score

Num

ber o

f Pro

gram

s

8

6

4

2

0

22

Mean = 61.46SD = 3.486N = 127

Mean = 64.13SD = 5.077N = 165

20

18

16

14

12

10

8

6

4

2

038 40 42 44 46 48 52 54 56 58 60 62 64 66 68 70 72 74 76 78 80 825050 52 54 56 58 60 62 64 66 68 70 72 74 76

C

Mean Total Test Percent Correct Score

Mean = 61.29SD = 5.077N = 194

Num

ber o

f Pro

gram

s

22

24

26

20

18

16

14

12

10

8

6

4

2

028 30 32 34 36 3840 42 44464850 52 54 56 586062 64666870 72 74 76 788082

Distribution of mean percent correct scores across U.S. cardiology training programs: (A) 2011; (B) 2012; (C) 2013.

J A C C V O L . 6 5 , N O . 1 2 , 2 0 1 5 Kuvin et al.M A R C H 3 1 , 2 0 1 5 : 1 2 1 8 – 2 8 Cardiovascular In-Training Examination

1225

STUDY LIMITATIONS. There are limitations noted inthe ITE experience thus far. Approximately 20% ofquestions are repeated across examination forms toallow for equating scores across administrationyears, and as such, it is possible that fellows mayimprove their score due to familiarity with previ-ously viewed questions. Although the number ofquestions available for reuse across years is limited

due to the ITE having only been in existence for 3years, we expect this to change in ensuing yearswith the development of additional questions.Another limitation of the examination is the lack of adefined section dedicated to electrocardiograms andimages. The test-writing committee, along with theNBME, is now working to develop new modulesto better mirror the present-day ABIM certifying

FIGURE 4 Examinee Mean Percent Correct Scores

130

A B

120

110

100

90

80

70

60

50

40

30

20

10

0

220210

200190180170160150140130120110

1009080706050403020100

0 100 200 300 400 500Total Test Scale Score

Num

ber o

f Exa

min

ees

Num

ber o

f Exa

min

ees

600 700 800 900 1000 0 100 200 300 400 500Total Test Scale Score

600 700 800 900 1000

C220210

200190180170160150140130120110

1009080706050403020100

0 100 200 300 400 500 600 700 800 900 1000Total Test Scale Score

Num

ber o

f Exa

min

ees

Mean = 494.41SD = 110.45N = 1,337

Mean = 505.83SD = 135.655N = 1,751

Mean = 518.93SD = 128.571N = 2,030

Distribution of examinee scale scores: (A) 2011; (B) 2012; (C) 2013.

Kuvin et al. J A C C V O L . 6 5 , N O . 1 2 , 2 0 1 5

Cardiovascular In-Training Examination M A R C H 3 1 , 2 0 1 5 : 1 2 1 8 – 2 8

1226

examination. Furthermore, there is potential foradditional ITEs in the subspecialty areas of cardio-vascular medicine, including electrophysiology,heart failure, interventional cardiology, and adultcongenital heart disease. The ITE has been used in afew countries outside the United States. In futureyears, further international penetration of the ITEappears warranted. Finally, after a few more years ofdata are accrued, ACC ITE data will be compared toABIM certification examination data. This will help

identify trainees at risk for failing the ABIM exami-nation and provide further guidance to program di-rectors and fellows.

CONCLUSIONS

The ACC ITE, developed in conjunction with theNBME, tests medical knowledge of cardiologyfellows in content areas that mirror the blueprintof the ABIM Cardiovascular Medicine certification

CENTRAL ILLUSTRATION Cardiovascular In-Training Examination: Assessment of Competency inCardiovascular Fellowship Training

The In-Training Examination (ITE) is a competency-based tool designed to help trainees and fellowship programs define knowledge gaps and

track improvement during training. The ITE complements other assessment tools, including Accreditation Committee for Graduate Medical

Education (ACGME) Milestones, American College of Cardiology Foundation (ACCF) Curricular Milestones, Core Cardiovascular Training

Statement (COCATS), and licensure/certification requirements.

PERSPECTIVES

COMPETENCIES IN MEDICAL KNOWLEDGE, PRACTICE-

BASEDLEARNING, AND IMPROVEMENT: The ACC ITE tool

assesses the medical knowledge of cardiovascular fellows-in-

training, identifies knowledge gaps and provides competency-

based feedback addressing each of the 6 core competencies

defined by the ACGME.

TRANSLATIONAL OUTLOOK: Additional work is needed

to identify test-related, trainee-related, and program-

related factors that correlate with subsequent passing of the

ABIM cardiovascular disease certifying examination or with

clinical competence, but further experience and analysis of

data from this examination will shed light on these

relationships.

J A C C V O L . 6 5 , N O . 1 2 , 2 0 1 5 Kuvin et al.M A R C H 3 1 , 2 0 1 5 : 1 2 1 8 – 2 8 Cardiovascular In-Training Examination

1227

examination. The ITE is a robust method of testingmedical knowledge of cardiology FITs across yearsof training and an invaluable tool for programdirectors.

ACKNOWLEDGMENTS The authors thank KristinDoermann and Julie Bainbridge, along with theleadership of the American College of Cardiology,for helping to develop this important teaching andlearning tool for training programs. In addition, theauthors thank the entire staff of the National Boardof Medical Examiners, in particular, Molly Meehan,Drew Dillon, and Tricia Manning.

REPRINT REQUESTS AND CORRESPONDENCE: Dr.Jeffrey T. Kuvin, Cardiovascular Center, Tufts MedicalCenter, 800 Washington Street, Box 315, Boston, Massa-chusetts 02111. E-mail: jkuvin@tuftsmedicalcenter.org.

Kuvin et al. J A C C V O L . 6 5 , N O . 1 2 , 2 0 1 5

Cardiovascular In-Training Examination M A R C H 3 1 , 2 0 1 5 : 1 2 1 8 – 2 8

1228

RE F E RENCE S

1. ACGME Common Program Requirements,focused revision. American Council for GraduateMedical Education. 2013. Available at: http://www.acgme.org/acgmeweb/Portals/0/PFAssets/ProgramRequirements/CPRs2013.pdf. AccessedJanuary 21, 2015.

2. ACGME Program Requirement for GraduateMedical Education in Cardiovascular Disease (In-ternal Medicine). American Council for GraduateMedical Education. 2013. Available at: https://www.acgme.org/acgmeweb/Portals/0/PFAssets/2013-PR-FAQ-PIF/141_cardiovascular_disease_int_med_07132013.pdf. Accessed January 21, 2015.

3. Garibaldi RA, Subhiyah R, Moore ME, et al. TheIn-Training Examination in Internal Medicine:an analysis of resident performance over time.Ann Intern Med 2002;137:505–10.

4. Rosner MH, Berns JS, Parker M, et al., for theASN In-Examination Training Committee. Devel-opment, implementation, and results of the ASNin-training examination for fellows. Clin J Am SocNephrol 2010;5:328–34.

5. Collichio FA, Kayoumi KM, Hande KR, et al.Developing an in-training examination for fel-lows: the experience of the American Societyof Clinical Oncology. J Clin Oncol 2009;127:1706–11.

6. American College of Cardiology com-petency statements. Available at: http://www.Cardiosource.Org/corecompetencies. AccessedJanuary 19, 2015.

7. Lord FM. Applications of Item ResponseTheory to Practical Testing Problems. Mahwah,

New Jersey: Lawrence Erlbaum Associates;1980.

8. Cronbach LJ. Coefficient alpha and the in-ternal structure of tests. Psychometrika 1951;16:297–334.

KEY WORDS cardiology, cardiovasculareducation, clinical competence, curriculum,education, fellow-in-training, graduate,medical

APPENDIX For a list of the In-TrainingExamination Writing Group, please see theonline version of this article.