circulation 2010 fornwalt 1985 91

8/12/2019 Circulation 2010 Fornwalt 1985 91

1/8

John D. Merlino, Derek A. Fyfe, Angel R. Len and John N. OshinskiBrandon K. Fornwalt, William W. Sprague, Patrick BeDell, Jonathan D. Suever, Bart Gerritse,

Resynchronization TherapyAgreement Is Poor Among Current Criteria Used to Define Response to Cardiac

Print ISSN: 0009-7322. Online ISSN: 1524-4539Copyright 2010 American Heart Association, Inc. All rights reserved.

is published by the American Heart Association, 7272 Greenville Avenue, Dallas, TX 75231Circulation doi: 10.1161/CIRCULATIONAHA.109.910778

2010;121:1985-1991; originally published online April 26, 2010;Circulation.

http://circ.ahajournals.org/content/121/18/1985

World Wide Web at:The online version of this article, along with updated information and services, is located on the

http://circ.ahajournals.org//subscriptions/is online at:CirculationInformation about subscribing toSubscriptions:

http://www.lww.com/reprints

Information about reprints can be found online at:Reprints:

document.Permissions and Rights Question and Answerthis process is available in theclick Request Permissions in the middle column of the Web page under Services. Further information aboutOffice. Once the online version of the published article for which permission is being requested is located,

can be obtained via RightsLink, a service of the Copyright Clearance Center, not the EditorialCirculationinRequests for permissions to reproduce figures, tables, or portions of articles originally publishedPermissions:

by guest on July 5, 2014http://circ.ahajournals.org/Downloaded from by guest on July 5, 2014http://circ.ahajournals.org/Downloaded from
http://circ.ahajournals.org/content/121/18/1985http://circ.ahajournals.org//subscriptions/http://circ.ahajournals.org//subscriptions/http://circ.ahajournals.org//subscriptions/http://www.lww.com/reprintshttp://www.lww.com/reprintshttp://www.lww.com/reprintshttp://www.ahajournals.org/site/rights/http://www.ahajournals.org/site/rights/http://circ.ahajournals.org/http://circ.ahajournals.org/http://circ.ahajournals.org/http://circ.ahajournals.org/http://circ.ahajournals.org/http://circ.ahajournals.org/http://circ.ahajournals.org/http://circ.ahajournals.org//subscriptions/http://www.lww.com/reprintshttp://www.ahajournals.org/site/rights/http://circ.ahajournals.org/content/121/18/1985


2/8

Arrhythmia/Electrophysiology

Agreement Is Poor Among Current Criteria Used to DefineResponse to Cardiac Resynchronization Therapy

Brandon K. Fornwalt, PhD; William W. Sprague, BS; Patrick BeDell, BBA, RDCS;Jonathan D. Suever, BS; Bart Gerritse, PhD; John D. Merlino, MD; Derek A. Fyfe, MD, PhD;

Angel R. Leon, MD; John N. Oshinski, PhD

BackgroundNumerous criteria believed to define a positive response to cardiac resynchronization therapy have been

used in the literature. No study has investigated agreement among these response criteria. We hypothesized that the

agreement among the various response criteria would be poor.

Methods and ResultsA literature search was conducted with the keywords cardiac resynchronization and

response. The 50 publications with the most citations were reviewed. After the exclusion of editorials and

reviews, 17 different primary response criteria were identified from 26 relevant articles. The agreement among 15

of these 17 response criteria was assessed in 426 patients from the Predictors of Response to Cardiac

Resynchronization Therapy (PROSPECT) study with Cohens -coefficient (2 response criteria were not

calculable from PROSPECT data). The overall response rate ranged from 32% to 91% for the 15 response criteria.

Ninety-nine percent of patients showed a positive response according to at least 1 of the 15 criteria, whereas 94%

were classified as a nonresponder by at least 1 criterion. -Values were calculated for all 105 possible comparisons

among the 15 response criteria and classified into standard ranges: Poor agreement (0.4), moderate agreement

(0.40.75), and strong agreement (0.75). Seventy-five percent of the comparisons showed poor agreement,

21% showed moderate agreement, and only 4% showed strong agreement.

ConclusionsThe 26 most-cited publications on predicting response to cardiac resynchronization therapy define response

using 17 different criteria. Agreement between different methods to define response to cardiac resynchronization therapy

is poor 75% of the time and strong only 4% of the time, which severely limits the ability to generalize results over

multiple studies. (Circulation. 2010;121:1985-1991.)

Key Words: heart failure cardiac resynchronization therapy pacing pacemakers

Predicting whether a patient will benefit, or respond, tocardiac resynchronization therapy (CRT) has been the focusof more than 500 publications during the last 5 years; however,

the definition of response to CRT varies widely between studies,

and numerous criteria to define a positive response to CRT exist

in the literature. Echocardiographic response is typically

assessed by quantifying the change in left ventricular ejection

fraction14 or left ventricular end-systolic volume (LVESV)2,510

3 to 6 months after CRT implantation. Clinical response is

assessed with the increase in the distance walked in 6 minutes11

or improvement in New York Heart Association functional

class2,1214 3 to 6 months after CRT implantation. Some studies

have defined response to CRT as a combination of several

clinical measures1517 or as a combination of both clinical and

echocardiographic measures.18

Editorial see p 1977Clinical Perspective on p 1991

The heterogeneous approach to defining response to CRT

is a potential barrier to progress in this field. No study has

addressed this issue by investigating the agreement among

the numerous published CRT response criteria. If these

different response criteria show poor agreement, then the

ability to generalize results from multiple studies is severely

impaired, and a standard needs to be developed. We hypoth-

esized that the agreement between the various published

response criteria would be poor. We tested this hypothesis by

identifying response criteria from a literature search and thenassessing the statistical agreement among the different crite-

ria in the 426 patients enrolled in the Predictors of Response

to Cardiac Resynchronization Therapy (PROSPECT) study.

Continuing medical education (CME) credit is available for this article. Go to http://cme.ahajournals.org to take the quiz.Received September 19, 2009; accepted February 17, 2010.From the Emory University School of Medicine (B.K.F., W.W.S., J.D.M., D.A.F., A.R.L., J.N.O.), Atlanta, Ga; Emory/Georgia Institute of

Technology, Department of Biomedical Engineering (B.K.F., J.D.S., J.N.O.), Atlanta, Ga; The Carlyle Fraser Heart Center (P.B., J.D.M., A.R.L.),

Division of Cardiology, Atlanta, Ga; Medtronic Bakken Research Center (B.G.), Maastricht, the Netherlands; and Sibley Heart Center Cardiology(D.A.F.), Atlanta, Ga.

Correspondence to Brandon Fornwalt, Emory University School of Medicine, Department of Radiology, MR Research, 1365 Clifton Rd, Atlanta, GA

30322. E-mail [email protected] 2010 American Heart Association, Inc.

Circulation is available at http://circ.ahajournals.org DOI: 10.1161/CIRCULATIONAHA.109.910778

1985 by guest on July 5, 2014http://circ.ahajournals.org/Downloaded from
http://circ.ahajournals.org/http://circ.ahajournals.org/http://circ.ahajournals.org/http://circ.ahajournals.org/http://circ.ahajournals.org/


3/8

Methods

Literature ReviewTo identify commonly used criteria to define response to CRT, a

literature search was conducted with the Web of Science ScienceCitation Index Expanded database19 using the topics cardiacresynchronization and response. The 50 publications with the

most citations were reviewed for relevance (Figure 1). Four reviewarticles and 20 publications that did not report individual responsecriteria were excluded.

Seventeen different primary response criteria were identified fromthe 26 remaining publications (Table 1).118,2027 Eight of these 17

response criteria were based on echocardiography, 8 were based onclinical measures, and 1 criterion was based on a combination ofboth echocardiographic and clinical measures. Six of the 17 response

criteria included either all-cause or heart failure mortality as acriterion to define a nonresponse, whereas the other 11 did not.

Patient PopulationAgreement between response criteria was assessed with informationfrom the baseline and 6-month follow-up visits for the 426 patients

in the PROSPECT study.10 Briefly, PROSPECT was a prospective,

multicenter study that was designed to test the ability of 12 differentechocardiographic dyssynchrony parameters to predict response to

CRT. Four hundred fifty-seven patients with standard CRT indica-tions (New York Heart Association class III/IV heart failure, leftventricular ejection fraction 35%, QRS 130 ms, and stablemedical regimen) were enrolled in PROSPECT at 53 centers

worldwide. After the exclusion of 31 patients who exited the studyearly and did not receive an implant, 426 patients remained and werefollowed up for 6 months after CRT implantation.

Statistics and Quantification of AgreementThe Cohen -coefficient was used to assess agreement between the

different response criteria. The -coefficient is an accepted statisticalcoefficient that is used to assess agreement between methodolo-gies.28,29 The -coefficient ranges from1 (perfect disagreement) to

1 (perfect agreement), and a -coefficient of 0 indicates that theamount of agreement was exactly that expected by chance.28 A

-coefficient 0.75 was defined as strong agreement, 0.40.75was defined as moderate agreement, and 0.4 was defined as poor

agreement (Table 2).29

The -coefficient was calculated with the following equation:

(1)

observed agreement (%)expected agreement (%)

100%expected agreement (%)

Expected agreement is the percentage of cases in which 2 response

criteria would agree on the basis of chance alone and can be

calculated as follows:

(2) expected agreement (%)responderscriterion 1

responderscriterion 2non-responderscriterion 1

non-responderscriterion 2

An example of a -value calculation is shown in Table 3 and

presented in the Results section.Two of the 17 response criteria24,25 could not be calculated from

PROSPECT data because (1) oxygen consumption at peak exercise

was not measured in PROSPECT and (2) height and weight at 6months were not measured in PROSPECT, which precludes the

Figure 1. Flow chart showing the process by which responsecriteria were identified from the literature.

Table 1. Seventeen Different Response Criteria Identified

From the 26 Relevant Publications

Response criteria

Echocardiographic

1. 1LVEF 5% (absolute)1,2

2. 1LVEF 15%3,4

3. 2LVESV

10% and did not die of progressive HF within 6months20,27

4. 2LVESV 15%2,510

5. LVESV 115% of baseline26

6. 2LVESVI 15%25

7. 2LVEDV 15%2

8. 1Stroke volume 15%4,21,22

Clinical

9. 2NYHA12,1214

10. 2NYHA1 and did not die of progressive HF within 6 months23

11. 2NYHA1 and 16MWD 25%15

12. 2NYHA1 and 16MWD 25% and did not die of progressive HFwithin 6 months16,17

13. 16MWD 10%, no heart transplant, did not die of progressive HFwithin 6 months11

14. (2NYHA1 or 1VO2max 10% or 16MWD 10%) and alive, nohospitalization for decompensated HF24

15. Two of 3:5

2NYHA1

16MWD 50 m

2QOL 15

16. Clinical composite score improved10

Combined

17. (1LVEF 5% absolute or 16MWD 30 m) and (2NYHA1 or2QOL 10)18

1 Indicates increase; LVEF, left ventricular ejection fraction;2, decrease;HF, heart failure; LVESV, left ventricular end-systolic volume; LVESVI, LVESVindexed by body surface area; LVEDV, left ventricular end-diastolic volume;

NYHA, New York Heart Association functional class; 6MWD, 6-minute walk

distance; VO2max, oxygen consumption at peak exercise; and QOL, quality-of-

life score.

If the authors did not specify whether death was considered a nonresponse,

then it was assumed that deaths were excluded.

1986 Circulation May 11, 2010

by guest on July 5, 2014http://circ.ahajournals.org/Downloaded from
http://circ.ahajournals.org/http://circ.ahajournals.org/http://circ.ahajournals.org/http://circ.ahajournals.org/


4/8

ability to calculate the LVESV index. Agreement among the 15

remaining response criteria was assessed by calculating a -value for

all possible pairs of criteria. Thus, 105 -values were calculated.

Group -values were then quantified (mean, median, and range) tosummarize the agreement among criteria from 4 different groups: (1)

All 105 comparisons, (2) comparisons between 2 echocardiographic

criteria, (3) comparisons between 2 clinical criteria, and (4) compar-

isons between an echocardiographic criterion and a clinical criterion.Mean group -values were compared with a permutation test, and a

Bonferroni correction was applied to account for multiple compari-

sons. To justify that the sample size to estimate -values was large

enough, a bootstrap resampling procedure was used with aKolmogorov-Smirnov test to assess the normality of the resulting

distribution. P0.05 was defined as statistically significant.

Subgroup Analysis Excluding Response CriteriaQuantified Short Term and at 3 MonthsOne response criterion (increase in stroke volume 15%4,21,22) was

used in the literature as a short-term response measure that was

quantified within 2 days of CRT implantation. In addition, 2response criteria from the literature were only used in studies that

assessed response at 3 months (criteria 15 and 17 in Table 1).

Short-term (2 days) and 3-month data for calculating all 15response measures were not collected in PROSPECT, so all criteriawere assessed with data from the 6-month visit. To ensure that the

present results were not confounded by calculating these short-term

and 3-month measures from 6-month follow-up data, we performeda subgroup analysis in which we excluded them and recalculated the

group -values.

Results

Response RatesThe percentage of patients defined as having a positive re-

sponse to CRT ranged from 32% to 91% for the 15 response

criteria (Table 4). All 15 criteria could be calculated in 250 of the

426 patients in PROSPECT. Of these 250 patients, 99% showed

a positive response according to at least 1 of the 15 criteria,

whereas 94% were classified as a nonresponder by at least 1

criterion. Similarly, 95% of patients showed a positive response

by at least 2 of the 15 criteria, whereas 87% were classified as

nonresponders by at least 2 criteria.

Example Calculation of -ValueAn example of the -value calculation between response crite-

rion 3 (decrease in LVESV

10%, no death due to heart failure)and response criterion 13 (increase in 6-minute walk distance

10%, no heart failure death, no transplant) is given in Table 3.

Response criterion 3 identified 62% of patients who received

CRT as responders (and thus, 38% of the patients as nonre-

sponders), and criterion 13 also identified 62% of patients as

responders. The expected agreement due to chance alone was

therefore 0.620.620.380.3853%. The observed agree-

ment was 0.390.1554%. Equation 1 then shows that 0.02

for Table 3, which suggests poor agreement after accounting for

the level of agreement expected due to chance.

Agreement Among the 15 Response Criteria

The 15 response criteria showed poor agreement as a group(Figures 2 and 3; mean 0.220.24, median0.14,

range0.2 to 0.97). Seventy-nine (75%) of the 105

-values were classified as having poor agreement, whereas

22 -values (21%) were classified as having moderate agree-

ment (Figure 3). Only 4 pairs of response criteria of the 105

total pairs (4%) showed strong agreement, and 2 of these 4

pairs were comparisons between a response criterion that

excluded mortality and the same exact criterion that defined

death as a nonresponse. The 7 echocardiographic response

criteria also showed poor agreement among each other

(Figure 4; mean 0.350.28, median0.29, range0.2

to 0.88). The 7 clinical response criteria showed moderateagreement (Figure 4; mean 0.440.23, median0.43,

Table 2. -Values Used to Define Strong, Moderate, and

Poor Agreement29

Level of Agreement

Strong (clinically acceptable) 0.75

Moderate (questionable value) 0.40.75

Poor (not clinically acceptable) 0.40

Table 3. Example Comparing Response Criterion 3 WithResponse Criterion 13

Criterion 13

Responders Nonresponders Sum

Criterion 3

Responders 101 61 162

Nonresponders 60 39 99

Sum 161 100 261

Example comparing response criterion 3 (decrease in LVESV 10%, no

death due to heart failure) with response criterion 13 (increase in 6-minute

walk distance 10%, no death due to heart failure, no transplant) shows that

0.02 based on Equations 1 and 2. This -value of 0.02 demonstrates poor

agreement between the 2 criteria despite the fact that each criterion had nearlyidentical response rates of approximately 62%.

Table 4. Response Rates for the Different Criteria

Response Criteria

Response

Rate, %

No. Evaluable

(% of Total)

Echocardiographic

1LVEF 5 units 51 286 (67)

1LVEF 15% (relative) 54 286 (67)

2LVESV

10%, no HF death 62 291 (68)2LVESV 15% 56 286 (67)

LVESV 115% of baseline 91 286 (67)

2LVEDV 15% 49 286 (67)

1Stroke volume 15% 34 286 (67)

Clinical

2NYHA1 71 385 (90)

2NYHA1, no HF death 70 390 (92)

2NYHA1 and 16MWD 25% 33 348 (82)

2NYHA1 and 16MWD 25%, no HFdeath

32 353 (83)

16MWD 10%, no HF death, no transplant 61 353 (83)

Two of the following 3: 2NYHA1, 16MWD50 m, 2QOL 15

63 339 (80)

Clinical composite score improved 69 426 (100)

Combined

1LVEF 5 units or 16MWD 50 m and2NYHA1 or 2QOL 10

71 250 (59)

Abbreviations as in Table 1.

Fornwalt et al Poor Agreement Among CRT Response Criteria 1987



5/8

range0.14 to 0.97). Agreement between echocardiographic

and clinical criteria was poor (Figure 4; mean 0.050.05,median0.04, range0.03 to 0.17), with all 49 -values

showing poor agreement. The agreement among the echocar-

diographic parameters was not significantly different from

the agreement among the clinical parameters (uncorrected

P0.35). The response criterion based on a combination of

both echocardiographic and clinical measures showed signif-

icantly better agreement (P0.003) with clinical response

criteria (0.440.10) than with echocardiographic responsecriteria (0.210.12). Bootstrap resampling of the -statistic

comparing clinical composite response and a 15% reduction

in LVESV justified that the sample size was large enough toestimate the -value (Kolmogorov-Smirnov test for normality

P0.15).

Subgroup Analysis Excluding Response CriteriaQuantified Short Term and at 3 MonthsExclusion of the short-term and 3-month response criteria did

not significantly affect the results. After exclusion of 1

short-term (criterion 8 in Table 1) and two 3-month (criteria

15 and 17 in Table 1) response measures, the agreement

among the 12 remaining response criteria was poor as a group

(mean 0.220.26, median0.14, range0.03 to 0.97).

Agreement among the 6 remaining echocardiographic re-

sponse criteria was moderate (mean 0.440.24, medi-

an0.47, range0.16 to 0.88). Agreement among the 6

remaining clinical response criteria was also moderate (mean

0.420.27, median0.32, range0.14 to 0.97). Finally,

agreement between echocardiographic and clinical criteria

remained poor (mean 0.050.05, median0.04,

range0.03 to 0.17).

DiscussionThe major findings of this study are as follows: (1) The 26

most-cited publications on predicting response to CRT used

17 different primary response criteria, and the level of

agreement not due to chance among 15 of these response

criteria was poor 75% of the time and strong only 4% of the time

Figure 2. Agreement among the 15

response criteria was poor. The

-axisshows the following ranges delineatedby dotted lines: strong agreement(0.75), moderate agreement(0.40.75), and poor agreement(0.4). The worst agreement wasbetween echocardiographic (Echo) andclinical (Clin) parameters. *P0.001 vsEcho vs Echo and Clin vs Clin.

Figure 3. Agreement among the 15response criteria was classified as poor for75% of the 105 possible comparisons.-Values are color-coded according to thefollowing ranges: greenstrong agreement(0.75), yellowmoderate agreement(0.40.75), and redpoor agreement(0.4). Echo indicates echocardiographicresponse criteria; Clin, clinical responsecriteria.




6/8

in the 426 patients enrolled in the PROSPECT study; (2)

agreement between echocardiographic and clinical response

criteria was poor and nearly equal to the level of agreementexpected by chance; (3) the percentage of patients defined as

having a positive response to CRT ranged from 32% to 91% for

the 15 response criteria; and (4) 99% of patients were classified

as a responder by at least 1 of the 15 criteria, whereas 94% were

classified as a nonresponder by at least 1 criterion.

Comparison to the LiteratureTo the best of our knowledge, no study has quantified

agreement among response criteria with a -coefficient;

however, a recent study by Bleeker et al2 aimed to quantify

the agreement between echocardiographic and clinical

measures of response to CRT. The authors compared a

decline in New York Heart Association class (clinical

response) with a 15% decrease in LVESV (echocardio-

graphic response) in 144 consecutive patients undergoing

CRT. The authors concluded that the agreement between

[clinical response and echocardiographic response] was

good based on the observed agreement of 76%. However,

their data show that clinical and echocardiographic re-

sponses would be expected to agree 52% of the time on the

basis of chance alone. The study did not calculate -values

to account for this expected level of agreement due to

chance. We estimated the -value to be 0.50 from their

data, which is higher than the value of 0.17 observed in the

present study. However, the main conclusion that shouldbe drawn from both studies is similar: The agreement

between echocardiographic and clinical criteria for defin-

ing a positive response to CRT is only slightly better than

that expected by chance alone.In the MIRACLE trial (Multicenter InSync Randomized

Clinical Evaluation), correlation between the change in left

ventricular end-diastolic volume and change in New York

Heart Association class after 6 months of CRT was weak

(r0.13).30 In addition, the correlation between the change

in distance walked in 6 minutes and change in left

ventricular ejection fraction was weak (r0.15).30 These

data are consistent with the present results, which show

poor agreement between clinical and echocardiographic

response criteria.

Previous studies have reported different rates of re-

sponse to CRT when different definitions of response are

used within the same population. For example, the PROS-

PECT study reported that 56% of patients were echocar-

diographic responders (defined by a reduction in LVESV

of at least 15%), whereas 69% of patients were clinical

responders (defined by an improvement in the clinical

composite score).10 Thus, one would expect these mea-

sures to show poor agreement because of the different

response rates. However, the actual response rate does not

tell the entire story: Table 3 shows 2 different response

criteria with identical response rates of 62%, and despite

the identical response rates, the criteria show very poor

agreement (0.02). Thus, assessment of agreement with

the

-statistic provides valuable information in addition tothe overall response rate of the population.

Figure 4. Agreement among the response criteria was poor 75% of the time and strong only 4% of the time. -Values are color-codedaccording to the following ranges: Greenstrong agreement (0.75), yellowmoderate agreement (0.40.75), and redpooragreement (0.4). LVEF indicates left ventricular ejection fraction; HF, heart failure; NYHA, New York Heart Association; LVESV, leftventricular end-systolic volume; LVEDV, left ventricular end-diastolic volume; 6MWD, distance walked in 6 minutes; and QOL, quality-of-life score.




7/8

Other Inconsistencies in Defining Responseto CRT

Length of Follow-UpAnother area of inconsistency in defining response to CRT is the

length of the follow-up period after which a patient is deemed

either a responder or a nonresponder. Some studies focused on

short-term (1 to 2 days) response,1,4,21,22

whereas most focusedon 3-month59,18 or 6-month24,1017,20,2327 response. CRT has

been shown to have persistent, increasing benefits with a longer

mean follow-up period of 29.5 months.31 We defined response at

6 months because this was the prespecified follow-up period for

the PROSPECT study. We also performed a subgroup analysis

after excluding criteria that were assessed in the literature at

short-term and 3-month follow-up only, and this did not change

the present results. Future studies will be needed to address

agreement among the different lengths of follow-up.

MortalityWhether death should be considered a nonresponse to CRT is

another area in which there is inconsistency. There are at least 3different methods that authors have used to incorporate death

into their response criteria: (1) Death due to worsening heart

failure is included in the nonresponder group,11,16,17,20,23,27 (2)

death due to any cause is included in the nonresponder group,24

and (3) deaths are excluded from analysis.3,5,6,8,9,26 Moreover,

numerous publications fail to specify how death was incorpo-

rated into response criteria despite enrolling consecutive patients

and following them for a 3- to 6-month period.2,1214,18 Although

inclusion of all-cause mortality as a criterion for nonresponse

may not be appropriate, a patient who dies of progressive heart

failure should, objectively, be classified as a nonresponder.

Regardless, there is no consistent method for incorporatingmortality into the definition of response to CRT, and this needs

to be standardized.

A Consensus Definition of Response to CRTBecause heart failure is a debilitating life-threatening disease,

an effective heart failure therapy should treat both symptoms

and quality and duration of life.32 Thus, measures of re-

sponse to CRT should either directly measure outcomes or

have a surrogate relationship with benefits in heart failure

symptoms, quality of life, and duration of life. The clinical

composite score33 is a measure of response that accounts for

all of these factors and may be the best overall choice for

defining response in future CRT trials.

Study LimitationsThe results of the present study are limited by the fact that we

used data from a single study (PROSPECT). However,

PROSPECT was a multicenter study that enrolled 457 wide-

QRS patients from 53 different centers across Europe, Hong

Kong, and the United States. We would expect similar results

from other large, multicenter databases.

The present results show that many different methods to

define a positive response to CRT are being used in the

literature and show poor agreement among each other. This

begs the question, which method should we use in the futureto determine whether a patient benefited from CRT? The

present study did not attempt to address this question, and

future studies will need to explore this important issue.

The definition of clinically acceptable agreement based on the

-coefficient is not standardized. Fleiss29 proposed a threshold of

0.75 to define strong evidence for agreement that is not due to

chance, which is what we used; however, this threshold is

somewhat arbitrary. Landis and Koch34 proposed that any value

of above 0.60 suggests substantial agreement, and 0.80

implies almost perfect agreement. However, the use of a

different threshold, such as 0.6, to define strong agreement

would not significantly alter the present results; 4 of the 105

-statistics that we calculated were greater than 0.8, and only 8

were 0.6. Thus, regardless of the threshold used, we observed

mostly poor agreement among the 15 different response criteria.

ConclusionsThe 26 most-cited publications on predicting response to

CRT define response using 17 different criteria. Agreement

between these different published methods to define response

to CRT is poor 75% of the time and strong only 4% of thetime. This inconsistency in the definition of response to CRT

severely limits the ability to generalize results over multiple

studies and hinders progress in the field.

Sources of FundingThis work was supported by grants from the American Heart Associa-tion (Grant-in-Aid No. 0855386E) and the National Institutes of Health(HL089160) to Dr Oshinski. Dr Fornwalt was supported in part byNational Institutes of Health training grant No. 5 T32 GM008169.

DisclosuresDr Gerritse is an employee of Medtronic Inc and owns companystock. The remaining authors report no conflicts.

References1. Bax JJ, Marwick TH, Molhoek SG, Bleeker GB, van Erven L, Boersma

E, Steendijk P, van der Wall EE, Schalij MJ. Left ventricular dyssyn-

chrony predicts benefit of cardiac resynchronization therapy in patients

with end-stage heart failure before pacemaker implantation. Am J

Cardiol. 2003;92:1238 1240.

2. Bleeker GB, Bax JJ, Fung JWH, van der Wall EE, Zhang Q, Schalij MJ, Chan

JYS, Yu CM. Clinical versus echocardiographic parameters to assess response to

cardiac resynchronization therapy.Am J Cardiol. 2006;97:260263.

3. Gorcsan J, Tanabe M, Bleeker GB, Suffoletto MS, Thomas NC, Saba S,

Tops LF, Schalij MJ, Bax JJ. Combined longitudinal and radial dyssyn-

chrony predicts ventricular response after resynchronization therapy.

J Am Coll Cardiol. 2007;50:1476 1483.

4. Suffoletto MS, Dohi K, Cannesson M, Saba S, Gorcsan J. Novel speckle-

tracking radial strain from routine black-and-white echocardiographic

images to quantify dyssynchrony and predict response to cardiac resyn-chronization therapy. Circulation. 2006;113:960 968.

5. Notabartolo D, Merlino JD, Smith AL, DeLurgio DB, Vera FV, Easley

KA, Martin RP, Leon AR. Usefulness of the peak velocity difference by

tissue Doppler imaging technique as an effective predictor of response to

cardiac resynchronization therapy. Am J Cardiol. 2004;94:817 820.

6. Yu CM, Chan YS, Zhang Q, Yip GWK, Chan CK, Kum LCC, Wu L, Lee

APW, Lam YY, Fung JWH. Benefits of cardiac resynchronization therapy

for heart failure patients with narrow QRS complexes and coexisting systolic

asynchrony by echocardiography. J Am Coll Cardiol. 2006;48:22512257.

7. Yu CM, Fung JWH, Chan CK, Chan YS, Zhang Q, Lin H, Yip GWK,

Kum LCC, Kong SL, Zhang Y, Sanderson JE. Comparison of efficacy of

reverse remodeling and clinical improvement for relatively narrow and

wide QRS complexes after cardiac resynchronization therapy for heart

failure. J Cardiovasc Electrophysiol. 2004;15:1058 1065.

8. Yu CM, Zhang Q, Chan YS, Chan CK, Yip GWK, Kum LCC, Wu EB,

Lee PW, Lam YY, Chan S, Fung JWH. Tissue Doppler velocity issuperior to displacement and strain mapping in predicting left ventricular




8/8

reverse remodelling response after cardiac resynchronisation therapy.

Heart. 2006;92:14521456.

9. Yu CM, Zhang Q, Fung JWH, Chan HCK, Chan YS, Yip GWK, Kong

SL, Lin H, Zhang Y, Sanderson JE. A novel tool to assess systolic

asynchrony and identify responders of cardiac resynchronization therapy

by tissue synchronization imaging.J Am Coll Cardiol. 2005;45:677684.

10. Chung ES, Leon AR, Tavazzi L, Sun JP, Nihoyannopoulos P, Merlino J,

Abraham WT, Ghio S, Leclercq C, Bax JJ, Yu CM, Gorcsan J, Sutton

MS, De Sutter J, Murillo J. Results of the Predictors of Response to CRT

(PROSPECT) trial. Circulation. 2008;117:26082616.

11. Diaz-Infante E, Mont L, Leal J, Garcia-Bolao I, Fernandez-Lozano I,

Hernandez-Madrid A, Perez-Castellano N, Sitges M, Pavon-Jimenez R,

Barba J, Cavero MA, Moya JL, Perez-Isla L, Brugada J; SCARS Inves-

tigators. Predictors of lack of response to resynchronization therapy. Am J

Cardiol. 2005;95:1436 1440.

12. Molhoek SG, Bax JJ, Bleeker GB, Boersma E, van Erven L, Steendijk P,

van der Wall EE. Comparison of response to cardiac resynchronization

therapy in patients with sinus rhythm versus chronic atrial fibrillation.

Am J Cardiol. 2004;94:15061509.

13. Molhoek SG, Bax JJ, Boersma E, Van Erven L, Bootsma M, Steendijk P,

Van Der Wall EE, Schalij MJ. QRS duration and shortening to predict

clinical response to cardiac resynchronization therapy in patients with

end-stage heart failure. Pacing Clin Electrophysiol. 2004;27:308313.

14. Molhoek SG, Bax JJ, van Erven L, Bootsma M, Boersma E, Steendijk P, van

der Wall EE, Schalij MJ. Comparison of benefits from cardiac resynchroni-

zation therapy in patients with ischemic cardiomyopathy versus idiopathicdilated cardiomyopathy.Am J Cardiol. 2004;93:860863.

15. Bax JJ, Bleeker GB, Marwick TH, Molhoek SG, Boersma E, Steendijk P,

van der Wall EE, Schalij MJ. Left ventricular dyssynchrony predicts

response and prognosis after cardiac resynchronization therapy. J Am Coll

Cardiol. 2004;44:1834 1840.

16. Bleeker GB, Kaandorp TAM, Lamb HJ, Boersma E, Steendijk P, de Roos

A, van der Wall EE, Schalij MJ, Bax JJ. Effect of posterolateral scar

tissue on clinical and echocardiographic improvement after cardiac resyn-

chronization therapy. Circulation. 2006;113:969 976.

17. Ypenburg C, Schalij MJ, Bleeker GB, Steendijk P, Boersma E, Dibbets-

Schneider P, Stokkel MPM, van der Wall EE, Bax JJ. Impact of viability

and scar tissue on response to cardiac resynchronization therapy in

ischaemic heart failure patients. Eur Heart J. 2007;28:3341.

18. White JA, Yee R, Yuan XP, Krahn A, Skanes A, Parker M, Klein G,

Drangova M. Delayed enhancement magnetic resonance imaging predicts

response to cardiac resynchronization therapy in patients with intraven-tricular dyssynchrony. J Am Coll Cardiol. 2006;48:19531960.

19. Thomson Reuters. Web of Science, Science Citation Index Expanded

database; ISI Web of Knowledge. Available at: http://apps.isiknowledge.

com. Available at: http://apps.isiknowledge.com. Accessed June 15, 2009.

20. Bleeker GB, Mollema SA, Holman ER, Van De Veire N, Ypenburg C,

Boersma E, van der Wall EE, Schalij MJ, Bax JJ. Left ventricular

resynchronization is mandatory for response to cardiac resynchronization

therapy: analysis in patients with echocardiographic evidence of left

ventricular dyssynchrony at baseline. Circulation. 2007;116:14401448.

21. Dohi K, Suffoletto MS, Schwartzman D, Ganz L, Pinsky MR, Gorcsan J.

Utility of echocardiographic radial strain imaging to quantify left ven-

tricular dyssynchrony and predict acute response to cardiac resynchroni-

zation therapy. Am J Cardiol. 2005;96:112116.

22. Gorcsan J, Kanzaki H, Bazaz R, Dohi K, Schwartzman D. Usefulness of

echocardiographic tissue synchronization imaging to predict acute response

to cardiac resynchronization therapy. Am J Cardiol. 2004;93:11781181.

23. Henneman MM, Chen J, Dibbets-Schneider P, Stokkel MR, Bleeker GB,

Ypenburg C, van der Wall EE, Schalij MJ, Garcia EV, Bax JJ. Can LV

dyssynchrony as assessed with phase analysis on gated myocardial perfusion

SPECT predict response to CRT? J Nucl Med. 2007;48:11041111.

24. Lecoq G, Leclercq C, Leray E, Crocq C, Alonso C, de Place C, Mabo P,

Daubert C. Clinical and electrocardiographic predictors of a positive

response to cardiac resynchronization therapy in advanced heart failure.

Eur Heart J. 2005;26:10941100.

25. Marcus GM, Rose E, Viloria EM, Schafer J, De Marco T, Saxon LA,

Foster E; VENTAK CHF/CONTAK-CD Biventricular Pacing Study

Investigators. Septal to posterior wall motion delay fails to predict reverse

remodeling or clinical improvement in patients undergoing cardiac resyn-

chronization therapy. J Am Coll Cardiol. 2005;46:22082214.

26. Stellbrink C, Breithardt OA, Franke A, Sack S, Bakker P, Auricchio A,

Pochet T, Salo R, Kramer A, Spinelli J; CPI Guidant Congestive Heart

Failure Research Group. Impact of cardiac resynchronization therapy

using hemodynamically optimized pacing on left ventricular remodeling

in patients with congestive heart failure and ventricular conduction dis-

turbances. J Am Coll Cardiol. 2001;38:19571965.27. Ypenburg C, Roes SD, Bleeker GB, Kaandorp TAM, de Roos A, Schalij

MJ, van der Wall EE, Bax JJ. Effect of total scar burden on contrast-

enhanced magnetic resonance imaging on response to cardiac resynchro-

nization therapy. Am J Cardiol. 2007;99:657660.

28. Hulley SB, Cummings SR, Browner WS, Grady D, Hearst N, Newman

TB. Designing Clinical Research: An Epidemiologic Approach. 2nd ed.

Philadelphia, Pa: Lippincott Williams & Wilkins; 2001.

29. Fleiss JL. Statistical Methods for Rates and Proportions. 2nd ed. New

York, NY: Wiley; 1981.

30. St John Sutton MG, Plappert T, Abraham WT, Smith AL, DeLurgio DB,

Leon AR, Loh E, Kocovic DZ, Fisher WG, Ellestad M, Messenger J,

Kruger K, Hilpisch KE, Hill MR. Effect of cardiac resynchronization

therapy on left ventricular size and function in chronic heart failure.

Circulation. 2003;107:19851990.

31. Cleland JG, Daubert JC, Erdmann E, Freemantle N, Gras D, Kappenberger

L, Tavazzi L. Longer-term effects of cardiac resynchronization therapy onmortality in heart failure [the CArdiac REsynchronization-Heart Failure

(CARE-HF) trial extension phase]. Eur Heart J. 2006;27:19281932.

32. Anand IS, Florea VG, Fisher L. Surrogate end points in heart failure.

J Am Coll Cardiol. 2002;39:1414 1421.

33. Packer M. Proposal for a new clinical end point to evaluate the efficacy

of drugs and devices in the treatment of chronic heart failure. J Card Fail.

2001;7:176182.

34. Landis JR, Koch GG. The measurement of observer agreement for cate-

gorical data. Biometrics. 1977;33:159174.

CLINICAL PERSPECTIVEA literature search revealed that the 26 most-cited publications on predicting response to cardiac resynchronization therapy

defined response using 17 different criteria. No study has investigated agreement among these various response criteria, and wehypothesized that this agreement would be poor. The agreement among 15 of the 17 response criteria was assessed in 426 patients

from the PROSPECT study using the Cohen -coefficient (2 of the 17 response criteria were not calculable from PROSPECT

data). Response rates for the entire population were highly varied and ranged from 32% to 91% for the 15 criteria. Ninety-nine

percent of patients showed a positive response by at least 1 of the 15 criteria, whereas 94% were classified as a nonresponder

by at least 1 criterion. -Values were calculated for all 105 possible comparisons among the 15 response criteria and classified

into standard ranges: Poor agreement (0.4), moderate agreement (0.40.75), and strong agreement (0.75).Seventy-five percent of the comparisons showed poor agreement, 21% showed moderate agreement, and only 4% showed strong

agreement. Thus, agreement between different methods to define response to cardiac resynchronization therapy is poor 75% of

the time and strong only 4% of the time, which severely impairs the ability to generalize results over multiple studies. This lack

of standardization hinders progress in cardiac resynchronization therapy research and needs to be resolved.

Go to http://cme.ahajournals.org to take the CME quiz for this article.



circulation 2010 fornwalt 1985 91

Documents