Maximal Reliability of Unit-Weighted Composites

Peter M. Bentler

University of California, Los Angeles

SAMSI, NOV 05

In S. Y. Lee (Ed.) (in press). Handbook of Structural Equation Models.

“Reliability” = Internal Consistency of the composite X based on p components

1 1 2 2 p pX=w Y +w Y +...+w Y (X=w Y)

(all 1); , uncorrelatedY C U p C U

ccov( ) ( psd, rank<p; diag or pd)cY

1cxx

w w w w

w w w w

(Bentler, 1968, eq. 12; Heise & Bohrnstedt, 1970, eq. 32;

this is H-B's coefficient - special case is McDonald's 1999 )

c

is a not necessarily a measure of homogeneity

or unidimensionality. A coefficient for uni-

dimensional common scores requires, in addition,

that rank( ) 1.

What , defined below, has to say (or, no

xx

t say)

about unidimensionality has been an issue

widely discussed in the literature.

Since uniqueness = specificity + error, i.e.,

. If both are p.d.,s e *

*

is a lower bound to "true" reliability,

where

1

xx xx

exx

w w

w w

In this talk I will emphasize

, the unit vector.

This excludes maximal reliability.

w 1

Under mild assumptions consistent with the above, Novick & Lewis (1967) showed that

xx

'1

1 '

where, with diag( ) and a unit vector,

is the well-known alpha coefficient. This says

nothing about as a lower bound to a uni-

dimensional component of a multidimensional

1 D 1p

p 1 1

D 1

c.

cWhat models for are consistent with ?

Let be the average covariance, and defineij

=( ), where =diag( ). ThenijD I D

. It follows thatc ijD I

.xx

cThis choice does not guarantee that has

any interesting properties, e.g., psd, rank 1.

This is not the only condition for .xx

If ,

where are communalities, we only needc H

H

D D

D

c

( ) /

Interesting seem not to be known.

Another well known case of equivalence

for a rank 1 is given below.

H ij

H

1 D 1 p

D

cActually, I will argue that should be psd,

which is not guaranteed for arbitrary .HD

This Hoyt-Guttman-Cronbach α is by far the

most widely used measure of the internal con-

sistency reliability of a composite X. In prac-

tice, one substitutes the sample covariance

matrix S and its diagonal DS to get

'ˆ 1 .

1 '

This is not necessarily a lower bound to

population internal consistency.

S1 D 1p

p 1 S1

Some Recent References• Becker, G. (2000). Coefficient alpha: Some terminological ambiguities and

related misconceptions. Psychological Reports, 86, 365-372.• Bonett, D. G. (2003). Sample size requirement for testing and estimating

coefficient alpha. Journal of Educational and Behavioral Statistics, 27, 335-340. • Enders, C. K., & Bandalos, D. L. (1999). The effects of heterogeneous item

distributions on reliability. Applied Measurement in Education, 12, 133-150. • Green, S. B., & Hershberger, S. L. (2000). Correlated errors in true score models

and their effect on coefficient alpha. Structural Equation Modeling, 7, 251-270. • Hakstian, A. R., & Barchard, K. A. (2000). Toward more robust inferential

procedures for coefficient alpha under sampling of both subjects and conditions. Multivariate Behavioral Research, 35, 427-456.

• Kano, Y., & Azuma, Y. (2003). Use of SEM programs to precisely measure scale reliability. In H. Yanai, A. Okada, K. Shigemasu, Y. Kano, & J. J. Meulman (Eds.), New developments in psychometrics (pp. 141-148). Tokyo: Springer-Verlag.

• Komaroff, E. (1997). Effect of simultaneous violations of essential tau-equivalence and uncorrelated error on coefficient alpha. Applied Psychological Measurement, 21, 337-348.

• Miller, M. B. (1995). Coefficient alpha: A basic introduction from the perspectives of classical test theory and structural equation modeling. Structural Equation Modeling, 2, 255-273.

• Raykov, T. (1998). Coefficient alpha and composite reliability with interrelated nonhomogeneous items. Applied Psychological Measurement, 22, 375-385.

• Raykov, T. (2001). Bias of coefficient α for fixed congeneric measures with correlated errors. Applied Psychological Measurement, 25, 69-76.

• Schmitt, N. (1996). Uses and abuses of coefficient alpha. Psychological Assessment, 8, 350-353.

• Shevlin, M., Miles, J. N. V., Davies, M. N. O., & Walker, S. (2000). Coefficient alpha: A useful indicator of reliability? Personality & Individual Differences, 28, 229-237.

• Schmitt, N. (1996). Uses and abuses of coefficient alpha. Psychological Assessment, 8, 350-353.

• Yuan, K.-H., Guarnaccia, C. A., & Hayslip, B. J. (2003). A study of the distribution of sample coefficient alpha with the Hopkins Symptom Checklist: Bootstrap versus asymptotics. Educational & Psychological Measurement, 63, 5-23.

Some Advantages of

• Widely taught and known

• Simple to compute and explain

• Available in most computer packages is a lower bound to population internal

consistency under reasonable conditions (not true of sample )

• Not reliant on researcher judgments (i.e., same data, same result for everyone)

Note: Independence of researcher judgment is not totally correct

• Different covariance matrices, or scalings of the variables, yield different

and hence different

• Sometimes alpha is computed from the correlation matrix, and not the covariance matrix, implying the sum X is a sum of standardized variables

Some Disadvantages of • α is not a measure of homogeneity or

unidimensionality• It may underestimate or overestimate uni- dimensional reliability• Since it will overestimate reliability if , the average covariance, is spuriously high;

or underestimate it if the average is spuriously low

2 /ijp 1 1 ij

This illustrates the advantages and disadvantages of alpha

• The plus: The average covariance is what it is, period. No judgment can change it. Hence, no judgment will change the reported .

• The minus: A researcher’s model of sources of variance does not influence the reported reliability (alpha) when, perhaps, it should.

• Some examples from Kano/Azuma (2003) illustrate this.

If for a one factor model and possibly correlated residuals ( not necessarily diagonal)

with (p 1), then c

Three examples from Kano & Azuma (2003) showing alpha as accurate, and as an overestimate. 1-factor reliability (rho= omega) ignoring correlated errors can overestimate.

Alpha as a unidimensional lower-bound: the comparison of in a 1-factor model

If , and is px1, is diagonal,

. Example: McDonald (1999) showed

with equality when

11,

11

1

11( ) { ( )} 01 i

p1 1 Var

p

c1

11

11

11a 11b 11c

If ,

there is no known general relation of to .

In fact, (= ) is itself not defined. We can

have , , etc., depending on how the

1-factor model is computed (ML,LS,GLS,etc.)

[Th

11

at is, how ( ) is fit to under this

misspecification. No sample data are involved.]

This nonuniqueness of under misspecification

is a weakness of this model-based coefficient.

Thus,

• α is a good indicator of association

but it is not clear what interesting sets of models it represents in the general case

• A model based coefficient such as

provides a clear partitioning of variance,

but it also can be (1) misleading (e.g., Kano

-Azuma), (2) not relevant

( )ij

11( )

(e.g., )

To give up α for a model-based coefficient, the model must be correct for Σ. That is, it should fit the data (say, sample covariance matrix S). I would conclude:

• If is computed, the researcher must also provide evidence of acceptability of the 1-factor model.

• If a modified estimator

is computed, the researcher should provide an argument for the variance partition.

11

11ˆ( with nondiagonal )

Should the correlated error be part of the residual covariance Ψ (on left), or part of the common variance Σc (on right)? Substantive reasoning should determine the variance partitioning. This is more than just model fit.

What other model-based coefficients could be

used instead? • Arbitrary latent variable model (Raykov &

Shrout, 2002; EQS 6)• Dimension-free lower bound (Bentler,

1972; bias correction Shapiro & ten Berge, 2000)

• Greatest lower bound (Woodhouse & Jackson, 1977; Bentler & Woodward, 1980 etc.; ten Berge, Snijders, & Zegers, 1981; bias correction Li & Bentler, 2001)

Arbitrary LV Model

c

model c

Suppose an arbitrary SEM (e.g., a LISREL

model) contains additive errors, whether

correlated or not. That is, = ( )= .

It must be acceptable statistically (fit ). Then

( ) /( )

is a

S

1 1 1' 1

11 more meaningful coefficient than (= ).

It is probably more meaningful than .

Dimension-free Lower Bound

+

*11 +

for some arbitrary, unknown

number of factors

is diagonal, and

min subject to above.

This has the property that ( , )

c

c

cxx

xx

1 1

1 1

Greatest Lower Bound

This is a constrained version of the dimension-free coefficient. In addition to

glb

+

*glb

for some number of factors,

is not only diagonal, but also psd. Thus

,

with equality when has no Heywood

variables (no negative variances).

Also, .xx

+ glb

Every researcher will get the same values of

and . They are based on a model that

does not depend on researcher choices.

Since off-diag( ) [off-diag( ), as replaces

in practice] is exactly rep

S S

+ glb

roduced, all

covariation is assumed to stem from common

variance. However, and do not allow

nondiagonal .

Maximal Unidimensional Reliability

• The problem with α and the multi-dimensional coefficients seems to be that they do not represent unidimensional reliability

• Although not obvious, unidimensional reliability can be defined for multi-

dimensional latent variable models. That is the main new result in this talk.

T E

1,

p

iX X 1

p

iT T 1

p

iE E

2

21T T E

Xxx

1 1 1 1

1 1 1 1

Repeating the Basic Setup

Xi = Ti + Ei,

X = T + E ,

2 221

2

but we compute something like

( )( )1d

p

T i u

X11

1 11 1 1

1 1 1 1 1 1 1 1

0. ., truth is ui e

Can we have something like1-Factor Based Reliability when the latent variables are multidimensional?

Maximal Unit-weighted Reliability x (p x k) for some k

(“small k” <

or “large k”)

.5(2 1 8 1)p p

for some acceptable k-factor model

[ | ] , where is (px1) and is (px(k-1))

contains unrestricted free parameters.01 , that is, the k-1 columns of sum to zero.

contains free parameters subject to (k-1)(k-2)/2 restrictions (usual EFA identification conditions)

Reliability under this Parameterization

X 1 x 1 1

1[ | ] [ | ] [ | 0] [ | 0]

[ | 0]

p

i

x

1 1 1 1 1

1X X d dX T E X is based on 1 factor!

2 2

2 2 2d

X

T X

X Xkk

2

2

( )

( ) ( )

1 1 1

1 1 1 1 1

2

2 2

ˆˆ ˆ ˆ1

ˆ ˆ ˆ ˆˆ

X

X

X

kk

1 1 1 1

1 1 1 1

This is Maximal Unit-Weighted Reliability

Let and let t be a normal vector ( 1)t t

Then the factor loading vector t that maximizes 2( )1

is given by 1/ 2( )1 1 1 and the residual factors

where have zero column sums

( 0).1

Applications to Arbitrary Structural Models

Any structural model with additive errors: ( )

Linear structural model with additive errors:

( )

Let 1 1( ) ( )I B I B with

1 1/ 2( )I B

Greatest lower bound:

1 max min 1 ,

with ( ) and

glb

1 1 1 1

1 1 1 1

psd

ˆ ˆ ˆS

EFA Example (It’s all in EQS)/TITLEMaximum Reliability EFA Model SetupNine Psychological Variables/SPECIFICATIONSVARIABLES= 9; CASES=101;MATRIX=COVARIANCE; METHOD=ML;/EQUATIONSV1=*F1+*F2+0F3+E1;V2=*F1+*F2+*F3+E2;V3=*F1+*F2+*F3+E3;V4=*F1+*F2+*F3+E4;V5=*F1+*F2+*F3+E5;V6=*F1+*F2+*F3+E6;V7=*F1+*F2+*F3+E7;V8=*F1+*F2+*F3+E8;V9=*F1+*F2+*F3+E9;

/VARIANCESF1 TO F3 = 1.0;E1 TO E9 = .5*;/CONSTRAINTS(V1,F2)+(V2,F2)+(V3,F2)+(V4,F2)+(V5,F2)+(V6,F2)+(V7,F2)+(V8,F2)+(V9,F2)=0;(V2,F3)+(V3,F3)+(V4,F3)+(V5,F3)+(V6,F3)+(V7,F3)+(V8,F3)+(V9,F3)=0;

/MATRIX1.00 .75 1.00 .78 .72 1.00 .44 .52 .47 1.00 .45 .53 .48 .82 1.00 .51 .58 .54 .82 .74 1.00 .21 .23 .28 .33 .37 .35 1.00 .30 .32 .37 .33 .36 .38 .45 1.00 .31 .30 .37 .31 .36 .38 .52 .67 1.00/END

for 9 Psychological Variables ˆ .886

Variable Number

1-Factor

Model

3-Factor

Model

1 .636 .727

2 .697 .738

3 .667 .754

4 .867 .789

5 .844 .767

6 .879 .803

7 .424 .492

8 .466 .597

9 .462 .635

Sum 5.942 6.302

227 190.6

ˆ .88011

212 1.6

ˆ .939kk