Coefficients Alpha and Omega, an Empirical Comparison

Larry R Nelson1

Curtin University, Western Australia

Document date: 14 October 2020 – working draft

website: www.lertap5.com DOI: 10.13140/RG.2.1.1957.5929 (now outdated)

Updated paper available as of 29 January 2021

The new paper includes more datasets and new methods (JASP and SPSS).

Update of 18 February 2018

This paper was originally published in June, 2016.

Since then, the special Omega1 macro found in Lertap5 has been enhanced and

is now substantially easier to use. The latest edition of the macro creates two

files for use with “R”, a free programming and data analysis system noted for,

among other things, its psychometric utility – numerous free resources in R

support a true variety of analysis tools useful for analysing results from tests and

surveys.

Interested readers will rush to consult this companion paper.

The original paper:

The standard index of reliability used in Lertap5 is a statistic commonly referred

to as “alpha”, “coefficient alpha”, and/or “Cronbach’s alpha”.

Alpha is discussed under the “Reliability” section of Chapter 7 in the Lertap5

manual (Nelson, 20002).

A somewhat more general discussion of “reliability” and Lertap5 is found at this

webpage. A favourite reference of mine, recommended basic reading, is the text

by Meyer (2010).

… new stuff, various citations which may be useful …

Meyer (2010, p.4): “Test score reliability refers to the degree of test score

consistency over many replications of a test of performance task.”

p.9: “reliability is situation-specific” & “a complete understanding of the meaning

of reliability is only possible through a complete specification of the process that

produced the test scores”

p.10 “Observed scores will vary due to the particular sample of items that

comprise ech test (i.e, measurement error due to the selection of ietms).

p.12 “there are many different interpretations of reliability and methods for

estimating it.”

1 Comments / questions may be sent to l.nelson@curtin.edu.au 2 References are found at this webpage.

An empirical look at coefficients alpha and omega, page 2.

p.16 the reliability coefficient is “an index that describes the similarity of true

and observed scores”.

p.20 “Reliability is defined as the squared correlation between observed scores

and true scores, and this index turns out to be the ratio of true score variance to

observed score variance.”

NOTE: the “index of reliability” is technically described in Hopkins, Stanley,

Hopkins (1990), p.117 as the “correlation of observed scores on a test

with corresponding universe scores”. Note that they use “universe score”

and “true score” interchangably. It’s output by Lertap5 where it’s labelled

as the “index of reliability”.

NOTE: Cronbach (his “current thoughts, 1997” Word file is in TechDocs).

has said “Just what is to be meant by reliability was a perrenial source of

dispute.” Violation of item independence usually makes the coefficient

somewhat too large. He points out that alpha is the squared correlation of

observed scores and true scores and then says “It might seem logical to

use the square root of alpha in reports of reliability findings, but that has

never become the practice.” (This is the index of reliability in Lertap.)

p.54: “The test-retest reliability coefficient is known as the coefficient of

stability.”

p.57: “… alternate forms reliability is … a coefficient of equivalence.”

p.63: reliability estimates based on part-tests (such as split-halves) “… are

referred to as internal-consistency estimates.” (Others may disagee.)

p.73: “Dimensionality refers to the number of factors or dimensions underlying

scores produced by the measurement procedure.”

p.74: “The assumption of unidimensionality is critical to selecting an appropriate

method for estimating reliability. This assumption is tested first, and it should be

tested prior to selecting a method for estimating reliability.”

p.75: “Cronbach’s alpha and other measures of internal consistency are not tests

of unidimensionality.”

p.75: “Reliability is underestimated when scores are not unidimensional.”

p.77: “… internal consistency estimates of reliability are artificially inflated in the

presence of postively correlated errors.”

p.86: “Cronbach’s alpha estimates the true reliability when measures are at least

essentially tau-equivalent (i.e., essentially tau-equivalent, tau-equivalent, or

parallel), the measure is unidimensional, and error scores are uncorrelated.” (It

becomes) “… a lower-bound to true reliability under the assumption of congeneric

measures.” And the simultaneous violation of these assumptions can lead to

alpha being an upper bound to true reliability.

Meyer does not refer to McDonald nor to omega. He does mention CFA.

Bandalos (2018)

p.156 “Reliability is a crucial property of test scores because it provides some

assurance that we would obtain similar scores if we were to measure again in the

same way.”

An empirical look at coefficients alpha and omega, page 3.

p.157 “… the term reliability in the context of measurement theory refers to the

dependability or consistency of measures across conditions.”

p.157 “Internal consistency measures of reliability reflect the degree to which the

items on a test are interrelated.”

p.162 “ … reliability can also be thought of as the the extent to which observed

scores reflect true rather than error scores.”

p.164 Here she also describes reliability as the “correlation between true and

observed scores”. (I think not.)

p.168 “… the reliability coefficient, a measure of the proportion of variance in the

observed scores that is due to differences in the true scores.”

p.169 Again she writes “… reliability is defined as the correlation between true

and observed scores”. (This statement is wrong.)

p.173 “Internal consistency coefficients assess the degree to which reponses are

consistent across items within a scale.”

… now returning to the original paper (with numerous modifications) …

Alpha is an extremely common measure of reliability, perhaps particularly appro-

priate when applied in cognitive testing, as in, for example, when assessing

student achievement in a particular subject area.

Coefficient omega is also used as a measure of reliability. It has gained promi-

nence after several authors have suggested, and shown, that it is a better index

of reliability. Here l refer readers to Dunn, Baguley, & Brunsden (2014), Geldhof,

Preacher, & Zyphur (2013)3, and Revelle & Zinbarg (2008).

In this paper I present results obtained from the application of five programs,

using them to get alpha and omega figures for several selected datasets. The

programs were: Lertap5; SAS Proc CALIS; two CRAN packages, MBESS and

psych, and JASP, a relative newcomer.

The datasets and results are shown below. In all cases I have reported results

without rounding them. Not all programs were used with all datasets. MBESS,

for example, proved to be somewhat problematic as I found it very slow when

working with large datasets. For SAS, I used the free University version.

3 The Geldhof et al. paper exhibits more balance than the others when it comes to the use of alpha.

An empirical look at coefficients alpha and omega, page 4.

Outcomes for selected datasets

1) Results for the MathsQuiz test with n=999 and items=15

Lertap5: alpha: 0.796; {95%CI: 0.78 to 0.81}; alpha(pc1): 0.807 SAS CALIS: omega: 0.798 MBESS: omega: 0.808; alpha: 0.796 psych4: omega: 0.803; alpha: 0.781; omega(h): 0.692

omegaSem (lavaan) could not invert matrix JASP: omega: 0.807; {95%CI: 0.778 to 0.813}

alpha: 0.796, standardized alpha: 0.781, alpha(glb): 0.853 links: the dataset; output from psych program; local data on L-sito C:\R

2) Results for the UniAA test with n=127 and items=30

Lertap5: alpha: 0.738; {95%CI: 0.67 to 0.80}; alpha(pc1): 0.769

SAS CALIS: omega: 0.746 MBESS: omega: 0.744; alpha: 0.736 psych: omega: 0.763; alpha: 0.737; omega(h): 0.407 omegaSem (lavaan) could not invert matrix JASP: omega: 0.744; {95%CI: 0.666 to 0.797}

alpha: 0.738, standardized alpha: 0.737, alpha(glb): 0.896

links: the dataset; output from psych program; local data on L-sito C:\R

3) Results for the UniBB test with n=132 and items=34

Lertap5: alpha: 0.816; {95%CI: 0.77 to 0.86}; alpha(pc1): 0.834 SAS CALIS: omega: 0.816 MBESS: omega: 0.819; alpha: 0.816

psych: omega: 0.831; alpha: 0.810; omega(h): 0.493 omegaSem (lavaan) could not invert matrix JASP: omega: 0.819; {95% CI: 0.767 to 0.857}

alpha: 0.816, standardized alpha: 0.814, alpha(glb): 0.937 links: the dataset; output from psych program; local data on L-sito C:\R

4) Results for the Zmed test with n=2,470 and items=100

Lertap5: alpha: 0.952; {95% CI: 0.95 to 0.96}; alpha(pc1): 0.958 with 25% sample: alpha: 0.95; {95% CI: 0.95 to 0.96} SAS CALIS: omega: 0.954 psych: omega: 0.957; alpha: 0.955; omega(h): 0.758 omegaSem: 0.957; alpha: 0.955; omega(h): 0.758 JASP: omega: 0.956; {95%CI: 0.953 to 0.958}

alpha: 0.955, standardized alpha: 0.955, alpha(glb): 0.972 links: the dataset; output from psych program; data on OSF

5) Results for the HalfTime test with n=424 and items=100

Lertap5 alpha: 0.935; {95%CI: 0.93 to 0.94}, alpha(pc1): 0.941 SAS CALIS omega: 0.915 MBESS5 omega: 0.934

psych omega: 0.939; alpha: 0.933; omega(h): 0.696 omegaSem (lavaan) could not invert matrix JASP: omega: 0.934; {95%CI: 0.925 to 0.943}

alpha: 0.935, standardized alpha: 0.934, alpha(glb): 0.982 links: the dataset; output from psych program; data on OSF

4 Psych omega values here are omega(t); alpha is standardized alpha 5 MBESS was unable to produce an alpha value for this dataset

An empirical look at coefficients alpha and omega, page 5.

6) Results for the NorthernRivers test with n=689 and items=40

Lertap5 alpha: 0.918; {95%CI: 0.91 to 0.93}, alpha(pc1): 0.920 SAS CALIS omega: 0.918 psych omega: 0.923; alpha: 0.918; omega(h): 0.739 omegaSem: 0.923; alpha: 0.918; omega(h): 0.739

JASP: omega: 0.919; {95% CI: 0.909 to 0.926} alpha: 0.918, standardized alpha: 0.918, alpha(glb): 0.951

links: the dataset; output from psych program; local data on L-sito

7a) Results for the LenguaBIg test with n=5,504 and items=50

Lertap5 alpha: 0.810; {95%CI: 0.80 to 0.82}; alpha(pc1): 0.846

SAS CALIS omega: 0.818 psych omega: 0.832; alpha: 0.822; omega(h): 0.717 omegaSem: 0.832; alpha: 0.822; omega(h): 0.717

JASP: omega: 0.815; {95%CI: 0.803 to 0.817}; I21, I29, I39 noted6 alpha: 0.810, standardized alpha: 0.810, alpha(glb): 0.856 Note: three items had negative CTT discrimination (I21, I29, I39) links: the dataset; output from psych program; local data on L-sito C:\R

7b) Results for the LenguaBIg test with n=5,504 and items=47

Lertap5 alpha: 0.829; {95%CI: 0.82 to 0.84}; alpha(pc1): 0.846 psych omega: 0.837; alpha: 0.827; omega(h): 0.718 (some probs) omegaSem: 0.837; alpha: 0.827; omega(h): 0.718 (some probs) JASP: omega: 0.832; {95%CI: 0.822 to 0.835}

alpha: 0.829, standardized alpha: 0.827, alpha(glb): 0.869 Note: excluding the three items from 7a) having negative discrimination links: the dataset; output from psych program; local data on L-sito C:\R

8) Results for the Negocios test with n=500 and items=60

Lertap5 alpha: 0.857; {95%CI: 0.84 to 0.87}; alpha(pc1): 0.873

with sample of 25%, alpha: 0.86; {95%CI: 0.83 to 0.90} SAS CALIS omega: 0.850 psych omega: 0.868; alpha: 0.862; omega(h): 0.567

omegaSem: 0.868; alpha: 0.862; omega(h): 0.567 JASP: omega: 0.859; {95%CI: 0.838 to 0.874}

alpha: 0.857, standardized alpha: 0.862, alpha(glb): 0.935 links: the dataset; output from psych program; data on OSF

9) Results for the DunnSES tech scale with n=201 and items=7

Lertap5 alpha: 0.936; {95%CI: 0.92 to 0.95}; alpha(pc1): 0.940

SAS CALIS omega: 0.936 MBESS omega: 0.938; alpha: 0.936 psych omega: 0.955; alpha: 0.939; omega(h): 0.924

omegaSem (lavaan) could not invert matrix JASP omega: 0.938; {95%CI: 0.920 to 0.953}

alpha: 0.936, standardized alpha: 0.939, alpha(glb): 0.950

note: JASP could not fit CFA and used PFA instead for omega links: the dataset; output from psych program; data on OSF

6 JASP’s output included a note about these three items having negative correlations

An empirical look at coefficients alpha and omega, page 6.

10) Results for the BFI Neuro scale with n=2,800 and items=5

Lertap5 alpha: 0.812; {95%CI: 0.80 to 0.82}; alpha(pc1): 0.815 SAS CALIS omega: 0.805 psych omega: 0.852; alpha: 0.815; omega(h): 0.729 omegaSem: 0.837; alpha 0.874; omega(h): 0.536

JASP omega: 0.813; {95%CI: 0.803 to 0.825} alpha: 0.814; standardized alpha: 0.815; alpha(glb): 0.850

links: the dataset; output from psych program; data on OSF

11) Results for the BFI Neuro scale with n=2,663 and items=5

Lertap5 alpha: 0.813; {95%CI: 0.802 to 0.824}; alpha(pc1): 0.817

SAS CALIS omega: 0.800 psych omega: 0.851; alpha: 0.814; omega(h): 0.727 omegaSem: unable to run correctly JASP omega: 0.813; {95%CI: 0.802 to 0.824}

alpha: 0.813; standardized alpha: 0.814; alpha(glb): 0.848 links: the dataset; output from psych program; data on OSF

12) Results for the Blirt8 scale with n=211 and items=8

Lertap5 alpha: 0.780; {95%CI: 0.732 to 0.822}; alpha(pc1): 0.794 SAS CALIS omega: 0.787 psych omega: 0.837; alpha: 0.784; omega(h): 0.536 omegaSem (lavaan) could not invert matrix JASP omega: 0.785; {95%CI: 0.731 to 0.821}

alpha: 0.780; standardized alpha: 0.784; alpha(glb): 0.835

Article omega: 0.785; alpha: 0.780 links: the article; the dataset; output from psych program; data on OSF

A link to one of the scripts used with the CRAN psych package is here; in this

case the script is that written to process the DunnSES data. A link to the

corresponding Dunn csv file is here. A helpful, concise guide to the use of CRAN

packages is found in Dunn, Baguley, & Brunsden (2014, pp. 407-408)7.

The program script used to analyse the DunnSES data with SAS University and

Proc CALIS may be seen here.

Comments

The main reason I undertook this work was to see if I ought to make an effort to

incorporate coefficient omega in Lertap5. This would, in all likelihood, be quite

possible, but I would not expect it to be a piece of cake to program.

However, given the results above, bringing omega into Lertap5 will not have

much priority. There is a resounding indication (in this paper) that it wouldn’t be

worth it – coefficients alpha and omega are nearly identical in all of the datasets

summarized here.

Appendix B indicates how much factor loadings can vary without seeming to

markedly diminish the agreement between alpha and coefficient omega.

In Geldhof, Preacher, & Zyphur (2013) mention is made of a six-item scale with

item loadings of 0.40, 0.40, 0.60, 0.60, 0.80 and 0.80, finding omega at 0.78

with alpha at 0.77; they report that differences between alpha and alternative

7 References are listed at this webpage.

An empirical look at coefficients alpha and omega, page 7.

reliability estimate are “… relatively minor … in applied research….”8.

Another factor mitigating against bringing omega into Lertap5 on a priority basis:

it is very easy to take results from Lertap and pass them over to the CRAN psych

package. The output from the psych package is well presented, formatted, and

annotated; it includes a useful graphical summary of how items load on factors

which I have not included here.

Caveat

Eight of the datasets used in this study involve multiple-choice cognitive tests.

For the most part, these tests were created by educators and professional test

developers with substantial experience – by and large, the quality of the tests,

from a statistical point of view, was quite good, a notable exception being the test

at 7a) above where three items had negative discriminations.

Dataset 9 above, was a short affective scale. The seven items in the scale were

similar in that their standard deviations did not vary much, ranging from 0.98 to

1.27 (in this case), and the correlations of each item with a composite formed by

summing scores on the other eight items in the scale were also similar, going (in

this case) from a low of 0.69 to 0.909. This scale, in other words, was a good

one, with similarly-behaving, highly inter-correlated items.

Dataset 10 was not “clean”. The BFI consists of five subscales, each having five

affective items. One of the subscales used in this study correlated negatively with

the others. It is probably the case that BFI users seldom if ever look at the total

BFI “score”, instead interpreting results based on scores for each of the five

subscales.

Alpha and omega can both be negatively affected by: items with standard

deviations out of line with the other items, and items whose correlations with the

other items are noticeably lower. In particular, the correlations should not be

negative – the LenguaBIc dataset, 7a above, had three items with negative

correlations; at 7b these items were removed and both alpha and omega

increased slightly. The LenguaBIc test had 50 items; the impact of removing

items with very low correlations will be more substantial when the test or scale

has fewer items.

8 Personal correspondence from Prof. Geldhof received 25 May 2016: “Your results match my personal

experience – alpha and omega give essentially the same results under most conditions”. 9 The “s.d.” and “cor.” columns in a Lertap5 Stats1b report provide this information.

An empirical look at coefficients alpha and omega, page 8.

The scatterplots above display the difficulty / discrimination values for each item

in the LenguaBIc (left) and Zmed (right) tests. Three LenguaBIc items had

correlations (discriminations) below zero; we might call the lowest two of these

“outliers”, or undesirables. There do not appear to be any outliers among the

Zmed items10, allowing us to say, perhaps, that the Zmed test was found to be

desirably free of undesirable items.

10 These scatterplots are found in Lertap5 Stats1b reports for cognitive tests.

An empirical look at coefficients alpha and omega, page 9.

Appendix A: Scree plots

I thought it might be illuminating to see what scree plots of the eigenvalues for

the first five principal components might look like, and present them below. I

have included omega(h) here as I was looking for a possible trend between it and

the size of the eigenvalue for the first principal component. (Trend not found.)

An empirical look at coefficients alpha and omega, page 10.

The “scree test” has, over many years, often been used as a test of “unidimen-

sionality”.

If a test or scale has just one common factor underlying it, the scree plot should

look like that for DunSES above: a very sharp, steep drop from the first eigen-

value to the second eigenvalue, with subsequent eigenvalues decreasing in a very

minor, gentle manner, with no bumps along the way.

Thompson (2004) wrote that the factor count ends “ … where there is an ‘elbow’,

or levelling of the plot.” The matter of determining where the elbow is can be

assisted, he wrote, by the “pencil” test: imagine laying a pencil over the scree

plot, starting from the right-hand side of the plot. The factor count ends where

the points in the plot are no longer covered by the pencil11.

In the graphs above, the pencil test might suggest there to be only one factor in

many of the scree plots, but not with HalfTime, and maybe not with UniAA, may-

be not with LenguaBIc.

A problem with the pencil test is that it might not take into account the steepness

of the plot, which is an important point. The steeper the plot, the bigger the dif-

ference between the first and second eigenvalues, the more likely there may be

just one underlying common factor. More support for the single-factor possibility

comes when the difference between the second and third eigenvalues is trivial12.

Note: the scree plots presented above are based on the percentage of variance

accounted for by each eigenvalue. This is not normal – users of the scree test

generally base the plot on actual eigenvalue values. I suggest that using per-

centages tends to normalise the plots, making it possible to better compare plots

from datasets having a varying number of items. And, using percentages makes

it easier to quantify the size of the drop from the first eigenvalue to the second.

Nelson (2005) has more comments on the use of scree plots.

11 No fair using a really fat pencil. 12 The plots above do not show all eigenvalues, just the first five – the number of eigenvalues will

equal the number of items (in a principal components analysis), so the complete plots would show values to the right of those plotted here, each one not greater than the previous one.

An empirical look at coefficients alpha and omega, page 11.

Appendix B: loadings / weights

Below is a graphical comparison of the loadings/weights of test items on the first

principal factor (unrotated), as output by Lertap5, and on the latent variable from

the SAS Proc CALIS.

C-f1 is the weight from CALIS on the latent variable, while p-f1 is the loading on

the first principal factor from Lertap5.

The pattern is similar; the two trace lines tend to rise and fall at the same items.

What is interesting is to notice how much the item weights on the latent variable

(C-f1) can swing without seeming to markedly serve to distance alpha from

omega. C-f1 is the top line in all of these graphs (blue when seen in colour).

An empirical look at coefficients alpha and omega, page 12.

An empirical look at coefficients alpha and omega, page 13.

Note negative values at items 21, 29, and 39.

References are listed at this webpage.