hs26 v.2006/06/29 Prn:29/06/2006; 12:35 F:hs26004.tex; VTEX/DL p. 1aid: 26004 pii: S0169-7161(06)26004-8 docsubty: REV

Handbook of Statistics, Vol. 26ISSN: 0169-7161 2006 Elsevier B.V. All rights reservedDOI: 10.1016/S0169-7161(06)26004-8

41 1

2 2

3 3

4 4

5 5

6 6

7 7

8 8

9 9

10 10

11 11

12 12

13 13

14 14

15 15

16 16

17 17

18 18

19 19

20 20

21 21

22 22

23 23

24 24

25 25

26 26

27 27

28 28

29 29

30 30

31 31

32 32

33 33

34 34

35 35

36 36

37 37

38 38

39 39

40 40

41 41

42 42

43 43

44 44

45 45

Reliability Coefficients and Generalizability Theory

Noreen M. Webb, Richard J. Shavelson and Edward H. Haertel

1. Introduction

When a person is tested or observed multiple times, such as a student tested for math-ematics achievement or a Navy machinist mate observed while operating engine roomequipment, scores reflecting his or her performance may or may not agree. Not only mayindividuals scores vary from one testing to another, calling into question the defensibil-ity of using only one score for decision-making purposes, but the rankings of individualsmay also disagree. The concern of reliability studies is to estimate the consistency ofscores across repeated observations. Reliability coefficients quantify the consistencyamong the multiple measurements on a scale from 0 to 1.

In this chapter we present reliability coefficients as developed in the framework ofclassical test theory, and describe how the conception and estimation of reliability wasbroadened in generalizability theory. Section 2 briefly sketches foundations of classicaltest theory (see the chapter by Lewis for a thorough development of the theory) and fo-cuses on traditional methods of estimating reliability. Section 3 reviews generalizabilitytheory, including applications and recent theoretical contributions.

2. Reliability Coefficients in Classical Test Theory

Classical test theorys reliability coefficients are widely used in behavioral and socialresearch. Each provides an index of measurement consistency ranging from 0 to 1.00and their interpretation, at first blush, is relatively straightforward: the proportion ofobserved-score variance attributable to true-scores (stable or nonrandom individual dif-ferences) (see Lewis chapter for definitions in Classical Test Theory). Coefficients at orabove 0.80 are often considered sufficiently reliable to make decisions about individu-als based on their observed scores, although a higher value, perhaps 0.90, is preferredif the decisions have significant consequences. Of course, reliability is never the soleconsideration in decisions about the appropriateness of test uses or interpretations.

Coefficient alpha (also known as Cronbachs alpha) is perhaps the most widelyused reliability coefficient. It estimates test-score reliability from a single test adminis-tration using information from the relationship among test items. That is, it provides an

1

Reliability Coefficients and Generalizability TheoryIntroductionReliability Coefficients in Classical Test TheoryTheoretical definition of reliabilityEstimation of the reliability coefficientReliability study designs and corresponding reliability coefficientsTest-retest reliability coefficientParallel or equivalent forms reliabilityDelayed parallel or equivalent forms reliabilityInternal-consistency reliabilityReliability coefficients for special occasions

Concluding comments: The need for more comprehensive reliability theory

Generalizability theoryOne-facet designsGeneralizability and decision studies with a crossed designGeneralizability and decision studies with a nested designA crossed generalizability study and a nested decision study

Multifacet designsRandom and fixed facetsSymmetryGeneralizability of group means

Multivariate generalizabilityAdditional issues in generalizability theoryVariance component estimatesHidden facetsNonconstant error variance for different true scoresOptimizing the decision (D) study design

Linking generalizability theory and item response theoryComputer programs

Concluding remarksUncited referencesReferences