lev-el/ - dtic · contract no. n00014-80-c-0402 contract authority identification number nr no....
TRANSCRIPT
RR-81-50LEV-EL/UNBIASED ESTIMATORS OF ABILITY PARAMETERS,
OF THEIR VARIANCE, AND OF THEIR
PARALLEL-FORMS RELIABILITY
o0
S-Frederic M. Lord
This research was sponsored in part by thePersonnel and Training Research ProgramsPsychological Sciences DivisionOffice of Naval Research, underContract No. N00014-80-C-0402
Contract Authority Identification NumberNR No. 150-453
Frederic M. Lord, Principal Investigator
Educational Testing Service D TICPrinceton, New Jersey ELECTE
November 1981 DEC 28 1981-.
DReproduction in whole or in part is permittedfor any purpose of the United States Government.
Approved for public release; distributionunlimited.
f3p112 28060,
UNBIASED ESTIMATORS OF ABILITY PARAMETERS,
OF THEIR VARIANCE, AND OF THEIR
PARALLEL-FORMS RELIABILITY
Frederic M. Lord
This research was sponsored in part by thePersonnel and Training Research ProgramsPsychological Sciences DivisionOffice of Naval Research, underContract No. N00014-80-C-0402
Contract Authority Identification NumberNR No. 150-453
Frederic M. Lord, Principal Investigator
AceesSion For--- Educational Testing Service
ITIS ORANIDTIC TAD N Princeton, New JerseyUnannounced 1JtitloatlO . .November 1981
Dist ribut tou/_Reproduction in whole or in part is permitted
Availability Codes for any purpose of the United States Government.Avall and/orDist S peciall,
II I Approved for public release; distribution
unlimited.
- 7 I J
UNCLASSIFIEDSECURITY CLASSIFICATION OF THIS PAGE (*%en Data Entered)
REPORT DOCUMENTATION PAGE BEOECOL NOM1. REPORT NUMBER 12. GOVT ACCESSION NO. 3. RECIPIENT-S CATALOG NUMUEER
4. TITLE (end Subtitle) 5. TYPE CF REPORT & PERIOD COVERED
Unbiased Estimators of Ability Parameters, of Technical ReportTheir Variance, and of Their Parallel-FormsReliability 6. PERFORMING G09. REPORT #V141111
Research Report 81-50
7. AUTHOR(&) S. CONTRACT ORl GRANT NUM11I[R1(s)
Frederic M. Lord N00014-80-C-0402
9. PERFORMING ORGANIZATION NAME AND ADDRESS 10. PROGRAM ELEMENT, PROJECT, TASKAREA & WORK UNIT NUMBERS
Educational Testing ServicePrinceton, NJ 08541 NR 150-453
II. CONTROLLING OFFICE NAME AND ADDRESS 12. REPORT OATE
Personnel and Training Research Programs November 1981Office of Naval Research (Code 458) IS. NUMBER OF PAGES
297T4 MONITORING AGENCY NAME & ADDRESS(II different from Controllinl Office) t5. SECURITY CLASS. of this report)
S1. DECLASSIFICATION/ DOWNGRADINGSCHEDULE
16 DISTRIBUTION STATEMENT (of this Report)
Approved for publiL release; distribution unlimited.
17, DISTRIBUTION STATEMENT (of the abstract entered In Block 20, If different from Report)
IS, SUPPLEMENTARY NOTES
19. KEY WORDS (Continue on reveree oide If necesesry and Identify by block number)
Bias (statistical), Estimation, Item Response Theory, Ability, True Score,Reliability, Maximum Likelihood, Mental Test Theory, Unbiased Estimate,
Standard Error, Asymptotics
20. ABSTRACT (Continue on everse side it necessary end Identify by block number)
G4.en known item parameters, unbiased estimators are derived 1) for anexaminee's ability parameter 0 and for his proportion-correct true scoreS~2
2) for s the variance of 0 across examinees in the group tested,2
also for s , and 3) for the parallel-forms reliability of the observed
test score, the maximum likelihood estimator 0
D I JAN°73 1473 EDITION OF I NOV , G IS OBSOLETE
S N 0102- LF- 014- 6601 SECURITV I No/SEUIYCLASSIFICATION OF TNIS PAGE (R11en Pat* Mature-f)
Unbiased Estimators
Abstract
Given known item parameters, unbiased estimators are derived
1) for an examinee's ability parameter 0 and for his proportion-
correct true score ,2) for s 2 the variance of 8 across examinees
2 -omin the group tested, also for s~ and 3) for the parallel-frm
reliability of the observed test score, the maximum likelihood estimator,
6.
Unbiased Estimators
2
Unbiased Estimators of Ability Parameters, of Their Variance,
and of Their Parallel-Forms Reliability
4--- his paper is primarily concerned with determining the statistical
bias in the maximum likelihood estimate e of the examinee ability .--
parameter ' in item response theory (IRT) [Lord, 1980]; also of
certain functions of such parameters. We will deal only with uni-
dimensional tests composed of dichotomously scored items. We assume
the item response function is three-parameter logistic (2).
Available results for the sampling variance ofC are currently
limited to the case where the item parameters are known; the present
derivations are limited, to this case also. This limitation is tolerable
in situations where the item parameters are predetermined, as in item
banking and tailored testing.
In the absence of a prior distribution for 8 , it is well known
that examinees with perfect scores h~ve',8 also that examinees
who perform near or below the chance level on multiple--choice •7•rms
may be gi.-.n large negative values of ,_Q. This (correctly) suggests
that 8 is positively biased for high-ability examinees and negatively
biased'for low-ability examinees. Will a correction of 0' for bias
be helpful in such cases?
*This work was supported in part by contract N00014-80-C-0402,project designation NR 150-453 between the Office of Naval Research andEducational Testing Service. Reproduction in whole or in part is permittedfor any purpose of the United States Government.
oI,
Unbiased Estimators
3
It is also 'well known' that for any ordinary group of examinees,
2the variance ( sý ) of 0 across examinees is larger than the variance
222( s• ) of the true 0 . The ratio sý/s_ is closely related to the
classical-test-theory reliability of 6 considered as the examinee's
2 2test score. Thus it i3 not enough for us to know that s• ÷ sa as the
number n of test items becomes large; we need to know how the rela-
2 2tion of sý to sa varies as a fbnction of n . We also need a
2 2better estimate of sa than its maximum-likelihood estimator s6
These objectives can be achieved by correcting s3 for bias.
--. -•The methods used to derive formulas for correction for bias are
presented herevin detail for at least two reasons: 1) experience
with similar derivations has shown that it is easy to reach erroneous
results if details are not spelled out. 2) The general methods used
here are easily transferred to solve other problems, such as a) cor-
rection of item parameters for bias, b) obtaining higher-order approxima-
tions to the sampling variance of e
1. Statistical Bias in 0 and r
The method used here to find the bias of 8 is adapted from the
'adjusted order of magnitude' procedure detailed by Shenton and Bowman
(1977). They assume their data to be a sample from a population divided into
a denumerable number of subsets. For them, the population proportion
of observations in a given sub3et is a known function of the param-
eter e whose value they wish to estimate. Their sample estimate of
o is therefore a function of observed sample proportions in the
t
Unbiased Estimators
4
various subsets. Since our data do not readily fit this picture, we
cannot uee their final published formulas but must instead derive our
own.
Thrtughout Section 1, we deal with a single fixed examinee whose
ability e is the parameter to be estimated. All item parameters are
assumed known.
i.1 Prelimninaries
The maximum likelihood estimate G is obtained by solving the like-
lihood equation
n (Lii - P~P/~ 1 - 0(1)i'l
where u 0 or 1 is the examinee's response to item i ( i - 1,2,...,
n ), P i P 1(0) is the response function for item i Q i E 1 - Pi
PIt is the derivative of P, with respect to e , and a caret indicates
that the function is to be evaluated at ; . We deal wizh the cese
where P is the three-parameter logistic function
+- c~ + (2)i i -Ai(6-b)
1+e
where A, bi ,and ci are item parameters describing item i
~. bi
Unbiased Estimators
5
We will assume
1. 6 is a bounded variable,
2. the item parameters ai and bi are bounded,
3. ci is bounded away from 1,
(thus Pi and Q, are bounded away from 0 and 1);
4. as n becomes large, the statistical characteristics of the
test stabilize.
Rather than trying to define this last assumption formally, the reader
may substitute the more restrictive assumption usually made in mental
test theory: that a test is lengthened by adding strictly parallel forms.
With these assumpttons, the conditions of Bradley and Gart (1962)
are satisfied. It follows from their theorpms that ; is a consistent
estimator of 8 and that 'nW (0 - 0) is asymptotically normally distri-
buted with mean zero and variance lim . niP2 /Pi . The existence ofr-mn i i Ii
this limit is guaranteed by assumpt'on 4.
For compactness, we will rewrite (1) as
n .LI 1 5 rii 0 (3)
where by definition
ii
Unbiased Estimators
rli (ui - P i)P/PiQi (4)i i
Now L1 considered as a function of 6 can be expanded formally in
powers of 0 - 6 as follows:
(61-6) 2 z 2 +L1 zii ÷i(( -) r i3iS i 1 +T3
where we define
o P Qiu8 s - 1,2,... ) . (5)8, de S
This definition is Lonsistent with (3).
Let x = a - o, r = P . Rather than proving the con-Ss sirsivergence of the power series, let us use a closed form that is always valid:
1 1 3 6 3L = r 4 xr ++ x r + L x4r (6)1 1 T2 + 3 6 4 24 5()
where r 5- Max r5 and 161 < I.
1.2 Derivatives and Expectations
To proceed further, it is necessary to evaluate the r . It iski
found that
IIVIVPM_
Unbiased Estimatots
7
S(_Ai)k-I *,' uicir ki ( -A si Sk-k- (i -Q' I ) (-I + -u' '- )(7)
ri
where - is a Stirling number of the second kind (Jordan, 1947,
pp. 31-32, 168).
Define
Ys i r 5si (8)
8s -rsi - srsi8 (9)
Since Sut M Pi D we find that
y1 i 0 , (10)
*'Pi' (21
AI Pi (P )201 c i) ( c (2y3 (1 c( ) 2 P 2
Eli r li (u i- Pi)P;/P1 QiQi (13)
A c P'(u - Pi)" ii I (14)21P
Let
YS y Ysi/n c -- E e sn /(15)
s si
S. .... . ... . >'4
Unbiased Es timators
8
We will denote the Fisher information by
I i -6(dLl/de) -ny 2 P /P (16)
Setting (6) equal to zero, the likelihood equation can now be written
in terms of the y and the c asM 1 x2 + 1 x3 (.4 + +_ 4r
-1 x(Y 2 + £2) + 2 (Y3 + Y3) +6 + £4) +C -2x4T 5 (17)
We will need some information about the order of magnitude of the
terms such as those in (17). It may be seen from (7) that each cs
has the form
C I Z Ki(usi (
where Ksi does not depend on n or on u . Since P, , QI and
1 - Ci are bounded, the K., and thus cs is bounded. By assumption
(4), the bound does not depend on n . The same conclusion holds for y.
Since /n x is asymptotically normally distributed with zero mean
and finite variance, it follows that Sxr ( r r 1,2,... ) is of order
n-.r/2. A similar statement is true of / e . Thus finally Oxrct <S 8 -
(fixr2 so that 6xrt is of order n-(r+t)/2 r,t = 1,2,... ).s S
Unbiased Estimators
9
1.3 First-Order Variance of e
To clarify the procedure, let us derive from (17) the familiar
formula for the asymptotic variance of 8 . Square (17) and take
expectations to obtain
2 2 2 23 3
"i: =y 2 ++ 2 + Y2r362 + Y2 5x + ,.. (18)
If we wish to neglect terms o(n-) (of higher order than n-),
equation (18) becomes
6x 2 = 1• S2 + o(n-1) (19)2 1 2 -
By (13) and (16), because of local independence,
2 P1
e1 5-- E i (uiPi J p (u P
"n -- i £ (u- )ui -j )
n 2 1J P i J j
1 2 2 Var u
22n i
n21 P t~n i P~
I2 (20)
n£ _ __
1 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Unbiased Estimators
10
Thus, finally
^' 1 n-1Var 0 + o( ) (21)
a well-known result. It is derived here to clarify the reasoning toA
be used subsequently. If 6 is substituted for 0 on the right side
of (21), the formula will still be correct to the spec'fied order of
approximation.
1.4 Statistical Bias of 8
Take the expectation of (17) to obtain
1 2 (22)
11 2 + 21 1X 2 +Y 3 ix 3 1
-1where 6 indicates an expectation in which only terms of order n are to
be retainedý Also multiply (17) by c 2 and take expectations to obtain
- CC22 2 = 1 xC2 (23)
By (9)
6 - 0 r - 1,2,... . (24)r
Trow (13) and (14)
Unbiased Estimators
11
1 A c P'2
c 1 i i i 21• 12 n 2 c, P32 ssui Pi)2
"1 2 1-c 3 2-
n i i PiQI
1 c 2 (25)=-2" :(1- ci) Pi
i
Substituting (16) and (25) into (23), we have the covariance
i A.,ci P' 2
x = -- c£(26)
Pi
Finally, substituting (16), (21), (24), and (26) into (22) and solving
for lX l , we have the bias
A Ac p'2B 9 ( - e i i-c 2L + ny3 ) (27)
i
This may be rewritten as
1 n1B 1(e) =2ZA I1(28)
where
Pi- ci a i p 2
1 - c i Q (29)
Unbiased Estimators
12
Since I is of order nl B1(6) is of order nl It may be of
interest to note that in the special case where all items are equivalent
(all P are the same), the bias simplifies to B 1(6 P/nP'
1.5 Numerical Results
A hypothetical test was designed to approximate the College
Entrance Examination Board's Scholastic Aptitude Test, Verbal Section.
This test is composed of n -90 five-choice items. Some information
about the distributions of the parameters of the 90 hypothetical item
is given in Table 1.
The standard error and bias of ; were computed from (21) and
from (27) respectively for various values of 6 The results are
shown in Table 2. It appears that the bias in e is negligible for
moderate values of e , but is sizable f or extreme values. Note that
the bias is positively correlated with e Because of guessing, zero
bias does not occur at 3 - 0 but at 0 .34 approximately.
1.6 Variance and Bias of Estimated True Score
Since the ability scale is not unique, any monotonic transformation
of e can serve as a measure of ability. Twio transformations aret
particularly useful: e0 and
=- P (W (30)-n
the proportion-correct true score (the number-right true score divided
Unbiased Estimators
13
TABLE 1
Range and Quartiles of the Item Parametersin 90-Item Hypothetical Test
ai&Al/3. 7 b c 1
Highest value 1.88 2.32 .47
Q1 1.07 1.15 .20
Median .83 .38 .15
Q3 .69 -. 41 .13
Lowest value .41 -3.94 .01
I
__i
Unbiased Estimators
14
by the number of items). One important reason for using the latter
transformation is the following.
Ordinarily, as in Table 2, we find large standard errors of
8 where 6 is extreme. Usually these large standard errors are no
more harmful to the user than are the smaller standard errors found
when e is near the level aimed at by the test. There is a reason
why this is so: If it were not, the user should have designed his test
so as to reduce those standard errors that were troublesome to him.
We see that from this point of view the size of a difference on
the 0 scale does not correspond to its importance. The discrepancy
is greatly reduced, however, if we measure ability on the C scale
instead of on the 8 scale. This is one reason, among several, wny
we are interestcd in the variance and bias of
E : ip(8)/n (31)
Although the proportion-correct true score
z u 1u1 /n (32)
is an unbiased estimator of C , z is never a fully efficient
estimator of C unless ci - 0 and ai , aj ( ij - 1,2,...,n ):
the sampling variance
nVar z r P Q (33)
n i2 1
Unbiased Estimators
15
TABLE 2
Standard Error and Statistical Bias in
e rVar e (0)
3.5 .60 .24
3.0 .43 .12
2.5 .31 .06
2.0 .23 .032
1.5 .19 .011
1.0 .19 .0032
0.5 .20 .0012
0 .22 -. 0028
-0.5 .25 -. 010
-1.0 .31 -. 025
-1.5 .41 -. 05
-2.0 .54 -. 09
-2.5 .70 -. 14
-3.0 .89 -. 22
-3.5 1.09 -. 31
.~-.--,---~~-~!
Unbiased Estimators
16
Aia not as small as the sampling variance of 4 , which we must now
derive.
By (31)
A n-~!- i PdeC .3 r
(34)
Using the 'delta' method
Var C ,- - ( E ,)2
n 0 P) Var2
By (21) and (16)
ii
Var C( 5n r Pi__ 2
(
i Pii
To find the bias of 4 we expand it In powers of x= -%:
- 2n 2n Epi +2"A- i 1 ''" (36)
Unbiased Estimators
17
where
pit d 2 P /do2
Taking expectations, and neglecting higher-order terms, we have for
the bias
(4 E + Cf()Z Var 0 (iP")i (37)1 n i 2 p
This can be rewritten as
,2I ~Aio ;2
SI •"" (l- + Z 1 +2 (38)S- i)Pi i
where C' Z •iP'/n and C" = P"/n . Let us note is passing that when
all items are equivalent (all P (6) are the same), 4 - z and its
bias (38) is zero.
1.7 Numerical Results
Table 3 shows the bias in C for the same hypothetical test con-
sidered in Section 1.5. The biases are all positive. However, they are
negligible at all except the lowest ability levels. This tends to con-
firm our choice of the 4 scale of ability rather than the 6 acale
for many purposes.
Unbiased Estimators
18
TABLE 3
Standard Error of z and of Cand Statistical Bias of
8 lVarz 'Var __
3.5 .981 .014 .014 .00045
3.0 .966 .019 .018 .00052
2.5 .937 .024 .023 .00064
2.0 .891 .031 .029 .00059
1.5 .812 .037 .035 .00021
1.0 .715 .042 .040 .00026
0.5 .608 .045 .042 .00061
0 .506 .046 .043 .00061
-0.5 .416 .047 .042 .00062
-1.0 .344 .046 .038 .00061
-1.5 .291 .045 .037 .00085
-2.0 .254 .044 .033 .0014
-2.5 .227 .042 .029 .0020
-3.0 .211 .042 .025 .0024
-3.5 .199 .041 .021 .0026
____t
Unbiased Estimatore
19
As a matter of incidental interest, for selected values of true
score Table 3 compares the standard error (35) of the maximum-likell-
hooJ estimator 4 with the standard error (33) of the unbiased estimator
z (proportion-correct score). There is little difference in accuracy
between the two estimators for 4 > .5 . At low tree-score levels, the
maximum-likelihood estimator is much better than the proportion of
cc rrect answers.
_o__ ____s2 o s2;Tt eiiiy
2. Unbiased Estimation o fa;Ts eiblt
2 2The symbols s2 and s are used for the sample variance of
8 and of C across the N examinees in the sample:
2 1 N 2 1 N 2sB N Pa N a) (39)
aan
an 2 22The maximum-likelihood estimators Of 8 ad9 are s^ and a?e
the sample variances across examinees of e and of C
2.1 Asmptotically Unbiased ;ýLIlmator of 2
Assume that t'ir examinees are a random sample of N from some
population. Denote by the population variance of 8 e Then Ne2 /(N-l)
2 2ia an unbiased estimator of 08 a Since s is unobservable, our first
task is to find a function of 0 that is an asymptotically unbiased
estimator of 2 2
0N
Unbiased Estimators
20
By the formula for the variance of a sum we have
a 0 2 a2 2+ 2a (40)e E +x 0 x ex
2where a denotes a variance across all examinees in the population
and ax is the corresponding population covariance. By a well-
known identity from the analysis of variance
2 2 + 2 (41)
x - xe 6(x18)
where • denotes an expectation across all examinees in the population.
Similarly,
c ol t ,s'xIB) (42)
Substituting (41) and (42) into (40), transposing, writing B, _ 1 (xle)
as in (28), and dropping the subscript from B for convenience, we
have
2 2 _ 2 a2 2 (43)e e-, OB exle 0B
Since by (28) B is of order n , its variance is of order n2
so aB can be neglected in (43). Since Section 1 deals with a singleB2
fixed examinee, the symbol a 2 in (43) has the same meaning as
Var e in (21):
2 1+o(a1 )oxle " o-
Unbiased Estimators
21
where I 1 1(0) is givme by (16). Since 2 -1 theXI i
effect of replacing e by e on the right in neglibible:
5c2 1n~~~+~ 1
8 xle -169)
By similar reasoning, we may replace a in (43) by a^ where B
is defined by (27) with 8 replaced by 8 • The result of these
approximations is that
2 2a8 -y - -2 - - + o(n-) . (44)
2
A useful estimator of oa can be calculated fromN
.2 N 2 2N N 1" -Ng-i N - 1 Si - N E1
(45)
a 16a
where
N N N-a;B E eB (a E ABas8BN •a-l a~a e ',/1;a)(i • a- )a-i aw a
and Ba is given by (27) with e replaced by e a , If we wish to
2estimate the sample variance of ability sB rather than the population
variance a, , we can use
"2 2 N-s 3 2s-B- (46)
a 6B N 2 a-i I(ea
* .. . .. ... .. . . ..
Unbiased Estimators
22
The second and third terms of (44)are of order n-1 , an order
of magnitude smaller than the first term but larger than the neglected
terms. The covariance of e and B is usually positive, as can be
readily seen from Table 2. Since 1(0) is necessarily positive, it
2 2appears that usually a ( , an inequality that is frequently assumed
without proof. It is not clear whether this inequality is necessarily
true.
Consider the parallel-forms reliability coefficient p'
the correlation between scores e and e' on two parallel tests.
For present purposes, two tests are parallel when for each item in one
test there is an item in the other test with the same item response
function. Let us estimate
a.;I a,
P - (47)
from a single test administration by substituting asymptotically unbiased
estimators of the numerator and of the denominator into (47).
As in (41),
coo + r (48
Unbiased Estimators
23
Because of local independence, the first term on the right vanishes.
Because of parallelism, the two expectations in the last term are
identical, so this term Is a variance. We thus have
a.2 a 2 + a 2 (49)901 S~ij) -B" B 6 + '01
From (49) and (43),
a _ xia (50)
We see that the parallel-forms reliability of 0 is
pai -- •S8 --- +o(n-1) (51)o 1 (e)
Priority in obtaining this result belongs to Sympson [Note 1].
Replacing population values on the right by the corresponding sample
statistics, we have a sample estimator of the parallel-forms reliability
coefficient of 8
N NP_ = 1 N-i 1 (52)
Be N 86 a-l 1(;
Since 8 is neither unbiased nor uncorrelated with 8 , we
should not expect the usual reliability formulas of classical test
theory to apply. A similar but not identical case is discussed in
• , -,, •,.•.•M•• • ,,,.".., .,,
Unbiased EstiWators
24
Lord and Novick (1968, Section 9.8). Thus p2 and
22a 8/a are not interchangeable definitions of reliability. Since
correlational measures are hard to interpret in the absence of
linearity and homoscedasticity, we will not now push this investigation
of reliability further.
2.3 Corresponding Results for True Score
By the same reasoning used to obtain (44) we have
2 2 2 2a* ;aZ- 2 e B14 - GBo a
- 2 __L2 -1a 2l _ + o(n- (53)
2A useful estimator of a can be calculated from
.. N 2 2N 1 N a54c{-N-1 s• N - 1 s••• - N 1T(a (54)
a-I T(e)a
To estimate 2 2 we can use
S2 N -l1 N {a,2
N I
'2 2s^.i() aN• a-i 10 (a
Unbiased Estimators
25
As in (50) - (52) we have
S(2 56)
() + o(n-l) (57)
28ý, E 1 (58)N sA a-l I(ea)
2.4 Numerical Results for True Scores
At moderate ability levels, (28) provides adequate but usually
neglible corrections for bias in e . Experience shows that at
very low ability levels, the usual test length ( n ) of 50 or 100
items is not long enough for the asymptotic results of (28) to apply.
For example, an examinee whose true e is -3 may easily obtain an
estimated ability 0 of -30 or of -= . For sufficiently long tests,
such extreme values of 0 %ould have negligible probability, but
with the usual values of n , equation (28) is totally inadequate for
correcting ; for bias at low ability levels.
This sair difficulty carries over to the unbiased estimation2
of o2 using (46). Since all ability levels are involved in (46),
the formula is useless in practice for any group that contains even
a few low-ability examinees. Fortunately, this difficulty does not
Unbiased Estimators
26
carry over to the estimation of ability on the true-score ( • )
scale.
The hypothetical SAT Verbal Test of Tables 1-3 was administered to
a typical group of 2995 hypothetical examinees. The bias in C
was estimated for each examinee and a corrected • obtained from
(51):
corrected C - B -lBI
In a few cases where the corrected • would have been below the
chance level En the corrected C was set equal to E c
The mean of the 2995 true C used. to generate the data was
.5280, the mean of the uncorrected Z was .5294, the mean of the
corrected Z was .5288. Thus the correction was in the right
direction, but not large enough. The uncorrected mean t was already
so accurate as to leave little room for improvement.
Next, (55) was used to estimate sa. The true value was
s4 .1610 , the standard deviation of • was s. - .1660 , the cor-
rected estimate from (55) was § - .1614 . The correction worked
very well here.
AAThe paralJlel-forms reliability of • was estimated from (58)
to be p^ M .9420 . We have no 'true' valuc against which this can
be compared, but the estimate seems a reasonable one. The Kuder-
Richardson formula-20 reliability of number-right scores for these
data is .9275.
a1
Unbiased Estimators
27
It should be remembered that both the formulas and the numerical
results In this report apply in situations where the item parameters
are known. These formulas may be satisfactory for situations where
the item parameters have been estimated from large groups not containing
the examinees whose ability estimates are to be corrected for bias.
These formulas will not be adequate for situations where the item
parameters and ability parameters are estimated simultaneously from
a single data set.
vi
Unbiased Estimators
28
Reference Note
1. Sympson, J. B. Estimating the reliability of adaptlve tests from
a •single test administration. Paper presented at the meeting ofthe American Educational Research Association, Boston, April 1980.
i iit[I
-gi1._
__r_ _ _
Unbiased Estimators
29
References
Bradley, R. A. & Gart, J. J. The asymptotic properties of ML estimators
when sampling from associated populations. Biometrika, 1962, 49,
205-214.
Jordan, C. Calculus of finite differences (2nd ed.). New York: Chelsea,
1947.
Lord, F. M. Applications of item response theory to practical
testing problems. Hillsdale, N.J.: Lawrence Erlbaum Associates,
1980.
Lord, F. M. & Novick, M. R. Statistical theories of mental test
scores. Reading, Mass.: Addison-4esley, 1968.
Shenton, L. R. & Bowman, K. 0. Maximum likelihood estimation i.n
small samples. Monograph No. 38. New York: Macmillan, 1977.
-2-
Technical DirectorNavy Personnel R & D I Dr. Bernard Rimland (030)San Diego, CA 92152 Navy Personnel Rt & D Center
San Diego, CA 921526 Commanding Officer
Naval Resevch Lahoratory I tr 1nold RubensteinCode 2627 Of7. Lt of Navel TechnologyWashington, DC 20390 600 N. Quincy Street
IPsychologist Arlington, VA 22217
ONi Branch Office 1 Dr. Worth Scauland, DirectorBuilding 114, Section D Research, Development, Test666 Smmer Street and EvaluationBoston, MA 02210 N-S
Naval Education and Training CommandOffice of Naval Research HASCode 437 Pensacola, FL 32508800 North Quincy StreetArlington, VA 22217 1 Dr. Robert G. Smith
Office of Chief of Naval Operations5 Personnel and Training Research OP-987H
Programs Washington, DC 20350Code 458Office of Naval Research I Dr. Alfred F. SmodeArlington, VA 22217 Training Analysis and Evaluation Group
Department of the NavyPsychologist Orlando, FL 32813ONR Branch Office1030 East Green Street 1 Dr. Richard SorensenPasadena, CA 91101 Navy Personnel R & D Center
San Diego, CA 92152SOffice of the Chief of Naval Operations
Research Development and Studies Branch I Mr. J. B. SympeonOP-115 Naval Personnel R & D CenterWashington, DC 20350 San Diego, CA 92152
LT Frank C. Petho, NSC, USN (Ph.D.) I Dr. Ronald WeitamanSelection and Training Research Division Code 54 WZHuman Performance Sciences Department Department of Administrative ServicesNaval Aerospace Medical Research Lab. U.S. Naval Postgraduate SchoolPensacola, FL 32508 Monterey, CA 93940
1 Dr. Robert Wherry562 Mallard Drive
Chalfont, PA 18914
L __ _ _ _ _ _ _
TI i -- -
DISTIIBJTION LIST
Navy
I Dr. 6d Aiken I Dr. Patrick i. MarrisonNavy Personnel R & D Center Psychology Course DirectorSan Diego, CA 94152 Leadership and Law Department (7b)
Division of Professional DevelopmentI Dr. Jack R. Borsting U.S. Naval Academy
Provost and Academic Dean Annapolis, MD 21402U.S. Naval Postgraduate SchoolMonterey, CA 93940 1 Dr. Norman J. Kerr
Chief of Naval Technical TrainingI Dr. kobert Breaux Naval Air Station Memphis (75)
Cod* N-711 Millington, TH 3d054NAVTKAEQU IPCENOrlando, FL 31613 1 Dr. William L. Naloy
Principal Civilian Advisor forI Chief of Naval Education and Education and Training
Training Liason Office Naval Training Command, Code OOAAir Force Human Resource Laboratory Pensacola, FL 32S06Flying Training DivisionWilliams Air Force Base, AZ 8I2'4 I Dr. Kneale Marshall
Scientific Advisor to DCNO(MPT)I CDR Mike Curran OpUIT
Office of Naval Research Washington, DC 203706U0 North 4uincy StreetCode 270 1 Dr. James McBrideArlington, VA 22217 Navy Personnel R & D Center
San Diego, CA 92152Or. Kichard ElaterDepartment of Administrative Sciences I Dr. William MontagueNaval Postgraduate School Navy Personnel I & D CenterMonterey, CA 9394U San Diego, CA 92152
1 or. Pat Federico I Mr. William NordbrockNavy Personnel K • U Center Instructional Program DevelopmentSan Diego, CA 91152 Building 90
NET-PDCDI Mr. Paul Foley Great Lakes NTC. IL 60088
Navy Personnel I & D Cent~erSan Diego, CA 9;152 1 Library, Code P2OIL
Navy Personnel i & U CenterDr. John Ford San Diego, CA 92152Navy Personnel K & 0 CenterSan Diego, CA 9l252
iI
4 .-- - - '
-3-
Dr. Robert Wisher 1 Dr. Hilton S. KatoCode 309 Training Technical AreaNavy Personnel R & D Center U.S. Army Research InstituteSan Diego, CA 92152 5001 Eisenhower Avenue
Alexandria, VA 22333Dr. Martin F. WiskoffNavy Personnel R & D Center 1 Dr. Harold F. O'Neil, Jr.San Diego, CA 92152 Attn: PERI-OK
Army Research InstituteMr. Ted M. 1. Yellen 5001 Eisenhower AvenueTechnical Information Office Alexandria, VA 22333Code 201Navy Personnel R & D Center 1 Dr. Robert SasmorSan Diego, CA 92152 U.S. Army Research Institute for
the Social and Behavioral Sciences5001 Eisenhower Avenue
Army Alexandria, VA 22333
I CommandantTechnical Director U.S. Army Institute of AdministrationU.S. Army Research Institute for the Attn: Dr. Sherrill
Behavioral and Social Sciences Ft. Benjamin Harrison, IN 462565001 Eisenhower AvenueAlexandria, VA 22333 1 Dr. Frederick Steinheiser
Department of the NavyDr. Myron Fischl Chief of Naval OperationsU.S. Army Rerearch Institute for the OP-113
Social and Behavioral Sciences Washington, DC 203505001 Eisenhower AvenueAlexandria, VA 22333 1 Dr. Joueph Ward
U.S. Army Research InstituteDr. Dexter Fletcher 5001 liseuhower AvenueU.S. Amy Research Institute Alexandria, VA 223335001 Eisenhower AvenueAlexandria, VA 22333
Air Force1 COL Frank Hart
Army Research Institute for theBehavioral & Social Sciences I Air Force Human Resources Laboratory
5001 Eisenhower Blvd. AIHRL/1@DAlexandria, VA 22333 Brooks Air Force Base, TX 78235
Dr. Michael Kaplan 1 U.S. Air force Office ofU.S. Army Research Institute Scientific Research5001 Eisenhower Avenue Life Sciences DirectorateAlexandria, VA 22333 Bolling Air Force Base
Washington. DC 20332
A+
i zq
-4b-
Air University Library 1 Director, Office of ManpowerAUL/LSE 76/443 UtilizationMaxwell Air Force Base, AL 36112 HQ, Marine Corps (MPU)
SIS, Building 2009Quantico, VA 22134
Dr. Earl A. AlluisiHQ, AFHRL (AFSC) I MAJ Michael L. Patrow, USMCBrooks Air Force Base, TX 78235 Headquarters, Marine Corps
Code HPI-20Mr. Raymond E. Christal Washington, DC 20380AFHRL/MOBrooks Air Force Base, TX 78235 1 Dr. A. L. Slafkosky
Scientific AdvisorDi. Genevieve Haddad Code RD-iProgram Manager HQ, U.S. Marine CorpsLife Sciences Directorate Washington, DC 20380AFOSRBolling Air Force BaseWashington, DC 20332 Coast Guard
Dr. Ross L. MorganAFIIRL/LR I Mr. Thomas A. WarmWright-Patterson AFB, OH 45433 U.S. Coast Guard Institute
P.O. Substation 18Research and Measurement Division Ozlahoma City, OK 73169Research Branch, AFHPC/HPCYPRRandolph Air Force Base, TX 78148
Other DoDDr. Malcolm RoeAFHRL/MPBrooks Air Force Base, TX 78235 1 DARPA
1400 Wilson BoulevardDr. Marty Rockway Arlington, VA 22209Technical DirectorAFHRL(OT) 12 Defense Technical Information CenterWilliams Air Force Base, AZ 58224 Cameron Station, Building 5
AtLn: TCAlexandria, VA 22314
MarinasI Dr. William Graham
Testing DirectorateDr. ". William Greenup IMECOM/MEPCT-PEducation Advisor (E031) Ft. Sheridan, IL 60037Education Center, ICDECQuantico, VA 22134
-. * " :. .. .• .. . . . . . 'T • " •; ' . . .
Military Assistant for Training I Dr. Joseph L. Youn", Directorand Personnel Technology Memory end Cognitive Processes
Office of the Under Secretary of National Science FoundationDefense for Research and Engineering Wash.ington, DC 2055U
Room 3D129, The PentagonWashington, DC 20301
Mon-GovernmentDr. Wayne SollmanOffice of the Assistant Secretary
of Defense (KRAL) I Dr. Erling R. Andersen2B209 The Pentagon Department of StatisticsWashington. DC 2030a1 Studiestraeds 6
1455 CopenhagenDENMAK
Civil GovernmentI Psychological Research Unit
Department of Defense (Army Office)Dr. Susan Chipman Campbell Park OfficesLearning and Development Canberra, ACT 26u0National Institute of Education AUSTRALIAI2UU 19th Street, NWWashington, DC 20208 1 Dr. Alan Baddeley
Medical reaerch CouncilHr. itichard HcKillip Applied Psychology Ur•.tPersonnel Rt & D Center 15 Chaucer RoadOffice of Personnel Management Cambridge CB2 2AF1900 E Street, NW ENGLANDWashington, DC 2U415
1 Dr. Isaac BeJarDr. Arthur Melmed Educational Testing ServiceNational Institute of Education Princeton, NJ Ul6i112OU 19th Street, N.W.Washington, DC 2020$ I Dr. Menucha birenbaum
School of EducationDr. Andre%. R. Nolnar Tel Aviv UniversityScience Education Development Tel Aviv, Ramat Aviv 69976
and Research ISAAELNational Science FoundationWashington, DC 205S0 1 Dr. Werner Birke
UesWPs im StreitkraeftsumtDr. Vern W. Urry Postfach 20 50 3Personnel R & D Center 0-5300 Ionn 2Office of Personnel Management WEST GERMANY1900 E Street, NWWashington, DC 20415
_
-6-
Or.r. Darrell bock I Dr. William I. CoffmnDepartment of Education Lirector, Iova Testing ProgramsUniversity of Chicago 334 Lindquist CenterChicago, IL 60037 University of Iowa
Liaison Scientists Iowa City, [A 52242Office of Naval Research I Dr. Allan M. Collinsbranch Office, London Bolt, Beranek, 4 Newman, Inc.Box 39 50 Moulton StreetFPO, NY 09510 Cambridge, MA 02136
Dr. Robert Brennan I Dr. Meredith P. CravfordAmerican College Testing Programs American Psychological AssociationP.O. Box 16b 1200 17th Street, NIowa City, IA J2240 Washington, DC 20036
Or. John B. Carrol) 1 Dr. Hans CrombagPsychometric Laboratory Education Research CenterUniversity of North Carolina University of LeydenDavis Hall U13A Boerhaavelaan 2Chapel Hill, NC 27514 2334 IN Leyden
THlE NETHERLANDSCharles Myers LibraryLivingstone House I Dr. Fritz DrasaovLivingstone Road Yale School of Organisation andStratforu ManagementLondon 915 ZLJ Yale UniversityENGLAND Box IA
Dr. Kenneth E. Clark New Haven, CT 0652UCollege of Arts and Sciences I LCOL J. C. EggenbergerUniversity of Rochester Directorate of Personnelliver Compus Station Applied ResearchRochester, NY 14627 National Defence Hq.
101 Colonel By DriveDr. Norman Cliff Uttava, KIA UB2Department of Psychology CANADAUniversity of Southern CaliforniaUniversity Park I Dr. Benjamin A. Fairbank, Jr.Los Angeles, CA 90007 McFann-Gray and Associates, lrec.
5825 CallaghanSuite 225San Antonio, TX 78226
77I
Dr. Leonard Feldt I Dr. Bert Green
Lindquist Center for Measurement Department of Psychology
University of Iowa Johns Hopkins University
Iowa City, IA 52242 Charles and 34th StreetsBaltimore, MD 212lS
I Dr. Richard L. Ferguson I Dr. Ron ambleton
The American College Testing Program School of EducationP.O. box Ibd University of Massachusetts
Iowa City, IA 52240 Amherst, NA O100
Dr. Victor Fields I Ur. Lloyd HumphreysDepartment of Psychology Department of Psychology
Montgomery College University of gllinoisRockville, MD 20d)O Champaign, IL 01820
Univ. Prof. Dr. Gerhard Fischer I Library
Psychologisches Institut der INavme/Wstern Division
Universitat Wien 27d57 Bervick Drive
Liebiggasse 5/3 Carmel. CA 93911A OIUl WienAUSTRIA I Dr. Steven Hunks
Department of Education
I Prof. Donald Fitzgerald University of Alberta
University of Now England Edmonton. Alberta
Armidale, Niow South Wales 2351 CANADA
AUSTRALIA1 Dr. Karl Hunt
Dr. Edwin A. Fleishman Department of Psychology
Advanced Research Resources Organization University of Washington
Suite VOU Seattle, WA 9d105
4330 East West HighwayWashington. DC 20014 1 Dr. Jack Hunter
2122 Coolidge Street
I Dr. John R. Frederiksen Lansing, MI 4890b
Solt, Beranek, and Newman50 Moulton Street I Dr. Huynh Muynh
Cambridge, MA 02138 College of EducationUniversity of South Carolina
Dr. Robert Glaser Columbia, SC 29204LAWCUniversity of Pittsburgh 1 Mr. Marlin KrogerJ939 O'Hara Street 1117 Via Goleta
Pittsburgh, PA 15113 Palos Verdes Estates, CA 90274'
-a1-
Dr. Michael Levine I Dr. Melvin A. Novickuepartment of tiducational Psychology 356 Lindquist Center for Measurement210 Education Building University of IowaUniversity of Illinois love City. IA 1242zChampaign, IL 61601
I Dr. Jesse Orlanakyur. Charles Levis Institute for Defense AnalysesFacultelt Social* Wetenechappen 40U Army Navy Driveaijlksuniversiteit Groningen Arlington, VA 2220OOude boteringestraat 239712GC Groningen I Dr. Wayne M. PatienceNTU•I.ANDS Americsn Council on Education
GED Testing Service, Suite IVUr. Robert Linn One Dupont Circle, NWCollege of Education Washington, DC 2003bUniversity of IllinoisUrbana, 1 bIOdU I Dr. James A. Paulson
Portland State UniversityDr. James Lumeden P.O. lox 751Uepartment of Psychol*gy Portland, ORt 97207University of Western AustraliaNedlands, Western Australia OUI I Mr. Luigi PetrulloAUSTRALIA 2431 North Idgevood Street
Arlington. VA 22207Dr. Gary MarcoEducational Testing Service I Dr. Diane H. .amsey-Kleeerinceton, NJ UdSui R-a- Research and System Design
4947 Ridgemont DriveOr. Scott Maxwell Malibu, CA 901b5Department of PsychologyUniversity of Houston I Hr. Minrat H. L. RauchHouston, TX 77U04 P II 4
lundessinisterium der VerteidigungI Or. ameuel T. Mayo Pontfach 1328
Loyola University of Chicago 0-53 Bonn 1d6U North Michigan Avenue GERMANYChicago. IL bU611
I Dr. Mark 0. ReckaseDr. Allen %tunro liducational Psychology Departmentbehavioral Technology Laboratories University of Missouri-Columbia1d45 Elena Avenue 4 dill HallFourth Floor Columbia, MO 65211Redondo leach, CA 9hJU77
,!I
Dr. Leonard L. Rosenbaum, Chairman I Dr. Kasuo ShigemasuDepartment of Psychology University of TohokuMontgomery College Department of Educational PsychologvRockville, MU ZUB50 Kawauchi, Sendai 960
JAPANDr. Ernst Z. RothkopfBell Laboratories I Dr. Edwin Shirkeyur iountailn Avenue Department of Psychology
Murray Hill, NJ 07974 University of Central FloridaDr. Lawrence Rudner Orlando, FL 32816
403 Elm Avenue I Dr. Richard SnowTakoma Park, HD 2UU12 School of EducationStanford University
Dr. J. Ryan Stanford, CA 9305Department of EducationUniversity of South Carolina 1 Dr. Robert SternbergColumbia, SC 29208 Department oi ;.ychology
Yale UniversityIProf. Fumiko jamejima Box 11A, Yale StationDepartment of Psychology New Haven, CT 06520University of TennesseeKnoxville, TN 37916 1 Dr. Albert Stevens
Bolt, Beranek, and Newman, Inc.Dr. Frank L. Schmidt 5U Moulton StreetDdepartment of Psychology Cambridge, MA 0213dBuilding GGGeorge Washington University 1 Dr. Hariharan SwaminathanWashington, DC 2UU02 Laboratory of Psychometric and
Evaluation ResearchDr. Robert J. Seidel School of EducationInstructional Technology Group University of HaasacuusettvHumRAO Amherst, MA 01003340 North Washington StreetAlexandria, VA 22314 I Dr. Kikumi Tatsuoka
Computer Based Education aesearch1 Committee on Cognitive Research Laboratoryc/o Dr. Lonnie K. Sherrod 252 Engineering Research LaboratorySocial Science Research Council University of Illinois6U5 Third Avenue Urbana, IL 61801New York, NY 10016
I Dr. David ThissenDepartment of PsychologyUniversity of KansasLawrence, KS 6b044
II
-' ' ~ t s b n - ~.~-',w,>
~ ~ ______________
-10-
Dr. Robert TsutakawaDepartment of StatisticsUniversity of MissouriColumbia, MO 6.520L
Dr. David ValeAssessment Systems
Corporation2395 University AvenueSuite 306St. Paul, MN 55114
Dr. Howard WainerEducational Testing ServicePrinceton, NJ 08541
Or. Thomas WallstenPsychometric LaboratoryDavis Hall 013AUniversity of North CarolinaChapel Hill, NC 27514
Dr. Phyllis WeaverGraduate School of EducationHarvard Universityý0U Larsen Hall, Appian WayCambridge, MA 02138
Dr. David J. WeissNb60 Elliott HallUniversity of Minnesota75 East River RoadMinneapolis, MN 55455
Dr. Susan E. WhitelyPsychology DepartmentUniversity of KansasLawrence, KS 66044
Dr. Volfgang WildgrubeStreitkraefteamtBox 20 50 03D-53OU Bonn 2WEST GER.MA