chapter7 item analysis

7/24/2019 Chapter7 Item Analysis

1/41

Chapter 7 Item Analysis In constructing a new test (or shortening or

lengthening an existing one), the final set of

items is usually identified through a process

known as item analysis.

Linda Croker


2/41

Both the alidity and the relia!ility of any

test depend ultimately on the characteristics of

its items.


3/41

Two Approaches of Item Analysis

Qualitative Analysis

Quantitative Analysis


4/41

Qualitative Analysis

includes the consideration of content

validity (content and form of items), as well as

the evaluation of items in terms of effective

item-writing procedures.


5/41

Quantitative Analysis

includes principally the measurement of

item difficulty and item discrimination.


6/41

1 Item ifficulty

1. Definition

!he item difficulty for item i, pi , is defined

as the proportion of e"aminees who get that

item correct.


7/41

"hough the proportion of examinees passing

an item traditionally has !een called the item

difficulty, this proportion logically should !e

called item easiness, !ecause the proportion

increase as the item !ecomes easier.


8/41

#. Estimation Methods

$ethod for ichotomously %cored Item

$ethod for &olytomously %cored Item

'rouping $ethod


9/41

Method for Dichotomously Scored

Items

(.1)

p is the difficulty of a certain item.

# is the numer of e"aminees who get that item

correct.$ is the total numer of e"aminees.

N

RP=


10/41

Example 1

"here are %& high school students attending a

science achieement test, and ' students pass

item , * students pass item &. +lease calculate

the difficulty for item and & separately.


11/41

Method for Polytomously Scored

Items

(.#)

max

X

XP=

X , the mean oftotal e"aminees* scores on one item

maxX , the perfect scores of that item


12/41

Example 2

"he perfect scores of one open ended item is*& points, the aerage score of total examinees

on this item is points. -hat is the item

difficulty

/ey0 .11


13/41

'rouping $ethod (+se of "treme 'roups)

2pper (2) and Lower (L) Criterion groups are selected

from the extremes of distri!ution of test scores or 3o! ratings.

!. . elley (1/0/) proposed that upper and lower

# could lead to the optimal point when the total test

scores are normally distriuted.


14/41

2

LU PP

P

+

=

UP is th proportion for e"aminees of upper groupwho get the item correct.

(.0)

LP is the proportion for e"aminees of lower groupwho get the item correct.


15/41

Example 3 "here are 4& examinees attending a language

test. /nown that '5 examinees of *46 upper

extreme group pass item 1, and examinees of *46

lower extreme group pass the same item. +lease

compute the difficulty of item 1.

/ey 0 .57


16/41

3. Correct Chance Effects on Item

Difficulty for Multiple-Choice Item

(.2)1

1

= K

KP

CP

CP ,corrected item difficulty

P ,uncorrected item difficulty

K , the number of choices for that item


17/41

Example 4

"he diffuculty of one fiechoice item is .1&,

the difficulty of another fourchoice item is .1.

-hich item is more difficulty


18/41

ANSWER

%o, the four-choice item is more difficulty.

38.0

15

15.05

1

11 =

=

=

K

KPCP

37.014

153.04

1

12 =

=

=

K

KPCP


19/41

4. Item Difficulty and Discrimination

iscrimination

ifficulty


20/41

If there are && persons in one population ,then ,we can calculate the discriminations as

following0

P=.01, 1 !! = !!P=.02, 2 !8 = 1!"

P=.3, 30 70 = 2100

P=.5, 50 50 = 2500


21/41

5. est difficulty and the

Distri!ution of est Scores 3ow to 4alculate the !est ifficulty 5

Two Methods

A calculate the mean of all item difficultiesof the test

B compute the ratio of mean of test scoresto perfect test scores


22/41

!est difficulty and the istriution of !est %cores

(a)

&ositive %6ewed istriution()

7egtive %6ewed istriution


23/41

# Item iscrimination

-hen the test as a whole is to !e ealuated !y means of

criterionrelated alidation, the items may themseles !e

ealuated and selected on the !asis of their relationships to

the external criterion.

-hen we identify an item for which high scoring

examinees hae a high pro!a!ility of answering correctly

and lowscoring examinees hae a low pro!a!ility ofanswer correctly, we would say such an item can

discriminates or differentiates the examinees.


24/41

1. Interpretation

Item discriminationrefers to the degree to

which an item differentiates correctly among

test takers in the !ehaior that the test is

designed to measure.


25/41

". Estimation Methods

Inde" of iscrimination

(used for dichotomously scored items)

D = PH- PL#7.5$

-e need to set one or two cutting scores to diide the examinees into

upper scoring group and lower scoring group.

+8 is the proportion in the upper group who answer the item

correctly and +L is the proportion in the lower group who answer theitem correctly.

9alues of : may range from .&& to .&&.


26/41

Example 1

"here are 5& students attending a world history test.

() If we use the ratio *46 to determine the upper andlower group, then how many examinees are there in the

upper and lower group separately (*)If % examinees

in upper group answer item 1 correctly, and ' examinees

in lower group answer it correctly, then calculate thediscrimination index for item 1.


27/41

Example 2 89 "aminees* !est ata on :-Item %cale Aout ;o%tress.

Item 1 # 0 2 8 < :

+8

+L

.82 .:1 .2 .0# .81 .1: .


28/41

'uidelines for Interpretation of =alue

:;.5&, the item is functioning


29/41

Correlation Indices of Item

Discrimination

(1) &earson &roduct $oment 4orrelation

4oefficient

"his formula is commonly used to estimate the degree of the

relationship !etween item and criterion scores

YX

XYsNs

xyr =


30/41

(#) &oint >iserial 4orrelation

If we use the total test score as the criterion, and test itemis scored & to , then we can use the following formula0

qp

s

XXr

t

tp

pbi %

= (.


31/41

Example 3

the !est ata of 18 "aminees"aminees 1 2 3 4 5 6 7 8 9 10 11

12 13 14 15

!est score

Item score

/9 :1 :9 : 9


32/41

72.2115

15%88258!3" 2=

=ts

48.4""7.%5333.72.21

80.585."8 =

=pbir

80.5815

10...8081!0=

++++=tX


33/41

!ransformation of ?ormula .iserial 4orrelation 4oefficient

or

Y

p

s

XXr

t

tp

b

=

Y

pq

s

XXr

t

qp

b

=


35/41

(2) 4orrelation >etween Items

a) Tetrachoric Correlation Coefcient

>ach aria!le is created through dichotomi?ing an

underlying normal distri!ution

$180cos# +

=BCAD

ADrt

(.:)

A >

4

Item i

9 1

Item

3

1

9

@AC BA:

@AB

CA:


36/41

b) PHI4oefficient

$$#$#$## DBDADCBAADBCr

++++

= (./)


37/41

#ariance for item

2

12

# $n

ij i

j

i

X X

sn

=

=

(.19)


38/41

ifficulty and iscrimination

&

1.99 9.99 9./9 9.#9

9.9 9.


39/41

0 Application 4ase of Item

Analysis

1. Procedures %elect a representative sample of e"aminees and

administer the test@

ifferentiate the e"aminees into upper # (or 09etc.) group and lower # group according to theirtest scores@

4alculate &+and &-, then estimate & and for each

item@

4ompare the responses on different choices for eachitem etween the upper group and lower group@

Bevise items.


40/41

". $nalysis Case

&

9.1 9.2# 9.8#

9.2# 9.0# 9.00

9.01 -9.9< -9.92

9.1# 9.92 9.9:

Item 'roup 7umer of "aminees on ach4hoice

ey

A > 4 Cmit

1 +pper 8 /# 1 # 9 >

ower ## 89 1# 1< 9

# +pper 8: 19 18 1< 1 A

ower #< #1 18 0< #

0 +pper 1 18 #: #: 1#

ower #8 11 1/ 02 11

2 +pper 1 22 12 0< 8 4

ower 1 8< 19 #: 8

br


41/41

Choice Analysis Dhether the e"aminees who choose the correct choice

is more than those who choose the wrong choices Dhether a lot of e"aminees choose the wrong choices

Dhether the e"aminees of upper group who choose

the correct choice is more than the e"aminees of lower

group Dhether the e"aminees of upper group who choose the

wrong choice is more than those of lower group

Dhether there is any choice that few e"aminees choose

Dhether there is any item that Euite a numer of

e"aminees ma6e no choices

chapter7 item analysis

Documents