fit analysis in rasch model university of ostrava czech republic 26-31, march, 2012
DESCRIPTION
3 Two approaches to assessing fit Fit statistics, based on standardized residuals Chi – square criteria, assesssing the closeness of model and empirical characteristic curvesTRANSCRIPT
![Page 1: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/1.jpg)
FIT ANALYSIS IN RASCH MODEL
University of OstravaCzech republic
26-31, March, 2012
![Page 2: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/2.jpg)
Two questions relating fit analysis:
How to assess fit the real data to the chosen model of measurement?
What should we do if the data don’t fit the model?
![Page 3: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/3.jpg)
3
Two approaches to assessing fit
• Fit statistics, based on standardized residuals
• Chi – square criteria, assesssing the closeness of model and empirical characteristic curves
![Page 4: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/4.jpg)
Let ani be a scored response for the interaction of the person n, n=1,…,N, and the item i, i=1,…,I (the 1’s and 0’s in dichotomous case);
xni – standardized residual;
Pni – the probability of a correct response for person n on item i.
( )( )
ni ni ni nini
ni ni ni
a M a a PxD a P q
( ) , ( )ni ni ni ni niM a P D a P q
![Page 5: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/5.jpg)
Properties of standardized residuals xni
M(xni)=0, D(xni)=1; In theory values vary in range (-∞,+∞); in practice values
usually range from -10 to +10. If Pni=0,99 and ani=0 than xni≈-9,94 ; similarily if Pni=0,01 and ani=1, than xni≈ 9,94 ;
Positive values represent correct responses: xni>0 if ani=1; Negative values represent incorrect responses: xni<0 if ani=0; The values are assumed to have normal distribution N
(0,1).
![Page 6: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/6.jpg)
Dependence of the standardized residual on
the difference θn - δi
![Page 7: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/7.jpg)
Statistics xni can have positive and negative values, so summing residuals across items and persons is informative. The solution of this problem is squaring standadized residuals:
22 ( )ni ni
ni nini ni
a Py xP q
![Page 8: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/8.jpg)
Properties of yni
Statistics yni has only non negative values; In theory values yni vary in range (0,+∞); in practice
values usually range from 0 to 100, at that most of values are in range (0,2);
The expected value and variance of yni are: M(yni)=1, D(yni)=2.
Statistics yni = xni2 can be evaluated as having χ2
distribution with one degree of freedom.
![Page 9: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/9.jpg)
Distribution features of statistics yni
These squared standardized residuals are only approximate chi-squares
Statistics yni would have exact χ2 distribution, if the following conditions were completed:
1) ani was continuous variable (rather than discrete);
2) exact values of the possibity Pni were known (indeed only estimates of this possibility are known that are based on parameter estimates);3) The data fit the measurement model.
![Page 10: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/10.jpg)
Person fit statistics for an examinee n:
This statistics is a sum of all values yni for the examinee across all items. It has approximately χ2 distribution with df=I.
A problem: for each degree of freedom there is a different crirical value, so no single critical value can be used
2
1
I
n nii
y
![Page 11: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/11.jpg)
Item fit statistics for an item i:
This statistics is a sum of all values yni for the item across all examinees. It has approximately χ2 distribution with df=N.
The same problem: for each degree of freedom there is a different crirical value, so no single critical value can be used.
2
1
N
i nin
y
![Page 12: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/12.jpg)
A possible solution of the critical value problem
Transformating the chi-squre statistics into a mean –square by dividing the chi-square by its degrees of freedom (Outfit MNSQ in Winsteps):
Person-fit statistics for an examinee n:
Item-fit statistics for an item i :
2(1)
1 1
1 1 ( )I Ini ni
n nii i ni ni
a PU yI I P q
2(1)
1 1
1 1 ( )N Nni ni
i nin n ni ni
a PU yN N P q
![Page 13: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/13.jpg)
Properties of mean square statistics Un
(1) and Ui(1)
Statistics Un(1) и Ui
(1) vary in range from [0,+∞); Expected value is 1: M(Un
(1))=M(Ui(1))=1 .
Statistics Un(1) и Ui
(1) are very sensitive to outliers (unexpected correct or incorrect responses).
![Page 14: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/14.jpg)
To counteract this sentivity to outliers the weighted versions of person-fit and item-fit statistics were developed:
2
(2) 1 1
1 1
( ) ( )I I
ni ni ni nii i
n I I
ni ni ni nii i
y D a a PU
P q P q
2
(2) 1 1
1 1
( ) ( )N N
ni ni ni nin n
i N N
ni ni ni nin n
y D a a PU
P q P q
![Page 15: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/15.jpg)
Properties of weighted fit statistics
Each squared standardized residual yni is weighted by the dispersion D(ani)=Pni·qni before it is summed. The value D(ani) is the least for the items difficulty of which don’t correspond to ability level of the examinee. Thus, contribution of these items to statistics Un
(2) and Ui(2) will
be reduced. Statistics Un
(2) and Ui(2) vary in range [0,+∞) and have
expected value of 1.
![Page 16: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/16.jpg)
Total fit statistics
The MNSQ statistics Un(1), Un
(2), Ui(1) и Ui
(2) are called total fit statistics.
Observed values of MNSQ statistics Un(1), Un
(2), Ui(1) and Ui
(2) are the more closed to expected value 1, the more the real data fir the Rasch model.
If the real data don’t fit the model, observed values of MNSQ statistics will differ from 1.
![Page 17: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/17.jpg)
The problem with critical values of total fit statistics
Critical values of mnsq statistics are different for different samples and different tests
The distributions of mnsq statistics are approximate and, as a rule, empirical distributions differ from the theoretical ones
So we can not use the same critical values for mnsq statistics defined from their theoretical distribution.
![Page 18: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/18.jpg)
Interpretation of total fit statistics values
The value of item-fit ststistics of 1.3 can be interpreted as indicating noise in the data in the item response pattern: there is 30% more variation in the data than it was predicted by the modal (underfit)
The value of item-fit ststistics of 0.8 can be interpreted as indicating Guttman pattern: there is 20% less variation in the data than it was predicted by the modal (overrfit)
![Page 19: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/19.jpg)
Recomendations on interpretation of fit statistics
![Page 20: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/20.jpg)
Transformating the mnsq statistics to standardized form (zstd in Winsteps)
There are two kinds of transformation that converts the mean-square to an approximate t-statistics:
Logarithm transformation
Cube-root transformation
1(ln 1)8
kt U U
3 3 ( )( 1)( ) 3
D Ut UD U
![Page 21: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/21.jpg)
Properties of standardized fit statistics
Standardized fit statistics t have approximately normal distribution N(0,1),
So with this statistics common critical values can be developed: for significance level 0,05 the acceptable values are in the range of (-2,+2)
Simulation studies have shown that the standardized fit statistics have more consistent distributional properties in the face of varying sample size than do the mnsq ststistics
![Page 22: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/22.jpg)
Statistics for Item Fit Analysis
Total item-fit statistics Ui(1) (Mnsq Outfit)
Standardized item-fit statistics ti(1) (Zstd Outfit)
Weighted total item-fit statistics Ui(2) (Mnsq Infit)
Standardized weighted item-fit statistics ti(2) (Zstd Infit)
A combination of item-fit statistics provides the best opportunity to detect poor fit items
![Page 23: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/23.jpg)
Recommended ranges for the total item-fit statistics for different test types
![Page 24: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/24.jpg)
Some reasons for poor item-fit (underfit - mnsq fit-statistics values are more than 1.2; zstd fit-statistics values are more than +2)
Test is not unidimensional Bed items (mistakes with keys, bed distractors in
MC items, item mistakes, etc.) Particular features of examinee behavior
(guessing, carelessness, etc.)
![Page 25: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/25.jpg)
About items that are overfit (mnsq fit-statistics values are less than 0.8; zstd fit-statistics values are less than -2)
Patterns of these items are too perfect (Guttman). It is not in agreement with a probabilistic nature of the model
Too perfect patterns of these items can be consequence of more high discrimination of these items.
A possible reason for it is violation of local independence: examinee’s response to one item affects his response to another one. Such items don’t contribute into measurement of ability.
![Page 26: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/26.jpg)
26
The second approach to item fit: chi-square statistics
2
2
1
( )( )
sn ni
i nn i n n
l l ff l
df = s - 1
![Page 27: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/27.jpg)
Problems with chi-square approach use
How many points for sample dividing should we take? 3 or 5 or 10 or more? In many test situations this statistics has low efficiency
In Rasch measurement the preference gives to item-fit statistics described above. In addition software Winsteps produces confidence intervals for ICC based on chi-square approach
![Page 28: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/28.jpg)
Example of fit analysis: test description
Test contains 50 items which are divided into three parts: part 1 has 32 MC items with 4 options (А1 – А32); part 2 has 12 opened items with a short answer (В1 – В12); part 3 has 6 items with free-constructed response (С1 – С6)/
Most of items were scored dichotomously, and a few items were scored polytomously.
The total sample size is 655.
![Page 29: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/29.jpg)
Item statistics
![Page 30: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/30.jpg)
ICC of a poor fit item
Dicriminating power of this item is lower than other items have
![Page 31: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/31.jpg)
ICC of a good item
![Page 32: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/32.jpg)
ICCs of the first 9 items (А1-А9)
![Page 33: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/33.jpg)
ICC of an item with two fit statistics with values below the left critical value
Discriminating power of the item is higher than other items have
![Page 34: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/34.jpg)
Statistics for Person Fit Analysis
Total person-fit statistics Un(1) (Mnsq Outfit)
Standardized person-fit statistics tn(1) (Zstd Outfit)
Weighted total person-fit statistics Un(2) (Mnsq Infit)
Standardized weighted person-fit statistics tn(2) (Zstd Infit)
A combination of person-fit statistics provides the best opportunity to detect poor fit items
![Page 35: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/35.jpg)
Some reasons of poor person fit:
Bed items Personal features of the examinee Gap in the examinee knowledge Violation of test conditions
![Page 36: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/36.jpg)
Example of person analysis
![Page 37: FIT ANALYSIS IN RASCH MODEL University of Ostrava Czech republic 26-31, March, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062412/5a4d1aff7f8b9ab0599861ff/html5/thumbnails/37.jpg)
Analysis of examinewe profiles