designing accessible reading assessments examining test items for differential distractor...
TRANSCRIPT
![Page 1: Designing Accessible Reading Assessments Examining Test Items for Differential Distractor Functioning Among Students with Learning Disabilities Kyndra](https://reader035.vdocuments.us/reader035/viewer/2022062404/5515172e5503465e608b4f1f/html5/thumbnails/1.jpg)
Designing Accessible Reading Assessments
Examining Test Items for Differential Distractor Functioning
Among Students with Learning Disabilities
Kyndra Middleton
The University of Iowa
April 10, 2007
![Page 2: Designing Accessible Reading Assessments Examining Test Items for Differential Distractor Functioning Among Students with Learning Disabilities Kyndra](https://reader035.vdocuments.us/reader035/viewer/2022062404/5515172e5503465e608b4f1f/html5/thumbnails/2.jpg)
Designing Accessible Reading Assessments
Purpose of the Study
• To examine whether different distractor choices functioned differentially for students with learning disabilities who did not receive an accommodation, students with learning disabilities who received a read-aloud accommodation, and students with learning disabilities who received an accommodation other than a read-aloud
• To help determine whether a test can be modified for students with learning disabilities by removing a distractor choice while maintaining adequate test validity and information
![Page 3: Designing Accessible Reading Assessments Examining Test Items for Differential Distractor Functioning Among Students with Learning Disabilities Kyndra](https://reader035.vdocuments.us/reader035/viewer/2022062404/5515172e5503465e608b4f1f/html5/thumbnails/3.jpg)
Designing Accessible Reading Assessments
Instrument Used
• 4th grade English Language Arts assessment from a criterion-referenced statewide test– Operational test data
– Reading (42 MC items)– Writing (33 MC items)
![Page 4: Designing Accessible Reading Assessments Examining Test Items for Differential Distractor Functioning Among Students with Learning Disabilities Kyndra](https://reader035.vdocuments.us/reader035/viewer/2022062404/5515172e5503465e608b4f1f/html5/thumbnails/4.jpg)
Designing Accessible Reading Assessments
Sample Used
• 30,000 non-LD students sampled from 298,622 students
• 9,056 LD students who did not receive an accommodation
• 4,727 LD students who received an accommodation based on their IEP/504 plan
• 1,371 LD students who received an accommodation based on their IEP/504 plus a read aloud accommodation
![Page 5: Designing Accessible Reading Assessments Examining Test Items for Differential Distractor Functioning Among Students with Learning Disabilities Kyndra](https://reader035.vdocuments.us/reader035/viewer/2022062404/5515172e5503465e608b4f1f/html5/thumbnails/5.jpg)
Designing Accessible Reading Assessments
Sample Used cont’d
Subgroup Far Below Basic
Below Basic
Basic Proficient Advanced Total Number of Students
No Disability 4% 11% 31% 30% 25% 30,000
Learning Disability—no
accommodation
32% 34% 25% 7% 2% 9,056
Learning Disability—IEP/5
04
35% 38% 23% 4% 1% 4,727
Learning Disability—IEP/504 & read aloud
26% 36% 30% 6% 1% 1,371
Percentage of Students at Each Proficiency Level
![Page 6: Designing Accessible Reading Assessments Examining Test Items for Differential Distractor Functioning Among Students with Learning Disabilities Kyndra](https://reader035.vdocuments.us/reader035/viewer/2022062404/5515172e5503465e608b4f1f/html5/thumbnails/6.jpg)
Designing Accessible Reading Assessments
Sample Used cont’dReference-Focal Comparisons
Reference Group Focal Group
No Disability (Group 0) Learning Disability—no accommodation (Group 20)
No Disability (Group 0) Learning Disability—IEP/504 (Group 21)
No Disability (Group 0) Learning Disability—IEP/504 & read aloud (Group 22)
Learning Disability—no accommodation (Group 20)
Learning Disability—IEP/504* (Group 21)
Learning Disability—no accommodation (Group 20)
Learning Disability—IEP/504 & read aloud (Group 22)
Note: IEP = Individualized Education Plan* = comparison did not show DIF so was not included in the DDF analyses
![Page 7: Designing Accessible Reading Assessments Examining Test Items for Differential Distractor Functioning Among Students with Learning Disabilities Kyndra](https://reader035.vdocuments.us/reader035/viewer/2022062404/5515172e5503465e608b4f1f/html5/thumbnails/7.jpg)
Designing Accessible Reading Assessments
Procedure
• Examine items that previously displayed DIF for DDF– DDF: when two groups that have been matched on
ability have different probabilities of selecting a distractor
• Standardized Distractor Analysis (SDA)
– Distinguishes between distractors
– Identifies uniformly and nonuniformly biased distractors
– An extension of standardized p-difference
![Page 8: Designing Accessible Reading Assessments Examining Test Items for Differential Distractor Functioning Among Students with Learning Disabilities Kyndra](https://reader035.vdocuments.us/reader035/viewer/2022062404/5515172e5503465e608b4f1f/html5/thumbnails/8.jpg)
Designing Accessible Reading Assessments
Procedure Used cont’d
• Equation used to test for DDF:
STD(i) =
• : negligible DDF
• : moderate DDF
• : large DDF
10.SDA05.
05.SDA
10.SDA
s
rsfss
W(i)P(i)PW
![Page 9: Designing Accessible Reading Assessments Examining Test Items for Differential Distractor Functioning Among Students with Learning Disabilities Kyndra](https://reader035.vdocuments.us/reader035/viewer/2022062404/5515172e5503465e608b4f1f/html5/thumbnails/9.jpg)
Designing Accessible Reading Assessments
Results
• 70% of the items that displayed DIF also displayed DDF
• 100% of DDF occurred with a comparison between the read aloud and some other group
• 64% of the distractors that displayed DDF were in favor of the read aloud group
![Page 10: Designing Accessible Reading Assessments Examining Test Items for Differential Distractor Functioning Among Students with Learning Disabilities Kyndra](https://reader035.vdocuments.us/reader035/viewer/2022062404/5515172e5503465e608b4f1f/html5/thumbnails/10.jpg)
Designing Accessible Reading Assessments
Option D Difference*(moderate DIF)
-50
-40
-30
-20
-10
0
10
20
30
40
50
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75
Total Score
Diffe
ren
ce in
Perc
en t
F-R
![Page 11: Designing Accessible Reading Assessments Examining Test Items for Differential Distractor Functioning Among Students with Learning Disabilities Kyndra](https://reader035.vdocuments.us/reader035/viewer/2022062404/5515172e5503465e608b4f1f/html5/thumbnails/11.jpg)
Designing Accessible Reading Assessments
Option B*: large DIF
-50
-40
-30
-20
-10
0
10
20
30
40
50
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75
Total Score
Dif
fere
nc
e in
Pe
rce
nt
F-R
![Page 12: Designing Accessible Reading Assessments Examining Test Items for Differential Distractor Functioning Among Students with Learning Disabilities Kyndra](https://reader035.vdocuments.us/reader035/viewer/2022062404/5515172e5503465e608b4f1f/html5/thumbnails/12.jpg)
Designing Accessible Reading Assessments
Option B Difference(no DDF)
-50
-40
-30
-20
-10
0
10
20
30
40
50
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75
Total Score
Diffe
ren
ce in
Perc
ent
F-R
![Page 13: Designing Accessible Reading Assessments Examining Test Items for Differential Distractor Functioning Among Students with Learning Disabilities Kyndra](https://reader035.vdocuments.us/reader035/viewer/2022062404/5515172e5503465e608b4f1f/html5/thumbnails/13.jpg)
Designing Accessible Reading Assessments
Option B: moderate DDF
-50
-40
-30
-20
-10
0
10
20
30
40
50
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75
Total Score
Dif
fere
nc
e in
Pe
rce
nt
F-R
![Page 14: Designing Accessible Reading Assessments Examining Test Items for Differential Distractor Functioning Among Students with Learning Disabilities Kyndra](https://reader035.vdocuments.us/reader035/viewer/2022062404/5515172e5503465e608b4f1f/html5/thumbnails/14.jpg)
Designing Accessible Reading Assessments
Option C Difference(large DDF)
-50
-40
-30
-20
-10
0
10
20
30
40
50
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75
Total Score
Diffe
ren
ce in
Perc
en
t
F-R
![Page 15: Designing Accessible Reading Assessments Examining Test Items for Differential Distractor Functioning Among Students with Learning Disabilities Kyndra](https://reader035.vdocuments.us/reader035/viewer/2022062404/5515172e5503465e608b4f1f/html5/thumbnails/15.jpg)
Designing Accessible Reading Assessments
Results cont’dComparison Groups
0-20 0-21 0-22 20-21 20-22 Ite
m
A B C D A B C D A B C D A B C D A B C D 3 *R
R
*R
R
10 + *F
F
+ *F
F 13 + + *F
F
- 25 ++ *F 32 *R
R
- 33 *F
F
+ 34 + *F 45 *R
R
56 *R 64 *R
R
- *R
R
- *R
R
Note: +: moderate DDF in favor of the focal group++: large DDF in favor of the focal group-: moderate DDF in favor of the reference group*R: DIF in favor of the reference group*F: DIF in favor of the focal groupShaded box: Items that did not exhibit DIF
![Page 16: Designing Accessible Reading Assessments Examining Test Items for Differential Distractor Functioning Among Students with Learning Disabilities Kyndra](https://reader035.vdocuments.us/reader035/viewer/2022062404/5515172e5503465e608b4f1f/html5/thumbnails/16.jpg)
Designing Accessible Reading Assessments
Results cont’d
• 17% that assessed reading standards showed DDF
• 9% that assessed writing standards showed DDF
• No observed pattern across content or cognitive area between groups
![Page 17: Designing Accessible Reading Assessments Examining Test Items for Differential Distractor Functioning Among Students with Learning Disabilities Kyndra](https://reader035.vdocuments.us/reader035/viewer/2022062404/5515172e5503465e608b4f1f/html5/thumbnails/17.jpg)
Designing Accessible Reading Assessments
Results cont’d
• Item that displayed large DDF was the most difficult item that displayed DIF
• One item displayed DDF in each of the distractors (two favoring the read aloud group and one favoring the non-LD group)
• Item that displayed DDF in two of its distractors was a spelling item– Both were homophones– Additional difficulty caused by read aloud
![Page 18: Designing Accessible Reading Assessments Examining Test Items for Differential Distractor Functioning Among Students with Learning Disabilities Kyndra](https://reader035.vdocuments.us/reader035/viewer/2022062404/5515172e5503465e608b4f1f/html5/thumbnails/18.jpg)
Designing Accessible Reading Assessments
Conclusions/Future Research
• Measurement dissimilarity between read aloud group and other groups
• Exploratory study: More research needed to determine whether read aloud actually alters test’s validity
• Matched on ability to provide more information at extremes