![Page 1: English Language Arts Performance Assessment: Its Fairness and Predictive Validity Jia Wang, David Niemi, Pete Goldschmidt, and Haiwen Wang UCLA Graduate](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a22f24/html5/thumbnails/1.jpg)
English Language Arts Performance Assessment:
Its Fairness and Predictive Validity
Jia Wang, David Niemi, Pete Goldschmidt, and Haiwen Wang
UCLA Graduate School of Education & Information StudiesNational Center for Research on Evaluation,Standards, and Student Testing (CRESST)
American Educational Research Association52.038–Applying Research-Based Performance Assessment
Models in Routine Practice in a Large Urban School District: The Pleasure-Pain PrincipleSan Diego, CAApril 15, 2004
![Page 2: English Language Arts Performance Assessment: Its Fairness and Predictive Validity Jia Wang, David Niemi, Pete Goldschmidt, and Haiwen Wang UCLA Graduate](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a22f24/html5/thumbnails/2.jpg)
Purpose
This study is to examine the English language arts performance assessment in the following two aspects:
1. Relative fairness
2. Predictive validity
![Page 3: English Language Arts Performance Assessment: Its Fairness and Predictive Validity Jia Wang, David Niemi, Pete Goldschmidt, and Haiwen Wang UCLA Graduate](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a22f24/html5/thumbnails/3.jpg)
Research Questions
1. Is the performance assessment a fair test?
2. Is the performance assessment predictive of the California High School Exit Exam (CAHSEE)?
![Page 4: English Language Arts Performance Assessment: Its Fairness and Predictive Validity Jia Wang, David Niemi, Pete Goldschmidt, and Haiwen Wang UCLA Graduate](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a22f24/html5/thumbnails/4.jpg)
Data Description
• 50 schools
• 5,427 students
• 9th graders in Spring 2001 and 10th graders in Spring 2002
![Page 5: English Language Arts Performance Assessment: Its Fairness and Predictive Validity Jia Wang, David Niemi, Pete Goldschmidt, and Haiwen Wang UCLA Graduate](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a22f24/html5/thumbnails/5.jpg)
School Description
Table 4. Means and Standard Deviations of School Level Variables (N=50)
Variables Mean Std. Deviation Average Class Size 27.19 1.63 Percent of Students in Lunch Program 0.56 0.21 School API Rank in 2001 2.88 2.09 School Enrollment Size (in 1,000s) 3.20 0.88
![Page 6: English Language Arts Performance Assessment: Its Fairness and Predictive Validity Jia Wang, David Niemi, Pete Goldschmidt, and Haiwen Wang UCLA Graduate](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a22f24/html5/thumbnails/6.jpg)
Student Description
• 21% of the students were English proficient;• 81% of the students were Hispanic;• 26% of the students were immigrant;• 75% of the students spoke Spanish at home;• 75% of the students were in Title 1;• 73% of the students were in lunch program;• 3% of the students were gifted; and• 1% of the students were in special education
program.
![Page 7: English Language Arts Performance Assessment: Its Fairness and Predictive Validity Jia Wang, David Niemi, Pete Goldschmidt, and Haiwen Wang UCLA Graduate](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a22f24/html5/thumbnails/7.jpg)
Hierarchical Linear Model(school level)
School level variables:
• Average class size• Percent of students in lunch program• School enrollment (in 1,000s)• API rank in 2001
![Page 8: English Language Arts Performance Assessment: Its Fairness and Predictive Validity Jia Wang, David Niemi, Pete Goldschmidt, and Haiwen Wang UCLA Graduate](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a22f24/html5/thumbnails/8.jpg)
Hierarchical Linear Model(student level)
Student level variables:
• Gender• Ethnicity• English language proficiency• Home language• Immigrant status• SES (lunch program and Title 1)• Special education program• Gifted status
![Page 9: English Language Arts Performance Assessment: Its Fairness and Predictive Validity Jia Wang, David Niemi, Pete Goldschmidt, and Haiwen Wang UCLA Graduate](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a22f24/html5/thumbnails/9.jpg)
Results on Fairness(Variance Partition)
Table 6Variance Component Results For SAT9 Reading, SAT9 Math and PA Scores
SAT9 Reading
SAT9 Math PA
SAT9 Reading
SAT9 Math PA
Level 1 - StudentProportion of variance attributable to students 93.4% 94.4% 92.3%
% variance reduced due to student variables 18.0% 12.5% 6.1%
Level 2 - SchoolProportion of variance attributable to schools 6.6% 5.6% 7.7%
% variance reduced due to school variables 65.3% 47.3% 9.8%
With PredictorsWithout Predictors
![Page 10: English Language Arts Performance Assessment: Its Fairness and Predictive Validity Jia Wang, David Niemi, Pete Goldschmidt, and Haiwen Wang UCLA Graduate](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a22f24/html5/thumbnails/10.jpg)
Results on Fairness(summary 1)
SAT9 Reading SAT 9 Math Performance Assessment
School-level VariablesSchool averageAverage classroom size% of students in lunch program
School API rank in 2001 Positive PositiveSchool enrollment size
Student-level Variables
Female Negative Negative PositiveEthnicity-Black Negative NegativeEthnicity-Hispanic NegativeEthnicity-Asian NegativeEthnicity-Other English language learners Negative Negative NegativeRe-designated Fluent E.P. PositiveHome Language - Spanish NegativeHome Language - Other
Immigrant NegativeFree/Reduced Fee Lunch NegativeTitle1 Negative NegativeSpecial Education Negative Negative NegativeGifted Positive Positive Positive
![Page 11: English Language Arts Performance Assessment: Its Fairness and Predictive Validity Jia Wang, David Niemi, Pete Goldschmidt, and Haiwen Wang UCLA Graduate](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a22f24/html5/thumbnails/11.jpg)
Results on Fairness(summary 2)
School level:• None of the 4 variables were significant
Student level:• Ethnicity and SES were not significant• Positive female effect• Positive gifted and former ELL effects• Negative ELL, Spanish (HL), immigrant,
and special education effects
![Page 12: English Language Arts Performance Assessment: Its Fairness and Predictive Validity Jia Wang, David Niemi, Pete Goldschmidt, and Haiwen Wang UCLA Graduate](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a22f24/html5/thumbnails/12.jpg)
Result on Fairness(school level)
Coefficients Effect Size
SAT9
Reading SAT 9 Math PA
SAT9 Reading
SAT 9 Math PA
School-level Variables
School average 22.38 36.58 1.80
(5.39) (5.33) (0.53)
Average classroom size 0.32 0.15 0.00 0.03 0.01 0.00 (0.19) (0.18) (0.02) % of students in lunch program 4.31 4.39 0.31 0.38 0.36 0.43 (2.47) (2.94) (0.21) School API rank in 2001 0.60 * 0.52 * 0.01 0.05 0.04 0.02 (0.18) (0.21) (0.02) School enrollment size 0.19 -0.03 0.01 0.02 0.00 0.01 (0.40) (0.45) (0.04)
![Page 13: English Language Arts Performance Assessment: Its Fairness and Predictive Validity Jia Wang, David Niemi, Pete Goldschmidt, and Haiwen Wang UCLA Graduate](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a22f24/html5/thumbnails/13.jpg)
Result on Fairness(student level 1)
Coefficients Effect Size
SAT9
Reading SAT 9 Math PA
SAT9 Reading
SAT 9 Math PA
Student-level Variables Female -0.80 * -2.10 * 0.10 * -0.07 -0.17 0.13 (0.26) (0.31) (0.02)
English language learners (ELL) -7.19 * -4.03 * -0.19 * -0.63 -0.33 -0.25 (0.72) (0.69) (0.04) Former ELL 1.25 0.90 0.10 * 0.11 0.07 0.14 (0.65) (0.70) (0.04) Home Language - Spanish -1.24 1.69 -0.12 * -0.11 0.14 -0.17 (0.85) (0.89) (0.05) Home Language - Other -0.48 2.91 0.02 -0.04 0.24 0.02 (1.77) (1.53) (0.07) Immigrant -0.11 0.15 -0.04 * -0.01 0.01 -0.05 (0.27) (0.41) (0.02) Special Education -10.04 * -8.73 * -0.42 * -0.88 -0.72 -0.58 (1.13) (1.72) (0.10) Gifted 11.67 * 14.84 * 0.40 * 1.02 1.22 0.54 (1.62) (1.72) (0.06)
![Page 14: English Language Arts Performance Assessment: Its Fairness and Predictive Validity Jia Wang, David Niemi, Pete Goldschmidt, and Haiwen Wang UCLA Graduate](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a22f24/html5/thumbnails/14.jpg)
Results on Fairness(Student level 2)
Coefficients Effect Size
SAT9
Reading SAT 9 Math PA
SAT9 Reading
SAT 9 Math PA
Student-level Variables Ethnicity-Black -5.32 * -5.57 * -0.06 -0.47 -0.46 -0.08 (1.09) (1.04) (0.06) Ethnicity-Hispanic -1.81 -3.35 * 0.07 -0.16 -0.28 0.09 (0.99) (1.40) (0.06) Ethnicity-Asian -3.82 * 3.16 -0.05 -0.33 0.26 -0.07 (1.22) (1.90) (0.09) Ethnicity-Other -1.65 0.33 0.06 -0.14 0.03 0.08 (1.31) (1.90) (0.07) Free/Reduced Fee Lunch -1.29 * 0.41 -0.04 -0.11 0.03 -0.06 (0.39) (0.45) (0.02) Title1 -1.87 * -1.83 * -0.05 -0.16 -0.15 -0.07 (0.69) (0.77) (0.03)
![Page 15: English Language Arts Performance Assessment: Its Fairness and Predictive Validity Jia Wang, David Niemi, Pete Goldschmidt, and Haiwen Wang UCLA Graduate](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a22f24/html5/thumbnails/15.jpg)
Results on Predictive Validity(cross-tabulation)
Figure 1. Percent of Grade 10 Students Passing CAHSEE (ELA) As Predicted
By Their Grade 9 Performance Assignment Scores (N = 5,427)
37.5%
56.4%
75.3%
92.2%
0%
25%
50%
75%
100%
Not proficient Partially proficient Proficient Advanced
2001 Performance Assignment Scores
Pe
rce
nt o
f Stu
de
nts
Pa
ssin
g th
e
20
02
CA
HS
EE
![Page 16: English Language Arts Performance Assessment: Its Fairness and Predictive Validity Jia Wang, David Niemi, Pete Goldschmidt, and Haiwen Wang UCLA Graduate](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a22f24/html5/thumbnails/16.jpg)
Results on Predictive Validity(school level)
Table 8
HLM Results on Performance Assessment's Predictive Validity
Coefficient Std. Error Log Odds
School-level Variables
Intercept -5.664 * 1.169 0.00
Average Class Size -0.009 0.040 0.99
Percent of Students in Lunch Program 0.103 0.389 1.11
School API Rank in 2001 0.023 0.037 1.02
School Enrollment Size 0.215 * 0.081 1.24
![Page 17: English Language Arts Performance Assessment: Its Fairness and Predictive Validity Jia Wang, David Niemi, Pete Goldschmidt, and Haiwen Wang UCLA Graduate](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a22f24/html5/thumbnails/17.jpg)
Results on Predictive Validity(student level)
Coefficient Std. Error Log Odds
PA 2001 0.385 * 0.042 1.47
SAT9 Reading 2001 0.123 * 0.006 1.13
SAT9 Math 2001 0.021 * 0.004 1.02
GPA 2001 0.365 * 0.067 1.44
Female 0.382 * 0.080 1.46
Ethnicity-Black -0.745 * 0.304 0.47
Ethnicity-Hispanic -0.394 0.313 0.67
Ethnicity-Asian -0.002 0.209 1.00
Ethnicity-Other -0.117 0.255 0.89
English Language Learners -0.397 * 0.121 0.67
Re-designated Fluent E.P. 0.369 * 0.108 1.45
Home Language - Spanish -0.188 0.162 0.83
Home Language - Other -0.278 0.296 0.76
Immigrant Status -0.118 0.076 0.89
Free/Reduced Fee Lunch -0.174 0.091 0.84
Title1 -0.129 0.120 0.88
Special Education -1.493 * 0.381 0.22
Gifted 0.152 0.193 1.16
![Page 18: English Language Arts Performance Assessment: Its Fairness and Predictive Validity Jia Wang, David Niemi, Pete Goldschmidt, and Haiwen Wang UCLA Graduate](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a22f24/html5/thumbnails/18.jpg)
Results on Predictive Validity(reading)
Figure 2:Probability of Passing CAHSEE by SAT9 Reading Scores
0%
20%
40%
60%
80%
100%
0 20 40 60 80 100
SAT9 Reading Scores
Pro
bab
ility
of
Pas
sin
g C
AH
SE
E
Not Proficient
Partially Proficient
Proficient
Advanced
Note: The calculation w as done assuming the student to be a non-immigrant male White English-proficient student w ho spoke English at home, paid for lunch at school, w as not classif ied as Title 1, gifted, and special education, and scored at the mean level in GPA and SAT9 mathematics test w hile enrolled in a school w ith mean school characteristics.
![Page 19: English Language Arts Performance Assessment: Its Fairness and Predictive Validity Jia Wang, David Niemi, Pete Goldschmidt, and Haiwen Wang UCLA Graduate](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a22f24/html5/thumbnails/19.jpg)
Results on Predictive Validity(mathematics)
Figure 3:Probability of Passing CAHSEE by SAT9 Mathematics Scores
0%
20%
40%
60%
80%
100%
0 20 40 60 80 100
SAT9 Mathematics Scores
Pro
bab
ilit
y o
f P
assin
g C
AH
SE
E
Not Proficient
Partially Proficient
Proficient
Advanced
Note: The calculation was done assuming the student to be a non-immigrant male White English-proficient student who spoke English at home, paid for lunch at school, was not classified as Title 1, gifted, and special education, and scored at the mean level in GPA and SAT9 reading test while enrolled in a school with mean school characteristics.
![Page 20: English Language Arts Performance Assessment: Its Fairness and Predictive Validity Jia Wang, David Niemi, Pete Goldschmidt, and Haiwen Wang UCLA Graduate](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a22f24/html5/thumbnails/20.jpg)
Conclusions
1. The performance assessment a fair test.
2. The performance assessment is predictive of students’ California High School Exit Exam results.