a statistical linkage between naep and ecls-k grade eight reading assessments enis dogan
DESCRIPTION
A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics Services Institute American Institutes for Research. Purpose. - PowerPoint PPT PresentationTRANSCRIPT
A Statistical Linkage Between
NAEP and ECLS-K Grade Eight Reading Assessments
Enis Dogan
Burhan Ogut
Young Yee Kim
Sharyn Rosenberg
NAEP Education Statistics Services InstituteAmerican Institutes for Research
A Statistical Linkage Between
NAEP and ECLS-K Grade Eight Reading Assessments
Enis Dogan
Burhan Ogut
Young Yee Kim
Sharyn Rosenberg
NAEP Education Statistics Services InstituteAmerican Institutes for Research
Purpose
In Spring of 2007, about 1300 students took the NAEP and ECLS-K grade eight reading assessments
The purpose of this study is to establish a statistical link between these two assessments using this common sample
We also test the validity of the link by comparing the model projected results to reported NAEP results
Early Childhood Longitudinal Study Kindergarten Cohort (ECLS-K)
Provides longitudinal data on students’ academic achievement in reading, mathematics, and science
Began in the fall of 1998 with a national sample of 21,000 kindergartners
Each student is tested by a series of two-stage, 30-minute adaptive tests, administered face to face, on seven occasions
Item parameters and individual scores are estimated using IRT
NAEP 8th Grade Reading 2007
Provides cross-sectional data on students’ academic achievement in reading
Nationally representative sample of more than 350,000 students participated in the 2007 reading assessment
Each student takes just a portion of the test (BIB Design), consisting of two 25-minute sections or one 50-minute section
Item parameters are estimated using IRT, but results are reported at group level. NAEP does not report individual scores
Differences between NAEP and ECLS-K
The two assessments have similar frameworks, but do not share any common items
NAEP uses a nationally representative sample ECLS-K is representative of children enrolled in first grade
in the U.S. in 1999-2000
Linking through a common sample
In Spring of 2007, about 1300 public school students took NAEP and ECLS-K grade 8 reading assessments. o About ¼ were part of the operational NAEP sample o The rest took the NAEP assessment for the purposes of
the common sample linking study
Method and challenges
NAEP scores in the form of plausible values were not available for about ¾ of the linking sample since they were not part of the operational NAEP assessment
Using item level data from the entire common sample, MML regression was used (with the AM software) to estimate a projection equation predicting NAEP score distributions from ECLS-K scores
The projection equation was
where y is the NAEP score and x is the ECLS-K score
i i iy a bx e
The projection equation
The projection equation was estimated using two different weights:o the original ECLS-K sampling weights, and o poststratification weights
Parameter estimates predicting NAEP scale scores from ECLS-K scale scores
1 Using original weights: RMSE = 18.84, F(1,168) = 326.47
2 Using poststratification weights: RMSE = 18.60, F(1,168) = 370.64
Parameter Original estimate1 Poststratified estimate2
Intercept 79.32 (10.46) 75.86 (9.92)
Slope 1.10 (0.06) 1.12 (0.06)
Variance of the projection
For the projection equation, the a, b parameter estimates are denoted
and their covariance as
The projected NAEP average reading score is
The variance of the projection at mean is
i i iy a bx e
2
2
ˆ ˆˆˆ( , )ˆ ˆa ab
ab b
Var a b
ˆˆ ˆy a bx
2 2 2 2ˆˆ ˆ ˆ ˆVar 2 Vara ab by x x b x
Comparing reported and projected NAEP mean
National average scores and 95% confidence intervals for projected and reported 2007 NAEP reading scores: grade 8, public
Scores by gender and race/ethnicity
Left bar: Confidence Interval for the mean based on the projection using the original weights
Right bar: Confidence Interval for the reported NAEP mean
N
AE
P S
cale
What is the relationship between reading performance at first, third and fifth grades and Proficiency in eighth-grade NAEP reading assessment?
Early childhood reading performance and proficiency in NAEP
Show an overall understanding of the text, including inferential as well as literal information
Extend the ideas in the text by making clear inferences from it, by drawing conclusions, and by making connections to their own experiences—including other reading experiences
Identify some of the devices authors use in composing text
Proficiency in eighth-grade NAEP reading assessment
Level 1 : Letter recognition: identifying upper- and lower-case letters by name
Level 2 : Beginning sounds
Level 3 : Ending sounds
Level 4 : Sight words
Level 5 : Comprehension of words in context
Level 6 : Literal inference
Level 7 : Extrapolation
Level 8 : Evaluation
Level 9 : Evaluating nonfiction
Level 10: Evaluating complex syntax: evaluating complex syntax and understanding high-level nuanced vocabulary in biographical text.
ECLS-K Reading performance levels
Percentage of students at Levels 5 through 9: Grade 5
Reading performance at grade 5 and Proficiency in 8th grade NAEP
Percentage of students at Levels 5 through 9: Grade 5
Reading performance at grade 5 and Proficiency in 8th grade NAEP
33 %
48 %
72 %Evaluation: demonstrating understanding of author’s craft …
13 %
Percentage of students at Levels 4 through 8: Grade 3
Reading performance at grade 3 and Proficiency in 8th grade NAEP
Percentage of students at Levels 4 through 8: Grade 3
Reading performance at grade 3 and Proficiency in 8th grade NAEP
11 %25 %
46 % 62 %
Extrapolation: identifying clues used to make inferences, …
Percentage of students at Levels 3 through 7: Grade 1
Reading performance at grade 1 and Proficiency in 8th grade NAEP
Reading performance at grade 1 and Proficiency in 8th grade NAEP
13 %
34 %
49 %
73 %
85 %
Percentage of students at Levels 3 through 7: Grade 1
Comprehension of words in context: reading words in context
Summary of findings
Using a common sample, a linking equation was estimated
Using the equation, projected NAEP scores were computed for ECLS-K 8th graders
Mean projected NAEP scores for the nation and the gender and racial/ethnic groups were close to reported NAEP results
Mean projected NAEP scores were used to examine the relationship between proficiency in grade eight NAEP reading and earlier reading performance