developing an early diagnostic assessment for reading skills: the research basis for the texas...
TRANSCRIPT
Developing an Early Diagnostic Assessment for Reading Skills: The Research Basis for
the Texas Primary Reading Inventory
David J. Francis
Texas Institute for Measurement, Evaluation, & Statistics
Department of Psychology
University of Houston
Collaborators in Initial Development of TPRICenter for Academic and Reading Skills
UT-Houston HSC
– Barbara R. Foorman
– Jack M. Fletcher
– Angeliki Mousaki
Univ. of Houston
– Chris Schatschneider
– Coleen D. Carlson
– Kimberly J. Wristers
– Dung-tsa Chen
Overview
• Background on Reading and Reading Problems
• Overview of TPRI
• Development of TPRI Screening Components
• Reliability and Validity of Screening Components
• Conclusions and Future Directions
What we know about reading, reading acquisition, and reading problems
Reading is • the process of extracting meaning from print• a developmental process• dependent on language• a complex behavior that is acquired
– playing the piano vs learning one’s native
language
What we know about reading, reading acquisition, and reading problems
• Reading outcomes vary across children. – some children learn almost magically (about
5%)– others learn easily no matter what method of
instruction is used (about 20%-30% )– about 30% of children struggle to learn to read
(equal for boys and girls)
What we know about reading, reading acquisition, and reading problems
• Most reading problems occur at the level of the single word
– reading is characterized by slow and labored decoding
– comprehension suffers due to inefficiency of decoding
• Phonemic awareness is an important skill that facilitates the development of word recognition skills
Importance of Early Assessment and Intervention for Reading Problems
• Reading problems are associated with poor educational and social outcomes
• Children do not simply outgrow reading problems.
• Early intervention is clearly effective (Torgesen, 1997; Vellutino et al., 1996)
• Not all children respond equally well to all interventions.
• Combining early identification with targeted intervention, the effectiveness of early interventions may be enhanced
Reading in Texas
• State Curriculum Standards (TEKS)
• State-wide Accountability System– TAAS (TEKS based CRT beginning in G3) – Academic Performance Indicators
• 1997 - Diagnostic Reading Instruments Bill– Link to TEKS– Teacher administered to individual students– Identify reading problems in K-2 Inform instruction– Inform instruction All children read by Grade 3– NOT part of State-wide Accountability System
Overview of the TPRI
• Teacher administered• Kindergarten to Grade 2• At each grade, test comprised of two major
sections:– Screening (identify children NOT at risk)
– Inventory (inform instruction)
Kindergarten TPRI
• Administered middle and end of year• Screening
– Letter Names & Sounds
– Phonemic Awareness (Blending Onsets & Rimes)
• Inventory– Book & Print Awareness
– Phonemic Awareness (Rhyming, Blending Word Parts, Blending Phonemes, Detecting Initial Sounds, Detecting Final Sounds)
– Graphophonemic Knowledge (Letter Name Identification, Letter to Sound Linking)
– Listening Comprehension
First Grade TPRI
• Screening (beginning and end of year)– Letter Names & Sounds (beginning of year only)
– Phonemic Awareness • Blending Word Parts (Beginning of year only)
• Blending Phonemes (End of year only)
– Word Reading (beginning and end of year)
• Inventory– Phonemic Awareness
– Graphophonemic Knowledge
– Word Reading (Placement on comprehension stories)
– Reading Comprehension
Second Grade TPRI
• Screening (beginning of year only)– Word Reading
• Inventory– Graphophonemic Knowledge
– Word Reading (Placement on comprehension stories)
– Reading Comprehension
Empirical Data Behind TPRIDesign of EARS Study
Cohort Year 1 Year 2 Year 3 Year 4
1 2
2 1 2
3 K 1 2
4 K 1 2
5 K 1
Sample Demographics
• Total N=945 across the 5 cohorts• 3 schools• All children in all K, 1, and 2 invited. (Random
sample of those consenting - 80+%)• Free lunch participation ranged from 13% to 30%• Boys and girls were equally represented• Caucasian (54%), African American (18%),
Hispanic (15%), Asian (12%)• SES - LC (9%), WC (43%), MC (48%)
Measures Assessed Four Times per Year
Skill Assessed K 1 2
Letter Names X
Letter Sounds X
Vocabulary X X X
Rapid Naming X X X
Phonemic Awareness X X X
Visual Motor Integration X X X
Recognition Discrimination X X X
Word Reading X X
Spelling X X
End of Year Outcomes in Grades 1 and 2
• Woodcock-Johnson Reading (LW, WA, PC)• KTEA Spelling• Gray Oral Reading Test• Formal Reading Inventory• WISC
• WJ Broad Reading Score was used to form criterion outcome in Grades 1 ( 1.4) and in Grade 2 ( 2.4), which correspond to the 18th and 35th percentiles, respectively
Longitudinal Sample Sizes in EARS Study
Screening Outcome n
Middle Kindergarten End of Grade 1 421
End of Kindergarten End of Grade 1 421
Beginning of Grade 1 End of Grade 1 599
End of Grade 1 End of Grade 2 376
Beginning of Grade 2 End of Grade 2 540
Steps Involved in Development of Each Screen
1) Scale Predictors Using IRT where possible and Estimate Ability for all subjects
2) Determine Criterion of Interest
3) Predict Criterion from Latent Ability Estimate
4) Manipulate Cut-Point to Achieve Desired False-Negative Rate
5) Determine Latent Ability at the Cut-Point (4)
6) Select Items to Discriminate at Latent Ability Determined in (5)
Hypothetical Item Characteristic Curves
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
-3.5 -3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3 3.5
Ability
Probability of a Correct
Response
1 2 3
IRT vs. Classical Test Theory
• Classical test theory defines true score as persons expected (i.e., average) score on the test– score is dependent on the items used in making up the
test. If the test is made more difficult, a person’s “abilty” goes down
• The IRT model is a latent trait, or strong true score model– ability is independent of the items used in the test
– because we know the fundamental relation between item responses and ability, we can estimate ability on this constant metric, even though the items in the test may change
Advantages of IRT utilized in the TPRI
• Focus the selection of screening items at the cut-point on the ability dimension
• Keep screening short and still afford accurate discrimination around the cut-point
• Develop new screening and inventory items through appropriate linking studies
• Develop word lists linked to stories for placement of students on comprehension stories
Definitions of Decisions
Identifications
Outcomes No Risk At Risk Total
No Risk Correct
d
False Positive
b d + b
False Positive
Rate = b/(b+d)
At Risk False Negative
c
Correct
a c + a
False Negative
Rate = c/(a+c)
Steps Involved in Development of Each Screen
1) Scale Predictors Using IRT where possible and Estimate Ability for all subjects
2) Determine Criterion of Interest
3) Predict Criterion from Latent Ability Estimate
4) Manipulate Cut-Point to Achieve Desired False-Negative Rate
5) Determine Latent Ability at the Cut-Point (4)
6) Select Items to Discriminate at Latent Ability Determined in (5)
Steps Involved in Development of Each Screen
1) Scale Predictors Using IRT where possible and Estimate Ability for all subjects
2) Determine Criterion of Interest
3) Predict Criterion from Latent Ability Estimate
4) Manipulate Cut-Point to Achieve Desired False-Negative Rate
5) Determine Latent Ability at the Cut-Point (4)
6) Select Items to Discriminate at Latent Ability Determined in (5)
Hypothetical Item Characteristic Curves
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
-3.5 -3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3 3.5
Ability
Probability of a Correct
Response
1 2 3
End of Grade 1 Basic Reading Using December Kindergarten Decision Rules
Identifications
Outcomes No Risk At Risk Totals Error Rates
No Risk 181 143 324 False Positive = .44
At Risk 5 92 97 False Negative = .05
Total 186 235 421
End of Grade 1 Basic Reading Using April Kindergarten Decision Rules
Identifications
Outcomes No Risk At Risk Totals Error Rates
No Risk 200 124 324 False Positive = .38
At Risk 10 87 97 False Negative = .10
Total 210 211 421
End of Grade 1 Reading Using October Grade 1 Word Reading and Phonological Awareness
Identifications
Outcomes No Risk At Risk Totals Error Rates
No Risk 305 175 480 False Positive = .37
At Risk 8 111 119 False Negative = .07
Total 313 286 599
End of Grade 2 using End of Grade 1 Word Reading
Identifications
Outcomes No Risk At Risk Totals Error Rates
No Risk 217 82 299 False Positive = .27
At Risk 3 74 77 False Negative = .04
Total 220 156 376
End of Grade 2 using End of Grade 1 Word Reading and Phonological Awareness
Identifications
Outcomes No Risk At Risk Totals Error Rates
No Risk 229 70 299 False Positive = .23
At Risk 6 71 77 False Negative = .08
Total 220 156 376
Beginning of Grade 2 to End of Grade 2
Identifications
Outcomes No Risk At Risk Totals Error Rates
No Risk 370 65 435 False Positive = .15
At Risk 9 93 102 False Negative = .09
Total 379 158 537
TPRI Field Study
• Investigate teacher use• Examine sources of variability in test performance
(Content, administrator, time)• Data Collected on 4 forms
– End of Kindergarten
– Beginning of First Gradea
– End of First Grade
– Beginning of Second Gradeb
aparticipants were at the end of Kindergartenbparticipants were at the end of Grade 1
Design of TPRI Field Study
Group Schools Number Form May 5Teacher
May 11Teacher
K 1-2 56 1 Own AlternateK 1-2 56 1 Alternate OwnK 3-4 55 2 Own AlternateK 3-4 55 2 Alternate Own1 1-2 44 3 Own Alternate1 1-2 44 3 Alternate Own1 3-4 62 4 Own Alternate1 3-4 62 4 Alternate Own
Field Study Estimates of Reliability for Screens
Alpha Time 1 Alpha Time 2 *Test-Retest
GeneralizabilityCoefficient
Kappa forTask
End of Kindergarten Screen
Letter-name .92 .92 .95 .75 .93
Letter-sound .93 .93 .87 .94 .61
Blending .92 .94 .66 .76 .64
Beginning of Grade 1 Screen
Letter-name .74 .80 .80 .96 .73
Letter-sound .86 .89 .76 .86 .60
Word Reading .91 .89 .87 .90 1.0
Blending Phonemes .80 .89 .70 .85 .35
End of Grade 1 Screen
Word Reading .88 .89 .93 .95 .69
Blending Phonemes .74 .75 .51 .58 .87
Beginning of Grade 2 Screen
Word Reading .92 .91 .90 .88 .69Note.*Test –Retest reliability estimate is a lower-bound estimate, since two different raters were used. One week passedbetween the two occasions.
Conclusions
• The TPRI is currently in use in over 80% of Texas Public Elementary schools
• Feedback from teachers has been widely positive• Teachers have asked for more training• Current emphasizing development of interventtion
strategies and developing more effective means for disseminating this information to teachers
• The screening device has proven helpful in optimizing assessment time for teachers
Conclusions (cont.)
• Opportunities for Research– linking to outcome assessments used in the
accountability system
– evaluation of instructional decision making and intervention strategies
– improved decision making through computer aided administration and student profiling
– development of comparable instruments for literacy instruction in Spanish (Tejas Lee)