assessing the english language proficiency of english learners with disabilities u.s. department of...
TRANSCRIPT
Assessing the English Language Proficiency of English Learners With DisabilitiesU.S. Department of Education
March 16, 2015
Copyright © 2015 American Institutes for Research. All rights reserved.
Agenda9:00 – 9:15am Welcome and Overview
9:15 – 11:05am Panel Session 1: Background – Differentiating Language and Literacy Acquisition From Disability
11:05 – 11:15am Break
11:15 – 1:05pm Panel Session 2: Fostering Valid and Reliable Assessments for ELs With Disabilities
1:05 – 2:00pm Lunch
2:00 – 3:00pm Panel Session 3: Assessing ELs With Significant Cognitive Disabilities
2
PanelistsPanel Session 1: Background – Differentiating Language and Literacy Acquisition From Disability Presenters: Aquiles Iglesias; Esther Geva; Sylvia Linan-Thompson; Alba Ortiz Moderator: Diane August
Panel Session 2: Fostering Valid and Reliable Assessments for ELs With Disabilities Presenters: Martha Thurlow; Jamal Abedi; Phoebe Winter; Gary Cook Moderator: Robert Linquanti
Panel Session 3: Assessing ELs With Significant Cognitive Disabilities Alternate ELP Standards and Assessments, and Growth and Attainment Criteria Presenters: Martha Thurlow; Gary Cook Moderator: Brian Gong
3
Meeting Purpose Build the knowledge base related to ELs with disabilities
and inform future research and development in this area Inform the work of a panel constituted by the US
Department of Education
• Panel constituted to develop a peer review guide related to Title III assessment and accountability provisions.
• Peer review guide will assist SEAs in preparing materials for the review and peer reviewers in conducting the review.
• Guide includes elements related to Title III assessment and accountability and evidence states must provide to show compliance with provisions of the law.
4
Background: Title III Title III of ESEA provides federal funding to state and local
education agencies to develop language instruction programs that assist ELs in acquiring English and meeting the same academic content standards as their English-proficient peers.
It also inaugurated important changes in assessment and accountability for EL students.
5
Background: Title III Title III of the law requires states to establish state standards for
English language proficiency (ELP) that correspond with state academic content standards required under Title I.
It requires an annually administered English language proficiency assessment based on those standards and measuring the four domains of reading, writing, listening, and speaking.
6
Background: Title III Title III also instituted new accountability requirements for
districts and states. New EL accountability provisions require states to:
• define criteria for progress in learning English
• establish a performance standard for English language proficiency and academic content knowledge
• set annually increasing performance targets for the number and percentage of ELs meeting these criteria
7
Background: Title III The Annual Measurable Achievement Objectives include:
• AMAO 1: annual increases in the number or percentage of ELs making progress in learning English;
• AMAO 2: annual increases in the number or percentage of ELs attaining English language proficiency (ELP) by the end of each school year as determined by a valid and reliable assessment of ELP; and
• AMAO 3: annual increases in the number or percentage of ELs making adequate yearly progress (AYP) on content area assessments.
8
DemographicsIn 2012–13: ELs comprised 9% of students enrolled in U.S. public schools There were 543,916 English learners with disabilities (this
represents 8.5% of ELs and 13% of all students with disabilities)
9
Identification of Speech-Language Impairment in ELsAquiles Iglesias
University of Delaware
March 16, 2015
10
Overview What we know about language acquisition in ELs
• Implications for assessment
Identification of:• Students with True Language Impairment
• Students with language learning difficulties not eligible for special services
• Typically developing students who are not making adequate progress
11
Knowledge Base and Implications for AssessmentLanguage acquisition • Innate capacity to learn
• Linguistic environment
Must assess:• Process of learning
• Product of learning
– (innate capacity + linguistic environment)
12
Knowledge Base and Implications for Assessment Growth trajectories of oral
measures related to literacy for TD K-3 ELs
Must assess using language development norms, trajectories, and growth rates of similar students• Gender
• Beginning L2 proficiency status
• Language learning ability
13
Knowledge Base and Implications for Assessment Children acquiring 2 languages Knowledge acquired is distributed across 2 languages
• Distribution is not always equal
14
Unequal Distributed Knowledge
VOCABULARY
15
Knowledge Base and Implications for AssessmentChildren acquiring two languages Knowledge acquired is distributes across two languages (not
always equal) Total score in one language only provides partial information. Best measure of language ability takes into consideration
distributed knowledge. Early identification and intervention results in better and more
cost effective outcomes. Assess early
16
The Ideal Assessment Should: Occur early Assess process and product Account for distributed knowledge Compare performance to similar students
• Static
• Growth
17
Language Assessments for Spanish-English Speaking Children* Preschool Computerized Language Assessment (PCLA)
• Funding: Institute of Education Sciences Grant R305A110284 Bilingual English Spanish Assessment (BESA)
• Funding: NIDCD N01-DC-8-2100 Systematic Analysis of Language Transcripts (SALT)
• Partial Funding NIH and IES Grants HD59321 and R305U010001
*Disclosure: Aquiles Iglesias has financial interests in all three assessments
18
Our Task
19
Preschool Computerized Language Assessment (PCLA)
20
Our Task – Screening With PCLAEnglish Total Scores and Component ScoresSpanish Total Score and Component Scores
Best Score Total
21
Our Task – Screening With PCLAEnglish Total Scores and Component ScoresSpanish Total Score and Component Scores
Best Score Total
22
Our Task – Identification of True Language Impaired
23
Bilingual English Spanish Assessment (BESA) Domains
• Semantics
• Morphosyntax
• Phonology
LI Markers vary across languages
• English: Past tense, Plural Nouns
• Spanish – Articles and Clitics
24
Bilingual English Spanish Assessment (BESA)
25
Our Task – Identification of True Language Impaired Older Students
Systematic Analysis of Language Transcripts (SALT)
Normed on over 4,000 English and Spanish Language Samples
Grades K-3, Ages 5 – 9
26
Elicitation ProcedureModel Retell Tell
27
Decision Making Process
Assess Language
X
Compare to Language X
Norms
WITHIN NORMAL LIMITS
Assess Language
Y
Compare to Language Y
Norms
WITHIN NORMAL LIMITS
28
English Spanish
29
Monitoring Progress-Language Samples
30
Monitoring Progress-Language Samples
31
Identification of Students Students with True Language Impairment
Students with language learning difficulties not eligible for special services
Typically developing students that are not making adequate progress
32
ReferencesIglesias, A., & Rojas, R. (2012). Bilingual language development of ELLs: Modeling the
growth of two languages. In B. Goldstein (Ed.), Bilingual language development and disorders: Past, present, and future (pp. 1-30). Baltimore, MD: Paul H. Brookes.
Miller, J. F., Heilmann, J., Nockerts, A., Iglesias, A., Fabiano, & Francis, D. (2006). Oral language and reading in bilingual children. Learning Disabilities Research and Practice, 2(1), 30-43.
Peña, E. (2007). Lost in Translation: Methodological considerations in cross-cultural research. Child Development, 78, 1255-1264.
Rojas, R., & Iglesias, A. (2009, March 3). Making a case for language sampling. The ASHA Leader, 14(3), 10-13.
Rojas, R., & Iglesias, A. (2013). The language growth of English language learners: Change over time, individual differences, and the impact of initial status on growth. Journal of Child Development, 84(2), 630-646.
33
AssessmentsGolinkoff, R., de Villiers, J., Hirsh-Pasek, K., Iglesias, A., & Wilson, M.
(under development). Preschool Computerized Language Assessment.
Iglesias, A., & Miller, J. (2012). Systematic Assessment of Language Samples. SALT Software, LLC, Middletown, WI. (saltsoftware.com)
Peña, E., Gutierrez-Clellen, V., Iglesias, A. Goldsteim, B., & Bedore, L. (2014). Bilingual English Spanish Assessment. A-R Clinical Publications, San Rafael, CA.
34
Aquiles IglesiasCommunication Sciences and DisordersUniversity of DelawareAquiles @UDEL.edu
35
Assessing Disabilities in ELs Struggling With Phonological Awareness, Word Recognition, and Decoding SkillsEsther Geva
University of Toronto
March 16, 2015
36
Research Agenda Clinical, assessment, and instructional issues related to:
• Over-identification (Cummins, 1991)
• Under-identification (Limbos & Geva, 2001)
The challenge:
• How do we tease apart L2 status from a possible learning disability?
37
Strategies Guiding My Research on Language and Literacy in ELs
Describe normal development
Examine universal/typology specific language and literacy
Study factors that explain development (typology, cognition, context…)
How can we identify reliably L2 learners who may be also LD?
38
What Do the Data Show About Typical Development of Reading in ELs? Longitudinal Design Cross-sequential (4 cohorts) A large task battery administered annually
• Each cohort tracked from Grade 1 to Grade 6
• 13 schools across 4 boards of education in a large metropolis
– mixed catchment area
– first generation immigrants
– communities average poverty incidence = 23%
– income below median for the metropolis
39
Test BatteryCognitive/ Linguistic
• non-verbal ability• rapid naming (letters)• phonological awareness• memory• auditory discrimination
Oral Language• vocabulary
• breadth• depth, academic• grammar
• listening comprehension
Word Level• pseudoword decoding
• word recognition
• word fluency
• spelling
Text Level• reading comprehension
• text fluency
• story writing
40
The Sample Home languages spoken in the EL group:
Punjabi, Tamil, Urdu, Cantonese, Portuguese, Spanish
Groups N Female MaleMean Age in Grade 4
EL 427 213 206 116.12
EL1 158 93 64 118.97
41
Receptive Vocabulary (PPVT): Average Growth Trajectories in EL and EL1
Grade1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 637.40
57.68
77.96
98.25
118.53
Growth From Grade 1 to Grade 6
Vo
cab
ula
ry
EL1
ESL
(Farnia & Geva, 2011)
Persistent EL1-EL Difference
EL1 Slope/Rate of Growth
EL Slope (steeper in early years)
42
Basic Reading Skills: Average Growth Trajectories in EL and EL1
Development of Word Recognition Skills
-45.00 -25.94 -6.88 12.18 31.2415.00
21.04
27.08
33.12
39.16
Growth from Grade 1 to Grade 6
Wor
d R
ecog
nitio
n
EL1
ELL
Wo
rd R
eco
gn
itio
n
Development of Decoding Skills
-45.00 -25.94 -6.88 12.18 31.240.15
8.52
16.89
25.26
33.63
Growth from Grade 1 to Grade 6
Dec
odin
g
EL1ELL
Dec
od
ing
Source: Geva & Farnia, LARCIC, 2009
43
Rapid Letter Naming (RAN): Average Growth Trajectories in EL1 and EL
Growth From Grade 1 to Grade 6
Rap
id N
amin
g
1.00
1.38
1.77
2.15
2.53
Rapid
Nam
ing
-45.00 -25.94 -6.88 12.18 31.24
Growth from Grade 1 to Grade 6
EL1ELL
Source: Geva & Farnia, LARCIC, 2009
44
Phonological Awareness: Average Growth Trajectories in EL1 and EL
-45.00 -25.94 -6.88 12.18 31.244.10
7.30
10.50
13.71
16.91
Growth from Grade 1 to Greade 6
Phonolo
gic
al A
ware
ness
EL1ELL
Growth From Grade 1 to Grade 6
Pho
nolo
gica
l Aw
aren
ess
No EL-EL1 difference
Source: Geva & Farnia, LARCIC, 2009
45
Relationship between Phonological Awareness and Word Reading SkillsGrade 1 Grade 6
X = EL1X = ESL
No EL-EL1 differences
46
Vocabulary - Strong Predictor of Reading Comprehension in Monolinguals and ELs
X = EL1X = ESL
Source: Geva & Farnia, LARCIC, 2009
47
Reading Comprehension*- Average Growth Trajectories in EL and EL1
-32.00 -22.10 -12.20 -2.30 7.60454.0
466.3
478.7
491.1
503.5
Growth from Grade 4 to Grade 6
Rea
din
g C
om
pre
hen
sion
EL1ELL
Source: Farnia & Geva, J of Res in Reading, 2013
48
*Gates-MacGinitie Reading Test-ESS
Persistent EL-EL1 difference
Summary of Developmental Trends: EL1 vs. EL
Skills EL vs. EL1
Rapid naming =
Phonological awareness =
Word reading =
Vocabulary <
Reading comprehension <
Farnia & Geva, 2013
49
Ways of examining “transfer” of skills between L1 and L2 Correlation between L1 and L2 skills (universal
perspective)
Effects of specific L1 features on L2 features (positive/negative)
Geva, E. Written Language & Literacy, 2014
50
Correlations Among English (L1) Predictors (Fall SK) and French Reading a Year Later (Grade 1)
English Predictors (Senior Kindergarten)
GR 1 FRENCH:
Vocab Letter
ID
Ltr-
Sound
Phono
Aware
Blend
Snd
Snd
Match
RAN
Word Reading
.25* .51** .55** .63** .45** .49** -.49**
Story Reading
.22 .61** .53** .50** .44** .48** -.35**
(Endler, M., Ph.D., 2007)
51
Spelling: EL1 Versus EL (Cantonese)
Spelling “th” in the word thick –
Gr. 1-Fall (T1) & Spring (T2)
Spelling “th” in the word thick –
Gr. 2 Fall (T3) & Spring (T4)
(Wang & Geva, 2003)
52
Research on ELs Who Might Have Learning Disabilities in the Following Areas:
Word Reading and Spelling (focus for today)
Reading Fluency
Reading Comprehension
Written Expression
Mathematics Calculations
53
Auditory Discrimination: Developing Sensitivity to New Phonemes Pseudoword Auditory Discrimination Task
• 17 Pseudoword pairs – keathe-keev
– thop-zop
– bish-biss
– shen-sen
“Did I say the same thing?”
54
Development of Auditory Discrimination: At- Risk for Dyslexia vs. No-Risk
Over time L1-EL differences disappear but reading group differences persist
55
Profiles of At-Risk and Not-at-Risk EL1 and EL (End of Grade 2)
-2
-1.5-1
-0.5
0
0.51
1.5
EL1-NAESL-NAEL1-ARESL-AR
Geva, Yaghoub-Zadeh, & Schuster, Annals of Dyslexia, 2000
56
Spelling Growth by Language Group
In Grade 1, Punjabi and Portuguese groups had lower scores, however, their growth was steeper over time.
From Grades 4 to 6, home language was not a predictor of initial status or growth. Geva & Lafrance, 2011
57
Phonological Processing Sub-Groups Phonological Processing (PP) Risk Status:
Below 25%ile on either PA, RAN, or both in Grade 1 (PA-risk, RAN-risk, Double Deficit-DD, Typically Developing)
58
Spelling Growth by PP Risk GroupRisk Groups: Grades 1–3 Risk Groups: Grades 4–6
Initially, all risk groups had lower scores than the No Risk group. L2 status not a good predictor but Phonological Processing (PP) risk
status is.
59
What Predicts Longitudinally Gr. 6 Spelling?
60
Conclusions
61
What Have We Learned? The overall profiles of ELs and EL1s who have a learning
disability are similar (despite differences in language proficiency)
Like monolinguals, ELs with persistent difficulties in word level skills have difficulties with:
• processing factors (e.g., phonological awareness, RAN, memory, auditory discrimination)
• accurate and fluent word reading and spelling
• these difficulties impact reading comprehension and writing
62
Food for Thought Be mindful of current norms Compare relevant performance to reference group What IS the relevant reference group? Consider developmental patterns Consider transfer from the L1(correlational; positive &
negative) Assess evidence based known predictors Expect same % of EL1 and EL with LD (systemic over-
and under-identification are problematic)
63
AcknowledgmentsFunding Support Social Sciences and
Humanities Research Council of Canada
Transfer Grants - Ontario Ministry of Education
Canadian Language and Literacy Research Network (CLLRNet)
University of Toronto
Former/Current Students Mahshid Azimi Todd Cunningham Dana Shafman (David) Fataneh Farnia Maureen Jean Emiko Koyama Adele Lafrance Kate Ndlovu (Herbert) Gloria Ramirez Barbara Schuster Zohreh Yaghoub-Zadeh Lesly Wade-Woolley Min Wang
64
ReferencesFarnia, F. & Geva, E., (2013). Growth and predictors of change in English
language learners' reading comprehension. Journal of Research in reading, 36(4), 389–421. ISSN 0141–0423 DOI:10.1111/jrir.12003,
Farnia, F., & Geva, E. (2011). Cognitive correlates of vocabulary growth in English language learners. Applied Psycholinguistics, 32(4), 711–738
Geva, E. (2006). Second–language oral proficiency and second–language literacy. In D. August & T. Shanahan (Eds.), Developing literacy in second–language learners: A report of the National literacy Panel on language–Minority Children and Youth (123–140). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
65
ReferencesGeva, E. (2014). The cross–language transfer journey – A guide to the perplexed.
In E. Zaretsky & M. Schwartz (Eds.), Cross–linguistic transfer in reading in multilingual contexts – Recent research trends, (1–15). Boston, MA: University of Massachusetts.
Geva, E., & Farnia, F. (2012). Developmental changes in the nature of language proficiency and reading fluency paint a more complex view of reading comprehension in ELL and EL1. Reading and Writing: an Interdisciplinary Journal, 25(8), 1819–1845.
Geva, E., & Herbert, K. (2012). Assessment and interventions for English language learners with learning disabilities. In B. Wong & D. Butler (Eds.), Learning about learning disabilities (4th ed.), (271–298). San Diego, CA: Elsevier Science.
66
ReferencesGeva, E., & Massey–Garrison, A. (2013). A comparison of the language skills of
ELLs and monolinguals who are poor decoders, poor comprehenders or normal readers. Journal of Learning Disabilities, 46(5), 387–401.
Geva, E., & Yaghoub-Zadeh, Z. (2006). Reading efficiency in native English–speaking and English–as–a–second–language children: The role of oral proficiency and underlying cognitive–linguistic processes. Scientific Studies of Reading, 10(1), 31–57.
Geva, E., Yaghoub–Zadeh, Z., & Schuster, B. (2000). Understanding individual differences in word recognition skills of ESL children. Annals of Dyslexia, 50(1), 121–154.
Jean, M., & Geva, E. (2009). The development of vocabulary in English as a second language children and its role in predicting word recognition ability. Applied Psycholinguistics, 30(1), 153–185.
67
ReferencesLimbos, M. & Geva, E. (2001). Accuracy of teacher assessments of second-
language students at risk for reading disability. Journal of Learning Disabilities, 34(2), 136–151.
Ndlovu, K. and Geva, E. (2008). Writing abilities in first and second language learners with and without reading disabilities. In J. Kormos & E.H. Kontra (Eds.), Language learners with special needs: An international perspective. Toronto, Canada: Multilingual Matters.
Saiegh–Haddad, E. & Geva, E. (2008). Morphological awareness, phonological awareness, and reading in English–Arabic bilingual children. Reading and Writing: An Interdisciplinary Journal, 21(5), 481–504.
68
ReferencesWang, M., & Geva, E. (2003). Spelling performance of Chinese children: Lexical
and visual–orthographic processes. Applied Psycholinguistics, 24, 1–25.
Wang, M., & Geva, E. (2003). Spelling acquisition of novel English phonemes in Chinese children. Reading and Writing: An Interdisciplinary Journal, 16(4), 325–348.
69
A Comparison of Global and Discrete Measures When Assessing ELs’ WritingSylvia Linan-Thompson
University of Texas at Austin
March 16, 2015
71
Rationale Recently, there has been increased interest in writing.
72% of fourth graders are below proficient level on the National Assessment of Educational Progress (2003).
Measures of writing that measure student progress and identify students who are struggling are available but ELs have not been included in much of the research.
72
Assessment Curriculum Based Measures: Multiple probes of equivalent
difficulty that are administered repeatedly (Deno, 1985; Jenkins, Deno, & Mirkin, 1979)
Progress monitoring: A nondiscriminatory RTI approach to track student progress (Fuchs & Fuchs, 2006; Speece, Case, & Molloy, 2003)
73
Research Words written, words spelled correctly, and correct letter
sequence were:• Sensitive to growth within year and across grades• Discriminated between student with and without disabilities
(Deno et al., 1982; Marston & Deno., 1981; Marston, Deno, & Tindal, 1983)
Analytic scores added to those measures discriminated between general education students and students with LD, at-risk, and low performance (Tindal & Hasbrouck, 1991).
Percent of words spelled correctly • Best screening tool (Parker et al., 1991b).
74
Research CBMs in reading are not as reliable with ELs as they are
for monolingual students.• (Linan-Thompson, 2010; Linan-Thompson, Vaughn, Prater, &
Cirino, 2006).
75
Exploratory Study To better understand the writing development of ELs in English
and Spanish, we are using global and discrete measures to identify measures that best discriminate among three groups of second-grade students in dual language classrooms.
Students received reading instruction in Spanish; writing instruction was in both languages.
Students had multiple opportunities to write: creative bilingual journals, independent writing, reader response.
Teachers used a writers’ workshop model for writing instruction.
Project ESTRE2LLA, US Department of Education, Office of Special Education Programs
Grant # H326M110010
76
Data Sources Journal samples
Stanford English Language Proficiency
Stanford Spanish Language Proficiency
77
Variables Total number of words
Correct word sequence
Correct word sequence without spelling
Number of correctly spelled words
78
Variables Bilingual strategies
• Discourse level
• Sentence/phrase level
• Word level
(Adapted from Soltero-González, Escamilla, & Hopewell, 2011). Holistic rating
• 1-5 scale
• Organization/content
• Cohesion
• Complexity
79
Findings: Group MeansStudent SELP WW
PreSELP WW
postSSLP WW
preSSLP WW
post
Dyslexia 33.5 38.2 48.25 58
Low language
proficiency27.2 59.4 50 56
Average language
proficiency52 48.3 63.3 64.5
80
Findings: English Journal SamplesStudent WW WW WSC WSC
Dyslexia 30 28 1 9
Low 83 30 23 13
Low 40 37 4 14
Average 46 54 42 45
81
Findings: English Journal Samples
Student CWS CWS w/o spelling CWS CWS w/o
spelling
Dyslexia 0 24 2 26
Low 2 52 5 23
Low 0 35 6 26
Average 34 38 40 48
82
Findings: English Journal SamplesStudent Holistic Holistic
Dyslexia 10 6
Low language proficiency 5 6
Low language proficiency 6 4
Average language proficiency 14 15
83
Findings: Bilingual StrategiesStudent Syntax Literal Translation
Dyslexia Mai tio hies
Low language proficiency
Average language proficiency
…rolacoster of water …another one of ginger
84
Findings: Bilingual StrategiesStudent Language
approximation Phonetic Transfer
DyslexiaSow (so), abautPle stesin, bak, flep
Pleis, ais cetResolors, febret
Low language proficiency
Flawr, gan (going), brouk
Aut sayd (outside), brader, lero, owis, jaga
Average language proficiency
Auncluse (uncles)
85
Summary and Next Steps Generally, average performing students write more words than
students in other groups.
Average proficiency students are better spellers than students in the other two groups as measured by both TWC and CWS.
Average proficiency students have higher holistic scores than students in the other two groups.
Most bilingual strategies are at the word level.
We will continue to code writing to determine whether there are differences between low performing students and students with dyslexia in student growth on any measures.
86
Conclusion (With Limited Data) Writing provides evidence of student development in the
process of becoming bilingual/biliterate. Student writing may be an alternative to parallel
monolingual assessments. Writing may be easier to use and more sensitive than oral
proficiency measures to monitor student language development.
87
ReferencesDeno, S. L. (1985). Curriculum-based measurement: The emerging alternative.
Exceptional Children, 52, 219-232.
Fuchs, D., & Fuchs, L. S. (2006). Instruction to responsiveness-to-intervention: What, why and how valid is it? Reading Research Quarterly, 41, 92-99.
Linan-Thompson, S. (2010). Response to instruction, English language learners and disproportionate representation: The role of assessment. Psicothema 2010, 22(4), pp. 970-974.
Linan-Thompson, S., Vaughn, S., Prater, K., & Cirino, P. T. (2006). The response to intervention of English language learners at-risk for reading problems. Journal of Learning Disabilities, 39, 390-398.
88
ReferencesMarston, D., & Deno, S. L. (1981). The reliability of simple, direct, measures of
written expression (Vol. IRLD-RR-50). University of Minnesota, Institute of Research on Learning Disabilities.
Martson, D., Deno, S. L., & Tindal, G. (1983). A comparison of standardized achievement tests and direct measurement techniques in measuring student progress (Vol. IRLD-RR-126). University of Minnesota, Institute of Research on Learning Disabilities.
McMaster, K., & Epsin, C. (2007). Technical features of curriculum-based measurement in writing: A literature review. The Journal of Special Education, 41, 68-84.
89
ReferencesParker, R. I., Tindal, G., & Hasbrouck, J. (1991b). Progress monitoring with
objective measures of writing performance for students with mild disabilities, Exceptional Children, 58, 61-73.
Soltero-Gonzalez, L., Escamilla, K., & Hopewell, S. (2011). Changing teachers’ perceptions about the writing abilities of emerging bilingual students: Toward a holistic bilingual perspective on writing assessment. International Journal of Bilingual Education and Bilingualism, 15, 71-94.
Speece, D. L., Case, L. P., & Molloy, D. E. (2003). Responsiveness to general education education instruction as the first gate to learning disabilities identification. Learning Disabilities Research & Practice, 18, 147-156.
90
Multitiered Systems of Support for English LearnersAlba A. Ortiz
University of Texas at Austin
March 16, 2015
92
Students Experience Learning Difficulties for a Variety of Reasons Learning problems result from deficiencies in the teaching-
learning environment.
Learning problems become more serious over time because instruction is not adapted to address identified needs or gaps (interplay of individual and environment).
Learning problems are associated with the presence of a disability (individual).
(Adapted from Adelman, 1992)
93
Multitiered System of Support (MTSS) This school improvement framework reflects an
interactional view of student success and failure. It focuses on all aspects of the educational process, and
those involved in this process, to ensure student success:• Effective schools
• Prevention of school failure
• Data-driven, multitiered systems of instruction
• Collaboration across programs, services, personnel
• Professional development
• Continuous evaluation of the effectiveness of system-level and student-level initiatives and supports
94
MTSS Essential Components:Positive School Environments Well-implemented special language programs
• Bilingual education and English as a Second Language Shared knowledge base about ELs and about effective
practices for their education Appropriate screening, assessment, and progress
monitoring systems for the native language (L1) and English (L2), depending on program model• Longitudinal record-keeping systems to track student progress
within and across grades
95
MTSS Essential Components: Response to Intervention (RtI) An essential component of a Multitiered System of
Support, Response to Intervention is a multitier approach to providing high-quality instruction and intervention, matched to student needs (Elliott, 2008).
Progress is closely monitored and changes in instruction are based on data collected from ongoing assessment (National Association of State Directors of Special Education, 2005).
96
Elements of an Effective RtI Model for ELs Universal screening in L1 and/or L2, depending on the
program model
• Measures validated for ELs
• Identification of students at risk of failure
97
Elements of an Effective RtI Modelfor ELs Tiered instruction
• Core instruction (Tier 1)– Universal design principles
– Language of Instruction In the native language and in English in bilingual education
programs In English as a Second Language programs, English instruction
with scaffolding across skill/content areas and settings
– Culturally responsive principles in delivery of instruction
– Continuous progress monitoring
98
Elements of an Effective RtI Modelfor ELs Supplemental instruction (Tier 2 and Tier 3)
• Increasingly intensive interventions for students experiencing learning difficulties
• Research-based interventions validated for ELs
• Intervention consistent with the language of core instruction
• Culturally responsive principles in delivery of instruction
• Provided by educators with expertise specific to ELs
Continuous progress monitoring in L1 and L2
99
Elements of an Effective RtI Modelfor ELs Data-based decision making
• Instructional planning for all students
• Grade, skill/content area, and school-level trends and issues
• Effectiveness of curriculum and instruction; fidelity of implementation
• Identification of students who are not meeting expectations
• Identification of students who may benefit from special education referral
100
Early Intervention for ELsWith Reading Difficulties (Tier 2) Supplement, do not replace, core instruction
Are based on screening, assessment, and progress
monitoring data in L1 and/or L2
Are differentiated to address identified language and
literacy needs Incorporate strategies to build oral language skills in the native language
and/or in English (with emphasis on vocabulary development).
• Provide systematic, explicit literacy instruction in identified areas of need,
consistent with the language of core instruction. (Cavazos & Ortiz,
2014)
101
Problem-solving Teams If specialized interventions do not resolve learning
problems, the student’s case should be presented to a campus-based problem-solving team (PST). PSTs for ELs must include members with expertise specific to this population.
Team members identify factors that contribute to learning difficulties (system- and/or student-level).
102
Indicators That Support Referral Multiple data sources corroborate
student difficulties Parents concur with educators’
concerns Significant life events have been
ruled out as causal factors (e.g., medical history, accidents, family issues)
Native language skills are atypical of peers with similar backgrounds
ESL skills are atypical of peers with similar backgrounds
Cultural characteristics are atypical of peers
No evidence of interrupted education; no excessive absences
Student has been enrolled in current placement long enough to benefit from intervention
General education interventions have not resulted in adequate progress
103
Linguistically and Culturally Responsive Special Education Processes Referral committees include parents and other members
with EL expertise Full and Individual Evaluations using instruments and
procedures that are valid and reliable for ELs Multidisciplinary teams with EL expertise
• Determine eligibility
• Develop IEPs that simultaneously address disability and language-related needs
• Assess results of special education intervention
104
Issues With EL/LD Identification ELs with limited oral language proficiency in both the
native language and in English have the highest special education placement rates ( Artiles, Rueda, Higareda, & Salazar, 2005).
ELs with specific learning disabilities (Ortiz, et al., 2011):• Were typically referred in 2nd or 3rd grade; many had already been retained or
socially promoted.
• Had limited oral language proficiency in the native language and in English at school entry and still had limited skills in both languages at the time of referral.
Yet, oral language development was not targeted in early intervention efforts.
105
Issues With EL/LD Identification Most states allow (or require) identification of specific
learning disabilities based on significant discrepancies between intelligence and achievement.• The instruments used to establish IQ-achievement
discrepancies are of questionable validity for ELs.
• Assessment personnel lack training in assessment of ELs and in interpretation of assessment results.
106
Issues With EL/LD Identification Neither RtI nor special education processes provide
appropriate data to determine the presence of a disability. Data are insufficient provide assurances that problems are not the result of:• Limited English language proficiency
• Cultural differences
• Economic disadvantage
• Lack of appropriate instruction in reading and math
107
Identification of LD As many as 75% of ELs identified as having learning
disabilities may be misclassified (Ortiz et al., 2011):• Some have learning problems that can be explained by factors other
than the presence of a disability (e.g., lack of access to appropriate instruction, lack of timely intervention, inappropriate assessment practices).
• Some have disabilities, but not the one they have been assigned.
In examining special education representation patterns (overrepresentation, underrepresentation, and proportionate representation), it is important to ask:
Are we serving the right students?
108
Indicators of LD for ELs In addition to the factors that supported referral:
• Full and individual evaluation corroborates reasons for referral.
• Results of RtI data corroborate those of formal assessments.
• The student’s performance is significantly different from that of EL peers.
• The student exhibits behaviors typically associated with the suspected disability.
• Parents/family concur that problems manifest in home and community contexts (i.e., they exist 24 hours a day, not just in school).
• There are no competing hypotheses to explain student difficulties.
109
Research Infrastructure Preparation of researchers with expertise in the interaction
of native language and English language proficiency/ development and disability
Routine inclusion of ELs in research on students with disabilities or specific explanation for their inclusion
Application of relevant theories and frameworks in studies involving ELs
110
Research Needs Documentation of special education representation
patterns and factors contributing to disproportionate representation by language group
Criteria to determine eligibility of ELs with disabilities for bilingual education and/or ESL/ELD programs and criteria for exit from these programs
Determining who is a “true” peer
Progress monitoring procedures for oral language development
111
Research Needs Effectiveness of RtI for ELs, including effect on special
education and placement rates Best practices in native language and English
assessments to identify disabilities Guidance for distinguishing differences from disabilities Effective instructional practices for core instruction,
supplemental intervention, and special education instruction• By age, grade, language proficiency, academic achievement,
disability/exceptionality, category, severity level
112
RtI Model Demonstration Projects Model Demonstration Project for English Language
Learners With or At-Risk of Having a Disability (84.326M)
• 3 Funded Projects
− The University of Texas at Austin (2)
− The University of Colorado at Boulder (1)
Sponsor: U.S. Department of Education, Office of Special Education Programs
Funding Period: 2012-2015
113
ReferencesAdelman, H. S. (1992). LD: The next 25 years. Journal of Learning Disabilities,
25, 17-22.
Artiles, A. J., Rueda, R., Salazar, J. J., & Higareda, I. (2005). Within-group diversity in minority disproportionate representation: English Language Learners in urban school districts. Exceptional Children, 71(3), 283-300.
Elliot, J. (2008, September). Response to Intervention: What and Why? School Administrator, 10-18.
Cavazos, L. & Ortiz, A. A. (2014, February). Response to Intervention for English Language Learners with Reading Difficulties. Presented at the meeting of the National Association for Bilingual Education, San Diego, California.
García, S. B., & Ortiz, A. A. (2008). A framework for culturally and linguistically responsive design of Response-to-Intervention models. Multiple Voices for Ethnically Diverse Exceptional Learners, 11(1), 24-41.
114
ReferencesLinan-Thompson, S., & Ortiz, A. A. (2009). Response to intervention and English
Language Learners: Instructional and assessment considerations. Seminars in Speech and Language, 30(2), 105-120.
National Association of State Directors of Special Education. (2005). Response to Intervention: Policy considerations and implementation. Alexandria, VA: Author.
Ortiz, A. A., Robertson, P. M., Wilkinson, C. Y., Liu, Y., McGhee, B. D., & Kushner, M. I. (2011). The role of bilingual education teachers in preventing inappropriate referrals of ELLs to special education: Implications for Response to Intervention. Bilingual Research Journal, 34(3), 316-333.
115
Alba A. Ortiz, Professor EmeritusDepartment of Special EducationThe University of Texas at [email protected]
116
Moderated Discussion
117
Questions
118
Break11:05 – 11:15 am
119
Assessing the English Language Proficiency of ELs With Disabilities:Implications From Research and PracticeMartha L. Thurlow
National Center on Educational Outcomes (NCEO)
March 16, 2015
120
Relevant Evidence Analyses of participation of ELs with disabilities in states’
assessments of English language proficiency
• Policies
• Public Reporting (including participation and performance)
Analyses of accessibility approaches and accommodations in English language proficiency assessments
Focus group study on perspectives from the field on challenges and successes in including ELs with disabilities in ELP assessments
121
Framework
High English language needs
Low disability-related needs
High English language needs
High disability-related needs
Low English language needs
Low disability-related needs
Low English language needs
High disability-related needs
122
State Assessment Policies: Participation Criteria for ELs With Disabilities 49 of 50 states provide criteria for the participation of ELs
with disabilities in their ELP assessments
123
Reporting on ELs With Disabilities ELP Participation and Performance ELs With Disabilities Data in Public Reports for
2012-13 ELP Assessments • California • Colorado• Louisiana • Minnesota • New York • Texas
None of these states reported a participation rate, although 2 reported percent not tested
All of these states reported percent proficient (and all but 1 reported by performance level)
124
State Policies: Reporting States rarely address what happens to the students’ scores
when the students are not able to participate in all domains of the ELP assessment.
When states do indicate what happens to scores, it is generally to indicate either that a student’s non-participation in some domains did not count against school participation rates or that, if an alternative means of assessing the student was used, the score was considered invalid.
125
Performance Reported Publicly 6 states reported data publicly, but 1 only reported on its
Alternate Assessment of ELP
126
GRADE
CAELSWD
CAELs
LAELSWD
LAELs
MNELSWD
MNELs
NYELSWD
NYELs
TXELSWD
TXELs
Grade 4
10%
40%
14%
35%
8%
31%
10%
21%
36%
78%
Grade 8
19%
56%
39% +
**
48%
1%
8%
11%
16%
69%
86%
Grade
10
18%
48%
13% +
**
41%
16%
30%
15%
16%
65%
82%
Implications Better national and state data are needed on ELs with
disabilities, including information on their disabilities, language background, assessment participation, and proficiency.
No student should be denied a score of language proficiency because of his or her disability.
127
State Policies: Accommodations Decision-making criteria differ across states, with the most
frequent criteria (cited by more than ½ of states) being:• IEP team decision – 46 states• Maintains validity – 41 states• Used in class – 39 states• Meets individual student needs – 39 states• Appropriate for domain – 28 states• Whether not recommended because of severity of disability – 27
states Some criteria, mentioned by fewer than ½ of states, refer
to specific disabilities:• Student is proficient in braille or lip reading – 22 states
128
State Policies: AccommodationsReading Writing Listening Speaking
Large Print (46)Proctor/Scribe (42)Braille (40)Magnification (36)Amplification (36)Directions-Repeat,
Re-read, Clarify (36)
Sign Interpret Directions (36)
Student Read Aloud (30)
Large Print (46)Braille (40)Proctor/Scribe (38)Magnification (38)Computer/Machine
(38)Amplification (35)Directions-Repeat,
Re-read, Clarify (36)
Student Read Aloud (29)
Large Print (44)Braille (38)Magnification (38)Sign Interpret Directions (38)
Directions – Repeat, Re-read, Clarify (36)
Amplification (35)
Large Print (42)Braille (38)Magnification (38)Sign Interpret
Directions (37)Directions-Repeat,
Re-read, Clarify (36)
Amplification (35)
129
More on Approaches for Deaf/HH Selective participation for ELP assessment allowed for
students who are deaf/hard of hearing in 26 states Least controversial accommodations – sign interpret
directions and amplification equipment Sign interpret questions prohibited across domains in
most states (30-32 states) Sign response prohibited across domains in most states
(26-27 states) Few state policies addressed visual cues (allowed in 4;
prohibited in 1)
130
More on Approaches for Blind/VI Selective participation for ELP assessment allowed for
students who are blind/visually impaired in 24 states Least controversial accommodations – braille, large print,
magnification equipment Few state policies addressed read aloud directions; more
likely to allow for writing than for listening and speaking Read aloud questions allowed in 30 states for writing
domain; rarely allowed for other domains Brailler prohibited for writing domain in 25 states
131
Implications Accessibility and accommodations policies should be based
on determinations about the construct being tested. More research and discussion may be needed on:• What does listening mean for a student who is deaf/hard of hearing?
• What does reading mean for a student who is blind/visually impaired?
• What does speaking mean for a student who has been deaf from birth or who has a speech impediment?
• What does writing mean for a student who has a significant motor disability?
Based on decisions about constructs, plans need to be made for obtaining a total score for all students
132
Perspectives From the Field: Challenges and Successes Participants stated that the IEP process served primarily to
make decisions about accommodations on content assessments, but less so for state ELP assessments and accommodations.
Participants described needs specific to ELs with disabilities for support and guidance from school and state education leaders on assessment and accommodations. The needs were for additional qualified staff and training, clear and consistent written assessment policies, and appropriate uses of state accountability test scores.
133
Implications More training is needed on appropriate decision making for
participation and accommodations for ELs with disabilities.
The IEP team must include professionals who know English language development.
The IEP team should make decisions about both language development and content development.
134
Information ResourcesChristensen, L. L., Albus, D. A., Liu, K. K., Thurlow, M., & Kincaid, A.
(2013). Accommodations for students with disabilities on state English language proficiency assessments: A review of 2011 state policies. Minneapolis, MN: U of MN, NCEO.
Christensen, L. L., Albus, D. A., Kincaid, A., Liu, K. K., & Thurlow, M. L. (2014). Including students who are deaf or hard-of-hearing in English language proficiency assessments: A review of state policies. Minneapolis, MN: U of MN, NCEO
Christensen, L. L., Albus, D. A., Kincaid, A., Christian, E., Liu, K. K., & Thurlow, M. L. (2014). Including students who are are blind or visually impaired in English language proficiency assessments: A review of state policies. Minneapolis, MN: U of MN, NCEO
135
Information ResourcesGuzman-Orth, D., Laitusis, C., Thurlow, M., & Christensen, L. (2014).
Conceptualizing accessibility for English learners and English learners with disabilities taking English proficiency assessments: What do we know and where do we go from here? Princeton, NJ: ETS.
Liu, K., Goldstone, L., Thurlow, M., Ward, J., Hatten, J., & Christensen, L. Voices from the field: Making state assessment decisions for English language learners with disabilities. Minneapolis, MN: U of MN, NCEO
NCEO. (May 2014). Participation of ELLs with disabilities in ELP assessments (NCEO Brief #8). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.
136
Training and Other ResourcesImproving the Validity of Assessment Results for English Language
Learners with Disabilities (IVARED) – Training Module: http://www.ivared.info/training.html.
Christensen, L., Shyyan, V., Rogers, C., & Kincaid, A. (2014). Audio support guidelines for accessible assessments: Insights from cognitive labs. Minneapolis, MN: U of MN, NCEO, GAAP Project.
Shyyan, V., Christensen, L., Rogers, C., & Kincaid, A. (2014). Sign support guidelines for accessible assessments: Insights from cognitive labs. Minneapolis, MN: U of MN, NCEO, GAAP Project.
137
Training and Other ResourcesShyyan, V., Christensen, L., Touchette, B., Lightborne, L., Gholson, M., &
Burton, K. (2013). Accommodations manual: How to select, administer, and evaluate use of accommodations for instruction and assessment of English language learners with disabilities. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.
138
ELP Assessment Accommodations for ELs With Disabilities:Relevance, Effectiveness, Feasibility, and ValidityJamal Abedi
University of California, Davis
March 16, 2015
140
Why Is It Important to Pay Special Attention to Assessment of ELP for ELs With Disabilities? They are faced with the highest level of challenges in their
academic career (the most at risk) They are often at the lowest level of ELP In content-area assessments, they are about 1 to 1 ½
standard deviations below mainstream students They may have the content knowledge but not the
language capabilities to express it The rate of misclassification of these students is extremely
high
141
What Types of Accommodations Are Used in the Assessment of ELs With Disabilities? ELs with disabilities may receive two different types of
accommodations:
1. Accommodations due to their EL status
2. Accommodations due to their disabilities (based on their IEP or 504 plans)
States should provide evidence on the effectiveness and validity of these accommodations
142
Can accommodations that are used in content-area assessments for ELs be used in ELP assessments for ELs with disabilities? The most useful accommodations for ELs in content-area
assessments where the focal construct is not language are language-based accommodations
Examples:
• English and bilingual dictionary
• English and bilingual glossary
• Native language assessments
However, the focal construct in ELP assessments is language; therefore, language-based accommodations may not be valid for ELP assessments
143
Issues/Questions To Be Addressed How can we make sure that accommodations and
accessibility features used in ELP assessments are effective in making ELP assessments more accessible for ELs with disabilities (effectiveness)?
How can we make sure that accommodations used for ELs with disabilities do not alter the focal construct?
How can we judge the feasibility of accommodations used for ELs with disabilities?
144
How Can Accommodations for ELs With Disabilities Be Examined for Validity*? Only through experimentally-controlled research where:
• ELs with disabilities and non-ELs without disabilities are randomly assigned to the accommodated conditions
• Both ELs with disabilities and non-EL students are observed under accommodated and non-accommodated assessments
An accommodation is valid if it does not impact performance of non-ELs without disabilities.
• If it does impact the performance of non-ELs without disabilities, then the accommodation provides an unfair advantage to the recipients
* Not altering the focal construct
145
How Can Accommodations for ELs With Disabilities Be Examined for Effectiveness? Only through experimentally-controlled research where:
• ELs with disabilities are randomly assigned to either experimental or control groups
• The experimental group receives an accommodation
• The control group is tested under the standard condition with no accommodation provided
A significant improvement in the performance of the experimental group indicates effectiveness
146
Are Accommodated and Non-Accommodated Assessments for ELs With Disabilities Comparable? The way to establish comparability between
accommodated and non-accommodated assessments for ELs with disabilities is to make sure that:
1. The accommodations used are valid (i.e., do not alter the focal construct (ELP))
2. The accommodations used are effective in making assessments more accessible for these students by controlling the construct-irrelevant sources.
147
Sample Accommodations Used for ELs With DisabilitiesAccommodation Examples
Presentation Repeat directions, read aloud, large print, braille
Equipment Calculator, amplification equipment, manipulatives
Response Mark answers in book, scribe records response
Setting Study carrel, student's home, separate room
Timing/Scheduling Extended time, frequent breaks
148
Issues/Questions To Be Addressed Can accommodations that are used in content-area
assessments for ELs be used in ELP assessments?
Can we assume accommodations that help reducing sources of construct-irrelevant variance in content-area assessments do the same in ELP assessments (validity)?
How can we make sure that accommodations used in ELP assessments do not alter the focal construct (validity)?
149
Sample Accommodations Used for ELs With DisabilitiesAccommodation Examples
Presentation Repeat directions, read aloud, large print, braille
Equipment Calculator, amplification equipment, manipulatives
Response Mark answers in book, scribe records response
Setting Study carrel, student's home, separate room
Timing/Scheduling Extended time, frequent breaks
However, not enough evidence on the effectiveness or validity of these accommodations for ELs with disabilities
Source: NCEO (http://www.cehd.umn.edu/NCEO/TOPICAREAS/ Accommodations/Accomtopic.htm
150
How Are We Doing in Practice Nationally? Are states and districts across the nation cognizant of
these important criteria for selecting and using accommodations that are effective and valid for ELs with disabilities?
Are there any objective national criteria to help states to select appropriate accommodations for EL students?
Or, is the assignment of accommodations to these students based on temporary and subjective decisions?
151
An Evidence-based System for Determining Appropriate Accommodations for ELs With Disabilities (Abedi & Ewers, 2013) Use: Supported by existing research as being effective in
making assessments more accessible and valid (i.e., does not alter the focal construct) for ELs with disabilities or supports the concept of differential boost
Use/Low Evidence: No clear evidence that the validity assumption is violated; however, additional support would strengthen their acceptance in the field
Do Not Use: There is enough consistent evidence suggesting an accommodation is not effective and alters the focal construct.
152
An Evidence-based System for Determining Appropriate Accommodations for ELs With Disabilities Unsure/Low Evidence Needed: Existing research-based
evidence is supportive of the accommodation but not sufficient to make a judgment about its effectiveness and validity
Unsure/Moderate Evidence: Existing research-based evidence is not quite sufficient to make a judgment about effectiveness and validity; some additional research-based evidence is needed
Unsure/High Evidence: Existing research-based evidence neither supports nor rejects the effectiveness and validity of the accommodation, substantial research-based evidence is needed
153
Overall Decision on Accommodation Use If valid/any level of effectiveness, then use
If valid or low evidence needed/any level of effectiveness, then use with minor risk
If validity is unsure with low evidence needed/any level of effectiveness, then use with minor risk
If validity is unsure with moderate evidence needed/any level of effectiveness, then use with moderate risk
If validity is unsure with high evidence needed/any level of effectiveness, then use with high risk
154
An Example of the Decision Process: Overall Decision – Use
EL Accommodation Extra time within the testing day
Research Findings This study indicated that extra time is both effective and valid for students in Grade 4 (Abedi et al., 2003b). Both EL and non-EL students in Grade 8 are helped by this accommodation on a mathematics assessment of 35 released NAEP items (Abedi, Hofstetter, Baker, & Lord, 2001a). Highly rated by a team of experts as helpful for the lowest English language proficient students. (Acosta et al., 2008).
Recommendation/Validity
Use
Recommendation/Effectiveness
Unsure/Moderate Evidence
Overall Decision Use - Access
155
Conclusions and RecommendationsAccommodations: Must be relevant in addressing assessment issues for ELs with
disabilities Must be effective in reducing the performance gap between
ELs with disabilities and those who are not ELs with disabilities Should not alter the construct being measured; under such
condition, the accommodated results can be aggregated with the assessments under standard conditions
Must be feasible in national and state assessments
156
ReferencesAbedi, J. (2013). Testing of ELL Students (Chapter 101). In K. F. Geisinger, APA handbook
of testing and assessment in psychology. Washington, DC: American Psychological Association.
Abedi, J. (2012). Validity issues in designing accommodations. In G. Fulcher & F. Davidson, The Routledge handbook of language testing. London, UK: Routledge, Taylor & Francis.
Abedi, J. (2007). English language learners with disabilities. In C. Cahlan-Laitusis & L. Cook (Eds.), Accommodating student with disabilities on state assessments: What works? Arlington, VA: Council for Exceptional Children.
Abedi, J. (2014). The use of computer technology in designing appropriate test accommodations for English language learners. Applied Measurement in Education.
Abedi, J., & Ewers, N. (2013). Accommodations for English language learners and students with disabilities: A research-based decision algorithm. Smarter Balanced Assessment Consortium. Retrieved from http://www.smarterbalanced.org/wordpress/wp-content/uploads/2012/08/Accomodations-for-under-represented-students.pdf
157
Accessibility and Accommodations on ELPA21Phoebe Winter
Independent Consultant
March 16, 2015
159
ELPA21 Developed by a multi-state consortium, USED EAG
Based on the English Language Proficiency (ELP) Standards (CCSSO, 2013)
• Correspondence to content area standards
Computer-based assessment
Use of technology to improve measurement
160
Designed for Accessibility Coordination among professionals with expertise in
• English language acquisition
• Students with disabilities
• Measurement Evidence centered design Test design Item development Research Test delivery
161
Developing Accessible Items Minimizing language-related barriers due to non-targeted
domains
Minimizing barriers to measuring all students’ proficiency/performance
• Universal design
• Designing with accessibility features and accommodation in mind
162
Building in Accessibility – Strategies Use of pictures and graphics
Multiple modes of presentation
Non-verbal response modes
APIP compatibility
Alt text
Reviews during and after item development
163
Sample Compare Pictures Set (Grade 4-5 Speaking)Written directions and narrator audio: Look carefully at the two pictures. Describe what is the same and what is different. Use as many details as you can. Include at least three things that are the same or different.
164
Sample Follow Instructions Set (G1 Listening)Written directions and narrator audio: Look at this classroom. Listen to the teacher. Follow the teacher’s directions.
(Teacher audio): Put the ruler on the bookshelf.
(Teacher audio): Choose the book about animals. Put it on the table by the window.
165
Sample Listen and Match Word (G1 Listening)Written directions and narrator audio: Listen to the word. Choose the picture that matches the word.
(Narrator audio): balloon
166
Conceptual Model
167
Blind and Visually Impaired Students Assess the same standards as the general ELPA21
Provide scores that are comparable in terms of acquisition of academic English
As parallel in structure as possible to the general ELPA21
• Minimize dependence on language skills other than those being assessed
• Be engaging and motivating
• Use the computer as much as possible, while being amenable to paper-based administration
168
Items Item “twins” developed Goal is to have 1.5 X items needed for each form
• Item review for accessibility
• Gaps filled by twins at the task type level
• Twins designed to measure same standards
Braille for reading items above K Braille and audio for other domains Manipulatives (realia) for response options and as stimuli Cognitive labs
169
Sample Item Twins Compare pictures
Follow instructions
Listen and match
Read and match
Word builder
Storyboard
170
Sample Compare Pictures Set General Assessment (Grade 4-5 Speaking)Written directions and narrator audio: Look carefully at the two pictures. Describe what is the same and what is different. Use as many details as you can. Include at least three things that are the same or different.
171
Sample Compare Pictures Twin (Grade 4-5 Speaking)Narrator audio and braille stimulus I’d like to hear about how playing at school and playing at
home are the same and different. Describe some things that are the same about playing at
school and playing at the home. Please give as many details as you can.
Now describe some things that are different about playing at school and playing at the home. Please give as many details as you can.
172
Sample Follow Instructions Set General Assessment (G1 Listening)Written directions and narrator audio: Look at this classroom. Listen to the teacher. Follow the teacher’s directions.
(Teacher): Put the ruler on the bookshelf.
(Teacher audio): Choose the book about animals. Put it on the table by the window.
173
Sample Listen and Match Word General Assessment (G1 Listening)Written directions and narrator audio: Listen to the word. Choose the picture that matches the word.
(Narrator audio): balloon
174
Sample Follow Instructions Twin (Grade 4-5 Listening)Script: Listen to the instructions. Follow my directions.
Script: Put the plant in the box.
Manipulatives: A small plant and a book; box, paper bag, backpack, vase
Sample Listen and Match Twin (Grade 1 Listening)Script: Listen to the word. Choose the object that matches the word.
Script: Balloon
Manipulatives: Ball, basket, balloon
175
Sample Read and Match Item General Assessment (G1 Reading)Written directions [and narrator at lower grades]: Look at the picture. Choose the word that matches the picture.
dotdogbug
176
Sample Read and Match Twin (Grade 4-5 Reading)Script: Touch the object. Read the words. Choose the word that matches the object.
Braille book: pail paper pan parent
Manipulative: paper
177
Sample World Builder Item General Assessment (G1 Writing)Written directions and narrator audio: Move the letter to complete the word.
(Narrator audio): bear
b p w
_ear
178
Sample World Builder Twin(G1 Writing)Script: Listen to the word. Complete the word.
Script: bear
Materials: Braille cards or tiles
_ear
b p w
179
Sample Storyboard General Assessment (Grades 6-8 Writing)Written directions and narrator audio: The four pictures below show a story about something that happened. What story do the pictures tell? Look at the pictures and prepare to write a paragraph. You may use the words in the Word Bank to help you. Type your story in the text box and click submit when you are finished.
Be sure to check your work. Make sure you… tell a complete story from
beginning to end organize ideas in a logical
way use the right style for the task
and the audience
Word Bankemptylitter
gardencleanplants
180
Sample Storyboard Twin (G4-5 Writing)Narrator audio and braille stimulus: Write a story about a time when you did something fun with your friends. You may use the words in the Word Bank to help you. Be sure to check your work. Make sure you… tell a complete story from beginning to end organize ideas in a logical way use clear language and complete sentences
Word Bank: friends fun exciting play
181
Considerations for Accessibility Understanding clearly the constructs being assessed Incorporating an understanding of the students in test
design and item development Providing general access to a suite of tools Carefully selecting/designing and considering the impact of
other accessibility features – designated features and accommodations
Incorporating empirical research into development
182
Resources English Language Proficiency Assessment for the 21st Century,
ELPA21, is a consortium of states committed to supporting educators, member states, and members of the public as they adopt and implement the English language proficiency standards and and college- and career-ready standards.
The consortium is developing an assessment system based on the ELP Standards
For more information see: http://www.elpa21.org/
183
Phoebe [email protected] more information about ELPA21 in general: http://www.elpa21.orgFor information specific to accessibility and accommodations: http://www.elpa21.org/assessment-system/features-benefits
184
WIDA Consortium, ACCESS 2.0Accommodations for ELs with disabilities H Gary Cook, Ph.D. Associate Research Scientist
Research 185
Overview The WIDA Consortium & ASSETS Grant
ACCESS 2.0 Accessibility Features and Accommodations
Creating Alternate Composites Scores for ELs with Disabilities
Research 186
ASSETS Grant Members (WIDA 2.0)
Research 187
ASSETS GrantNext Generation ELD Standards & Assessments to support Carrier & College Ready Language Expectations
Research 188
ACCESS 2.0 Features ACCESS 2.0 will be provided both in a paper & pencil &
online format
Speaking test computer delivered and scored remotely (online)
Writing above grade 3 is provided online
Research 189
Categorization of Accommodations is Different: ACCESS for ELLs & ACCESS 2.0
What’s Different Reframed the 40+ Access
for ELLs accommodations into 3 categories: • More streamlined approach
to accommodation
• Added additional supports within online test
• Expanded accessibility for all ELLs
Research 190
Introduction of New “Middle” Category: Accessibility Features Include tools and supports
that are available to all ELLs taking the tests, based on need or preference.
May either be embedded in the computer-based test or provided to ELLs by test administrators on online or paper-based tests.
Research 191
UDL Principles Applied Throughout Test Items: Balance Accessibility and Usability Increased multimodality
• Add supporting prompts with appropriate animations and graphics
Uncomplicated and predictable as possible, low extraneous processing demands
• The test will look uncluttered and function intuitively for the test taker. Sample Reading Item Online Layout
Research 192
Creating Alternate Composite Scores Problem
• Some EL/SWD do not get composite scores (e.g., deaf or blind students)
• Lack of composite scores effects AMAOs Solutions
• Create conjunctive expectations based on administered tests
• Create alternate composite scores
Research 193
A Proficient Performance Criterion
U.S. Dept. of Education (2012). Prepared by Cook, Linquanti, Chinen, Jung.
Conversationsabout proficiencycan start here.
Research 194
Conjunctive Expectations
List_P Speak_P Read_P Writ_P
Research 195
Alternate CompositesAlternate Composite Comparison - Math & Reading Grade 3
PLOT LitAltCompMath OralAltCompMathLitAltCompRead OralAltCompRead
Research 196
Creating Alternate Composite Scores It is possible to create composite score analogs which can
be used to exit ELs who are SWD who cannot participate on all domain tests.
These methods show a predictive relationships to content assessments.
Research 197
Moderated Discussion
199
Questions
200
Lunch1 – 2 pm
201
Alternate Assessments of ELP for ELLs With Significant Cognitive Disabilities:Considerations From the FieldMartha Thurlow
National Center on Educational Outcomes (NCEO)
March 16, 2015
202
Overview What Have We Learned From Alternate Assessment
Consortia?
Who Are ELs With Significant Cognitive Disabilities?
What Else Do We Know From the Field?
What Might an Alternate Assessment of ELP Look Like?
203
Delphi Study Results“Schools must address the language development needs of ELs with significant cognitive disabilities. This is a fundamental civil rights issues….they must have high-quality information about these students’ language skills and needs.”
204
DLM and NCSC Findings Most students in the AA-AAS use oral speech, read sight
words, and do math with a calculator, and can learn more than “rote academic skills.” Still there is a small group who do not.
Communication systems are essential for access to the general curriculum, including “language” development.
It is important to develop a description of the student population and a theory of learning for these students – these should drive the nature of the assessment.
205
DLM and NCSC Findings Many students with significant cognitive disabilities have
not had access to the curriculum or to language development efforts.
It is important to take a three-pronged approach for these students – curriculum, instruction, and assessment.
206
Misperceptions About StudentsCommon Misperceptions1 about who the students are who participate in the AA-AAS include:
They function more like infants or toddlers than their actual age.
They have life-threatening medical conditions or are not able to communicate.
They can learn only rote academic skills.
1 From Quenemoen, Kearns, Quenemoen, Flowers, & Kleinert (2010), NCEO Synthesis Report 73.
207
Primary Disability Categories of Students in the AA-AAS
Alternate Assessment Participants
MR
MD
Autism
From: National Alternate Assessment Center (NAAC)
208
Research on Characteristics of Students With Significant Cognitive Disabilities Most students (about 72%) in AA-AAS had expressive
communication (used verbal or written words, signs, braille, or language-based augmentative/alternative communication system) – symbolic language users
About 18% of AA-AAS students were emerging symbolic language users
Another 10% of AA-AAS students were pre-symbolic language users
1 From Kearns, Towles-Reeves, Kleinert, Kleinert, & Thomas. (2011). The Journal of Special Education.
209
Meet ShelleyShelley also has multiple disabilities and is using a switch to indicate that she wants “to go more.” It took about 8 trials in one session to get this consistent response.
210
Meet JordanJordan, who has autism, participates in the AA-AAS. He uses symbolic language and reads sign words. In this clip, he is learning about idiomatic expressions.
211
Meet BruceBruce is a student who was identified as having significant cognitive disabilities with no oral communication. After a communication system was identified for him, educators no longer believed he should be taking an AA-AAS.
212
Meet Axel and ThasyaThese two students are highlighted on the NCSC website. They are both ELLs with significant cognitive disabilities.
The videos show how complex the identification of these students is and how determining the communication that works for each student is critical.
Find the videos under Multimedia athttp://www.ncscpartners.org
213
Characteristics of ELLs With Significant Cognitive DisabilitiesCategory of Disability Category of Disability Count Percent
Mental retardation 3559 59.2%Autism 1038 17.3%Multiple disabilities 599 10.0%Other health impairment 154 2.3%Speech/language impairment 91 1.5%Hearing impairment 91 1.5%Orthopedic 62 1.0%Emotional disability 45 0.7%Traumatic brain injury 36 0.6%Deaf-Blind 26 0.4%Visual impairment 16 0.3%Other 246 4.1%No data 45 0.7%
214
Characteristics of ELLs With Significant Cognitive Disabilities
Expressive Communication
Expressive Communication Count Percent
Uses symbolic language 3675 61.2%
Uses intentional communication 1388 23.1%
Communicates primarily through cries, etc. 850 14.1%
No data 95 1.6%
215
Participation of ELLs With Significant Cognitive Disabilities in ELP Assessments
216
Instruction of ELLs With Significant Cognitive Disabilities Language of instruction – almost exclusively English;
special education teachers for the most part do not have backgrounds in language development; some have paraprofessionals who know other languages to provide needed scaffolding
Successful approaches – picture and photo cues, realia, identify objects and pictures in both English and student’s home language
More curricula and instructional strategies are needed
217
What Might an Alternate ELP Look Like? Some Hints from AA-AAS HistoryEarly AA-AAS:
Primarily portfolios, with teachers collecting evidence
Current AA-AAS:
Technology-based assessments, with allowances for teacher administration
Item-based assessments
Assessments based on grade-level achievement standards
218
Intercultural Considerations Students and their families are characterized by
diverse communication styles Non-verbal communication misunderstandings are
more likely to occur in intercultural settings Disability is perceived differently across cultures Students’ cultures of origin and prior intercultural
experiences have implications for curriculum, instruction, and assessment decisions
Educators need intercultural competence skills
219
Concluding ThoughtsAn alternate assessment of ELP needs to be: Based on strong college and career ready standards of
ELP Carefully planned in terms of how it addresses the
assessment of reading, writing, speaking, and listening domains, so that those with disabilities in one or more areas can still earn a score and demonstrate proficiency
Designed so that ELs with significant cognitive disabilities can demonstrate proficiency in English, with solid exit criteria based on alternate performance criteria.
220
ResourcesKearns, J., Towles-Reeves, E., Kleinert, H., Kleinert, J., & Thomas, M. (2011).
Characteristics of and implications for students participating in alternate assessments based on alternate academic achievement standards. Journal of Special Education, 45(1), 3-14. doi:10.1177/0022466909344223
Liu, K. K., Goldstone, L. S., Thurlow, M. L., Ward, J. M., Hatten, J., & Christensen, L. L. (2013). Voices from the field: Making state assessment decisions for English language learners with disabilities. Minneapolis, MN: U of MN, IVARED.
NCEO. (2014). Exploring alternate ELP assessments for ELLs with significant cognitive disabilities (NCEO Brief 10). Minneapolis, MN: U of MN, NCEO.
221
ResourcesQuenemoen, R., Kearns, J., Quenemoen, M., Flowers, C., & Kleinert, H.
(2010). Common misperceptions and research-based recommendations for alternate assessment based on alternate achievement standards (Synthesis Report 73). Minneapolis, MN: U of MN, NCEO.
Quenemoen, R. (2008). A brief history of alternate assessments based on alternate achievement standards (Synthesis Report 68). Minneapolis, MN: U of MN, NCEO.
Thurlow, M. L., Liu, K. K., Ward, J. M., & Christensen, L. L. (2013). Assessment principles and guidelines for ELLs with disabilities. Minneapolis, MN: U of MN, IVARED.
222
What we’re learning about Alt ACCESSH Gary Cook, Ph.D. Associate Research Scientist
Research 224
Overview About Alt ACCESS
What we’re learning about attainment
What we’re learning about growth
Research 225
What Is Alt ACCESS? Alternate ACCESS for ELLs (Alt ACCESS)
• Assessment based on WIDA’s Alternate Model Performance Indicators (AMPIs) www.wida.us/assessment/alternateaccess.aspx
• Meant to assess ELs with significant cognitive disabilities
Adaption of an assessment created from a 2008-2011 USDE Enhanced Assessment Grant
Research 226
What is Alt ACCESS? Locally administered assessment Four grade clusters: 1-2, 3-5, 6-8, 9-12 Four domains: listening, speaking reading, writing Measures social
instructional language, and the language of ELA, mathematics, and science
Measures five proficiency levels
Research 227
Who participates in Alt ACCESS?
Eligible to participate on Alt ACCESS
Research 228
Attainment on Alt ACCESS What does it mean to be proficient on Alt ACCESS? What
does language proficiency look like for ELs with significant cognitive disabilities?
One idea…
• Conceptually, apply methods used to identify English language proficiency on general ELD assessments
Research 229
A Proficient Performance Criterion
US Dept. of Education (2012). Prepared by Cook, Linquanti, Chinen, Jung.
Conversationsabout proficiencycan start here.
Research 230
Alt ACCESS to State Alternate Assessment Proficiency (in 4 states)
Logistic Probability Curves
Domain A1/A2 A2/A3 A3/P1 P1/P2
Overall Composite 924 931 938 944
Research 231
What Do We See About Attainment? This method seems to provide information consistent with
a similar method used for ELD and content assessments.
This method defines the English proficient performance criterion with reference to the state’s alternate assessment.
Research 232
Overall Composite Score Level Change on Alt ACCESS
262 154 279 180 96 Total = 971
Research 233
Overall Composite Score Level Change on Alt ACCESS
337 227 448 550 292 Total = 1854
Research 234
Overall Composite Score Level Change on Alt ACCESS
198 113 283 394 383 Total = 1371
Research 235
Overall Composite Score Level Change on Alt ACCESS
160 119 322 338 319 Total = 1258
Research 236
What Do We See About Growth? Proportionally, most students did not move levels (except
for level A2)
Largest level gain occurs at A2 level across all grade clusters
Students in level A1 grew the least across all grade clusters (discounting level P2)
Scale Score growth (not shown) very small
Research 237
Some Questions Given the large number of student not progressing in
proficiency on Alt ACCESS….
• How long would it take to be proficient for these students?
• What should our growth expectations be vis-à-vis AMAO 1 for these students?
• Is there a point where a student’s lack of growth suggests that Title III services are no longer helpful?
Research 238
Moderated Discussion
240
Questions
241
