center for applied linguistics standards, structure, & field testing of access for ells jessica...
TRANSCRIPT
Center for Applied Linguistics
Standards, Structure, & Field Testing of ACCESS for ELLs™Jessica Motz
Center for Applied Linguistics
Washington, DC
January 2005
Illinois 28th Annual Statewide Conference for Teachers Serving Linguistically and Culturally Diverse Students
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 2
Outline
1. Background on the WIDA Project
2. WIDA Standards for English Language Proficiency – language vs. content
3. Structure of the ACCESS for ELLs™ Test
4. Sample Items
5. Pilot Test Results
6. Field Testing Update
7. Training for ACCESS for ELLs™
8. Other Issues/Questions/Discussion
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 3
Origins of the WIDA Consortium
Ten states, representing approx. 275,000 ELLs
2002: U.S. Dept. of Education Enhancement Grant Competition
3 Original States:
Wisconsin
Delaware
Arkansas
Early Additions:
District of Columbia
Rhode Island, Maine, New Hampshire, Vermont
Later Additions:
Illinois, Alabama
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 4
Multiple WIDA Components
Language Proficiency• English Language Proficiency Standards
• Large-Scale Assessment (ACCESS for ELLs)• Classroom Instruction & Assessment
Academic Content• Alternate Assessment (Alternate ACCESS)
Professional Development• Standards• Assessment• Instruction
Validation and Research
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 5
WIDA PartnersState Leadership
Tim Boals, Wisconsin DPI, Chair Steering Committee
Lead Developer (Standards and Project PI)
Margo Gottlieb, Illinois Resource Center
Item Specification Development
Fred Davidson, UIUC
Test Development
Language Testing Division, CAL
Professional Development
Lorraine Valdez-Pierce, George Mason University
Technical Applications (Database, Desire2Learn)
University of Wisconsin - Oshkosh
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 6
Centrality of the ELP Standards
ClassroomAssessmentFramework
Large-ScaleAssessmentFramework
English LanguageProficiencyStandards
& PerformanceDefinitions
ModelPerformance
Indicators:Classroom
ModelPerformance
Indicators:Large-Scale
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 7
Overall Organization of Standards
Frameworks for Classroom & Large-Scale Assessment (2)
English Language Proficiency Standards (5)
Language Domains (4)
Grade Level Clusters (4)
Language Proficiency Levels (5)
Model PIs are the lowest level ofexpression of the standards
Model Performance Indicators
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 8
The WIDA ELP StandardsStandard 1—SI
• English language learners communicate in English for social and instructional purposes in the school setting.
Standard 2— LA• English language learners communicate information, ideas
and concepts necessary for academic success in the content area of Language Arts.
Standard 3—MA• English language learners communicate information, ideas
and concepts necessary for academic success in the content area of Math.
Standard 4—SC• English language learners communicate information, ideas
and concepts necessary for academic success in the content area of Science.
Standard 5— SS• English language learners communicate information, ideas
and concepts necessary for academic success in the content area of Social Studies.
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 9
The Levels of English Language Proficiency
ENTERING
BEGINNING
DEVELOPING
EXPANDING
Never LEP
1
2
3
4
5
6
7
Formerly LEP
BRIDGING
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 10
Criteria for Proficiency Level Definitions
Comprehension and use of the technical language of the content areas
Extent of language (text or discourse) control
Development of phonological, syntactic, and semantic understanding or usage
ENTERING BEGINNING DEVELOPING EXPANDING BRIDGING
1 2 3 4 5
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 11
Large-Scale Standards: SC Reading
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 12
Large-Scale Standards: SC Reading
Classify living organisms (such as birds and mammals) by using pictures or
icons
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 13
Large-Scale Standards: SC Reading
Interpret data presented in text and
tables in scientific studies
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 14
Large-Scale Standards: SC Reading
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 15
Assessment Forms
Non-secure form for initial screening (July 1)• One for each grade level cluster - with items at all 5
proficiency levels
• Kindergarten form - individually administered
Secure forms for annual testing • Two (initially: 100 and 200) for each grade level
cluster• Tier A: Proficiency levels 1-3• Tier B: Proficiency levels 2-4• Tier C: Proficiency levels 3-5
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 16
Tier Alignment with Proficiency Levels
ENTERING BEGINNING DEVELOPINGEXPANDING BRIDGING
1 2 3 4 5
Annual ACCESS for ELLs
Tier A
Tier B
Tier C
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 17
WIDA Steering Committee Administration Decisions
Listening (15%): 20-25 minutes, machine scored
Reading (35%): 35-40 minutes, machine scored
Writing (35%): Up to 1 hour, rater scored
Speaking (15%): Up to 15 minutes, administrator scored
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 18
Structure of ACCESS for ELLs
Grade Leveland Tier
K
1-2
3-5
6-8
9-12
A (adaptive)
A B C
A B C
A B C
A B C
100 (roll-out Spring 2005)
999 (used to produce screener)
200 (roll-out Spring 2006)
Listening — group admin, machine scored
Reading — group admin, machine scored
Speaking — individual admin, adaptive, TA scored
Writing — group admin, rater scored
Domains
Forms
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 19
Item Creation Process (Fall 2003 – present)
Item specifications drafted
Item writers assembled from nominated ESL teachers in consortium states
Item writers trained using Blackboard distance-learning management software
Item writers submitted items electronically
Items reviewed by External Reviewers, also trained via Blackboard
Items reviewed and revised internally & organized into themes…
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 20
Thematic Folders of items arranged onto pilot test forms
Forms piloted in 5 WIDA districts
Pilot analysis & feedback incorporated; items created, revised by ESL teachers
Thematic Folders of items arranged onto field test forms
Forms field tested in 8 WIDA states
Field test analysis occurring presently
For more information on item development process: Jim Bauman’s session
Item Creation Process, continued… (Fall 2003 – present)
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 21
Pilot Testing Results
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 22
Pilot Test Participation
April – May 2004
5 Districts:
Kenosha & Milwaukee, WI; Chicago & Cicero, IL; Washington, DC
Approx. 1100 students in grades 1-12
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 23
Listening Test
Multiple choice
20-25 minutes
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 24
Sample Items: Listening, Science 1-2
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 25
Science Listening 1-2 P1
PI: “identify objects according to chemical or physical properties from pictures and oral statements”
SCRIPT: “A seed is small. Find the small seed.”
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 26
Science Listening 1-2 P2
PI: “match objects with their chemical or physical properties from pictures and oral statements”
SCRIPT: “One day the seed will grow into something large, round, and heavy. Find what the seed grows into.”
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 27
Science Listening 1-2 P3
PI: “identify and group objects according to chemical or physical properties from oral statements”
SCRIPT: “Seeds grow into plants. Find something else that grows.”
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 28
Percent Correct on Each Item (Grades 1-2: n = 173)
P1scored P2scored P3scored
0
0.2
0.4
0.6
0.8
1
Mean
0.940.89
0.79
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 29
Percent Correct on Each Item (by Tier)
A B C
Tier
0
0.2
0.4
0.6
0.8
1
Mea
n
0.839
0.983 1 0.98
0.589
0.897 0.882
P1scored
P2scored
P3scored
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 30
Percent Correct on Each Item (by Grade Level)
1 2
Grade
0
0.2
0.4
0.6
0.8
1
Mea
n
0.931 0.949
0.851
0.713
0.872
P1scored
P2scored
P3scored
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 31PLEVEL
54321
Mean ITEMDIFF
1000
900
800
700
600
500
400
300
200
GRADELEV
g12
g35
g68
g912
Average Listening Item Difficulty Across Grade Clusters (by Proficiency Levels)
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 32
Average Reading Item Difficulty by Grade Level Cluster across P-Levels
1 2 3 4 5
Plevel
0
200
400
600
800
1,000
Mea
n It
emD
iffi
cult
y
GradeLevelg12
g35
g68
g912
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 33
Reading Test
Multiple choice
35-40 minutes
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 34
Examining Reading Items Across a Strand for the Different Tiers
Language Arts, Reading, Grades 3-5
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 35
Sample Item (Field Test): Reading, Lang. Arts, Grades 3-5, Tier C
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 36
#11: “To Jessica, he was the best dog in the world.” This sentence shows Jessica’s opinion. Which of the following also shows an opinion?PI (p3): Identify language associated with stating opinions found in fiction or non-fiction text.
#12: When Jessica saw the sign for the lost dog, why did she believe it was Blue?PI (p4): Differentiate between statements of fact and opinion found in various reading selections.
#13: Why does the woman say at the end of the story, “He’s your dog, all right!”?PI (p5): Identify author’s reasons or intent for selecting facts or opinions found in fiction or non-fiction from grade-level language arts text.
Items Tied to Performance Indicators: Tier C Items
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 37
Reading Items Adapted for Tier A Simpler text and more graphic support
Items at proficiency levels 1, 2, and 3• For example:
1. Which is Blue?PI (p1): Match labels or identify facts from pictures and phrases.
2. “I know he has white spots.” Which words in this sentence tell you it is a fact?PI (p2): Identify language associated with stating facts found in short fiction or non-fiction text supported by pictures or graphics
3. [Same item from Tier C] “To Jessica, he was the best dog in the world.” This sentence shows Jessica’s opinion. Which of the following also shows an opinion?PI (p3): Identify language associated with stating opinions found in fiction or non-fiction text.
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 38
Tier Alignment with Proficiency Levels on Test Forms (L, R, W)
ENTERING BEGINNING DEVELOPINGEXPANDING BRIDGING
1 2 3 4 5
Annual ACCESS for ELLs
Tier A
Tier B
Tier C
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 39
Writing Test
Up to 1 hour
4 tasks per tiered form: 1. SI
2. MA
3. SC
4. LA/SS
Model provided to give background and structure for the task
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 40
Lang. Arts Writing 6-8 P5 PI: “defend positions or stances using original ideas with supporting details”
Sample Item: Writing Grades 6-8
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 41
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 42
Standard 1 2 3 4 5
(1) SI 1 1 1 0 0
(2) LA/SS 1 1 1 1 1
(3) MA/SC 1 1 1 1 1
Speaking Test: Adaptive Format
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 43
Sample Item: Speaking
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 44
Task (Proficiency Level) 1 Example: Grades 3-5
SI Speaking 3-5:
P1 “Respond to WH- questions”
First let’s talk about things people do outside. This is a picture of people in a park. I’m going to ask you some questions about this picture.
Q1: (Point to TREE) What is this?
Q2: (Point to BALL) What is this?
Q3: (Point to DOG) What is this?
Q4: (If necessary) What else do you see in this picture (OR) What other things do you see in this picture?
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 45
Task (Proficiency Level) 2 Example:
SI Speaking 35:
P2 “Ask and respond to questions”
Now listen carefully. I’ve just asked you some questions about this picture. Now I want you to ask me some questions about it. (OR) Pretend you are the teacher and want to ask me some questions about this picture. For example, you could ask me, “Where are the people?” OK?
Q1: (Point to BOY ON BIKE) What do you want to know about him? (OR) Ask me a question about him.
Q2: (Point to PICNIC TABLE) What do you want to know about this? (OR) Ask me a question about this.
Q3: What other things do you want to know about his picture? (OR) What’s another question you can ask me about (anything in) this picture?(Answer student’s question.)
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 46
Task (Proficiency Level) 3 Example:
SI Speaking 35:
P3 “Exchange personal information”
Now let me tell you something about these children. (Point to CHILDREN PLAYING CATCH) Their names are Alex and Leticia. They like to play catch.
Q1: Do you like to play catch?
Q2: (If “Yes”) What else do you like to do?
Q3: (If “No”) What do you like to do?
Q4: What do you like about __________? (OR) Tell me something about ___________.
Q5: (If necessary) Tell me more.
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 47
Field Testing: Overview
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 48
Field Test Participation
Request:• Group-administered sections, 600 per form
(minimum 400 per form)
• Individually-administered sections, 225 per form
Includes 8 WIDA states
Proportional representation of states (approx. 5.5%)
Approx. 8700 students total (~3500 from Illinois)
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 49
Initial Field Test Analyses
Concurrent calibration of MC items (Rasch)
Separate scale construction for each domain
Raw score to scale score on the screener
NOTE: Steering Committee determined the following weights when a single level designation is needed:
Writing 35% (e.g 2)
Reading 35% (e.g. 2)
Listening 15% (e.g. 5)
Speaking 15% (e.g. 4)
Composite = 2.75 (not yet 3)
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 50
Sample Feedback received to date
Input from test administrators and coordinators:
• Test Length: The test is taking longer to administer than anticipated.
• Test Difficulty: The test is more difficult than anticipated.
• Grades 1-2 Test: Too challenging for fall first graders; these students should take K test.
• Grades 9-12 Test: There is not enough authentic literature on the Reading Test.
CAL and DPI responded to these concerns via e-mail and the D2L (online training) discussion board.
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 51
Timeline for Development and Implementation
Field Testing (incl. English speakers): September - January 2005
Analysis of Field Test Data: November 2004 - January 2005
Setting Cut Scores: January 2005
Spring Roll-Out of Form 100 (AL, ME, VT): February - May 2005
Practice Items for Fall Roll-out Available: March 15, 2005
Screener Available: July 2005
Fall Roll-Out of Form 100 (IL, DC): Fall 2005
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 52
Training for ACCESS for ELLs
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 53
Training for ACCESS
August 2004: SEA WebEx conference on logistics and data needed for Field Test
September 2004: LEA Coordinators trained online via Desire2Learn at UW – Oshkosh
September – November 2004: Field Test Administrators trained online via D2L
December 2004 – January 2005: Speaking Field Test Administrators trained in-person (DC, WI)
March – May 2005: Operational Test Administrators giving spring roll-out test trained
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 54
WIDA Field Test Administrator Workshop
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 55
Operational ACCESS Test Administrator Training: Possible Scenarios
Beginning March 2005
2-hour total online training for all sections of the test
Additional, as-needed training on scoring the Speaking test, with online or live (state/district) trained facilitators.
For more information and updates:
www.wida.us
IRC Conference Jan 24-27, 2005 Standards, Structure & Field Testing of ACCESS for ELLs™ Jessica Motz [email protected] 56
Other Issues/Questions/Discussion