practical issues in computerized testing: a state perspective patricia reiss, ph.d hawaii department...
Post on 21-Jan-2016
222 Views
Preview:
TRANSCRIPT
Practical Issues in Computerized Testing:
A State Perspective
Patricia Reiss, Ph.DHawaii Department of Education
Agenda
Contextual Information on CAT Implementation in Hawaii Shadow Tests Scoring of Incomplete Adaptive Tests Speededness
Contextual Information
- Hawaii State Assessment: Online CAT since 2010- Smarter Balanced Assessment: CAT and PT- Computer Adaptive Algorithm
- Blueprint alignment: content (claims, targets, CCSS), cognitive complexity, and item types
- Maximize achievement level classification for total test
and reporting categories - Minimize overall measurement error- Stopping rule – test length
Flow
Select Eligible Items (not seen before)
Assign Starting Scores
Calculate Value for Each Item
Recalculate score
Administer item
Select highest valuedItem (or randomly from
highest)
Recalculate score
Field Test?
Go Back?
Recalculate score
Capture responsechange
Next item alreadyAdministered?
Recalculate score
Select field testitem
Update Constraints
Used with permission from American Institutes for Research
Practical Considerations
To allow for adaptations items in CAT are selected sequentially. But each of these practical problems implies constraints on item selection that can only be realized if all items are selected simultaneously.
Wim van der Linden
Practical ConsiderationsComputing resourcesNumber and cost of serversNetwork Bandwidth Reduce number of reassembled shadow
tests by requiringfixed intervals between reassembliesa certain minimal change in the interim
ability estimate
AIR Summative Simulation Report for Smarter Balanced March 2015
AIR Summative Simulation Report for Smarter Balanced March 2015
AIR Summative Simulation Report for Smarter Balanced March 2015
AIR Summative Simulation Report for Smarter Balanced March 2015
Scoring of Incomplete Adaptive Tests
For the Smarter Balanced online assessments, A student is required to provide a
response to an item before s/he can go on to the next item
However, the student is able to go back and change his or her answers
Scores for Incomplete Tests
For incomplete Smarter Balanced Assessments for which students answer at least 10 CAT items, the Consortium will derive a score based on the students’ actual responses to items and will consider as incorrect any item to which students do not provide a response. The difficulty of a student’s omitted items from the combination of online computer adaptive and performance tasks will be estimated based on the average difficulty of the items in the item pool.
For incomplete tests for which students answer fewer than 10 CAT items, the Consortium will assign the lowest possible scale score for the grade and content area.
Sub-scores are not reported for incomplete tests.
Differential Speededness
“Test items vary considerably in the amount of time they require.”
Examinees vary in how much time they require to answer test items.
Actual Testing TimesSmarter
Average testing time Max Test Time Min Test Time StdDev of Test TimeGrade 3 ELA 01:54:45 10:43:05 00:01:30 00:58:50Grade 3 Math 01:29:11 14:30:45 00:01:40 00:52:51Grade 4 ELA 02:05:19 12:09:37 00:02:10 00:59:41Grade 4 Math 01:39:38 10:51:03 00:01:42 00:54:07Grade 5 ELA 02:13:39 17:56:36 00:02:35 01:13:40Grade 5 Math 01:49:35 10:53:13 00:01:46 01:01:33Grade 6 ELA 02:08:01 12:17:17 00:01:44 01:01:32Grade 6 Math 01:43:07 13:58:16 00:01:36 00:48:51Grade 7 ELA 01:44:40 07:23:41 00:03:59 00:42:54Grade 7 Math 01:39:49 09:19:03 00:04:03 00:42:49Grade 8 ELA 01:45:30 09:19:16 00:04:11 00:42:56Grade 8 Math 01:39:58 19:20:24 00:02:22 00:45:04Grade 11 ELA 01:25:54 06:15:35 00:02:14 00:36:26Grade 11 Math 01:14:00 05:46:47 00:01:59 00:35:25
All test taken by 6/12/2015
Implementation Issues Related to “Time” Number of Computers
Bandwidth Prioritization and Management Highest priority given to assessments
Gaming, audio and video sites were blocked
Start times of tests staggered
School level feedback
Impact on duty free time for teachers
Bathroom breaks for teachers
Move students who require more time or keep all students in the same room
Conflict with lunch schedule
Loss of instructional time, extra-curricular activities, assemblies, field trips, etc.
top related