Florida Interim Assessment Item Bank and Test Platform

(FL IBTP)

Scoring Overview

2

Agenda

• Overview of scoring process– Identifying and using anchor sets

– Identifying and applying calibration sets– Scoring to a rubric– Working with exemplars– Avoiding bias in scoring

• Panel discussion

3

Approach to Scoring• Score supportively.

• Look for what is correct and done well in the response, as opposed to what is incorrect or done poorly.

• View each response as a snapshot of the student’s ability to respond to the item; look at the response strictly for what it is.

4

The Scoring Process

5

Anchor Sets

• The anchor set is an item-specific guide to help the scorer identify the correct score point for each response.

• Determine which group of specialists in the district will identify the anchor sets.

• Manage logistics of anchor set development: How many responses will be in each set? How many copies are required? How will security be maintained? Who will store the anchor sets?

6

Anchor Sets Diagram

7

Determining Anchor Sets

• Establishing anchor sets is often called range-finding.

The goal is to– Select from available student responses examples

that best match the description of each score point in the rubric.

– Demonstrate a range of student performance within each score point. Scorers will compare each live paper to the corresponding anchor set responses to determine the best-fit score for the live response.

– Maintain historical scoring standards for similar items.

8

Using Anchor Sets

• Use anchor sets to train scorers, explaining how each anchor response illustrates that score point in the rubric.

• Encourage scorers to compare each response to the anchor responses.

9

Calibration Sets

• Like anchor sets, calibration sets are groups of papers for which the scores have been agreed upon previously by a select group of specialists.

• Responses in a calibration set are clear representatives of one score point in the rubric.

• Calibration sets level-set teams of scorers and ensure inter-rater reliability throughout the scoring process.

10

Calibration Sets Diagram

11

Using Calibration Sets

• Ongoing training during scoring• Distributed to scorers without being identified as

anything but additional responses to score• Correct a drift in scoring• Clarify or provide further training

12

Scoring to a Rubric• The rubric is a set of predefined criteria that

delineate the important characteristics on which responses will be scored.

• The rubric allows each scorer to remove subjectivity from his or her scoring and reduces the variation in scores created by individual biases and personal preferences.

• Traits not mentioned by the rubric must not affect the score a response receives.

13

• Assign a response to the score point where that response best fits. It may not fit perfectly into any one score point.

• Focus on the broad fit of each response to a score point. Do not get caught up in the small details.

• Remember that many different types of responses may be grouped into the same score point.

Applying a Rubric

14

Using Exemplars

• The rubric allows for a general or generic description of the level of understanding shown at each score point. The exemplar defines those levels of understanding in an item-specific manner.

• The exemplar is a representative response chosen to clarify the rubric, similar to an anchor response.

• The exemplar will not show all possible correct answers.

• The top score responses need not be perfect. Each score point, even the top score, shows a range of understanding.

15

Bias•Guard against biases, assumptions, preferences, and pre-conceived expectations when scoring.

•Look holistically for the parts of the response that can receive credit.

16

Assumptions

• Student identity

• Gender• Socioeconomic class• Regional location• Student personality• Sensitive or emotional content

17

Expectations

• Handwriting

• Response length

• Regionalism

• Five-paragraph essay

• Alternate approaches

• Subject-matter expertise

18

District Experiences with Scoring

19

Panel Discussion

• District sharing• Project overview

– Managing scoring logistics– Developing scoring policies– Developing and maintaining protocols

• Questions from the audience

20

Thank You

NOTE: These materials were developed under a grant from the U.S. Department of Education. However, the contents do not necessarily represent the policy of the U.S. Department of Education, and you should not assume endorsement by the Federal Government.

Copyright © 2012 Florida Department of Education. All rights reserved.