Holistic Scoring in Social Studies

Curriculum, Instruction, and Instructional Technology Team

New York State Education DepartmentAlbany, NY

2005

NYS Education Department 2005

Contact InformationCurriculum &

Instruction

Jo Ann Larsonjlarson@mail.nysed.gov

Lawrence Paskalpaska@mail.nysed.gov

Phone: (518) 474-5922

State Assessment

Donna Merlaudmerlau@mail.nysed.gov

Gary Warrengwarren@mail.nysed.gov

Phone: (518) 474-3860

NYS Education Department 2005

Purposes of the Regents

• Accountability: To measure student attainment of the Learning Standards for Social Studies at the commencement level

• Diagnostic: To help school districts identify those students who are in need of Academic Intervention Services (AIS)

NYS Education Department 2005

Goal• To achieve consistent and accurate scores

Note: not simply to have everyone assign the same scores,

but to assign them for the right reasons.

NYS Education Department 2005

Purpose of This Workshop

• Make the rating task as simple and painless as possible

NYS Education Department 2005

Cast of CharactersPRINCIPAL /

SOCIAL STUDIES COORDINATOR• Appoints a Scoring Coordinator who will

manage the training and logistics of the scoring process

• Appoints a Trainer

NYS Education Department 2005

Cast of CharactersSCORING COORDINATOR

• Manages the training and logistics of scoring• Organizes the movement of papers throughout

the entire scoring process• Records scores on Part I answer sheets

NYS Education Department 2005

Cast of CharactersTRAINER

• Responsible for the overall training session, including time management (starting on time, moving from step to step of training procedures, keeping to schedule)

• Responsible for the substance of scoring• Leads review of: specific rubric, procedures for

assigning holistic scores, each anchor paper and commentary, practice papers

NYS Education Department 2005

Cast of CharactersTRAINER

• Follows the procedures and adapts them when necessary to ensure that everyone can assign valid and consistent scores

• Answers questions that arise over how to evaluate papers or assign scores

• Monitors scoring session progress; helps keeps Raters on schedule

• Assists the Table Leaders with back-reading

NYS Education Department 2005

Cast of CharactersTABLE LEADERS

• Distributes the bundles of papers to be scored (with Support Staff help)

• Leads review of the generic rubric and specific rubric with reference to the task by matching evidence from the student responses to the rubric

• Leads review of each anchor paper and commentary

NYS Education Department 2005

Cast of CharactersTABLE LEADERS

• Leads discussion until Raters feel confident to move on to actual rating; discusses any discrepancies

• Helps the Trainer with the substantive issues that arise during discussion (e.g., why a paper received a certain score, how to explain a distinction between score levels)

NYS Education Department 2005

Cast of CharactersTABLE LEADERS

• Back-reads several papers for each scorer, to confirm that Raters assign scores consistently

• Serves as a third Rater to resolve scores that differ by more than one point

• Calculates a final score for each student’s essay• Ensures that Raters fill out score sheets correctly• Assembles all score sheets and training materials

at end of scoring session

NYS Education Department 2005

Cast of CharactersRATERS

• Scores the five practice papers without looking at the scores and commentaries provided

• Records own individual rating for each student’s essay on the rating sheet provided (not directly on the student’s paper/answer sheet), when the actual rating begins; the scores of each scaffold question may be recorded in the student’s examination booklet

NYS Education Department 2005

Cast of CharactersRATERS

• Should not correct the student’s work by making insertions or changes of any kind

• Note: Each essay must be scored by two different Raters; scaffold questions only need scoring by one Rater

NYS Education Department 2005

Cast of CharactersRATERS: QUALIFICATIONS

• Teachers of Grades 7-12 social studies• Special Education Teachers who are

knowledgeable about the GHG or USHG social studies curriculum

• Should have previous school, district, and/or regional training on scoring social studies essays or scaffold questions as part of the turnkey training process

NYS Education Department 2005

Cast of CharactersSUPPORT STAFF

• Distributes the bundles of papers to the tables• Moves the papers so that all parts and all papers

are completely scored• Collects the score sheets and training materials,

moving them to shipping centers or a storage facility, at the end of the session

NYS Education Department 2005

The Rating Process1. BEGIN• Select a Table Leader.• Review each scaffold question and/or essay task.• Discuss possible student responses.• Review and discuss the specific rubric.• Discuss the nature of responses that would be

expected at each score point of the rubric.Remember: This is a holistic scoring. A student does not need to achieve a “5” for each bullet in the rubric in order to be awarded a “5” on the essay. A “5” is not necessarily a perfect paper.

NYS Education Department 2005

The Rating Process2. REVIEW THE SET OF ANCHOR PAPERS• Read the student anchor papers individually.• Starting with the first anchor paper, the group

should discuss the scores that were awarded. Using the specific rubric, explain why each anchor paper earned the score that it did.

• Read the commentaries at the back of the packet for each of the anchor papers, to find the rationale for each paper’s score.

NYS Education Department 2005

The Rating Process3. REVIEW THE PRACTICE PAPERS• Read the student practice papers individually.• Using the specific rubric, each Rater at the table

awards a score to each practice paper without looking at the scores and commentaries provided after the papers.

• After each Rater has separately scored the practice papers, the Table Leader charts the scores and leads discussion on the scores that were awarded.

NYS Education Department 2005

The Rating Process3. REVIEW THE PRACTICE PAPERS• Where there are discrepancies, the Table Leader

should ask individuals with the most discrepant scores to discuss their rationale for awarding credit. The group should continue to discuss the scores until consensus is reached. (See the commentaries at the back of the packet for the scores awarded for each practice paper.) Discuss any discrepancies between the SED commentaries and the scores awarded above.

NYS Education Department 2005

The Rating Process3. REVIEW THE PRACTICE PAPERS• If additional practice is required to reach scoring

consensus, a sample of actual student answer papers from the test may be used.

NYS Education Department 2005

Setting the Scene …• Provide frequent breaks.• Provide incentives (e.g., water, snacks, meals).• Conduct the scoring session in a pleasant

environment.• Check for understanding of rating procedures

frequently.• Review the rubric after any long-term break

(e.g., a lunch period, overnight).• Work toward exact agreement in awarding credit.

NYS Education Department 2005

Materials Needed• Photocopiers• Fax machines• Computers and printers• Paper• Chart paper• Paper clips and rubber bands• Pens, pencils, markers, and highlighters

NYS Education Department 2005

Checking for Accuracy• Do the Raters share a common interpretation of

the rubric, or do some Raters have their own personal construct of the rubric?

• Are Raters applying the rubric appropriately, or are they influenced by irrelevant factors when scoring student performance?

• Are Raters using all of the criteria, or are they ignoring some aspects?

• Are Raters assigning scores reflective of the rubric?

NYS Education Department 2005

Other Reminders• “Scoring Key and Rating Guide” may not be

used until after the Uniform Statewide Admission Deadline (45 mins. after exam start time).

• Set aside one room as a central rating room for collecting, storing answer sheets/essay booklets, and for preparing/maintaining records, for the exam.

• Avoid having a teacher rate his/her own students’ responses.

• Provide adequate time for training and for rating.