Regents Distributed Scoring

School Training

Preparing the January 2013 Regentsfor Distribution to Scoring Sites

1

Office of Assessment OperationsDecember 2012

Agenda

• Background

• School Preparation

• Preparation for Day and Per Session non-Electronic Scoring

• Preparation for Electronic Scoring

• Scoring

• Scanning

2

3

Background

• In October 2011, the Board of Regents voted to prohibit teachers from scoring their own students’ state assessments, beginning in SY 2012-13.

• NYC has moved to strengthen this requirement for Regents exams by shifting to a distributed scoring system (DSS), where teachers will not score the exams of any student that attend their school.

• Requirement: Teachers will not score the exams of students attending their school.

• Preference: Each student’s exam is marked by staff from at least two different schools.

• To prepare for this shift, NYC implemented distributed scoring in select schools and for select exams during 2012 administrations.

4

January 2013: Framework

2012 January 2013

Scoring Sites • Regional• Campus• School-Based

• Regional

Scoring Format • Electronic• Non-electronic (Paper/Pencil)

• Electronic• Non-electronic (Paper/Pencil)

Scoring Model • Day Scoring • Day Scoring• Per Session Scoring

Exams • Select subjects• Exams written in English

• All 10 Regents exams• English + translated exams

5

January 2013: Scoring Format & ModelsDay-Scoring Per Session

Non-electronic

Integrated Algebra EnglishLiving Environment

Algebra 2/TrigonometryGeometryUS History Gov’tGlobal History & GeographyPhysics

Electronic n/a Earth ScienceChemistry

Day-Scoring: Only schools administering these exams in January will be required to contribute staff to meet the scoring requirements.

Per-Session: Staff from schools citywide will be eligible to apply for per session scoring positions.

6

January 2013: Exams & Scoring Timeline

• Scoring staff are expected to be at the scoring site by 8:30 a.m. each day and will be expected to stay for the full length of the teacher work day (i.e., 6 hours and 20 minutes plus extended time).

• Staff serving in site management roles should be available before and after the scoring day.

• The exact days and times for per session scoring of each exam have yet to be determined. Scoring periods for exams scored in per session will be communicated later this fall.

Exam Tue.1/22

Wed.1/23

Thu.1/24

Fri.1/25

Sat.1/26

Sun.1/27

Mon.1/28

Tue.1/29

Integrated Algebra AM Admin Day Scoring

Chancellor’s PD Day

(No Scoring Activities)

Spring Semester

Begins(No Scoring Activities)

Comprehensive English PM Admin Day Scoring

Physics PM Admin

Per Session Scoring (tentative)

Living Environment AM Admin Day Scoring

Algebra 2/Trig PM Admin Per Session Scoring (tentative)

US History & Gov’t. AM Admin Per Session Scoring (tentative)

Geometry AM Admin Per Session Scoring (tentative)

Earth Science PM Admin Per Session Scoring (tentative)

Chemistry PM Admin Per Session Scoring (tentative)

Global History & Geog. AM Admin Per Session Scoring (tentative)

Key Information for January 2013• All exams (including translated versions) will be scored via the DSS model.

• All answer documents will be two (2) pages except Physics*.

• Schools print ATS information labels and affix them to answer booklets for all exams except Earth Science and Chemistry or handwrite Name and Student ID on the booklet.

• Schools administering the Earth Science and/or Chemistry Regents will participate in the electronic scoring process, which have distinct packaging directions.

• Test booklets for Earth Science and Chemistry must be labeled with pre-printed CTB labels.

• Schools will print pre-populated rating sheets for the English, Global and US History Regents.

• Schools will RETAIN and scan all page 1 answer documents except Physics* by 12 noon the day following administration..

• Only page 2 will be scored and scanned at central scoring sites for all exams except Earth Science and Chemistry**.

• Each school’s scorers and exams are sent to different scoring sites.

• Schools will provide staff to score the Comprehensive English, Integrated Algebra and Living Environment Regents exams during the day.

• All other exams will be scored as a per session activity.

• Schools that host a YABC should package those exams with their own.

• Read DAPS Regents Memo #4 at https://portal.nycenet.edu/Accountability/Assessment/YearlyTesting/MemorandaProcedures/&AuthResend1908BC2350124b5095AB75012FA405BA .

7

*See section on Physics. **See section on electronic scoring

School Preparation

8

School Preparation Required Prior to Exam

9

Deadline Activity

December 11 - 21 Principal assigns teachers to serve as scorers using web-based tool

January 4 School complete exam scheduling in STARS

January 7 Planning • Develop proctoring assignments that account for the availability of teachers who will be involved in scoring

• Reach out to network assessment liaison if you need assistance developing an appropriate proctoring schedule.

January 14 Schools receive and store packaging materials

January 18 Train proctors on administration proceduresPrinting• Complete printing of answer documents via ATS•Print ATS Student Information Labels for English, history, math, Living Environment and Physics Regents to affix to test and essay booklets. • Print section attendance rosters

•Print Rating Sheets for English, US and Global History Regents (directions to follow

• Print documents for packaging materials (see memo attachments)

Important Administration Procedures

• Program students in STARS by room; each room is a section.

• Distribute exam invitations produced in STARS.

• NEW! Print Student Information labels and affix to the appropriate

area of Regents test booklets and essay booklets of all math,

English, history, Living Environment and Physics Regents. (More

information to follow.)

• Earth Science and Chemistry test booklets must be labeled with

bar-coded labels provided by CTB.

• Track attendance.

• Ensure students use appropriate writing utensil.

• Ensure students sign declaration.

• Place student answer documents inside student test booklets.

• Collect test booklets and answer documents.

10

Pen/Pencil PolicyComprehensive

EnglishMath Science History

Pencil Multiple Choice

Student Declaration*

Multiple Choice

Student Declaration*

Graphs and Drawings

Multiple Choice

Student Declaration*

Diagrams and Drawings

Multiple Choice

Student Declaration*

Pen Short-Constructed Response

Essay

All responses written in test booklet except, graphs and drawings

All responses written in test booklet except, diagrams and drawings

DBQ

Essay

11

*Students are permitted to sign the declaration in pen.

12

Track Attendance

• Proctors should track student attendance using the section attendance roster. See DAPS Regents Memo #4 attachment for sample. Schools may use an equivalent school generated document.

• Indicate the following on the section attendance roster:> The testing school’s DBN and the name of the exam on the

top of the roster, including each students’ name, NYC Student ID number.

> Attendance (absent or present).> If the student wrote the exam in a language other than

English .> If the exam was invalidated i.e. missed labs, cheating, etc.

Section Attendance Rosters

13

Preparations for Day and Per Session (Non-Electronic) Scoring

English, US and Global History, Integrated Algebra, Geometry, Algebra2/Trigonometry,Living

Environment and Physics

14

Label Test Booklets• Schools will run student labels and affix them to Regents test and essay

booklets in the upper left hand corner of all exams EXCEPT Earth Science and Chemistry (see directions in packing for electronic scoring).

• Do not put labels on the answer documents.

• Use Avery 5160 (30 labels per sheet) or equivalent.

15

Sample Only English, US and Global History have an essay booklet.

Sample

Labels for all Regents except Chemistry and Earth Science

16

Collecting Student Answer Documents

• Place student answer documents and, if appropriate, essay booklets inside the Regents test booklet.> To ensure that answer documents and test booklets remain

appropriately matched, proctors should ask students to place their answer document (and essay booklet) inside their Regents test booklet upon completion of the exam.

> Essay booklets are used in the English, US and Global History Regents.

17

Collecting Student Answer Documents

• When collecting a Regents exam every proctor must:• Check that the student has signed the declaration.

• Ensure that an answer document, and if appropriate, an essay booklet, are received from every student and inserted into the Regents test booklet.

• Alphabetize booklets and check against student roster.

Essay booklets are for English,

Global and US History only. This

step N/A for all other Regents

exams.

Declaration must be

signed

Packaging Test Materials for Scoring

• Page 1 answer documents are removed from the test booklet.

• Page 2 answer documents with the ELL and Special Education accommodations completed, as appropriate, are inserted into the correct students’ test booklets and packaged for shipment to the scoring sites.

• Page 2 documents for Earth Science and Chemistry are retained and will be scanned at the school.

• The performance score for Earth Science must be bubbled in on page 2 before scanning.

• Essay booklets for the English, US and Global History Regents are also included inside the test booklet.

• Retain and scan ALL page 1 answer documents by 12 noon the day following administration.

• Retain test booklets and scan the page 2 answer documents of absent students and students with invalidated tests.

18

19

Packing Sections

• When packing Regents exams for Day (non-electronic scoring):

• Complete any IEP/504 accommodations.

• Bubble in alternative language if administered to a student.

• Ensure that the page 2 answer document and, if appropriate, an essay booklet,

are inserted into the Regents test booklet*.

Essay booklets are for

English, Global and US

History only.

Bubble IEP/504 and/or

ELL accommodations,

and alternate language

versions administered to

students. *See special directions for Physics.

20

Packaging Sections

• A section is defined as group of students who write the test in the same room.

• Each section will be packaged in its own polybag, which should include the

following in the order listed:

> A completed Section Header (see DAPS Memo #4 Attachment).

> A Section Attendance Roster (see DAPS Memo #4 Attachment).

> (English, US and Global History only) school produced preslugged Essay

Rating sheet that corresponds to the section list.

> Test booklets – except Physics

Page 2 answer document, and

(English, US and Global History only) Essay booklet are placed inside the

test booklet, for all tested students in that section.

Do not send Page 1 answer documents to the scoring site.

Do not send test booklets for students who were absent or had invalidated

tests.

• The exams of graduating seniors are not separated for January.

21

Section List

Be sure that:> All information on the Section List form is completed by the

testing school including day and cell phone numbers.> Each student test booklet includes the corresponding page 2

answer document.> The count of answer documents and test booklets on the

Section List match the number of answer documents and test booklets in the polybag.

> All answer documents and test booklets for absent students and students with invalidated tests are retained at the testing school for scanning.

Section List

22

Submission of Scoring Materials: Non-electronicPacking the Polybag

23

STEP 1: Page 2 answer document is placed inside the

same student’s Regents booklet

STEP 2: Alphabetize each class.

English, US and Global History: Insert Page 2

answer document and essay booklet

Math and Living Environment exams: Insert Page 2 answer document

only

Physics: Answer document and answer book

Do not send Page 1 answer documents or booklets or

documents of absent students or invalidated tests.

STEP 4: The Section

Attendance Roster is placed on top of

the class pack. Annotate with notes

about absentees, translations, etc.

STEP 6: Each section is placed in a large polybag. Do not mix sections in the same polybag.

STEP 5: The Section List is placed on top of

the Roster

STEP 3: English, US and Global History will also have a school produced preslugged Essay Rating Sheet included in each polybag. Skip this for math and science exams

Physics Packaging

The Physics Regents is a secure exam that requires special handling.

• Each student answer booklet is numbered and included in the same envelope as the

exam booklet. Students are to write their answers to all of the exam questions, multiple-

choice and open-ended, directly in the answer booklet.

• No answers are recorded by students on the one page school generated answer

document.

• At the conclusion of the exam students place their exam booklets, reference tables and

scrap paper in an SED supplied secure return envelope and seal according to SED

directions.

• After bubbling any accommodations the school generated answer document is placed

inside the numbered SED Physics answer booklet.

• Do not scan the school generated answer document.

• Pack answer books for scoring using directions for all other non-electronic exams.

• Bundle together the envelopes containing ALL used and unused exam booklets and pack

with the answer documents for shipment to the scoring site.

• All Physics testing materials will be returned to SED directly from the scoring site.

24

Physics Answer Document

The Physics school generated scannable answer document is one page and is NOT distributed to students.

25

26

Packaging Boxes

• Each exam subject is packaged in its own box. If there is more than one box for an exam subject, indicate the box number out of the total shipment (e.g. Integrated Algebra – Box 1 of 2, 2 of 2, etc.) on the outside of the box.

• Every box should include the following in the order listed:> Packing List.> Polybags with exams (equal to the number of testing

sections).

• Be sure that:> All information on the Packing List to be completed by the

testing school is completed.> The number of sections listed matches the number of

polybags included in the box.

Packing List

27

Please complete all sections of the form

Packing the Box: All exams except Earth Science and Chemistry

28

Each section is packed in a large polybag. Do not mix sections in

the same polybag.

ALL sections of an exam are put into one or more boxes DO NOT MIX

subjects in boxes!

Complete information on each side of the box

One Packing List is included in every

box.

Exam Pick-Up Schedule

• Deluxe Courier will pick-up non-electronic scoring test materials for

delivery to the scoring site

• Assessment Distribution Services (ADS) will pick up Chemistry

and Earth Science test materials for electronic scoring.

• A school staff member must be at the school and available during

the delivery window to provide the test materials to the courier.

29

* Electronic scoring - Assessment Distribution Services (ADS) will pick up Chemistry and Earth Science test materials.

Exam Administration Pick-up from schoolIntegrated Algebra January 22 (9:15 AM) January 22 (3:00 PM) Comprehensive English January 22 (1:15 PM) January 23 (7:30 AM) Physics January 22 (1:15 PM) January 23 (3:00 PM)Living Environment January 23 (9:15 AM) January 23 (3:00 PM) Algebra 2/Trigonometry January 23 (1:15 PM) January 24 (3:00 PM)Geometry January 24 (9:15 AM) January 24 (3:00 PM)U.S. History January 24 (9:15 AM) January 25 (3:00 PM)Global History January 25 (9:15 AM) January 25 (3:00 PM)Chemistry* January 24 (1:15 PM) January 25 (7:30 AM)Earth Science* January 24 (1:15 PM) January 25 (7:30 AM)

Exams Not Ready

• Exams not included in the initial pick up will have to be hand delivered to the BAO with an explanation as to why they were not included in the scheduled pick up i.e. student had extended time, student was in a conflict room, etc.

30

Borough BAID Contact Address

Manhattan-Districts  1, 2, 3, 4, 5, 6

Marie BusielloMBusiel@schools.nyc.gov

212 356-3784212 356-7523 fax

333 Seventh AvenueSeventh FloorNew York,  NY

Bronx- Districts  7, 8, 9, 10, 11, 12

Sharon CahrSCahr@schools.nyc.gov

718 741-5559718 741-7971 fax

1 Fordham PlazaSeventh FloorBronx, NY

Brooklyn-Districts 13, 14, 15, 16, 19, 23, 32

David RaphealDRaphea@schools.nyc.gov

718 935-5965718 935-5941 fax

131 Livingston StreetRoom 608Brooklyn, NY

Queens-Districts 24, 25, 26, 27, 28, 29, 30

Barbara MarcisakBMarcis@schools.nyc.gov

718 391-8352718 391-6088 fax

28-11Queens Plaza NSecond FloorLong Island City, NY

Staten Island/ Brooklyn - Districts 17, 18, 20, 21, 22, 31

Jose GarciaJGarcia17@schools.nyc.gov

718 390-1579718 420-5665 fax

715 Ocean TerraceBuilding A Staten Island, NY

Electronic Scoring

Chemistry and Earth Science

31

Preparing for Electronic Scoring

• Earth Science and Chemistry will be scored electronically.

• Electronic scoring does not affect the administration of the exams.

• Affix all pre-slugged CTB bar-coded labels to all test booklets.

• Page 1 and 2 answer documents for Earth Science and Chemistry will be scanned at the school.

• Complete all Accommodations and Alternate Language Information on Page 2.

• Ensure that Earth Science performance scores are bubbled on the page 2 answer document.

• Do not score any constructed response questions at the school.

• Scan Earth Science and Chemistry answer document pages 1 and 2 at your school.

32

Packaging Exams for Electronic Scoring

• Detailed directions for packaging exams for electronic scoring can also be found in DAPS Regents Administration Memo #4 at http://schools.nyc.gov/Accountability/resources/testing/MemorandaProcedures.htm

• Ensure that a bar-coded label with the student information is attached to every test booklet.

• Remove any answer documents from inside the test booklet and scan them at your school.

33

Label Earth Science and Chemistry Booklets• Chemistry and Earth Science Test Booklets- affix a bar-coded

label provided by CTB is placed in the upper LEFT hand corner of the test booklet.

• Generic labels will provided for students who do not have a pre-slugged label i.e. walk-ins

34

Pre-slugged CTB bar-coded Chemistry and Earth Science labels

Sample

Prepare Earth Science and Chemistry for Packaging

Prior to packaging collected exams for delivery to CTB schools should take the following steps:

• Remove answer documents for scanning at the school

• Ensure the following items are not in the test booklets:• scratch paper (retain in school for 1 year)• tape of any kind• Post-it™ Notes• Staples• Pins• paper clips

35

Room Header Form- Documenting how many completed test/essay booklets are returned to CTB.

• Complete a Room Header Form for each

assessment subject and booklet administered

within his or her class. It is essential that the

Room Header Forms be returned filled in

completely and accurately. It is recommended

that the school make a copy of each Room

Header Form.

• Make sure to complete the following steps:

• Verify that the information on each Room

Header Form is correct by checking the sheet

against the actual number of student test

booklets returned by the Room Proctor.

• Make any necessary adjustments required to the

total number of test booklets listed on the Room

Header Form.

36

Stack Cover Card- Documenting Exam Stacks per Section

Complete one Stack Cover Card for each section and

place it on top of the stack of test/essay booklets and

Room Header Form(s).

• Check the upper right corner of the stack cover card

for complete DBN, section, room, and exam subject

(content) information.

• Validate the number of assessments returned for

scanning matches the information on the Room

Header Form.

• Place the first stack cover card on top of the Room

Header Form.

• Secure the test books with paper bands as illustrated

on the stack cover card.

• A room may have one room header sheet but result in

several stacks. > Fill out an additional Stack Cover Cards for each stack of X

exams.

> Record the total number of stacks for that room in the

“Stack __ of ___” fields.

37

Band the Test Booklets into Stacks

After removing all materials from test booklets books must be bundled into stacks. Use the paper bands sent by CTB to bundle the used test booklet together after completing the Stack Cover Card and the Room Header Form.

A section may have one or more stacks of exams.

38

Used test booklets

Paper Band

Completing the Box Packing List

The Test Coordinator completes the Box Packing

List by listing the total number of stacks being

returned to CTB/McGraw-Hill from each room.

• IMPORTANT:

• Document the box label number from the

return shipping label in the space provided.

• List the test coordinator name and phone

number.

• Enter the total number of rooms returned in

the box.

• List the room, subject, and total number of

stacks for each room.

• Make a copy of the Box Packing List for the

school’s records.

39

Packing Sections for Electronic ScoringEach section should be packaged separately, and include the following in the

order listed:

• A completed Stack Cover Card.

• A completed Room Header Form.

• Test booklets, secured with a paper band, for all tested students in that section.> Be sure that:

all information on the Stack Cover Card and the Room Header Form is complete

each student test booklet has the pre-coded barcode label placed in the top right

corner of the test booklet

the count of test booklets on the Room Header Form matches the number of test

booklets in the polybags

all page 1 answer documents are scanned and retained in the school

all page 2 answer documents with accommodations performance scores and

alternative languages bubbled are scanned and retained in the school

all answer documents and booklets for absent students and students with

invalidated tests are retained at the school for scanning

40

Packing Boxes for Electronic Scoring

• Each exam subject (Earth Science or Chemistry) is

packaged in its own box. If there is more than one box for

an exam subject, indicate the box number out of the total

shipment (e.g., Chemistry– Box 1 of 2, 2 of 2, etc.) on the

outside of the box. Every box should include the following

in the order listed:> a subject-specific Box Packing List placed directly on top of the

stacks of test booklets.

> Be sure that:

all information on the Box Packing List to be completed by the

testing school is completed

the number of sections listed matches the number of stacks

included in the box

41

test booklets Section 1

Packing Process

42

Stack cover card for each section goes on top of Room Header form – One per Section. Please note: this form should go under the paper band

Room Header Form goes on the top stack of each section – one per section

Bundled stacks of each test section’s booklets go into box for CTB (repeat for each section)

Section 1

test booklets Section 1

Box packing list goes in the box on top of the stacks– one per box

Shipping label provided by CTB goes on top cover of the box

To CTB

Scoring

43

44

Scoring Schedule

Day Scoring• Scoring period for English and Integrated Algebra will be during the school day

and for three days or until scoring of all included exams is complete.

• Living Environment will be scored during the day for two days or until scoring of all included exams is complete.

• Scoring staff are expected to be at the scoring site by 8:30 a.m. each day and will be expected to stay for the full length of the teacher work day (i.e., 6 hours and 20 minutes plus extended time).

• Staff serving in site management roles should be available before and after the scoring day.

Per Session Scoring• Regents to be scored per session are US and Global History, Chemistry, Earth Science,

Physics, Geometry and Algebra 2/Trigonometry

• Scoring period for per session is scheduled to begin on Friday, January 25 and end on

Sunday, January 27.

• Exams may begin and end scoring at different days and times.

• Exact days and hours will be sent to all selected scorers.

45

Scorer Training

• Scorers will be trained on the first day of scoring by the subject area Scoring Content Leaders. Scorers are required to attend the entire training session.

• Day scorers must bring a copy of the Rating Guide released on the day of the exam for the subject area to be scored. > TCs can help by ensuring there is a plan for generating

scoring guides.> Use of electronic copies of the Rating guides will not be

allowed at the scoring sites to protect the confidentiality of student responses.

46

Scoring

• Scoring committees will be established by the Scoring Content Leaders, in collaboration with the Site Supervisor, and structured according to SED policy. No teacher will be allowed to grade an exam written by a student at his/her school.

• Scorers are expected to:> Score accurately.> Complete a sufficient number of test books to finish scoring within

the allotted time frame.> Maintain student confidentiality and objectivity while scoring> Maintain security of test materials.

• Scorers will sign their school’s Examination Scoring Certificate at the scoring site. The scoring certificate will be returned with each school’s test materials for return to SED in the school’s Regents boxes.

Scanning

47

Scanning - Schools

• Schools will scan:• All page 1 answer documents by 12 noon the day following

administration.• All page 2 answer documents for Earth Science, Chemistry

and exams of students who are absent or have invalid tests.

• All other page 2 answer documents will be scanned at the scoring sites.

• Schools will be responsible for data capture errors (RGTU) on all documents scanned at their school.• Schools will not have access to RGTU for page 2 answer

documents scanned at scoring sites except to change accommodations or language bubbles.

• Scoring sites will not have access to RGTU for answer documents scanned at home schools.

48

Scanning – Scoring Site

• All page 2 answer documents will be scanned at the scoring site.

• In some cases, scanners may not accurately capture the information that a teacher provided on page 2 of an answer document.  In those cases, corrections will be made at the scoring site using the RGTU function in ATS to manually correct the errant data.

• Please note that scoring sites scan in batches and all of a school’s answer documents may not be scanned at the same time. If schools do not see all of their data initially it is because scanning has not been completed. Scores should be available by the end of the scoring window.

49

50

Access to Data

• Schools will be able to access a number of reports in ATS, including REDS:• Status Report• Item Report• Item Distribution Report.

• Scores for non-electronic exams will be available in ATS after both the page 1 and page 2 answer documents have been scanned.

• Scores for electronically scored exams will be available after page1 and page 2 answer documents have been scanned at the school and the scorers grade each exam electronically.

•  More details on scanning will be available shortly.

The Finality of Examination Scores

• Per SED guidelines, when the teacher scoring committee completes the scoring process, test scores must be considered final and will be entered onto students’ permanent records via the scanning process.  

• Scoring committees, once they have rated an exam, are not permitted to re-score them, regardless of the final score. Specifically, exams receiving a final score of 60-64 or 50-54 may not be re-scored.

• Principals and other administrative staff in a school or district do not have the authority to set aside the scores arrived at by the teacher scoring committee and rescore student examination papers or to change any scores assigned through the procedures outlined in the scoring materials provided by SED.

51

52

More Information

• For questions contact your network assessment liaison.

• See the NYC Distributed Scoring of New York State January 2013 Regents Exams memo for detailed instructions regarding preparations for exam administration.