1 new england common assessment program (necap) setting performance standards
TRANSCRIPT
![Page 1: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/1.jpg)
1
New EnglandCommon Assessment Program
(NECAP)Setting Performance Standards
![Page 2: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/2.jpg)
2
Purpose
• Provide data to establish the following three cut scores for Reading, Grades 3-8 and Writing, Grades 5 and 8:– Proficient/Proficient with Distinction– Partially Proficient/Proficient– Substantially Below Proficient/Partially
Proficient
![Page 3: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/3.jpg)
3
What is Standard Setting?
• Set of activities that result in the determination of threshold or cut scores on an assessment
• We are trying to answer the question:– How much is enough?
![Page 4: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/4.jpg)
4
What is Standard Setting
• Data collection phase
• Policy/Decision making phase
![Page 5: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/5.jpg)
5
Many Standard Setting Methods
• Angoff
• Body of Work
• Bookmark
![Page 6: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/6.jpg)
6
Choice of Method is Based on Many Factors
• Prior usage/history
• Recommendation/requirement by some policy making authority
• Type of assessment
![Page 7: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/7.jpg)
7
Choice of Method is Based on Many Factors
• Weighing all these factors, it was determined that the methods to be used are:– Reading: Bookmark– Writing: Body of Work
• They are both well-established procedures that have been successfully used on many assessments
• They have produced defensible results
![Page 8: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/8.jpg)
8
Choice of Method is Based on Many Factors
• Bookmark is appropriate for assessments that consist primarily of multiple-choice items but also include some constructed-response items
• Body of Work method works well for assessments that consist primarily of constructed-response items
![Page 9: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/9.jpg)
9
Standard Setting vs. Standards Validation
• Standard setting– Generally three rounds:
• Round 1: individual ratings
• Round 2: table group discussion of Round 1 results, followed by second round ratings
• Round 3: whole room discussion of Round 2 results, followed by final ratings
![Page 10: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/10.jpg)
10
Standard Setting vs. Standards Validation
• Standards validation– Starting cut points are provided to panelists at the
beginning of the process
– Two rounds:• Round 1: table group discussion of starting cut points,
followed by first round ratings
• Round 2: whole group discussion of Round 1 results, followed by second round ratings
• The process you will be following will be closer to a standards validation process
![Page 11: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/11.jpg)
11
Details for Standards Validation using the Bookmark and Body of Work Procedures
![Page 12: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/12.jpg)
12
Starting Cut Points
• Starting cuts for Reading and Writing were determined using teacher judgment data:– At the time of testing, teachers were asked to categorize
each student into an achievement level category, based on classroom performance (not test performance)
– Using these categorizations as well as the students’ test scores, Measured Progress staff calculated starting cut points
![Page 13: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/13.jpg)
13
Starting Cut Points
• Analyses of teacher judgment data indicate:– Good participation rates– Teachers appear to have been conscientious in
completing the task
• In other words, available evidence supports the validity of the starting cuts
![Page 14: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/14.jpg)
14
Starting Cut Points
• However, standard setting is also a critical part of the process because:– the teachers based their judgments on classroom
performance, not performance on NECAP
– different teachers have different conceptions of what it means to be, for example, Partially Proficient
– they did not have the benefit of discussion with their colleagues from across the three states
– etc.
![Page 15: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/15.jpg)
15
Starting Cut Points
• Both sources of information are important. Therefore:– You are free to recommend changing the
starting cuts, but we don’t expect to see major changes
– The changes you recommend should be based on the test content and the Achievement Level Descriptions
![Page 16: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/16.jpg)
16
The Bookmark Procedure(Reading)
• The Bookmark procedure is a standard setting method that uses a book of items (ordered from easiest to hardest)
• Panelists place bookmarks in that book of items
![Page 17: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/17.jpg)
17
What is the bookmark procedure?
![Page 18: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/18.jpg)
18
What is the bookmark procedure?
• For purposes of NECAP standard setting, you will be provided with starting cut points and will either validate those cuts, or recommend modifications to them
![Page 19: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/19.jpg)
19
A Technical Detail regarding the Bookmark Method
• Note that the ordered item cut point for a given cut does not equal the raw score a student must obtain to be categorized into the higher achievement level
• For example, the Substantially Below Proficient/ Partially Proficient starting cut for grade 3 reading is between ordered items 6 and 7; however, a student must obtain at least 20 points on the test in order to be classified into the Partially Proficient level
![Page 20: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/20.jpg)
20
How to Place a Bookmark
• A few concepts you will need to know:– The starting cut points– The achievement level descriptions– ‘Borderline’ students– What knowledge, skills, and abilities (KSAs)
are needed to answer each question
![Page 21: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/21.jpg)
21
How to Place a Bookmark
• Start at the beginning of the ordered item book• Evaluate whether at least 2 out of 3 students demonstrating
skills at the ‘borderline’ of Partially Proficient would correctly answer item 1
• Moving through the book, make this evaluation of each item• The bookmark should go where you no longer think 2 out of
3 Partially Proficient ‘borderline’ students would correctly answer the question.
• You may decide that the starting cuts are in the right place, or you might recommend moving them.
![Page 22: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/22.jpg)
22
How to Place a BookmarkItem Number
Would at least 2 out of 3 students who demonstrate skills at the Partially Proficient 'borderline' correctly answer this question?
1 Yes
2 Yes
3 Yes
4 Yes
5 Yes
6 Yes
7 Yes
8 Yes
9 No
10 No
11 No
12 No
13 No
14 No
15 No
… No
![Page 23: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/23.jpg)
23
How to Place a Bookmark
• In the example, the bookmark would go between items 8 and 9
• However, it won’t be that easy; there will be gray areas
• Place one bookmark for each cut score
• You will have the opportunity to discuss your bookmark placements and change them if desired
![Page 24: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/24.jpg)
24
How to Place a Bookmark
• To place your bookmarks you will need to be familiar with the achievement level descriptions and the assessment items
![Page 25: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/25.jpg)
25
How to Place a Bookmark
• Don’t worry, we have procedures, materials and staff to assist you in this process.
![Page 26: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/26.jpg)
26
Any questions about the Bookmark Procedure?
![Page 27: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/27.jpg)
27
The Body of Work Procedure (Writing)
• A standard method that uses approximately 30 complete sets of student work, including their answers to open-ended items and a display of their responses to the multiple-choice items
• Bodies of work cover the range of possible scores and are presented in order from lowest to highest total score
![Page 28: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/28.jpg)
28
What is the Body of Work procedure?
• You will classify each BOW into the achievement level in which you feel it belongs:– Proficient with Distinction– Proficient– Partially Proficient – Substantially Below Proficient
![Page 29: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/29.jpg)
29
The Body of Work Procedure
• Prior to beginning the process of rating the BOWs, you will:– take the assessment– become very familiar with the Achievement
Level Descriptions– define, with your grade-level group,
‘borderline’ students– individually review the full set of BOWs
![Page 30: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/30.jpg)
30
Making Your Ratings
• Beginning with the first BOW, and referring to the Achievement Level Descriptions and the definition of ‘borderline’ students, determine into which achievement level the BOW should be categorized
• Continue through each BOW in turn, assigning each to an achievement level
• You may agree with the categorizations based on the starting cuts, or you may recommend changing some of them
![Page 31: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/31.jpg)
31
Any questions about the Body of Work Procedure?
![Page 32: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/32.jpg)
32
What Next?
• After this session, you will break into grade-level groups, where you will:– take the assessment to familiarize yourself with the test
items
– Reading groups: complete the item map, which is a document that will help you with the bookmark placement process
– discuss the Achievement Level Descriptions and develop definitions of “borderline” Partially Proficient, Proficient, and Proficient with Distinction students
![Page 33: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/33.jpg)
33
What Next?
• You will:– discuss the starting cut points in table groups
and do the first round of ratings– discuss the first round ratings as a whole room
then do the second round of ratings
![Page 34: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/34.jpg)
34
What Next?
• After the second round of ratings, you will have an opportunity to provide feedback about the Achievement Level Descriptions
• As the final step, we will ask you to complete an evaluation of the standard setting process
![Page 35: 1 New England Common Assessment Program (NECAP) Setting Performance Standards](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649ce35503460f949af834/html5/thumbnails/35.jpg)
35
Good Luck!