1 4sight benchmark assessments a data tool in pennsylvania’s standards aligned system summer 2008...

1

4Sight Benchmark Assessments

4Sight Benchmark Assessments

A Data Tool in Pennsylvania’s Standards Aligned System

Summer 2008

© 2006 Success for All Foundation ZZ4146

© 2006 Success for All Foundation 2

Initial Training ObjectivesInitial Training Objectives

• Understand the design and purpose of 4Sight Reading and Math Benchmark Assessments as part of the Standards-Aligned Systems.

• Facilitate the administration of 4Sight Reading and Math Benchmark Assessments.

• Facilitate the scoring of 4Sight Reading and Math Benchmark Assessments.

• Begin to develop plans to prepare staff to administer, score, and utilize 4Sight.

• Generate and read proficiency and subscale reports in the Member Center.

Pennsylvania’s Standards-Aligned System

Pennsylvania’s Standards-Aligned System

April 2008


Assessment ActivityAssessment Activity

• Brainstorm and list all of the assessments you administer (district, building, grade level, classroom).


Fundamental to ensuring usefulness of an assessment is:

Fundamental to ensuring usefulness of an assessment is:

I. Understanding the purpose for which the assessment was developed;

II. Understanding the purpose for which it is being administered; and

III. Understanding how the data is used and fits in school structures.


Types of Assessment in Standards Aligned System

Types of Assessment in Standards Aligned System

The assessment was developed as:

• Summative/Outcome: “Reaching Our Goals”

• Formative/Progress Monitoring: “Growth Charts”

• Diagnostics: “In-depth View”

• Benchmark: “Progress Towards Standards”

http://www.portal.state.pa.us/portal/server.pt?open=512&objID=3188&&level=1&css=L1&mode=2






Summative/OutcomeSummative/Outcome• Purpose

•Reports how a student compares to a certain standard

•State or local accountability•Helps to draw conclusions regarding

groups of students•Cannot inform current instructional

practices• An integral part of an assessment plan

• Occurs at end of school level, grade, or course or at certain grades

• Provides outcome data


Summative/Outcome Measures (continued)Summative/Outcome Measures (continued)

• Examples

• PSSA• Unit exams• Final exams• End of year grades• Terra Nova


• Purpose• Provides data regarding impact of instruction on

learning• Data used to adapt instructional practices to meet

individual student need

• An integral part of an assessment plan• Given frequently to determine student growth• Classroom based assessments• May range from formal instruments to informal

observations• Results used to shape teaching and student learning

Black and Wiliam 1998

Formative/Progress MonitoringFormative/Progress Monitoring

Conducted at Student’s Instructional Level


Formative/Progress Monitoring (continued)Formative/Progress Monitoring (continued)

• Examples:• Response Cards, White Boards, Homework• Curriculum-based measurement (Fuchs and Fuchs; Deno and Merkin; etc.)

• Dynamic Indicators of Basic Early Literacy Skills (DIBELS)

• AIMSWeb• Edcheckup


DiagnosticDiagnostic•Purpose

• Identifies student’s strengths, weaknesses, knowledge and skills prior to instruction

• Provides additional information to plan effective instruction & targets for intervention

•An integral part of an assessment plan• Administered infrequently• Used to supplement information from

screening, formative/progress monitoring, and summative/outcome


Diagnostic Tests (continued)Diagnostic Tests (continued)

•Examples:• GMADE, GRADE• DRAs• Running Records• Cognitive Ability Scales (WISC, Stanford-

Binet, etc.) • Achievement Tests (WIAT, Woodcock-

Johnson, etc.)• Brigance Diagnostic Inventory or Basic Skills• Comprehensive Test of Phonological

Processing


Benchmark AssessmentBenchmark Assessment•Purpose

• Provides feedback on student progress towards proficiency on grade level standards

• Measures concepts, skills, applications (Anchors)

•Data informs instructional decision-making

•An integral part of an assessment plan• Administered periodically during school year• Standards-based Assessments• Strong reliability and validity


Benchmark AssessmentsBenchmark Assessments

• Examples• 4Sight Benchmark Assessments• Riverside Publishing• Dynamic Indicators of Basic Early

Literacy Skills (DIBELS)


Student Planning Process

Focus: Classroom of Students

Who: Teacher

Periodic Grade Level Planning Process

Focus: Groups of StudentsWho: Teacher Teams

How: Regular 1-2 hour meetings

Annual Building-wide Planning ProcessFocus: All Students

Who: School-wide TeamHow: Data Retreat, School Planning Process

District-Level Support(Budgetary Support, Professional Development, Resources and Time)

Student Learning Data

School Structures for Data-Informed Decision Making

School LevelPSSA & PVAAS

Standardized AssessmentsDistrict End-of-Year Tests

Final Benchmark Test

Classroom Level Initial: PSSA/PVAAS/finals, studentCyclical:

•4Sight, Benchmark Data – Student LevelContinuous•Individual Classroom Assessments•Progress Monitoring

Grade/Course LevelInitial: PSSA/PVAAS/final tests – class/subgroup levelsCyclical:

•4Sight, Benchmark Data - grade level•District quarterly assessments•Common Classroom Data

Demographic/Perceptual/Process Data

School Level •School Demographics•Discipline Data•Attendance Data•Mobility Rate

Grade/Course Level•Class Demographics •Class Engagement Data•Satisfaction Data•Attendance Data

Classroom LevelQualitative Data•Student Historical Information•Student Medical Information•Student Learning Information


Assessment ActivityAssessment Activity

•Review list created earlier.•Review or revise the purpose of each assessment.


What are 4Sight What are 4Sight Benchmark Benchmark

Assessments? Assessments? Where do they fit?Where do they fit?How do they fit?How do they fit?


What do each of these items have in common?

• The dipstick under the hood of a car• A blood pressure cuff • A weather thermometer

4Sight Benchmarks4Sight Benchmarks


4Sight Benchmarks4Sight Benchmarks

Another tool that can help direct our actions is

4Sight Benchmarks.


Alignment of 4Sight with State Expectations


How do the 4Sight Benchmarks mirror the look and feel of the PSSA

assessment?




Particularly note: • Lengths of passages • Types of passages • Types of questions (what skill or strategy is being addressed)

• Types of question stems/or the format of the question

• Types of problems


PSSA Student Reports for Reading Include:

PSSA Student Reports for Reading Include:

Student reports include:a) Scale score b) Performance level: advanced, proficient, basic, below basicc) Reporting categories:

• Comprehension and Reading Skills (standards 1.1 and 1.2)

• Interpretation and Analysis of Fiction and Nonfiction Text

(standards 1.1, 1.2, and 1.3)


PSSA Reading Blueprint 2007PSSA Reading Blueprint 2007

Reporting Category Gr. 3 Gr. 4 Gr. 5 Gr. 6 Gr. 7 Gr. 8 Gr. 11

Comprehension and Reading Skills 1.1 Learning to Read Independently 1.2 Reading Critically in All Content Areas

65 - 85%

60 -80%

60 – 80%

50 – 70%

50 – 70%

40 – 60%

40 -60%

Interpretation and Analysis of Fiction and Nonfiction Text1.1 Learning to Read Independently 1.2 Reading Critically in All Content Areas1.3 Reading, Analyzing and Interpreting Literature

15 – 35%

20 – 40%

20 – 40%

30 – 50%

30 – 50%

40 – 60%

40 – 60%


PSSA Student Reports for Math Include:

PSSA Student Reports for Math Include:

Student reports include:a) Scale score b) Performance level: advanced, proficient, basic, below basicc) Reporting categories:

• Numbers and Operations (standards 2.1 and 2.2)

• Measurement(standard 2.3)

• Geometry(standards 2.9 and 2.10)

• Algebraic Concepts (standard 2.8)

• Data Analysis and Probability (standards 2.6 and 2.7)


PSSA Math Blueprint 2007PSSA Math Blueprint 2007

Reporting Category

Gr. 3 Gr. 4 Gr. 5 Gr. 6 Gr. 7 Gr. 8 Gr. 11

Numbers and Operations

40 – 50%

43 – 47%

41 – 45%

28 – 32%

20 – 24%

18 – 22%

12 - 15%

Measurement 12 – 15%

12 – 15%

12 – 15%

12 – 15%

12 – 15%

12 – 15%

12 – 15%

Geometry 12 – 15%

12 – 15%

12 – 15%

15 – 20%

15 – 20%

15 – 20%

12 – 18%

Algebraic Concepts

12 – 15%

12 – 15%

13 – 17%

15 – 20%

20 – 27%

25 – 30%

38 – 42%

Data Analysis and

Probability

12 – 15%

12 – 15%

12 – 15%

15 – 20%

15 – 20%

15 – 20%

12 – 18%


Data Provided by 4SightData Provided by 4Sight

1. Estimate of student performance on the PSSA• Advanced, Proficient, Basic, Below Basic• Scaled Scores

2. Subscale data• Reporting Categories


Performance Level ReportPerformance Level Report


Reading Subscale DataReading Subscale Data

Subscale Information Provided on 4Sight: • Comprehension and Reading Skills • Interpretation and Analysis of Fiction and

Nonfiction Text• 1.1 Reading Independently • 1.2 Reading Critically in All Content Areas • 1.3 Reading Analyzing and

Interpreting Literature• Open-Response Items


Reading Subscale ReportReading Subscale Report

©2006 Success for All Foundation 15


Mathematics Subscale DataMathematics Subscale Data

Subscale Information Provided on 4Sight:


Mathematics Subscale Report

Mathematics Subscale Report

ADD SCREEN SHOT OF SUBSCALE REPORT SHOWING REPORTING CATEGORIES


Administering 4Sight Benchmarks


Who takes 4Sight Benchmarks?

• Students in grades 3–11 take an on-grade-level assessment.

• Use the same modifications/exemptions that are used for the PSSA.




Who, What, When, Where, How Long?

• 60-minute testing period per subject

• Homeroom or other grade-level groups

• Grades 3-8: 5 tests; Grades 9-11: 4 tests

• Baseline and quarterly as needed

• School sets common date and time(s)

• Directions in Administration and Scoring Guide




Why 60-minutes per test?• Correlations

• Test Validity

• Pilot

• Follow IEP for students w/ disabilities

• Differs from PSSA


4Sight To-Do List:4Sight To-Do List:

Preparing Staff to Administer 4Sight

1. Overview: What is it? Why are we doing it? Who takes it? What does it mean for me?

2. Schoolwide testing and scoring plan3. Materials: test booklets, answer

sheets/teacher score sheets, scoring masks, administration and scoring guide



Developing Your School’s 4SightAdministration Plan

1. Staff 4Sight orientation: set date and draft agenda.

2. Select dates for first benchmark administration.3. Decide who will administer the test such as

homeroom teacher, etc.4. Decide who will distribute 4Sight materials.


Scoring the 4Sight Benchmarks


Options for Administering & Scoring 4Sight Benchmark Assessments

1. Hand-Scoring2. Scanning3. On-line

School must score open-response items for all options.




HAND-SCORING ACTIVITY: Scoring Multiple-Choice Items

1. Understanding Source of Subscales2. Understanding Calculation of Proficiency


Scoring Open-Response ItemsScoring Open-Response Items

Reading & Mathematics

1. The 3-point rubric used for PSSA reading is also used with 4Sight.

2. The 4-point rubric used for PSSA math open-response items is also used for 4Sight.

3. Each open-response item has item-specific examples to assist scoring, just as the PSSA.

4. Plan for establishing inter-rater reliability.

5. Transfer open-response item scores to answer sheet.


Inter-Rater ReliabilityInter-Rater Reliability

Inter-rater reliability is established when different judges/teachers rate the similar performance or open-ended response items the same way.


Establishing Inter-Rater Reliability

Establishing Inter-Rater Reliability

• Inter-rater reliability increases when all raters have been trained in the use of a common rubric.

• Inter-rater reliability increases when two or more people score each response.

• At a minimum, if only one rater is available, teachers should not score their own students’ tests.




ACTIVITY: Establishing inter-rater reliability

WHY?



Developing a Scoring Plan• Who will score the test items?

- Multiple choice

- Open-response (Is training needed? Will scoring be done in teams? Etc.)

- Timeline

• Who will collect and organize the data?- Scoring, Scanning & Entering


How do we get the data, now that we scored the

assessments? How do we analyze and interpret data in PA’s

Standards Aligned System?

How do we get the data, now that we scored the

assessments? How do we analyze and interpret data in PA’s

Standards Aligned System?


Data Analysis Identify Proximity to Goal(s).Identify Areas of Concern. Develop Target(s).

DiscoveryBrainstorm Root Causes.Prioritize and verify Root Causes with data.Select the Root Causes that are most important and doable. Keep the number low.

SolutionsDesign interventions to address identified Root Causes.Evaluate results and readjust.

Steps for Data Analysis, Discovery, & Solution Identification

Data-Informed Decision-Making Process

Data AnalysisIdentify Proximity to Goal(s)Data AnalysisIdentify Proximity to Goal(s)


Our Goals for AYP 2008 have been set:

Reading: 63% ProficientMathematics: 56% Proficient


AYP Plan for PennsylvaniaAYP Plan for Pennsylvania

PennsylvaniaConsolidated State Application

Accountability WorkbookMay 1, 2003

Revised May 30, 2003Revised May 8, 2004

(Revised sections: 3.2, 5.4, 7.1, 7.2, 9.1, 9.2, and 10.1)for State Grants under Title IX, Part C, Section 9302 of the

Elementaryand

Secondary Education Act (Public Law 107-110)

Available at: http://www.ed.gov/admins/lead/account/stateplans03/index.html

http://www.ed.gov/admins/lead/account/stateplans03/index.html


AYP Variables AYP Variables

• Which students count for AYP?

• What are the identified subgroups?

• What is the minimum number of students for a subgroup to be included in AYP calculations?

• What is the date for determining which students are continuously enrolled?

• What is the participation rate?


Calculating AYP in PACalculating AYP in PA

AYP Subgroups: Number of students/subgroup for reporting purposes: 10

Number of students/subgroup for AYP calculation purposes: 40

1. White2. Black3. Hispanic/Latino4. Asian/Pacific

Islander5. Native American

6. Multicultural7. Low SES8. ELL9. Special

Education

Continuous Enrollment Date: October 1

Participation Rate Required: 95%


Calculating AYP in PACalculating AYP in PA

Continuous Enrollment Date: October 1

Participation Rate Required: 95% overall and for each subgroup

Academic

Measure

2006–07 2007–08 2008–09

% Proficient

SafeHarbo

r

% Proficient

SafeHarbo

r

% Proficient

Safe Harbo

r

Reading PSSA

54%10% less Basic and Below Basic



MathPSSA

45% 56% 56%


4Sight AYP Reports4Sight AYP Reports

• RP 3533: Preliminary AYP Counts

• RP 3379: Projections by Grade

• RP 3434: 4Sight Proficiency Projections


RP 3533: Preliminary AYP Projections

RP 3533: Preliminary AYP Projections


RP 3379: AYP Projections by Grade

RP 3379: AYP Projections by Grade


4Sight Member Center4Sight Member Center

On-line tool to capture 4Sight data.


To get a User ID & Password:https:// members.successforall.org To get a User ID & Password:

https:// members.successforall.org


Member CenterMember Center

Allows schools to:• sort and analyze data by:

- whole school - grade level - Homeroom and/or class sections and periods- reading group- individual student

• track data over time to assess progress.


Member Center web address:https://members.successforall.net

Demo Site: http://demo.successforall.net

Help Line: (800) 548-4998, ext. [email protected]

Setting Up Your School in the Member Center

Setting Up Your School in the Member Center

https://members.successforall.net/

http://demo.successforall.net/

mailto:[email protected]


Member Center Home PageMember Center Home Page

Check the News section of the Member Center every time you log into the

site – this is where we will communicate important information

about the Member Center and 4Sight to our member schools.

You can change your logon and password via Profile

Settings.

See QSG page 4.


4Sight – Two Types of Data4Sight – Two Types of Data

1. Proficiency Levels

2. Subscale Data


Data DialoguesData Dialogues

PRIMARY QUESTIONSHow many proficient?

SECONDARY QUESTIONSStrengths/Weakness

TERTIARY QUESTIONSWhy/Discovery/Root Causes


Primary questions are those that closely match your most essential goals.

They tend to be very, very simple.


Data Analysis • Identify Proximity to Goal(s).• Identify Areas of Concern. • Develop Target(s).

Discovery• Brainstorm Root Causes.• Prioritize and verify Root Causes with data.• Select the Root Causes that are most important and

doable. Keep the number low.

Solutions• Design interventions to address identified Root Causes.• Evaluate results and readjust.

Steps for Data Analysis, Discover, and Solution Identification

Steps for Data Analysis, Discover, and Solution Identification


Analyzing Proficiency ReportsPrimary Questions

Analyzing Proficiency ReportsPrimary Questions

• How many students are proficient in reading/math?

• How many students in this subgroup are proficient?

• How many students do you need in that subgroup to make AYP?

• How many more quarters until the PSSA is given?

• Who are the students just below proficiency?

• How many points do these students need to move up to be proficient? How many points per quarter?


Member Center Primary Question Reports

Member Center Primary Question Reports

Under Reports – School Reports:

• RP 3434: 4Sight Proficiency Projections

• RP 3869: Test Results Chart & Graph (performance levels)

• RP 3508: Predicted State Results

Under Testing Center:

• Proficiency Report


RP 3434: 4Sight Proficiency Projections

RP 3434: 4Sight Proficiency Projections


Primary QuestionsPrimary Questions

• How many students are proficient school-wide in Reading?

• Who are the students performing at the Basic Level in Reading?


Activity: Primary Questions

Activity: Primary Questions

• How many students are proficient school-wide in Mathematics?

• Who are the students performing at the Basic Level in Mathematics?


RP 3869:Test Results Chart and Graph

RP 3869:Test Results Chart and Graph

• Snag it from MC


Proficiency ReportsProficiency Reports


RP 3508: Predicted State Results RP 3508: Predicted State Results


Primary Question Tool:Administration & Scoring

Guide

Primary Question Tool:Administration & Scoring

Guide

4Sight Pennsylvania Benchmark Answer and Alignment Guide:

• Estimation of Student Performance on PSSA chart


PA Reading Grade 5 Numbers 1, 3, and 5Estimation of Student Performance on PSSA

PA Reading Grade 5 Numbers 1, 3, and 5Estimation of Student Performance on PSSA

4SightTotal Score

PSSA Performance

Level Score

5 832

6 865

7 898

8 931

9 964

10 996

11 1029

12 1062

13 1095

14 1128

15 1161

16 1194

17 1226

18 1259

19 1292

20 1325

21 1358

Performance Level Score Ranges for PSSAPerformance Standards, Grade 5

Performance Level Score

Advanced 1497 and above

Proficient 1275-1496

Basic 1137-1274

Below Basic 1136 and below


Data DialoguesData Dialogues

PRIMARY QUESTIONSHow many proficient?

SECONDARY QUESTIONSStrengths/Weakness

TERTIARY QUESTIONSWhy/Discovery/Root Causes


Secondary questions discover what strengths and weaknesses are according to student learning.

They tend to be more specific.



Discovery• Brainstorm Root Causes.• Prioritize and verify Root Causes with data.• Select the Root Causes that are most important and doable.

Keep the number low.


Steps for Data Analysis, Discovery, and Solution Identification


Analyzing Subscale Data Secondary Questions

Analyzing Subscale Data Secondary Questions

• What are the strengths identified on the graph?

• What are the areas for concern?

• If the graph is flat, are the students proficient readers or close to nonreaders?


Member Center Secondary Question Reports

Member Center Secondary Question Reports

Under Reports – School Reports:

• RP 3304: Subscale Test Results

• RP 3861: Subscale Averages Graph

Under Testing Center:

• Subscale Test Results


Subscale ReportsSubscale Reports


4Sight Data Analysis4Sight Data Analysis

+


RP 3861Subscale Averages Graph RP 3861Subscale Averages Graph

• Snag it from MC


RPT 3304: Subscale Test ResultsRPT 3304: Subscale Test Results


ACTIVITYACTIVITY

• ADD ACTIVITY – 3 QUESTIONS HERE…



Discovery• Brainstorm Root Causes.• Prioritize and verify Root Causes with data.• Select the Root Causes that are most important and doable.

Keep the number low.


Steps for Data Analysis, Discovery, and Solution Identification


Member Center Secondary Question Tools

Member Center Secondary Question Tools

• 4Sight Pennsylvania Benchmark Reporting Categories: (Admin. Guide – PA standards condensed version)

• 4Sight Pennsylvania Benchmark Answer and Alignment Guide: (Admin. Guide – Reporting Category, PA Standard, Assessment Anchor)

• RP 3746: 4Sight Item Analysis by Subscale: (Under Reports – School Reports)


Reporting CategoriesReporting Categories


Pennsylvania 4Sight Reading Benchmark: Grade 5 Number 1 Answer and Alignment Guide

Pennsylvania 4Sight Reading Benchmark: Grade 5 Number 1 Answer and Alignment Guide

Item

No.

4Sight Scale Reporting Category

Assessment

AnchorDescriptor Correct

Answer (MC)PA Reporting

CategoryPA Standard

1Interpretation & Analysis

Reading Critically

1.2.5.A

R5.B.3.3.2 Headings C

2 Comprehension & Reading Skills

Reading Critically

1.2.5.A

R5.A.2.4.1Supporting details

B

3Interpretation & Analysis

Reading Critically

1.2.5.A

R5.B.3.1.1 Fact/opinion D

4Comprehension & Reading Skills

Reading Independently

1.1.5.F

R5.A.2.1.1 Multiple meanings

C


Reading Critically

1.2.5.A

R5.A.2.4.1 Supporting details

A

6 Comprehension & Reading Skills

Reading Independently

1.1.5.A

R5.A.2.6.1Author’s purpose

D


Reading Critically

1.2.5.A

R5.A.2.5.1 Main idea A


RP 3746: Item Analysis by Subscale – Snag it

from MC

RP 3746: Item Analysis by Subscale – Snag it

from MC


4Sight To-Do List:4Sight To-Do List: 4Sight Reading and Math Benchmark Assessment Initial Set-up

Checklist 2008-2009 School Year

• We placed an order for 4Sight on-line for each building using 4Sight.• We decided how 4Sight will be scored (hand-scored, scanned, on-line) and ordered the scanner, if necessary

(and have reviewed the tech requirements and ordered scanning software from Member Center if scanning OR 800.548.4998 ext. 2563)).

• We identified a main school district and building level contact for 4Sight.• We have registered our district and schools in the 4Sight Member Center, including assigning teachers log-in

information and appropriate security level access.• We have input student demographic information in the 4Sight Member Center.• We have developed a training plan to introduce 4Sight to the staff and explain its design, purpose,

administration, scoring and how it fits in existing school structures.• We have completed the plan for receiving the shipped 4Sight assessments in our buildings throughout the

school year (see sample tracking log).• We identified a testing plan and testing schedule (EAP Districts refer to the testing schedule in the EAP

Guidelines) for 4Sight.• Will the entire staff be involved?• How many times per year will we administer 4Sight?• When will 4Sight be administered?• Will we set a common time across the building for 4Sight administration?• How will we address the needs of students with accommodations and modifications in their IEPs or

504 Plans?• Will students be placed in the same groups for 4Sight administration as they are for PSSA?

• We have developed a training plan for analysis of 4Sight (and other relevant) data that follows the 4Sight testing schedule so staff has the opportunity to analyze and respond to the data.

• We have a plan for scoring (and scoring training including inter-rater reliability) 4Sight including the open-ended response items.

• We have determined a plan for sharing this data with students, parents, etc (as necessary).


III. What 4Sight is and is not…III. What 4Sight is and is not…

4Sight is… 4Sight is not…

a low stakes assessment. a high stakes assessment.

administered on grade level.

administered on instructional level*. (*unless student on grade level)

an assessment that gives information on progress towards proficiency on PSSA.

a diagnostic assessment that gives information on specific skills.

administered 3-5 times a year as a snapshot.

administered regularly as a progress monitoring tool for students performing below grade level.

1 4sight benchmark assessments a data tool in pennsylvania’s standards aligned system summer 2008...

Documents