what is nceo?

+

Monitoring Access Tools and Accommodations in a New Assessment Context

Laurene Christensen, Ph.D.National Conference on Student Assessment

+ What is NCEO?

Federally-funded technical assistance center

Support SEAs and consortia on issues related to inclusive assessmentStudents with disabilitiesELLsELLs with disabilities

Website: nceo.info

+ Changing Assessment Context

Common Core State Standards

PARCC AssessmentDLM Assessment

College and Career Readiness

ELPA21

Smarter Balanced Assessment

NCSC Assessment

ASSETS

+ From Accommodations to Accessibility Tools and Accommodations

Accommodations are adjustments to the testing situation, test format, or test administration that provide equitable access during assessments for students with disabilities and students who are English learners.

+ Smarter Balanced Framework

+ PARCC Framework

+ Accommodations on DLM

Dynamic Alternate Assessment System Universal design Variety of response formats Compatible with variety of assistive

technologies Allows for varying levels of teacher assistance

+ Accommodations on NCSCFormat of assessment (AA-AAS):

4 complexity levels; 30 items per subject; can be delivered online

Most items are multiple choice Some constructed response; can use picture

symbols or graphic organizersMost accommodations are embedded

within itemsAllowed accommodations are included

in test administrators’ scripts

+ Accommodations on ELPA21

Still being decided

Administrative ConsiderationsUniversal FeaturesDesignated FeaturesAccommodations

+ Accommodations on ASSETS

ELP assessments Computer delivered Accommodations and

accessibility features built-in

Designed to be appropriate for a range of needs

+ How do all of these pieces come together?

+ First steps: The Need for MonitoringMonitoring requirements of IDEAMonitoring requirements from ESEAExpectations from

peer review (uncertain right now)

+ What is monitoring accommodations?Monitoring accommodations is a

process of evaluating policies, procedures, and practices in order to ensure that a state’s assessment system is inclusive of students with disabilities and English language learners.

+ Monitoring is about complianceThe State has analyzed the use of specific

accommodations for different groups of students with disabilities and has provided training to support sound decisions by IEP teams.

The State routinely monitors the extent to which test accommodations are consistent with those provided during instruction.

But it is about more than compliance….

+ Monitoring is about improving outcomesSystematic attention to the provision of

accommodations can ensure that students are able to show what they know and can do

+ Monitoring Accommodations: Overview of Tool Developed in collaboration with CCSSO Reviewed extensively by ASES SCASS group Based on a review of publicly available

materials 5 Step process Each step includes examples, a checklist,

additional resources Appendices include additional examples

from states Extensive glossary at the end

+ 5 Steps to Monitoring Accommodations

1. Know the rules and regulations for accommodations monitoring

2. Document decisions about access tools and accommodations

3. Document the use of access tools and accommodations

4. Review access tool and accommodations decisions and use

5. Evaluate and report on access tools and accommodations

+Document Decisions about Accessibility Tools and Accommodations

Keeping track of trainingKeeping track of decisions

General ed students ELLs Students with disabilities

Are access tools and accommodations used for instruction consistent with those used for assessments?

Keeping track of requests for unique accommodations

+ Document the use of accessibility tools and accommodationsDocumenting on the PNP/ISAAPDocumenting inappropriate use of tools and accommodations on test day

Documenting that students received tools and accommodations on test day

+ Review accommodations decisions and useDirect observation on test dayOnline record reviewsOn-site visits on a day other than test day

Interviews with students, teachers, and administrators about the use of accommodations

+ Evaluate and Report on Accommodations Analyze accommodations

Report on accommodations

Revise accommodations policies

+ Questions to Consider

How can we support educators in keeping up with the different policies of each assessment?

What additional support is needed to effectively implement the PNP?

With all of the data these new systems can generate, what are the questions we should ask of the data?

+ Other Questions?

Thank you!

Laurene Christensenlaurene@umn.edu

Beyond the PNP:Designing a Comprehensive State System

for Systematically Monitoring Accommodations

Melissa Gholson, CoordinatorOffice of Assessment and

Accountability

Common Core EssentialElements

Our System

English Language Proficiency Standards

2014 and Beyond

• Stakeholders meeting• Reviewing current policies, processes and

procedures• Develop short and long term goals• Supporting an ongoing research plan• Focus on improving our system

A Systems Perspective

A Systems Perspectives

Research Agenda

Peer Review

Participation Guidelines & Policies

WVEIS Data Process, Codes & Descriptions

IEP, 504, SAP and LEPPlans

Monitoring and the WVS 326 Process

Improving Accommodations

• Ensure that your state has a valid method for gathering data on which students are to receive specific accommodations in assessment, and a form to document what they receive on test day. (Electronic WVS.326 Process)

• Document how your state analyzes its accommodations data, including timelines ofanalysis. (State & District Reports)

• Establish a specific monitoring procedure that identifies issues in the selection of accommodations for individual students or the provision of accommodations for instruction or assessment. (Participation Guidelines)

• Include information on any consequences that result from any irregularities in the administration of assessment accommodations. (Report analysis, investigation team and invalidation procedures)

• Conduct studies that examine the link between instructional accommodations and assessment accommodations (Research agenda)

Accommodations Research Plan

2009Paper

version of

monitoring

2010Pilot

Electronic WVS

326ACT

2011Seconda

ry Feasibilit

yW2

2012All WV-Maps

ELL Research

Development

2013State & District

Reporting

2014 and Beyond

Focus on improvin

g the process

Accommodations Research – Using last 3 years of W2 data– Will be able to look for trends– The primary purpose of this report is to describe the accommodations assigned to SWD,

LEP, and 504 students for the 2011 – 2013 school years. • Discussion/Interpretation: How appropriate are the accommodation assignments based upon

student subgroup/performance?• How do these trends vary across time?

– Report may be able to - Describe: Relationship of WT2 to Accommodation Assignments• Does WT2 performance predict accommodation assignments? 2010 assessment and 2011

accommodations– Do accommodation assignments influence WT2 performance? 2010 accommodations

and 2011 assessment– Examine 2013-2014 online W2 data independently– After 2-3 years may be able to do DIF– Continue analyzing ELL data by ELP proficiency, achievement, by years in program,

include ELL who have met exit criteria for two years.

Accommodation Code Accommodation Shorthand n of Students Recommended to Receive Accommodation

% of Sample Recommended to Receive Accommodation

P02 Have test read aloud: Math, Science, SS Tests

13,456 58.8%

T04 Use extra time (not applicable to WESTEST 2)

6,750 29.5%

P26 Have directions/passage and prompt read aloud -- on-line writing

6,472 28.3%

P15 Have directions only read aloud--RLA

5,686 24.8%

T03 Provide more breaks 5,085 22.2%P18 Have directions rephrased by

trained examiner4,381 19.1%

T07 Flexible scheduling 4,104 17.9%R02 Indicate response to a scribe

(selected response and on-line writing)

499 2.2%

R04 Indicate response to a scribe (constructed response)

445 1.9%

P19 Use a large-print edition of the test

139 0.6%

Most Frequent Accommodations for Students with Disabilities

Peer Review Requirements • Providing an appropriate variety of assessment accommodations • Ensuring that the use of assessment accommodations yields meaningful scores• Ensuring that the use of accommodations is consistent with instructional

approaches, as determined by a student’s plan• Determining that assessment scores for students when administrated under

accommodated conditions, allow for valid inferences about the students’ knowledge and skills

• Establishing clear guidelines for including all students in assessment• Differentiate between accommodation policies for students with disabilities and

English Language Learners.

Participation Guidelines

• The document includes references to current state policies and procedures.

• The document is updated and revised annually by a stakeholder group

Accommodations Review

Smarter Balanced

Smarter Balanced

Universal ToolsAvailable to all

Designated SupportsAvailable for Students with LEP, IEP and 504 Plans (ISAAP)

AccommodationsAvailable for Students with IEP and 504 plans.

EmbeddedInto the platform

BreaksCalculatorDigital NotepadEnglish DictionaryEnglish GlossaryExpandable PassagesGlobal NotesHighlighterKeyboard NavigationMark for ReviewMath ToolsSpell CheckStrikethroughWriting ToolsZoom

Color Contrast/P28Masking /P37Text-to-Speech /P01Translated Test Directions/P30Translations(Glossary)/P31Translations (Stacked)/P32Turn off Any Universal Tools/P33

American Sign Language/P34Braille/P03 or P35Closed Captioning/P36Text-to-Speech/P13

Non-EmbeddedProvided locally

BreaksEnglish DictionaryScratch PaperThesaurus

Bilingual Dictionary /R15Color Contrast/P38Color Overlay/P39Magnification/P23Read Aloud/P02Scribe/R04Separate Setting/T09Translations(Glossary)/P41

Abacus/R05Alternate Response Options/R11Calculator/R19Multiplication Table/R20Print on Demand/P40Read Aloud/P14Scribe/R04Speech-to-Text/R21

Accommodations ProcessDecision to provide

an accommodation is documented in the

student plan.

The documentation is entered into the

WVEIS data system

Data must be verified and

reviewed prior to assessments

Accommodations Application provides the data on a daily

delta

Accommodations data is collected analyzed, reported to

districts and used to inform policies, practices and provide targeted technical assistance.

Accommodations Electronic Application• Accommodations application supports the process for

identifying students who need accommodations• Accommodations are updated on a daily delta and are

available at the school, district and state level.• The accommodations application allows users to filter

and sort to by school, student ID, group, name, grade or accommodation.

• The application allows for printing and the creation of spreadsheets

Verification Process

• Accommodations are verified locally prior to assessment.

• WVEIS Accommodations App are compared to current student plans

• The WVS. 326 is reviewed to prepare the testing environment for the student.

WVEIS Data System:Accommodations Application

Data Sorting &Filtering

The WVS.326 Process:Accommodations Provision

and Monitoring

• The WVS.326 form is provided for all state and district assessments.• One form for each student receiving identified supports and

accommodations.• Electronic process which feeds accommodations research

information to support better student outcomes.• Accommodations data collected informs accommodations policies,

practice and research agendas.• The WVS.326 process provides feedback on accommodations

provision for student plans.• Documentation of the WVS. 326 form is available http://

wvde.state.wv.us/oaa/videos/WV_326/Documentation%20Procedures.zip

WVS.326• The WVS.326 form includes all

the accommodations • All pre-slugging and data entry

one side• Directions are on the back• Coding process includes data

for accommodation refusals and is used for invalidations of over and under accommodations.

• Information about WVS. 326 process is included in the Participation Guidelines.

Let’s Talk About CodesIf you mark Provided “Yes”- this means the student was given the identified accommodation.If you mark “No”- this means the student was not provided the identified accommodation. This is an accommodations breach UNLESS it is a:• Code 1-the student refused; or• Code 2-the accommodation is not allowed/applicable for

this testIf “no” is not marked with a code 1 or 2, then it constitutes an accommodations error that requires documentation. See the PG for specific details.

What to do about students who do not receive accommodations?

All incidents for failing to provide an accommodation requires written documentation.

• The CTC will review all accommodation breaches with principals/building level coordinators and report any findings to the Special Education, Title III and/or 504 director(s) and to the Office of Assessment and Accountability.

• If a student has been denied accommodations listed within the IEP, 504 or LEP plan, there has been a breach in the integrity/accuracy of test results. Therefore, the CTC or county special education director should contact the principal who must inform the parent/guardian of the testing administration breach and provide the following options:

– Test is invalidated and the student is retested using a breach form (if the testing window is still open). – Test is scored. – Test is invalidated and student receives a score of ―0.

If the second or third option is selected—there must be a signed written agreement between the parent/guardian and principal/building level coordinator. Copies of the agreement must be kept on file with the CTC/county special education director, Section 504 director, or Title III director.

WVS 326 State and District Accommodations Report

• Analyzed provision data for 2012 WESTEST 2 by content area• 7 reports were generated for each content area to summarize

(1) provision rate, (2) refusal rate, (3) not allowable rate, and (3) number of students over-accommodated– State, district, school-level reports – Intended uses of these data– Set specific targets for provision rates– Identify common errors to inform your training– Eliminate under- and over-accommodation

WVS. 326Accommodations Reporting

1. Count of Students with Accommodation

This report provides the total number of students with the accommodation listed on the front of the 326 form (this number serves as the denominator for all rate calculations in reports 2 – 6).

2. Provision Rate

This report provides the percentage of students with the accommodation for whom the accommodation was marked “provided.”

3. Not Allowed Rate

This report provides the percentage of students with the accommodation for whom the accommodation was marked “not provided” and the “not allowed” code was provided in the reason field.

4. Refusal Rate

This report provides the percentage of students with the accommodation for whom the accommodation was marked “not provided” and the “refused” code was provided in the reason field.

5. Not Provided (no reason indicated)

This report provides the percentage of students with the accommodation for whom the accommodation was marked “not provided” but no reason code was provided.

6. Not Provided (blank)

This report provides the percentage of students for whom no information was provided on whether or not the accommodation was provided.

7. Count of Students Over-accommodated

This report provides the number of students who do not have the accommodation listed in the pre-slug file, but for whom the accommodation was marked “provided.”

State Reports

District Reports

Lessons Learned During the Transition

• Collaboration is critical. • Review and update WVEIS codes and descriptions in all policy and practice documents.• The codes we use for a PNP to identify specific students’ needs for example, braille, large print,

etc. will correspond to the accommodations data map that must be provided to test vendors.• The Participation Guidelines must be updated and cover the parameters of the designated

systems such as TIDE and define the allowable universal tools designate supports and accommodations .

• OAA must provide training to districts, including CTC’s, Special Education Directors, 504 Coordinators, ELL Coordinators, principals/BLC, special education teachers, student assistance teams, teachers and test examiners.

• The LEP, 504, SAP Plan & the Online IEP system must be reviewed and incorporate available tools, designated supports and accommodations.

• Communicate plan changes to all impacted, including parents.• Student plans must be reviewed, amendments made or meetings held in order to address the

changes.• The WVS.326 form must be adjusted, a new program created for pre-slugging and reading the

form to provide monitoring data for accommodations

Melissa Gholsonmgholson@k12.wv.us

54

Beyond the PNP:Designing a Comprehensive State System for Systematically Monitoring Accommodations

Brian M. TouchetteDirector

Office of Assessment

55

State Profile• 3 counties

o 19 districtso 25 charterso ~220 schools

• 130,000 total students (15,000 w/ IEPs)• 80,000 students assessed (10,000 w/ IEPs)• DE transitioned from paper based assessment to

on-line assessment 2008-09 school year

56

On-line IEP• State mandated form• Accom. Form included• Form is printed for

student’s audit File• State can view any

student’s IEP and attached forms

• Universal design vs. accommodations

57

Accommodations Data-baseOn-line IEP

• Paper version of Accom. Form is entered

• Separates accoms. for each content area

• Often entered by office admin.• Functional vs. Non-functional

58

Functional vs Non-Functional Accommodations• Functional Accommodations

o These are provided within test delivery system to student during online assessment

o These include text-to-speech, font size (can be pre-set), print-on-request, color choices (available to all)

• Non-Functional Accommodationso These accommodations are provided to student by test

administratoro These include: human reader, external mouse (available to

all), special equipment

59

Universal Design vs. Accommodations

• Some universal design features are also included on Accommodation Form

• UD features still available even if IEP team did not select

• Way to ensure students with IEPs still received• UD features not counted as accommodation in

federal or state reporting

60

Accommodations Data-base

Accommodation Data Sent to

Vendor

On-line IEP

• Functional and Non-Functional are sent

• Daily upload• Accoms can change

during a students test (may take up to 72 hours to sync)

61

Review of Assessment Accommodations in DCAS TIDE • Review DCAS TIDE for

accommodationso Review

accommodations within TIDE

o Accommodation summary displayed next to each student name

62

Functional Assessment Accommodations in DCAS TIDE • Review individual student

accommodationso Review functional

accommodations within TIDEo Select VIEW button next to a

student name.

63

Non - Functional Assessment Accommodations • Non-Functional

Accommodationso Review non-functional

accommodations within TIDEo Select VIEW button next to a

student name. o This will display all of the

accommodations assigned to student.

64

Student Accommodations Review• Validate all accommodations for testing • Make sure all new IEP accommodations have been

modified or added 3 days before testing• Please provide a hard-copy of student

accommodations for Test Administrator to validate when in the test administration window

• Additional accommodation information can be found in the Test Administration Manual (TAM)

65

Accommodations Data-base

Test Session

Accommodation Data Sent to

Vendor

On-line IEP

• Mandatory PD • All accoms must be

provided, available and accessible during test

• Accoms appear on Test Administrator’s screen– Must verify both functional

and non-functional

66

Test Administration Session – Validate Accommodations• If student has accommodations, select the See/Edit

details button

67

Test Administration Session – Validate Accommodations

• Review Accommodations against Accommodations Form (paper version)

68

Accommodations Data-base

Accommodations Data Returned to

DOETest Session

Accommodation Data Sent to

Vendor

On-line IEP

•Provides accoms at time of test•Long string of information•Does not show if student actually used an accom or not

69

Accommodations Data-base

Accommodations Data Returned to

DOETest Session

Accommodation Data Sent to

VendorMonitoring

On-line IEP

70

Monitoring throughout the System• Test Security:

o From teachers, students and parentso From data forensics o From special education state complaints

• Primarily completed off-site• Random reviews of accommodations used during

testing vs what is in DE data base• Do not currently compare what is on IEP paper

version what is in the system (unless there is complaint)

71

Accommodations Data-base

Accommodations Data Returned to

DOETest Session

Accommodation Data Sent to

Vendor

On-line IEP

Future Change:•New IEP forms for accoms•Does student need additional practice or Preparation?

72

Accommodations Data-base

Accommodations Data Returned to

DOETest Session

Accommodation Data Sent to

Vendor

On-line IEP

Future Change:Approval of Read Aloud (passages) through IEP audit

73

Accommodations Data-base

Accommodations Data Returned to

DOETest Session

Accommodation Data Sent to

Vendor

On-line IEP

Future Change:Direct link: On-line IEP Accoms to DE Accoms Data-base

74

Accommodations Data-base

Accommodations Data Returned to

DOETest Session

Accommodation Data Sent to

Vendor

On-line IEP

Future Change:PD for validating accoms On-site monitoring

75

Accommodations Data-base

Accommodations Data Returned to

DOETest Session

Accommodation Data Sent to

Vendor

On-line IEP

Future Change:Accom data will be sent in way we can delineate accoms

76

Thank you

Carolyn LazarCarolyn.lazar@doe.k12.de.us

forBrian M. Touchette

Brian.touchette@doe.k12.de.us

+ Discussant

Jan Sheinker, Ed. D.jansheinker@gmail.com