National Center and State Collaborative

General Supervision Enhancement Grant(NCSC GSEG)

Kick-off Meeting Washington, DC

December 17, 2010The contents of this presentation were developed under a grant from the Department of Education (PR/Award #: H373X100002, Project Officer, Susan.Weigert@Ed.gov). However, the contents do not necessarily represent the policy of the Department of Education and no assumption of endorsement by the Federal government should be made.

National Center and State Collaborative

Building an assessment system based on research-based understanding of:

- technical quality of AA-AAS design- formative and interim uses of assessment data- summative assessments - academic curriculum and instruction for students

with significant cognitive disabilities- student learning characteristics and

communication- effective professional development

NCSC GSEG 04/19/23 2

NCSC PartnersCentersNational Center on

Educational Outcomes

National Center for the Improvement of Educational Assessment

University of KentuckyUniversity of North

Carolina-CharlotteedCount, LLC

StatesAlaska Arizona

Connecticut District of Columbia

Florida Georgia

Indiana Louisiana

Massachusetts Nevada

New York North Dakota

Pacific Assessment Consortium (PAC-6)

Pennsylvania

Rhode Island South Carolina

South Dakota Tennessee

WyomingNCSC GSEG 04/19/23 3

A Comprehensive Model

All partners share a commitment to the research-to-practice focus of the project and the development of a comprehensive model of curriculum, instruction, assessment, and supportive professional development.

NCSC GSEG 04/19/23 4

HISTORY AND CONTEXT OF THE PROJECT DESIGN

Overview

5

Shift to Inclusion• NCEO research on outcomes for ALL students

with disabilities leading to OSEP Technical Assistance Center (21 years)

• 1990s work in Kentucky and Maryland• Range of participation from 0 – 85%• Separate systems for assessment, separate

systems for curriculum, separate systems for instruction

• Very poor outcomes for students with disabilities, although range was wide

NCSC GSEG 04/19/23 6

Key Reports Along the Way• Testing, Teaching, and Learning – Elmore and

Rothman (1999) Committee on Title I Testing and Assessment – NRC– Theory of Action in SBR based 1994 ESEA

reauthorization: Set standards, build assessments, hold accountable = Increased achievement

– Evidence emerged– to get increased achievement, actually had to intervene on curriculum and instruction – Professional Development, Pre- and Inservice

– Quenemoen, Lehr, Thurlow, and Massanari (2001) Implications of theory underlying reform for students with disabilities

NCSC GSEG 04/19/23 7

Building Assessments – But Worrying About Curriculum and Instruction

• Knowing What Students Know – Pellegrino, Chudowsky, and Glaser (2001) Committee on Assessment – NRC

• New Hampshire Enhanced Assessment Initiative (NHEAI)

• National Alternate Assessment Center (NAAC)• Triangle off the triangle CIA/KWSK• Poking around the C and the I while working

on the A

NCSC GSEG 04/19/23 8

OBSERVATION INTERPRETATION

COGNITIONStudent PopulationAcademic contentTheory of Learning

Assessment SystemTest DevelopmentAdministration Scoring

ReportingAlignmentItem Analysis & DIFMeasurement errorScaling and Equating Standard Setting

VALIDITY EVALUATIONEmpirical evidenceTheory & logic (argument)Consequential features

The Assessment Triangle & Validity EvaluationMarion & Pellegrino (2006)

NCSC GSEG 04/19/23 9

Ten Challenges Identified by Partners and States

• College and career readiness • Learning progressions• Formative and interim uses of assessment data• Instruction and curriculum tools – concrete supports• Differences within the 1% population – communicative

competence 70-30? 70-15-15? Teacher capacity issues• Flexibility and standardization balance• Growth• Technology• Comparability• Costs

NCSC GSEG 04/19/23 10

Key Ideas for Building the Foundation

• Articulating CCR• Defining the construct• Instructional models – Principle of

Uncertainty; Least Dangerous Assumption• Communicative competence• Delivering PD, building capacity• Validity argument

NCSC GSEG 04/19/23 11

Theory of Action

Long-term goal: To ensure that students with significant cognitive disabilities

achieve increasingly higher academic outcomes and leave high school ready for post-secondary options.

A well-designed summative assessment alone is insufficient.

To achieve this goal, an AA-AAS system also requires:

Curricular & instructional frameworks Teacher resources and professional development

NCSC GSEG 04/19/23 12

Students respond to tasks/prompts

as intended

Students respond to tasks/prompts

as intended

Students get greater exposure to grade-

level academic curriculum

Students get greater exposure to grade-

level academic curriculum

Teachers have the resources and

supports necessary to administer a

standards-aligned AA-AAS

Teachers have the resources and

supports necessary to administer a

standards-aligned AA-AAS

AA-AAS scores appropriately reflect student knowledge

and skills

AA-AAS scores appropriately reflect student knowledge

and skills

The content assessed by the AA-AAS is appropriately

rigorous and aligned with grade level content

standards

The content assessed by the AA-AAS is appropriately

rigorous and aligned with grade level content

standards

Teachers provide instruction aligned

with the grade-level content standards and

academic expectations

assessed

Teachers provide instruction aligned

with the grade-level content standards and

academic expectations

assessed

The AA-AAS has been designed to allow

students to demonstrate their knowledge and skills in relation to prioritized common core standards

The AA-AAS has been designed to allow

students to demonstrate their knowledge and skills in relation to prioritized common core standards

AA-AAS scores provide

information that is useful for teachers

in building and maintaining

instruction aligned with academic expectations

AA-AAS scores provide

information that is useful for teachers

in building and maintaining

instruction aligned with academic expectations

Negative consequences are

minimized

Negative consequences are

minimized

Students with significant cognitive disabilities achieve increasingly higher

academic outcomes

Students with significant cognitive disabilities achieve increasingly higher

academic outcomes

Instructional material is aligned to

common core state standards

Teachers are given resources for and training on instruction in

academic material needed for college and

career readiness

Students with significant cognitive

disabilities leave high school ready

for college and careers

Students with significant cognitive

disabilities leave high school ready

for college and careers

Curricular & Instructional Frameworks

Summative assessment

design

Short term outcomes

AA-AAS scores provide

information that allows educators

and parents to track student

progress toward college and career

readiness

AA-AAS scores provide

information that allows educators

and parents to track student

progress toward college and career

readiness

Long term outcomes

Appropriate communication

methods/ resources are

recognized and made available to the student

and teacher

A wide range of communication methods are available for the AA-

AAS

A wide range of communication methods are available for the AA-

AAS

The appropriate students are

identified for AA-AAS

The appropriate students are

identified for AA-AAS

04/19/23 13

Validity Evaluation

• Determining that the C, I, and A strategies are effective, implemented as intended, and producing desired results.

NCSC GSEG 04/19/23 14

NCSC Organizational Structure

• States participate in one or more work groups• Each work group nominates one or more

states to serve on the management team• Work groups interact and collaborate as

appropriate• The management team and Technical

Advisory Committee oversee the work of all four groups

NCSC GSEG 04/19/23 15

Perie/Domaleski

Technical Issues

Quenemoen

PI / Proj. Director

Kearns

PD/Training

Forte

Validity

Evaluation

Work Group 1 Work Group 2

Partner Organizations

NCIEA UNCC UKY

Hess

Content

Wakeman

C&I

Towles-Reeves

Process Evaluation

edCount

RI

Work Group 3 Work Group 4

Project

Management

Team

NY

LAFLAK

SCNV

NDINGACT

Project

Management

Thurlow

PI/TAC Lead

TNSD WY

NCEO

Work Groups

Lead State Representative annually selected from each Work Group to serve on the

Project Management Team; each State committed to no less than 1 Work Group, but

can be involved in as many as they choose

AZ DC

PA

Derek Briggs

George Engelhard

Mike Kolen

Suzanne Lane

Jim Pellegrino

TAC

MA

PAC604/19/23 16

NCSC Work Group Structure

Management Team

Lead: NCEO andState

Representatives

Assessment Design Work Group

Lead: NCIEA and State Representatives

Assessment Design Work Group

Lead: NCIEA and State Representatives

Professional DevelopmentWork Group

Lead: UKY and State Representatives

Professional DevelopmentWork Group

Lead: UKY and State Representatives

Curriculum & Instruction Work Group

Lead: UNCC and State Representatives

Curriculum & Instruction Work Group

Lead: UNCC and State Representatives

Evaluation Work Group

Lead: edCount, LLC and State Representatives

Evaluation Work Group

Lead: edCount, LLC and State Representatives

NCSC GSEG 04/19/23 17

ASSESSMENT DESIGNWork Group 1

18

Assessment Design: Major Goals

• Articulate college- and career-readiness• Establish construct definitions• Design assessment frameworks• Develop/field-test assessment items and draft

PLDs• Establish technology platform• Develop reporting system• Establish operational assessment system

NCSC GSEG 04/19/23 19

Assessment Design: Key Ideas

• Assessment Triangle (Pellegrino, Chudowsky, & Glaser, 2001)

• College- and career-readiness for SSCD• Evidence-centered design • Balancing standardization and flexibility (Gong

& Marion, 2006)

NCSC GSEG 04/19/23 20

Technology (1)• Proposed development of a comprehensive system to

support instruction and assessment to include:– Facilitating summative assessment that is enhanced

by appropriate assistive technology – Providing support for formative assessment tools and

strategies, and supporting interim uses of assessment data

– Supporting professional development and providing instructional resources to include curriculum modules

– Enabling flexible, dynamic reporting of student performance

NCSC GSEG 04/19/23 21

Technology (2)

• Currently exploring partnership with DLM GSEG (Kansas) for technology solution– An Ad Hoc Technology Committee is currently

investigating the possibility of collaboration with DLM GSEG

– The committee will provide a recommendation by mid-January, 2011

04/19/23 NCSC GSEG 22

CURRICULUM & INSTRUCTIONWork Group 2

23

Curriculum & Instruction: Major Goals

• Validate the learning progressions frameworks and entry points

• Develop skill sequences within each learning progression• Develop generalizable skill sequences for each content

area• Pilot and validate formative assessments and interim

uses of assessment data• Develop content support for special ed teachers• Partner with WG 1 to develop test blueprint aligned to

the CCSS

NCSC GSEG 04/19/23 24

Curriculum & Instruction: Key Ideas

• Learning progressions• Big ideas/enduring understandings and

prioritization of content• Entry points• Alignment• Curricular modules

NCSC GSEG 04/19/23 25

PROFESSIONAL DEVELOPMENTWork Group 3

26

Professional Development: Major Goals

• Establish state PD communities of practice (COPs)

• Implement Common Core State Standards• Communication Triage K-8• Develop assessment administration and

assessment results trainings• Develop teacher/principal evaluation tools

NCSC GSEG 04/19/23 27

Professional Development: Key Ideas

• Communities of Practice• Scaling up use of CCSS-aligned academic

curriculum• Communication by Kindergarten/

Communication Triage• Technology and training• Teacher/principal effectiveness

NCSC GSEG 04/19/23 28

EVALUATIONWork Group 4

29

Evaluation: Major Goals

• Establish the project Theory of Action and Interpretive Argument

• Prioritize issues and develop study designs• Implement validity studies• Synthesize results and produce reports

NCSC GSEG 04/19/23 30

Evaluation: Key Ideas

• Argument-based approach (Kane, 2006)• Theory of Action• Validity evaluation and process evaluation• External evaluation

NCSC GSEG 04/19/23 31

State Commitments• Agree to the Theory of Action in principle and practice• Active participation in one or more topical area Work

Groups • Involvement of state stakeholders in development

processes (e.g., item review, standard-setting) • Active participation in pilot and field testing of all

components of the systems • Participation in validity and evaluative studies• Provision of communication and practice linkages to

existing RTTA funded consortia.

NCSC GSEG 32

Project Management• Management team coordinates and oversees work

groups• Each work group annually selects one or more states

to represent them on the management team• Ongoing process evaluation to identify challenges

and opportunities, bring to management team for action

• One-on-one quarterly state transition planning• TAC meetings• Other expert advisory roles

NCSC GSEG 04/19/23 33

Management Team Purpose

• Monitor and evaluate attainment of goals, objectives, and timelines

• Identify barriers and solutions to problems encountered by work groups or individual collaborative members

• Ensure that the research-to-practice efforts honor the contributions, insights, needs, and unique concerns of all collaborative members

04/19/23 NCSC GSEG 34

Management Team Process

• Meet via phone/web monthly, with one face-to-face annual meeting each year, in conjunction with a full project team/state annual face-to-face meeting

• Full project team/states will meet 2-4 times per year via distance technology as well

• Support cross-GSEG and RTTA project collaboration

04/19/23 NCSC GSEG 35

Developing a system of assessments supported by curriculum, instruction, and professional development to ensure that

students with significant cognitive disabilities achieve increasingly higher academic outcomes and leave high school

ready for post-secondary options.

For more information, contact Project Director Rachel Quenemoen at quene003@umn.edu or 612-708-6960. The contents of this presentation were developed under a grant from the Department of Education (PR/Award #: H373X100002, Project Officer, Susan.Weigert@Ed.gov). However, the contents do not necessarily represent

the policy of the Department of Education and no assumption of endorsement by the Federal government should be made.04/19/23 NCSC GSEG 36

NCSC GSEG