NHSAA MEETINGSMARTER BALANCED

ASSESSMENT

September 26, 2014

Scott J Mantie, Ph.D. Bureau of Accountability & Assessment

New Hampshire Department of Education

The Context and PerspectiveWhy assess?

For many reasons, but the most important are:

1. Improve teaching and learning2. Evaluate and improve programs3. Support accountability uses

In serving all three of these purposes, assessments should reflect intended learning targets, but can also serve as an important signal to what should be learned

Learning and Assessment

Meaningful assessment scores depend on tight linkages among learning targets and assessment design (ELA/Math Blueprints).

In other words, we want students to develop deep understanding of important knowledge and skills and we want to know if/how students are acquiring those skills/knowledge.

Assessing for Deep Understanding Students cannot develop deep understanding

unless they are provided opportunities on both learning and assessment tasks.

A major component of Next Generation Learning is the ability to solve novel problems—this requires deep understanding!

Assessment conveys what’s important to learn (a signal) as well as providing an opportunity to check on students’ understanding and evaluate achievement

How we got here…

The Assumption:

“Assessment must support ongoing improvements in instruction and promote meaningful learning experiences for students that lead to outcomes valued by all stakeholders.”

Creating our own…

Assessment Test Construction Assumptions:

“Apples to Apples” assessment (i.e. accommodations, glossaries, adaptive, number of items, etc.)

300 items/grade/content area for summative (4200 total)

150 items/grade/content area for interim (2100 total)

Estimated cost for fully fleshed out item: $2,000 Summative estimated cost: $8,400,00 Interim estimated cost: $4,200,00 Ongoing administrative yearly costs: up to

$50/student

How we got here…

SBAC Underlying Principles:

SBAC learning and assessment system grounded in Theory of Action

Research-supported instructional practice Technology-supported tools Innovative assessments State-of-the-art classroom support mechanisms Have the broadest access for the greatest number of

students Also, it is essential to have comprehensive and

detailed evidence to support the validity and reliability of SBAC assessment.

How We Got Here…

Based on Research:

Demonstrating evidence – an ongoing process

Materials: Standards for Educational and Psychological Testing (AERA, 1999) U.S. Department of Education’s Standards and Assessment Peer Review

Guidance (2009b) Program Evaluation Standards (Yarbrough, Shulha, Hopson & Caruthers,

2011) Guiding Principles for Evaluators (American Evaluation Association,

2004)

Evidence-based to determine what can be done to improve learning – this evolves operationally

How We Got Here…

SBAC: Evidence-Centered Design

Domain Analysis – content Domain Modeling – structure of components Conceptual Assessment Framework –

constructs, evidence needed and items needed to collect the evidence

Implementation – development of the assessment items or tasks using specifications

Delivery – delivery of the assessment

How We Got Here…

Reliability & Validity

The test development process of Smarter Balanced seeks to make explicit the claims that test users can make when interpreting student scores.

Detailed & Rigorous Framework Planning, assessing construction, delivery,

accessibility and outcomes Constant process of evaluating and modifying

Constant Test Development & Test Evaluation processes

Assessments in New Hampshire

Assessment in New Hampshire SBAC: THE statewide assessment for mathematics and English

language arts NECAP Science: THE statewide assessment for science NH ALPs Science: THE statewide alternate assessment for

science Dynamic Learning Maps (DLM): THE statewide alternate

assessment for mathematics and English language arts – pending G&C approval

Data Privacy: NO student identifiable information collected! NO non-academic questions are asked!

On-line, Adaptive Assessment vs. Paper/Pencil We want to take advantage of the engaging aspects that adaptive,

on-line tests can have for students.* Assessment results: 4-6 weeks

Emphasis: Removing the test as being the focus of the assessment

NH Assessment: Important Points

SBAC Operational Vendor Selected In conjunction with NEAC States (CT & VT) Approval through G&C in October Training, documentation and portion of reporting Interim Assessments (K12) & Digital Library ARE

included

NHDOE Assessment Communication Plan ALT information has begun to be disseminated SBAC materials are being developed (more in a

moment)

Practice & Training Tests, Scoring Guides, Documentation

See: http://sbac.portal.airast.org/

Emphasis: Removing the test as being the focus of the assessment

Student Data Privacy

NHDOE worked closely with legislators (esp. Rep. Neal Kurk) since 2005 to ensure that laws protect student data

Data is protected by state and federal laws: Federal law prohibits disclosure of student identifiable data,

even to the US ED State law guides NHDOE efforts:

Name and student data must be kept separate Student ID is a completely random number (SASID) Few at NH DOE can access student identifiable data to even assist

schools Violation of student data laws: $25,000/violation, Class b Felony

and involuntary termination of employment New law (189:68): prevents collection of parents name, criminal

records, etc. Increased transparency

Student Data Privacy

Data stored onsite at the NH DOE on secure servers

Smarter Balanced Consortium: States own the/their data. No data can be released without state permission

Smarter Balanced Assessment: Online assessment has even greater security (unique SBAC-ID – no name and no SASID)

Primary purpose of collecting data: aid schools in improving instruction

Online vs. Paper/Pencil

• Adaptive (online) adjusts to student vs. non-adaptive (p/p) fixed

• Online – greater security (p/p lost/misplaced paper, changes numerous hands, etc.)

• Online – faster results (p/p must wait until all test results are in)

• Online – Performance Tasks (p/p no performance tasks)• Online – more usability, accessibility and accommodations

(p/p fewer)• Language supports (glossaries in 10 languages &

dialects, full Spanish translations of math assessment)

• Deaf or hard of hearing receive tests in American Sign Language, signed by recorded human interpreters

• Refreshable Braille keyboards and real-time embossers allow blind students to receive online tests in Braille

• Accommodations uploaded (IEP & 504)

Online vs. Paper/Pencil

• Online – will assess a broader range of students because of the many adaptive and accessibility features

• Online resources

• P/P is only temporary and shortly must move to online

• Most other assessments are going online

• Students are accustom to electronic interactions – it is part of their current experience

Smarter Balanced: A Next Generation Assessment

Rigorous assessments of progress toward “college and career readiness”

Summative, Interim, Performance assessments & Digital Library

Summative & Performance Assessments: 3-8, 11, Interim & Digital Library: K-12

Common cut scores across all Consortium states Information about grade-level achievement and growth Valid, reliable, fair and comparable for all students

(students with “significant cognitive disabilities" will be assessed using another valid, reliable and fair assessment)

Administered online, using multiple measures Economy of scale: 19 states

Smarter Balanced: Interim

Interim to Provide valid, reliable and fair information about:

Student progress toward mastery of skills measured by summative assessment

Performance at content cluster level so teachers and administrators can track student progress throughout the year and adjust instruction accordingly

Individual/group (e.g., school/district) performance at the claim level to determine whether teaching and learning are on target

Teacher-moderated scoring of performance events as a professional development vehicle to enhance teacher capacity to evaluate student work aligned to standards

Student progress toward the mastery of skills across all students and subgroups of students

Smarter Balanced: Digital Library

Digital Library (AKA: Formative Assessment Resources) to Provide measurement tools and resources to:

Improving teaching and learning

Monitor student progress throughout the school year

Help teachers and other educators align instruction, curricula and assessment

Help teachers and other educators use the Summative, Performance and Interim assessments to improve instruction at the individual and classroom levels

Illustrate how teachers and other educators can use assessment data to engage students in monitoring their own learning

This will be made available to NH teachers/educators soon

Assessment Details and Reactions

Performance tasks and summative items delivered by adaptive engine, yielding more precise, valid, and timely measures of achievement and growth for all students to support improved decision making and a fair accountability system.

Pilot & Field Test Pro/Con: Generally, positive and productive assessment experience for all students, including those from under-represented groups, through the use of universal design, embedded digital accessibility tools and external accommodations supported by research and practice.

Some student anxious test takers

Positive reaction to assessment interface

Age appropriate items and questions?

Adaptive Assessment

Adjusts to student’s ability by basing difficulty of future questions on previous answers

More accurate measurement of student achievement – particularly for high and low performing students

Creates an individually tailored set of questions

Tested (mathematically) in the Field-Test

CAT implemented 2001, Supporting Research: A Framework for the Development of Computerized Adaptive Tests,

Nathan A. Thompson, Assessment Systems Corporation, and David J. Weiss, University of Minnesota

The Road Ahead for State Assessments, Rennie Center for Education Research & Policy, Policy Analysis for California Education (PACE)

22

Here is how the Smarter Balanced system fits into the school year

Scope, sequence and timing of interim assessments locally determined

School year

Fast, detailed and digital with year-on-year measurement.

Student resultsComputer-adaptive accurate, aligned to Common Core and built to promote great instruction

Performance Tasks

Computer Adaptive Assessment

Summative Assessments

Optional interim assessments

Optional interim assessments

Instruction

Teacher Development / NH Networks (Formative)

22

SBAC Issue Checklist:

Asking non-academic questions…

Student privacy… Validity & reliability of

assessment… Accessibility to more students… Timeliness of scores…

SBAC Issue Checklist:

Asking non-academic questions…

Student privacy… Validity & reliability of

assessment… Accessibility to more students… Timeliness of scores…

DONE!

The Assumption & Foundation

“Assessment must support ongoing improvements in instruction and promote meaningful learning experiences for students that lead to outcomes valued by all stakeholders.”

Testing Window

Window can begin: 66% instructional days: Grades 3-8 80% instructional days: Grade 11

Where States are at: ..\..\..\..\Documents\A_SBAC Materials\SBAC State Test Windows\2015_Test_Admin_Windows_20140814_Final.xls

Idea/Suggestion for NH (not final): Grades 3-4, weeks 1-3 Grades 5-6, weeks 4-6 Grades 7-8 & 11, weeks 7-9 Makeup, weeks 10-12 Your thoughts? (Accountability Task Force)

The Test: What We’ve Learned

Are You Ready? Questions?

Smarter Balanced Information and FAQs

SBAC Network: www.education.nh.gov/networks/

Smarter Balanced pages on NHEON.org www.nheon.org/oet/readiness/

Smarter Balanced Website: You will find the most current information here (www.smarterblanced.org)

Are You Ready? Questions?

Your school readiness questions: Technology Information and communication

technologies (ICT) program standards are equal to or beyond SBAC technology requirements (i.e. Technology Plans).

Planning room(s), schedule, installing browser(s), etc.

Practice/Experience test teachers & students

30

Visit: SmarterBalanced.orgfor current information

For more information…

603-271-3844

603-271-3844

Scott.Mantie@doe.nh.gov

Please contact Dr. Scott Mantie if you have any questions!

Information & Practice/Training Test Resources:

http://sbac.portal.airast.org/

Communication Planning – Assessment Transition

Stakeholder Advisory Group Assisting with the development of a

communications plan around standards, competencies and assessments

Preparing communication resource documents

FAQs; talking points; parent resources

Organizing and redesigning public information on the NH DOE website

Collecting and redesigning educator resources on the NH Network