state policies for science and mathematics assessment: … state assessment... · 2016. 10. 3. ·...

State Policies for Science and Mathematics Assessment:

Implementation of STEM K-12 Education Indicators

An Online Reporting System developed by NORC at the University of Chicago With support from the National Science Foundation

Year-One Results: October 2016

Developed under NSF Grant # 1544123

Rolf K. Blank, Principal Investigator

NORC at University of Chicago

Education and Child Development Department

[email protected]

mailto:[email protected]

NORC | State Policies for Science and Mathematics Education: Year One Results

1

A System for Reporting State Science and Mathematics Assessment Policies

NORC at the University of Chicago is developing and implementing an online reporting system

on state assessment policies with the goal of establishing one of the STEM K-12 education

indicators recommended by a National Research Council committee of leading science and

mathematics educators and researchers (Monitoring Progress toward Successful K-12 STEM

Education: A Nation Advancing? (NRC, 2013). This first-year report summarizes the progress of

implementation and initial findings from analysis of state policies across 20 states that

participated in the project in 2015-16 school year.

The NORC plan for developing and implementing the state assessment policies reporting system

has three key objectives:

1. With cooperation of the state departments of education, NORC will design and implement an online reporting instrument that will address the need for 50-state

information on the types of student assessments, alignment of assessments to content

standards, and state initiatives to develop approaches for science and math assessment.

2. Information will be collected across states using existing state websites and an online survey system to support cooperation with state education staff for annual updates on

policy changes and assessment initiatives.

3. NORC will design and implement a central web-based reporting resource that is accessible and useful to educators, leaders, and researchers, and NORC will provide

summary analysis of trends in assessment policies across the states.

The progress report is organized for access and use through the NORC project website. Each of

the main sections on assessment policies are reported through summary tables with state-by-state

information. This initial report demonstrates how information on state policies across the states

will be reported using the policy information for the 2015-16 school year from 20 states that

voluntarily participated in the first year of the project.

The selection and organization of assessment policy measures was based on results of a design

pilot study conducted in 2014-15 and through review and input from the project technical

advisers comprised of education researchers and science and mathematics education specialists.

Each of the sections of the first-year progress report provide a brief initial analysis of the cross-

state patterns in state assessment policies, and these initial results are reported with the

http://www.nap.edu/catalog.php?record_id=13509http://www.nap.edu/catalog.php?record_id=13509


2

understanding that more complete and meaningful analysis will follow in years two and three as

more states are participating in the system.

State Science and Mathematics Assessments: Which Assessments? Intended Uses?

What Types of Assessment Items and Tasks?

Assessment Administration and Reporting: What are State Policies?

Content of Mathematics and Science Assessments: What is the degree of alignment to

Standards?

Rationale for a STEM K-12 Indicator on State Assessment Policies

Summary: Year-One Results and Recommendations


3

State Science and Mathematics Assessments: Which Assessments? Intended Uses?

For the first year of the state assessment polices reporting system, the 2015-16 school year,

NORC requested information on the types of student assessments in mathematics and science

education that are funded and provided under state policy. Questions about state policies were

organized under two kinds of state policies for student assessments. First, states reported on

state-administered assessments that were required of most students in mathematics and science

(except students with disabilities taking alternate assessments). The information was reported

for grades 3-8 and high school assessments. Second, NORC requested information on

assessments that were supported by the state but were not required – the assessments were

developed as an option for local district or school decision on application. The state-supported

assessments might be assessments at the K-2 grade levels, diagnostic assessments, formative

assessments, benchmark assessments, or high school level assessments supported as an option,

including end of course, AP exams, or college or career readiness tests such as ACT, or SAT.

A key policy question for math assessments that was addressed with the NORC reporting system

was the role of consortia, i.e. whether assessments in mathematics were adopted by state policy

from a state assessment consortium (either Smarter Balanced or PARCC), or assessments were

adopted for the specific state. A second major question addressed by the NORC reporting

system was the intended uses of the assessment information, and several categories of uses were

outlined, based input from the project technical advisers including state education specialists.

The intended uses were reported for each of the state-administered and state-supported math and

science assessments.

Elementary & Middle Mathematics Assessments

The assessments listed in Table 3 for elementary and middle grades (grades 3-8) math show that

nine of the 20 states reporting for 2015-16 used assessments from the Smarter Balanced

Consortium, and two states used assessments in mathematics from the PARC Consortium. The

other nine states had elementary math assessments that were developed specifically for their

state. (Note: The NORC survey did not include information on assessment contractors by state.)

The format for reporting on state assessments and intended uses is shown below.


4

State Name of Assessments

Used for School

Accountability?

Used for Teacher

Accountability?

Used for Student

Accountability?

Used to Inform

Instruction/ Curriculum?

Arkansas ACT Aspire Yes Yes Yes Yes

Connecticut Connecticut Smarter Balanced Assessments

Yes No No Yes

District of Columbia

Partnership for Assessment of Readiness for College and Careers (PARCC)

Yes No No Yes

Delaware Smarter Mathematics (SBAC)

Yes Yes Missing Yes

Idaho

Idaho Standards Achievement Test (ISAT) by Smarter Balanced

No Yes No Yes

Kentucky

Kentucky Performance Rating for Educational Progress (K-PREP) (Vendor-Pearson)

Yes No No Yes

Stanford 10 (Norm Reference Test)

No No No No

Insert Table 3 followed by Examples of C/I uses in Table 4

Of the 20 states reporting on policies for 2015-16, 18 states reported an intended use of the

grades 3-8 math assessments for school accountability, 12 states reported they were to be used

for teacher accountability, four states reported a use for student accountability, and 16 states

reported the assessment results would inform curriculum and instruction in mathematics. State

education staff were asked to provide an example of how the math assessment results would be

used by educators in their state, and the most common use cited was that schools use the results

to make decisions about curriculum and how to adjust instrument to better meet needs.


5

Definitions: Intended Uses of Assessments in Mathematics and Science

1 Accountability

Student level – determine whether students advance to the next grade level, receive course credit,

or receive a diploma

Teacher level – determine how teachers are categorized into a teaching performance category or

whether teachers receive a bonus

School level – determine whether schools are meeting annual goals or are placed on an

improvement plan

2 Inform instruction or curriculum – provide feedback about instructional practice, guide

curriculum decisions, or determine appropriate instructional intervention for students

3 Student placement – determine which students are placed in a specific instructional program,

grade-level, or curriculum

4 College readiness – determine whether students are prepared for college level work

5 Career readiness -- determine whether students are prepared for jobs or careers

6 Graduation requirement – Passing assessment required for high school graduation (may be one

or more EOC assessments)

For grades 3-8, several states reported their agency had a role in supporting additional

assessments. For example, nine states reported they support interim benchmark assessments

(including states in Smarter Balanced consortia), several states provide diagnostic assessments

(e.g., MN, PA, WV), digital library was provided for use in classroom assessment (e.g., CT, DE),

and one state (Utah) provides a formative assessment online system for use by districts.

Table 5

High School Mathematics Assessments

Tables 6-8 provide lists of the state high school assessments in mathematics given in 2015-16.

The reporting on EOC assessments in Table 6 shows that eight of the 20 states have required end

of course assessments, primarily at the level of Algebra 1/ Math I/Integrated Math 1, and

Geometry/Math 2/ Integrated Math 2. Six of the states with EOC high school assessments

require passing the assessment as a graduation requirement. Seven of these states define college

and career readiness as an intended use, and only two states report that the assessment is

intended for use in student placement.

Table 7 reports the list of state high school Comprehensive assessments which cover multiple

math subject areas. Of the 20 reporting states, 16 have a policy requiring a Comprehensive HS


6

assessment in 2015-16. (Thus, four states provide a Math EOC assessments and a

Comprehensive HS assessment in math.) Only three states require passing their assessment for

graduation, 11 states define intended use for college or career readiness, and one state intends a

use in placement. The Table indicates the grade in which the Comprehensive assessment is

given (grade 11 primarily). Across the 16 states, seven are using the ACT, three the SAT, four

use the Smarter Balanced assessment, and two states require their own in-state assessment.

In Table 8 states list additional HS math assessments, and 10 states report at least one additional

assessment is supported for use at the local level or by students. The additional assessments

include college and career readiness options for assessment Kentucky in 12th grade (e.g., ACT

WorkKeys, ACT Compass, and KYOTE from University of KY), diagnostic assessments (PA),

interim benchmark assessments (Smarter Balanced, Utah), EOC assessments (NC, DE), and AP

and IB assessments in District of Columbia.

Insert Tables 6, 7, 8

K-2 Assessments

States were asked to report on state administration or support of assessments for students in

kindergarten, grade 1, or grade 2. Seven states reported one or more assessments that can be

used at one of these grades, although none are required for all students. State contacts may be

able to provide further information about the content of these assessments.

Insert Table 1

Elementary & Middle Science assessments

The state assessments listed in Table 11 for elementary science show that each of the 20 states

reporting for 2015-16 used a different science assessment instrument: one state (AR) used ACT

Aspire, one used Stanford 10 (KY), one state (VT) is part of the New England Common


7

Assessment program, and 17 states listed an assessment specific to their state. (Note: The

NORC survey did not include information on assessment contractors by state.)

State Name of Assessments

Used for School

Accountability?

Used for Teacher

Accountability?

Used for Student

Accountability?

Used to Inform

Instruction/ Curriculum?

Grades Assessment

Administered?

Arkansas ACT Aspire Yes Yes Yes Yes 3rd–8th

Connecticut Connecticut Mastery Test in Science

Missing Missing Missing Missing 5th and 8th

Delaware DCAS Science

Yes No No Yes 5th and 8th


DC Science Assessment

No No No Yes 5th and 8th

Idaho

Idaho Standards Achievement Test (ISAT) for Science

Yes Yes No Yes 5th and 7th

Also shown in the table, 16 states assess science currently at two grade levels, generally grade 4

and 7 or grade 5 and 8, while two states assessed science at each of the grades 3 through 8, one

assessed grades 3 through 7, and one grades 4 through 8.

Insert Table 11-12

Of the 20 states reporting elementary/middle (grades 3-8) science assessments, 14 states reported

the intended use of results for school accountability, 10 states reported intended use for teacher

accountability, three states intend to use science results for student accountability, and 14 states

intended the assessment results be used to inform curriculum and instruction in science.

Responses from state education specialists regarding examples of how data are used locally

focus mainly on informing decisions about curriculum and instruction and aligning instruction to

standards.

In four states, support was provided for additional science assessment options that are available

for local application, as reported in Table 12. Utah provides formative and diagnostic science

assessments, Pennsylvania has diagnostic assessments, and North Carolina provides final exams

for elementary science.


8

High School Science Assessments

Tables 13 - 16 provide lists of the state science high school assessments for 2015-16. The

assessment reporting is organized by end of course (EOC) assessments and Comprehensive

assessments. In Table 13 the reporting on EOC science HS assessments shows that 13 of the 20

states have end of course assessments, primarily at the level of Biology 1 and several states have

assessments for chemistry, physics, and technology/engineering, and earth science. Eight states

require passing at least one EOC high school assessment for course credit for graduation; and

several states require passing a combination of at least two science EOCs. Of the 13 states, 11

have school accountability as an intended use, eight use them in teacher accountability, and six

states define college and career readiness as intended uses. Examples of use of the assessment

data in curriculum and instruction are reported in Table 14 as well as state policies on taking and

re-taking the assessments for course credit.

Table 15 lists the state high school science Comprehensive assessments which cover multiple

science subject areas. Of the 20 reporting states, 13 have a policy requiring a Comprehensive HS

assessment in 2015-16. (Thus, six states provide a Science EOC assessment and a

Comprehensive HS assessment in science.) Only one state requires passing the assessment for

graduation, while 10 states define the intended use for college or career readiness and eight of

these states use ACT for science assessment. Six states report informing curriculum and

instruction as an intended use of the results, and these are states with a state-specific science

assessment or the NE Common Assessment. The information in Table 16 reports examples of

use with curriculum and instruction and state policies on taking or re-taking the comprehensive

science assessments.

What Types of Assessment Items and Tasks?

States were asked to report on the types of assessment items or tasks that comprise the

assessment in mathematics and science for one elementary grade and for one assessment in high

school (either comprehensive or EOC). The objective in reporting on types of items is to provide

an indication of the state policy toward use of multiple methods of assessment particularly tasks

with complexity that will increase the degree of alignment of the assessment to state content

standards for mathematics and science.


9

The participating states reported on their use of mathematics items/tasks developed by one of the

State Assessment Consortia –Smarter Balanced or PARCC. With the sign-off of most states

with the Common Core State Standards, and the initial involvement of many states with the

planning and development of mathematics assessments with one of the two state consortia,

interest is high in tracking the current assessment providers for states. Interest is also high

because states have changed their policies over the past years and decided to contract with other

developers of assessments in math. The technical advisers for the NORC project recommended

questions to track the degree of use of items/tasks developed through the state consortia. To

reduce burden, the reporting for 2015-16 focused on item/task design for two grade levels

(elementary—grade 5 math, grade 4 or 5 science, and high school – the most advanced

assessment completed by the most students). NORC and the technical advisers decided that

information on assessments at two levels could provide important information on policy

differences and trends among the states.

Table 19 reports on the states use of mathematics assessments from the Smarter Balanced or

PARCC consortia for 2015-16 assessments. Half of the 20 participating states used assessments

at the elementary or high school level. Other states provided comments on the development of

math assessment items/tasks and vendors contracted.

Insert T 19


10

Table 19. Development of State Mathematics Assessments and Use of State Consortia Items/Tasks

State

Use SBAC or PARCC

Percentage of Items

Use SBAC or PARCC

Percentage of Items

Comments Grades 3–8 3–8

Assessment High

School HS

Assessment

Arkansas No No

Connecticut SBAC 100% No The state uses the SAT as the High School Mathematics assessment.


PARCC 100% PARCC 100%

Delaware SBAC 100% SBAC

Idaho SBAC 100% SBAC 100%

Kentucky No No

Louisiana PARCC 47–49% varied by grade level

No The percentage of PARCC items used in 2015-16 forms was legislatively mandated to be no more than 49.9%.

Massachusetts PARCC 75% students took PARCC

No Districts selected PARCC or MCAS assessments for grades 3-8

Minnesota No No

Michigan SBAC 90% No High School is SAT Only.

North Carolina No No

Oklahoma No No

Pennsylvania No No

Tennessee No No Items were developed to TN state specific standards, and were licensed from SAGE, Kansas performance tasks, and a few items from the AIR bank.

Utah No No SAGE assessment was developed under a contract with AIR, including a pilot with teachers and administrators.

Vermont SBAC 100% SBAC 100%

Washington SBAC 100% SBAC 100%

Wisconsin No No The Forward math items for grades 3–8 designed by Data Recognition Corp. ACT, Aspire, and Work Keys used for high school.

West Virginia SBAC 100% SBAC 100%

Notes: For the state administered grades 3–8 Math assessment, does the assessment use SBAC or PARCC items? and, if so, Indicate percentage that are SBAC items and PARCC items. For high school Math, same questions.


11

Elementary Mathematics Items/tasks

The data in Table 20 report the results on percentage of assessment points for grade 5

mathematics by type of item or task. The technical advisers and feedback from states from the

pilot study produced a list of 11 categories of assessment item or task types. One change from

the pilot study was to provide more specific categories describing the task or item, rather than

general categories such as performance task, technology enhanced, or select response. The

reporting categories for types of math items or tasks were the following:

Multiple choice, single correct response Multiple choice, multiple correct responses

Matching tables

Drag and drop

Table Fill In

Gridded response

Graphing

Equation/Numeric

Hot Spot

Short answer

Extended response

Across the states, three of the 20 reporting states (KY, MS, OK) reported using only multiple

choice, single answer items. Two states used two types of assessment items, and nine states used

four or more types of assessment items or tasks. Five states reported that the assessment

item/task types could not be tabulated from the state assessment blueprint or framework, or the

assessment used a computer-adaptive platform (Utah). The states using Smarter Balanced

items/tasks at grade 5 (seven states) and Minnesota used computer-adaptive assessment

platforms but were able to provide estimates of the range of assessment points across students for

each category of item type or task. The reported policies on types of grade 5 math items and

tasks indicate that the states participating in the two assessment consortia are assessing students

with multiple methods and wide range of item and task designs, and thus these assessments

address a broader range of mathematics knowledge and skills in applying knowledge than other

states’ assessments.


12

Insert table 20

High School Mathematics Items/tasks

The state reporting on types of items/tasks shown in Table 21 indicates a divergence among the

states in how high school assessments are constructed. Six states reported that all high school

assessment items were multiple-choice, single response items, and five of these states had

adopted the ACT as the required high school assessment. Assessments in seven states include

four or more types of items/tasks, and six of these seven states were members of the assessment

consortia. Two states used two types of assessment items (using SAT as the high school

assessment). At the high school level, four states reported that the assessment item/task types

could not be tabulated from the state assessment blueprint or framework, or the assessment used

a computer-adaptive platform (Utah). Similar to the grade 5 assessments, the states using

Smarter Balanced high school items/tasks and Minnesota used computer-adaptive assessment

platforms, and they were able to provide estimates of the range of assessment points across

students for each category of item type or task. The reported policies on high school

assessments’ types of items/tasks indicate that the states participating in the two assessment

consortia are assessing students with multiple methods and a range of item and task designs, and

thus these assessments address a broader range of mathematics knowledge and applications than

other state assessments.

Insert T 21

Table 22 lists the comments about the state Mathematics assessments provided by state

specialists. The comments include explanation of the use of estimated ranges by category for the

computer-based adaptive assessments, and the development of items for the assessments.

Elementary Science Items/tasks

As with mathematics, the types of reporting categories for types of science items and tasks were

expanded following the results from the pilot study. A set of 12 categories were recommended

for science assessment. For example, the performance tasks category was expanded to three

categories—interactive computer tasks, simulations, and hands-on tasks. describing the task or


13

item, rather than general categories such as performance task, technology enhanced, or select

response. The science categories are the following:

Multiple choice, single correct response Multiple choice, multiple correct responses

Matching tables

Drag and drop Fill In blank Hot spot Short answer Extended response Interactive computer tasks Simulations Hands-on tasks

Gridded response

The types of science assessment items or tasks used across the 20 states (Table 23) were

predominantly multiple-choice, single-answer items in 2015-16. In eight states, all (100

percent) of the points on the state assessment for grade 4 or 5 were based on multiple-choice,

single answer items, and two states had 95 percent of points from this item type. Four states

assessments incorporated three or more types of science items/tasks in the elementary assessment

(DC, MN, WA), and three states included extended response science items (CT, DC, MA) that

comprised a significant portion of the assessment. Five states did not report on the types of

science items/tasks.

Insert T 23


14

High School Science Items/tasks

The state Science assessment items/tasks for 2015-16 were primarily multiple choice, single-

answer items, with eight states assessment comprised of only this type of item. Nine states

reported on Biology 1 EOC assessment as the highest level science course and assessment

completed by most students. Only three states used more than three types of items or tasks, and

three states had extended response items. Two states (DC, MN) were using technology enhanced

items through the computer. Minnesota used hot-spot and simulation tasks in the high school

assessment. One state (MI) reported that the state pilot tested-based technology enhanced items

in 2015-16. Six states did not report on the types of high school science items/tasks.

Insert T 23 a

Assessment Administration and Reporting Policies

Computer Technology in Administering Assessment

Increasingly, states are moving towards computer technology to administer student assessments.

[In mathematics, five states use a computer-adaptive format. In science, only one state uses

computer-adaptive format. Only two of the pilot states administer the math assessments solely in

a paper-and-pencil format, and one state administers science in paper and pencil format. Three

states administer assessments with computer technology and two of these states also offer the

paper-and-pencil format.

[results in preparation: One question not resolved by the pilot study was the extent to which

methods of administration are actually used, since in some states both paper-pencil and

computer-based platforms are in place. It would be useful to track the percentage of students

assessed with each method.]


15

Dates for Administration

In 2015-16 a majority of the 20 participating states administered assessments close to the end of

the school year (e.g., late May). One state in the pilot study conducting their assessments in

March. Policies on timing are important in relation to the total time for teaching and learning

prior to the assessments. The information should be reported and tracked over time by state.

Guidance on time for completing assessment

NORC collected information on state policies regarding time allowed to students for math and

science assessments. Most of the 20 states provided guidance about expected time for

completion, however only a few states had state requirements limiting time for assessments. A

revised version should address the question of what state guidance is provided to districts and

schools with respect to the expected time needed for most students to complete the assessment.

Use of Calculators in Assessment

State policies vary widely on the extent to which calculators are allowed for use in mathematics

assessments. The results of reporting on calculator use in math assessment for 2015-16 in Table

25 show that some states do not allow calculators use in the elementary assessments (e.g.,

Smarter Balanced states), while other states allow calculators on a portion of the assessments,

and six states have no policy at the elementary level. At the middle level, a majority of states

allow calculators use on 40 to 60 percent of the items. In high school math assessments, eight

states allow calculators on the entire math assessment, and nine states allow the calculator on

about half of the state assessment.

In the 2015-16 reporting, calculator use in science assessment was added to the policies survey.

The reporting in Table 24 shows that many states do not have a policy on use of calculators in

science assessments (shown as NR). A majority of states allow use of calculators in science at

each grade level, and in the states with computer-based assessments (e.g., ID, UT) the tool is

provided when it is suggested for use on specific items.

Insert T 25, T 24


16

Release of Assessment Items/Tasks

One method of providing educators with additional information and professional development

regarding assessments is to release assessment items or tasks. In 2015-16, 8 states released a

portion of science items/tasks from a prior year. The results (Table 27) show that states that

release items generally provide a small percentage of the actual test items, and some states share

similar example items. Massachusetts stands out by releasing 50 percent of the grades 3-8 items,

and 100 percent of high school science and engineering/technology EOC assessment items. Nine

states reported a website that is provided for teachers and educators to access example student

assessment items.

Insert Table 27

Mathematics items were released in only six of 20 states and a portion of the assessment

items/tasks were made available to the public (Table 26). Nine states provided example items on

a website designed for teachers and educators.

Timing of Reporting on Assessment Results

The participating states reported on the time period when math and science assessment results

are reported for three different purposes: 1) use of data by teachers, 2) review by students and

parents, and 3) school accountability. The month when the assessment results are reported for

these purposes are shown in Table 28. The amount of time between administration of tests to

students and the scoring, analysis, and reporting of results is one of the primary reasons for state

policies to favor computer-based assessments and use of computers in scoring and tabulation of

results. The use of assessment scores in school accountability has become another major state

policy issue and particularly in providing the status of schools meeting performance goals prior

to the start of the next school year. The results show that many states are providing quick

turnaround of scores for students and parents following spring testing, and the school

accountability reporting is provided in early fall for most states.

Insert T 28


17

Content of Mathematics and Science Assessments: What is the degree of

alignment to Standards?

The Math-Science Reporting system is being supported by NSF in relation to the recent support

for improved STEM K-12 education indicators. The series of EAGER grants from NSF were

intended to support design and research for these indicators, and NORC completed a design

study for a state assessment policy indicator and it was awarded in light of the strong interest in

how states are responding with assessments that would be consistent with the Common Core

State Standards in mathematics and science assessments that respond to the Next Generation

Science Standards.

For the implementation of an assessment reporting system for 2015-16, two strategic decisions

were made by NORC in the instrument design with the Technical Advisers panel. First, we

decided it is important to identify the official policy position of each state concerning the CCSS-

M, and the relationship of the state standards to CCSS-M. Then, states would be asked several

questions regarding assessment design and development that would provide indicators of the

degree of alignment to mathematics assessments to a common set of mathematics content topics

at the elementary and high school levels. Second, for science, we decided to similarly track the

official state policy position concerning the Next Generation Science Standards (NGSS) and the

NRC Framework for K-12 Science Education –that is, whether state science standards have been

developed or revised in relation to these documents. Then, a series of questions would be given

to SEA contacts to determine whether science assessments are being developed in response to

the key themes of these documents, particularly since 2013 when the NGSS was released by the

consortium of states developing the standards.


18

Mathematics Assessment Content

The question of state policy on mathematics standards and the Common Core Standards is

addressed with the results from 20 participating states in Table 19. Eleven of the states reported

that their state standards are the same as the CCSS-M, and seven states made minor revisions in

CCSS-M for their state math standards. Two states (MN, OK) developed standards separately

from the CCSS-M.

Insert T 29

Mathematics Assessment Content Topics, Grade 5

To analyze math content in assessments across states, states were asked to report on the

percentage of the grade 5 assessment that focuses on each of six topics, and the selected topics

are reporting are consistent with content recommended in the Common Core Standards. These

topics provide a way to provide comparable data across states. Additionally, states were asked to

report on one additional topic for grade 5 assessment emphasized in their state. Grade 5 was

recommended by the expert panel because it is a key transition grade between math instruction in

the elementary grades K–4 and middle grades mathematics instruction. Example data are below.

State Name of Assessment

Numerical Expression

Add/sub Fractions

Multi/div Fractions

Multi-digit

Volume Add/ mult.

Graph 1st

Quad Data

Distrib. Add.

Topic**

% points % points % points %

points %

points %

points %

points %

points

Connecticut CT Smarter Balanced Assessments

0-15 5 to 20 5 to 25 5 to 20 5 to 20 5 to 20 0 5 to 20


PARCC 18-23 11 to 38 17-44 12 to 36 8 to 29 5 2

Delaware Smarter Balanced

0-15 5 to 20 5 to 25 5 to 20 5 to 20 5 to 20 0 5 to 20

Idaho ID Standards Achievement by SBAC

0-15 5 to 20 10 to 25 5 to 20 5 to 20 5 to 20 0 5 to 20

Kentucky K-PREP (Vendor-Pearson)

15 5 2 13 12 5 0 30

Louisiana LEAP 3 8 6 8 6 3 2 8

Minnesota

MN Comprehensive Assessments, Series III

2 to 15 6 to 15 0 24-50 2 to 14 2 to 9 14-17 7 to 10


19

Insert Table 30 – 31 for all 20 states

The percentage of grade 5 assessment points reported show that the six topics are being assessed

across the 20 states, and that percentage of the assessment varies widely for each topic. For

example, the topic “Write and interpret numerical expressions” varies from 0-15 percent to 26

percent of the grade 5 points. The percentages are a range for about half the states. Due to the

adaptive test design in many of the reporting states, the percentages are reported as a range since

students see different combinations of assessment items and topics based on their performance.

Across the six topics, a total of 50 to 80 percent of the grade 5 points are included. The

additional topic included for grade 5 included early geometry concepts, measurement, and real-

world applications.

Levels of Cognitive Demand, Grade 5 Math

Analysis of content alignment of standards and state assessments has typically included both

content topics and cognitive demand expectations (or levels of complexity). Many states refer to

the “depth of knowledge (DOK)” levels. Based on the design pilot study, the states were asked to

report on what cognitive levels are used in developing the assessments and whether the levels are

used in reporting assessment results. The information was collected for the grade 5 assessments.

Insert T 32

The policies reported by 20 states show that all of the states use some levels for the development

of math assessments. Many states plan their assessments with three cognitive levels. Less than

half indicate that the cognitive levels are used in reporting assessment results. Comments provide

further information about how levels are used and weblinks to the state definitions.

Elementary Mathematics Progression of Content Topics

Consistent with providing indicators of mathematics assessment content, states reported on the

grades when topics that are specified in the CCSS-M are first assessed. This indicator addresses

the issue of content progression as taught through the elementary grades, and the indicator can

compare the progressions in assessments used across the states. Results from seven of the states

across the elementary topics are shown below (see Table 33).


20

State/Topics* Fractions Word

Problem Multiply

Fractions Volume Divide

Fractions Proportions

Connecticut 3 3 5 5 6 7

Delaware 3 3 5 5 6 7


3 3 4 5 6 7

Idaho 3 3 4 5 6 7

Kentucky 3 3 4 5 6 7

Louisiana 3 3 5 5 6 6

Mass. 3 3 5 5 6 6

Across all of the 20 states (Table 33), the responses show high consistency on assessing three of

the topics—understanding fractions, word problems with multiplication/division, and dividing

fractions. The grades differ slightly for multiplying fractions (grade 4 to grade 6) and using

proportions (mostly grade 7). Overall, the reporting on elementary topics assessed between

grades 3 and 7 show a high degree of consistency across the states.

Content of High School Mathematics Assessments

The Technical Advisory panel recommended that NORC request assessment content information

for the mathematics assessments that most students take in the most advanced assessment taken

by most students (often about grade 11). The reporting involved two steps: first, what

percentage of high school students take each of the assessments offered—either a

Comprehensive assessment or an EOC assessment, and the state policy on a minimum score

required was also reported. Second, for the Comprehensive Assessments, states were asked to

report on the percentage of the assessment points that are addressed by each of seven topics. One

additional topic was requested and the percentage covering that topic. The topics for reporting

were selected from the recommended mathematics content in the high school mathematics

Common Core Standards.

The results from the first question on what assessments students take in high school in Table 34

show that a majority of the 20 states now require a Comprehensive Assessment in math, and

more than 95 percent of students take the assessment in the typical state. Eight of the reporting

states require students to complete an EOC assessment, with five requiring Algebra 1

assessment, one requiring Algebra 2, and two requiring Geometry or Secondary math II

assessment. Only six states require a minimum score on the math assessment be attained for


21

students to be able to graduate from high school. These assessments have replaced the basic

skills or minimum competency tests required formerly by many states.

Eleven states reported on the content topics by percentage of assessment points in the

Comprehensive High School Math Assessment used by their state in 2015-16. The content

topics analyzed are:

a) Applying trigonometry to solve problems

b) Summarizing & interpreting data and making inferences

c) Transformational geometry

d) Setting up and solving systems of equations and inequalities

e) Building, interpreting, and analyzing functions in different representations

f) Interpreting and working with structure in equations and expressions

g) Working with complex numbers.

The percentages reported by 11 states (Table 35) show that the Smarter Balanced

Comprehensive assessment and the ACT were used by a majority of the states, with two states

reporting on the SAT and one state (MN) reporting on a specific state assessment. The

percentage of the assessment points differ significantly between these ACT and Smarter

Balanced comprehensive high school assessments on three topics—Summarizing and

interpreting data, Transformational geometry, and Working with complex numbers. The

additional high school math topics assessed by the ACT (Table 36) are Geometry and Pre-

algebra. The additional topic reported for the Smarter Balanced assessment is “Understand the

concept of a function and use function notation,” and from 5 to 20 points of the total assessment

are covered by this topic.

On reviewing the reporting of assessment content topics, the Technical Advisors recommended

that in the next version of the policies reporting instrument, further information be collected on

the assessments being reported. The end of course assessments in mathematics are not given for

a specific grade, but rather at completion of a high school course. They recommended that states

report on content of the most advanced high school math assessment that most students take

before graduating, either a comprehensive assessment (e.g., the HS assessment offered by the


22

Smarter Balanced consortium) or the highest level end-of-course assessment most students take

in high school.

Math—Formative Assessments

Science Assessment Content

The reporting on science assessment content first examines the state policy position concerning

the NGSS and the NRC Framework, and if standards are being revised or updated in relation to

these documents. From input of the advisory panel, reporting on specific content alignment

indicators will wait until more states have made a transition. For 2015-16 reporting, states were

asked to report on the timing and nature of the transition to new assessments.

The results reported on state policy position and standards (T38) indicate that 11 of the 20 states

adopted new science standards since 2013. Among the 11 states:

Seven states adopted the NGSS standards as a whole, without changes

Three states have standards that are “similar” to NGSS

One state made revisions to NGSS before adopting

Additionally, seven states have a plan and timeline for adopting new state science standards

between 2016 and 2019. The table provides comments from states reporting on their adoption

process as well as weblinks to the state standards.

State Planning for New State Science Assessments

In responses to questions on state plans for New Science Assessments, a series of questions

addressed the timing, content, and organization of development of new assessments. First,

(Table 39) states reported on the whether there is a plan for new science assessments, the target

date, states in collaboration, and comments about the process. The modal year for implementing

new science assessments is 2018-19 for these 20 states. Only a few states reported on

collaborations with other states.

Next, (Table 40) states reported on the question of planning for assessing three dimensions of

science content and strategies for planning content of new assessments. Several states reported

they are planning assessment clusters or bundles in order to assess across the three dimensions.

Fourteen states reported that their plan calls for assessing students on the three dimensions of


23

content, with a majority indicating that the dimensions will be assessed together. Following are

examples of state reporting on their plans for new assessment design and use of sampling to

assess the three dimensions.

State

State Approach to Design and Use of Sampling

Connecticut Some use of assessment item clusters. Sampling uncertain at this time.


Outlined in blueprint

Delaware Our approach is modeled very closely with the recommendations of the BOTA report. Beyond that we are in developmental stages of our plan and not yet ready to share further information.

Idaho Currently still in the planning stage

Kentucky Bundled PEs around a phenomenon will be used to develop a cluster, in which each item will assess as least 2 dimensions.

Louisiana Unable to answer until we have revised standards

Minnesota Currently researching future approach. Test specifications after new standards adopted.

Michigan

Working with the topic bundles as the PE bundles for assessment and using the CCCs, DCIs, and SEPs, as elements that can be used in any combination to address the phenomenon. Matrix sampling will allow multiple Topic bundles to be assessed over the state.

States also reported on the planned organization of new assessments (Table 41) including the

grades to be assessed, use of grade span or grade specific assessments, and high school science

assessment plans. About half the states were still making decisions about the organization of the

grades and structure of the new assessments as of reporting in June 2016.

Current Reporting of Science Assessment Results

An additional indicator of content of science assessment is how results are reported. States were

asked to report on sub-scores and achievement levels for 2015-16 science assessments. The

results (T42) show that 11 states gave categories for the science sub-scores with varying number

of sub-scores. Additionally, 13 states reported that achievement levels are defined and reported

for the science assessments, and the websites for the levels are provided. Examples of science

reporting categories are shown below.


24

State Report

Sub-scores Science Categories

Report Achiev. Levels

Arkansas No

Connecticut Yes

Life Science; Physical Science; Earth Science; Content Knowledge; Scientific Inquiry, Literacy and Numeracy

Yes


New Based on domains Not yet

Idaho Yes Nature of science, life science, physical science, earth science, and technology

Yes

Kentucky No

Louisiana Yes Life Science, Physical Science, Earth and Space Science, Science in the Environment, Science as Inquiry

Yes

Massachusetts Yes Earth and Space, Life, Physical, and Technology/Engineering

Yes

Minnesota Yes Nature of Science and Engineering, Physical Science, Earth Science, Life Science

Yes

Michigan Yes Domain level (Life Science, Physical Science, Earth Science, Inquiry)

Yes

Formative Science Assessments

State leadership with use of classroom-focused formative assessments is an additional measure

of state assessment activity. Nine of the 20 reporting states (Table 43) described a current

initiative or plan for supporting use of formative assessments or professional development with

teachers on formative assessment.


25

Rationale for a STEM K-12 Indicator on State Assessment Policies

In 2015, NORC at the University of Chicago (NORC) submitted a proposal to the National

Science Foundation program on Promoting Research and Innovation in Methodologies for

Evaluation (PRIME) of the Division of Research on Learning. A three-year grant was awarded

to NORC for the project, STEM Indicators: Implementing a Model for Reporting and

Research on State Assessment Policies for K-12 Science and Mathematics Education

(#1544123).

The goal of the project is to develop and implement an online system for reporting state

assessment policies across states, and a new grant has been funded to extend the pilot work into a

three-year implementation project. The reporting system will inform mathematics and science

educators and leaders at state, local, and national levels about the status of state assessment

policies across the nation and provide comparable policy-relevant information that can be

tracked as changes are implemented over time. NORC is implementing the system of reporting

through an online, web-based instrument that provides accessible information across the states

and regular updates.

The STEM Indicator on state assessment policies reflects the strong interest in advancing the

quality of student assessments in mathematics and science education. Research and analysis of

the content of state achievement tests has yielded concerns about their academic rigor, as the

items tend to assess lower-level cognitive skills (such as recall, recognition, and application of

procedures) as opposed to higher-level cognitive skills (such as analysis, evaluation, and

synthesis of ideas) (Darling-Hammond, et al., 2013; Resnick, et al., 2003; Webb, 2002). Even in

the NCLB era, with its emphasis on accountability, researchers have found state tests generally

assess lower-level cognitive skills. In a recent study of tests in 17 states that were selected

because the tests were reputed to be more cognitively demanding, Yuan and Le (2012) found that

fewer than 2 percent of mathematics items assessed a higher level of cognitive demand. Using a

different methodology on 19 state tests, Polikoff, Porter, and Smithson (2011) found that only 7

percent of mathematics items required students to use higher-order cognitive skills, and fully 80

percent of mathematics items assessed memorization, recall, and use of routine procedures.


26

The lack of test items assessing more advanced analysis and knowledge has resulted in reforms

to state content standards and assessments, with the most prominent reform being the

development of the Common Core State Standards (CCSS-M) for mathematics and the Next

Generation Science Standards (NGSS) for science (NGA/CCSSO, 2010; Achieve & Lead States,

2013). The CCSS-M and NGSS have the potential to improve the degree to which deeper

learning is assessed through state achievement tests. The CCSS-M establishes a single set of

educational standards for kindergarten through 12th grade that identifies the concepts and

knowledge that students should acquire to show that they have attained the skills necessary for

college and career success (Yuan & Le, 2014). Similarly, the NGSS initiative focuses on

coherency and progression of core conceptual science skills from early grades through high

school, with an emphasis on preparing students for college, career, and citizenship. A recent

study report and recommendations from the National Research Council is informing the

development of new designs and systems for science education in the states to support the goals

of the NGSS (NRC, 2013).

A key issue analyzed in the design study was how to track states use of innovative features of the

mathematics assessments being provided through the consortia--Smarter Balanced (SB) and

PARCC, including performance tasks (with multiple steps and explanation of work), balance of

items across the different levels of depth of knowledge, use of computer-based testing to

improve turnaround time for scoring and reporting, measures of learning content and skills, use

of adaptive testing to measure a full range of student knowledge, benchmark testing to track

progress through the year, and a digital assessment library for formative classroom assessment.

To work toward an indicator of state assessments alignment to standards especially relative to

content coverage and levels of cognitive demand, NORC and the advisers considered the Criteria

for High-Quality Assessments defined by the chief state school officers (CCSSO, 2014).

Reporting of state policies and trends over time is equally important for science education as part

of the proposed state policies tracking system because of the recent shifts in science education

standards and the strong interest in new forms of student assessment to match the direction of the

standards. Some states are following the recommendations of the NRC committee on NGSS

science assessments (NRC, 2014), including moving toward a system of science assessment with


27

varied methods, assessing the three dimensions of science instruction, and maintaining annual

reporting of student progress.

Results of Design Pilot Study

The NORC plan for a design and pilot study was intended to test a model for collecting

information on states’ policies for state-administered student assessments to meet several needs.

The pilot study (2014–15 school year) tested a design for collection of state policies information

from three sources—state websites, survey of state assessment staff (focusing on design and

methods of reporting), and survey of state content specialists in math and science (focusing on

content and alignment of assessments and standards). The design options and issues of focus in

the pilot study are summarized in a paper, developed from the recommendations of a project

expert panel (NORC, 2015).

The design study and pilot project addressed two questions about state assessment policies for

mathematics and science and development of a system for tracking and reporting across the

states:

1) What core information on state assessment programs and policies should be collected and reported across states, i.e., what information on types of assessments and characteristics of assessment

programs are important to have available in a 50-state report or online resource?

2) What is an effective method for reporting comparable information on the extent to which student assessments in math and science are aligned to state-adopted content standards, and what is the

relationship to the Common Core State Standards-Mathematics and Next Generation Science

Standards?

The design pilot study was carried out with support and participation by staff of ten state

departments of education. The results of the design study were reported by NORC to NSF and

the report is available (http://www.norc.org/Research/Projects/Pages/developing-a-system-for-

tracking-state-assessment-policies-in-science-and-mathematics-education.aspx).

http://www.norc.org/Research/Projects/Pages/developing-a-system-for-tracking-state-assessment-policies-in-science-and-mathematics-education.aspxhttp://www.norc.org/Research/Projects/Pages/developing-a-system-for-tracking-state-assessment-policies-in-science-and-mathematics-education.aspx


28

Selection of State Assessment Policies for the Reporting System

The project Technical Advisers (see Exhibit 1) reviewed the findings of the pilot study report

and the research on needs for information on assessment policies in math and science education

to recommend implementation of a reporting system beginning with the 2015-16 school year. A

key decision in planning for a state assessment policies indicator is the degree to which reporting

and analysis across states should be comparable and quantifiable. For purposes of cross-state

reporting, a common metric and organization of information in tabular format facilitates

comparisons; however, with greater depth of information, key differences in state policies and

design of state assessments could be highlighted and shared among users in different states. It

was decided that some policies related to assessments should not be included in this reporting

system. For example, school accountability has been the primary focus of uses of state

assessments recently, however, the advisers recommended that the reporting system should

include the different types of intended uses of state assessments in mathematics and science, and

that this reporting system should forego details about state accountability plans and the formulas

by which schools, teachers, or students are determined to meet expected performance levels.

A) Characteristics of State Assessments in Science and Mathematics

The panel recommended collecting and reporting information for the following categories of

state assessment policies:

■ List of types of state assessments by grade level

■ Intended uses of assessment data

■ Item or task design

■ Timing and methods of administration

■ Methods of assessment reporting and dissemination.

B) Relationship of State Assessments to Standards

The pilot study examined several options for reporting measures of the degree of alignment of

mathematics and science assessments in relation to content standards. The central question in

studying alignment is the degree to which assessments are consistent with the content and depth

of teaching and learning called for in the standards (Webb, 2002; Polikoff, Porter, and Smithson,


29

2011). Many states have conducted alignment studies for the assessment instruments used for

accountability reporting as part of the requirements under federal No Child Left Behind funding,

and a recent study has reported new information on the quality of the state consortia assessments

based on the Common Core State Standards and two other assessments used by states, based on

the criteria defined by CCSSO (Doorey & Polikoff, 2016).

For this project, NORC determined that it would not be possible to conduct an analysis of the

alignment of each state’s assessments in math and science. Instead, the advisers recommended a

design for collecting information on several measures of the degree of alignment that could be

compiled and reported across states that would address key differences and similarities in the

state assessments’ content breadth and depth relative to standards. A design for reporting on

grade-specific content topics, cognitive levels of assessment items, and the range of methods

used in assessing content knowledge and practices was tested in the pilot study, and with some

changes and additions, the measures were implemented for the year-one reporting for 2015-16.

The measures used are designed to provide analysis and comparison of state assessments’

content rigor and breadth specific to a grade, and the degree to which the items/tasks address the

depth of knowledge and cognitive demand of the standards. Thus, regardless of the standards

defined by state policy, and the assessments that are selected by states, the alignment indicators

should provide a common basis for reporting across states.

The following indicators of content alignment were identified for testing in the pilot study:

■ State policy on content standards, and relationship of state standards to CCSS and NGSS

■ When standards were developed and approved

■ Source of state assessment framework or blueprint (consortia, state, other)

■ Percentage of grade 5 mathematics assessments that cover selected content topics

■ Levels of cognitive demand used in developing math assessment items/tasks, and use of

levels in reporting student results

■ Elementary grades at which selected topics were assessed

■ Percentage of high school math assessments that cover selected content topics

■ State plans and methods for development of new science assessments


30

■ Major content topics for new science assessments.

■ Content for new science assessments aligned to standards


31

Summary: Year-One Results and Recommendations

Process Recommendations

Changes and edits to Questions

Cross-state results


32

References

Achieve & Lead States (2013). Next Generation Science Standards: By States, For States. Washington,

D.C.: National Academies Press. http://www.nextgenscience.org/next-generation-science-

standards

CCSSO’s (2014) Criteria for Procuring and Evaluating High-Quality Assessments

http://www.ccsso.org/Documents/2014/CCSSO%20Criteria%20for%20High%20Quality%20Ass

essments%2003242014.pdf

CCSSO/ATP (2013) Operational Best Practices for Statewide Large-Scale Assessment Programs.

http://www.ccsso.org/Resources/Publications/Operational_Best_Practices_for_Statewide_Large-

Scale_Assessment_Programs.html

Darling-Hammond, L., Herman, J., Pellegrino, J., Abedi, J., Baker, E., Bennett, R. et al. (2013). Criteria

for high-quality assessment. Stanford, CA: Stanford Center for Opportunity Policy in Education.

Doorey, N. & Polikoff, M. (2016) Evaluating the Content and Quality of Next Generation Assessments.

Washington, DC: Fordham Institute. http://edexcellence.net/publications/evaluating-the-content-

and-quality-of-next-generation-assessments

National Governors Association Center for Best Practices and Council for Chief State School Officers.

(2010). Common Core State Standards. Washington, DC: National Governors Association Center

for Best Practices and Council of Chief State School Officers. http://www.corestandards.org/

National Research Council (2013). Monitoring Progress toward Successful K–12 STEM Education: A

Nation Advancing? Washington, DC: National Academy Press.

National Research Council. (2013). Developing Assessments for the Next Generation Science Standards.

James W. Pellegrino, Mark R. Wilson, Judith A. Koenig, and Alexandra S. Beatty, Editors;

Committee on Developing Assessments of Science Proficiency in K–12. Board on Testing and

Assessment. Division of Behavioral and Social Sciences and Education. Washington, DC: The

National Academies Press.

NORC (2015) Study Design Paper: Developing a System for Tracking State Assessment Policies in

Science and Mathematics Education Instrument Design and Pilot Study Plans, Bethesda, MD.

March.

Polikoff, Porter, and Smithson (2011) How well aligned are state assessments of student achievement

with state content standards? American Educational Research Journal, 48(4), 965–995.

http://www.nextgenscience.org/next-generation-science-standardshttp://www.nextgenscience.org/next-generation-science-standardshttp://www.ccsso.org/Documents/2014/CCSSO%20Criteria%20for%20High%20Quality%20Assessments%2003242014.pdfhttp://www.ccsso.org/Documents/2014/CCSSO%20Criteria%20for%20High%20Quality%20Assessments%2003242014.pdfhttp://www.ccsso.org/Resources/Publications/Operational_Best_Practices_for_Statewide_Large-Scale_Assessment_Programs.htmlhttp://www.ccsso.org/Resources/Publications/Operational_Best_Practices_for_Statewide_Large-Scale_Assessment_Programs.htmlhttp://edexcellence.net/publications/evaluating-the-content-and-quality-of-next-generation-assessmentshttp://edexcellence.net/publications/evaluating-the-content-and-quality-of-next-generation-assessmentshttp://www.corestandards.org/http://www.nap.edu/catalog.php?record_id=13509http://www.nap.edu/catalog.php?record_id=13509http://www.nap.edu/openbook.php?record_id=18409


33

Resnick, L.B., Rothman, R., Slattery, J.B., & Vranek, J.L. (2003). Benchmark and alignment of standards

and testing. Educational Assessment, 9, 1–27.

Schmidt, W. H., McKnight, C. C., Houang, R. T., Wang, H. C., Wiley, D. E., Cogan, L. S., et al. (2001).

Why schools matter: A cross-national comparison of curriculum and learning. San Francisco:

Jossey-Bass.

Webb, N. L. (2002). Alignment study of language arts, mathematics, science, and social studies of state

standards and assessments in four states. Washington, DC: Council of Chief State School

Officers.

U.S. Dept. of Education’s Peer Review Guidance (issued 9/25/15)

http://www2.ed.gov/policy/elsec/guid/assessguid15.pdf

Yuan, K. & Le, V. (2012). Estimating the percentage of students who were tested on cognitively

demanding items through the state achievement tests. Santa Monica, CA: RAND Corporation.

http://www2.ed.gov/policy/elsec/guid/assessguid15.pdf


34

Appendix: Technical Advisers

Catherine Bowler, Science & Engineering Assessment, Massachusetts Department of Education

Pete Goldschmidt, CSU-Northridge (formerly New Mexico State Assessment Director)

Brian Gong, Executive Director, Center for Assessment, Dover, NH

Robin Hill, Mathematics, Kentucky Department of Education

Tammy Howard, State Accountability Services Director, North Carolina

Peter McLaren, State Science Specialist, Achieve, Inc. (formerly Rhode Island)

Hillary Michaels, HumRRO, Education Research & Evaluation, Alexandria, VA

Jessica Mislevy, Education Research Scientist, SRI International

Brett Moulding, Science Education Professional Development Director, Utah

Tiffany Neill, Science Education Specialist, Oklahoma State Department of Education

Scott Norton, CCSSO Strategic Initiative Director for Standards, Assessment, & Accountability

Cinda Parton, Assessment Specialist, Washington State Office of Education

Jim Pellegrino, University of Illinois at Chicago, Learning Sciences Institute

William Schmidt, Center for the Study of Curriculum, Michigan State University

Ellen Whitesides, Mathematics Educator & Math SCASS, Washington, DC