Moving to the Next Generation of Science Assessments:Science Assessments:

Lessons Learned from the National A t f Ed ti l PAssessment of Educational Progress

D P C A i t C i iDr. Peggy Carr, Associate CommissionerNational Center of Educational StatisticsJune 27, 2012

2012 National Conference on Student Assessment

1

Elements of Science Assessment

S11 14

I09

SC T IHNAEP

ScienceNAEPTEL

NAEPICT-HOTS

12

T11

P TS2

PISAScience

TIMSSScience

11

SC11

SCNAEP

S i

NAEP Science Framework

Science

NAEP Science Framework

NAEP Science Framework

Science PracticesScience Content Areas

• Identifying SciencePrinciples

• Physical SciencePrinciples

• Using Science • Life Science

• Earth and Space Principles

• Using Scientific Inquiry

• Earth and SpaceSciences

Using Scientific Inquiry

• Using TechnologicalDesignDesign

Grade 8 Science Question

Sample Question: Physical Science

Grade 8 Science Question

09

IHIHNAEP

ICT-HOTS

Science in Action

A New Generation of Testing

• Part of 2009 NAEP science assessmentH d T k• Hands-on Tasks:• National sample of 4th-, 8th-, and 12th-

gradersg• Two 40-minute tasks at each grade

• Interactive Computer Tasks:• National sample of 4th-, 8th-, and 12th-

graders• Two 20-minute tasks and one 40-minute taskTwo 20 minute tasks and one 40 minute task

at each grade• National results only

8

NAEP Science In Action 2009Hands-on Tasks

9

10

NAEP Science In Action 2009

11

Science In Action 2009 Results

Three Key Findings

12

NAEP Science In Action 2009:Inability to Explain Conclusionsy p

Grade 12

13

14

TTNAEPTEL

Technology and Engineering Literacy (TEL) Assessment

3 Dimensions of knowledge and abilitiesab t es

• Disciplinary Core Ideas In Science• Scientific And Engineering Practices (new!)• Crosscutting Concepts (new!)

TEL Definition

Technology and engineering literacy:

the capacity to use, understand, and evaluate

technology as well as to understand technologicaltechnology as well as to understand technological

principles and strategies needed to develop

solutions and achieve goals

16

TEL Content Areas and Competencies

17

Ch ll f h NChallenges for the Next Generation of Assessments

Measurement Challenges

• Developing tasks that match the framework

• Developing interfaces that are appropriate d iand engaging

• Developing assessment, NOT teaching tasks

20

Conclusions

• Technology enhances our ability to measure and report student scientific ability beyond what paper and pencil assessments can provide.

• Piloting new items and delivery systems were crucial steps toward obtaining valid results.

• Using uniform platforms were key to having successful test administrations.successful test administrations.

• Rapidly changing technology is an important p y g g gy pconsideration for future test development.

21

Item Development Integrating HOTs and ICTs

• Develop a sufficient number of HOTs and ICTs to complement each other and the discrete items incomplement each other and the discrete items in ways that are consistent with the framework

• Ensure the constructs assessed by the HOTs andEnsure the constructs assessed by the HOTs and ICTs extend what can be measured by discrete items

• Balance efficiency with depth of measurement for each component

• Collect evidence progressively throughout the development cycle (e.g., play testing, cog labs, and small-scale tryouts) to confirm mapping to cognitive models

About PISA

• Assesses 15-year-olds every three years

• In 2012, focusing on math, reading, science, problem solving, and financial literacy

• Approximately 70 countries will participate, including United States

23

NAEP-TIMSS Linking Study

• Scores for all 50 states on TIMSSTIMSS• 8th grade math and science

• Will allow for international benchmarking of states

• Anticipated release: Winter 2012-2013

• Expected replication in 2015

24

Our Changing Environment

• Move toward Common Core State Standards

• Workplaces requiring fluency in technological knowledge

• Major shift from paper-and-pencil tests to computer-based testsp

25