Using the PADI Design System to Examine the

Features of a NAEP Performance Assessment

Kathleen C. Haynie Kathleen Haynie Consulting

Andrea A. Lash, Geneva D. Haertel, Edys S. Quellmalz, Angela Haydel DeBarger

SRI InternationalAERA April 2005

Background Use of 21st century collaborative

workplace tools is on the rise Technologically rich “dispersed”

collaboration requires (Olson & Olson, 2000): Common ground Coupling of work Technological readiness Collaborative readiness

2

PADI Project Seeks to improve assessment of

inquiry in science learning

Networked collaboration

Development and use of Web-based tool

3

Central ProcessThe central process for project strands working with the PADI Design System can be stated as follows: The interaction between assessment materials (e.g., assessment tasks) and the Evidence-Centered Design framework (Mislevy, Steinberg, & Almond, 2002) is mediated by various tools, representational forms, and representations utilized by collaborative workgroups. 4

5

PADI Assessment Design

Goals: To reverse engineer a performance task

from a large-scale, national assessment To use the PADI System as an analytical

tool for understanding task features In reverse engineering and analyzing

a performance assessment task via PADI, what types of knowledge were created?

6

NAEP Performance Assessment

1. Selection of a Set of Items (July – August)

2. Exploration of the Floating Pencil Task (August – December)

3. Development of a Task Specification (December – March)

7

Reverse Engineering & Analysis Process

We selected a set of items for analysis NSES inquiry standards, PADI Design

Patterns

Defined our Student Model Exploration of multiple student models Choice of NAEP framework

Refined the Student Model8

The Student Model

We explored evaluation & measurement issues via the PADI representational forms Evaluative Sub-Model based on NAEP

rubric Multiple levels of conditional dependencies MRCML psychometric model

Clarified our Measurement Model Based closely on NAEP practices Elements of PADI

9

The Evidence Model

We explored attributes of the Floating Pencil task Content-lean, inquiry-constrained Use of lab materials; High verbal

demand Constructed Task Model Sought to define a ‘Family of Tasks’

10

The Task Model

Process: Floating Pencil team engaged in collaboration using technological workplace tools

Results: Analysis of the Floating Pencil task A ‘trace’ on PADI: Floating Pencil Task Specification The impact of our work on PADI Design System

Next Steps: Task family and TMVs Empirical data analysis Technical report on Floating Pencil work Consideration of alternative student models 11

So What?