![Page 1: Using the PADI Design System to Examine the Features of a NAEP Performance Assessment Kathleen C. Haynie Kathleen Haynie Consulting Andrea A. Lash, Geneva](https://reader038.vdocuments.us/reader038/viewer/2022100514/5a4d1b487f8b9ab0599a44c4/html5/thumbnails/1.jpg)
Using the PADI Design System to Examine the
Features of a NAEP Performance Assessment
Kathleen C. Haynie Kathleen Haynie Consulting
Andrea A. Lash, Geneva D. Haertel, Edys S. Quellmalz, Angela Haydel DeBarger
SRI InternationalAERA April 2005
![Page 2: Using the PADI Design System to Examine the Features of a NAEP Performance Assessment Kathleen C. Haynie Kathleen Haynie Consulting Andrea A. Lash, Geneva](https://reader038.vdocuments.us/reader038/viewer/2022100514/5a4d1b487f8b9ab0599a44c4/html5/thumbnails/2.jpg)
Background Use of 21st century collaborative
workplace tools is on the rise Technologically rich “dispersed”
collaboration requires (Olson & Olson, 2000): Common ground Coupling of work Technological readiness Collaborative readiness
2
![Page 3: Using the PADI Design System to Examine the Features of a NAEP Performance Assessment Kathleen C. Haynie Kathleen Haynie Consulting Andrea A. Lash, Geneva](https://reader038.vdocuments.us/reader038/viewer/2022100514/5a4d1b487f8b9ab0599a44c4/html5/thumbnails/3.jpg)
PADI Project Seeks to improve assessment of
inquiry in science learning
Networked collaboration
Development and use of Web-based tool
3
![Page 4: Using the PADI Design System to Examine the Features of a NAEP Performance Assessment Kathleen C. Haynie Kathleen Haynie Consulting Andrea A. Lash, Geneva](https://reader038.vdocuments.us/reader038/viewer/2022100514/5a4d1b487f8b9ab0599a44c4/html5/thumbnails/4.jpg)
Central ProcessThe central process for project strands working with the PADI Design System can be stated as follows: The interaction between assessment materials (e.g., assessment tasks) and the Evidence-Centered Design framework (Mislevy, Steinberg, & Almond, 2002) is mediated by various tools, representational forms, and representations utilized by collaborative workgroups. 4
![Page 5: Using the PADI Design System to Examine the Features of a NAEP Performance Assessment Kathleen C. Haynie Kathleen Haynie Consulting Andrea A. Lash, Geneva](https://reader038.vdocuments.us/reader038/viewer/2022100514/5a4d1b487f8b9ab0599a44c4/html5/thumbnails/5.jpg)
5
PADI Assessment Design
![Page 6: Using the PADI Design System to Examine the Features of a NAEP Performance Assessment Kathleen C. Haynie Kathleen Haynie Consulting Andrea A. Lash, Geneva](https://reader038.vdocuments.us/reader038/viewer/2022100514/5a4d1b487f8b9ab0599a44c4/html5/thumbnails/6.jpg)
Goals: To reverse engineer a performance task
from a large-scale, national assessment To use the PADI System as an analytical
tool for understanding task features In reverse engineering and analyzing
a performance assessment task via PADI, what types of knowledge were created?
6
NAEP Performance Assessment
![Page 7: Using the PADI Design System to Examine the Features of a NAEP Performance Assessment Kathleen C. Haynie Kathleen Haynie Consulting Andrea A. Lash, Geneva](https://reader038.vdocuments.us/reader038/viewer/2022100514/5a4d1b487f8b9ab0599a44c4/html5/thumbnails/7.jpg)
1. Selection of a Set of Items (July – August)
2. Exploration of the Floating Pencil Task (August – December)
3. Development of a Task Specification (December – March)
7
Reverse Engineering & Analysis Process
![Page 8: Using the PADI Design System to Examine the Features of a NAEP Performance Assessment Kathleen C. Haynie Kathleen Haynie Consulting Andrea A. Lash, Geneva](https://reader038.vdocuments.us/reader038/viewer/2022100514/5a4d1b487f8b9ab0599a44c4/html5/thumbnails/8.jpg)
We selected a set of items for analysis NSES inquiry standards, PADI Design
Patterns
Defined our Student Model Exploration of multiple student models Choice of NAEP framework
Refined the Student Model8
The Student Model
![Page 9: Using the PADI Design System to Examine the Features of a NAEP Performance Assessment Kathleen C. Haynie Kathleen Haynie Consulting Andrea A. Lash, Geneva](https://reader038.vdocuments.us/reader038/viewer/2022100514/5a4d1b487f8b9ab0599a44c4/html5/thumbnails/9.jpg)
We explored evaluation & measurement issues via the PADI representational forms Evaluative Sub-Model based on NAEP
rubric Multiple levels of conditional dependencies MRCML psychometric model
Clarified our Measurement Model Based closely on NAEP practices Elements of PADI
9
The Evidence Model
![Page 10: Using the PADI Design System to Examine the Features of a NAEP Performance Assessment Kathleen C. Haynie Kathleen Haynie Consulting Andrea A. Lash, Geneva](https://reader038.vdocuments.us/reader038/viewer/2022100514/5a4d1b487f8b9ab0599a44c4/html5/thumbnails/10.jpg)
We explored attributes of the Floating Pencil task Content-lean, inquiry-constrained Use of lab materials; High verbal
demand Constructed Task Model Sought to define a ‘Family of Tasks’
10
The Task Model
![Page 11: Using the PADI Design System to Examine the Features of a NAEP Performance Assessment Kathleen C. Haynie Kathleen Haynie Consulting Andrea A. Lash, Geneva](https://reader038.vdocuments.us/reader038/viewer/2022100514/5a4d1b487f8b9ab0599a44c4/html5/thumbnails/11.jpg)
Process: Floating Pencil team engaged in collaboration using technological workplace tools
Results: Analysis of the Floating Pencil task A ‘trace’ on PADI: Floating Pencil Task Specification The impact of our work on PADI Design System
Next Steps: Task family and TMVs Empirical data analysis Technical report on Floating Pencil work Consideration of alternative student models 11
So What?