some implications of expertise research for educational assessment robert j. mislevy university of...
TRANSCRIPT
Some Implications of ExpertiseResearch for Educational
Assessment
Robert J. Mislevy
University of MarylandNational Center for Research on Evaluation,Standards, and Student Testing (CRESST)
2005 CRESST ConferenceUCLA, Los Angeles, CASeptember 8-9, 2005
The Central Idea Expertise research seeks to understand in
detail the knowledge, representations, development, and contexts of expertise.
Results can ground assessment arguments & task design.
Must recast in terms of purpose, perspective, principles, and structures of assessment.
Especially valuable with complex proficiencies, situations, or performances.
Outline
An example: Architectural design
Implications of expertise research
Assessment arguments
Generative schemas for task design
Conclusion
The Architectural Registration Examination
Architectural design; CAD-like environment.
(ETS: Katz, Bejar, Braun, Hone, Brittingham, Bennett, et al.)
To replace 10-hour hand-drawn design problem
Reflects changing of profession to CAD
Premium on thinking, not drawing
Some Results
Differences between novices and experts• Success rate: 98% vs. 88%
• Planning time & execution time
• Patterns of revision involving rework
Assessment Arguments
What complex of knowledge, skills, or other attributes should be assessed?
What behaviors or performances should reveal those constructs?
What tasks or situations should elicit those behaviors?
(Messick, 1994)
Implications for task design (1)
Experts perceive/understand/act via fundamental principles rather than surface features• Chi, Feltovich, & Glaser on physics
In architecture, Constraints• Number • Variation in importance/difficulty• Degree of conflict• Implicit constraints
Implications for task design (2)
Importance of interaction with situation Intelligence built into situations, tools,
processes “Attunement to affordances” Cycles of …
Approach to solution, as in AREHypothesize, test, revise, as in
inquiry & troubleshooting
Implications for task design (3)
External knowledge representationsMaps, insurance forms, symbol systems, diagrams,
Punnett squares, wiring charts, blueprints
Roles in practice Gather/create/share/hold/transform knowledge Embody generative principles of domain Central to practice in the domain
Roles in assessment Environment / stimuli / work of tasks Design KRs to create / implement assessment
CTA for Assessment
CTA can help ground assessment, re validity argument and principled/explicated task construction. Must aim to specify elements of Student-,
Evidence-, and Task-models
in light of the intended assessment’s purpose,
and constraints of the assessment-building and assessment-delivery contexts.
Design patterns
Look across domains to find recurring difficulties, blockages, overloads (Salthouse, 1991)
Classes of expertise / observables / situations that arise in many domains.
Can create “Design patterns” in, e.g.,• Design under constraints (engineering)
• Problem-solving in finite domains (troubleshooting)
• Model-based reasoning (science)
Advances in assessment design
A cognitive design system approach to generating valid tests (Embretson,1998)
Model-basel assessment (Baker, 1997)
Constructing measures (Wilson, 2004)
Understanding by design (Wiggins, 1998)
On the structure of educational assessments (Mislevy, Steinberg, & Almond, 2003)