show me an evidential approach to assessment design michael rosenfeld f. jay breyer david m....
TRANSCRIPT
Show Me an Evidential Approach to Assessment Design
Michael RosenfeldF. Jay Breyer
David M. WilliamsonBarbara Showers
A Brief Introduction to Evidence Centered Design (ECD)
Michael Rosenfeld, Ph.D.Rosenfeld & Associates
What is Evidence Centered Design? A formal framework for assessment
design Emphasizes assessment as an
evidential argument Encourages models and
representations that strengthen validity arguments
Focuses the test development process to better target decisions of interest
The Basic Models of ECDProficiency Model Evidence Models
Statmodel
EvidenceRules
Task Models
Features1.
xxxxx2.
xxxxx
3. xxxxx
Proficiency Model – What you want to measure
Evidence Model – How to recognize & interpret observable evidence of unobservable proficiencies
Task Models – How to elicit valid and reliable evidence
ECD Validity Chain
Evidence
Evidence
Evidence
Proficiency
Tasks
Tasks
Tasks
Tasks
Tasks
Task Model
Task Model
Task Model
Task Model
What is Gained from ECD? Better understanding of the domain, for new
constructs Explicit chain of evidence from assessment goals,
to models of evidence, to assessment tasks Strong construct validity argument Rapid and efficient item generation from models
automatic item generation Opportunities for innovation, task types other than
multiple choice Philosophical shift from content-authoring to
evidential argument
Typical Test DevelopmentContent Specification
Item Writing
Test Blueprint
Analysis
Cut Scores
The Job
Knowledge & Tasks
Major DimensionsJob Analysis
Incumbent Survey
Content Linking
Statistical Modeling
Typical Test Development and Evidence Centered Design
Content Specification
Item Writing
Test Blueprint
Analysis
Cut Scores
Typical Test Development Evidence Centered Design
Proficiency Model
Task Model
Evidence Model
Tasks
Evidence Centered Design
Proficiency Model
Task Model
Evidence Model
Tasks
Claims (Criterion)
Knowledge, Strategies & Skills
Reporting Variables
Content Linking
Evidential Argument
Job Analysis
Cognitive Task Analysis
Statistical Modeling
Task Design
Still to Come
Example of the Proficiency Modeling process (Jay)
Illustration of Evidence Modeling and Task Modeling (David)
Commentary from an applied perspective (Barbara)
Proficiency Model Combination of
Formalization of assessment goals, population and purpose
Formal claims to be made on the basis of the assessment
He/she is competent to operate an automobile
Proficiencies of interest Knowledge of rules, procedures, and
laws Ability to implement driving
regulations in practice Skill in automobile operation
What is a Proficiency? Proficiencies are knowledge, skills
and abilities that provide the basis for making claims about people
Building a Proficiency Model: Assessment Goals
Assessment Goals Protect the public health, safety & welfare
Examinee population All eligible (age, prior training requirements, etc.)
citizens self-selecting into the program Test users/stakeholders
State regulatory bodies State resident drivers Other citizens exposed to driving
Result Usage/Decisions State issuing the license (to issue or deny) Means of official identification
Building a Proficiency Model: Assessment Claims
Formal Claims are developed for assessment goals Claims can be course-grained and general
Can safely operate a motor vehicle These general claims can be be supported by a
variety of more specific sub-claims in a hierarchy Operates a vehicle in conformance with posted
signs Stops at a posted stop sign Obeys posted speed limits
The final hierarchy of claims drives the evidential argument and the score reporting needs of the test
Building a Proficiency Model: Proficiencies Proficiency variables are the basis of
making claims Proficiency variables, like claims, may be
organized hierarchically Values or levels of proficiency variables
may be selected to support claim distinctions
Informed by investigations of the domain and learning within the domain job analyses, cognitive task analyses, learning
models, etc.
Example of a Hypothetical Proficiency Model for Driving
Can drive safely
Understands rulesand regulations
Can operate amotor vehicle
DrivingAbility
KnowRotR
Psycho-motor
Weather& light
Inter-sections
TrafficFlow
Parking
Turning
Speed
Still to Come
Illustration of Evidence Modeling and Task Modeling (David)
Commentary from an applied perspective (Barbara)
Building Evidence Models Objective: valid inference
Understanding what constitutes evidence and why
Situations that elicit evidential behaviors Task design Statistical representation of evidential value
of observations Integration of scoring with design
ECD models – conceptual scoring
Establishing Evidence in a “Perfect World”
Proficiency Evidence Task
Targets of Inference
Links behaviorsand inference
Behavior Elicitation
Driving Ability•Knowledge of RotR•Psychomotor
Evidence•Knowledge declarations•Performance
Task Requirements•Situation•Outcomes
Establishing Evidence: Driver’s Exam
Proficiency Evidence Task
•Knowledge of RotR•Weather & Light•Traffic Flow•Intersections•Speed
•Psychomotor•Intersections•Speed•Turning•Parking
Driving Ability•Knowledge of RotR•Psychomotor
Intersections•Recognizing signs
•Specifying intersection traffic behaviors
•Following proper procedure
Building Task Models: Why? Elicits behaviors by predictably
manipulating Focus Difficulty Context
Targets what we want to measure Provides a framework for generating
multiple tasks targeting same proficiencies
Building Task Models: What?
To target elicitation of particular evidence, the task model specifies:
Tools provided Directions and stimulus materials Indication of data collected and retained
for scoring Relevance of specified elements to
particular evidence, and particular proficiencies
Evidence as Inference about Proficiency
Proficiency Evidence Task
IdentificationAccumulation
Stage 1: Evidence Identification Identify and summarize response elements
Stage 2: Evidence Accumulation Using these elements to estimate ability
Beneficial Outcomes of ECD
Development direction w evidential focus Integration of expertise/activities (TD, stat,
etc.) in the design process Chain of reasoning
Integrated design – implications of design changes Validity argument
Item modeling AIG Defensibility Directed item writing – future forms
Somewhat more expensive (Initially)