in the science and practice of standard setting, where is the science? barbara s. plake, university...
TRANSCRIPT
In the Science and Practice of Standard Setting, Where
is the Science?Barbara S. Plake,
University of Nebraska-Lincoln, Emeritus
Purpose of PresentationStandard setting involves applications of
Social science Psychometrics Content expertise Politics economics
Decisions based on scientific studies but many based on human judgment or for streamlining process
Presentation will focus on a few areas where additional research is needed to support many of these practical decisions
Overview of topics Selection of panelists
Panelists’ training
Panelists’ operational ratings
Feedback to panelists during rounds of ratings
Cross panel/vertical articulation of cutscores
Selection of PanelistsWhat is the impact of involving “outside”
stakeholders on panels?
What is the efficacy of using crowd-sourcing to create panel of raters?
Panelist TrainingWhat is the effect of providing pre-meeting
materials to panelists?
Panelists’ TrainingHow critical is it for the panelists to “take the
test”?
Is it sufficient for them just to “experience the test”?
Should panelists have a fuller practice in the method during training, including practice with feedback?
Panelists’ Operational Ratings
Does it matter which performance category is rated first?
Should the ratings be sequential or hierarchical?
Should the same RP value be used across content areas?
What is the best strategy for setting cutscores on computerized adaptive tests?
Feedback to PanelistsHow does the type and timing of feedback affect
panelists’ ratings?
How do graphical displays impact panelists’ interpretation of feedback data?
Cross-grade/vertical articulation of cutscores
What are the optimal ways to conduct cross-grade/vertical articulation of cutscores?
How much flexibility should the cross-grade panels be allowed in adjusting cutscores?
ConclusionMultiple areas for research to support validity of
cutscore interpretations and uses
Standard setting plays a crticial role in assessment practices
Results provide meaning to test scores
Validity of score interpreations relies directly on the quality of the standard setting process.
Research is needed to help ensure validity is a priority