in the science and practice of standard setting, where is the science? barbara s. plake, university...

In the Science and Practice of Standard Setting, Where

is the Science?Barbara S. Plake,

University of Nebraska-Lincoln, Emeritus

Purpose of PresentationStandard setting involves applications of

Social science Psychometrics Content expertise Politics economics

Decisions based on scientific studies but many based on human judgment or for streamlining process

Presentation will focus on a few areas where additional research is needed to support many of these practical decisions

Overview of topics Selection of panelists

Panelists’ training

Panelists’ operational ratings

Feedback to panelists during rounds of ratings

Cross panel/vertical articulation of cutscores

Selection of PanelistsWhat is the impact of involving “outside”

stakeholders on panels?

What is the efficacy of using crowd-sourcing to create panel of raters?

Panelist TrainingWhat is the effect of providing pre-meeting

materials to panelists?

Panelists’ TrainingHow critical is it for the panelists to “take the

test”?

Is it sufficient for them just to “experience the test”?

Should panelists have a fuller practice in the method during training, including practice with feedback?

Panelists’ Operational Ratings

Does it matter which performance category is rated first?

Should the ratings be sequential or hierarchical?

Should the same RP value be used across content areas?

What is the best strategy for setting cutscores on computerized adaptive tests?

Feedback to PanelistsHow does the type and timing of feedback affect

panelists’ ratings?

How do graphical displays impact panelists’ interpretation of feedback data?

Cross-grade/vertical articulation of cutscores

What are the optimal ways to conduct cross-grade/vertical articulation of cutscores?

How much flexibility should the cross-grade panels be allowed in adjusting cutscores?

ConclusionMultiple areas for research to support validity of

cutscore interpretations and uses

Standard setting plays a crticial role in assessment practices

Results provide meaning to test scores

Validity of score interpreations relies directly on the quality of the standard setting process.

Research is needed to help ensure validity is a priority