Using Standing Setting in evaluation: Exploring

differences between the ordinary and excellent

Janet Clinton University of Auckland, New Zealand

University of Connecticut 2009

The chocolate chip cookie exercise

Chocolate chip standards of success

Characteristic

Poor Basic Good Excellent

Chewy centre

Health factor

Number of chips

Size

The x factor

Aims

Explore judgment of success

Illustrate standard setting as an approach

to systematically judge levels of success

Demonstrate that standard setting can be

a powerful tool in program evaluation in

either large or small evaluations

Standard setting particularly useful in

health settings

What is a standard?

A standard is a statement about

whether a performance is good

enough for a particular purpose

Evaluation is about judgments

The judges

How have we done this in the past?

The use of benchmarks to understand success:

Common practice Developed from literature reviews &

input by stakeholders however Often inaccurate or not representative

The lack of strong evidence and diverse cultures often makes the process of determining standards difficult

Praying……….

What is standard setting?

Standard setting is a means of

systematically harnessing human judgment

to define and defend a set of standards

Standard setting methodology

Describes the attributes of each category e.g., what makes a program fall into the category of excellent

A process of rationally deriving, consistently applying, and describing procedures on which judgments can be made

The process decides on a “cut-score” that then creates distinct and defensible categories e.g., pass/fail, allow/deny, excellent/poor.

Theoretical basis & method

Based on theories of judgment analysis & attribution theory

Used in educational measurement e.g. Jaegar, 1997

Methods◦ Angoff◦ Bookmark◦ Hofstee◦ Lens Modelling

The method has to be:DefensibleCredible Supported by body of

evidence in the literature

Feasible Acceptable to all

stakeholders

All about the process and the right fit

Standard Setting in an Evaluation of

National Chronic Care Management

Initiatives

The project

Understand best practice of chronic care management programs --- nationally.

Develop a user friendly best practice workbook: e.g. COPD, Stroke, CVD & CHF

◦ The evidence---Literature review◦ What is happening---Stock take survey◦ Key stakeholders--Expert interviews◦ Acceptable practice--Standard setting

exercise◦ Evaluation of individual programs or sites◦ Development of a best practice work book

Applying the methodology

Conducted Stand Setting Workshops

◦5 sites around NZ ◦Invite experts/stakeholders◦Group and individual rating exercises ◦Analyze the assessments◦Define a standard ◦Determine success indicators

Preparation work

1. Understand the evidence base for success

2. Develop a set of dimensions

3. Develop profiles of programs

4. Plan the workshops

5. Determine the composition of stakeholders

6. Develop assessment rubrics—scoring schedule

Exercise 1

Importance of each factor

◦Rank the importance of the factors

that make up best practice

◦Present the information back

◦Discuss as a group e.g. missing

dimensions

DIMENSION EXPLANATION

LEVEL OF SUPPORT FOR CHRONIC CARE

Limited Basic Reasonably Good Fully Developed

1 2 3 4 5 6 7 8 9 10

Conceptual understanding of CCM

Appropriate levels of collaboration

Active engagement of leadership

Development of sustainable community links

Focus on health (inequalities)

Decision support systems in place

Appropriate delivery design system

Knowledge transfer

Attention to efficiency/ cost/ output

Attention to effectiveness/ outcomes

Adherence to clinical guidelines

Overall perception of the programme

ID CODE: PROFILE NAME:

Site W

0

2

4

6

8

10

12

Con

cept

ual

unde

rsta

ndin

g of

CC

MP

atie

nt

App

ropr

iate

leve

lsof

col

labo

ratio

n

Act

ive

enga

gem

ent

of le

ader

ship

App

ropr

iate

deve

lopm

ent o

fsu

stai

nabl

eco

mm

unity

link

s

Foc

us o

n he

alth

(ineq

ualit

ies)

Dec

isio

n su

ppor

tsy

stem

s in

pla

ce

App

ropr

iate

del

iver

yde

sign

sys

tem

Kno

wle

dge

tran

sfer

Atte

ntio

n to

effic

ienc

y / c

ost /

outp

ut

Atte

ntio

n to

effe

ctiv

enes

s /

outc

omes

Adh

eren

ce to

clin

ical

gui

delin

es

Av

era

ge

Ra

tin

g (

+1

SD

)

Regional Comparison

0

1

2

3

4

5

6

7

8

9

10

11C

once

ptua

lun

ders

tand

ing

ofC

CM

App

ropr

iate

leve

lsof

col

labo

ratio

n

Act

ive

enga

gem

ent

of le

ader

ship

App

ropr

iate

deve

lopm

ent o

fsu

stai

nabl

e

Foc

us o

n he

alth

(ineq

ualit

ies)

Dec

isio

n su

ppor

tsy

stem

s in

pla

ce

App

ropr

iate

del

iver

yde

sign

sys

tem

Kno

wle

dge

tran

sfer

Atte

ntio

n to

effic

ienc

y / c

ost /

outp

ut

Atte

ntio

n to

effe

ctiv

enes

s /

outc

omes

Adh

eren

ce to

clin

ical

gui

delin

es

Av

era

ge

Ra

tin

g Rotorua

Whangarei

Christchurch

Auckland

Wellington

Regions

The Exercise 2: Round 1

• Form representative groups • Describe and discuss the dimension• Read the profiles & make a judgment about

the standard of the dimensions• Rate the dimensions• Each individual records the reason or

explanation for their choice

DIMENSION EXPLANATION

LEVEL OF SUPPORT FOR CHRONIC CARE

Limited Basic Reasonably Good Fully Developed

1 2 3 4 5 6 7 8 9 10

Conceptual understanding of CCM

Appropriate levels of collaboration

Active engagement of leadership

Development of sustainable community links

Focus on health (inequalities)

Decision support systems in place

Appropriate delivery design system

Knowledge transfer

Attention to efficiency/ cost/ output

Attention to effectiveness/ outcomes

Adherence to clinical guidelines

Overall perception of the programme

ID CODE: PROFILE NAME:

Analyzing the information

Analyze all the information

◦Present the information back to the established groups of mixed stakeholders

◦Individual information was returned

Exercise 2: Round 2

Information returnedGroups discuss their individual

choicesAim to get group consensus on

rating best practice of dimensionsInformation recorded along with

the explanations

DHB1 Self Perceptions - Average Ratings for Dimensions

012

345678

91011

Empo

werm

ent

and

Self-

Man

agem

ent

Colla

bora

tion

Lead

ersh

ip

Com

mun

ityLin

kage

s

Redu

cing

Ineq

ualiti

es

Decis

ionSu

ppor

t

Deliv

ery:

orga

nisat

ionof

hea

lthca

re

Deliv

ery:

prog

ram

me

deliv

ery

Know

ledge

Tran

sfer

Effic

iency

Effe

ctive

ness

Clini

cal

Guid

eline

s

Roles defined,

attendance at meetings;

program spokes person

No collaborati

on with PHO or NGOs

No plan to disseminate info to community

Results for each profile

Analysis of results

Regression to determine importance of dimensions

Create a matrix of explanations first by individuals then by group consensus

Consult the evidence & the experts Determine the ‘cut scores’ for each

assessment area—used the bookmark method

N Mean sd

Engagement of leadership 479 5.23 1.81Focus on health (inequalities) 479 4.97 2.09Collaboration 478 4.96 1.86Adherence to clinical guidelines 477 4.88 1.95Conceptual understanding 475 4.76 2.00Community links 475 4.67 1.86Attention to effectiveness / outcomes 476 4.65 2.03Attention to efficiency / cost / output 468 4.63 2.01Delivery design system 469 4.40 1.70Decision support systems 478 4.39 1.79Knowledge transfer 476 4.21 1.81

Overall 480 4.72 1.76

Overall means

Factor analysis Individuals Groups

Program & Organization 1 2 3 1 2 3Community links .92 .06 -.18 .69 .06 .06Collaboration .74 .04 .05 .90 -.00 -.07Focus on health inequalities .72 .01 .00 .75 -.10 .17Conceptual understanding .61 -.07 .21 .86 .06 -.08Delivery design system .54 .05 .22 .52 .36 -.01Engagement of leadership .40 .06 .39 .48 .24 .24

Effectiveness/EfficiencyAttention to efficiency -.03 .97 -.03 .04 .91 -.02Attention to effectiveness .10 .73 .12 .15 .79 -.03

InformationDecision support systems .08 .10 .72 .30 .19 .70Adhere to clinical guidelines .05 .21 .59 -.11 .67 .32Knowledge transfer .14 .36 .39 .43 .48 -.00

Factor inter correlationsProgram 1 1Efficiency/Effectiveness .62 1 .68 1Information .66 .68 1 .38 .51 1

Across the exemplars

1 5 10 3 8 6 7 2 4 9 Total1.00

2.00

3.00

4.00

5.00

6.00

7.00

Grp- Final 4 Ind - Final 4 Gp - All Ind - All

Reasonably good

Fully Developed

Limited

Basic

DIMENSION

LEVEL OF SUPPORT FOR CHRONIC CARE

LIMITED BASICREASONABLY

GOODFULLY

DEVELOPED

1 2 3 4 5 6 7 8 9 10

CONCEPTUAL UNDERSTANDING

COLLABORATIONACTIVE LEADERSHIPCOMMUNITY LINKSFOCUS ON INEQUALITIESDECISION SUPPORTDELIVERY SYSTEMKNOWLEDGE TRANSFER

EFFICIENCY COST/ OUTPUT

EFFECTIVENESS-OUTCOMES

USE OF CLINICAL GUIDELINES

OVERALL

The standard

Dimension Limited Basic Reasonably good

Excellent

Collaboration No work with communityNo evidenceHospital focusPoor referral system

Low engagement with Maori low levels of trustLittle primary/ secondaryintegration

Recognises weaknessesEvidence of partnershipsGood initiatives

Community approach evidencedWhole system collaborationLots of alternatives

Leadership No championPoor managementLack of evidence

Some leadershipNo championsFoundation of leadership

Strong clinical leadershipWeak championsIdentified problems but no change

Evident leadership at programStrong championsEvidence of vision

Did it work?

Analysis illustrates that we can

create cut scores for basic through

to excellent.

We can define the judgment

Have an explanation or criteria for

each level

Reactions to the workshops

◦ We have a voice

◦ Systematic

◦ Inclusive for roles,

regions & view

◦ Useful for self review

◦ Hard work

◦ We got nothing

◦ This is dangerous

◦ Stupid!

Variable

Draw backs

Resource intense

Analysis can be problematic

Time

Pressures

Setting league tables in health

Scary

Consequences

Creates a benchmark for judgment

Validates research through practice

Understand different views of success

Facilitates self-review

Encourages learning

Encourages an evaluation perspective

Working with Community groups

Adapt the processNeed 8-10 in groupGreat focus of the explanation Qualitative analysisPreparation takes community into

accountUse of evidence to build rubrics &

exemplarsFeasibility-Propriety- Accuracy-Utility

Final word

Not so much◦the METHOD as the PROCESS

The judgment