A Rational Approach to Outcome

Measurement Duncan Raistrick

Leeds Addiction Unit

Leeds and York Collaborations for Leadership in

Applied Health Research and Care (CLAHRC)

Funded by the NIHR

Payment by Results Conference

Friday 28th September 2012

NIHR Nine CLAHRCs approx £10million each

CLAHRC for Leeds/York/Bradford Addiction

Research in Acute Settings (ARiAS) 1 of 5 themes

ARiAS outcome measurement 1 of 6 strands

CLAHRC research group: Duncan Raistrick, Gillian

Tober, Christine Godfrey, Charlie Lloyd, Steve Parrot,

Jude Watson, Veronica Dale

Co-opted Expert Group convened March 2011: Owen

Bowden-Jones, Alex Copello, Ed Day, Eilish Gilvarry,

Don Lavoie, Damian Mitchell, Julia Sinclair, John

Strang, and Alex Whincup

Summary

o Views and expectations of different stakeholders

Service user and carers

Politicians and commissioners

Practitioners

o Substance use and scales

Self report

Biological measures

Scales EQ5D LDQ CORE SSQ

Societal Impact Measures

o Further interesting things to find out

Relationship between measures

Clinically significant change

Outcome measures answer the question

“how do I know this person is getting better?”

and they also...

summarise complex information in a clinically meaningful

and ‘real world’ way

communicate complex information in a clear and simple

way using minimal data

can be integrated into routine practice in a clinically useful

way

but.............

......different stakeholders have different ideas of ‘getting

better’

...... the same stakeholders have different needs

o Service users and carers – abstinence

o Politicians and commissioners – costs and benefits

o Practitioners

Health workers – physical and mental health

Criminal justice workers – offending behaviour

Social workers – safeguarding children

......so, there is no ideal measure, just the mix of

measures that best suits the purpose.

Service user and carer views on what

outcomes are important

Agency Type Type of Activity SU F&F

FG1 NHS Treatment 7 4

FG2 NHS/3rd sector DRR 3 2

FG3 NHS/3rd sector Harm reduction 0 0

FG4 SMART group Recovery mutual aid 7 n/a

FG5 3rd sector Recovery SU only 7 n/a

FG6 3rd sector Recovery F&F only n/a 6

TOTAL 24 12

Source: unpublished CLAHRC study 2012

Six focus groups (FGs) were held to elicit service user (SU)

views and views of Family and Friends (F&F) on what

constitutes a good outcome.

Analysis of Focus Groups FG1 recovery is abstinence, social life, support for family and friends, relationships, lifestyle with

partner, being in control, life style changes during recovery process, physical and mental

health, confidence

FG2 positive environment to maintain abstinence and move away from drugs and alcohol, no good

outcome – it’s with you for life, not using methadone or buprenorphine, social support, doing

something to fill time, safe accommodation, children back from care, confidence, physical

health and appearance

FG4 physical and mental health, ability to distract and not act upon cravings, self-worth, energy, not

feeling isolated, personal responsibility, living a normal life, engaging in treatment, improved

relationships

FG5 self esteem, eating healthily, abstinence, sleep, physical health, relationships, social networks,

feeling valued, new interests and skills, courses.

FG6 employment, safe and supportive social networks, something to do during the day, self-worth,

physical and psychological well-being, having a good addiction, not feeling ashamed, financial

situation

There was a distinct view that abstinence or perhaps moderation is the

first step but the social support to maintain change is all important. Other

gains such as health and relationships were seen as good but not the

essence of a good outcome.

Source: unpublished CLAHRC study 2012

Substance Use as the Primary Outcome

Justification for substance use as the

primary outcome measure.......

Viewed by service users as most important

Viewed by careers as most important

Most convincing for general public and policy

makers

Substance misuse is the condition

Correlation with societal costs

Different Ways to Measure Substance Use

Frequency

Categories

ASSIST &

AUDIT

Objective

Testing

blood tests

toxicology

Episodes

of Use

OTI

Quantity /

Frequency

MAP & TOP

Actual

Frequency

ASI

Composite measures are restrictive: • always a compromise – too much/too little detail eg ASI

• typically rater completed eg TOP, ASI, MAP

• item selection bias eg TOP

• predetermined categories not suited to particular needs eg AUDIT

• not easy to update/modify and revalidate eg TOP, ASI

0

10

20

30

40

50

60

% o

f sam

ple

in d

rin

kin

g v

olu

me g

roup

Interview year

Abstinent Sensible (<14/21 units) Hazardous (14-35 / 21-50)

Heavy (35-70 / 50-100) Very heavy (70+ / 100+)

Birmingham Heavy Drinkers 10yr Follow-up Uses UNITS of alcohol grouped in ranges

n=259 followed at 10yrs: >50% reported major life events in previous 2yrs:

typically health, employment, shift of attitude. 18 known deaths: 3 CVS, 4

liver, 3 cancer, 1 suicide, 2 diabetes. Abstinence increased from 0-10%,

weekly units decreased from 90-59 (m) and 60-36 (f)

Source: Rolfe et al.(2009) Report to DH

UK Alcohol Treatment Trial FREQUENCY and UNITS of alcohol

0

20

40

60

80

100

Pe

rce

nt o

f D

ays A

bstin

en

t

Baseline 12 Months

MET SBNT

0

5

10

15

20

25

30

Drin

ks p

er

Drin

kin

g D

ay

Baseline 12 months

MET SBNT

Source: UKATT Research Team, BMJ (2005)

Problems of the substance itself as an

outcome measure……

o Is it how much (quantity) or how often (frequency) that matters? How reliably can these be measured?

o Is it the range of substances used or just the presenting main substance that should be the outcome? Does this include prescribed drugs (eg methadone, diazepam)?

o Are some drugs associated with more harmful routes of administration than others (eg heroin)?

o Does substance use itself matter or is it the related harms? Are some harms clearly attributable to particular substances (eg alcohol) some much less so (eg heroin)?

o How to measure it?

72-hour detection window (drugs that can be detected up to 72hrs)

Frequency of

drug use

Frequency of testing

2x /wk 1x /wk 2x /mth 1x /mth 8x /yr

Every day 3±2 7±2 15±10 30±13 46±40

Every other day 4±3 8±3 18±12 35±17 51±50

1x /wk 5±4 11±7 23±18 48±31 71±66

1x /wk 9±9 18±14 40±33 80±64 118±106

2x /mth 19±21 39±35 91±88 160±124 272±260

1x /mth 36±42 71±66 150±141 306±283 560±598

Toxicology Screening as an Indirect Measure Days (mean ± sd) before a positive test would be expected given different drug use

frequency and different testing frequency

Source: Crosby et al., (2003) J Addictive Diseases

Correlations with Societal Costs example data for units of alcohol and health status

0

10

20

30

40

50

60

70

80

90

100

0 10 20 30 40 50 60

one2one

increase

decrease

weekly units of alcohol

health a

s %

of best

possib

le h

ealth

moderate drinking not

associated with major health

problems

health may not recover

once damage is done

Correlations with Substance Use....

Baseline Follow-up

EQ5D LDQ CORE SSQ EQ5D LDQ CORE SSQ

Alcohol

freq -0.32** 0.48** 0.28** -0.08* -0.30** 0.62** 0.39** -0.20**

Alcohol

units -0.38** 0.47** 0.25** -0.19** -0.30** 0.62** 0.38** -0.20**

Heroin

freq -0.11 0.39** 0.09 -0.10 -0.21 0.40** 0.40** -0.60**

**p<.001 *p<.01 Source: unpublished CLAHRC clinical sample

but....

substance use is difficult to measure – LDQ good proxy

other scales are needed to paint a picture of outcome

Conclusion There is a scientific and political case for the primacy of substance

use as an outcome measure....

Choosing Scales as Secondary Outcome

Measures

Types of Outcome Scales

Generic Measures (Health) Is treatment cost effective? How ill are people with addiction problems compared

to other users of health care? How complex are the health problems? What is

the illness profile of people with addictions?

Dimension Measures (Addiction) How severe is the addiction? How difficult is treatment likely to be? How good is

one addiction service compared to another? Do problems persist?

Condition Specific Measures (Depression, Pregnancy) How severe is the specific condition? How do services targeting the condition compare? How effective is treatment for this specific problem?

Personal Goal Measures (Me) The personal outcome goals agreed with each service user.

Source: Fitzpatrick et al. (1998) Health Technology Assessment

Societal Impact Measures - SIMs

Treating the individual has benefits at the societal

level – SIMs are service level

Political interest is in societal costs (a research

exercise) – SIMs are a headline contribution

SIMs need to be: objective, easy to collect, capable

of showing variability, reflect the impact of the service

in the fewest possible measures...............

Pregnancy and Parenting i) birth weight ii) child with mother at

12months

Hospital In-reach i) A&E attendances ii) in-patient admissions in

last 12months

Detoxification i) % completing ii) % supervised disulfiram

A Quality Framework for Outcome Scales

Self completion scales are the gold standard

Scales evaluated by scoring for:

Evidence Base (including independent evaluations)

Psychometric Properties

Normative Data

Availability (free and supported by website)

Ease of Use

Universality (all substances and all socio-economic

groups)

Service User Evaluations Source: NIMHE (2009)

Scales that best match quality criteria...

Evidence

(0-2)

Number of

validation

publications

Psychometric

properties

(0-6)

Availability

(0-2)

Practicality

(0-3)

Universality

(0-2)

Population

norms

(0-2)

EQ-5D

health 2 tbc 6 2 2 2 2

LDQ

dependence 2 3 5 2 3 2 1

CORE-10

mental health 2 2 5 2 3 2 2

SSQ

satisfaction 1 1 5 2 3 2 1

APQ

problems 2 2 6 2 3 0 0

HoNOS

mental health 2 tbc 4 2 3 2 tbc

PHQ9

depression 2 tbc tbc 2 3 2 tbc

GAD7

anxiety 2 tbc tbc 2 3 2 tbc

IRS

impulsivity 2 0 5 2 2 2 0

FMQ

family coping tbc tbc tbc 2 tbc 2 0

PCQ

parenting 1 1 2 2 3 2 0

EQ5D weighted scores baseline and 3mth higher score = better health weighted score max = 1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

All Alcohol Heroin & Opiates

Methadone Other drug Population 25-35

Baseline Follow up Gen Pop

Source: unpublished clinical data

Frequency of scores at assessment and 3mth

LDQ CORE SSQ

Source: unpublished clinical data

Conclusion The EQ5D LDQ CORE SSQ are clinically useful, used in routine

practice, have predictive value and paint a comprehensive

picture....

0

20

40

60

80

100

0

5

10

15

20

25

30

Your score now…

…this is what happens

in treatment

Interesting Things to Find Out

Domains (or components) of Addiction

psychological well being

dependence

social well being

substance use

Factor analysis with #4 factor solution

EQ5D LDQ CORE-10 SSQ

F#1 F#2 F#3 F#4 F#1 F#2 F#3 F#4 F#1 F#2 F#3 F#4 F#1 F#2 F#3 F#4

.77 .13 .07 -.09 .10 .80 .25 -.10 .19 .36 .71 -.12 .11 -.04 -.01 .66

.67 -.02 .15 -.08 .12 .81 .24 -.15 -.15 -.01 .15 -.40 -.01 -.05 -.04 .70

.57 .28 .31 -.15 .12 .80 .27 -.10 .02 .26 .44 -.22 -.15 -.19 -.03 .57

.68 .22 .08 -.07 .11 .83 .20 -.11 .04 .19 .60 -.14 -.05 -.22 -.06 .59

.25 .31 .69 -.12 .18 .65 .20 -.11 .28 .25 .75 -.10 -.17 -.13 -.30 .56

.15 .75 .18 -.13 .34 .14 .57 -.20 -.11 -.07 -.30 .60

.04 .75 .25 -.08 .26 .30 .55 -.11 -.06 -.05 -.20 .64

.04 .66 .22 -.12 .22 .31 .77 -.22 -.02 -.11 -.25 .57

.09 .78 .22 -.12 .20 .30 .73 -.23

.15 .74 .35 -.13 .27 .22 .69 -.17

8% of variance 23% of variance 17% of variance 9% of variance

Relationship between EQ5D LDQ CORE SSQ

LDQ and CORE close to

each other. Predicts if

dependence is treated then

mental health improves.

SSQ is most separated.

Predicts it will change

most differently to other

measures.

EQ5D is generic but

independent and

between the other

measures. Has less

influence than LDQ on

mental health more on

social satisfaction.

Clinically Significant Change - ‘Gold Standard’

Reliable Change

Score

Well Functioning

Population

LDQ >= 4 < 12

CORE-10 >= 6 < 14

SSQ >= 4 > 10

Source: CLAHRC submitted

Jacobson et al. (1999) proposed that in order to take account of baseline

scores and measuring error, clinically significant change should a) be

statistically reliable b) end scores be in a well functioning population range

reteste r1SS 1

2ediff )2(SS

RC = reliable change 95% probability if RC >=1.28

Sdiff = standard error of difference between means of LDQ

scores

Se = standard error of measurement of LDQ

S1 = standard deviation of mean1

rretest = test/retest reliabilty of LDQ

Clinically Significant Change - example

0

10

20

30

40

50

60

70

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

LDQ scores

reliable change >=4 if pushed into

well functioning population range

then Clinically Significant Change

achieved

well functioning population is 2standard

deviation above general population

mean LDQ<12

Reliable (RC) and clinically significant (CS) change (n-925)....

n=925 LDQ % CORE % SSQ %

RC improved 61.0 46.0 36.7

RC worse 4.6 5.7 12.0

Too small for RC 14.3 6.1 5.2

CS improvement 50.1 30.5 31.9

clinically significant change at 3mth (drinking n=396).....

CS change % Yes No Yes No Yes No

Drinking 22.0 31.1 13.4 39.6 16.7 36.4

Abstinent 35.4 11.6 24.2 22.7 17.7 29.3

P<.001 P<.001 n.s.

95%

probability

of real

change social

situation

most difficult

to recover abstinence

associated

with the most

CSC. LDQ

works across

drinking

outcomes

CSC is a

tough test of

improvement

The GOLD

standard

Thank you. Questions?

Conclusion

multiple measures are needed to answer different stakeholder

questions

always include substance misuse

avoid composite measures – tailor to suit the agency

choose scales that meet Quality Framework criteria

choose clinically useful measures

add Societal Impact Measures

check that the package is ethical and easy for routine use