Value for money

How to design RCTs to ensure their compatibility with economic evaluation

September 2006

Kevin Marsh

Introduction and overview

Challenges

• measuring the cost of interventions

• standardised measure of effects

• generalising

Source of evidence

• two recent NICE public health reviews

Question:

• what are the methodological lessons from attempts to build economic evaluations on RCTs?

2 requests

1 question

Why are we concerned with value for money?

Campbell Collaboration Economic Methods Group (CCEMG) Policy Brief (in press):

Provision of evidence on cost-effectiveness, alongside evidence on the benefits and adverse effects of interventions, can significantly enhance […] decision-making by managers and policy makers.

Effectiveness plane

y x

+ve-veeffectiveness

Should we invest in intervention ‘x’?

• yes – ‘x’ is more effective than ‘y’

• but what is the opportunity cost of ‘x’?

Cost-effectiveness plane

y +ve-veeffectiveness

+ve

-ve

cost

Cost-effectiveness plane

y +ve-veeffectiveness

+ve

-ve

cost

Should we invest in ‘x’?

• yes - ‘x’ dominates ‘y’

Cost-effectiveness plane

y +ve-veeffectiveness

+ve

-ve

cost

Should we invest in ‘x’?

• no - ‘y’ dominates ‘x’

Cost-effectiveness plane

y +ve-veeffectiveness

+ve

-ve

cost

Should we invest in ‘x’?

• ? – cost-effect trade off?

Aren’t we already doing economic evaluation?

Review Effect

studies Econ

studies Ratio

Promotion of physical activity 24 8 3:1

Community-based substance misuse prevention in young vulnerable people

222 5 44.4:1

Number of studies identified in 2 recent reviews of effectiveness and economic studies

How do we model value for money from RCTs?

Intervention

£ pp

risk factor (e.g. cigarette use)

probability substance misuse

probability problematic SM

health, crime, unemploy.

£ saved pp QALY gain pp

How do we model value for money from RCTs?

Intervention

£ pp

risk factor (e.g. cigarette use)

probability substance misuse

probability problematic SM

health, crime, unemploy.

£ saved pp QALY gain pp

(£ pp- £ saved pp) / QALY gained pp

Cost / risk factor

How do we model value for money from RCTs?

Intervention

£ pp

risk factor (e.g. cigarette use)

probability substance misuse

probability problematic SM

health, crime, unemploy.

£ saved pp QALY gain pp

Review of effectiveness

(RCT)

How do we model value for money from RCTs?

Step 1: cost intervention?

Intervention

£ pp

risk factor (e.g. cigarette use)

probability substance misuse

probability problematic SM

health, crime, unemploy.

£ saved pp QALY gain pp

Good description of a simple intervention

Facilitate referral to an external treatment agency  

1. discuss negative consequences of personal drug use

2. discuss impediments to reducing negative consequences

3. advice about appropriate services

4. discuss possible impediments to treatment

5. arrange the first appointment with a named counselor

6. reminder telephone call prior to the first appointment

7. offer of transport to first appointment

8. offer to accompany then to their first appointment

Example: intervention description

1. Who provides the intervention?

Facilitate referral to an external treatment agency  

1. discuss negative consequences of personal drug use

2. discuss impediments to reducing negative consequences

3. advice about appropriate services

4. discuss possible impediments to treatment

5. arrange the first appointment with a named counselor

6. reminder telephone call prior to the first appointment

7. offer of transport to first appointment

8. offer to accompany then to their first appointment

Example: intervention description

2. How long does each element take?

Facilitate referral to an external treatment agency  

1. discuss negative consequences of personal drug use

2. discuss impediments to reducing negative consequences

3. advice about appropriate services

4. discuss possible impediments to treatment

5. arrange the first appointment with a named counselor

6. reminder telephone call prior to the first appointment

7. offer of transport to first appointment

8. offer to accompany then to their first appointment

Example: intervention description

3. % take up optional elements?

Facilitate referral to an external treatment agency  

1. discuss negative consequences of personal drug use

2. discuss impediments to reducing negative consequences

3. advice about appropriate services

4. discuss possible impediments to treatment

5. arrange the first appointment with a named counselor

6. reminder telephone call prior to the first appointment

7. offer of transport to first appointment

8. offer to accompany then to their first appointment

Request 1: resource use data

1. describe resource use: who does what, what equipment?

2. measure resource use: hours, units?

3. value resource use: £

How do we model value for money from RCTs?

Step 2: CE?

Intervention

£ pp

risk factor (e.g. cigarette use)

probability substance misuse

probability problematic SM

health, crime, unemploy.

£ saved pp QALY gain pp

Step 2: calculating cost-effectiveness

Intervention 1:

Cost - £100

Effect – YP not use cannabis in last month when would otherwise

Intervention 2:

Cost - £100

Effect – YP move from use cannabis 20 times/yr to only 5 times/yr

How do we model value for money from RCTs?

Step 3: long-term effect?

Intervention

£ pp

risk factor (e.g. cigarette use)

probability substance misuse

probability problematic SM

health, crime, unemploy.

£ saved pp QALY gain pp

Step 3: modelling long-term effect

RCT:

Change in YP risk factors

Epidemiological:

Impact risk factors on LR sub. misuse

Use cannabis > 31 times/yr? Y/N

Same risk factors

Different measures

E.g. Use cannabis in the last month? Y/N ?

NICE econ review: matching effect studies to epidemiological data

Population type Results of

effectiveness review

Quality and effect

criteria

Outcome variable criteria

General at risk 98 17 3

Behavioural / aggressive young people

7 0 0

BME 46 8 3

Young offenders 10 0 0

Young substance users 23 3 0

Families with drug using members

18 3 0

School drop-outs, truants and underachievers

11 0 0

Other 9 4 0

222 35 6

6 only model fraction of the effects identified

Request 2: use standard measures of effect

• compare between studies

• link with epidemiological data

How generalisable are our results?

1. RCTs: conflicting + inconsistent results

2. Heterogeneity

How generalisable are our results?

Study Intervention Population / inclusion

criteria Context Location

Griffin et al

(2003)

Life Skills Training (LST) v

normal educational curriculum

General at risk:

Low grades in school

Peers use substances

School New York,

USA

Hawkins et al

(1999)

Teacher training, parent

education, and social

competence training for

children v normal educational

curriculum

General at risk:

School

Seattle and

Washington,

USA

Zavela et al

(2004)

‘Say Yes First’ v normal

educational curriculum General at risk School

Colorado,

USA

Botvin et al

(1995)

Life Skills Training (LST) v

Normal educational curriculum

(information on drugs)

BME group:

students in six New York

City public schools who

consented to participate

School New York,

USA

Botvin et al

(2001)

Life Skills Training (LST) v

Normal educational curriculum

(information on drugs)

BME group:

Seventh graders from 29

New York schools. School

New York,

USA

Campbell et al

(2002)

The Abecedarian Project v

normal child rearing

BME group:

High risk index created

from 13 socio-demographic factors

School /

home USA

Question: how do we overcome the challenge of heterogeneity?

Can an RCT only tell us about intervention X if implemented in way A, for population B, in social context C?

If yes

RCT for each combination of

intervention, control,

population, context?

Alternative method?

Quasi-exp?

Other?