1

Lecture 20: Non-experimental studies of interventions

• Describe the levels of evaluation (structure, process, outcome) and give examples of measures of each level

• Describe the applications of cohort and case-control designs to the evaluation of interventions.

• Describe advantages and disadvantages of randomization versus:

• - Historical controls• - Simultaneous, non-randomized controls• Describe the following quasi-experimental designs:• - Time series (trend) design• - Non-equivalent control group design

2

Design of an intervention study

• Study objectives:– Define intervention– Define target population – Define evaluation measures

• Study design:– Experimental– Non-experimental

3

Levels of evaluation• STRUCTURE:

– Drugs, devices, staff, equipment needed to provide intervention

• PROCESS:– Interaction between structure and patient/client– Adherence/compliance

• OUTCOMES: – Expected or unexpected results, positive or negative, e.g.:

• Death, disease, disability• Attitudes, behaviors • Costs

4

Levels of evaluation• Create hypothetical diagram linking

structure, process, and outcome

• Based on goals of study, select measures of structure, process, and/or outcome

5

Levels of evaluation: example• Hypothetical diagram:

– HIV/AIDS educational intervention for drug injectors (describe planned structure)

Process (attendance/quality of participation)Outcome 1: Improved knowledge/attitudesOutcome 2: Lower risk behaviourOutcome 3: Lower HIV incidence rate

6

Levels of evaluation• Example:

– Exercise program to reduce CHD risk

• STRUCTURE?

• PROCESS?

• OUTCOMES?

7

Epidemiological observational study designs

• Cohort and case-control studies• Independent and dependent variables:• Studies of risk factors:

– independent variable (exposure): risk factor – dependent variable: disease

• Studies of interventions:– independent variable (exposure): intervention – dependent variable(s): selected “outcomes” (could be

measures of process and/or outcomes)

8

Cohort study

• Study population:– Cohorts with and without “exposure” to

intervention (or different levels of exposure)– Control (unexposed) cohort - concurrent or

historical• confounding by changes over tine in patient

population, aspects of treatment other than intervention; measurement of confounders

• Follow-up to measure outcomes

9

Cohort study

• Selection of controls: could they receive either treatment?

• Example: medical vs surgical treatment of CHD

• Some sources of bias:– Selection bias– Information bias: detection bias, other– Confounding: by indication, other

10

Examples of cohort studies• Effectiveness of new cancer treatment

– Historical controls

• Do HMOs reduce hospitalization in terminal cancer patients, during 6 months before death?

– Administrative databases and tumor registry from Rochester NY

– Cancer deaths in 100 pairs of HMO members and non-members

– Matched by age, cancer site, months from diagnosis to death

11

12

Case-control study

• Study population:– Cases (with outcome)

– Controls (without outcome) Limited to single, categorical outcome

• Data collected on prior “exposure” to intervention • Some sources of bias

– Selection bias

– Information bias

– Confounding: by indication, other

13

Case-control study: Examples

• Screening programs:– screening Pap test and invasive cervical cancer– screening mammography and breast cancer

deaths– screening sigmoidoscopy and colon cancer

deaths

• Vaccine effectiveness (e.g., BCG)

• Neonatal intensive care and neonatal deaths

14

Quasi-experimental study designs

• Investigator has “some control” over timing or allocation of intervention – Non-randomized or quasi-randomized trials– Non-equivalent control group designs:

• pre-test and post-test

• post-test only

– Time series designs• single or muliple

15

Diagramming Intervention Study (Evaluation) Designs

Campbell and Stanley

• X = program

• O = measurement

• R = randomization

16

Randomized (Experimental) Designs

• Randomized pre-test post-test control group design

R O1 X O2

R O3 O4

• Post-test only control group design

R X O1

R O2

17

Some Weak Observational Designs: Cross-sectional

• One-shot case-study

X O

• Static group comparison:

X O1

O3

18

Some Weak Observational Designs: Longitudinal

• Before-after (pre-post) study

O1 X O2

19

Some quasi-experimental designs: with control/comparison group

Pre-test post-test non-equivalent controlgroup design

O1 X O2

O3 O4

Recurrent institutional cycle

X O1

O2 X O3

20

Health insurance in Quebec• 1961: universal hospital insurance

– included ER care for accidents

• 1970: universal health insurance (Medicare)

– added MD care including hospital outpatient clinics and ERs

• Population surveys before and after

• Effects on:

– use of physician services by general population

– physician workload

– use of emergency rooms

– hospitalization and surgery

21

MD visits/person/year by income(household surveys)

0

1

2

3

4

5

6

7

8

All visits <3000 3000- 5000- 9000- 15000+

PrePost

22

MD visits/person/year (household surveys)

0

1

2

3

4

5

6

All visits Office ODP/ER Home

PrePost

23

MD visits/person/year by income(household surveys)

0

1

2

3

4

5

6

7

8

All visits <3000 3000- 5000- 9000- 15000+

PrePost

24

% adults with cough 2+ weeks who consulted MD (household surveys)

0

10

20

30

40

50

60

70

<$5000 $5000- $9,000 Total

PrePost

25

% children (<17) with tonsilitis or sore throat and fever who consulted MD

(household surveys)

0

10

20

30

40

50

60

70

80

<$5000 $5000- $9,000 Total

PrePost

26

% pregnancies with visit in first trimester (household survey)

0

10

20

30

40

50

60

<$5000 $5000- $9,000 Total

PrePost

27

% Tried to contact MD before ED visit; of these, % successful (6 hospital sample)

010203040506070

Tri

ed t

o co

ntac

t

Spok

e to

MD

Off

ice/

answ

erin

gm

achi

ne

Uns

ucce

ssfu

l

PrePost

28

Examples of pre-post non-equivalent control group design

• Stanford 5-city study of CHD prevention

• Intervention included mass media education and group interventions for high-risk

• 5 cities selected - similar characteristics – those with shared media market were allocated

to intervention – isolated cities allocated to control group

29

Time series designs

Time series desgn

O1 02 O3 X O4 O5 O6

Multiple time series design

O1 O 2 O 3 X O 4 O 5 O 6

O7 O8 O9 O10 O11 O12

30

Example of time series study:Tamblyn et al, 2001

• Evaluation of prescription drug cost-sharing among poor and elderly

• Methods:– Trend study: Multiple pre- and post-

measurements– Cohort study:

31

Source: Tamblyn et al, JAMA 2001, 285(4): 421-429

32

Source: Tamblyn et al, JAMA 2001, 285(4): 421-429

33

Time-series design: Home care in terminal cancer

• Evaluation of home-hospice programme in Rochester, NY

• Expansion of home-care benefits in 1978

• Hypothesis: home-hospice care in last month of life reduces hospital days and costs

• Data sources: Linkage of tumor registry and health insurance claims databases

34

35

36

Differences between quasi-experimental and epidemiological cohort study designs

• Quasi-experimental designs often use ecological rather than individual level of measurement

• Serial cross-sectional studies over time vs follow-up of individuals:– advantages and disadvantages?