Childhood Obesity Scenario: Quasi-Experiments and Natural Experiments

Versus RCTs

Steven Gortmaker, Ph.D.

Harvard School of Public Health /Harvard Prevention Research Center

Definitions

• Quasi-experiments: can have all the attributes of randomized controlled trial with pretest and posttest data; key difference: no random assignment to intervention versus control

• Natural experiment: can have all the attributes of randomized controlled trial with pretest and posttest data; key differences: – no random assignment

– experimenter does not control intervention

The Research Question

• Can be same in Quasi-experiment and Natural experiment as in Randomized Controlled Trial

• However, random assignment can set limits:

– Long lead time for the study/results; may preclude rapid evaluations of innovations

– Funding agency often pays for the intervention - may preclude evaluations of expensive or complex interventions or policy changes

Evaluation Data

• Quasi/Natural Experiment

– Quasi-experiment has similar data requirements to RCT

– Natural experiments may need to rely on surveillance data

• Group Randomized Trial– Want pre-post data on

key measures

Internal Validity: Control of Selection/Confounding

• Quasi/Natural Experiment

– Success dependent on selection of control sample (e.g. propensity matching)

– Multiple pre-intervention and post-intervention data points can strengthen design

• Group Randomized Trial– The major strength of

RCT’s

– Multiple pre-intervention and post-intervention data points can strengthen design

Hypothetical Quasi Experimental Design to Evaluate Impact of School Food Service Change, With Single Data

Point Pre and Post-Intervention

0

10

20

30

2002 2003

Year

Mean

BM

I

Intervention Begins

Control

Intervention

Hypothetical Quasi Experimental Design to Evaluate Impact of School Food Service Change, With Multiple

Data Points Pre and Post-Intervention

0

10

20

30

0 1 2 3 4 5

Year

Mean B

MI

Intervention Begins

Control

Intervention

Loss to Follow-up

• Quasi/Natural Experiment

– Similar concerns to RCTS

– Can study predictors of loss, model loss

– Natural experiment: a concern is lack of detailed baseline data

• Group Randomized Trial– As with Quasi-experiment,

can study predictors of loss, model loss

External Validity: Generalizability

• Quasi/Natural Experiment

– Quasi-experiment: similar to RCT but may have broader sample of participant sites

– Natural experiment: concerns re selection of intervention sites

• Group Randomized Trial– Participating sites may

differ substantially from target population

– As with Quasi-experiment, can study participant sites versus non-participants

Intervention Costs

• Quasi/Natural Experiment

– Quasi-experiment: can be similar to RCT, or can capitalize on other funding

– Natural experiment: intervention costs generally not borne by funding agency

• Group Randomized Trial– Intervention costs often

borne by funding agency; can limit cost of intervention program

– If costs borne by outside agency, difficulty with randomization (but not always!)

Evaluation Costs

• Quasi/Natural Experiment

– Quasi-experiment: can be similar to RCT re data collection

– Natural experiment: can be less expensive by using extant data

• Group Randomized Trial– Similar data collection

costs re Quasi-experiment.

– A hypothesis: typically data collection costs in RCTs are much larger than intervention costs

Summary of Trade-Offs

• Quasi/Natural Experiment– Worse internal validity; but

more data points can help

– Some limited generalizability - particularly in natural experiments

– Potential to study more innovative, expensive, difficult to implement programs/policies

• Group Randomized Trial– Better internal validity – Often limited

generalizabilty; but potential to improve

– More limited programs/policies to study with RCT due to costs and difficulty of randomizing