UNPACKING THE BLACK BOXENGINEERING MORE POTENT BEHAVIORALINTERVENTIONS TO IMPROVE PUBLIC HEALTH

Linda M. Collins, Ph.D.

Outline What is a behavioral intervention? What is the black box, and why unpack it? What is optimization? A few examples Concluding remarks

What is a behavioral intervention?

Definition: A program aimed at modifying behavior for the purpose of preventing/treating disease, promoting health, and/or enhancing well-being.

What is a behavioral intervention?

A few examples from PSU (out of many):

PROSPER (Greenberg, Bierman, Feinberg, Welsh, Perkins, Mincemoyer, Corbin)

Keepin’ it REAL (Hecht, Miller-Day, Graham)

Project ACT (Turrisi)

Active MOMS (Downs)

Interventions for at-risk caregivers (Zarit)

Head Start REDI (Bierman, Domitrovich)

HealthWise (Caldwell, Smith)

RESERVE (Kolanowski, Fick)

What is a behavioral intervention?

Most behavioral interventions are made up of multiple components.

Some components may be pharmaceutical or medical.

Outline What is a behavioral intervention? What is the black box, and why unpack it? What is optimization? A few examples Concluding remarks

Treatment package approach

component

component

component

RCT

component component

What is the black box, and why unpack it?

If RCT finds significant effect, it is UNKNOWN Which components are making positive

contribution to overall effect Whether all the components are really needed Whether a component’s contribution offsets its

cost How to make the intervention more effective,

efficient, cost-effective

What is the black box, and why unpack it?

If RCT finds non-significant effect, it is UNKNOWN Whether any components in the box are worth

retaining

Whether one component in the box had a negative effect that offset the positive effect of others

Specifically what went wrong and how to do it better the next time

What is the black box, and why unpack it?

The treatment package approach Encourages stuffing the black box with as

many components as possible to get a significant effect

Downplays considerations such as efficiency, cost-effectiveness, time-effectiveness

Places focus on attaining statistical significance rather than meeting a criterion

What is the black box, and why unpack it?

This is NOT how engineers build products. They take an approach that is

► Systematic

► Efficient

► Focused on the clear objective of optimizing the product

Can we borrow ideas from engineering… … and build optimized behavioral interventions?

Resource management principle How engineers think: This is what I need to find out: ______ These are the resources I have: ______ How can I manage my resources strategically

to find out what I need to know?

Resource management principle Logic: Objective is to identify ONE OF THE

TWO OR THREE BEST approaches

0

1

2

3

4

5

6

7

8

9

10

Hypothetical distribution of outcome variable for different combinations of components

Score on outcome variable

Freq

uenc

y

Resource management principle Logic: Objective is to identify ONE OF THE

TWO OR THREE BEST approaches Manage research resources strategically

► Decide what information most important, and target resources there

Choose efficient experimental designs Take calculated risks

Resource management principle Note that the starting point is the resources

you have By definition, MOST does not require an

increase in research resources► But may require a realignment of research

resources

The Multiphase Optimization Strategy (MOST)

component

component

component

Optimizedintervention

component

component

component

RCT

component component

Screening (component

selection) and refinement

experiment(s)

Note: MOST is a framework, not an off-the-shelf procedure.

Outline What is a behavioral intervention? What is the black box, and why unpack it? What is optimization? A few examples Concluding remarks

Definition of optimized “The best possible solution… subject to given

constraints” [emphasis added] (The Concise Oxford Dictionary of Mathematics)

► Optimized does not mean best in an absolute or ideal sense

► Instead, realistic because it includes constraints

Optimization always involves a clearly stated optimization criterion

► A working definition of what YOU mean by “better”

Selecting an optimization criterion: what you mean by “best”

Your definition of “best possible, given constraints”

This is the goal you want to achieve Suppose you are developing a behavioral

intervention to encourage HAART adherence in HIV+ people called “Living with HIV”

One possible optimization criterion: Intervention with no “dead wood”

► Example: Health care settings are finding it difficult to fit Living with HIV into their busy day, and are watching costs carefully. The investigators want to be confident that every component is necessary so that no time or money is wasted.

► Achieve this by selecting only active intervention components.

Another possible optimization criterion:

Most effective intervention that can be delivered for ≤ some $$

► Example: To have a realistic chance of being adopted by HMOs, Living with HIV must cost no more than $200/participant to deliver, including materials and staff time.

► Achieve this by selecting set of components that represents the most effective intervention that can be delivered for ≤ $200.

Another possible optimization criterion:

Most effective intervention that can be delivered in ≤ some amount of time

► Example: Interviews with health care clinic staff suggest that Living with HIV has the best chance of being implemented well if it takes no more than 15 minutes to deliver.

► Achieve this by selecting set of components that represents the most effective intervention that can be delivered in ≤ 15 minutes.

Evaluation and optimization:Both important;

not the same thing.

Evaluation: Is the intervention’s effect

statistically significant?Optimization: Is the intervention the best possible, given constraints?

No Yes

NoMay wish to

optimize using effect size as

criterion

Intervention can probably be

improved

Yes Different intervention

strategy neededWhat we should

be aiming for

Outline What is a behavioral intervention? What is the black box, and why unpack it? What is optimization? A few examples Concluding remarks

Three examples of MOST All currently in the field All three have the same optimization criterion:

No inactive components

Example 1: Clinic-based smoking cessation study funded by NCI

Tim Baker Mike Fiore University of Wisconsin

Team also includes B.A. Christiansen, L.M. Collins, J.W. Cook, D.E. Jorenby, R.J. Mermelstein, M.E. Piper, T.R. Schlam, S.S. Smith

Project funded by NCI grant P50CA143188

Example 1: Clinic-based smoking cessation study funded by NCI

Tim Baker Mike Fiore University of Wisconsin

Objective: Develop an effective “lean” clinic-based smoking cessation intervention (no inactive

components)

Some interesting features of Example 1

Study being implemented in health care settings

Involves both behavioral and pharmaceutical components

Baker & Fiore’s model of the smoking cessation process

PRECESSATION(3 weeks prior up to

quit day)

CESSATION(quit day to 2 weeks

after quit day)

MAINTENANCE(2 weeks to 6 months

after quit day)MOTIVATION

Six components being considered for the smoking cessation intervention

Precessation nicotine patch (No, Yes) Precessation ad lib nicotine gum (No, Yes) Precessation in-person counseling (No, Yes) Cessation in-person counseling (Minimal,

Intensive) Cessation phone counseling (Minimal, Intensive) Maintenance medication duration (Short, Long)

Experiment to examine individual component effects

We decided to conduct a factorial experiment► Special type called a fractional factorial

N=512 subjects TOTAL provides power ≥ .8

Engineering the intervention Experiment will give us

► Main effect of each individual intervention component on outcomes► e.g. number days abstinent in 2-wk post-quit period

► Selected interactions between intervention components

This information will be used to select components/component levels

Result: optimized clinic-based smoking cessation intervention

Plan to conduct an RCT to establish statistical significance of effect

Example 2: School-based drug abuse/HIV prevention study funded by NIDA

Linda L. Caldwell Edward A. SmithPenn State

Team also includes D. Coffman, L. Collins, J. Cox, I. Evans, J. Graham, M. Greenberg, J. Jacobs, D. Jones, M. Lai, C. Matthews, R. Spoth, L. Wegner, T. Vergnani, E. Weybright

Project funded by NIDA grant R01DA029084

Example 2: School-based drug abuse/HIV prevention study funded by NIDA

Linda L. Caldwell Edward A. SmithPenn State

Objective: To develop a strategy for maintaining implementation fidelity in which all components contribute

Some interesting features of Example 2

Components being examined relate to how the intervention is delivered

► “sealed intervention”

Cluster randomization

Background HealthWise school-based ATOD/HIV prevention

intervention Has previously been evaluated in South Africa Metropolitan South Education District in South

Africa wants to implement HealthWise in all its schools

Question: how to maintain fidelity? Metro South allowed us to conduct an experiment

Caldwell & Smith’s model of implementation fidelity

Components Enhanced teacher training

► Standard training (one and one-half days) vs. enhanced (three days + two additional days four months later)

Structure, support, and supervision► No additional vs. additional (e.g., weekly text messages; monthly

visits from support staff; option to call support staff with questions as needed)

Enhanced school climate► No climate enhancement vs. climate enhancement (e.g., form

committee of parents and teachers to promote HealthWise; develop visuals; issue newsletter)

How to conduct an experiment to examine individual component

effects We decided to conduct a factorial experiment. Why? Enables examination of individual component effects

AND Statistical power achieved with smaller sample sizes

than alternative designs► Yes, I mean smaller

BUT they also usually require more experimental conditions than we may be accustomed to

Experiment uses all 56 schools in district

Factorial experiments 101 Example: 2 X 2, or 22, factorial design

Factorial experiments can have► ≥ 2 factors

► ≥ 2 levels per factor

Component A

Component B Off On

Off A,B off A on, B off

On A off, B on A,B on

Experi-mental

conditionN of

schoolsHealthWise

program Training

Structure, support, & supervision

Enhanced school climate

1 7 2 7 3 7 4 7 5 7 6 7 7 7 8 7

HealthWise experiment in South African school district. 56 schools in all; 7 schools assigned to each experimental condition

Are 7 schools per experimental condition enough?

We estimated power ≥ .8 for main effects Remember that each main effect estimate is

based on ALL schools In a factorial experiment you DO NOT

compare individual conditions

Experi-mental

conditionN of

schoolsHealthWise

program Training

Structure, support, & supervision

Enhanced school climate

1 7 2 7 3 7 4 7 5 7 6 7 7 7 8 7

Main effect of Training is mean of (5,6,7,8) vs. mean of (1,2,3,4).

Note that all 56 schools are used in estimating the main effect.

Main effect of Structure, support, & supervision is mean of (3,4,7,8) vs. mean of (1,2,5,6).

Note that all 56 schools are used in estimating the main effect.

Experi-mental

conditionN of

schoolsHealthWise

program Training

Structure, support, & supervision

Enhanced school climate

1 7 2 7 3 7 4 7 5 7 6 7 7 7 8 7

Experi-mental

conditionN of

schoolsHealthWise

program Training

Structure, support, & supervision

Enhanced school climate

1 7 2 7 3 7 4 7 5 7 6 7 7 7 8 7

Main effect of Enhanced school climate is mean of (2.4.6.8) vs. mean of (1,3,5,7).

Note that all 56 schools are used in estimating the main effect.

Engineering the intervention Experiment will give us

► Main effect of each individual intervention component on outcome variables

► Also interactions between intervention components

This information will be used to select the best set of the three components

Result: Intervention engineered to optimize fidelity according to our criterion

Note: optimized ≠ the best possible

Example 3: Internet-delivered drug abuse prevention program aimed at NCAA

athletes

David Wyrick UNC Greensboro

Prevention Strategies, LLC

Melodie Fearnow-KenneyPrevention Strategies, LLC

Team includes L. M. Collins, J. Milroy, K. RulisonProject funded by NIDA grant R44DA023735

Example 3: Internet-delivered drug abuse prevention program aimed at NCAA

athletes

OBJECTIVE: No inactive components. Secondary aim: Maximize overall program effect within available research

resources.

David Wyrick UNC Greensboro

Prevention Strategies, LLC

Melodie Fearnow-KenneyPrevention Strategies, LLC

Some interesting features of Example 3

Intervention is internet-delivered Experimental design is a cluster-randomized

fractional factorial Takes an iterative approach

Background of MyPlaybook Study

Theoretical Model for MyPlaybook

Social NormsSocial Norms Theory

Positive & Negative Expectancies

Health Belief Model

Intentions to Avoid Use and Prevent Harm

Theory of Planned Behavior

Alcohol and Other Drug Use

Consequences

Intervention components Five components, each corresponding to a substance:

I. Alcohol

II. Tobacco

III. Marijuana

IV. Performance enhancers

V. Prescription and OTC drugs Each component aimed at

► social norms

► positive and negative expectancies

► intentions to avoid use and prevent harm

MOST as implemented in the My Playbook study

Optimizedintervention

Alcohol

Performance enhancers

Tobacco

RCT

Marijuana

Prescription/OTC drugs

Experiment 1

Revision of componentsExperiment 2

Design of MyPlaybook experiments: 25-1 fractional factorial. 64 schools assigned to conditions; overall N approx. 3500

Condition Number

Intro Component

Intervention Components Manipulated

I II III IV V

1 On Off Off Off Off On

2 On Off Off Off On Off

3 On Off Off On Off Off

4 On Off Off On On On

5 On Off On Off Off Off

6 On Off On Off On On

7 On Off On On Off On

8 On Off On On On Off

9 On On Off Off Off Off

10 On On Off Off On On

11 On On Off On Off On

12 On On Off On On Off

13 On On On Off Off On

14 On On On Off On On

15 On On On On Off Off

16 On On On On On On

Engineering the intervention Each experiment will give us

► Main effect of each individual intervention component on outcome variables

► e.g. substance use at 30-day follow-up

► Also selected interactions between intervention components

We will select the best version of each component for inclusion in the optimized intervention

The optimized intervention will be evaluated via an RCT

Outline What is a behavioral intervention? What is the black box, and why unpack it? What is optimization? A few examples Concluding remarks

Frequently asked questions Nice idea, but will it be fundable? Can this approach be carried out with the level of

funding typically available? Will I be able to publish based on the results of the

screening experiments? I don’t see how I can implement all the experimental

conditions required by a factorial experiment. What if the outcome of interest is in the distant

future?

If you are interested in learning more about MOST

FACULTY► “A Taste of Methodology” brief workshop, Monday,

April 30, 10:30 am to 1:30 pm (co-sponsored by the Methodology Center and SSRI; includes lunch)

GRAD STUDENTS► HDFS 597x Quantitative Methods for Intervention

Science

The future I would like to see Behavioral interventions systematically

engineered to meet specific criteria Standards of intervention effectiveness,

efficiency, cost-effectiveness Cumulative base of scientific knowledge Behavioral intervention engineering a subfield

in industrial engineering

THANK YOU!