designs and types of evidence for dissemination and...
TRANSCRIPT
![Page 1: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/1.jpg)
Designs and Types of Evidence for Dissemination and
Implementation Research C Hendricks Brown
University of Miami Miller School of Medicine
Director, Center for Prevention Implementation Methodology
Director, Prevention Science and Methodology Group Director, Social systems Informatics
chbrown@med miami edu
1
![Page 2: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/2.jpg)
Acknowledgements
• Center for Prevention Implementation Methodology (Ce-PIM) for Drug Abuse and HIV Sexual Risk Behavior (NIDA) PI C Hendricks Brown
• Advanced Center to Improve Pediatric Mental Health Care (P30 MH074678) aka Implementation Methods Research Group (IMRG) PI John Landsverk
• Scaling up MTFC in California NIMH: R01MH076158, PI Patti Chamberlain
• Methods for MH/DA Prevention and Early Intervention, NIMH/NIDA R01-MH040859, PI C Hendricks Brown
• Collaborative Synthesis of Adolescent Depression Trials, NIMH R01-MH040859-
22 PI C Hendricks Brown • NIDA R01 DA019984 A Follow-up of Classroom Services to Prevent Drug Use,
PI Jeanne Poduska,
• NIDA R21DA024370, Scaling-up Prevention Services for Early Drug Abuse Risk in School Systems, PI Jeanne Poduska
2
![Page 3: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/3.jpg)
Acknowledgements Co-Authors John Landsverk, Rady Children’s Hospital Patricia Chamberlain, Oregon Social Learning
Center Lawrence Palinkas, USC Wei Wang, University of South Florida Lisa Saldana, Center for Research to Practice Jeremy Goldhaber-Fiebert, Stanford Sarah Horwitz, Stanford
3 NIH Training Institute for D&I Research Ce-PIM
![Page 4: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/4.jpg)
Outline I. Factors Affecting Design of the D & I Study II. Three Dimensions for Typing Study
Designs for Implementation Research Illustrations - Microsimulation
III. Increasing the “Evidence” in Implementation Research
Roll-Out Randomized Implementation Trials Example of the CAL-OH Randomized
Implementation Trial IV. Summary
4 NIH Training Institute for D&I Research Ce-PIM
![Page 5: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/5.jpg)
Where We Are in the Science of Implementation
• Just beginning • All of this is in flux • Role of Methodology Similar to That for
Prevention Science 20 Years Ago IOM Preventing Mental, Emotional, & Behav Disorders 2009
5
![Page 6: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/6.jpg)
I. Factors Affecting Design of the D & I Study
• Mutually Agreed Upon Research Question
6
![Page 7: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/7.jpg)
System to Support Adoption
1. What designs are used in implementation studies?
Changing Research Questions Effectiveness vs Implementation
7
Intervention Intervention
System to Support Adoption
NIH Training Institute for D&I Research Ce-PIM
![Page 8: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/8.jpg)
General Influences on the Design of a Study
NIH Training Institute for D&I Research Ce-PIM 8
Determine Research Question(s) Research Team
Institutional/Community other Stakeholders & Partners
Intervention Conditions / Implementation Strategies
Population
Measures and Follow-up/ Assessment Procedures
Sampling, Sample Size and Enrolling
Assignment to Intervention Condition / Implementation Strategies
Ethics/Human Subjects/Values Financial/Logistic
s
Policy Makers, Funders
![Page 9: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/9.jpg)
Handy Terminology
• Implementation (or Dissemination) Strategy – system-level program:
Glisson’s ARC Hawkins Communities that Care Community Development Team (CiMH) • Intervention – evidence-based program
9
![Page 10: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/10.jpg)
Illustrative Classes of Research Questions for D&I Research and Common Types of Designs
1. What System Level Factors Impede or Facilitate implementation?
Descriptive Epidemiologic - Cross-Sectional Do More Evidence-Based Organizations
Adopt? 2. What implementation strategies lead to faster uptake, successful adaptation and sustained effects on target population? Interventional (Testing) -- Experimental 3. How does a Strategy Affect Implementation? Etiologic (Mediational) -- Multilevel Longitudinal
10
![Page 11: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/11.jpg)
II. Three Useful Dimensions for Typing of Studies for D & I Research
NIH Training Institute for D&I Research Ce-PIM 11
How “Real World”
How much control over Implementation
Assignment What Levels Are Measured?
![Page 12: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/12.jpg)
II. A Three Useful Dimensions for Typing of Studies for D & I Research
NIH Training Institute for D&I Research Ce-PIM 12
How “Real World” In Vivo -- full implementation as it takes
place In Vitro – implementation facilitated by
researchers In Silico –studies involving simulation
![Page 13: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/13.jpg)
Examples • In Vivo : CMHS/SAMHSA Implementation
of the Good Behavior Game in 24 School Districts
Very Modest Federal Funding Districts Responsible for TA • In Vitro : IES/NIDA Funded Professional
Development Trial in Houston – Poduska PI Training, Supervision Supported through Research
• In Silico: MicroSimulation of Expected Impact of Project KEEP for a State Child Welfare System
13
![Page 14: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/14.jpg)
Microsimulation
• Start with completed efficacy or effectiveness trial data of an intervention
• Model how this intervention would behave in a larger system if implemented broadly
• Design – Needs: 1. Individual level data from trial 2. Longitudinal data from a large system 3. Programming to model and extrapolate
findings
14
![Page 15: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/15.jpg)
KEEP Efficacy Trial - Chamberlain
• Study design: Randomized controlled trial • Population: Children in 700 foster homes in
San Diego county (ages 5-12) • Outcome: Efficacy of a foster parenting
intervention (KEEP) in reducing placement changes over 200 days
• Findings: – Positive exits (reunification with biological families,
adoption, etc) increased – Negative exits (other foster homes, group care, etc)
decreased 15
![Page 16: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/16.jpg)
KEEP Applied to Statewide Data – Jeremy Goldhaber-Fiebert
16
-0.60
-0.50
-0.40
-0.30
-0.20
-0.10
0.00
0.10
0.20
0.30
Effe
ct o
f KEE
P on
cu
mul
ativ
e in
cide
nce
Positive exits Negative exits Lateral moves
![Page 17: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/17.jpg)
II. B Three Useful Dimensions for Typing of Studies for D & I Research
NIH Training Institute for D&I Research Ce-PIM 17
How much control in implementation
strategy and/or intervention assignment Research Influence Observational - NONE Quasi-Experimental SOME QI-Designs Randomized - Randomizes
implementation strategy to systems (components)
![Page 18: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/18.jpg)
Standard Names for Experimental and Quasi-Experimental Designs (Cochrane,
EPOC)
• Randomized Controlled Trial (RCT) – random allocation to implementation/intervention arms
• Controlled Clinical Trial (CCT) – assigned in some controlled way (even-odd)
• Controlled Before and After Study (CBA) – Pre-post data same time for different arms Comparable controls More than 1 site in each arm • Interrupted Time Series (ITS) – sites
change from non-intervention to intervention 18
![Page 19: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/19.jpg)
Biglan et al., AJCP 2006
NIH Training Institute for D&I Research Ce-PIM 19
![Page 20: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/20.jpg)
Other Communities
NIH Training Institute for D&I Research Ce-PIM 20
![Page 21: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/21.jpg)
Important Research Questions:
NIH Training Institute for D&I Research Ce-PIM 21
Which communities/organizations get invited an
— What percent of those invited participate? — What are the characteristics of
participants? — What are barriers to participation in this context?
![Page 22: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/22.jpg)
Useful Nonrandomized Designs for Dissemination RE-AIM components
NIH Training Institute for D&I Research Ce-PIM 22
1. Broadcast of an Intervention Standardize Invitation and See Who Comes Kellam SG, Branch J, Brown CH, Russell G. Why teenagers come for treatment: A ten-year prospective
study of Woodlawn. Journal of the American Academy of Child Psychiatry 20:477-95, 1981.
2. Narrowcast of an Intervention Social Marketing to a Targeted Audience
3. “Timecast” of an Intervention Multiple Baseline design = Interrupted Time Series o Repeated measures over time of a community outcome o Introduce the intervention to a community midway through o Check whether community outcome differs before and after introduction
![Page 23: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/23.jpg)
Hawkins NG et al., AJPM 2007
NIH Training Institute for D&I Research Ce-PIM 23
![Page 24: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/24.jpg)
II. C Three Useful Dimensions for Typing of Studies for D & I Research
NIH Training Institute for D&I Research Ce-PIM 24
What Levels Are Measured? System Leaders County MH Agency Leader Implementation Agents Agency Hiring Clinicians Intervention Agents Clinicians Delivering Services Intervention Target Families Receiving Services
![Page 25: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/25.jpg)
Example
25
Four Levels of Change for Assessing
Performance Improvement
Assumptions about Change
Larger System / Environment
Organization
Group / Team
Individual
Reimbursement, legal, and
regulatory policies are key
Cooperation, coordination, &
shared knowledge are key
Structure and strategy are key
Knowledge, skill, and
expertise are key
From Shortell, 2004
![Page 26: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/26.jpg)
Types of Evidence for Implementation
NIH Training Institute for D&I Research Ce-PIM 26
• Formal Causal Inference • Hierarchy of Evidence
Generative Modeling
![Page 27: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/27.jpg)
III. Increasing the “Evidence” in Implementation Research
• Evidence is Elusive Concept 1. Formal use of the causality in
philosophy/statistics 2. Intuitive Ideas: Ordering -- Hierarchy of evidence 2 trials with replicated results
better than 1 trial
27
![Page 28: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/28.jpg)
Hierarchies of Evidence
NIH Training Institute for D&I Research Ce-PIM 28
• Preventive Task Force Expert Opinion … Pre-Post Design … Single Randomized Trial Multiple Randomized Trial
![Page 29: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/29.jpg)
Caution: Hierarchy Not Always True
NIH Training Institute for D&I Research Ce-PIM 29
• Date from “Observational Studies” May Be More Valuable than that From Randomized Trials
Example: Do antidepressants cause suicide in
youth? RCT’s – no suicides occurred, few
attempts
![Page 30: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/30.jpg)
Intuitive Idea About Evidence • Study 1: Implementation Strategy A significantly better
than Strategy B Is A truly better than B? Other Explanations -- imbalanced assignment -- differential support staff… • If we can modify Study 1’s Design in such a way to
– reduce the risk of other explanations – Leave all other factors the same
• Then the new Study Design has Stronger Evidence
30
![Page 31: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/31.jpg)
Example
• Study 1: Offer Implementation A to 40 Counties and Compare the 23 Who Select A against the 27 Who Do Not Select A (B).
• Study 1’ : Randomly assign 20 of the 40 Counties to Receive A and 20 to Receive B. And All 40 Counties Do Adhere to this Strategy.
31
![Page 32: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/32.jpg)
Roll-Out Randomized Implementation Trials
• Straightforward way to improve the evidence over interrupted time series or multiple baseline designs
• Generally Acceptable by Communities • Beneficial properties in statistical power
and logistics Brown et al. Clinical Trials 2006 Brown et al. Ann Rev Pub Health 2008
32
![Page 33: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/33.jpg)
Biglan et al., AJCP 2006
NIH Training Institute for D&I Research Ce-PIM 33
![Page 34: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/34.jpg)
Inferential Challenge with this Design
NIH Training Institute for D&I Research Ce-PIM 34
• What if and Exogenous Factor Happens at one of these Times of Transition?
• What if you Select the Most Promising Communities to Work with First?
• What if there are only a small number of communities?
• Harder to Conclude that Intervention Caused Change.
![Page 35: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/35.jpg)
Turning a Multiple Baseline Design into a True Randomized Experiment: Roll-Out Design
NIH Training Institute for D&I Research Ce-PIM 35
ROLL-OUT DESIGN • Divide Available Communities into Comparable
Batches • Start Measuring Outcomes for All Communities • Randomly Assign Each Comparable Batch to
WHEN the Implementation Begins • At the end, ALL Communities Are Exposed
• Analysis Uses All Communities and All Times
• Communities Still Serve as Own Controls • Communities Compared by Exposure Status Across Time
![Page 36: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/36.jpg)
Roll-Out Randomized Trials for Dissemination Research (Brown et al.,
2006 2008)
NIH Training Institute for D&I Research Ce-PIM 36
Population
1
2 3 4
5
Time of Transition in Dissemination Randomly Determined
R
Equivalent Subsets that are Ordered
Randomly
![Page 37: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/37.jpg)
Timing of Dissemination ( 0 to x)
NIH Training Institute for D&I Research Ce-PIM 37
• 1 0 0 x x x x x
• 2 0 0 0 x x x x
• 3 0 0 0 0 x x x
• 4 0 0 0 0 0 x x
• 5 0 0 0 0 0 0 x
![Page 38: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/38.jpg)
Random Assignment Reduces Bias in the Long Run
NIH Training Institute for D&I Research Ce-PIM 38
• Example: mPowerment Community Intervention Yr 1 Yr 2 . . . Yr 10
Treat Control Treat Control
R
R
R
![Page 39: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/39.jpg)
Implication of Roll-Out Designs for Community Research
NIH Training Institute for D&I Research Ce-PIM 39
• Units (communities) that get randomized are Large and often few are available at a time
A Single Trial with a Small Number of Communities is
Nearly Always Underpowered • Randomize small numbers of communities now • Randomize small numbers next year • … • Randomize small numbers in following years • Combine results across the years
• Brown et al., Ann Rev Public Health 2009 • Brown et al., Prev Science 2011
![Page 40: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/40.jpg)
Roll-Out Randomized Trials Are Often Acceptable to Communities and
Researchers • Traditional Control Group is Nonstarter for Many
Communities
Why Communities Are Comfortable Participating • Going Early – get a potentially valuable intervention • Going Later – potentially improved intervention • Randomization is Fair
Statistical and Logistical Reasons for Roll-Out Design • Higher statistical power over traditional wait-listed designs • Robust to External Influences • Higher Training Achieved When Training in Small Groups
40 Family Research Consortium San Juan
July 2011
![Page 41: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/41.jpg)
Examples of Roll-Out Trials Now Being Conducted
• 2 Generations of School-Based Suicide Prevention Trials
• Evaluation of the Training of Suicide Prevention Hot Lines
• Randomized Implementation Trial of 40 Counties in California for an Evidence-Based Foster Care Program (MTFC)
Family Research Consortium San Juan July 2011 41
![Page 42: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/42.jpg)
Randomize 40 Counties in CA Chamberlain et al., Adm Pol MH 2008
Family Research Consortium San Juan July 2011 42
40 CA Counties
26 Wait LIsted
CDT
Stnd
Wait Listed
13 Wait LIsted
COHORT 1 COHORT 2 COHORT 3
![Page 43: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/43.jpg)
Where These Designs Are Likely to Go
• Implementation Trials of Evidence-Based Programs
• Inclusion of Programs in Health Reform through Research Based Practice Organizations
Family Research Consortium San Juan July 2011 43
![Page 44: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/44.jpg)
Summary • Implementation Designs Classified Along How “Real World” How much control over Implementation
Assignment What Levels Are Measured?
• Numerous Opportunities to Improve Evidence
through Design Enhancements – Obtain wider distribution of communities – Strategic use of random assignment – Roll-Out Randomized Trials
• Community Buy-In • Improved Statistical Properties
44
![Page 45: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/45.jpg)
Where are we going in Implementation Research?
NIH Training Institute for D&I Research Ce-PIM 45
![Page 46: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/46.jpg)
Your Role in the Next Stage of Implementation Research: Relying
on What We Now Know
NIH Training Institute for D&I Research Ce-PIM 46
![Page 47: Designs and Types of Evidence for Dissemination and ...conferences.thehillgroup.com/OBSSRinstitutes/TIDIRH2011... · Designs and Types of Evidence for Dissemination and Implementation](https://reader031.vdocuments.us/reader031/viewer/2022011806/5bec62d809d3f26e208c769d/html5/thumbnails/47.jpg)
Your Role in the Next Stage of Implementation Research: Your
Ideas in Building This Field
NIH Training Institute for D&I Research Ce-PIM 47