innovations in prevention & implementation science: implications for improvement in schools

1

Innovations in Prevention &

Implementation Science:

Implications for Improvement in

SchoolsC Hendricks Brown, Northwestern University Feinberg School of Medicine

Sheppard Kellam, Johns Hopkins UniversityJeanne Poduska, American Institutes for Research

Juan Villamar, Northwestern University Feinberg School of MedicineGracelyn Cruden, Northwestern University Feinberg School of Medicine

2

Acknowledgments: for Ce-PIM • National Institute on Drug Abuse: CENTER FOR

PREVENTION IMPLEMENTATION METHODOLOGY FOR DRUG ABUSE AND SEXUAL RISK BEHAVIOR (PI Brown P30 DA027828)

• Substance Abuse and Mental Health Services Administration

SAMHSA Ce-PIM Partnership

3

Acknowledgements: Work to DateFunding Agencies:• IES•NIDA•NIMH•NICHD

4

Process in this Workshop: Active

Engagement

5

1. A Challenging Tour of Implementation Science and its Home in Service Delivery Systems

2. Public Education and Public Health: Toward a System of Systems

3. Systems Thinking and Partnerships in Daily Life4. Illustrative Innovations of Ce-PIM5. Summary: Building an Integrated Structure to

Support Children and Youth Reaching their Full Potential

Workshop Outline

6

1. A Challenging Tour of Implementation Science and its Home in Service Delivery Systems The Perspective at Ce-PIMA. Implementation as an action strategy working on systems changeB. Why Study Implementation?C. Approaches to Moving Effective Interventions into PracticeD. Three Interacting and Evolving Components of ImplementationE. A Systems Approach involving 3 interacting components Small group exercise

7

Ce-PIM Mission: Building scientific and methodologic rigor into implementation research and practice.AIMS: 1. Develop Systems Methods (Systems Science, Engineering,

Computational) Methods for Implementation Science2. Infuse systems methods into the field of Implementation

Research3. integrating methods within the practice of prevention

implementation at the federal, state, county, and local levels.

8

A. What is Implementation Research?According to NIH (2008):The use of strategies to adopt and integrate evidence-based health interventions and change practice patterns within and across specific systems

Action OrientedWithin Settings or SystemsAND collects data

Chambers DA. Advancing the science of implementation: A workshop summary. Administration and Policy in Mental Health and Mental Health Services Research. 2008;35(1-2):3-10.

9

B. Why Study Implementation?1. We know a lot about what works 10K reviewed studies in What Works Clearinghouse

2. We are short on implementation action strategies to put what works into practice: WWC Practice Guides:

Reducing Behavior Problems in the Elementary School Classroom 2008

3. It takes too long for research to affect practice

10

Closing the Gap between What We “Know” Works

and What We Do.“17 Year Gap” in Health Care

11

17 Years in Implementing an Statistical MethodologyGeneralized Estimating Equations (GEE: Brown et al., APMH 2012)

12

•Is the gap between research and practice similar in education to that existing in health?Types of Gaps?As long as? As important to shorten? Which way?As resistant to change?

13

C. Approaches to Moving Effective Programs into Practice

Making a Program

Work

Does a Program Work?

Could a Program Work?

Traditional Research Pipeline for

Implementation

14

IOM 2009

Landsverk, Brown et al. 2012

Aarons et al., 2011

Implementation

Exploration

Adoption / Preparation

Implementation

Sustainment

Effectiveness Studies

EfficacyStudies

Preintervention

Traditional Translation Pipeline

Rea

l Wor

ld R

elev

ance

Local knowledge

Generalized knowledge

Intervention

15

A Focus on the Intervention itself is Not Sufficient•The use of effective interventions without implementation strategies is like serum without a syringe; the cure is available, but the delivery system is not

Fixsen, Blase, Duda, Naoom, Van Dyke,2010.

16

D. Three Interacting and Evolving Components of Implementation (Chambers et al., 2013 Imp Sci)• Intervention: Program, Practice, Policy, Principles• Practice Setting: Delivery Support System• Ecological System: Population and Community/Cultural Context

17

Key Parameters on Implementation ScienceA. Transitions of Research Questions:

Effectiveness – Can a Program Work in real life systems? Implementation – Making a Program Work

B. 4 Stages: Exploratory, Adoption, Implement w/ Fidelity, SustainabilityC. Focus on Interactions of Program, Delivery System, and EcologyD. Science:

Produce Generalizable Knowledge rather than Local KnowledgeMeasurement, Modeling, and Testing Using Rigorous Methods

E. Partnerships

18

Definition of System

A set of connected parts forming a complex whole

19

Group Exercise: Produce a System Map of the Educational System Three Steps:1. Who needs to be at the table? What are the components

needed represented?2. How do they Interact with one another?3. Where would you intervene to affect intervention usage?

Education Policy, i.e. Common Core Standards

Identify one person from your small group to bring this back to the group

20

Follow-up Question on Systems:Would your list of players and location intervention differ based on implementing

New policyReplacement of an existing curriculumRemoval of a expulsion policy

21

Summary of Part 1• Multi-goal, Interacting, Dynamic and Multilevel

System

• System View Looks at Interactions between components.

• Behavior of a system is not linear: complex.

• A useful way to understand a system is to intervene and study its behavior.

2. Public Education and Public Health: Toward a

System of Systems

Sheppard G. Kellam, MDCenter for Prevention Implementation Methodology

Johns Hopkins Bloomberg School of Public Health

Society for Research on Educational EffectivenessWashington, DC, September 4-5, 2014

23

The Baltimore Education and Prevention Partnership

The Baltimore City Public School System (BCPSS) has collaborated in 3 generations of education and prevention field trials.

Trials were directed at helping children master obeying rules of behaving, attending, academic learning, socializing appropriately in 1st grade classroom.

Interventions were tested separately in 1st generation (our focus today), then together in later trials.

24

Two Dimensional View of Mental Health

•Social Adaptational Status—level of success or failure in main social fields at each stage of life

•Psychological/physical Well-Being—includes affect, cognitive development, i.e., internal condition of the individual

25Kellam/Langevin, 2003

DEVELOPMENTALEPIDEMIOLOGY:

directed at early proximal targets

MORE IMMEDIATE RISK: directed at

more recent proximal targets

COMMUNITY PREVENTION:

directed at Community & School proximal targets

COMMUNITY / SOCIETAL: directed at

Policies & Laws as proximal targets

Prevention Research and Service Strategies

26

Universal

Selective

IndicatedRxMed, MH,Soc Welfare

Levels of Prevention

and TreatmentEntire

Population

27

Current Problems in System Coordination

•Prevention and treatment not integrated•Service Agencies are Silos—minimal collaboration

•No shared information system monitoring developmental progress

•Mystery as to who gets “referred” for services•Little follow-up of “no-shows”

28

Early Risk in Prevention Research Over the last four decades much has been learned about early risk factors and paths leading to drug abuse, and other behavioral, mental health, and school problems.

Most if not all are strongly related to school failure, also a major risk factor for later drug abuse, alcohol, tobacco, depression, anti-social and other problem outcomes.

Aggressive, disruptive behavior as early as 1st grade has been repeatedly found a risk factor for later drug and alcohol abuse and disorders, delinquency, violence, tobacco use, high risk sex, school failure and other high risk behaviors.

29

When a child misbehaves…

30

The Baltimore Education and Prevention Partnership

• The Baltimore City Public School System (BCPSS) has collaborated in 3 generations of education and prevention field trials.

• They were directed at helping children master key social task demands in 1st grade classroom.

• Interventions were tested separately, then together.

• The 1st generation will be our main focus here, where the Good Behavior Game (GBG) was tested by itself and the children were followed up at ages 19-21 and beyond.

31

High Risk Children in Well vs. Poorly-Managed Classrooms (control classrooms only)

• If the top 25% of children on aggressive behavior were in disrupted classrooms, their risk of severe aggressive behavior by middle school was up to 59 times the average child’s.

• If similar children were in well-managed classrooms, the risk was up to 2.7 times the average child’s.

32Kellam et al., Addiction Science & Clinical Practice; 6:73-84, 2011

33

The Role of School Information Systems in our New System of Systems• Following individual children over time, schools and

communities is required for assessing their progress, needs and for integrating services across agencies

• Information systems tracking each child exist in most school districts and can be extended and used to integrate services and monitor progress

• Confidentiality is by law guaranteed by school authorities and can be expanded to include data as needed for services

34

Partnerships for Building and Overseeing the New System of Systems •Analyze political constituencies•Engage and work through trust with each•Learn their mission and vision• Identify mutual self-interests and•Encourage their coming together as oversight committee

35

Whole Group Activity: Partnerships•What do we mean by “who” needs to be included?

•The nature of the process of engaging and developing the base of support/collaboration.

36

3. Systems Thinking and Partnerships in

Daily Life Jeanne Poduska, Sc.D.

American Institutes for Research

Society for Research on Educational EffectivenessWashington, DC, September 4-5, 2014

37

Systems Concepts • Introduction of programs and action strategies requires adaptation to systems, e.g. staffing, fiscal allocations

•We operate in complex systems: multi-level, multi-actor

•Change occurs locally (even if simultaneously)

•Systems thinking focuses on interactions

Working Definition of Scaling-Out

“Scaling out is the deliberate use of strategies to implement and sustain evidence-based interventions through or across settings to promote the greatest public health impact”.

Ce-PIM Workgroup on Scaling Out, unpublished, 2014

38

Scaling Out Perspective on Implementation Research

39Scale Out Across Diverse ContextsLocal knowledge

Generalized knowledge

Scale Up

ExploreAdopt/

Prepa

re

Implem

entSusta

in

Baltimore

Houston

NE urban

NE rural

40

AIR’S mission is to conduct and apply the best behavioral social science research and evaluation towards improving peoples’ lives, with a special emphasis on the disadvantage.

41Chambers et al, 2013 Implementation Science

Prevention Science into….

Johns Hopkins: 1983-present

RCT 1: GBG | MLRCT 2: GBG + curricula | family

AIR: 2003-2007RCT 3: GBG + reading + family Develop model of training and support for GBG

Baltimore ’83-’07

Case Study: Good Behavior Game at AIR

Implementation Science

RCT 1: -GBG: 2 models of PD-Implement/Sustain-”Adaptation”

Service contract to develop local GBG capacity: Face-to-face training and support

Houston ‘08- present

Implementation Science

Service contract: Face-to-face training and support

Under review: GBG for Middle School

Nebraska urban ’11- present

Implementation Science

Development grant: Develop and pilot web-based tools to deliver training to teachers

Under review: Develop and pilot web-based tools for coaching and on-going support to teachers

Nebraska rural ’13- present

JHSPH American Institutes for Research

Domitrovich et al., 2008; Poduska et al., 2009; Poduska et al., 2012; Poduska & Kurki, 2014

Scaling Out Perspective on Implementation Research

43Scale Out Across Diverse ContextsLocal knowledge

Generalized knowledge

Scale Up

ExploreAdopt/

Prepa

re

Implem

entSusta

in

Baltimore

Houston

NE urban

NE rural

Who needs to be at table at AIR?

Leadership:CEO/President

Board of DirectorsDir, Ed Program

LegalContracts

Human Resources

Information Technology

(IT)

Financial

Institutional Review Board

(IRB)

Publication Services

Communications

Web Services

??

??

Colleagues:Researchers

Practitioners

Data Use

Professional Development

Adolescence

Ed Tech

Early Childhood

Middle School

Social Emotional Learning

Mental Health ServicesQuantitative Analysts

Qualitative Analysts

45

Additional Systems Concepts

•Systems have developmental trajectories •Systems are adaptive, fluid •Adaptation occurs locally •A note about silos: people may know one another, talk with one another and not be working towards a common goal

46

4. Innovations: Activities of Ce-PIM

Juan Villamar, MS.EdGracelyn Cruden, M.A.

C Hendricks Brown, Ph.D.

Center for Prevention Implementation MethodologyDepartment of Psychiatry and Behavioral Sciences

Northwestern University

47

5. Improve efficiency and effectiveness of implementation delivery

Implementation Agency or Agencies

Target

Developer/ Purveyor

Broker, Intermediary

1. Person, Place, and Time, Reach, Engagement

2. Delivery medium

3. Simplify and support intervention delivery

4. Training, fidelity monitoring and feedback

6. Intervention R&D

7. Monitoring Large-Scale Implementation

Funder / Oversight

8. Modeling and instituting implementation and oversight policies

General Model to Support Implementation of Evidence Based InterventionBrown et al, 2013 JAIDS

48

Using a mobile application to improve fidelity of the GBG – Rapid Development of a Prototype

49

Visualization Tool – Classroom Disruptiveness Comparison Model

2 classrooms, 24 students

Chance that students will act disruptively and their disruptiveness will spread.

50

Major Gap – Sustainable Partnerships •Partnerships between content experts and technology developers

“What do you recommend we use?” VS “Tell me what you want to see in a report”

•Bring to the table those players that will interact with the report FIRST

•Partnerships between Research and PracticeIterative testing with feedback loops

•Technology System, Educational System, Public Health SystemIntegrated System?

Understanding the System: Policy• Problem: Dissemination of Evidence-Based Interventions• Who can pay, sustainably USPSTF• How to influence this powerful entity?

• Understand how they came to power, to whom they report, and who is required to listen

• Review their decision making process• Concrete challenge: # of prevention programs in parenting that are

linked to primary care nearly 0 (despite IOM 2009 report)

• Is primary care even the best environment? What about schools, who have a larger denominator?

52

Strategic Engagement• Identify areas for involvement

• Short term – • engage with AHRQ and USPSTF; identify brokers

• Long term – • Scaling out research designs (primary care)• Research design for increased knowledge utilization (ex: harmonized

measures)• Publications in strategic journals

• Is there a similar influential agent in Education? How would you begin to engage in the short and long term?

53

Innovation: Opening integration between primary care and research

AHRQ

USPSTF

NIH

Schools- Nurses- School Wide

Programs- Information Systems

Primary Care Clinics

CMMI grants ODP

54

Implementation Science can be conducted with high rigor through:•Measurement•Modeling•Testing

CAL-OH Head-to-Head Randomized Implementation Trial• Single evidence-based intervention:

• Multidimensional Treatment Foster Care (MTFC)

• 2 alternative strategies of implementing this same programStandard implementation (Stnd)

Community Development Team (CDT)

• Randomize 51 Counties to implementation strategy*• Evaluate implementation success using Stages of Implementation Completion

(Saldana IS 2014)• Faster implementation• more (# families served), • better (fidelity)

Chamberlain P, et al. (2008). Engaging and Recruiting Counties in an Experiment on Implementing Evidence–Based Practice in California. Administration and Policy in Mental Health and Mental Health Services Research, 35(4): 250-260.

55

Existing Implementation

Supports for County, Agency,

Group Home

MTFC Intervention

MTFC Implementation

Supports for County, Agency, Clinicians,

Parent

Two-Arm TrialsEffectiveness vs.

Implementation

56

MTFC Intervention MTFC

Intervention

CDTImplementation

Supports for County

Control Condition

Standard Implementation

Supports for County

Youth Youth

Youth Youth

57

Randomize 51 Counties in CA and OH to Implementation Strategy and Time (Cohort)

Randomized Roll-Out Design*

40 CA Counties

26 Wait LIsted

CDT

Stnd

Wait Listed

13 Wait LIsted

COHORT 1 COHORT 2 COHORT 3 COHORT 4

*Brown, et al. 2009 Ann Rev PH

11 OH Counties

Summary of Findings (Brown et al., under review)• Mixed Results

• No evidence that • CDT affected rate of adoption• CDT changed speed of implementation

• Evidence that • CDT increased numbers

of families served• CDT counties completed

implementation more thoroughly58

0 5 10 15 20 25 30

05

10

15

20

25

30

Figure 4. Comparison of PlacementQuantiles for CDT and IND Counties

CDT

IND

Number Served Quantiles for CDT versus IND (EQQ Plot)

59

5. Building an Integrated Structure to Support

Children and Youth Reach their FULL Potential

Opportunities, Strengths, Next Steps and Closing

60

1. Implementation Science involves a systematic study of actions within and across systems

2. Requires partnerships that need to built on mutual self interest 3. Systematic way to build such partnerships that coalesce the

interests of diverse systems, including public education and public health

4. Implementation lives inside the organizations and communities that are responsible for delivering or overseeing services.

5. There are rigorous methodologic approaches to Implementation Science that address different stages of implementation with well define measures, modeling approaches and research designs.

61

Contact Information:• C Hendricks Brown,

hendricks.brown@northwestern.edu • Sheppard G Kellam, skellam@jhu.edu • Jeanne Poduska, jpoduska@air.org • Juan Villamar, juan.villamar@northwestern.edu• Gracelyn Cruden, gracelyn.cruden@northwestern.edu

Slides and References are Available:http://cepim.northwestern.edu/products/presentations-and-workshops/