from bench science to systems change · children’s services,3(2), 53-61. bumbarger, b., perkins,...

From Bench Science to Systems Change: Reconciling Evidence-based Practice with

Practice-based Evidence

Brian Bumbarger & Stephanie Bradley

PRC Seminar

January, 2013

Investigators and Authors:

Brian Bumbarger Mark Greenberg

Mark Feinberg Brittany Rhoades

Louis Brown Wayne Osgood

Michael Cleveland Damon Jones

Jennifer Sartorious Julia Moore

Brendan Gomez Richard Puddy

Stephanie Bradley Elizabeth Campbell

Collaborative Policy Innovators:

Mike Pennington Clay Yeager

James Anderson Keith Snyder

The EPISCenter and research described here are supported by grants from the Pennsylvania Commission on Crime

and Delinquency. Special thanks to the staff of the Office of Juvenile Justice and Delinquency Prevention (OJJDP)

2

We know a great deal about how youth problems develop, and how to effectively prevent them (& reduce prevalence)

• Known risk & protective factors

• Multiple domains of influence (community, family, school, peer, individual)

• Multifinality and equifinality

(a public health approach to public safety)

• Different trajectories (early vs. late starters)

• Criminogenic impact of intervention

From Lists to Improved Public Health…

• Synthesis and translation of research to practice, (and practice to research)

• EBP dissemination, selection, and uptake

• Ensuring sufficient implementation quality and fidelity

• Understanding adaptation and preventing program drift

• Measuring and monitoring implementation and outcomes

• Policy, systems, and infrastructure barriers

• Coordination across multiple programs and developmentally

• Sustainability in the absence of a prevention infrastructure

Bumbarger, B. and Perkins, D. (2008). After Randomized Trials: Issues related to dissemination of evidence-based interventions. Journal of Children’s Services,3(2), 53-61. Bumbarger, B., Perkins, D., and Greenberg, M. (2009). Taking Effective Prevention to Scale. In B. Doll, W. Pfohl, & J. Yoon (Eds.)

Handbook of Youth Prevention Science. New York: Routledge.

Moving From Prevention Science . . .

Problem Response

Define the

Problem

Identify Risk

& Protective

Factors

Develop &

Test

Interventions

Implement &

Evaluate

Programs

…To Prevention Service

Provide

Technical

Assistance

Set & Collect

Performance

Measures

Monitor Quality

of Program

Implementation

Assess

Public Health

Impact

Response

Pennsylvania’s EBP Dissemination Model

• Prevent dependency, delinquency, and ATOD use to the greatest

degree possible (primary prevention)

• Intervene effectively with youth for whom primary prevention is not

sufficient

• Allow communities flexibility to select strategies that best meet local

needs

• Create community-level infrastructure for strategic prevention

planning and coordination

• Provide accountability and use scarce resources efficiently

ULTIMATELY….

• To “move the needle” on key indicators of (behavioral)

health at the POPULATION level 9

Intermediary and State-level Prevention Support System

Support to Community Prevention Coalitions

Improve Quality of Local Innovative

Programs and Practices

Support to

Evidence-based

Programs

Multi-Agency Steering Committee

(Justice, Welfare, Education, Health)

The EPISCenter is a project of the Prevention Research Center, College of Health and Human Development, Penn State University, and is funded by the Pennsylvania Commission on Crime and Delinquency and the Pennsylvania Department of Public Welfare

as a component of the Resource Center for Evidence-Based Prevention and Intervention Programs and Practices.

A unique partnership between policymakers, researchers, and communities to bring science to bear on issues of public health and

public safety

The Menu of EBPs in This Project*

• Olweus Bullying Prevention Program (OBPP)

• Promoting Alternative Thinking Strategies (PATHS)

• Big Brothers Big Sisters of America (BBBS)

• Multidimensional Treatment Foster Care (MTFC)

• Strengthening Families Program 10-14 (SFP)

• Project Towards No Drug Abuse (Project TND)

• Life Skills Training (LST)

• Incredible Years (IYS)

• Functional Family Therapy (FFT)

• Multisystemic Therapy (MST)

• Aggression Replacement Training (ART)

Multifinality: There are known risk

factors, each of which is associated

with a range of poor outcomes

Equifinality: Poor outcomes are

associated with a combination of risk

factors and developmental pathways

Creating Fertile Ground for EBPs Risk-focused Prevention Planning

(the Communities That Care model)

Collect local data on

risk and protective

factors

Use data to

identify priorities

Select and implement

evidence-based program that

targets those factors

Re-assess risk

and protective

factors

Form local coalition

of key stakeholders

Leads to community

synergy and

focused resource allocation

Pennsylvania’s CTC coalitions 2012

Improve Quality of Juvenile Justice Programs and

Practices

Support to

Evidence-based

Programs




Creating “fertile ground” for EBP

selection and adoption



Practices

Support to

Evidence-based

Programs




Ensuring high-quality implementation and

sustainability


Pennsylvania’s EBP dissemination in 1999…

Measuring Population-level Impact

• Cross-sectional quasi-experimental study of

98,000 students in 147 communities

• Used propensity score matching to minimize potential

selection bias

• Found youth in CTC communities reported lower rates of

risk factors, substance use, and delinquency than youth

in similar non-CTC communities (7x as many as by

chance)

• Communities using EBPs showed better outcomes on

twice as many R/P factors and behaviors (14x as many

as by chance)

-20

-10

0

10

20

30

40

-10.8

33.2

-10.8

16.4

Delinquency AcademicPerformance

Nega vePeerInfluence SchoolEngagement

5 year Longitudinal Study of PA Youth

% Change of CTC/EBP Youth Over

Comparison Group

419 age-grade cohorts over a 5-year period:

youth in CTC communities using EBPs had significantly lower rates of delinquency,

greater resistance to negative peer influence,

stronger school engagement and better academic achievement

Feinberg, M.E., Greenberg, M.T., Osgood, W.O., Sartorius, J., Bontempo, D.E. (2010). Can Community Coalitions Have a

Population Level Impact on Adolescent Behavior Problems? CTC in Pennsylvania, Prevention Science.

Impact on Juvenile Court Placement Rates: Comparison of Placement Rates for Counties* With and Without an EBI

9.07

9.76

10.05

10.53

9.89

8.70

7.79 7.78

7.00

7.50

8.00

8.50

9.00

9.50

10.00

10.50

11.00

2007 2008 2009 2010

No EBI

AdoptedEBI

Bumbarger, B. K., Moore, J., & Rhoades, B. (2010). Impact of evidence-based interventions on delinquency placement rates.

Presentation at 2011 Society for Prevention Research annual meeting. Washington, DC.

24

Justice Reinvestment Realized

26

Programs/services can be placed along a continuum of confidence based on their evidence or theory

How confident are we that this program or practice is a good use of resources AND improves outcomes for children and families?

Very Confident

Evidence-based “This program has been rigorously evaluated and shown to work”

Research-based “This program is based on sound theory informed by research”

Promising Approaches “We really think this will work… but we need time to prove it”

Best Practices “We’ve done it and we like it”

EFFECTIVE HARMFUL

Iatrogenic (Harmful) “This program has been rigorously evaluated and shown to be harmful”

Ineffective “This program has been evaluated and shown to have no positive or negative effect”

Promising Ineffective unknown Very

Confident

*Bumbarger & Rhoades, 2012


Practices

Support to

Evidence-based

Programs




Developing “practice-based evidence”


• Theoretically-based

• Demonstrated effects

(confidence)

• Sponsored lists

– E.g., Blueprints

• Funding requirements

• Not an EBP for every community need/context

• Many

programs/services already in widespread use

• Some of those may be effective

• Local expertise/fit

A 2-pronged approach…

Promoting Evidence-based Programs

Developing Practice-based Evidence

Background

• More than one way to establish “evidence-based practice”:

1. Model program implementation

2. Evaluation of local program

3. Meta-analysis of research on program “types”

Background

• Generic intervention types abundant in juvenile justice

• Research shows some types effective at reducing recidivism

• Main disadvantage of generic programs is they…

“lack the specificity that comes with the protocol for a

model program and the associated training and support

systems that are also often available from the developer”

(p. 3).

Howell, J.C., & Lipsey, M. W. (2012) Research-based guidelines for juvenile justice programs.

Justice Research and Policy, (14) 1, p.1-18.

Integrated Meta-analysis of Program Types

• Mark Lipsey, Peabody Research Institute, Vanderbilt Univ.

• 548 studies - program impact on recidivism

• Juveniles aged 12-21, in programs to reduce delinquency

• Most effective program types? Characteristics of them?

Lipsey, M. W. (2009). The primary factors that characterize effective interventions with juvenile

offenders: A meta-analytic overview. Victims and Offenders (4), 124-147.

Meta-analysis Results



Meta-analysis Results

• Four factors most strongly related to recidivism reduction:

• Program type: therapeutic vs. control-oriented

• Quality of service delivery: clear protocol, monitoring,

staff training

• Amount of service: duration and total contact hours

• Risk: greater impact with higher risk youth

Service/program groups (35pts):

1. Individual counseling, job-

training

2. Community service, remedial

academics

3. Family couns., social skills

training, challenge programs

4. Group counseling, mentoring,

contingency mgmt

5. Cognitive-behavioral therapy

Amount of Service (20pts):

% of youth meeting targets for

1. Duration (10pts)

2. Contact Hours (10pts)

Example: Mentoring

Target weeks: 26

Target hours: 78

Quality of Service Delivery (20pts):

1. Written Program Protocol

2. Staff Training

3. Staff Supervision

4. Response to Drift

Youth Level of Risk (25pts):

% of youth served at target risk

Youth Level of Service (YLS)

42 item, interview-based

Risk factors assessed:

• Prior & current offenses

• Family circumstances/Parenting

• Education/Employment

• Peer Relations

• Substance Use

• Leisure/Recreation

• Personality/Behavior

• Attitudes/Orientation

SPEP Score Validity

*Quality of service delivery not scored in this sample.



Advantages

• Utilizes wider research base, and broadens scope of

vetted programs

• e.g., literature on “social skills training”

• e.g., other mentorship models, beyond BBBS

• Fills practical and political gap between “model

programs” and everything else

• Not a model program for every need

• Potential for significant public health impact

• Creates new possibilities in program research and

development

• Moving beyond the binary mode of “on-the-list” or not

Limitations

• Scoring is intensely qualitative, time consuming

• e.g., 53.5 man hours (2-3 staff, 3-4 days)

• What’s “behind the curtain”?

• Confusion around what “evidence-based” means now…

• Field just getting comfortable with vetted lists

EPISCenter’s Role

• “Ride along” with Berks & Lehigh during pilot phase

• How well does SPEP work in PA context?

• Prepare for later statewide rollout

• Develop infrastructure and streamline processes from

pilot phase to scale-up

• Facilitate the process of scoring across PA counties

(probably in phases/cohorts)

• Develop and support Program Improvement Plans

• Collect lessons learned and facilitate CQI of SPEP in PA

EPISCenter’s Role

• Main disadvantage of generic programs is they…

“lack the specificity that comes with the protocol for a

model program and the associated training and support

systems that are also often available from the developer”

(p. 3).



Thank You!

Evidence-based Prevention and Intervention Support Center

Prevention Research Center, Penn State University

206 Towers Bldg.

University Park, PA 16802

(814) 863-2568

[email protected]

www.episcenter.psu.edu

from bench science to systems change · children’s services,3(2), 53-61. bumbarger, b., perkins,...

Documents