karen a. blase, phd dean l. fixsen, phd, melissa van dyke, lcsw, allison metz, phd, sandra naoom,...

Karen A. Blase, PhDDean L. Fixsen, PhD, Melissa Van Dyke, LCSW,

Allison Metz, PhD, Sandra Naoom, MSPH,

Michelle Duda, Ph D

From Outcomes to Implementation


July 31, 2010

2

ECO Logic Model (simplified)Early Childhood Outcomes Center

ECO Center

Activities

Strategic use of data

High quality state

outcomes measure-

ment systems

Improved services, supports,

and practices

Improved outcome

s for children

and families

Celebrate!

Copyright © Dean L. Fixsen and Karen A. Blase, 2008

Early Childhood Outcomes Center3

State Early Childhood Accountability and Improvement System

A system of standards-based assessments of

(a) children’s development and learning and

(b) program quality,

designed to inform state policy decisions, investments, and improvement efforts for early education programs for three- and four-year-old children,

linked to a continuum of kindergarten through third grade standards, curriculum, assessments, and program improvement efforts.

National Early Childhood Accountability Task Force


4 Early Childhood Outcomes Center

ECO Center

Activities


High quality state

outcomes measure-

ment systems


and practices

Improved outcomes

for children and families

Implementation Science


Goals TodayUnderstand

how implementation science and frameworks can positively impact program improvement efforts

what functions data serves in the implementation process

Think about

implementation-informed efforts to improve program quality

and how to align policy to support effective programs and practices.

6 Early Childhood Outcomes Center

ECO Center

Activities


High quality state

outcomes measure-

ment systems


and practices

Improved outcomes

for children and families

Implementation ScienceWhat is implementation?

How do we use the science?


Early Childhood Programs, Services, and Data Systems

….Through an Implementation Lens


But FIRST… What Do We Mean by Implementation?

A specified set of activities designed to put into practice a policy, activity, or program of known dimensions.

Implementation processes are purposeful and defined in sufficient detail such that independent observers can detect the presence and strength of these “specified activities”


What Works

Effective NOT Effective

Effective

NOT Effective

IMPLEMENTATION – The HOW

INT

ER

VE

NT

ION

Th

e W

HA

T

High Fidelity = Good

Outcomes

Low Fidelity = Poor Outcomes

Getting Here and Staying Here is the Challenge!



Review and synthesis of the implementation evaluation literature (1970 – 2004)

Multi-disciplinary

Multi-sector

Multi-national


Implementation

Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M. & Wallace, F. (2005). Implementation Research: A Synthesis of the Literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231).

Download all or part of the monograph at:

http://www.fpg.unc.edu/~nirn/resources/detail.cfm?resourceID=31

Implementation Research: A Synthesis of the Literature

http://www.fpg.unc.edu/~nirn/resources/detail.cfm?resourceID=31



Implementation science is universal (like physics, chemistry)


There are Shared Challenges!!

Implementation is implementation in:• Agriculture and Forestry • Manufacturing and Business• Substance abuse• Child welfare, • Education, • Engineering, • Health and Medicine • Juvenile Justice, • Mental health, • Nursing, …and more

They have similar implementation problemsThey have similar implementation solutions We can learn more …more quickly


Data Show These Methods, When Used Alone, Do Not Result In Implementation As Intended:

Diffusion/ Dissemination of information

Training

Passing laws/ mandates/ regulations

Providing funding/ incentives

Organization change/ reorganization

Business as Usual ≠ Impact

Necessary But Not Sufficient


What Works

Effective NOT Effective

Effective

NOT Effective

IMPLEMENTATION – The HOW

INT

ER

VE

NT

ION

Th

e W

HA

T

Performance Implementation (High Fidelity)

Paper ImplementationProcedure Implementation (Low Fidelity)

Getting Here and Staying Here is the Challenge!


Pre-Requisite: Defining “IT”

From what current state to what future state?

The “it” must be operationalized whether it is:

An evidence-based practice or program

A best practice Initiative

A broad early childhood systems change initiative

OperationalizePart of Speech: verb Definition: to define a

concept or variable so that it can be measured or expressed quantitativelyWebster's New Millennium™ Dictionary of English, Preview Edition (v 0.9.7)Copyright © 2003-2008 Lexico Publishing Group, LLC


Implementation = Putting “It” Into Action

Who will do what differently?

Who will help change occur?

AND How will change happen?

At all levels….

Practice (individual child data)

Program (summative “unit” child data)

Organization(s) (agency or coalition data)

System (state data)

Higher Fidelity is correlated with better outcomes across a

wide range of programs and practices

I didn't have potatoes, so I substituted rice.

Didn't have paprika, so I used another spice.

I didn't have tomato sauce, so I used tomato paste.

A whole can not a half can - I don't believe in waste.

My friend gave me the recipe - she said you couldn't beat it.

There must be something wrong with her, I couldn't even eat it!

~ Senior Center Newsletter

Policy Implications

1. IF THERE IS NOT THE “WILL” OR“MEANS” TO DO IT RIGHT YOU WON’T GET RESULTS

2. YOU CAN’T DO A COST-BENEFIT ANALYSIS IF THERE IS NO BENEFIT!

WHAT WORKS ~IMPLEMENTING PROPERLY MATTERS

Higher Fidelity is correlated with better outcomes across a

wide range of programs and practices

I didn't have potatoes, so I substituted rice.

Didn't have paprika, so I used another spice.

I didn't have tomato sauce, so I used tomato paste.

A whole can not a half can - I don't believe in waste.

My friend gave me the recipe - she said you couldn't beat it.

There must be something wrong with her, I couldn't even eat it!

~ Senior Center Newsletter

Measure Fidelity AND Measure Outcomes BECAUSE you need to know:1. Are we having an

implementation problem? Low fidelity & Poor outcome = implementation problem

2. Or are we having an effectiveness problem?

High fidelity & Poor outcome = effectiveness problem

Program Improvement Implications


Implementation Frameworks

Multi-dimensional, Fully integrated

Implementation Drivers

Improvement Cycles

Implementation Teams

Implementation Stages

Need attention to all but today focus on just the first two





Improvement Cycles




Implementation Drivers…

HOW to:

Build Competency and ConfidenceDevelop, improve, and sustain competent & confident use of innovations.

Change Organizations and SystemsCreate and sustain hospitable organizational and systems environments for effective instructional and educational services

Provide Leadership that Matches the Challenge

Core Implementation

Components

Core Implementation

Components

© Fixsen & Blase, 2008

Improved Early Childhood Outcomes

Com

pete

ncy

Driv

ers

Com

pete

ncy

Driv

ers O

rganization Drivers

Organization D

rivers

LeadershipLeadership

Evidence-Based and Evidence-Informed Programs and Practices

How:

What:

Why:

Capacity to provide direction, vision, & the

“right” leadership approach

Staff capacity to support children/families with the selected practices

Institutional capacity to support staff in implementing practices with fidelity


Performance Assessment

Coaching

Training

Selection

Com

pete

ncy

Driv

ers

Com

pete

ncy

Driv

ers


Implementation Drivers =

Infrastructure

Implementation Lens

Evidence-Based and Evidence-InformedPrograms and Practices


Coaching Driver

OUTCOMES% of Participants who Demonstrate Knowledge, Demonstrate New Skills in a Training Setting,

and Use new Skills in the Classroom

TRAINING

COMPONENTSKnowledge Skill

DemonstrationUse in theClassroom

Theory and Discussion

10% 5% 0%

..+Demonstration in Training 30% 20% 0%

…+ Practice & Feedback in

Training

60% 60% 5%

…+ Coaching in Classroom

95% 95% 95%

Joyce and Showers, 2002

Copyright © Dean L. Fixsen and Karen A. Blase, 2008© Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008

Sobering Observations

"All organizations [and systems] are designed, intentionally or unwittingly, to achieve precisely the results they get."

R. Spencer DarlingBusiness Expert



Coaching

Training

Selection

Systems Intervention

Facilitative Administration

Decision Support Data SystemC

ompe

tenc

y D

river

s

Com

pete

ncy

Driv

ers

Organization D

rivers

Organization D

rivers

Graphics by Steve Goodman,2009


Infrastructure




EXISTING SYSTEM

Effective Innovations are Changed to Fit the

System

Or Operate in the Shadows

(The Ghost System)

System Stability


EXISTING SYSTEM

Effective Innovations are Changed to Fit the

System

Or Operate in the Shadows

(Ghost System)

Effective System Change

EXISTING SYSTEM IS

CHANGED TO SUPPORT

THE EFFECTIVENESS OF

THE INNOVATION

(Host System)

EFFECTIVE INNOVATION



Coaching

Training

Selection



Decision Support Data System

Adaptive

TechnicalCom

pete

ncy

Driv

ers

Com

pete

ncy

Driv

ers O

rganization Drivers

Organization D

rivers



Infrastructure




Types of Leadership Needed

Different strategies for different challenges

Technical Leadership

Adaptive Leadership

According to Ron Heifitz and his colleagues, one of the biggest mistakes “leaders” make is to incorrectly identify the type of challenge they are facing

Using technical approaches for adaptive issues


The Adaptive Work of the Leader

Get on the Balcony

Identify the Adaptive Challenge

Regulate Distress

Maintain Disciplined Attention

Give the Work Back to the People

Protect All Voices» R. Heifetz and D. Laurie: The Work of

Leadership. Harvard Business Review, 1998.



Coaching

Training

Selection




Adaptive

Technical

Integrated & Compensatory

Com

pete

ncy

Driv

ers

Com

pete

ncy

Driv

ers O

rganization Drivers

Organization D

rivers



Infrastructure

Improved Child Outcomes

EBP/Program/Initiative/Framework for Effective Practices & System Change


Integrated and CompensatoryImplementation Drivers

IntegratedConsistency in philosophy, goals, knowledge and skills across these processes (S/T/C/SE/DSDS/FA/SI)

CompensatoryAt the practitioner level

At the program level



Coaching

Training

Selection




Adaptive

Technical

Integrated & Compensatory

Com

pete

ncy

Driv

ers

Com

pete

ncy

Driv

ers O

rganization Drivers

Organization D

rivers



Infrastructure for Data Systems

Improved Early Childhood Outcomes System

Reliable, Valid, Timely, Actionable Child, Fidelity, and Program-Level Data


Skill and “Will”

Current Skills Will Do

New Skills Needed

Sufficient “Will”Exists

Data as Feedback more likely to improve program quality

Data necessary but not sufficient. Need to Build Competence and Confidence even if ‘incentives’ and policy are aligned

Lack of “Will”New Policies, Funding, Incentives will be needed to ‘motivate’ and enable the use of current skill sets

Need all the Drivers in play to:• Increase confidence and competence• Be guided by data• Create a hospitable environment• Provide leadership for adaptive challenges


Skill and “Will”

Current Skills Will Do

New Skills Needed

Sufficient “Will”Exists

Data as Feedback is more likely to improve program quality

Data necessary but not sufficient. alone won’t work. Need to build Competence and Confidence even if ‘incentives’ and policy are aligned

Lack of “Will”New Policies, Funding, Incentives will be needed to ‘motivate’ use of current skill sets and create a hospitable environment

Need all the Drivers in play to:• Increase confidence and competence• Be guided by data• Create a hospitable environment• Provide leadership for adaptive challenges





Improvement Cycles




Improvement Cycle Uses

Practice-Policy Feedback Loops

Transformation Zones

Usability Testing

Rapid Cycle Teams

Problem-solving

Practice Improvement

What Do They Have In Common?


PDSA CyclesShewhart (1924); Deming (1948); Six-

Sigma (1990)

Plan – Develop specific things to do (based on DATA)

Do – Do them (make sure) (DATA)

Study – See what happens (DATA)

Act – Make adjustments (based on DATA)

Cycle – Do over and over again until the goal is reached (DATA) or the problem is clearer (go again)


Improvement Cycle Uses

Practice-Policy Feedback Loops

Transformation Zones

Usability Testing

Rapid Cycle Teams

Problem-solving

Practice Improvement


Policy Practice

Policy (Plan)

Practice (Do)

Policy

Practice

Po

licy

En

able

d P

ract

ices

(PE

P)

“Ext

ern

al”

Sys

tem

Ch

ang

e S

up

po

rt

Structure

Procedure


Policy Practice Feedback Loops

Policy (Plan)

Practice (Do)

Structure

Procedure

Policy

Practice

Form Supports Function at every level (National, State, District, Region, Agency, Practitioner)

Fee

db

ack

Stu

dy - A

ctP

oli

cy E

nab

led

Pra

ctic

es(P

EP

)

Pra

ctic

e In

form

ed P

oli

cy(P

IP)

“Ext

ern

al”

Sys

tem

Ch

ang

e S

up

po

rt


Policy Practice Feedback Loops

Policy (Plan)

Practice (Do)

Form Supports Function at every level (National, State, District, Region, Agency, Practitioner)

Fee

db

ack

Stu

dy - A

ctP

oli

cy E

nab

led

Pra

ctic

es(P

EP

)

Pra

ctic

e In

form

ed P

oli

cy(P

IP)

“Ext

ern

al”

Sys

tem

Ch

ang

e S

up

po

rt Requires:• Buy – in from all• Facing Adaptive Challenges • Linked Teams (right issues at the right level)• Linking Communication Protocols

• Transparent• Formal• Regular Intervals• Rewards Honesty• Able to be Evaluated


System Alignment

State Departments

Communities, Alliances, Councils,

Provider Agencies

Practitioners

Effective Practices

AL

IGN

ME

NTFederal

Departments

Imp

lem

en

tati

on

Te

ams

FORM SUPPORTS FUNCTION


For More InformationDean L. Fixsen, Ph.D.

919-966-3892

[email protected]

Karen A. Blase, Ph.D.

919-966-9050

[email protected]

At the Frank Porter Graham Child Development Institute

University of North Carolina

Chapel Hill, NC

www.scalingup.org http://nirn.fpg.unc.edu/

http://www.fpg.unc.edu/~nirn/resources/publications/Monograph/

mailto:[email protected]

mailto:[email protected]

http://www.scalingup.org/

http://nirn.fpg.unc.edu/



For More Information

Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M. & Wallace, F. (2005). Implementation Research: A Synthesis of the Literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231).

Download all or part of the monograph at:


To order the monograph go to:

https://fmhi.pro-copy.com/



blocked::https://fmhi.pro-copy.com/

karen a. blase, phd dean l. fixsen, phd, melissa van dyke, lcsw, allison metz, phd, sandra naoom,...

Documents

implementation fixsen

effective implementation

implementation lens

implementation processes

implementation research

practices improved outcomes

copyright dean

implementation science