Developing, Measuring, and Improving Program Fidelity:

Achieving positive outcomes through high-fidelity implementation

SPDG National Conference

Washington, DC

March 5, 2013

Allison Metz, PhD, Associate Director, NIRN Frank Porter Graham Child Development Institute

University of North Carolina

Goals for Today

• Define fidelity and its link to outcomes• Identify strategies for developing

fidelity measures• Discuss fidelity within a stage-based

context• Describe the use of Implementation

Drivers to promote high fidelity • Provide case example

“PROGRAM FIDELITY”

“The degree to which the program or practice is implemented ‘as intended’ by the program developers and researchers.”“Fidelity measures detect the presence and strength of an intervention in practice.”

Context, Compliance, and Competence

• Three components– Context: Structural aspects that encompass

the framework for service delivery– Compliance: The extent to which the

practitioner uses the core program components

– Competence: Process aspects that encompass the level of skill shown by the practitioner and the “way in which the service is delivered”

Definition of Fidelity

Purpose and Importance

• Interpret outcomes – is this an implementation challenge or intervention challenge?

• Detect variations in implementation • Replicate consistently• Ensure compliance and competence• Develop and refine interventions in the

context of practice • Identify “active ingredients” of program

Fidelity

Socially Significant Outcomes

Effective Interventions

A well-operationalized

“What”

Effective Implementation

Methods

Enabling Contexts

Formula for Success

• Clear description of the program

• Clear essential functions that define the program

• Operational definitions of essential functions (practice profiles; do, say)

• Practical performance assessment

Usable Intervention Criteria

Practice ProfilesDeveloping Fidelity Measures

Practice Profiles Operationalize the Work• Describe the essential functions that allow a model to be

teachable, learnable, and doable in typical human service settings

• Promote consistency across practitioners at the level of actual service delivery

• Consist of measurable and/or observable, behaviorally-based indicators for each essential function

Gene Hall and Shirley Hord, (2010) Implementing Change: Patterns, Principles, and Potholes (3rd Edition)

Measuring Competency

Practice Profiles

For each Essential Function: Identifies “expected” activities Identifies “developmental” variation(s)in

practice Identifies “unacceptable,” incompatible, or

undesirable practices

Sample TemplatePractice Profiles

Case ExampleDifferential ResponseImplementation Science

• Engagement• Assessment • Partnership• Goal Planning• Implementation

Communication Evaluation Advocacy Culturally Competent

Service Delivery

Functions

Case Example Differential ResponsePractice Profiles

Multiple Purpose for Implementation Practice Profiles

If you know what “it” is then:•You know the practice to be implemented•You can improve “it” •Increased ability to effectively develop the Drivers•Increased ability to replicate “it”•More likely to deliver high quality services •Outcomes can be accurately interpreted•Common language and deeper understanding

When are we ready to assess fidelity?

Stages of Implementation

Practice profiles are a part of stage-based work. When we are engaged in program development work, practice profiles operationalize the intervention so that installation activities can be effective and fidelity can be measured during initial implementation.

Stages

When are they developed?Practice Profiles

In order to create the necessary conditions for…

Creating practitioner competence and confidenceChanging organizations and systems

….we need to define our program and practice adequately so that we can install the Drivers necessary to promote consistent implementation of the specific activities associated with the essential functions of the new service(s)

Drivers

Fidelity MeasuresPractice Profiles

Start with the Expected/Proficient column

Develop an indicator for each Expected/Proficient Activity Identify “evidence” that this activity has taken place Identify “evidence” that this activity has taken place with

high quality Identify potential data source(s)

Performance Assessment

Practice Profiles

Engagement

Indicator Data Source Evidence that engagement is happening

Evidence that engagement is happening WELL

Occurrence of visitsInitialAssessmentGoal Planning

Database X visits in y months Time spent in those

visits What engagement

interventions were used during the visits?

Outcome of visits/what happened?

1.

2.

3.

4.

5.

Establishing FidelityPractice Profiles

II. Engagement- The ongoing ability to establish and sustain a genuinely, supportive relationship with family while developing a partnership, establishing healthy boundaries, and maintaining contact as mutually negotiated. The ability to identify risk and protective factors, including family’s positive supports and assess the best way to engage the family and/or who in the family to engage first.

A. Initial Engagement

SC explains service; ensures family’s understanding that service is voluntary; and describes the scope of service

Engagement Survey (ES)

SC clearly establishes the purpose of involvement with the family

ES

Family signs service agreement DB

Family signs consents DB

Health status forms completed (by SC with family) DB

Where appropriate, SC alerts Educational Advocate (EA) Educational Advocate & DB

Conducted an Engagement Activity within 60 days of the service agreementA.Who was there (list)B.What activity

DB 

SC demonstrates respect, genuineness, and empathy for all family members, as defined by the family

ES (Q:1,3,&18)

Establishing FidelityPractice Profiles

B. Ongoing Engagement (SC creates trust and buy-in) Providing services that clients view as relevant and helpful ES (Q:3,10,12,13,&14)

Parents/caregivers to listen carefully, obtain information, and begin to develop trust

ES(Q:1&25)

SC is consistent, reliable and honest with families ES (Q: 35)

SC maintains contact as negotiated with family DB(Add to DB)

Respecting the culture, racial, ethnic, linguistic, and religious/spiritual backgrounds, and sexual orientation of children, youth, and families and uses as protective factors

ES (Q:14)

Motivational Interviewing (use to elicit change) DB

Family-centered case planning and management: includes all family members in process, reflected in SP

DB/SP-SCS review(Checkbox/tab)

Use of family satisfaction measures feedback to inform work SCS

5 Steps

New or established criteria1. Assure fidelity assessors are available,

understand the program or innovation, and are well versed in the education setting

2. Develop schedule for conducting fidelity assessments

3. Assure adequate preparation for teachers/practitioners being assessed

4. Report results of the fidelity assessment promptly

5. Enter results into decision-support data system

Fidelity Data Collection

Implementation Supports

Promote High Fidelity

Fidelity is an implementation outcome

How can we create an implementation infrastructure that supports high fidelity implementation?

IMPROVED OUTCOMES FOR CHILDREN AND FAMILIES

Performance Assessment (fidelity)

Coaching

Training

Selection

Integrated & Compensatory

Systems Intervention

Facilitative Administration

Decision Support Data System

AdaptiveTechnical

Com

pete

ncy

Driv

ers O

rganization Drivers

Leadership Drivers

Integrated & Compensatory

Building the InfrastructurePractice Profiles

Differential Response Essential FunctionsSelection Engagement Assessment Partnership Communication Evaluation

What are prerequisites (skills, value, knowledge) practitioners ideally would have when hired or redeployed to implement DR?

What features of DR practice would be helpful to assess through behavioral rehearsals during the selection process? What aspects of DR practice would be important to include in the caseworker job descriptions?

Building the InfrastructurePractice Profiles

Differential Response Essential FunctionsFacilitative

AdministrationEngagement Assessment Partnership Communication Evaluation

Will new policies or procedures need to be developed by the State or County to support DR essential functions?

What role does leadership need to play at State and County levels to reduce administrative barriers to DR practice?

How can State leadership institute policy-practice feedback loops?

Function X Driver ExamplePractice Profiles

Success Coach Example

Engagement KSAs Training

The ongoing ability to establish and sustain a genuinely supportive relationship with family while developing a partnership, establishing healthy boundaries, and maintaining contact as mutually negotiated.

The ability to identify family’s positive supports and assess the best way to engage the family and/or who in the family to engage first.

• Develop rapport and build relationship with family members

• Listen actively and openly to families’ perspective, needs and story

• Complete life circles, genograms, eco maps, case mapping, and other engagement/assessment tools

• Use basic Motivational Interviewing techniques with families to overcome resistance

• Motivational Interviewing (MI)

• Family Preservation Training

• Cultural Competency Training

Improvement Cycles

Shewhart (1924); Deming & Juran (1948); Six-Sigma (1990)

Practice Profiles

Plan

DoStudy

Act

Make Adjustments Decide what to do

Do it (be sure)Look at the results

Cycle Do over and over

again until intended benefits realized

Improvement CyclesPractice Profiles

PDSA cycles•Competency Drivers•Organization Drivers•Leadership•Essential Functions of the Profile•Data Collection activities

Practice Profiles and accompanying implementation supports will change multiple times during initial implementation

Cycles

Program Improvement

Program Review Process •Process and Outcome Data•Detection Systems for Barriers •Communication protocols

Questions to Ask •What formal and informal data have we reviewed? •What is the data telling us?•What barriers have we encountered?•Would improving the functioning of any Implementation Driver help address barrier?

Fidelity Data

Results from Child Wellbeing Project

Case Example

Component T1

Selection 1.44

Training 1.33

Coaching 1.27

Perf. Assessment 0.78

DSDS 0.18

Fac. Administration 1.38

Systems Intervention 1.29

Average Composite Score 1.1

Fidelity (% of cases) 18%

Case management model involved intense program development of core intervention components and accompanying implementation drivers.

Clinical case management and home visiting model for families post-care.

Case Example

• How did Implementation Teams improve fidelity?

– Intentional action planning based on implementation drivers assessment data and program data

– Improved coaching, administrative support, and use of data to drive decision-making ; adapted model

– Diagnosed adaptive challenges, engaged stakeholders, inspired change

Using Data to Improve Fidelity

Results from Child Wellbeing Project

Case Example

Component T1 T2 T3

Selection 1.44 2.00* 1.89*

Training 1.33 1.5* 1.10

Coaching 1.27 1.73* 1.83*

Perf. Assessment 0.78 1.34 2.0*

DSDS 0.18 1.36 2.0*

Fac. Administration 1.38 2.00* 2.0*

Systems Intervention 1.29 1.86* 2.0*

Average Composite Score 1.1 1.68* 1.83*

Fidelity (% of cases)

18% 83% 83%

Success Coach model involved intense program development of core intervention components and accompanying implementation drivers

Positive Outcomes

• Stabilized families• Prevented re-entry of children

into out of home placements

High Fidelity

Did high fidelity implementation lead to improved outcomes? Early outcomes include…

Methods, Resources and Feasibility

If fidelity criteria are already developed1. Understand reliability and validity of instruments

a. Are we measuring what we thought we were?b. Is fidelity predictive of outcomes?c. Does fidelity assessment discriminate between programs?

2. Work with program developers or purveyors to understand the detailed protocols for data collection

a. Who collects the data (expert raters, teachers)b. How often is data collectedc. How are data scored and analyzed

3. Understand issues (reliability, feasibility, cost) in collecting different kinds of fidelity data

a. Process data vs. Structural data

Fidelity Data Collection

Program Fidelity

• Fidelity has multiple facets and is critical to achieving outcomes

• Fully operationalized programs are pre-requisites for developing fidelity criteria

• Valid and reliable fidelity criteria need to be collected carefully with guidance from program developers or purveyors

• Fidelity is an implementation outcome; effective use of Implementation Drivers can increase our chances of high-fidelity implementation

• Fidelity data can and should be used for program improvement

Summary

Program Fidelity

Examples of fidelity instruments

•Teaching Pyramid Observation Tool for Preschool Classrooms (TPOT), Research Edition, Mary Louise Hemmeter and Lise Fox•The PBIS fidelity measure (the SET) described at http://www.pbis.org/pbis_resource_detail_page.aspx?Type=4&PBIS_ResourceID=222Articles •Sanetti, L. & Kratochwill, T. (2009). Toward Developing a Science of Treatment Integrity: Introduction to the Special Series. School Psychology Review, Volume 38, No. 4, pp. 445–459.•Mowbray, C.T., Holter, M.C., Teague, G.B., Bybee, D. (2003). Fidelity Criteria: Development, Measurement and Validation. American Journal of Evaluation, 24 (3), 315-340.•Hall, G.E., & Hord, S.M. (2011). Implementing Change: Patterns, principles and potholes (3rd ed.)Boston: Allyn and Bacon.

Resources

Stay Connected!

nirn.fpg.unc.edu www.scalingup.org

www.implementationconference.org

Allison.metz@unc.edunirn@unc.edu