Program Theory and Logic ModelsGareth Parry PhDSenior Scientist

Program Theory– Content Theory– Execution Theory

Logic ModelsPractical

Welcome and Introduction

Program Theory

What Is A Program Theory?

A conceptual explanation of how the program, as designed, will theoretically result in the desired improvement.

Comprises two major components:– Content Theory– Execution Theory

5

Kirkpatrick Framework

Kirkpatrick Evaluation of Learning Framework has four levels:

1) Experience What was the participants’ experience? Did the participants have an excellent experience working on the improvement project?

2) Learning What did participants learn? Did they learn improvement methods and begin testing?

3) Process/Behavior Did participants modify their behavior? Did they work differently and see change in their process measures?

4) Outcomes Did the organization improve its performance (via outcome measures)?

InputsResources

invested and activities

performed by each

improvementagent/entity

Level 1Participant Experience

Level 2Learning

Level 3 Process/Behavior Changes

Level 4Clinical, personal,

organizational outcomes

Content Theory:What changes will teams make that will result in improved outcomes?

Execution Theory:What will the improvement initiative do that will lead teams to adopt the process changes?

Parry et al. Recommendations for Evaluation of Health Care Improvement Initiatives, 2013, Acad Peds.

Content Theory

Rapid Response Teams

MEWS >=5Use a reliable method to identify

deteriorating patients in real

time.

When a patient is deteriorating,

provide the most appropriate

assessment and care as soon as

possible

MEWS >=4

2 Nurses1 Physician

1 Nurse1 Physician

1 Physician

ActionTheory

Core Concepts Detailed Tasks and Local

Adaptations

PRIMARY DRIVERS SECONDARY DRIVERSAIM

Apply the MEWS Score

Reduce inpatient mortality

Establish appropriate criteria for the MEWS Score

Establish a communication system for an alert

Identify the team who will respond to an alert

Train staff in how to proceed following an alert

Driver Diagram: Example

Use a reliable method to identify

deteriorating patients in real

time.

When a patient is deteriorating,

provide the most appropriate

assessment and care as soon as

possible

Train staff in the use of the MEWS Score

Execution Theory

11

1) Generating the pressure (will) for ICUs to take part2) A networked community3) Re-framing BSIs as a social problem4) Approaches that shaped a culture of commitment5) Use of data as a disciplinary force6) Hard edges

Milbank Quarterly, 2011

SummaryA well described theory:– Provides a common understanding of the intention of the model.– Allows an evaluator to understand the core concepts that

underpin the more detailed tasks carried out by those at the front lines.

– Encourages testing and local amendment, informed by underlying principles.

– Shows how all the pieces of a program are related.– Provides a high-level summary that can be the starting point for

testing in other settings.

Rossi PH, Freeman HE, Lipsey MW. Evaluation: A Systematic Approach, 7th Edition. Sage, California 2004.

Ovretveit J, Leviton L, Parry GJ. Increasing the generalisability of improvement research with an improvement replication programme. BMJ Qual Saf 2011;20:i87-i91.

Logic Models

What Are They?Diagram connection between what we want to accomplish and the way we plan to do that

Teams use them to:– Lay out plan– Expose gaps– Get multiple parties on the same page– Increase intentionality

What Are They?The Basic Logic Model: “If-Then”

Mid-term Outcomes

5

If participants start testing,

then participants will change

their behavior/ processes

Long-term Outcomes

6

If participants change

processes, then you will see changes in patient and organizational

outcomes

Resources/Inputs

Certain resources

are needed to operate

your program

1

Activities

If you have access to

them, then you can

use them to

accomplish your

planned activities

2

Outputs

If you accomplish

your planned activities,

then you will deliver the intended

amount of product/

service to the intended audience

3

Short-term Outcomes

If you accomplish

your planned activities to

the extent you intended, then

participants will see

changes in knowledge and start testing

4

Execution Theory Content Theory

Source: Adapted from WK Kellogg Foundation, Logic Model Development Guide

Project JOINTS: BackgroundAim: To assess whether a rapid spread approach applied at a state-level, will lead to increased adoption of the three interventions to reduce SSI in arthroplastic hip and knee surgery.

1. Preoperative bathing or showering with chlorhexidine

2. Preoperative nasal screening for Staphylococcus aureus carriage followed by decolonization of S. aureus carriers

3. Preoperative skin preparation with a long-acting antiseptic agent in combination

Project JOINTS: Execution Theory Summary

Uptake will occur:

1. When interventions align with local and national priorities

2. By using charismatic leaders, highly respected within their field, to describe the changes.

3. By establishing clear and practical and ambitious aims at a local level.

4. By allowing for and encouraging local tailoring.

5. Through facilitation of a networked community.

Medium Term

In participating hospitals, >90% adoption of:

Preoperative bathing or showering with chlorhexidine for 3 days prior to surgery

Preoperative a)nasal screening for Staphylococcus aureus carriage followed by b) decolonization of S. aureus carriers

Preoperative skin preparation w/a long-acting antiseptic agent in combination w/alcohol

Long Term

Reduce surgical site infections in hip & knee patients in 10 states in the U.S.

Short Term

IHI: Establish & Support Rapid Spread Network

Node- Recruit hospitals-Strengthen/ develop relationships with & among hospitals- Assist hospitals with implementation & questions-Increased capacity to coach hospitals with subject experts-Create links to related initiatives- Track hospital participation- Link hospitals to IHI resources- Provide feedback- Suggest potential exemplar hospitals-Raise awareness of Project JOINTS within state (media)

Hospital- Set aims; form teams- Access and use tools- Increased knowledge of importance of bundle- Increased leadership buy-in- Increased surgeon, admin staff, lab, & patient buy-in- Test & implement new processes & tools- Share and learn from other hospitals- Develop tools to share w/other facilities- Customized implementation (e.g., Pre-ops joints class- Provide feedback to nodes & IHI

External factors: - Other SSI focused projects e.g. Partnership for Patients , Dept of Health HAI reduction efforts and QIO 10th SOW detracted attention due to competing priorities and also prioritized reduction of SSIs- Shifting payment mechanisms for reimbursement for SSIs

Assumptions: SSI reduction is a priority; IHI & nodes are a trusted source of content; to achieve goal must effectively recruit and then engage to support flexible implementation; developing an infrastructure for teams to learn from each other will support successful implementation

Activities

Recruitment-Recruit nodes, hospitals, national partners-Build/strengthen state-level relationships-States assigned to cohort 1 or cohort 2

Ongoing Development & Refinement of Content Materials -Understand and summarize evidence of interventions-Gather existing materials from early adopters -Continuous development & refinement of materials to guide the work

Support ImplementationBuild infrastructure where nodes & hospitals can share learning and adapt implementation to their setting: in-person, phone, web-based support

Revise activities and approach based on learning from cohort 1

Outputs

Nodes in 10 states recruited & assigned to cohorts; hospitals recruited for participation; relationship & support letters from national partners

Tools and communication structure for nodes and hospitals that support implementation• IHI in-person site visits • How-to guide• Improvement tools• Patient/family 1-pager• Surgeon/MD materials &

calls• Measurement tools• Node meetings at 2011 and

2012 forum• Monthly node calls• Webinar call series• State-specific calls• Electronic communications

(website, listserv, email)• Exemplar hospital

application & awardees• Business Case • Hospital stories• Certificate of project

completion • AORN and AAOS Support

Letters• Hospital Town Hall

Meetings

Inputs

$500,000 funding

Project & communication management team: PC, PM, communications, director, field manager, clinical director

Content experts: Surgical, infection control, nursing, and improvement faculty from IHI

Network of state nodes, hospitals & national partners

Experience of sites that have done this before

Communication & technology infrastructure

Learning from campaign tools

Evidence that is ready for spread

Context

3yr federal grant funded project designed in two waves to work with all hospitals in 10 states that perform hip/knee replacements to reduce hip and knee SSIs

The project is designed to use IHI’s strong relationships w/existing nodes, hospitals and nodes’ strong relationships w/hospitals to speed adoption of evidence based practices

Project JOINTS: Logic Model

Practical

ExerciseYou have been asked to design a one year collaborative that aims to improve the uptake of three evidence-based interventions for stroke care:1. Brain imaging within 24 hours of admission to hospital (CT scan) to

confirm stroke type (ischaemic or haemorrhagic) and determine management.

2. Delivery of aspirin or an alternative antiplatelet (for patients where an antiplatelet is clinically indicated) within 24 hours of admission to modulate stroke complications and improve outcomes. For shorthand, we refer to this as “aspirin”.

3. Swallow screen within 24 hours of admission, to prevent unnecessary withdrawal of nutrition, support timely administration or modification of aspirin/antiplatelet delivery and highlight patients who need on-going management of swallow safety.

20

Exercise (30 minutes)

Use the logic model template to begin to map out, how you would describe the:– Inputs– Activities– Outputs– Short, medium and long-term outcomes

Work with others on your tableThink about the outcome you want to impact and work backwards

21

Exercise (15 minutes)10 minutes to talk through your logic models. Reflect on:– What did you find helpful about drafting a logic model?– What was challenging?– What questions do you have?– How can you move forward?

5 minute report out, for each table:– 1 thing that was helpful– 1 challenge– 1 question

22

Thank you!

Please send any follow up questions or suggestions to:

Gareth Parry email: gparry@ihi.orgTwitter: @gjparry03

23

What are you doing?

e.g., training in QI, monthly data analysis calls

What changes in organizational and patient outcomes do you expect ultimately?

e.g., 20% decrease in under-5 mortality

What changes in learning do you expect to come from your activities and outputs?

e.g., teams complete PDSA cycles

What are the reach and products of the activities?

e.g., 3 QI teams trained in improvement methods

What changes in processes do you expect as a result of your short-term outcomes?

e.g., blood protocol implemented with 80% reliability

What are you investing to get the work done?

e.g., staff, funding, space, external partners, materials, tools

External factorsWhat factors outside of the project may be a barrier or facilitator to reaching your desired outcomes?e.g., new incoming government that prioritizes x, high staff turnover in health centers,

AssumptionsWhat are you assuming about what is needed to be able to deliver the program in the above planned way? e.g., leadership is on board and the will is strong, surgeons will make the time to attend trainings and lead their QI team

LOGIC MODEL TEMPLATE: Project Title (Month Year – Month Year)

Context

What is the 2-3 bullet elevator speech? What background info is necessary to understand this project plan?

Inputs Activities Outputs Short term outcomesMedium term outcomes

Long term outcomes