Using Audit and Feedback to Improve Quality of Care

December 7-8, 2012

What are we talking about here?

A definition of Audit and Feedback:

summary of clinical performance over a specified period of time and provision to health care provider(s)/organization(s)

NOT: reminders at the point of careNOT: relay of individual, patient-specific clinical data between providers

Why Talk about Feedback?

“the limiting factor to Self-Directed Learning for practicing physicians may be their limited ability to accurately self-assess”

Davis et al. JAMA 2006 Sep 6;296(9):1094-1102.

“You can’t manage what you can’t measure”

Feedback and Behaviour Change

Old behaviours

NewbehavioursPerformance

FeedbackPerformance

Feedback

ObjectiveTo develop a research agenda to improve the effectiveness of audit and feedback interventions to reliably improve quality of care.

Goals:1. Establish the foundation for a program of research aimed at improving

the effectiveness of AF interventions1a) prioritize elements of AF design that should be tested1b) identify settings that provide opportunities for testing AF design1c) utilize methodology to efficiently test and improve upon AF interventions

2. Develop a knowledge translation plan:2a) integrated collaboration with clinical, administrative, policy stakeholders2b) disseminating the results of the meeting 2c) developing a wiki-based platform to support ongoing collaboration 2d) establishing plans for new operating grants that will carry out the agenda developed in the meeting

INTRODUCTIONS

Agenda

Results from the Cochrane Review

• What do we know about the impact of AF?– Cochrane Review and Meta-regression– Cumulative Meta-analysis (not yet published)

Findings of 2006 Cochrane Review

• 88 comparisons from 72 studies• dichotomous outcomes median adjusted risk difference of

compliance with desired practice = 0.05 (IQR 0.03 to 0.11)

• “Intensity of audit and feedback might also help to explain variation in the absolute effect (p = 0.04).”

• “Intensive”(individual recipients) AND ((verbal format)OR (a supervisor or senior colleague as the source)) AND (moderate or prolonged feedback)• “Non-intensive” ((group feedback) NOT (from a supervisor or senior colleague)) OR ((individual feedback) AND (writ-ten format) AND (containing information about costs or numbers of tests without personal incentives))• “Moderately intensive”(any other combination of characteristics than described in Intensive or Non-intensive group).

Jamtvedt G et al. Cochrane Database of Systematic Reviews 2006, Issue 2. Art. No.: CD000259. DOI: 10.1002/14651858.CD000259.pub2.

Unclear how to “do” AF

“an unreliable approach to quality improvement until we learn how and when it works best”

Foy R. et al. BMC Health Services Research, 2005;5, 50.

2012 Update: Changes in Review Methodology

ANALYSIS OF HETEROGENEITY

Meta-regression• Format (verbal; written; both; unclear)• Source (supervisor or senior colleague; review organization or employer;

investigators; unclear)• Frequency (weekly; monthly; less than monthly; one-time)• Instruction for improvement (explicit goal; action plan; both; neither)• Direction of change required (increase behaviour; decrease; mix/unclear)• Recipient (physician; other health professional)• Baseline performance (continuous)• Risk of bias (high; unclear; low)

Plus exploratory analyses:• Targeted behaviour (prescribing; test ordering; dm/cvd)

Characteristic N % Characteristic N %Publication Year Classification of Intervention 2006-2010 32 23 AF alone 49 35 1996-2005 76 54 Multifaceted 91 65 1986-1995 20 14 case mgmt/team change 3 2 before 1986 12 9 clinician education (not outreach) 48 34Risk of Bias educational outreach 28 20 Low 45 32 clinician reminders, cdss 17 12 Unclear 70 50 patient intervention 8 6 High 25 18 continuous qi 9 6Number of Arms in Trial financial incentives 5 4 Two 98 70 Three 22 16 Targeted Behaviour(s) Four 20 14 DM/CVD mgmt 30 21Clinical Setting Laboratory testing/radiology 21 15 Outpatient 94 67 Prescribing 31 22 Inpatient 36 26 Other 50 41 Other/unclear 10 7 Targeted Health Professional(s) Medical Specialty(s) Physician 121 86 GP 84 60 Nurses 16 11 Internists 60 43 Pharmacists 5 4 Other 40 29 Other 3 2

2012 Update: Study Characteristics

Characteristics of Included Studies:Features of Feedback Design

Characteristic N % Characteristic N %

Format

verbal only 13 9

written only 84 60

both 32 23 Instructions for Improvement unclear 11 8 Goal-setting 11 8Source Action planning 41 29 supervisor/colleague 13 9 Both 4 3 employer 15 11 Neither 84 60 investigators/unclear 112 80 Direction of Change RequiredFrequency Increase current behaviour 57 41 weekly 11 8 Decrease current behaviour 29 21 monthly 19 14 Mix or unclear 55 39 less than monthly 36 26

once only 68 49

Meta-RegressionCharacteristic Effect Characteristic EffectFormat of feedback p=0.020 Instructions for improvement p<0.001Verbal 3.4 Target/Goal 2.52Written 9.5 Action plan 9.57Both Verbal and Written 11.2 Both Goal and Action plan 11.09Not clear 5.3 Neither 6.2Source of feedback p<0.001 Direction of change required p<0.001A clinical supervisor or colleague 16.5 Increase current behaviour 4.34A ’PSRO' or employer 2.4 Decrease current behaviour 10.54The investigators 5.0 Chg behaviour/mix/unclear 7.16Not clear 5.5 Baseline performance p=0.007Frequency of feedback p<0.001 at 25% 9.11Frequent (up to weekly) 1.4 at 50% 7.07Moderate (up to monthly) 9.8 at 75% 5.03Infrequent (less than monthly) 4.8 Profession of recipient p=0.561Once only 2.6 Physician 7.9Unclear; 18.1 Non-physician 6.8

Risk of bias p=0.679Low risk of bias 7.68Unclear 7.02High risk of bias n/a

Meta-Regression - Exploratory

Characteristic Effect

Type of professional practice P<0.001Diabetes/CVD 5.91Laboratory testing/radiology referrals 4.21Prescribing 11.11Other 4.71

Direction of change required P=0.525Increase current behaviour 6.64Decrease current behaviour 7.13Change behaviour or mix or unclear 5.7

Summary

• AF is effectiveo One quarter of studies find relatively large effect (>16% aRD)

• AF may be more effective when:o baseline performance is low, o the source is a supervisor or senior colleague, o delivered both verbally and written, o provided more than once, o and includes both explicit targets and an action plan

• Targeted behavior plays an important role

Growing Literature; Stagnant Science?

1985 1990 1995 2000 2005 20100

20

40

60

80

100

120

140

160

Cumulative Meta-Analysis

Feedback and Behaviour Change

Old behaviours

NewbehavioursPerformance

FeedbackPerformance

Feedback

Questions?

Clinical: noah.ivers@wchospital.caPersonal: noahivers@gmail.com

Agenda

Role of Theory

• What can it offer?

Illustrative Examples

• Good, bad, and ugly

Moving forward

Tension between theory and practice?

Agenda

Small groups: part 1

• Building a complete list of potential effect modifiers

• What are the things to consider when thinking about doing audit and feedback interventions?

Small groups: part 2

• Prioritizing effect modifiers for testing

• What are the top five things that we need to know in order to sort out how to make audit and feedback more effective?

Agenda

Agenda

Agenda

• What is the impact of engaging the recipient in design of AF

• What is the impact of adding to AF

5 Research Questions1. What is impact of engaging the recipient in design implementation vs.

engaged in none of them2. What is the impact of adding to AF:

• Incentives/penalties – financial, CME credit + licensing• Tools/practise aid – eg. Clinical decision tool…

• Practise redesign, coaches, facilitation, mentorship3. Skill of the person providing feedback make a significant difference4. Take the top quintile of AF studies and replicate their interventions5. The 5 most important aspects of AF to study are:

1. Frequency, individual vs. group, evidence based, in person vs electronic delivery, number of targets

2. Developing strategies for replicating and implementing successful intervention in other settings, looking at

Dinner 6.30pmEmpire Grill 47 Clarence Street (in Byward Market) (About 20

min walk)