the calgary audit and feedback framework: a practical

RESEARCH Open Access

The Calgary Audit and FeedbackFramework: a practical, evidence-informedapproach for the design andimplementation of socially constructedlearning interventions using audit andgroup feedbackLara J. Cooke1* , Diane Duncan2, Laura Rivera2, Shawn K. Dowling3, Christopher Symonds4 and Heather Armson5

Abstract

Background: Audit and feedback interventions may be strengthened using social interaction. The Calgary office ofthe Alberta Physician Learning Program (CPLP) developed a process for audit and group feedback for physicians.This paper extends previous work in which we developed a conceptual model of physician responses to audit andgroup feedback based on a qualitative analysis of six audit and group feedback sessions. The present studyexplored the mediating factors for successfully engaging physician groups in change planning through auditand group feedback.

Methods: To understand why some groups were more interactive than others, we completed a comparativecase analysis of the six audit and group feedback projects from the prior study. We used framework analysisto build the case studies, triangulated our observations across data sources to validate findings, compared thecase studies for similarities and differences that influenced social interaction (mediating factors), and thematicallycategorized mediating factors into an organizing framework.

Results: Mediating factors for socially interactive AGFS were a pre-existing relationship between the program teamand the physician group, projects addressing important, actionable questions, easily interpretable data visualization inthe reports, and facilitation of the groups that included reflective questioning. When these factors were in place (cases1, 2A, 3), the audit and group feedback sessions were dynamic, with physicians sharing and comparing practices, andraising change cues (such as declaring commitments to de-prescribing, planning educational interventions, andimproving documentation). In cases 2C–D, the mediating factors were less well established and in these cases, thesessions showed little physician reflection or change planning. We organized the mediating factors into a frameworklinking the factors for successful sessions to the conceptual model of physician behaviors which these mediatingfactors drive.

(Continued on next page)

* Correspondence: [email protected] of Clinical Neurosciences, Cumming School of Medicine,University of Calgary, UCMC Area 3, 3350 Hospital Drive NW, Calgary, AB T2N4N1, CanadaFull list of author information is available at the end of the article

© The Author(s). 2018 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link tothe Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Cooke et al. Implementation Science (2018) 13:136 https://doi.org/10.1186/s13012-018-0829-3

http://crossmark.crossref.org/dialog/?doi=10.1186/s13012-018-0829-3&domain=pdf

http://orcid.org/0000-0001-5417-571X

mailto:[email protected]

http://creativecommons.org/licenses/by/4.0/

http://creativecommons.org/publicdomain/zero/1.0/

(Continued from previous page)

Conclusions: We propose the Calgary Audit and Feedback Framework as a practical tool to help foster sociallyconstructed learning in audit and group feedback sessions. Ensuring that the four factors, relationship, questionchoice, data visualization, and facilitation, are considered for design and implementation of audit and groupfeedback will help physicians move from reactions to their data towards planning for change.

Keywords: Audit and feedback, Feedback, Social learning theory, Framework, Practice improvement, Professionaldevelopment, Comparative case study, Physician learning, Implementation, Knowledge translation

BackgroundAudit and feedback (AF) is a widely published method ofproviding performance data to physicians to help themtranslate knowledge into practice [1]. It has been shownto be more effective in helping physicians change their be-havior than many traditional models of professional devel-opment [2]. However, the effectiveness of published AFinterventions is variable [1]. Several authors have called at-tention to this issue, citing the need for further study sothat the reasons for varied effectiveness of AF can be morefully understood and addressed [3–7].Here, we extend our previous work, which examined

physician responses to a novel type of audit and groupfeedback (AGF) and presented a conceptual model ofphysician responses to AGF sessions (AGFS) [8]. We ob-served that physicians engaged in planning for changemore robustly in some AGFS than in others. Thepresent study explores factors that influenced the socialinteractions in those AGFS.Our understanding of what influences AF effectiveness

is informed by three main areas of literature: implemen-tation science, motivational and behavior change theory,and the educational feedback literature [5]. Colquhounet al. have emphasized the need to draw from thesedifferent domains in order to optimize AF design andimplementation [5].Frameworks and theories from these fields can help us

to understand the factors that influence implementation[9–19]. One widely cited, evidence-based framework isiPARIHS [17], which identifies four key domains thatcan be used to determine why an intervention may ormay not be successful: the innovation, the recipient, thecontext, and the facilitation [17].The authors of iPARIHS described facilitation as the

“active ingredient” for implementation [17]. It was de-fined as “the construct that activates implementationthrough assessing and responding to characteristics ofthe innovation and the recipients” in context ([17], p 8).iPARIHS situates the success of implementation uponwhether the facilitator can enable the recipients to makethe desired change [17].Because many published AF interventions describe

passive, non-facilitated feedback in the form of physician‘report cards’, the lack of attention to facilitation of

feedback in AF is a potential criticism and may explainsome of the variation in AF interventions [1, 4, 5, 7].Brehaut et al. published 15 recommendations to en-

hance effectiveness of “practice feedback” [7]. Thoserelevant to the current study include choosing the rightitems on which to provide feedback (aligned with localpriorities, actionable, specific), providing individualizeddata with relevant comparators, integrating summarymessages and data visualization, using social interactionto construct feedback, managing cognitive load, and ad-dressing barriers to change [7].The use of social interaction to construct feedback un-

derpins the design of AGFS described in the presentstudy and stems from Social Learning Theory [20] andthe work of Vygotsky and Ajzen, who emphasized thatefficient learning can occur through observation ofothers’ behaviors, the rewards or consequences of others’behaviors, and the social norms that develop in groupsettings [7, 20–22].Similarly, the Theoretical Domains Framework

(TDF) explores factors that impact behaviors [23].The TDF identifies 14 domains to consider in imple-mentation [23]. These include knowledge and skills,beliefs about capability for change, goal setting, theenvironmental context and exploration of social influ-ences on implementation [23].The medical education literature about feedback up-

take highlights several aspects of feedback delivery.Ideally, feedback occurs in an in-person, facilitated,coaching-oriented manner within the setting of a trust-ing, respectful relationship between the provider and re-cipient of feedback [24–27].In the R2C2 model, Sargeant et al. emphasize the pri-

macy of the relationship between feedback providers andrecipients [25]. Likewise, the R2C2 model focuses on un-derstanding and accepting feedback prior to coachingfor change [25, 28, 29]. Watling et al. highlight the valueof the credibility of the feedback provider as perceivedby the recipient [24, 26]. In an overview of the educa-tional feedback literature, Telio et al. proposed a con-struct of “educational alliance,” which may influencefeedback uptake [27].The Calgary office of the Alberta Physician Learning

Program (CPLP) delivers AF to groups of physicians.

Cooke et al. Implementation Science (2018) 13:136 of 18

This team has delivered over 30 AF projects on variousclinical topics, locally and provincially, addressing prac-tice variation and appropriateness. Some projects en-gaged physicians more than others and we wished toexplore why.In a previous study, we described our approach to AF:

audit and group feedback sessions (AGFS) [8]. These AGFSwere designed based upon the principles of social learningtheory and best practices from the education feedback andimplementation science literature [7, 17, 23–27]. In theAGFS, physician groups participated in face-to-face, [25,27] facilitated group feedback sessions with peers, duringwhich they reviewed reports containing their own individu-alized performance data (along with anonymized peer com-parators) in order to identify opportunities, barriers, andenablers for making change [7, 17, 23, 25–27].We investigated the behaviors of physicians in AGFS in

order to capture their reactions and engagement with thedata and how the presence of peers influenced the direc-tion of group discussions [8]. Through a qualitative ana-lysis, we developed a conceptual model of how physiciansreact and interact in AGFS. Physicians expressed initial re-actions to the data (skepticism, interest) and then transi-tioned through several discrete behaviors: understandingand questioning, justifying and contextualizing the data,and reflection and sharing of practices, before beginningto raise ‘change cues’ and make change plans. Change cueswere defined as “turning points in the group discussion,initiated by a brief comment highlighting the importanceof a performance gap revealed by the data reports,” andwere usually raised by a group participant rather than thefacilitator, who was not a group member [8].Qualitative analysis of the AGFS transcripts showed

that the degree of interaction and engagement of thephysicians varied between AGFS, as did the groups’ ten-dency to raise change cues and plan for change [8]. Thisfollow-up study sought to explore the factors mediatingsocial interaction during the six AGFS.We present the results of a comparative case study of

the AGFS described in the previous study [8]. The aimwas to understand what factors contribute to the suc-cessful engagement of physicians in change planningand to develop a practical, evidence-informed tool toguide AGF design and implementation.

MethodsEthics approval for this work was received for each casefrom the Conjoint Health Research Ethics Board:REB13-0075 (case 1); REB14-0484 (cases 2a, 2b, 2c, 2d);REB13-0459 (case 3).

SettingThis analysis of the work of the CPLP between 2014 and2016 took place at the Cumming School of Medicine at

the University of Calgary. The CPLP is funded by theprovincial medical association, to deliver audit and feed-back reports about individual practice performance toAlberta physicians. Most projects are initiated when aphysician group (such as a department or clinic) ap-proaches the program with a clinical question. Programstaff clarify the question; if it is amenable to AF, an algo-rithm for data extraction is developed and data isaccessed from relevant provincial repositories to createthe AF report. Each project has a unique, multi-pageaudit and feedback report reflecting the amount of dataand information needed to answer the physician groups’questions. Confidential reports for participating physi-cians contain individual data, anonymous group compar-ators, and relevant references reflecting best practices.The project culminates in a facilitated AGFS in which

consenting physicians have their AF reports (provided atleast 1 week before the AGFS), work as a group with aCPLP facilitator to review them, and identify opportun-ities, barriers and enablers for change [10, 23]. In thisstudy, across all AGFS, the facilitator was the CPLPmedical director, who was not a member of any of thephysician groups. The sessions were attended by CPLPstaff who observed the process and served as projectmanagers for each case from conceptualization todelivery.The process is depicted in Fig. 1.

Study designWe wished to understand why social interaction be-tween physicians in AGFS varied across cases [8]. Com-parative case analysis is an appropriate approach whencontext, culture, and system factors may influence a pro-gram [30, 31]. We used framework analysis to build theindividual cases for this research [32–35]. The overalldesign of the project was as follows: (1) identification ofdata sources, (2) familiarization with data sources, (3)development of a program model or “change theory,” (4)creation of a framework table from the program model,(5) framework analysis to extract and index data to buildindividual case studies, (6) comparisons across cases toidentify similarities and differences believed to influ-ence social interaction, (7) thematic organization ofsimilar findings into key “mediating factors,” (8) cre-ation of an organizing framework linking the mediat-ing implementation and design factors for socialinteraction in AGFS to the conceptual model of phys-ician behaviors [8].

ParticipantsParticipants included the physicians who participated inthe AGFS and the CPLP staff who created the AF re-ports and delivered the AGFS through collaborative


relationships with the physicians. AF participants pro-vided written informed consent to allow access to theiradministrative health data for purposes of creating AFreports and to record, evaluate, and study the AF ses-sions in which they participated.

Characteristics of staff who contributed are describedin Table 1. The staff worked closely with the physiciangroups to develop the AGFS and observed the sessions.A typical project would take 1 year to complete. TheCPLP staff had extensive longitudinal contact with

Fig. 1 The CPLP process from clinical question to AGFS. Physician groups bring clinical questions of interest for review by the CPLP. The CPLPteam reviews the questions for appropriateness for audit and feedback. Consideration is given to impact, reach, actionability, and accessibilityof the data. CPLP collaborates with data custodians to make individualized AF reports for consenting doctors. The confidential reports includeindividual data with anonymous peer comparators and relevant best practice information. Consenting physicians then participate in a facilitatedgroup feedback session with their peers, led by a CPLP and/or participant facilitator. As a group, the physician peers review each aggregate datapoint, along with their own performance reports and seek opportunities for practice improvement

Table 1 Descriptions of training and roles of CPLP staff and the research team

CPLP staff participants Description

Project managers (2) The two CPLP project managers were experienced in audit and feedback project development and hadformal project management training.

CPLP facilitator (CS) A physician with experience in education and clinical research, who worked at the CPLP for 3 years as theprogram medical director.

Research team members

LC An academic clinician educator with training in medical education research and experience in knowledgetranslation and audit and feedback, who oversaw the CPLP program at the time when these AGFS occurred.

CS See above. CS was the medical director at the time the AGFS were completed.

DD CPLP project and program manager during the time that these AGFS took place.Has training in education and knowledge translation

LR Research associate in CPLP when the AGFS were conducted. Training in epidemiology, quantitative andqualitative methodologies.

SD A physician with training in epidemiology and knowledge translation who became the program medicaldirector after the AGFS were developed.

HA An academic professor of family medicine with training in knowledge translation, medical education andqualitative methodologies with extensive research experience with physician learning and feedback.


members of the physician groups and familiarity withprocesses and contextual and cultural elements thatwere observed over the course of the projects.Descriptions of the research team are also included in

Table 1 in order to acknowledge their orientation, posi-tions, and perspectives at the outset of this research asthese perspectives likely inform our analysis.It is important to note that while CS, LR, DD, and LC

participated in the development and evolution of theCPLP processes over time, HA and SD were not in-volved in the building of the original projects or evolu-tion of the processes described in this study, but ratherjoined the research team afterwards. Their perspectiveswere routinely sought in an effort to balance and miti-gate any biases or pre-conceived ideas of other teammembers more intimately involved with the cases.

Case definitionFor purposes of this study, a “case” was defined as anAGFS with an individual physician group, including theprocesses of working with the group to refine the ques-tion, prepare the report and coordinate, and facilitatethe AGFS.Cases selected for this study included all AGFS that

took place through the CPLP between January 2015 and

January 2016. These cases were selected because theywere offered in short succession, such that researcherscould begin to understand the respective cultures, pat-terns, and influences within each group.

Data collectionThe research team began by identifying possible datasources from which to derive the six case studies. Datasources are listed and described in detail in Table 2.These included sample AF reports from each case, thetranscripts and qualitative analysis from the prior study[8], basic data about the projects from a CPLP trackingdatabase (number of reports, dates, ethics approvals,etc.), a process evaluation document that was written forcase 1, field notes that were collected directly into ourframework table to capture the observations of theresearch team as they explored the cases, and notesfrom structured interviews with CPLP staff who werepresent at the AGFS to corroborate and validate thefindings of the research team (detailed later in themethods section).Next, the research team built a program model

comprised of possible elements influencing the AGFprojects [30, 31].

Table 2 Description of data sources for the framework analysis and how they were used by the research team

Data sources Description of how data was collected/used

Sample anonymous AF reports for each project The research team reviewed the AF reports and described the quality of data visualization foreach case. Through familiarization with the data from the first study, the team capturedparticipant responses to the reports which could support or refute the team observations aboutthe reports. Observations were noted in the framework table (Exemplar graphs from AF reportsare shown in Figs. 3 and 4).

Process evaluation for case 1 A formal process evaluation of case 1 was conducted by a senior CPLP team member at thetermination of that project. This was a seven-page document outlining processes, procedures,stakeholders, and lessons learned for this project. This report was reviewed by the researchers(LC, DD) information from the report that provided information about influencing socialinteraction in case 1 was added to the case 1 description in the framework table.

Transcripts and qualitative analysis of AGFS An inductive thematic analysis of the transcripts for the six AGFS was conducted in a priorstudy [8]. Team members who reviewed these transcripts repeatedly for the first study madeobservations about the interactivity, collegiality, and change orientation of the groups whichwere included in the case analysis. These observations were collected as “field notes” recordeddirectly into the framework table during research team meetings to discuss the case studies.They were corroborated by returning to review the coding in the transcripts and in interviewswith program staff who were present at those AGFS.

Structured interviews of CPLP staff A staff member who observed each AGFS was interviewed using a the framework table as astructured guide. They were asked to comment about each element in the framework for eachcase. Their responses were captured in notes entered directly into the framework analysis table.Likewise, the facilitator of the six sessions was interviewed and all responses were captured inthe same document.

CPLP tracking document Basic information about each AGFS was captured by the CPLP staff in a tracking documentmaintained by the program. These included key performance indicators such as numbers ofreports distributed, timing of AGFS.

Observations of the research team A consensus meeting of members of the research team (LC, HA, LR, DD) was held to shareand compare observations of the AGFS and AGF projects. These observations were capturedand noted directly into the framework table. This content was reviewed iteratively during thecase analysis and during the development of the CAFF to ensure accuracy and consensus aboutthe findings for each case that was analyzed. Observations of the research team were triangulatedwith the other data sources for corroboration.


Building the program modelDeveloping a program model is an important early stepin a comparative case analysis [30, 31]. The model iscomprised of elements that are expected to influence thecases [30, 31]. The research team worked collaborativelyover several meetings to diagram the program modelbased on literature that describe factors that influenceimplementation success and acceptance of feedback [7,17, 23–27]. Based on the team’s tacit knowledge derivedfrom their collected experience in developing and deliv-ering audit and feedback and educational feedback aswell as their familiarity with the literature, additional ele-ments not specified in published frameworks were addedto the program model. The team met, drafted, andre-drafted the model iteratively until there was consen-sus on the likely factors influencing social interaction inAGFS. This program model, described in the “Results”section, was used as the a priori framework for collect-ing and organizing information from all of the datasources (Table 3).The program model became the a priori framework

into which all data was later indexed in order to buildeach case study. The framework was put into a table(the “framework table”) with the elements of the frame-work on the vertical axis and the cases on the horizontalaxis.Three 60-min interviews structured upon the a priori

framework comprised one source of data about thecases. The interview participants (the CPLP medical dir-ector and two CPLP project managers, detailed inTable 1) were asked to describe their observations of thecases with respect to each of the framework compo-nents. Their responses were recorded directly into theframework table. They were used to corroborate and/oradd to the observations and findings of the research

team. The notes from the interviews were reviewed, con-densed, and reviewed again by the research team andsummarized in the case analysis table for purposes ofthe publication.

Data analysisFramework analysis was the primary mode of data ana-lysis in this project [32–35]. This is a method of organiz-ing textual data according to an a priori “coding”framework and is commonly used in social sciences re-search [32–35]. The first step involves familiarizationwith the data through iterative reading and discussion.Familiarization with the AF reports and the qualitativedata from the prior study took place during project de-velopment within the CPLP and in conducting the priorstudy [8]. Next was developing the program model and apriori framework as outlined in the previous section.Data were indexed in the framework table by searchingthe data sources for evidence for each element of the apriori framework in each case and populating the tablewith findings. If observations from the data sources didnot align with the framework, there was opportunity toadd additional elements.The research team then met to review the case studies

and develop consensus on the case descriptions. Corrob-orating evidence for the observations were sought by tri-angulating the evidence from different data sources.The next step in data analysis was the case compari-

son. Once consensus was reached on the case descrip-tions, the research team worked collaboratively toidentify similarities and differences across the cases.Each of these was considered by the team as to whetheror not they influenced the social interactions in the re-spective AGFS.

Table 3 Program model for the comparative case analysis

Program model elements Components

Innovation [17] The clinical question upon which the project was based

Style of report [7] Design of the report, co-creation of report design with CPLP and physician leadsfrom participating groups

Available gold standard/benchmark for the clinical question [7]

Recipients [17] Who participants were and how they interacted with one another [7, 17, 23]

Context [17] Project origin/historyGroup dynamic (collegiality/culture) and leadership involvement in the sessions [23]Pre-work/relationship building between PLP and participant group [25]Co-creation of data/metrics [7]Proximity of doctor to data/patient (perceived control) [23]

Facilitation [17, 24–26] CPLP led or co-facilitated with a participant physician-leadCoaching-oriented facilitation [25]

Physician engagement/change talk/change planning [8] Interactivity of the AGFS, extent of change talk

Outputs Further project development with CPLPEmergence of autonomous data reporting by the physician group (dashboards)Measured/reported behavior change by the physicians


The final step in the analysis was to create an organizingframework. The research team iteratively diagramed aframework that captured and organized factors that wereidentified as likely influences on physician engagement inthe AGFS. Similar elements were grouped thematicallyunder a single “mediating factor” for AGFS success. Eachmediating factor was then ordered to reflect the CPLP pro-cesses and the role these processes might play in the phys-ician behaviors identified in the previous study [8].

ResultsParticipantsA total of 99 AF reports were distributed and 50 physi-cians participated across the six AGFS. Two CPLP pro-ject managers and the CPLP Medical Director wereinterviewed for data collection.

Program modelThe program model, shown in Table 3, is comprised ofeight elements. These elements were derived from twowidely cited frameworks that describe influencing factorsin implementation [17, 23], Brehaut’s recommendationsfor enhancing “practice feedback”, and elements fromseveral models of educational feedback [7, 24–27]. Inaddition to the factors derived from the literature, theteam added these elements: AGFS interactivity andchange talk and group dynamic.

Case descriptionsCase 1 was a project exploring practice variation withspecialists involved in a specific surgical procedure. Thegroup demonstrated a collegial relationship during theAGFS with a high frequency of sharing and reflecting oncommon practices and raising change cues which led tochange talk between the participants during the AGFsession. Representative examples are listed in Table 4.Participants in case 1 were familiar with the CPLP ap-

proach and staff from a prior project with the program.Their AF reports were co-created with input from thephysician lead and three other group members. Thephysician lead was a highly respected leader in thegroup. These “group champions” were involved with allaspects of project design from the clinical question tothe AGFS. The project lead shared their individual datawith the group while co-facilitating the AGFS. Thisgroups’ clinical question related to treatment decisionsthat were largely under their direct control.Cases 2A–D addressed a de-prescribing question de-

rived from Choosing Wisely Canada recommendations.These cases took place at four adult hospitals, each withseparate groups of physicians, leaders, institutional cul-tures, and serving unique patient populations in differ-ent parts of the city. For these reasons, these AGFS aretreated as separate cases. These physician groups were

reviewing baseline data. While there were varying de-grees of change talk and interactivity across these fourAGFS, it was uncommon across all four AGFS for physi-cians to compare and share their practices with oneanother.Case 2A occurred at a hospital serving an area with a

large immigrant population. This physician group alsoprovided palliative care. The AGFS focused heavily ondiscussion about the uniqueness of the patient popula-tion. There was skepticism about the data on the basisof the uniqueness of this group’s patient population ac-counting for the variations in prescribing. The projectlead was from this group but was unable to attend thissession. The group discussion was interactive and somechange cues arose (see Table 4).Case 2B took place in a very large tertiary academic

health center serving a complex, high-acuity population.The group was comprised of physicians who worked to-gether for many years. The session was interactive andled to change planning. A nurse with a formal qualityimprovement role attended this session. A physician par-ticipant prompted several interactions around changeplanning with the allied health staff who sharedin-patient care.Case 2C took place in a smaller, community-based

hospital serving a predominantly elderly population.One of the physicians for the project was based withinthis group, attended the session, helped facilitate, andshared their data with the group. This session was lessdynamic than cases 2A and 2B. During the AGFS, therewere few instances of comparing and sharing practiceshowever, two change cues were raised and led to changetalk around improving documentation and dischargesummaries as a way to improve community prescribing.Case 2D took place in a newer, smaller hospital on the

outskirts of the city. Because of its short history, thishospital has a unique physician culture and medical cul-ture with a focus on patient and community-centeredcare. Participants expressed that they believed that themake-up of their group and their consultant colleaguesinfluenced their prescribing behaviors (Table 4).This session was the least interactive of our six cases;

physicians did not raise change cues, share their practices,or discuss making changes. The facilitator posed few ques-tions to the group in this session, and few questions wereraised beyond trying to understand the data that was pre-sented. Participants pointed out that the AF report wasquite difficult to interpret; something that was reflectedacross cases 2A–D in the amount of time spent question-ing the facilitator to understand the data during theseAGFS (see Table 4). The facilitator expressed that thisgroup did not seem “interested” in the AGFS.Case 3 was a project with a group of specialists ad-

dressing broad questions about practice variation in


Table

4Represen

tativeexam

ples

ofdata

that

wereused

inbu

ildingthecase

stud

ies.In

thistable,theway

inwhich

raw

data

was

inde

xedto

thecompo

nentsof

theprog

ram

mod

elisshow

n.Whe

rerelevant,h

owthisdata

was

linkedto

thekeyelem

entsof

theCAFF

mod

elisalso

show

nin

orde

rto

demon

strate

theway

inwhich

inferences

were

draw

nbe

tweentheoriginalraw

data,the

prog

ram

mod

el,and

,ultimately,theframew

ork

CAFF

mod

elelem

ent(overarching

them

e)Com

pone

ntof

prog

ram

mod

elExam

plefinding

sfro

mcase

analysisextractedfro

mtranscrip

tsRepresen

tativequ

otations

Facilitation:

Facilitator

encouraged

andcreatedspacefor

thesechange

cues

tobe

raised

anddiscussed

bygrou

p.

Physicianen

gage

men

t/change

talk/chang

eplanning

[8]

Ahigh

degree

ofsocialinteractionin

case

1.Data

captured

from

originalAGFS

transcrip

tsReflectingandsharingof

practices

arou

ndusingintraven

ous

anesthesia:

“Ifindthat

thedistinctionbe

tweende

epandaw

akehas

becomeblurredwith

TIVA

.Ijustturn

downtheTIVA

andjust

extubate

them

andthey’re

almostaw

akebu

tthey’re

notas

deep

asforgasextubatio

n.Iu

suallycallitde

epbu

tit’sno

treallyade

epextubatio

nin

thetruestsense…

itkind

ofis

halfw

ayin

betw

een.”

Achange

cuefro

manothe

rparticipant:

“Our

bigg

estprob

lem

ispo

st-oppain”

Thischange

cueledto

adiscussion

betw

eentheparticipants

abou

tstrategies

forim

provingpain

managem

entwith

out

worsening

nausea

andvomiting

.

Facilitation:

Facilitator

encouraged

andcreatedspacefor

thisdiscussion

tooccur.

Physicianen

gage

men

t/change

talk/chang

eplanning

[8]

Case2A

Datacaptured

from

originalAGFS

transcrip

tsAparticipantexpressedtheird

esire

toimprovetheirp

rescrib

ing

habits:

“…Letus

say3calls

that

Iget

atnigh

tarerequ

ests,thispatient

wantsasleeping

pill.Im

ight

besuccessful

in1in

3in

convincing

thepatient

that

nothey

dono

treallyne

edthis.O

rconvincing

thenu

rse,no

they

dono

treallyne

edthis.M

aybe

I’llb

esuccessful

with

1on

3.Bu

tits’w

orth

trying

.”Thischange

cueledto

adiscussion

arou

ndthene

edforthe

grou

pto

developconsen

suson

de-prescrib

ingthecompo

unds

inqu

estio

n.

Recipien

tsandcontext

Case2D

:Theph

ysiciangrou

pin

thisho

spitalw

asyoun

gand

they

believedtheirgrou

pmake-up

andrelatio

nships

with

consultantsat

that

site

influen

cedtheir

prescribinghabits.

Datafro

moriginalAGFS

transcrip

ts

“Wearearelativelyyoun

gergrou

pcomparedto

some...And

Ithinkalotof

ustryto

utilize…

restraintin

antip

sychotic

prescriptio

ns.W

ehave

hadgo

odteam

sinvolved

with

alotof

our

patientsandwetryandavoidalotof

that.”

Datarepresen

tatio

nStyleof

repo

rtCase2D

:Participantsraised

concerns

abou

tthe

complexity

andcogn

itive

load

intheirAFrepo

rts.

Datafro

moriginalAGFS

transcrip

ts

“But

whe

nyoufirstlook

atthedo

cumen

t…itisvery

difficultto

unde

rstand

alotof

it,be

causethere’salotof

inform

ation.”

Exam

ples

ofdata

extractedfro

minterviewswith

CPLPStaff

Represen

tativequ

otations

Relatio

nshipbu

ildingandqu

estio

nchoice

Recipien

ts,inn

ovation,and

context

Whe

naskedabou

tho

wtheprojectoriginated

and

nature

oftheinno

vatio

n(case1):

CPLPfacilitator:“Manyof

thesedo

cshadbe

eninvolved

in[our

first]project,bu

tthereweresomeyoun

gerrecent

grads

who

weremoreinvolved

inchoo

sing

questio

ns—NAMEwas

seen

asastrong

leader,g

roup

hadbo

ught

into

hisvision

ofed

ucationin

gene

raland

theidea

ofauditandfeed

back.M

aybe

thegrou

pge

nerally

hasacultu

reof

mon

itorin

ganddata

collection—

…no

tabigsellto

thisgrou

p–they

werekeen

,thou

ghtful,interested,

andacadem

icin

theirthinking

….they

hadtheadvantageof

having

alreadybe

enthroug

htheprocess.

Lotsof

grou

ndworkdo

neby

ourteam

working

with

theclinic


Table

4Represen

tativeexam

ples

ofdata

that

wereused

inbu

ildingthecase

stud

ies.In

thistable,theway

inwhich

raw

data

was

inde

xedto

thecompo

nentsof

theprog

ram

mod

elisshow

n.Whe

rerelevant,h

owthisdata

was

linkedto

thekeyelem

entsof

theCAFF

mod

elisalso

show

nin

orde

rto

demon

strate

theway

inwhich

inferences

were

draw

nbe

tweentheoriginalraw

data,the

prog

ram

mod

el,and

,ultimately,theframew

ork(Con

tinued)

CAFF

mod

elelem

ent(overarching

them

e)Com

pone

ntof

prog

ram

mod

elExam

plefinding

sfro

mcase

analysisextractedfro

mtranscrip

tsRepresen

tativequ

otations

staff,daycarestaff,data

analyst,etc...lotsof

face

timefro

mPLP

onsite—seem

edto

behe

lpfulw

ithbu

yin.”

Relatio

nshipBu

ilding&QuestionCho

ice

(thisgrou

pwas

notdirectlyinvolved

inco-creationof

questio

n)

PhysicianEngage

men

t/Chang

eTalk/Chang

ePlanning

[8]

Whe

naskedabou

tgrou

pdynamic(case2D

):Facilitator:“Po

orparticipationfro

mthegrou

p.Thegrou

pcultu

reat

thisho

spitalseemed

different

from

theothe

rgrou

ps…very

silent

durin

gthesession…

notvery

manyshow

edup

..for

their

repo

rts”.Project

manager

1:“[T

heparticipantswere]

youn

g.Did

they

feelun

comfortablespeaking

up?”

Questionchoice

Inno

vatio

nWhe

naskedabou

tnature

oftheprojectcase

3:Projectmanager

2:“The

goalwas

hazy…very

broad—

things

that

werecommon

”.

Relatio

nshipbu

ilding

Recipien

tsandcontext

Describingtherecipien

ts(case3):

Facilitator:“Thisgrou

pissupe

ren

gage

d.Abu

nchof

XXXreside

ntswho

may

have

been

frien

dsbe

fore,or

becamefrien

dsafterw

ards—they

areallabo

utthesameage,

early

millen

ialsmostly

allo

nthesamepage

.The

grou

pwas

onthesamepage

.NAMEisacompe

tent,q

uiet

leader

…very

practical,d

ownto

earth,that’swhy

NAMEwas

askedto

dothis.

Thegrou

pistig

htbe

causethey

coverforon

eanothe

rand

cross-cover.Themeanageof

thisgrou

pwas

particularly

youn

g–senseof

buyin

todata,com

puters,d

igitaln

atives.H

admanysugg

estio

ns…cameup

with

atleasttenpo

tentialp

rojects.

They

werekeen

–lotsof

sharingof

oneanothe

r’sdata

inthe

sessions”.


relation to anesthesia for various surgical procedures. Inthis AGFS, there was a high degree of interactivityaround change cues which led to change planning, andalso comparing and sharing of practices. Participantsraised ideas about additional rounds and education, im-provements in charting, and working with other system“actors” to foster improvements in care. The participantswere young and very collegial with one another. Severalof them had reportedly trained together. The medicallead for the project co-created the questions with theCPLP medical director, helped with report design,pre-circulated relevant journal articles to the participantgroup, and shared their data with the group during thepresentation.

Comparative case analysisThe results of the comparative case analysis are summa-rized in Table 5. While the authors recognize that phys-ician behavior change was the goal of undertaking theAGF projects, the focus of this study was the factors in-fluencing whether physicians engaged with the data andchange planning during the AGFS. Thus, the “success”of the individual sessions was gauged based on the fol-lowing three criteria: (1) The physicians engaged in thegroup discussion about the data, (2) Change cues thatarose were followed by change talk, and (3) Further ac-tion was taken based on the AGFS to address identifiedgaps. There were numerous additional outputs thatarose following the AGFS studied, and these are cap-tured in Table 3 but not elaborated here as they are out-side the scope of this paper.In comparing across cases, the most change talk and

planning occurred during cases 1, 2A, and 3. Compara-tive case analysis highlighted several key elements foster-ing physician engagement with the data in these groups.These included the pre-existing relationship between theCPLP and the physician group (case 1), the active in-volvement of a physician leader from within each AGFgroup (cases 1, 2A, and 3), co-creation (between groupmembers and CPLP) of the clinical questions and AF re-port designs (cases 1 and 3), perceived control over thebehaviors being measured in the AF report (cases 1 and3), intrinsic group dynamic or cohesiveness (cases 1, 2A,2B, and 3), and the approach of the facilitator leadingthe sessions (co-facilitation in cases 1 and 3 andcoaching-oriented facilitation with prompts and ques-tions in cases 1, 2A, 2B, and 3).The physicians’ tendency towards sharing and compar-

ing practices during AGFS were greater in AGFS 1 and3, which were based on clinical questions about practicevariation, than they were in cases 2A–D. The datavisualization for the AF reports in cases 2A–D wasoverly complex. During AGFS 2A–D, a disproportionateamount of time was spent by the participants on

understanding the reports. Participants in AGFS 2Dcommented specifically on this finding. In addition, theextent of interactivity in AGFS 2A-D diminished witheach successive case. In reviewing the transcripts, and ininterviewing the facilitator and CPLP staff who observedthe sessions, it became apparent that the facilitation stylechanged across those four cases. In cases 2A and 2B, thefacilitator asked many questions of the participants,prompting them to consider their data. In contrast, incase 2D, the facilitator asked very few questions and theparticipants made few inquiries about the data. Indeedthe transcriptionist for the AGFS described the AGFSfor 2D as a “lecture rather than a focus group” aftertranscribing the session.

Developing an organizing frameworkIn reviewing the similarities and differences across thecases, and grouping similar items, four overarchingthemes emerged: relationship building, question choice,data visualization, and facilitation. In organizing these ina way that reflected both the processes used by theCPLP and the observed physician behaviors in AGFS, itwas possible to link the design/implementation factorswith the cycle of predictable physician behaviors [8] tocreate a practical framework for AGF project planningand design. The Calgary Audit and Feedback Framework(CAFF) is shown in Fig. 2.

DiscussionThe audit and feedback literature consists of many pub-lished interventions demonstrating that AF can be a use-ful means to change physician behavior [1]. However, ithas been pointed out that there are gaps in the designand implementation of AF and that these may accountfor some of the variability in the effectiveness of pub-lished interventions [1, 3–7].The CPLP developed a novel way to deliver AF that

aimed to address some of the design and implementa-tion elements emerging from the AF and educationalfeedback literature: AGFS [4–7, 9, 24–27]. Recognizingthat social learning was a key ingredient in pivotingAGFS towards change planning, we wished to explorewhy some AGFS were more socially interactive thanothers [8, 20].In this study, we present the results from a compara-

tive case analysis of six AGFS and based on thesefindings, we propose a practical, evidence-informedframework for the development, and implementation ofAGFS. We have termed the product of this research a“framework” based on Nilson’s definition:“Frameworks in implementation science often have

a descriptive purpose by pointing to factors believedor found to influence implementation outcomes.”([36], p 2).


Table

5Summaryof

results

ofcomparativecase

analysisforsixauditandfeed

back

sessions

with

different

physiciangrou

ps.The

prog

ram

mod

elelem

ents,w

hich

form

edthe

framew

orktableused

fordata

analysisandinde

xing

areon

theverticalaxisof

thistableandthecasesarelistedacross

theho

rizon

tal.Theresults

presen

tedbe

low

are

summarized

,for

purposes

ofthepu

blication,

from

thede

tailedno

tatio

nscaptured

intheoriginalframew

orktables.The

data

sourcesfro

mwhich

thedata

wereextractedare

describ

edin

Table1

Prog

ram

mod

elelem

ents

Case1

Case2a

Case2b

Case2c

Case2d

Case3

Inno

vatio

nReview

ofpractice

variatio

nandou

tcom

esof

anesthesiafor

proced

urex

Review

ofprescribingpractice

variatio

nformed

ications

inpo

pulatio

nxat

hospitalA

Review

ofprescribing

practicevariatio

nfor

med

ications

inpo

pulatio

nxat

hospital

B

Review

ofprescribing

practicevariatio

nfor

med

ications

inpo

pulatio

nxat

hospitalC

Review

ofprescribing

practicevariatio

nfor

med

ications

inpo

pulatio

nxat

hospitalD

Survey

ofpracticevariatio

nin

anesthesiafor5proced

ures

atHospitalA

.Asurvey/overview

ofpracticetype

ofprojectin

prep

arationforamore

subseq

uent

specificproject.

Intent

tointrod

ucethegrou

pto

A&F

Forthisproject,A&F

was

onecompo

nent

ofamultifaceted

interven

tion:Itwas

preced

edby

adidacticinter-

profession

al(doctors,nurses,ph

armacists)con

tinuing

med

icaled

ucationsession,andsubseq

uentlyinclud

eda

commun

icationcampaignandchange

sto

physicianorde

ren

trysets,w

hich

stem

med

from

thisseriesof

AGFS

(cases

2A–D

)

Styleof

auditrepo

rtHighlyrefined

with

heavygrou

pinpu

tTherepo

rtsacross

cases2A

–Dpresen

tedthesameaggreg

atedata,b

uthadindividu

alph

ysiciandata

forthe

participatingph

ysicians

ateach

site.Ineach

ofthesefour

AGFS,a

sign

ificant

amou

ntof

timewas

spen

twith

the

participantsasking

questio

nsto

tryto

unde

rstand

thedata

asitwas

presen

ted

Engage

dgrou

plead

contrib

uted

substantially

torepo

rtde

sign

One

ofthetw

oprojectleads

was

from

thissite

andhe

lped

torefinethequ

estio

nforthe

AFrepo

rts

One

ofthetw

oproject

leadsforthisworkcame

from

thissite

andhe

lped

toco-createandrefine

thequ

estio

nfortheAF

repo

rts

2DParticipantscommen

ted

durin

gtheAGFS

that

the

repo

rtwas

difficultto

interpretdu

ringtheAGFS

Goldstandard

for

prop

osed

best

practice

Non

eNo,bu

tseveralexistinggu

idelines

andrecommen

datio

nsNo,bu

texistin

geviden

ceand

guidelines.

Recipien

tsSpecialistsat

hospital1.

18ph

ysicianrepo

rts,13

physicians

attend

edfeed

back

sessions

Gen

eralistph

ysiciangrou

p,nu

rsemanagers,QIleadat

hospitalsA,B,C

,and

Drespectively(64ph

ysicianrepo

rts,28

physicians

attend

edfeed

back

sessions)

Specialistgrou

p,ho

spitalD

17ph

ysicianrepo

rts,9

physicians

attend

edfeed

back

session

Participantsin

thisgrou

piden

tifiedthat

theirpatient

popu

latio

ninclud

edho

spice

patientsandfeltthat

this

influen

cedprescribing

behaviors

Participantsin

this

grou

pcaredforvery

high

volumeservices

with

high

degreesof

acuity

andcomplexity

Participantsat

thissite

iden

tifiedthat

they

care

foran

olde

rpatient

popu

latio

n.Thissite

had

access

toage

riatric

psychiatry

servicewhich

may

have

influen

ced

prescribingpatterns.The

ywerepo

sitivelyoriented

,cohe

sive,non

-judg

men

tal

Participantsat

thissite

self-

iden

tifiedas

being

“you

nger”andfeltthat

their

prescribingbe

haviorswere

influen

cedhe

avily

bytheir

training

andby

thelocal

consultantswith

who

mthey

collabo

rated

Con

text

Project

origin

Asecond

projectwith

CPLPforthisgrou

pAqu

estio

nraised

bycity-w

ideInno

vatio

nCom

mittee

who

invitedCPLPto

developtheAFrepo

rtsandde

liver

AGFS

across

the4sites

CPLPinvitedby

thisgrou

pafterthey

learne

dabou

tcase

1

Group

dynamicand

leadership

involvem

ent

fortheA&F

Expe

rienced

,respe

cted

senior

physicianwith

strong

commitm

entto

profession

alde

velopm

ent.

One

projectlead

was

amem

berof

thisgrou

pbu

twas

notableto

attend

thissession.

Thesite

lead

was

presen

tbu

tdidno

thave

facilitator

rolein

Site

lead

presen

t.Lotsof

questio

ning

ofdata

and

discussion

abou

tho

wthedata

was

analyzed

.Site

lead

expressed

One

ofthetw

oproject

leadswas

from

thissite.

Somego

oddiscussion

insession,bu

tCPLPstaff

notedthat

participants

Youn

gph

ysiciangrou

pin

ane

who

spitalw

orking

in2weekshiftswith

lotsof

hand

over.Poo

rattend

ance,

few

questio

nsat

session.

Very

motivated

,dynam

icsite

projectlead

with

anadmin

positio

nfored

ucationwith

inthegrou

p.Ayoun

gph

ysician

coho

rt,d

igitally

literate,op

en


Table

5Summaryof

results

ofcomparativecase


back

sessions

with

different

physiciangrou

ps.The

prog

ram

mod

elelem

ents,w

hich

form

edthe

framew

orktableused

fordata

analysisandinde

xing

areon

theverticalaxisof


theho

rizon

tal.Theresults

presen

tedbe

low

are

summarized

,for

purposes

ofthepu

blication,

from

thede

tailedno

tatio

nscaptured

intheoriginalframew

orktables.The

data

sourcesfro

mwhich

thedata

wereextractedare

describ

edin

Table1(Con

tinued)

Prog

ram

mod

elelem

ents

Case1

Case2a

Case2b

Case2c

Case2d

Case3

sessions

Senior

physicianand3

colleaguesinvolved

inallaspectsof

project.

Coh

esive,collegial

grou

p.Senior

physicianand1

colleague

ledsession

andshared

owndata.

session.Non

ethe

less,C

PLP

staffiden

tifiedthat

thesite

lead

was

“astrong

lead”and

contrib

uted

tothesession

meaning

fully.Lotsof

questio

ning

anddiscussion

abou

twhy

thissite

might

bedifferent

vsothe

rs,skepticism

regardingdata,req

uestsfor

data

that

wereno

tinclud

edin

originalproject.

CPLPmed

icaldirector-led

session

concernre

potential

defensiven

essof

grou

ppriorto

AGFS.G

ood

attend

ance.

CPLPmed

icaldirector-

ledsession

begansharingand

comparin

gdata

asthe

CPLPteam

lefttheroom

.Goo

dattend

ance.Project

lead

shared

owndata.

CPLPmed

icaldirector-led

session

CPLPmed

icaldirector-led

session

toself-reflection,keen

todiscusschange

andun

derstand

finding

s.Group

cohesio

nand

collegiality

high

,inpartbecause

ofcallstructure.

CPLPmed

icaldirector-led

session.Ph

ysicianproject

helped

prom

ptgrou

pdiscussion

andleddiscussion

arou

ndeviden

ce

Pre-work,

relatio

nship-

building

Thiswas

asecond

projectforthisgrou

p.Lotsof

interactionwith

CPLPteam

inproject

developm

ent

Thiswas

afirstprojectforeach

ofthefour

hospitalg

roup

s.Therewereseveralm

eetin

gswith

thetw

oprojectleads

(From

cases2A

and2C

)and

ameetin

gwith

thecity-w

ideQIg

roup

(atsite

2B),bu

tno

twith

grou

pmem

bers

Firstcontactwas

byinvitatio

n:CPLPpresen

tedto

theen

tire

physiciangrou

pto

introd

uce

concep

tof

A&F

prog

ram.

Projectlead

met

frequ

ently

with

MDgrou

pandCPLP

team

Co-creatio

nof

data/

metrics

Aworking

grou

pwith

astrong

lead

andthree

departmen

tmem

bers

andinpu

tfro

mMD

grou

pde

velope

dmetricsandrepo

rt

Thiswas

onepartof

alarger

project.Individu

alph

ysicians

ateach

site

didno

tchoo

sewhatqu

estio

nwou

ldbe

addressed;

however,the

questio

nwas

brou

ghtto

theCPLPby

thetw

oprojectleads(From

sites2A

and2C

)and

was

aligne

dwith

aprovincialinitiativeto

de-prescrib

ein

thispo

pulatio

n.Theprojectleadshe

lped

theCPLPteam

tode

term

inewhich

data

pointsto

build

into

therepo

rts

Strong

leadership

from

site

lead

andcollabo

rative

developm

entof

project

betw

eensite

lead

andMD

grou

pmem

bers

Proxim

ityof

doctor

todata

and

patient

Direct

physicians

administerin

gorde

rsat

bedside

In-direct:A

dmittingph

ysicianwas

notalwaysattend

ing,

manyorde

rswerePRNandat

discretio

nof

nursingstaff.

MDsdidno

tfeelthey

hadgo

odcontrolo

vernu

rsediscretio

nno

rorde

ren

trysystem

Direct,asin

case

1,andMDs

haddirect

access

todatabase

design

ersfordata

capture/

metrics

Facilitationof

Session

Co-ledby

CPLPand

grou

pcham

pion

who

presen

tedow

ndata

togrou

p.Prom

ptingand

questio

nsfro

mCPLP

andfro

mco-facilitator

CPLPfacilitator,w

howas

aph

ysicianbu

tno

tagrou

pmem

ber,ledeach

session.Facilitationfollowed

structureof

therepo

rt,tableby

table.Forcase

2C,exemplar

data

from

thesite

lead’srepo

rtwas

presen

tedbu

tthesessionwas

stillbasedon

structureof

therepo

rt.Som

ede

gree

ofinpu

tfro

msite

lead

occurred

atsites2A

,B,b

utno

tat

site

Cor

D.The

facilitationstyleat

siteDwas

different

from

theprevious

sessions.Thissessionwas

largelyadidactic

presen

tatio

nfro

mthefacilitator

tothegrou

p.Thiswas

thelastof

four

sessions

inthisproject.Thefacilitator

asked

fewer

questio

nsthan

inothe

rgrou

ps.W

heninterviewed

,the

facilitator

describ

edthefeelingthat

thisgrou

pwas

‘not

asinterested

’intherepo

rtandthat

attend

ance

atthissessionwas

poor

Ledby

CPLPfacilitator

with

substantiveinpu

tfro

mproject

lead.Promptingqu

estio

nsfro

mbo

thCPLPfacilitator

and

projectlead

Engage

men

t/change

talk/actionplanning

Very

interactive.Lotsof

sharingof

person

alpractices

andplanning

arou

ndadditio

nal

change

activities

Mod

erateinteraction.Some

change

cues

wereraised

,leadingto

plansforchange

,andsomesharingof

practices

occurred

Mod

erate.Aparticipant

spon

tane

ouslyraised

need

forchange

and

thisledto

considerable

discussion

ofho

wthis

couldbe

achieved

Minim

al.The

rewas

little

change

discussion

durin

gthisAGFS,b

utparticipants

didshareandcompare

practices

tosomeextent

Minim

al.The

grou

pasked

questio

nsrelatedon

lyto

unde

rstand

ingthedata.N

ochange

cues

arosein

this

AGFS

Very

dynamicsessionwith

lots

ofqu

estio

nson

eviden

cedirected

totheprojectlead

anddiscussion

betw

eengrou

pmem

bers


Table

5Summaryof

results

ofcomparativecase


back

sessions

with

different

physiciangrou

ps.The

prog

ram

mod

elelem

ents,w

hich

form

edthe

framew

orktableused

fordata

analysisandinde

xing

areon

theverticalaxisof


theho

rizon

tal.Theresults

presen

tedbe

low

are

summarized

,for

purposes

ofthepu

blication,

from

thede

tailedno

tatio

nscaptured

intheoriginalframew

orktables.The

data

sourcesfro

mwhich

thedata

wereextractedare

describ

edin

Table1(Con

tinued)

Prog

ram

mod

elelem

ents

Case1

Case2a

Case2b

Case2c

Case2d

Case3

includ

ingen

gage

men

tof

nursingstaffand

makingacollective

commitm

entto

de-

prescribing

Other

outputs

Ledto

7subseq

uent

projectswith

same

specialty

indifferent

settings

with

different

inno

vatio

nsandto

developm

entof

continuo

usrepo

rting

platform

foralld

octors

inthisspecialty

ThisAGFS

occurred

aftertheed

ucationcampaignthat

was

apartof

thelarger

initiative.Therewas

nochange

inprescribingfro

mbaselineto

thisAGFS.H

owever,1

year

afterthisAGFS,the

rewas

arelativeredu

ctionin

prescribingover

allfou

rsitesof

39%

durin

gstud

ype

riod.

Engage

men

tof

grou

pin

afollow-upprojectlookingat

anothe

rinno

vatio

narou

ndinsulin

prescribing,

also

inconjun

ctionwith

alarger,health

-autho

rityinitiated

educationalintervention.

Chang

esin

orde

r-sets,eng

agem

entof

multi-disciplinaryteam

s

Specificfeed

back

infollow-up

from

onegrou

pmem

berwho

change

dop

ioid

practice

subseq

uentlyafterlearning

abou

tpracticevariatio

nvs

othe

rmem

bers.Led

tofollow-

upprojectswith

samegrou

plookingat

othe

rclinical

questio

nsandultim

ately

additio

nalp

rojectsin

city-w

ide

departmen

t(see

case

1)


In the Calgary Audit and Feedback Framework(CAFF),the key findings from the case analyses were groupedinto four “factors” and aligned with the conceptualmodel of physician behaviors to demonstrate linkagesbetween the design and implementation elements andthe desired progression of the physicians towards changeplanning in AGFS (Fig. 2).

The Calgary Audit and Feedback FrameworkThe intent in developing the CAFF was to provide aconcise way to organize our findings and to delineatehow the various design elements could drive physicianstowards planning for change in a socially constructedlearning environment (Fig. 2). The first two factors ofthe framework, relationship building and questionchoice, help physicians in AGFS overcome potential bar-riers to acceptability of the feedback (for example:skepticism, mistrust, non-actionable feedback). Qualitydata representation is needed to facilitate understandingand interpretation of the data. Facilitation as a mediatingfactor is dependent on the trusting collaborative relation-ships between the feedback providers and recipients andhelps participants move from understanding the data tochange planning. Each of the identified mediating factorsare supported by existing literature and can be used to

help us understand how social interaction drives physi-cians towards change planning.

Building relationshipsBing-You et al. identified that feedback recipients in-corporate feedback into their learning inconsistently[37]. This may be mediated by failure to create an “edu-cational alliance” between the providers and recipientsof feedback, the credibility and constructiveness of thefeedback, and/or the nature of the relationship and trustbetween the providers and the feedback recipients [25–28]. To mitigate skepticism and enhance feedback ac-ceptability, the primacy of establishing respect and trustbetween the “provider” and “recipient” of in-person feed-back cannot be over-emphasized [25, 27–29].We found that when present, skepticism about the

data was a barrier to moving physicians in AGFS to-wards change talk. Groups who had prior experiencewith the CPLP program had a working relationship withour team and appeared to engage with their data andthe facilitator readily.Key elements of relationship building in our program

include empowering a group member to co-facilitate theAGFS, clarifying use of and the confidential nature ofthe AF reports, and the data limitations. These elements

Fig. 2 The Calgary Audit and Feedback Framework (CAFF) for the design and delivery of AGF. This model organizes the key findings from ourcase analysis that were identified as important drivers in moving physicians through the cycle of discrete behaviors that occur in AGFS towardsthe end goal of planning for change. Under each “mediating factor” are listed the distinguishing elements between the cases with more or lesssocial interaction that emerged from the comparative case analysis. The framework is linked to the conceptual model of physician behaviors inAGFs to show how the different mediating factors drive the behaviors towards change


necessitate advance planning and direct contact betweenthe individuals developing the AGF project and the pro-spective AGF recipients in order to forge an “educationalalliance” [27]. We suggest the successful creation of thisalliance is reflected in the subsequent engagement of theparticipating groups in additional projects with theCPLP. Co-creation also appeared to be a critical compo-nent in the development of the educational alliance:Cases where co-creation was emphasized in project de-sign had the most interactive AGFS.

Question choiceA key contributing factor to intervention effectiveness isensuring that the intervention selected is appropriate tothe desired behavioral change [10, 11, 19]. TheKnowledge-to-Action Cycle and the “Behavioural ChangeWheel” provide guidance for the choice of interventionfor fostering change depending on the nature of the de-sired behavior change [18, 19].It follows that not all questions about physician per-

formance are appropriately dealt with by using AF inter-ventions [7, 10, 11, 19, 23]. Choosing metrics over whichphysicians have little control or for which there is no goldstandard is unlikely to result in feedback acceptance [7].Brehaut and others emphasize the importance of

actionability of the feedback provided in AF [7, 10, 19,26]. Watling et al. found that perceived “constructive-ness” of feedback mediates its acceptance by learners tosome extent, and we suggest that the perceived “action-ability” described in the AF literature [7] and perceivedconstructiveness of educational feedback are similar con-structs [7, 26]. Likewise, the availability of best practiceevidence or gold-standards in addition to anonymizedpeer data with which to compare an individual’s per-formance may be an important component of the per-ceived “constructiveness” of the feedback [7, 24, 26].In AGFS (cases 1 and 3) in which the physicians had dir-

ect control over the outcome in question, participants fo-cused readily on the evidence, reflection, and changeplanning. In contrast, in cases 2A–D, physicians expressedreservations about their ability to make change because ofother “actors” in the system who could influence the out-come (e.g., allied health). We observed that this perceived“actionability” seemed to be mediated through the prox-imity between the physicians’ behavior or decision and theoutcome being measured.

Representation of the dataHow the data is presented in an AF report affects howeasily the participants can understand the report andmove on to making meaning from the findings and look-ing for change opportunities. Our analysis revealed thatthe data reports for cases 1 and 3 were simpler in theirdesign than the others (Figs. 3 and 4); this may be

because the project leads were more intimately involvedin co-creation of the questions the reports. This high-lights the importance of managing cognitive load, build-ing “end-user” testing of the AF reports into the designof an AGF project, and the value of co-creation of thelearning [7, 38].

Facilitation of the AGF sessionThe approach to facilitating the AGFS is pivotal to en-sure that participants in the session move from inter-preting the data to planning for change. The iPARIHSframework identifies the facilitator as playing the centralrole of fostering interaction between the participant,their data, and their context [17]. The iPARIHS authorsemphasize that experienced facilitators can grasp the nu-ances of system and organizational factors (context) thatmay influence implementation of a change intervention[17]. The need for an understanding of context lendsfurther support for the role of co-facilitating feedbackwith a group member. A non-group member facilitator,while expert at interpreting the data risked missing rele-vant change cues [8]. Physician engagement was highestwhen a respected group member co-facilitated AGFS,and we speculate that this resulted in positive credibilityjudgements by the other participants [10, 11, 19, 23, 26].We suggest that training a member of the recipientgroup to co-facilitate or lead AGFS will enhance accept-ability of the feedback and that with their privilegedknowledge (as a group member) of the context and cul-ture of the group, they will be better able to capturechange cues and incorporate elements such as barriersand enablers of change to support action planning [17].Sargeant et al. highlight that the facilitator’s role in a

feedback session should be coaching-oriented [25]. Thecoach-facilitator helps participants to navigate throughtheir reactions to data, to understand their data, andthen plan for change [25]. The “coaching-oriented ap-proach,” with prompts and questioning is essential. Inour study, when prompts and questioning were not usedby the facilitator (perhaps because of facilitator fatigueover the course of four AGFS), there was very limitedsocial interaction in the AGFS [25].

Limitations and future researchThere are several limitations of this study. Our findingsof physician behaviors in AGFS represent the collectedobservations over the course of six AGFS, five hospitals,and three specialties, but included only consenting phy-sicians. The authors acknowledge the risk of selectionbias in our participants. Another potential limitation isthe perspectives of some of the research team members,who were intimately involved with the leadership ofCPLP when this work was carried out. The authors


attempted to balance important tacit knowledge that in-formed the program development with the risk of biasfrom pre-conceived ideas by acknowledging their per-spectives, bringing on experienced research team mem-bers who were not intimately involved with the CPLP atthe time of the analyses and triangulating across data

sources to corroborate our observations. Finally, it maybe argued that cases 2A–D should be treated as one casebecause they all addressed the same clinical question.However, the authors were interested to explore context-ual and cultural factors that influenced social interactionin AGFS, and because cases 2A-D occurred in different

Fig. 4 This graph is an aggregate exemplar of how the data for cases 2A–D were initially presented. The goal was to show physicians whetherthey were discontinuing sedatives and anti-psychotics in patients who were admitted with a pre-existing prescription, whether the patients werestarted on these medications in hospital, and whether they remained on them after discharge. A desirable outcome for a patient on sedatives oranti-psychotics either before or during their hospital stay was considered to be discharge from hospital without a prescription for thesemedications. Physicians found these graphs challenging to interpret, resulting in disproportionate AGFS time being spent on clarifyingand questioning

Fig. 3 Representative example of a single data table from a generic report from the case 1 project. Physicians appeared to be readily able tointerpret these graphs during the AGFS, allowing more time for reflection and planning for change


settings with different participants, it was felt that thesecase should be treated separately [17, 23].The framework we present is a synthesis of our find-

ings from analysis of AF projects designed and deliveredin a novel way to address recommendations for enhan-cing AF in the literature [7]. While it appears that designand implementation elements used by the CPLP to de-liver AGFS promote social interaction, prospectiveevaluation and refinement of the CAFF will be import-ant next steps.

ConclusionBased on the findings of our study, we present a prac-tical, evidence-informed approach for the design anddelivery of AGFS in a way that links design and imple-mentation elements (relationship building, choice ofquestion, quality of data representation, and facilitationstyle) to the anticipated behaviors of physicians partici-pating in AGFS in order to promote social learning andbehavior change.

AbbreviationsAF: Audit and feedback; AGF: Audit and group feedback; AGFS: Audit andgroup feedback sessions; CAFF: Calgary Audit and Feedback Framework;CPLP: Calgary Physician Learning Program

AcknowledgementsThe authors would like to acknowledge the tremendous work of Ms. IneldaGjata and Ms. Wenxin Chen in support of the six audit and feedback projectsthat were studied in the course of this research.

FundingThis research was not funded.

Availability of data and materialsThe qualitative datasets generated for and analyzed during the current studyare not publicly available due to the need for anonymity of participants inthe AGFS, which could be compromised by sharing of transcripts in theirentirety. The complete comparative case analysis document, which wassummarized for this publication, is available from the corresponding authoron reasonable request.

Authors’ contributionsLC led the study design, qualitative and comparative case analysis, andconceptual model of physician behaviors and wrote the draft of themanuscript. DD was a major contributor to the concept and design ofthe CAFF, qualitative and comparative case analysis, conceptual modelof physician behaviors, and editing of the manuscript. LR contributed tothe research methodology, design of the CAFF, program model for thecomparative case analysis, and editing of the manuscript. SKD contributed tothe design of the CAFF and editing of the manuscript. CS contributed to thecomparative case analysis, design of the CAFF, and editing of the manuscript.HA contributed to the research design, conceptual model of physicianbehaviors, comparative case analysis, the CAFF, and edited the manuscript. Allauthors read and approved the final manuscript.

Ethics approval and consent to participateEthics approval for this work was received for each case from the ConjointHealth Research Ethics Board: REB13-0075 (case 1); REB14-0484 (cases 2a, 2b,2c, 2d); and REB13-0459 (case 3).

Consent for publicationNot applicable

Competing interestsThe authors declare that they have no competing interests.

Publisher’s NoteSpringer Nature remains neutral with regard to jurisdictional claims in publishedmaps and institutional affiliations.

Author details1Department of Clinical Neurosciences, Cumming School of Medicine,University of Calgary, UCMC Area 3, 3350 Hospital Drive NW, Calgary, AB T2N4N1, Canada. 2Cumming School of Medicine, University of Calgary, 3330Hospital Dr NW, Calgary, AB T2N 4N1, Canada. 3Physician Learning Program,Cumming School of Medicine, University of Calgary, HSC G302, 3330 HospitalDr. NW, Calgary, AB T2N 4N1, Canada. 4Division of Endocrinology &Metabolism, Cumming School of Medicine, University of Calgary, 3330Hospital Dr NW, Calgary, AB T2N 4N1, Canada. 5Department of FamilyMedicine, Cumming School of Medicine, University of Calgary, 3330 HospitalDr NW, Calgary, AB T2N 4N1, Canada.

Received: 7 May 2018 Accepted: 18 October 2018

References1. Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, et

al. Audit and feedback: effects on professional practice and healthcareoutcomes. Cochrane Database Syst Rev. 2012;6:CD000259. https://doi.org/10.1002/14651858.CD000259.pub3.

2. Mansouri M, Lockyer J. Meta-analysis of continuing medical educationeffectiveness. JCEHP. 2007;27(1):6–15.

3. Ivers NM, Grimshaw JM, Jamtvedt G, Flottorp S, O’Brien MA, French SD, etal. Growing literature, stagnant science? Systematic review, meta-regressionand cumulative analysis of audit and feedback interventions in health care.J Gen Intern Med. 2014;29:1534–41.

4. Colquhoun HL, Brehaut JC, Sales A, Ivers N, Grimshaw J, Michie S, Carroll K,Chalifoux M, Eva KW. A systematic review of the use of theory in randomizedcontrolled trials of audit and feedback. Implement Sci. 2013;8:66.

5. Colquhoun HL, Carroll K, Eva KW, Grimshaw JM, Ivers N, Michie S, Sales A,Brehaut JC. Advancing the literature on designing audit and feedbackinterventions: identifying theory-informed hypotheses. Implement Sci. 2017;12:117.

6. Colquhoun H, Michie S, Sales A, Ivers N, Grimshaw JM, Carroll K, et al.Reporting and design elements of audit and feedback interventions: asecondary review. BMJ Qual Saf. 2017;26:54–60.

7. Brehaut JC, Colquhoun HL, Eva KW, Carroll K, Sales A, Michie S, et al.Practice feedback interventions: 15 suggestions for optimizing effectiveness.Ann Intern Med. 2016;164:435–41.

8. Cooke LJ, Duncan D, Rivera L, Dowling SK, Symonds C, Armson H. How dophysicians behave when they participate in audit and feedback activities ina group with their peers? Implement Sci. 2018;13:104.

9. Sales AE, Fraser K, Baylon MA, O’Rourke HM, Gao G, Bucknall T, et al.Understanding feedback report uptake: process evaluation findings from a13-month feedback intervention in long-term care settings. Implement Sci.2015;10:20.

10. French SD, Green SE, O’Connor DA, McKenzie JE, Francis JJ, et al.Developing theory-informed behaviour change interventions to implementevidence into practice: a systematic approach using the theoretical domainsframework. Implement Sci. 2012;7:38.

11. Cane J, O’Connor D, MIchie S. Validation of the theoretical domainsframework for use in behaviour change and implementation research.Implement Sci. 2012;7:37.

12. Ames C, Archer J. Achievement goals in the classroom: students’ learningstrategies and motivation processes. J Educ Psych. 1988;80:260–7.

13. Babenko O, Szafran O, Koppula S, Au L. Motivations for learning of familymedicine residents trained in competency-based education. Education forPrimary Care. 2018;29(2):86–93.

14. Anderman LH, Anderman EM. Oriented towards mastery: promotingpositive motivational goals for students. In: Gilman R, Huebner ES, FurlongMJ, editors. Handbook of positive psychology in schools. New York:Routledge/Taylor & Francis Group; 2009. p. 161–73.


https://doi.org/10.1002/14651858.CD000259.pub3

https://doi.org/10.1002/14651858.CD000259.pub3

15. Urban TC. Achievement of goal theory: past results, future directions. In:Maehr ML, Pintrich PR, editors. Advances in motivation and achievement,vol. 10. Greenwich: JAI Press; 1997. p. 99–141.

16. Darnon C, Dompnier B, Poortvliet PM. Achievement goals in educationalcontexts: a social psychology perspective. Soc Personal Psychol Compass.2012;6(10):760–71.

17. Harvey G, Kitson A. PARIHS revisited: from heuristic to integrated frameworkfor the successful implementation of knowledge into practice. ImplementSci. 2016;11:33.

18. Graham I, Logan J, Harrison M, Straus S, Tetroe J, Caswell W, et al. Lost inknowledge translation: time for a map? JCEHP. 2006;26(1):13–24.

19. Michie S, Atkins L, West R. The behaviour change wheel: a guide todesigning interventions. Great Britain: Silverback Publishing; 2014.

20. Bandura A. Social learning theory. New Jersey: Prentice Hall; 1977.21. Vygotsky L. Mind in society: the development of higher psychological

processes. Cambridge: Harvard University Press; 1978.22. Ajzen I. The theory of planned behaviour. Organ Behav and Hum Decis

Process. 1991;50(2):179–211.23. Abraham C, Michie S. A taxonomy of behaviour change techniques used in

interventions. Health Psychol. 2008;27:379–87.24. Watling C, Driessen E, Van Der Vleuten CPM, Vanstone M, Lingard L. Beyond

individualism: professional culture and its influence on feedback. Med Educ.2013;47(6):585–94.

25. Sargeant J, Lockyer J, Mann K, Holmboe E, Silver I, Armson H, Driessen E,MacLeod T, Yen W, Ross K, Power M. Facilitated reflective performancefeedback: developing an evidence- and theory-based model that buildsrelationship, explores reactions and content, and coaches for performancechange (R2C2). Acad Med. 2015;90:1698–706.

26. Watling C, Driessen E, Van Der Vleuten CPM, Lingard L. Learning fromclinical work: the roles of learning cues and credibility judgements. MedEduc. 2012;46(2):192–200.

27. Telio S, Ajjawai R, Regehr G. The “Educational Alliance” as a framework forreconceptualising feedback in medical education. Acad Med. 2015;90(5):609–14.

28. Sargeant JM, Mann KV, van der Vleuten CP, Metsemakers JF. Reflection: alink between receiving and using assessment feedback. Adv Health Sci EducTheory Pract. 2009;14(3):399–410.

29. Eva KW, Armson H, Holmboe E, Lockyer J, Loney E, Mann K, et al. Factorsinfluencing responsiveness to feedback: on the interplay between fear,confidence, and reasoning processes. Adv Health Sci Educ Theory Pract.2012;17(1):15–26.

30. Yin RK. Case study research: design and methods. Los Angeles: Sage; 2014.31. Goodrick D. Comparative case studies: Methodological briefs – Impact

evaluation No. 9. 2014;36. Downloaded from: https://www.unicef-irc.org/publications/754-comparative-case-studies-methodological-briefs-impact-evaluation-no-9.html. Accessed 13 Aug 2018.

32. Srivastava A, Thomson SB. Framework analysis: a qualitative methodologyfor applied policy research. JOAAG. 2009;4(2):72–29.

33. Ritchie J, Spencer L. Qualitative analysis for applied policy research. In:Bryman A, Burgess RG, editors. Analyzing qualitative data. London:Routledge; 1994. p. 173–94.

34. Gale N, Heath G, Cameron E, Rashid S, Redwood S. Using the frameworkmethod for the analysis of qualitative data in multi-disciplinary healthresearch. 2013 13:117. BMC Med Res Methodol. 2013;13:117.

35. Ward D, Furber C, Tierney S, Swallow V. Using framework analysis in nursingresearch: a worked example. J Adv Nurs. 2013;69(11):2423–31.

36. Nilsen P. Making sense of implementation theories, models and frameworks.Implement Sci. 2016;10:53.

37. Bing-You RG, Trowbridge RL. Why medical educators may be failing atfeedback. JAMA. 2009;302(12):1330–1.

38. Noel G. Health design: mapping current situations, envisioning next steps.Design J. 2017;20(Sup1):S2304–14.


https://www.unicef-irc.org/publications/754-comparative-case-studies-methodological-briefs-impact-evaluation-no-9.html



the calgary audit and feedback framework: a practical

Documents