preventing the voltage drop: keeping practice-based

ABOUT PRACTICE-BASED RESEARCH NETWORKS

Preventing the Voltage Drop: KeepingPractice-based Research Network (PBRN) PracticesEngaged in StudiesBarbara P. Yawn, MD, MSc, Allen Dietrich, MD, MPH, Deborah Graham, MSPH,Susan Bertram, BSN, MSN, Marge Kurland, BSN, Suzanne Madison, MPH,Dawn Littlefield, CSC, Brian Manning, MPH, Craig Smail, MPH,and Wilson Pace, MD

Introduction: Practice-based research continues to evolve and has become a major methodology formany pragmatic studies. While early practice-based network projects were usually short term, currentstudies often introduce or compare practice innovations that require long-term evaluation. That changerequires that practice sites remain engaged in research work for up to 5 years, a time that can allow fora significant “voltage drop,” or decline in active participation.

Methods: Over the past 15 years we have developed and adapted several strategies to facilitate andencourage the continued active engagement of practices in practice-based research network studies ofup to 5 years’ duration. The concepts, details, evaluation, and results (when available) of the strategiesare described.

Results: Eight strategies that enhance practice sites’ attention to enrollment, data collection and con-tinued use of the implemented practice change are described.

Conclusion: The loss of momentum, or “voltage drop,” that happens in longer-term practice-basedresearch network studies can be addressed using multiple strategies. (J Am Board Fam Med 2014;27:123–135.)

Keywords: Methods, Practice-based Research Network, Program Effectiveness, Randomized Controlled ClinicalTrials, Training

Practice-based research network (PBRN) researchhas become a valuable method of identifying andanswering questions relevant to primary care pa-tients and practices. As PBRN research has ma-tured, studies have evolved from simple, short-termobservational studies to longer-duration interven-tional studies that involve all members of the prac-

tice and enrolled patients.1 PBRN methodologyliterature also has progressed from reports of howto begin PBRNs2–6 to publications relevant to con-ducting longer-term and more complex studies inestablished PBRNs.7–12

Practices usually begin a study with significantenthusiasm and willingness to learn and fulfill thestudy requirements.13,14 As time progresses, thereality of everyday life in a busy practice may usurpthe time and effort needed for continued studyparticipation. We have observed this phenomenonin our own studies, and we have labeled this par-ticipation decline as “voltage drop” (A. Dietrich,MD, personal communication).

Regular study visits to local or regional PBRNpractices by practice enhancement assistants, prac-tice enhancement and research coordinators, or re-gional study team members may reduce the loss ofstudy momentum.15,16 Such activities seem to im-

This article was externally peer reviewed.Submitted 14 January 2013; revised 19 July 2013; accepted

12 August 2013.From the Department of Research, Olmsted Medical

Center, Rochester, MN (BPW, SB, MK, SM, DL); theNational Research Network, American Academy of FamilyPhysicians, Leawood, KS (DG, BM, CS, WP); and theDartmouth Medical Center, Dartmouth, NH (AD).

Funding: none.Conflict of interest: none declared.Corresponding author: Barbara P. Yawn, MD, MSc, De-

partment of Research, Olmsted Medical Center, 210 NinthSt. SE, Rochester, MN 55904 (E-mail: [email protected]).

doi: 10.3122/jabfm.2014.01.130026 Keeping PBRN Practices Engaged in Studies 123

prove practice outcomes17–19 but are not feasiblefor large, geographically dispersed studies thatwork to achieve a nationally representative sampleof practices.

Over the past 10 years of working with practicesto complete pragmatic trials3 we have identifiedseveral methods to facilitate initial engagement bythe practices and to maintain the initial high levelof enthusiasm and willingness to participate overthe long study period. The purpose of this report isto describe these strategies and their impact withthe purpose of supporting other researchers in im-plementing and enhancing their work in PBRNs.Other PBRNs have been engaged in similar trans-lational and practice change implementation stud-ies, so the strategies presented here may not beunique to our work but to date have not beenpublished in the PBRN literature.

MethodsThe strategies described in this report come frominformation collected over the past 10 years in 4large, federally funded studies, each lasting up to 5years (Table 1). We believe that these 4 long-termstudies are most representative of the difficulties wehave encountered in sustaining the practices’ inter-est and commitment over several years in projectsthat introduced a significant practice change. De-spite the variety of topics and varying study designs,many of the same implementation strategies andsupport techniques proved to be useful across stud-ies. Each of the studies introduced a practicechange: using spirometry to evaluate patients withasthma and chronic obstructive pulmonary disor-der20; introducing screening, diagnosis, and man-agement support for postpartum depression21–24;using new asthma tools (asthma APGAR tools) inmanaging children and adults with asthma25; andassessing the effect of 2 classes of long-acting bron-chodilators on exacerbation rates in black adultswith asthma.

We included 96 practices in the four large prag-matic studies mentioned above. Of the total, 87were family medicine practices, 5 were pulmonol-ogy practices, and 4 were pediatric practices. Foreach study except the spirometry study, we havelost from 1 to 4 practices because of a variety ofproblems, including dissolution of practices, physi-cian illness with prolonged disability, and inabilityto recruit patients (Table 1). The practices included Ta

ble

1.Th

eFo

urPB

RNSt

udie

sIn

clud

edin

this

Stud

y

Stud

yN

ame

Pra

ctic

es(n

)St

udy

Des

ign

Stud

yD

urat

ion

(mon

ths)

Cen

tral

Site

sSt

atus

Fund

ing

Pra

ctic

eC

hang

eD

isco

ntin

ued

Pra

ctic

es(n

)

Intr

oduc

tion

ofSp

irom

etry

into

Pri

mar

yC

are

18B

efor

e/af

ter

obse

rvat

iona

l18

OM

Can

dA

AFP

NR

NC

are

Net

Com

plet

edin

2006

20N

HL

BI

Usi

ngsp

irom

etry

toev

alua

tepa

tient

sw

ithas

thm

aan

dC

OP

D1

(una

ble

toen

roll

patie

nts)

TR

IPP

D(T

rans

latin

gR

esea

rch

into

Pri

mar

yC

are

Pra

ctic

efo

rP

ostp

artu

mD

epre

ssio

n)

28R

ando

miz

edco

ntro

lled

tria

l60

OM

Can

dA

AFP

NR

NC

ompl

eted

in20

1021

,22

AH

RQ

Intr

oduc

ing

scre

enin

g,di

agno

sis,

and

man

agem

ent

supp

ort

for

post

part

umde

pres

sion

4(2

prac

tices

diss

olve

d,1

prac

tice

had

long

-ter

mill

ness

oflo

cal

prim

ary

inve

stig

ator

,1pr

actic

eco

uld

not

mee

ten

rollm

ent

goal

s)T

heA

sthm

aT

ools

Stud

y28

Ran

dom

ized

cont

rolle

dtr

ial

60O

MC

and

AA

FPN

RN

Inpr

ogre

ssA

HR

QU

sing

new

asth

ma

tool

sin

man

agin

gch

ildre

nan

dad

ults

with

asth

ma

1(P

ract

ice

dest

roye

dby

torn

ado,

unab

leto

cont

inue

whi

letr

ying

tore

build

)B

EL

T(B

lack

Ast

hmat

ic’s

Exa

cerb

atio

nsw

ithL

AB

Avs

Tio

trop

ium

)

30R

ando

miz

edco

ntro

lled

tria

l36

OM

C,A

AFP

NR

N,

and

Har

vard

Med

ical

Scho

ol

Inpr

ogre

ss23

AH

RQ

Ass

essi

ngth

eim

pact

onex

acer

batio

nra

tes

of2

clas

ses

oflo

ng-a

ctin

gbr

onch

odila

tors

inbl

ack

adul

tsw

ithas

thm

a

3(1

prac

tice

unab

leto

dow

ork,

2si

tes

unab

leto

mee

ten

rollm

ent

goal

s)

AA

FPN

RN

,Am

eric

anA

cade

my

ofFa

mily

Phy

sici

ans

Nat

iona

lRes

earc

hN

etw

ork;

AH

RQ

,Age

ncy

for

Hea

lthca

reR

esea

rch

and

Qua

lity;

NH

LB

I,N

atio

nalH

eart

,Lun

gan

dB

lood

Inst

itute

;O

MC

,Olm

sted

Med

ical

Cen

ter.

124 JABFM January–February 2014 Vol. 27 No. 1 http://www.jabfm.org

576 physicians and 52 other clinicians (nurse prac-titioners and physician assistants), and each practiceincluded one or more other practice study leaders,usually a practice nurse, social worker, or medicalassistant.

While no specific evaluation activities were de-signed to rigorously test the value of any of thestrategies, formal and informal evaluation strategieswere used across all studies. Following centraltraining, each attendee was asked to use a Likertscale to answer questions about the perceived valueof the training, timing of the training, specific con-tent related to study procedures, and informationon the practice change. At the close of each of thestudies, qualitative data were collected using semi-structured interviews with the practice leaders and,in some cases, one or more other members of thepractice teams. The interview included questionsabout the support provided by the central site;sessions to help solve implementation issues in thelocal practice; time constraints; which, if any, partsof the study intervention were now a “routine” partof the practice; and which elements would be con-tinued and which elements were not feasible. Theinterviews also included open-ended questions al-lowing the practice leaders to make any other com-ments or suggestions they desired. Data from theinterviews were analyzed by members of the re-search group who were familiar with qualitativeresearch. Informal data sources included anecdotesand comments during case reviews, and calls by theliaison and principal investigator with the site lead-ers during the course of the studies.

ResultsWe report 8 general strategies (Table 2) that ourteam identified as useful for maintaining engage-ment in PBRN studies of up to 5 years’ duration.Each strategy is designed to help prevent or over-come practice study fatigue and disengagement.Table 2 expands on the characteristics of each ofthe strategies beyond the comments presentedhere.

1. Centralized TrainingGarnering sufficient time from busy practice stafffor education about study details is difficult. Tryingto train an entire practice about all the details ofany study is unrealistic. To overcome these prob-lems, we identified 2 study champions or site lead-

ers at each practice: one a physician and the other amember of the nursing staff. These 2 people be-came the study “leads” and attended the centralizedtraining.

Centralized training was usually done over aweekend in conjunction with another primary caremeeting or at a “vacation” site to enhance interestin attendance. The sessions were designed to intro-duce study background and need, study proceduressuch as enrollment and informed consent, and thepractice change or intervention; the format allowedinteraction among site leaders and between siteleaders and the central staff of the study. Fundingfor the central training was included in the budgetfor each study.

Training sessions were led by different membersof the central team to allow the site champions tomeet everyone on the central team. While the first3 to 4 hours were primarily didactic, the rest of thesessions were interactive, practicing informed con-sent or use of the intervention tools.

When appropriate, randomization assignmentswere announced at the training session so thatsupport could be provided to the “usual care” orcontrol group, explaining their important role andreminding them that they will move to the inter-vention arm within 18 months. The control prac-tices often realize quickly that they have a smallerinitial study burden and will get the benefits of anysuccessful interventions after the study interventionis complete.

At the close of the training session, each site wasprovided with a condensed slide set prepared forthe site leaders to use to educate the rest of theirpractice members. Asking the site leaders to leadthe presentation ensured that they understood theproject sufficiently well to present the project totheir colleagues and supported their identificationas the site study leaders and champions.

All materials presented during the central train-ing sessions were provided to each site in a binderthat included all slides and handouts from the cen-tral training. The materials supported study re-freshers between practices and central study teammembers on an as-needed basis.

Immediately after its completion, the centralizedtraining received reviews of “very good” to “excel-lent” for meeting expectations, understanding ofthe intervention, and clear, readable slides. Whenpresenting to their local practices, the practiceleaders repeatedly commented that they were able


Table 2. Summary of Elements of the Eight Methods

Method Main Elements Subcategories

Centralizedtraining

1. Select an attractive city a. Link to another meetingb. Choose a “vacation” city

2. Brief and focused presentations3. All central team members make presentations4. Introduce all site champion teams and

announce randomizationa. Celebrate with both groupsb. Extra support for usual care initial disappointment

5. Introduce site champions to the central siteliaison

6. Introduce the topic of the protocol and reasonsfor the study

7. Provide detailed information in the slides andhandbooks

a. Continually refer to the handbook when championsreturn home

b. Use the handbook to reinvigorate, remind, andrefresh

8. Introduce the intervention to appropriate sites a. Use practical examples based on patient careb. Let the group brainstorm anticipated problems and

solutionsc. Introduce the concept of implementation flexibility

9. Provide time for each set of site champions todiscuss their local implementation strategieswhile central team members are present toanswer questions.

10. Take and share pictures of site champions andcentral teams

a. It may be harder to ignore calls from someonewhose picture is right over the phone

b. Helps central team to remember more detailsabout each champion

11. Provide time to complete human subjectstraining with central team mentors

a. Reinforce that support is readily availableb. Helps when the champions may have to repeat

training in 2 to 3 years12. Provide concise slide set for site champions to

use to train practices when they go home13. Use humor and interactive, supportive sessions

to model behavior for interactions for the restof the study period

Liaisons 1. Assign to specific practices for the entire study2. Serve as primary contact for practices to the

central team3. Include in all study meetings4. Become an early warning system for practice

problems and “voltage drop”FAQs 1. Develop and use a consistent format with

study logo at top2. Send to sites on a consistent day of the week3. Share by fax or E-mail as sites request4. Use for continuing education, maintaining

same style as central training5. Reinforce complex or apparently confusing

steps in implementation6. Recognize accomplishments such as enrollment

numbers7. Share local sites’ stories a. Successes

b. New strategies or tools developed by sitesc. Occurrence and burden of natural disastersi. Floods, earthquakes, fires, hurricanes, tornadoes

ii. Share interpractice support given8. Use humor and cartoons to illustrate issues

and celebrations

Continued


Table 2. Continued

Method Main Elements Subcategories

9. Develop and use mini crossword puzzles orother study-related “contests”

10. Share case reviews and questions11. Celebrate holidays

Incentives 1. Foster recognition and acknowledgement a. Press releases for local mediab. Framed certificates of participation for display in

the local office2. Set achievable goals and recognize them when

they are met3. Provide small but fun rewards, eg, boxes of

microwave popcorn4. Use variety to continue encouraging

participation5. Increase the frequency at lulls in enrollment or

other dips in activity6. Provide larger rewards for major achievements

like meeting total enrollment goalsa. Pizza partyb. Ice cream social

7. Provide personalized thank you from the PIfor special recognition

Case reviews 1. Assume sites have important insight to offerand expect to learn from sites

2. Highlight positive aspects of care3. Ask what could be done to make care even

better4. Relate the care questions to the intervention5. Allow site attendees to teach each other6. Present alternatives from other sites or the PI’s

experience or expertise7. Guide problem solving8. End by setting up the next review and perhaps

follow up on the cases presentedAppreciation 1. Respect is mainstay

2. Acknowledge all members of the practice3. Praise the work of the site champions during

case reviews and in FAQs4. Schedule interactions at the convenience of the

sites5. Consider using emails for some interactions—

especially PI to site physician champions6. Give site champions study shirts or mugs7. Give modest holiday gifts for all practice

membersa. Candy or fruit for winter holidaysb. Cookies or gift card for an ice cream party in

summerIRB support 1. Offer to do initial submission

2. Provide support for all IRB interactions a. Serious adverse events reportingb. Annual reports

3. Copy and mail new consent forms requiredannually

4. Offer to provide telephone support for all site-required IRB interactions

5. As requested, teach sites to do its own IRBsubmissions with oversight and support

FAQ, frequently asked question; IRB, institutional review board; PI, primary investigator.


to do this only because they had attended the cen-tralized training. Site leaders also commented onthe importance and value of interacting with otherPBRN practices; for many sites this was the firstPBRN study in which they had participated. One ofthe most common requests was to have a secondmeeting of practice leaders at the mid-point orclose to the end of the study. We interpreted theserequests as an endorsement of this type of training.The centralized training engages the attendees andhelps them gain an in-depth understanding of theresearch question and methods for the study.

2. Practice LiaisonsEach practice was assigned a central site study liai-son, with each liaison working with 4 to 10 prac-tices, depending on the study complexity and otherduties of the liaison. The liaison maintained weeklyto biweekly contact with the practice leaders andwas the first line of communication between thecentral site and the practice.

The relationship between the liaisons and thepractice leaders was initiated at the centralizedtraining and was instrumental to continued practiceengagement. The liaison worked to develop a long-term personal relationship with the practice to al-low easy and honest communication between thepractices and the central study site. The liaison wasusually the first to notice signs of slowed enroll-ment or failure to use the intervention suggested bydelays in scheduling or completing regular liaisonsite contact. By brainstorming with the site leaders,the liaison, with the assistance of the study centralinvestigators, was able to find reasons for slowedparticipation, disengagement, or “voltage drop”and developed strategies to solve the identifiedproblems.

The importance and value of the site liaisons wasa recurrent theme in the exit interviews for each ofthe completed studies. In all interviews, the prac-tice leaders knew their liaisons by name, often com-mented that they kept her phone number immedi-ately available, and enjoyed having the long-termrelationship.

3. Frequently Asked Questions (FAQs)Frequently asked questions (FAQs) began as justthat—responses to a question asked by one site butwith answers shared with all sites. They evolvedinto a method to communicate weekly with siteswithout phone calls that could be disruptive to the

practice. The FAQs were sent to each participatingpractice on a specific day of the week, using aconsistent format with the study logo at the top ofeach FAQ. FAQ communications evolved fromfaxes to E-mails, which allowed the use of color.Site champions were asked to print the FAQs andpost them in an area where they could be seen by allstaff members. Examples of FAQs are shown inFigures 1 and 2.

FAQ topics varied from updates on study imple-mentation details to reminders of proper tech-niques for specific study procedures such as signinginformed consents or completing enrollment logs.Studies with both intervention and usual care armsoften required arm-specific FAQs at least part ofthe time, for example, when discussing interventionspecific topics. To maintain interest and preventthe FAQs from being seen as nagging, FAQs alsowere used to recognize and congratulate sites fortheir efforts or to celebrate national and faith-basedholidays, recognizing that sometimes the sitesneeded a break from our educational endeavors.

Acknowledgment that sites were aware of receiv-ing the FAQs is illustrated in a comment we re-ceived following a blizzard and power outage at thecentral site that prevented sending of the FAQ onthe assigned day: “We did not get our FAQ—areyou there and OK?” Evidence of reading theFAQ content came when we reported some ofthe study practices’ experiences with tornadoes,floods, earthquakes, and hurricanes. Followingreports of these occurrences in the weekly FAQs,other practices responded with offers of help andemergency supplies.

As we became more creative with our FAQgraphics and used more color, sites responded withrequests to use less color and simpler pictures toconserve color printer ink, demonstrating that thesites were printing out the FAQs. FAQs promptedcalls from practices asking the liaisons to clarifyimprovements requested on the FAQ, such as mak-ing sure the “witness” signs the required lines onthe informed consent. These anecdotal findingsconfirm that the FAQs generated interactions be-tween the central study staff and the practices.

During exit interviews for the 3 studies that havebeen completed, the practice leaders reported thatFAQs were read “almost every week and saved inthe notebook” provided by the central site for FAQcollection. The practice leaders further reportedthat both the “funny” FAQs and the serious ones


were useful and kept them thinking about the studyand their roles of “identifying and enrolling pa-tients” and “facilitating the intervention.” FAQsthat included case reviews generated discussionamong the physician leaders across practices. Ex-amples included an asthma case that generatedcomments from 50% of the 29 physician practiceleaders and a postpartum depression case that gen-erated responses from 30% of the physicians and70% of the nursing practice leaders.

4. IncentivesThere are many types of incentives that can be usedin PBRN studies. Using the FAQs to acknowledgesites that meet their individual monthly enrollmentgoals can be an incentive. Simple contests thatquery the site leaders about protocol details or evencurrent medical events and reward correct answerswith boxes of microwave popcorn or gift cards ofnominal value to local restaurants are easy to ac-complish. Most incentives are provided to the prac-

tice as a whole to reemphasize that it is the wholepractice and not just the champions who are par-ticipating in the study.

Response rates to the simple contests have beenhigh. In one contest all the practice leaders plusanother 5 practice members answered (110% re-sponse). When enrollment slows, incentives thatchallenge sites to reach their individual monthlyenrollment goals have resulted in increasing enroll-ments (13 of the 16 times such FAQs were sent).Incentives of recognition and praise— for example,an FAQ naming all the “winning” sites in an en-rollment challenge—were also well accepted. Theincentives were in addition to the site payments forstudy participation.

5. Case ReviewsCase reviews evolved from specific patient encoun-ters that prompted discussion in an FAQ as beingof general interest to all practice sites. Case reviewsallowed the central site to better understand how

Figure 1. Example of frequently asked questions.


Figure 2. Example of frequently asked questions.


the practice incorporated the intervention and toprovide the sites with an opportunity to ask anexpert about difficult cases. Case reviews were lim-ited to the intervention sites.

Each case review was completed by conferencephone call between the study’s principal investiga-tor (PI), the site’s liaison, and all the members atthe site who were available to attend. The case waspresented by the enrolled patient’s physician, resi-dent, or nurse. The presenters from the site chosetheir own presentation formats, although we devel-oped a written outline for anyone requesting it.Most cases required 3 to 5 minutes for the initialpresentation, with another 10 to 15 minutes fordiscussion. The questions that followed the presen-tation included how the study intervention did ordid not affect the patient’s care, what might be thenext steps in care, and other management issuespertinent to the specific patient, such as immuni-zation status or adherence issues. The participationof the central PI allowed the site practice membersto query about anything from how the investigatorwould have used the intervention tools to furtherdiagnostic testing or alternative therapeutic strate-gies.

Case reviews were not an instant success: manypractices were reluctant to participate in the firstcase reviews. However, almost every practice hadsomeone brave enough to have their patient carediscussed, and participation increased as the prac-tices learned that the case reviews were intended tobe learning sessions rather than judgmental ses-sions.

At the first case review it was often just the studyPI and the presenter interacting. But once the in-teraction was shown to be positive and truly inter-active, such as asking the presenter what they thinkis good and what could be improved in the care ofthe patient, others often joined the discussion. Thediscussions were intended to focus on the practiceor systems changes and tools introduced during thestudy. Asking for additional suggestions often re-vealed other aspects of patient care, such as clari-fying requirements for pneumococcal immuniza-tions in people with asthma or querying about theyearly influenza immunization for the pregnant andpostpartum women.

After the initial 1 or 2 case reviews the casereviews were well attended, especially in residencyprograms, and the practice staff asked questionsabout other complex cases. The PI, central site

staff, and practice staff usually learned at least onenew thing from these interactions.

One of the best indications that the case re-views were valued was the decreasing resistanceto scheduling future case reviews. It was commonat the end of the second or third case review tohave the site staff make recommendations for thedate, time, and patient for the next case. Thepractice’s queries regarding the care of nonen-rolled patients also suggests that the practicesvalued the time and collaborative attitude thatthese sessions exemplified.

6. AppreciationIn health care settings, expressions of appreciationmay be uncommon. Many members of the studypractices reported that they heard little praise butbore the brunt of many complaints. We appliedseveral methods for expressing our appreciation forthe site’s participation in the studies. At the centraltraining session, each site was provided with aframed certificate of appreciation for its participa-tion in the study. These plaques can be displayed inthe office waiting room and are often as simple as aframed page designed and printed at the centralsite. To facilitate acknowledgment of the work ofthe practice in their local community, we providedeach site with tailored press releases to explain thestudy and the local clinic’s role. These were de-signed for the practice leaders to forward to localnewspapers or other media outlets.

The study liaison for each site provided encour-agement, support, and praise during each interac-tion with the site leaders. All interactions with thesites ended with the liaison or investigator thankingthe practice leaders or other practice staff for theircontinued work.

Sites have repeatedly told us they displayed theirstudy participation certificates in their offices orfront lobbies. Certificates from previous studies arefrequently encountered during site visits. Five sitesfrom the postpartum depression study sent us cop-ies of newspaper articles that resulted from pressreleases. Accompanying the articles were the fol-lowing comments: “Our patients really liked seeingus in the newspaper” and “some of our subspecialtycolleagues were impressed with what we are do-ing.” The central site received many E-mails, cards,and notes from the practices, usually addressed tothe practice liaison, acknowledging and thankingthe central site for the small boxes of cookies or


candy and holiday greetings sent in appreciation ofthe practice’s work.

7. Institutional Review Board Submission SupportInstitutional review board (IRB) submission sup-port from the central study team was very impor-tant since many PBRN practices in our studies hadlimited experience with IRB submissions. The ideaof submitting the protocol application to the localIRB or sending a yearly report to the IRB of recordcan be overwhelming for a staff not accustomed topreparing such reports. Learning to complete IRBreports can take time away from other importantstudy activities such as enrolling patients or sup-porting an intervention. Having one person on thecentral team assigned to manage the IRB submis-sions freed the sites of this concern and assured thecentral team that there would be no lapse in theability to enroll patients.

A proxy outcome of the effectiveness of the IRBsupport provided to the practices was the numberof IRB refusals, queries, and delays that our studiesexperienced. In our submissions and yearly reportsto �55 IRBs for periods up to 5 years each (�200interactions), we have had 3 initial protocol approv-als delayed by 30 to 60 days, 2 yearly reports thatresulted in temporary holds on enrollment, andonly one IRB request for additional data on a pa-tient outcome. Considering the large number ofIRBs and the requirements for up to 5 yearsof approval from each of the IRBs, the number ofdelays and requests for additional informationhas been minimal, and we believe this was thedirect result of the support provided by the cen-tral study team during initial and yearly submis-sions of the reports.

8. Meeting the Academic Needs of Site PersonnelMany family physicians report the need for aca-demic stimulation in their regular practices. Forsome physicians, such as residents and physiciansseeking recertification, the requirements are morespecific.

For example, all family medicine residents arerequired to complete some type of scholarly activityduring their training. In 2 of our studies we havebeen able to provide the support and data for res-idents to complete research projects for this re-quirement that are complementary and parallel tothe main study.

All physicians are required to complete trainingfor maintenance of board certification throughtheir appropriate medical specialty board. We de-veloped a Maintenance of Certification program,sponsored by the American Academy of Pediatrics,based on one study’s intervention by expanding itsquality improvement focus. The cost of the pro-gram was minimal and provided the physician withMaintenance of Certification credit for 3 years.This program has been directly responsible foractivating 2 of the pediatricians who were not prac-tice leaders to incorporate the study interventioninto their daily practices and to add further modi-fications that they believe enhance the interventionin their practices.

One of our studies has been able to support thedoctoral dissertation of the practice’s nonphysicianleader. We have previously published other bene-fits of participation in PBRN studies as reported bythe practices.1

DiscussionWhile each of our individual strategies seems tohave had a favorable impact on re-energizing andsupporting continued activity in one or more of ourstudies, we have used the combined strategies in amulticomponent approach. Waning interest is apotential problem for many types of practice activ-ities and has been described most often in qualityimprovement activities.26–33 Over periods of 18 to60 months, as required by our studies, currentmedical practices are likely to be faced with manyexternal requirements for change, including theintroduction or update of electronic medical re-cords systems, payment restructuring, clinical sys-tems changes, and staff turnover. Designing andimplementing methods to maintain interest andenthusiasm for research requirements and practicechanges seemed to prevent some of the “voltagedrop” for the practices in the 4 studies discussedhere.

The importance of building relationships be-tween central team members and the practices byusing a practice facilitator, practice enhancementassistant, or practice enhancement and research co-ordinator has been reported by others.15,17,18,34,35

While we agree that the ability to send facilitatorsdirectly into the practices can be very helpful, inwidely dispersed practices, such as those in ournational studies, the contact must be continued


using distance communication formats such as thetelephone, E-mails, and faxes. We also have morerecently used video communication with free orlow-cost shared services such as Skype.

Our attention to the learning and supportiveenvironment was explicitly begun during our 2002spirometry study funded by the National Heart,Lung and Blood Institute, and it might be consid-ered similar to the concept of the formal “learningcollaborative” introduced by the Institute forHealthcare Improvement (IHI) in 2003. However,unlike the early design of IHI’s learning collabor-ative, we found it more feasible to have a singleface-to-face meeting and to work with each site ata distance after that.36,37 Our system perhaps fitsmore closely with the Collaborative NetworkedLearning first described in 1987 by Findley,38

which is designed to occur via electronic dialoguebetween learners and experts sharing a commonpurpose—in our case implementation of practicechanges and study-related activities.38 Our strate-gies of centralized training, regular contact, andinteractive discussion with the site leaders, as wellas the FAQs and case reviews, could be consideredelements of collaborative learning. We consciouslychose not to include some of the elements requiredby the IHI learning collaborative concept, such asthe large time and personnel commitments. TheIHI learning collaborative can require severalmonths to work out a solution for an identifiedproblem. Our approach with busy practices usedmuch shorter time frames to determine how best tointroduce practice tools and systems that had beenpreviously developed and tested. The focus was onflexibility in implementing the tools and systemchanges rather than developing those tools andsystems.39

Quality improvement initiatives also encountersimilar experiences with “voltage drop.” The re-sponse to declining participation in quality im-provement work initially was to declare that “whatgets measured gets done” and develop multiplemetrics to assess quality processes.40,41 Similargoals can be accomplished in research settings withaudit and feedback.42 While some of our studies,such as the Asthma Tools study, use specific prac-tice audit and feedback, we also consider the casereviews to be a form of assessing what is actuallybeing done in the practice followed by an interac-tive discussion between the practices and the studystaff.

Few other specific strategies can be found in themedical literature.26–33 The RE-AIM (Reach, Effi-cacy, Adoption, Implementation, and Maintenance)framework clearly denotes the importance of “main-tenance,” which might be equated with continuedparticipation in our research projects.43 Suggestionsfor supporting maintenance often include recom-mendations for continuing to support and em-power those doing the quality improvement pro-cesses but without methods of operationalizingthose recommendations. Our strategies incorpo-rated support and empowerment while focusingheavily on implementing and sustaining the ele-ments necessary to complete the studies.

The challenge of working with multiple IRBshas been discussed frequently.8 However, little hasbeen published about IRB continuing reviews andthe need to identify a central team member tooversee this work. With many IRBs required foreach study, the need for an organized system be-comes obvious. While some practices assisted withthe IRB submissions, all were happy to have centralsupport. Although this may be considered usualprocedure for any PBRN study, it removes some ofthe time demand from the site and allows greatertime for the core study activities.

Incentives and mini contests were used toheighten awareness of the need for ongoing patientenrollment into the studies and often as a simplefun break from everyday practice routines. Theresponse rate to the incentives and mini contestswas high and usually resulted in a note or picturewhen the incentive or prize was enjoyed by theentire practice.1,10,44

To support the need for health professionals toregularly complete continuing education require-ments,45 we worked to provide continuing medicaleducation credit for attendance at the in-personcentralized training sessions through the AmericanAcademy of Family Physicians. We most recentlyadded an opportunity for pediatricians to complete amodule required for maintenance of certification, andwe assisted family medicine residents in completingtheir required academic activities during their final 2years of training.46,47 Supporting continuing medi-cal education and academic activities has been fa-vorably received by residency directors, residents,and practice physicians involved in our PBRNstudies, and we believe that this has contributed tothe willingness of the physicians to sustain theirinvolvement with the studies.


ConclusionPBRN, translational, and implementation researchrequires unique methods to assure success, espe-cially as the studies become more complex and oflonger duration. Each practice is a unique researchpartner whose strengths should be highlighted andenhanced. The practices are often inexperiencedpartners who require special attention when iden-tifying and overcoming barriers to consistentlycompleting all the requirements of the study anddoing so over a sustained time period, especially themid-study lag that we have labeled a “voltage drop.”The strategies we have outlined here were successfulin sustaining a high level of practice involvement inour studies, which is essential for high-quality proj-ects producing valid results. We believe that PBRNresearch will continue to be extremely valuable intransforming medical practice, and the use of ourstrategies can contribute to the overall success ofthe PBRN concept.

References1. Yawn BP, Pace W, Dietrich A, et al. Practice benefit

from participating in a practice-based research net-work study of postpartum depression: a NationalResearch Network (NRN) report. J Am Board FamMed 2010;23:455–64.

2. Green LA, Hickner J. A short history of primary carepractice-based research networks: from concept toessential research laboratories. J Am Board Fam Med2006;19:1–10.

3. Woolf SH. The meaning of translational researchand why it matters. JAMA 2008;299:211–3.

4. Hayes H, Parchman ML, Howard R. A logic modelframework for evaluation and planning in a primarycare practice-based research network (PBRN). J AmBoard Fam Med 2011;24:576–82.

5. DeVoe JE, Gold R, Spofford M, et al. Developing anetwork of community health centers with a com-mon electronic health record: description of theSafety Net West Practice-based Research Network(SNW-PBRN). J Am Board Fam Med 2011;24:597–604.

6. DeVoe JE, Likumahuwa S, Eiff MP, et al. Lessonslearned and challenges ahead: report from theOCHIN SafetyNet West practice-based researchnetwork (PBRN). J Am Board Fam Med 2012;25:560–4.

7. Mold JW, Lipman PD, Durako SJ. Coordinatingcenters and multi-practice-based research network(PBRN) research. J Am Board Fam Med 2012;25:577–81.

8. Yawn BP, Graham DG, Bertram SL, et al. Practice-based research network studies and institutional re-

view boards: two new issues. J Am Board Fam Med2009;22:453–60.

9. Yawn BP, Bertram S, Wollan P. Introduction ofAsthma APGAR tools improve asthma managementin primary care practices. J Asthma Allergy 2008;1:1–10.

10. Pace WD, Fagnan LJ, West DR. The Agency forHealthcare Research and Quality (AHRQ) practice-based research network (PBRN) relationship: deliv-ering on an opportunity, challenges, and future di-rections. J Am Board Fam Med 2011;24:489–92.

11. Williams RL, Rhyne RL. No longer simply a prac-tice-based research network (PBRN) health im-provement networks. J Am Board Fam Med 2011;24:485–8.

12. Gilbert GH, Richman JS, Gordan VV, et al. Lessonslearned during the conduct of clinical studies in thedental PBRN. J Dent Educ 2011;75:453–65.

13. Graham DG, Spano MS, Stewart TV, Staton EW,Meers A, Pace WD. Strategies for planning andlaunching PBRN research studies: a project of theAcademy of Family Physicians National ResearchNetwork (AAFP NRN). J Am Board Fam Med 2007;20:220–8.

14. Fagnan LJ, Handley MA, Rollins N, Mold J. Voicesfrom left of the dial: reflections of practice-basedresearchers. J Am Board Fam Med 2010;23:442–51.

15. Nagykaldi Z, Mold JW, Aspy CB. Practice Facilita-tors: a review of the literature. Fam Med 2005;37:581–8.

16. Nagykaldi Z, Mold JW. The role of health informationtechnology in the translation of research into practice:an Oklahoma Physicians Resource/Research Network(OKPRN) study. J Am Board Fam Med 2007;20:188–95.

17. Hogg W, Baskerville N, Nykiforuk C, Mallen D.Improved preventive care in family practices withoutreach facilitation: understanding success and fail-ure. J Health Serv Res Policy 2002;7:195–201.

18. Frijling BD, Lobo CM, Hulscher ME, et al. Multi-faceted support to improve clinical decision makingin diabetes care: a randomized controlled trial ingeneral practice. Diabet Med 2002;19:836–42.

19. Dietrich AJ, O’Connor GT, Keller A, Carney PA,Levy D, Whaley FS. Cancer: improving early detec-tion and prevention. A community practice ran-domised trial. BMJ 1992;304:687–91.

20. Yawn BP, Enright PL, Lemanske RF, et al. Spirom-etry can be done in family physicians’ offices andalters clinical decisions in management of asthmaand COPD. Chest 2007;132:1162–8.

21. Yawn B, Dietrich A, Wollan P, et al. TRIPPD: apractice based network effectiveness study of post-partum depression screening and management. AnnFam Med 2012;10:320–9.

22. Yawn BP, Pace W, Wollan PC, et al. Concordanceof Edinburgh Postnatal Depression Scale (EPDS)and Patient Health Questionnaire (PHQ-9) to assess


increased risk of depression among postpartumwomen. J Am Board Fam Med 2009;22:483–91.

23. Yawn B, Madison S, Bertram S, et al. Automatedpatient and medication payment method for clinicaltrials. J Clin Trials 2012;4:1–9.

24. Yawn B, Dietrich A, Wollan P, et al. TRIPPD: Apractice-based network effectiveness study of post-partum depression screening and management. Annof Fam Med. 2012;10(4).

25. Yawn B, Bertram S, Kurland M, et al. Protocol forthe asthma tools study: a pragmatic practice-basedresearch network trial. Pragmat Observ Res 2013;4:1–12.

26. Hughes RG. Chapter 44. Tools and strategies forquality improvement and patient safety. In: Patientsafety and quality an evidence-based handbook fornurses. Rockville, MD: Agency for Healthcare Re-search and Quality; 2008. Available from: http://www.ahrq.gov/professionals/clinicians-providers/resources/nursing/resources/nurseshdbk/nurseshdbk.pdf. Accessed November 11, 2013.

27. Bryant D. Maintaining your quality improvementachievements. May 7, 2009. Available from: http://ezinearticles.com/?Maintaining-Your-Quality-Improvement-Achievements&id�2312493. Ac-cessed November 11, 2013.

28. Rabinow P, Vilela M. Section 1. Achieving andmaintaining quality performance. In: Nagy K,Berkowitz B, Loewenstein M, editors. Maintainingquality performance. Lawrence: Work Group forCommunity Health and Development, University ofKansas; 2013. Available from: http://ctb.ku.edu/en/tablecontents/sub_section_main_1387.aspx. AccessedNovember 11, 2013.

29. Institute of Medicine. Crossing the quality chasm: anew heath system for the 21st century. Washington,DC: National Academies Press; 2001: 164–80.

30. Wallin L, Bostrom AM, Wikblad K, et al. Sustainabil-ity in changing clinical practice promotes evidence-based nursing care. J Adv Nurs 2003;41:509–18.

31. Shojania K, McDonald K, Wachter R, et al. Closingthe quality gap: a critical analysis of quality improve-ment strategies: volume 1—series overview andmethodology. Technical review, no. 9. AHRQ pub-lication no. 04-0051-1. Rockville, MD: Agency forHealthcare Research and Quality; 2004.

32. Horbar JD, Plsek PE, Leahy K; NIC/Q 2000.NIC/Q 2000: establishing habits for improvement inneonatal intensive care units. Pediatrics 2003;111(4Pt 2):e397–410.

33. Kim GR, Chen AR, Arceci RJ, et al. Error reductionin pediatric chemotherapy: computerized order en-try and failure modes and effects analysis. Arch Pe-diatr Adolesc Med 2006;160:495–8.

34. Cook R. Primary care. Facilitators: looking forward.Health Visit 1994;67:434–5.

35. Nagykaldi Z, Mold JW, Robinson A, Neibauer, FordA. Practice facilitators and practice-based researchnetworks. J Am Board Fam Med 2006;19:506–10.

36. Collaboratives. Cambridge, MA: Institute for Health-care Improvement. Available from: http://www.ihi.org/offerings/MembershipsNetworks/collaboratives/Pages/default.aspx. Accessed November 11, 2013.

37. Wikipedia. Collaborative learning. Available from: http://en.wikipedia.org/w/index.php?title�Collaborative_learning&printable�yes. Accessed November 11, 2013.

38. Quik WH, Wright N. Information sharing to trans-formation: antecedents of collaborative networkedlearning. World Acad Sci Engin Technol 2012;68:1477–86.

39. Roland M, Torgerson DJ. Understanding controlledtrials: what are pragmatic trials? BMJ 1998;316:285.

40. Williamson RM. Available from: http://www.swspitcrew.com/articles/What%20Gets%20Measured%201106.pdf. Accessed December 2, 2013.

41. Six Sigma. Community. Available from: http://www.isixsigma.com/community. Accessed July 10,2013.

42. Ivers N, Jamtvedt G, Flottorp S, et al. Audit andfeedback: effects on professional practice and health-care outcomes. Cochrane Database Syst Rev 2012;(6):CD000259.

43. Glasgow RE, Vogt TM, Boles SM. Evaluating thepublic health impact of health promotion interven-tions: the RE-AIM framework. Am J Public Health1999;89:1322–7.

44. Gibson K, Szilagyi P, Swanger CM, et al. Physicianperspectives on incentives to participate in practice-based research: a greater Rochester practice-basedresearch network (GR-PBRN) study. J Am BoardFam Med 2010;23:452–4.

45. ABFM maintenance of certification for family med-icine. Lexington, KY: American Board of FamilyMedicine. Available from: https://www.theabfm.org/moc/index.aspx. Accessed July 10, 2013.

46. Maintenance of certification (MOC) four-part structure.Chapel Hill, NC: American Board of Pediatrics. Availablefrom: https://www.abp.org/abpwebsite/moc/aboutmoc/maintenanceofcertification(moc)four-partstructure.htm.Accessed July 10, 2013.

47. American College of Graduate Medical Education.Scholarly activity guidelines. Review Committee for Fam-ily Medicine. Available from: https://www.acgme.org/acgmeweb/Portals/0/PFAssets/ProgramResources/120_Family_Medicine_Scholarship_Guidelines.pdf. AccessedJuly 10, 2013.


preventing the voltage drop: keeping practice-based

Documents