effective in-service training design and delivery - human resources

Bluestone et al. Human Resources for Health 2013, 11:51http://www.human-resources-health.com/content/11/1/51

RESEARCH Open Access

Effective in-service training design and delivery:evidence from an integrative literature reviewJulia Bluestone1*†, Peter Johnson1†, Judith Fullerton2, Catherine Carr1†, Jessica Alderman3 and James BonTempo1

Abstract

Background: In-service training represents a significant financial investment for supporting continued competenceof the health care workforce. An integrative review of the education and training literature was conducted toidentify effective training approaches for health worker continuing professional education (CPE) and what evidenceexists of outcomes derived from CPE.

Methods: A literature review was conducted from multiple databases including PubMed, the Cochrane Library andCumulative Index to Nursing and Allied Health Literature (CINAHL) between May and June 2011. The initial reviewof titles and abstracts produced 244 results. Articles selected for analysis after two quality reviews consisted ofsystematic reviews, randomized controlled trials (RCTs) and programme evaluations published in peer-reviewedjournals from 2000 to 2011 in the English language. The articles analysed included 37 systematic reviews and 32RCTs. The research questions focused on the evidence supporting educational techniques, frequency, setting andmedia used to deliver instruction for continuing health professional education.

Results: The evidence suggests the use of multiple techniques that allow for interaction and enable learners toprocess and apply information. Case-based learning, clinical simulations, practice and feedback are identified aseffective educational techniques. Didactic techniques that involve passive instruction, such as reading or lecture,have been found to have little or no impact on learning outcomes. Repetitive interventions, rather than singleinterventions, were shown to be superior for learning outcomes. Settings similar to the workplace improved skillacquisition and performance. Computer-based learning can be equally or more effective than live instruction andmore cost efficient if effective techniques are used. Effective techniques can lead to improvements in knowledgeand skill outcomes and clinical practice behaviours, but there is less evidence directly linking CPE to improvedclinical outcomes. Very limited quality data are available from low- to middle-income countries.

Conclusions: Educational techniques are critical to learning outcomes. Targeted, repetitive interventions can resultin better learning outcomes. Setting should be selected to support relevant and realistic practice and increaseefficiency. Media should be selected based on the potential to support effective educational techniques andefficiency of instruction. CPE can lead to improved learning outcomes if effective techniques are used. Limited dataindicate that there may also be an effect on improving clinical practice behaviours. The research agenda calls forwell-constructed evaluations of culturally appropriate combinations of technique, setting, frequency and media,developed for and tested among all levels of health workers in low- and middle-income countries.

Keywords: In-service training, Continuing professional education, Continuing medical education, Continuingprofessional development

* Correspondence: [email protected]†Equal contributors1Jhpiego Corporation, 1615 Thames Street, Baltimore, MD 21231, USAFull list of author information is available at the end of the article

© 2013 Bluestone et al.; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the CreativeCommons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, andreproduction in any medium, provided the original work is properly cited.

mailto:[email protected]

http://creativecommons.org/licenses/by/2.0

Bluestone et al. Human Resources for Health 2013, 11:51 of 26http://www.human-resources-health.com/content/11/1/51

BackgroundThe need to increase the effectiveness and efficiency ofboth pre-service education and continuing professionaleducation (CPE) (in-service training) for the health work-force has never been greater. Decreasing global resourcesand a pervasive critical shortage of skilled health workersare paralleled by an explosion in the increase of and accessto information. Universities and educational institutionsare rapidly integrating different approaches for learningthat move beyond the classroom [1]. The opportunitiesexist both in initial health professional education andCPE to expand education and training approaches be-yond classroom-based settings.An integrative review was designed to identify and re-

view the evidence addressing best practices in the designand delivery of in-service training interventions. The useof an integrative review expands the variety of researchdesigns that can be incorporated within a review’s inclusioncriteria and allows the incorporation of both qualitativeand quantitative information [2]. Five questions were for-mulated based on a conceptual model of CPE developedby the Johns Hopkins University Evidence-Based PracticeCenter (JHU EPC) for an earlier systematic review ofcontinuing medical education (CME) [3]. We askedwhether: 1. particular educational techniques, 2. frequencyof instruction (single or repetitive), 3. setting where instruc-tion occurs, or 4. media used to deliver the instructionmake a difference in learning outcomes; and, 5. if there wasany evidence regarding the desired outcomes, such asimprovements in knowledge, skills or changes in clinicalpractice behaviours, which could be derived from CPE,using any mixture of technique, media or frequency.

MethodsInclusion/exclusion criteriaArticles were included in this review if they addressedany type of health worker pre-service or CPE event, andincluded an analysis of the short-term evaluation and/orassessment of the longer-term outcomes of the training.We included only those articles published in Englishlanguage literature. These criteria gave priority to articlesthat used higher-order research methods, specificallymeta-analyses or systematic reviews and evaluations thatemployed experimental designs. Articles excluded fromanalysis were observational studies, qualitative studies,editorial commentary, letters and book chapters.

Search strategyA research assistant searched the electronic, peer-reviewedliterature between May and June 2011. The search wasconducted on studies published in the English languagefrom 2000 to 2011. Multiple databases including PubMed,the Cochrane Library and Cumulative Index to Nursingand Allied Health Literature (CINAHL) were utilized in

the search. Medical subject headings (MeSH) and keysearch terms are presented below in Table 1.

Study type, quality assessment and gradeAn initial review of titles and abstracts produced 244 re-sults. We identified the strongest studies available, using arange of criteria tailored to the review methodology. Initialselection criteria were developed by a panel of experts.Grading and inclusion criteria are presented in Table 2.The grading criteria were adapted from the Oxford Centrefor Evidence-Based Medicine (OCEMB) levels of evidencemodel [4]. Grading of studies included within systematicreviews was reported by authors of those reviews and wasnot further assessed in this integrative review. Therefore,reference to quality of studies in our report refers to those apriori judgments. Only tier 1 articles (grades 1 and 2) wereincluded in our analysis.After prioritization of the articles, 163 tier 1 articles

were assessed by a senior public health professional todetermine topical relevance, study type and grade. A totalof 61 tier 1 studies were selected to be included in theanalysis following this second review. An additional handsearch of the reference lists cited in published studieswas conducted for topics that were underrepresented,specifically on the frequency and setting of educationalactivities. This search added eight articles for a total of69 studies, including 37 systematic reviews and 32 ran-domized controlled trials (RCTs), see inclusion processfor articles included in analysis, Figure 1.A data extraction spreadsheet was developed, following

the model offered in the Best Evidence in Medical Educa-tion (BEME) group series [5] and the conceptual modeland definition of terms offered by Marinopoulos et al. inthe JHU EPC earlier review of CME [3]. Categorizationdecisions were necessary in cases when the use of termin-ology was inconsistent with the Marinopoulos et al. defini-tions of terms for CPE [3]. For example, an article thatanalysed ‘distance learning’ as a technique and used thecomputer as the medium to deliver an interactive e-learning course was coded and categorized as an ‘inter-active’ technique delivered via ‘computer’ as themedium of instruction. See illustration of categorizationterminology in panels A, B, and C, Figure 2, for an il-lustration of how terminology was used to categorizeand organize articles for analysis.

ResultsSelected articles that best represent common findingsand outcomes (effects) of CPE are discussed in the resultsand discussion sections; the related tables present allthe articles analysed and categorized for that topic, andeach article is included only once. Relevant informa-tion obtained from educational psychology literature isreferenced in the discussion.

Table 1 Medical subject headings (MeSH) and key search terms

Group-based education Asynchronous distance learning Nursing education

Facility-based education Synchronous distance learning Medical education

On-the-job education Online learning Teaching methods

Group-based training Distance learning Health care professionals

Facility-based training Continuing medical education Education methods

On-the-job training Continuing nursing education Continuing education methods

Point-of-care training Nursing education methods

Mobile technologies Medical education methods


TechniquesThe articles or studies that specifically addressed educa-tional techniques are summarized in Table 3. Techniquerefers to the educational methods used in the instruction.Technique descriptions are based on the Marinopouloset al. definitions of terms [6] and reflect the approachesdefined in the articles analysed.

Case-based: use of created or actual clinical cases thatpresent materials and questionsThough case-based learning was not specifically comparedwith other techniques in the literature reviewed, it wasoften noted as a method in articles that discussedinteractive techniques. Case-based learning was alsonoted as a technique used for computer-delivered CPEcourses. Triola et al. compared types of media utilizedfor case-based learning and found positive learning out-comes both with the use of a live standardized patientand a computer-based virtual patient [7].

Didactic/lecture: presenting knowledge content; facilitatordetermines content, organization and paceLecture was often referred to in the literature as traditionalinstruction, lecture-based or didactic teaching. Didacticinstruction was not found to be an effective educationaltechnique compared with other methods. Two studies[8,9] found no statistical difference in learning outcomes,and three studies found didactic to be less effective thanother techniques [10-12]. Reynolds et al. compared didactic

Table 2 Grading criteria

Design Type of groups

Meta-analysis or systematic review NA

Experimental Between subjects (experim

Within subjects (crossover

Quasi-experimental Non-equivalent control gr

Repeated measures

Pre-experimental Comparison group

Pre-test/post-test

Post-test only

NA not applicable.

instruction with simulation. The study was limited by smallsample size (n = 50), but still demonstrated that the simula-tion group had a significantly higher mean post-test score(P <0.01) and overall higher learner satisfaction [12].Several systemic reviews that compared didactic in-

struction to a wide variety of teaching approaches alsoidentified didactic instruction as a less effective educa-tional technique [13-15].

Feedback: providing information to the learner aboutperformanceMultiple articles identified feedback as important foroutcomes [16-18]. Herbert et al. compared individualizedfeedback in the form of a graphic (a prescribing portraitbased on personal history of drug-prescribing practices) tosmall group discussion of the same material and foundthat both the feedback and the live, interactive sessionwere somewhat effective at changing physician’s pre-scribing behaviours [16]. The Issenberg et al. systematicreview of simulation identified practice and feedback askey for effective skill development [17]. A Cochrane reviewof the evidence to support CPE suggested the importanceof feedback and instructor interaction in improvinglearning outcomes [18].

Games: competitive game with preset rulesThe use of games as an instructional technology wasaddressed in one rigorous systematic review. The authorsfound only a limited number of studies, which were of low

Literature grade Tier

1

ental and control) 2 1

) 2

oup 3

3

4

4 2

5

37Systematic

Reviews

32Randomized Control Trials

Journal Database

244

133Grade One

30Grade Two

61First Tier

69First Tier

81Excluded for data

quality controlFirst Quality Review

Second Quality Review

Additional “Time and Setting” Hand Search

Titles and Abstract Review

102Excluded for data

quality control

8 Added afterhand review

Figure 1 Inclusion process for articles included in the analysis.

Figure 2 Illustration of categorization terminology in panels a-c.


to moderate methodological quality and offered inconsist-ent results. Three of the five RCTs included in the reviewsuggested that educational games could have a positiveeffect on increasing medical student knowledge and thatthey include interaction and allow for feedback [19].

Interactive: provide for interaction between the learner andfacilitatorFive articles specifically compared interactive CPE toother educational techniques. De Lorenzo and Abbotfound interactive techniques to be moderately superiorfor knowledge outcomes than didactic lecture [10]. Twoother studies found interactive techniques were moreeffective when feedback from chart audits was added tothe intervention [16,20].Three systematic reviews and one meta-analysis specific-

ally noted the importance of learner interactivity or engage-ment in learning in achieving positive learning outcomes[21-24] (refer to summary of articles focused on outcomes).

Point-of-care (POC): information provided as needed, at thepoint of clinical careTwo articles and one systematic review specificallyaddressed point-of-care (POC) as a technique. The sys-tematic review included three studies and concludedthat while the findings were weak, they did indicate thatPOC led to improved knowledge and confidence [25]. Inan examination of media, Leung et al. determined thathandheld devices were more effective than print-based,POC support, although outcome measures were self-reported behaviours [26]. You et al. found improvedperformance on a procedure among surgical residents who

Table 3 Summary of articles focused on techniques

Citation Study design Participants Intervention Key findings

Aki E et al. 2010 Systematic review: five articles reviewed todetermine the effectiveness of educationalgaming on learning

Mostly medical students Technique: educational games Findings in three of the five RCTs suggestedbut did not confirm a positive effect of thegames on medical students’ knowledge.Media: multiple

Frequency: NR

Blaya J et al. 2010 Systematic review: 45 articles included forreview, only three related to POC support,included qualitative and quantitative data

Nurses in developing countries Technique: didactic vs POC POC findings: studies were weak but indicatedknowledge improved and increased rapport intrusting personal judgment.Media: computer-based vs live

Frequency: NR

Bruppacher H et al. 2010 Prospective, single-blinded RCT todetermine if simulation or interactivetechniques are better for teachingweaning a patient from anaesthesia

Anaesthesiology trainees,post-graduate year 4

Technique: simulation vs interactive The simulation group scored significantlyhigher than the seminar group at bothpost-test and retention test. Clinicaldecision-making/psychomotor skills canbe acquired via simulation.

I = 10, C = 10 Media: live

Country: China Frequency: single

Intervention group received simulation-basedtraining; control group received an interactiveseminar.

Daniels K et al. 2010 Prospective RCT to determine if simulationis more effective than didactic in obstetricemergency management

Residents and labour anddelivery nurses

Technique: simulation vs interactive Simulation-trained teams had superiorperformance scores when tested in a labourand delivery drill. In an academic trainingprogramme, didactic and simulation-trainedgroups showed equal results on written testscores.


Country: USA Frequency: single

Intervention group received simulation-basedtraining; control group received an interactiveseminar.

De Lorenzo R andAbbott C 2004

RCT to determine if the adult learningmodel improves student learning in termsof cognitive performance and perceptionof proficiency in military medic training

Army medic students Technique: interactive vs didactic The adult learning model offered only amodest improvement in cognitive evaluationscores over traditional teaching. Additionally,students in the traditional teaching modelassessed themselves as proficient morefrequently than instructors, whereasinstructor and student perception ofproficiency were more closely matched inthe adult learning model.

n = 150, I = 81, C = 69 Media: live


Intervention group emphasized the principlesof adult learning including small groupinteractive approach, self-directed study,multimedia didactics and intensive integratedpractice of psychomotor skills; control groupreceived a traditional, lecture-based course.

Harder BN 2010 Systematic review: 23 articles reviewed toevaluate the use of clinical simulation inhealth care education

Health professionals Technique: simulation Inconclusive evidence about the use ofsimulation due to a low number of studies.However, the use of simulation, as opposedto other education and training methods(motor skills laboratory sessions with tasktrainers, computer-based instruction andlecture classes), increased students’ clinicalskills in the majority of studies.

Media: multiple

Frequency: single

Bluestoneet

al.Hum

anResources

forHealth

2013,11:51Page

5of

26http://w

ww.hum

an-resources-health.com/content/11/1/51

Table 3 Summary of articles focused on techniques (Continued)

Herbert C et al. 2004 RCT to assess the impact of individualizedfeedback and live, interactive groupeducation on prescriptive practices

Physicians Technique: audit and feedback vs interactiveplus audit and feedback vs interactive sessiononly vs nothing

Increase in prescribing preference for correctdrug class in module and “prescribingportraits” (graphic comparisons betweenindividual, group and evidence basedprescribing practices) group. Evidence-basededucational interventions combiningpersonalized prescribing feedback withinteractive group discussion can lead tomodest but meaningful changes in physicianprescribing.

I1 = 48, audit and feedback only;I2 = 47, interactive module only;I3 = 49, interactive plus auditand feedback; C = 56, nothing

Media: live

4,394 charts reviewed Frequency: single

Country: Canada

Issenberg S et al. 2005 Systematic review: 109 studies reviewedto determine the use of high-fidelitymedical simulations that lead to mosteffective learning

Health professionals Technique: simulation The weight of the best available evidencesuggests that high-fidelity medicalsimulations facilitate learning under the rightconditions. These conditions include:providing feedback, repetitive practice,curriculum integration, range of difficulty,multiple learning strategies, capture clinicalvariation, controlled environment,individualized learning, defined outcomesand simulator validity.

Media: multiple

Frequency: both single and multiple

Lamb D 2007 Literature review: nine articles reviewed todetermine effectiveness of experiential(focused on simulations) learning

Health professionals Technique: simulation None of the studies showed conclusivelythat simulated learning improves patientoutcome; however, evidence suggestshuman patient simulators to beadvantageous over other modalities. Theyhave been proven to be at least as effectiveas traditional teaching by didactic methods.Both human patient simulators (models)and computer-simulations may be effective.

Media: multiple


Laprise R et al. 2009 Cluster randomized trial of 122 generalpractitioners to determine if chart auditsand feedback reminders after a CMEevent lead to better adherence to clinicalguidelines

General practitioners Technique: audit and feedback plusinteractive vs interactive only

This study demonstrated significantlyimproved adherence in the interventiongroup using chart audits vs CME alone. Themagnitude of the difference observedbetween the two groups in absolute pre-postintervention change is consistent withprevious studies on the effectiveness ofchart prompting in preventive care.

n = 122, I = 61, C = 61 Media: live

Chart audit of 2,344 consentingpatient charts

Frequency: single vs multiple

Country: Canada Intervention group and control group receivedthe same CME intervention, a 2-hour live,interactive workshop. The intervention groupalso received six monthly follow-up visits froma nurse that included chart screening, auditsand feedback, and a print-based checklistdistribution and print summary of expertrecommendations.

Lin C et al. 2010 RCT to determine if peer-tutored, PBL ispreferable to didactic-based instructionfor teaching nursing ethics

Nursing students Technique: PBL vs didactic Peer-tutored, PBL was shown to be moreeffective than conventional lecture-typeteaching. Peer-tutored, PBL has the potentialto enhance the efficacy of teaching nursing


Country: Taiwan Frequency: single

Bluestoneet

al.Hum

anResources

forHealth

2013,11:51Page

6of

26http://w

ww.hum



ethics in situations in which there arepersonnel and resource constraints.

Intervention group received PBL technique;control group received didactic-basedinstruction.

McGaghie W et al. 2009a Systematic review: nine of the JHU EPCsystematic review articles reviewed todetermine the effectiveness of simulationmethods in medical education outsideof CME

Health professionals Technique: simulation Due to a low number of studies, evidence onsimulation methods is inconclusive. However,the direction of evidence points to theeffectiveness of simulation training, especiallyfor psychomotor and communication skills.Data analysis revealed a highly significant‘dose-response’ relationship among practiceand achievement, with more practiceproducing higher outcome gains.

Media: multiple


Merien A et al. 2010 Systematic review: eight articles reviewedto determine the effectiveness ofteam-based training for obstetric care

Health professionals Technique: team-based Due to a low number of studies, evidenceon teamwork training in simulation isinconclusive. However, introduction ofmultidisciplinary teamwork training withintegrated acute obstetric traininginterventions in a simulation setting ispotentially effective in the prevention oferrors, thus improving patient safety inacute obstetric emergencies.

Media: live

Frequency: NR

Murad MH et al. 2010 Systematic review: 59 articles (enrolled8,011 learners) reviewed to determineeffectiveness of self-directed learning

Health professionals Technique: self-directed Moderate-quality evidence suggests thatself-directed learning in health professionseducation is associated with moderateimprovement in the knowledge domaincompared with traditional teaching methods,and may be as effective in the skills andattitudes domains.

Media: multiple

Frequency: NR

Perry M et al. 2011 Systematic review: six articles representingfive studies were reviewed to determinethe effect of educational interventions inprimary dementia care

Health professionals Technique: multiple Interactive workshops and decision supportsystems led to increased detection rates.Evidence shows moderate improvementsin knowledge and techniques that requiredactive participation tended to improvedetection rates.

Media: multiple


Reynolds A et al. 2010 RCT to compare students’ knowledgeusing either simulation or didactic lecture

Midwifery students Technique: simulation vs didactic A significantly higher short-term reinforcementof knowledge and greater learner satisfactionwas obtained using simulation-based trainingcompared to image-based lectures whenteaching routine management of normaldelivery and resolution of shoulder dystociato midwives in training.


Country: Portugal Frequency: single

Intervention group received simulation-basedtraining; control group received didacticlectures with print visuals.

Bluestoneet

al.Hum

anResources

forHealth

2013,11:51Page

7of

26http://w

ww.hum



Smits P et al. 2003 RCT to compare effectiveness of PBL vsdidactic for management of mentalhealth problems

Post-graduate medical trainees Technique: PBL vs didactic The study found that both PBL anddidactic-based instruction were effective, buthad no statistical difference. The PBLprogramme appeared to be more effectivethan the lecture-based programme inimproving performance, but received lessfavourable evaluations.


Country: the Netherlands Frequency: single


Steadman R et al. 2006 RCT to determine if simulation is betterthan PBL for teaching assessment andmanagement skills

4th year medical students Technique: simulation vs PBL Simulation-based teaching was superior toPBL for the acquisition of critical assessmentand management skills.I = 15, C = 16 Media: live


Intervention group received simulation-basedteaching; control group received PBL.

Sturm L et al. 2008 Systematic review: 11 articles reviewed todetermine if skills acquired bysimulation-based training transfer to theoperative setting

Surgeons Technique: simulation Due to limited quality and methodology anda lack of relevant studies, a weak conclusioncan be made supporting the transfer of skillsdeveloped in simulation to the operativesetting. Evidence from one study showedbetter performance for participants whoreceived simulation-based training beforeundergoing patient-based assessment thantheir counterparts who did not receiveprevious simulation training.

Media: multiple


Werb S and Matear D 2004 Systematic review: three systematicreviews and nine original research articlesreviewed to examine evidence-basedclinical teaching and faculty continuingeducation

Allied health professionals Technique: PBL PBL and evidence-based health careinterventions were effective in increasingstudents’ knowledge of medical topics andtheir ability to search, evaluate and appraisemedical literature. Dental students in a PBLcurriculum, emphasizing evidence-basedpractices, scored higher on the NationalDentistry Boards, Part I, than students intraditional curricula.

Media: multiple


White M et al. 2004 RCT to investigate effectiveness of PBL vsdidactic for asthma management

Physicians Technique: PBL vs didactic There was no significant difference inknowledge gained or satisfaction with thefacilitator between the PBL group and thelecture-based group. The PBL group ratedthe educational value higher than thedidactic group.


Country: Canada Frequency: single


Young J and Ward J 2002 Randomized trial to determine the effectof self-directed (distance) learning onknowledge, attitudes and practices relatedto smoking cessation

Family physicians Technique: self-directed vs reading Modest changes from baseline to post-testfor both the distance learning group andself-directed group suggest a lack ofsignificant evidence to support a distance orself-directed approach to address changesin practice.

I = 26, C = 27 Media: print

Country: Australia Frequency: single

Intervention group received a self-directedlearning module; control group receivedguidelines only.

Bluestoneet

al.Hum

anResources

forHealth

2013,11:51Page

8of

26http://w

ww.hum



Yuan H et al. 2008 Systematic review: 10 studies reviewed todetermine the evidence to support PBL

Nursing students Technique: PBL Inconclusive evidence to support PBL. Whileseveral studies showed increased reportedself-confidence in ability to make decisions,and several showed increased skills in criticalthinking questions from the PBL group,overall findings were inconclusive due to alack of quality studies.

Media: multiple


Zurovac D et al. 2011 Cluster RCT at 107 rural health facilities todetermine if text-message reminderswould improve provider adherence tonational malaria treatment guidelines

Health professionals Technique: reminders The use of mobile technology showedsignificant improvement in case managementpractice for paediatric malaria cases amongphysicians with repetitive text-messagereminders compared to control group.

119 health workers Media: mobile phone

Case-management practiceswere assessed for 2,269 childrenwho needed treatment

Frequency: repetitive

I = 1,157, C = 1,112 Intervention group received repetitive textmessages over a 6-month period; controlgroup received nothing.Country: Kenya

aJHU EPC systematic review. C Control, CME Continuing medical education, I Intervention, JHU EPC Johns Hopkins University Evidence-Based Practice Center, NR Not reported, PBL problem-based learning,POC point-of-care, RCT randomized controlled trial.

Bluestoneet

al.Hum

anResources

forHealth

2013,11:51Page

9of

26http://w

ww.hum



received POC mentoring via a video using a mobile de-vice, compared with those who received only didacticinstruction [27].

Problem-based learning (PBL): present a case, assigninformation-seeking tasks and answer questions about thecase; can be facilitated or non-facilitatedFour articles specifically compared problem-based learning(PBL) to other methods. One study identified PBL asslightly better [11], and two studies indicated it to berelatively equal to didactic instruction [8,9]. A systematicreview of 10 studies on PBL reported inconclusive evidenceto support the approach, although several studies reportedincreased critical thinking skills and confidence in makingdecisions [28].

Reminders: provision of remindersThe Zurovac et al. study conducted in Kenya found thatusing mobile devices for repetitive reminders resulted insignificant improvement in health care provider’s casemanagement of paediatric malaria, and these gains wereretained over a 6-month period [29]. Intention-to-treatanalysis showed that correct management improved by23.7% (95% confidence interval (CI) 7.6 to 40.0, P <0.01)immediately after intervention and by 24.5% (95% CI 8.1 to41.0, P <0.01) 6 months later, compared with the controlgroup [29]. Reminders were also noted as an effectivetechnique by two of the systematic reviews [13,14].

Self-directed: completed independently by the learner basedon learning needsThis term was difficult to extract for analysis due towidely varying terminology. Some authors used the term‘distance learning’, and some used it to define the mediumof delivery, rather than technique. This analysis specificallydiscusses articles that were consistent with the descriptionfor self-directed learning, even if the authors used dif-ferent terminology.A recent systematic review identified that moderate-

quality evidence suggests a slight increase in knowledgedomain compared with traditional teaching, but notesthat this may be due to the increased exposure to content[30]. One RCT found modest improvements in knowledgeusing a self-directed approach, but noted it was less effect-ive at impacting attitudes or readiness to change [31].Multiple studies focused on use of the computer as the

medium to deliver instruction and noted that self-directedinstruction was equally (or more) effective as instructor-led didactic or interactive instruction and potentiallymore efficient.Simulation may include models, devices, standardized

patients, virtual environments, social or clinical situationsthat simulate problems, events or conditions experiencedin professional encounters [17]. Simulation was noted as an

effective technique for promotion of learning outcomesacross the systematic reviews, particularly for the devel-opment of psychomotor and clinical decision-makingskills. The systematic reviews all highlighted inconclusiveand weak methodology in the studies reviewed, but notedsufficient evidence existed to support simulation as usefulfor psychomotor and communication skill development[32-34] and to facilitate learning [35]. The systematic reviewby Lamb suggests that patient simulators, whether com-puter or anatomic models, are one of the more effectiveforms of simulations [36].Outcomes of the four separate RCTs indicated simulation

was better than the techniques to which they werecompared, including interactive [37,38], didactic [12] andproblem-based approaches [35]. A study by Daniels et al.found that although knowledge outcomes were similar be-tween the interactive and simulation groups, the simulationteam performance in a labour and delivery clinical drill wassignificantly higher for both shoulder dystocia (11.75 versus6.88, P <0.01) and eclampsia (13.25 versus 11.38, P = 0.032)at 1 month post-intervention [38].Simulation was also found to be useful for identifying

additional learning gaps, such as a drill on the task ofmixing magnesium sulfate for administration [39]. A sys-tematic review focused on resuscitation training identifiedsimulation as an effective technique, regardless of mediaor setting used to deliver it [40].

Team-based: providing interventions for teams that providecare togetherArticles discussed here focused on the technique of pro-viding training to co-workers engaged as learning teams.One systematic review of eight studies found that there islimited and inconclusive evidence to support team-basedtraining [41]. Two of the articles reporting on the sameCPE study did not identify any improvements in per-formance or knowledge acquisition with the addition ofusing a team-based approach [39,42].

FrequencyThis review included consideration of frequency, compar-ing single versus repetitive exposure. The findings regardingfrequency are summarized in Table 4.The three articles focused on frequency all support the

use of repetitive interventions. These studies evaluatedrepetition using the Spaced Education platform (now calledQstream), an Internet-based medium that uses repeatedquestions and targeted feedback. The evidence from thesethree articles demonstrated that repetitive, time-spacededucation exposures resulted in better knowledge outcomes,better retention and better clinical decisions compared withsingle interventions and live instruction [43-45].The use of repetitive or multiple exposures is supported

in other systematic reviews of the literature, as well as one

Table 4 Summary of articles focused on frequency


Kerfoot BP et al. 2007 RCT to determine if spacing principlescan improve acquisition and retentionof medical knowledge

Five cohorts with 76 to 80 urologyresidents in each cohort

Frequency: multiple vs single Conclusive evidence to support repetitive,spaced education in online learning, sinceresidents in the spaced education cohortdemonstrated significantly greater onlinetest scores than those in the bolus cohort.The scores for the spaced cohort remainedstable with no overtime, while test scores inthe bolus cohort demonstrated a significantlinear decrease.

Of 537 participants, 400 (74%)completed the online staggered testsand 515 (96%) completed theIn-Service Examination

Technique: self-directed

Cohort 1 = bolus, single intervention;Cohort 2 =multiple, spacedintervention

Media: Internet-based

Country: USA and Canada Cohort 1 received bolus education of 96 studyquestions (June 2005); Cohort 2 received dailyemails over 27 weeks (June to December 2005),each with one to two questions in spacedpattern. In November 2005, all participantscompleted the urology exam. Participants wererandomized to five cohorts and completed a32-item online test at staggered time points(1 to 14 weeks) after completion of SpacedEducation.

Kerfoot BP et al. 2009 RCT to determine if Spaced Educationis an effective form of CME

Urologists and urology residents Frequency: multiple vs single Conclusive evidence to support the use ofISE programmes. Knowledge scores of ISEintervention were statistically significantlyhigher than those of the control bolusmethod.

Completed by 71% of urologistsand 83% of residents


Cohort 1 = 80 urologists, 160residents, completed by 196;Cohort 2 = 80 urologists, 160residents, completed by 182

Media: Internet-based

Country: USA (March to July 2007) A total of 160 urologists and 320 urology residentswere randomized to one of two cohorts.Participants were stratified by training level(urologist in practice vs resident) and urologytraining year (residents only) and were blockrandomized (block size = 8) to one of twocohorts. Participants in Cohort 1 received the3-cycle ISE course on the HP CPGs, with 24control items on the SIA CPGs in cycle 3.Participants in Cohort 2 received the 3-cycleISE course on SIA CPGs, with 24 control itemson HP CPGs in cycle 3. The trial was structuredin this manner to allow the topic-specificlearning gains from the ISE courses to beidentified in cycle 3. Since the 24 items arepresented simultaneously to both cohorts incycle 3, the learning gains of physicians whohad completed two cycles of the ISEprogramme could be directly compared withthose physicians who were presented with thematerial for the first time (controls).

Bluestoneet

al.Hum

anResources

forHealth

2013,11:51Page

11of

26http://w

ww.hum


Table 4 Summary of articles focused on frequency (Continued)

Kerfoot BP et al. 2010 RCT to determine if Spaced Educationcan effect knowledge transfer and theability to make diagnostic decisions

Urology residents Frequency: multiple vs single Conclusive evidence to support spaced,web-based education compared to WBT.Spaced education demonstrated a staticallysignificant increase in knowledge andlong-term retention of knowledgecompared with bolus web-based modulesthat delivered the same content ofhistopathology diagnostic skills.

Cohort 1 = 164; Cohort 2 = 194 Technique: self-directed

Country: USA (June 2007 to June 2008) Media: Internet-based

Transfer and retention of diagnostic skillsbetween Spaced Education vs bolus, WBT

All residents were sent both spaced educationand WBT, but the set of topics delivered byeach method varied by cohort. Residents inCohort 1 received three cycles of spacededucation on prostate-testis (weeks 1 to 4, 5 to8, and 13 to 16) and three WBT modules onbladder-kidney (weeks 14 to 16). Residents inCohort 2 received three cycles of spacededucation on bladder-kidney (weeks 1 to 4, 5 to8, and 13 to 16) and three WBT modules onprostate-testis (weeks 14 to 16). The spacededucation items were delivered each weekdaythrough emails containing one question/answer,and the spaced education material wasdistributed in three cycles or repetitions to takeadvantage of the spacing effect. The WBT usedthe identical content and delivery system, withthe questions aggregated into three20-question modules delivered throughseparate emails in week 14. The trial wasspecifically structured to ensure that within agiven set of topics (bladder-kidney orprostate-testis) the only difference betweenintervention cohorts was the spacing of content.

CME Continuing medical education, CPG Clinical practice guideline, HP Haematuria and priapism, ISE Interactive spaced education, RCT Randomized controlled trial, SIA Staghorn calculi, infertility, and antibiotic use,WBT Web-based teaching.

Bluestoneet

al.Hum

anResources

forHealth

2013,11:51Page

12of

26http://w

ww.hum



RCT conducted in Kenya that used repeated text remindersand resulted in a significant improvement in adherence tomalaria treatment protocols [29].

SettingSetting is the physical location within which the instructionoccurs. We identified three articles that looked specificallyat the training setting. The findings regarding setting aresummarized in Table 5. Two of them stemmed from thesame intervention. Crofts et al. specifically addressed theimpact of setting and technique (team-based training) onknowledge acquisition and found no significant differencein the post-score based on the setting [42]. A systematicreview of eight articles evaluating the effectiveness ofteam-based training for obstetric care did not find signifi-cant differences in learning outcomes between a simulationcentre and a clinical setting [41].Coomarasamy and Khan conducted a systematic review

and compared classroom or stand-alone versus clinicallyintegrated teaching for evidence-based medicine (EBM).Their review identified that classroom teaching improvedknowledge, but not skills, attitudes or behaviour outcomes;whereas clinically integrated teaching improved all out-comes [46]. This finding was supported by the Hamiltonsystematic review of CPE, which suggests that teaching in aclinical setting or simulation setting is more effective(Table 1), as well as the Raza et al. systematic review of 23studies to evaluate stand-alone versus clinically integratedteaching. This review suggested that clinically integratedteaching improved skills, attitudes and behaviour, not justknowledge [18].

MediaMedia refers to the means used to deliver the curriculum.The majority of RCTs compared self-paced or individualinstruction delivered via computer versus live, group-basedinstruction. The findings regarding media are summarizedin Table 6.

Live versus computer-basedLive instruction was found to be somewhat effective atimproving knowledge, but less so for changing clinicalpractice behaviours. When comparing live to computer-based instruction, a frequent finding was that computer-based instruction led to either equal or slightly betterknowledge performance on post-tests than live instruction.One of the few to identify a significant difference in out-comes, Harrington and Walker found the computer-basedgroup outperformed the instructor-led group on the know-ledge post-test and that participants in the computer-basedgroup, on average, spent less time completing the trainingthan participants in the instructor-led group [47].Systematic reviews indicate that the evidence supports

the use of computer-delivered instruction for knowledge

and attitudes; however, insufficient evidence exists to sup-port its use in the attempt to change practice behaviours.The Raza Cochrane systematic review identified 16randomized trials that evaluated the effectiveness ofInternet-based education used to deliver CPE to practicinghealth care professionals. Six studies showed a positivechange in participants’ knowledge, and three studiesshowed a change in practice in comparison with traditionalformats [18]. One systematic review noted the importanceof interactivity, independent of media, in achieving animpact on clinical practice behaviours [48].

MobileOne article assessed the use of animations against audioinstructions in cardiopulmonary resuscitation (CPR) usinga mobile phone and found the group that had audiovisualanimations performed better than the group that receivedlive instruction over the phone in performing CPR; how-ever, neither group was able to perform the psychomotorskill correctly [49]. Leung et al. found providing POCdecision support via a mobile device resulted in slightlybetter self-reporting on outcome measures compared withprint-based job aids, but that both the print and mobilegroups showed improvements in use of evidence-baseddecision-making [26].

PrintThe systematic review of print-based materials conductedby Farmer et al. did not find sufficient evidence to supportthe use of print media to change clinical practice be-haviours [50]. A comparison of the use of print-basedguidelines to a live, interactive workshop indicated thatthose who completed live instruction were slightly betterable to identify patients at high risk of an asthma attack.However, neither intervention resulted in changed practicebehaviours related to treatment plans [51].Multiple systematic reviews caution against the use

of print only media, concluding that live instruction ispreferable to print only. Another consistent theme wassupport for the use of multimedia in CPE interventions.

OutcomesOutcomes are the consequences of a training intervention.This literature review focuses on changes in knowledge,attitudes, psychomotor, clinical decision-making orcommunication skills, and effects on practice behavioursand clinical outcomes. All of the articles that focused onoutcomes were systematic reviews of the literature and aresummarized in Table 7.The weight of the evidence across several studies in-

dicated that CPE could effectively address knowledgeoutcomes, although several studies used weaker meth-odological approaches. Specifically, computer-based in-struction was found to be equally or more effective than

Table 5 Summary of articles focused on setting


Coomarasamy A and Khan K 2004(link to the follow-up study,Raza A et al. 2009)

Systematic review: 23 articles reviewedto determine the effect of stand-alonecompared to clinically integratedteaching in EBM

Post-graduate physicians,allied health professionals

Technique: multiple, focus on case-based Sufficient evidence to support the use ofclinically integrated teaching over stand-aloneeducation. While stand-alone teaching improvedknowledge, there were no improvements inskills, attitudes or behaviours, whereas clinicallyintegrated teaching showed improvements inknowledge, skills, attitude and behaviour.

Media: live


Crofts J et al. 2007 Prospective RCT to explore if knowledgeacquisition is influenced by trainingsetting or teamwork training

Senior doctors, junior andsenior midwives

Technique: team-based vs interactive Statistical evidence supported the use of live,multi-professional, obstetric emergency trainingto increase midwives’ and doctors’ knowledgeof obstetric emergency management. However,neither the location of training either in asimulation centre or in local hospitals, nor theinclusion of teamwork training, made anysignificant difference to the acquisition ofknowledge in obstetric emergencies.

Total of 140 participants;interdisciplinary teams offour or six in four blocks

Media: live

Country: UK Frequency: single

I1 = 1-day interactive at hospital(no team-based training); I2 = 1-dayinteractive at simulation centre(no team-based training); I3 = 2-dayteam training at hospital; I4 = 2-dayinteractive in simulation centre

Main outcome measured by a 185multiple-choice questionnaire completed3 weeks before and 3 weeks after thetraining intervention.

Ellis D et al. 2008 Same study design as Crofts et al. 2007 Same participants asCrofts et al. 2007

Same intervention as Crofts et al. 2007 Statistical evidence to support the use of live,eclampsia training to increase providers’performance rate for completion of basic tasks.Neither the location (simulation centre or inlocal hospitals), nor the inclusion of teamworktraining made any significant difference to theperformance results for basic task completion.

EBM Evidence-based medicine, I Intervention.

Bluestoneet

al.Hum

anResources

forHealth

2013,11:51Page

14of

26http://w

ww.hum


Table 6 Summary of articles focused on media used to deliver instruction


Augestad K andLindsetmo R 2009

Systematic review: 51 articles reviewedto determine usefulness ofvideoconferencing as a clinical andeducational tool

Surgeons Media: video Review discussed primarily observational dataon the use of videoconferencing for provisionof lecture, mentoring and POC support foremergencies or trauma settings. Methodologyof studies is weak, but shows promise forproviding POC and mentoring to rural settingsfrom specialists in other geographical areas.

Country: Norway and developedcountries

Technique: multiple

Frequency: NR

Bloomfield J et al. 2010 RCT to test if the theory and skill ofhandwashing can be taught moreeffectively when taught usingcomputer-assisted learning comparedto conventional face-to-face teaching

Nursing students Media: computer-based vs live The computer-assisted learning module wasan effective strategy for teaching both theoryand practice of handwashing to nursingstudents and was found to be at least aseffective as conventional, face-to-face teachingmethods. However, this finding must beinterpreted with caution in light of samplesize and attrition rates.

n = 242, I = 113, C = 118 Techniques: multiple

Country: UK Frequency: single

Intervention group received theory viacomputer-based module; control group viainstructor-led. The objectives and contentwere the same, both groups includedpractice opportunities.

Bradley P et al. 2005 Prospective RCT and qualitativeevaluation to compare self-directed,computer-based learning to traditional,live, interactive education techniques

Medical students Technique: self-directed vs interactive There were no differences in outcomes for thecomputer-based group compared to the live,interactive group in knowledge acquisition,critical appraisal skills or attitudes toward EBM.This trial and its accompanying qualitativeevaluation suggest that self-directed,computer-assisted learning may be analternative format for teaching EBM.

I = 85, C = 90

Country: Norway Media: computer-based vs live

Frequency: single

Intervention group received self-directed,computer-based modules on EBM; controlgroup received live, interactive sessions.

Choa et al. 2008 Single-blinded, cluster randomized trialto compare the effectiveness ofaudiovisual animated CPR instructionwith audio, dispatcher-assistedinstruction in participants with noprevious CPR training; both via mobilephones

Allied health professionals,hospital employees

Media: mobile, audiovisual animation vs audioinstructions from live dispatcher

Audiovisual animated CPR instruction viamobile phone resulted in better scores inchecklist assessment and time intervalcompliance in participants without CPR skillcompared to those who received CPRinstructions from a dispatcher. However, theaccuracy of important psychomotor skillmeasures was unsatisfactory in both groups.

Technique: POCI = 44, C = 41

Country: Korea Frequency: single

Intervention group used mobile phoneapplication with audiovisual animationinstructions for CPR; control group receivedaudio guidance from a live dispatcher.

Chui S et al. 2009 Experimental research design with twogroups, one pre-test and two post-tests,to determine the effectiveness ofcomputer-based interactive instructionvs video didactic instruction

Nurses Media: computer-based vs video Interactive, computer-assisted instructionincreased student assessment correctnesscompared to video didactic instruction forin-service neurological nursing educationafter statistical adjustments for length ofexperience.

I = 44, C = 40 Technique: self-directed interactive vs didactic

Country: Taiwan Frequency: single

Intervention group received computer-based,interactive educational module; control groupwatched a video of a lecture.

Bluestoneet

al.Hum

anResources

forHealth

2013,11:51Page

15of

26http://w

ww.hum


Table 6 Summary of articles focused on media used to deliver instruction (Continued)

Curran V and Fleet L 2005 Systematic review to evaluate thenature and characteristics of theweb-based CME, based on Kirkpatricklevels of evaluation; 86 studies wereidentified, majority were descriptive

Physicians Media: Internet Inconclusive evidence to identify the mosteffective characteristics of web-based CMEdue to a lack of studies focusing onperformance change. Findings suggestweb-based CME is effective in enhancingknowledge and attitudes. Several studiessuggest interactive CME that requiresparticipant activity and the chance to practiceskills can effect changes in practice behaviours.

Technique: multiple


Farmer A et al. 2008 Systematic review: 23 studies reviewedto determine the usefulness ofprint-based materials in practicebehaviours or clinical practiceoutcomes

Health care professionals Media: print Insufficient information to support theeffectiveness of print-based educationalmaterials compared to other interventions.Print materials may have a beneficial effecton process outcomes compared to nointervention, but not on clinical practiceoutcomes.

Technique: didactic

Frequency: single

Fordis M et al. 2005 RCT to determine if Internet-basedCME can produce changes comparableto those produced via live, small group,interactive CME with respect tophysician knowledge and behavioursthat have an impact on patient care

Physicians Media: Internet-based vs live, interactive Internet-based CME can produce objectivelymeasured changes in behaviour as well assustained gains in knowledge that arecomparable or superior to those realizedfrom an effective, live, group-based activity.The Internet-based intervention wasassociated with a significant increase in thepercentage of high-risk patients treated withpharmacotherapeutics according to guidelinescompared to the live, group-based controlgroup.

n = 97; I = 49, randomly assignedInternet-based over 2 weeks;C1 = 44, single, live, interactivesession; C2 = 18, from same sitesreceived nothing

Technique: self-directed vs interactive

Frequency: single

Intervention group received Internet-basedmodules over 2 weeks; one control groupreceived a live, interactive session and theother control group received nothing.

Country: USA

Hadley J et al. 2010 Cluster RCT to evaluate the educationaleffectiveness of a clinically integratede-learning course for teaching basicEBM among post-graduate medicaltrainees compared to a traditionallecture-based course of equivalentcontent

Post-graduate medical trainees,interns

Media: Internet vs live An e-learning course in EBM was as effectivein improving knowledge as a standardlecture-based course. There was no statisticallysignificant difference in knowledge ofparticipants in the e-learning coursecompared to the lecture-based course. Thebenefits of an e-learning approach includestandardization of teaching materials and it isa potential cost-effective alternative tostandard, lecture-based teaching.

Techniques: multiple

Frequency: single

Intervention group received clinical integrated,e-learning course on EBM; control groupreceived live, didactic-based course.

Seven clusters of 237

I = 88, C = 72

Country: UK

Harrington S andWalker B 2004

RCT to determine effectiveness ofcomputer-based training comparedwith the traditional, instructor-ledformat

Nurses Media: computer-based vs live The computer-based group significantlyoutperformed the instructor-led group on theknowledge sub-test at post-test (gain of 28%vs 26%). Participants reported linked,computer-based learning and researchersnoted potential for efficiencies and costreduction.

n = 1,294, I = 670, C = 624 Technique: didactic vs self-directed


Intervention group received self-directed,computer-based instruction; control groupreceived instructor-led, live instruction. Bothgroups had the same objectives and content.

Bluestoneet

al.Hum

anResources

forHealth

2013,11:51Page

16of

26http://w

ww.hum



Horiuchi S et al. 2009 RCT compared web-based to liveinstruction

Nurses or midwives Media: Internet vs live No significant differences in knowledge wereobserved between the web-based andface-to-face group. However, the web-basedinstruction was rated as more flexible andaffordable and had a lower drop-out rate thanthe face-to-face programme.

n = 93; C = 45, web-based;I = 48, live

Techniques: multiple

Frequency: single

Intervention group received web-basedinstruction; control group received didacticlive instruction.

Country: Japan

Kemper K et al. 2006 National randomized 2 x 2 factorial trial Health professionals Media: Internet There were statistically significant improvementsin knowledge, confidence and communicationscores after the course for each of theInternet–based delivery methods, with nosignificant differences in any of the threeoutcomes by delivery strategy. Outcomes werebetter for those who paid for continuingeducation credit.

n = 1,267; completion rate = 62%;Group 1 = 318; Group 2 = 318;Group 3 = 318; Group 4 = 313


Frequency: single

Group 1: four modules delivered weekly over10 weeks by email (drip-push); Group 2:modules accessible on web site with fourreminders weekly for 10 weeks (drip-pull);Group 3: 40 modules delivered within 4 daysby email (bolus-push); and Group 4: 40modules available on the Internet with oneemail informing participants of availability(bolus-pull).

Country: USA

Leung G et al. 2003 RCT to compare the effectiveness ofmobile, POC support vs print-basedjob aids

4th year medical students Media: mobile vs print Both the PDA and pocket card groupsshowed improvements in scores for personalapplication and current use of EBM. The PDAgroup showed slightly higher scores in all fiveoutcomes, whereas those for the pocket cardgroup were not appreciably different fromthe previous rotation.

Technique: POCn = 169; I = 54; C/pocketcard = 55; C/nothing = 55

Frequency: single

Intervention group given PDA devices withclinical decision support tools; one controlgroup was given a pocket card containingguidelines and the other control groupreceived no intervention.

Country: China

Liaw S et al. 2008 Cluster randomized trial to determinethe effectiveness of locally adaptedpractice guidelines and educationabout paediatric asthma management,delivered to general practitioners usinginteractive, small group workshops

General practitioners Media: live vs print only Using interactive small group workshops todisseminate locally adapted guidelines wasassociated with improvement in generalpractitioners’ knowledge and confidence tomanage asthma compared to receivingguidelines alone in the control arm, but didnot change their self-reported provision ofwritten action plans.

n = 29, randomly assigned; I = 18,live, interactive plus guidelines;C/guidelines only = 18;C/nothing = 15

Technique: interactive vs reading

Country: Australia Frequency: single

Intervention group received live, interactivesessions plus guidelines; control groupsreceived guidelines only and no intervention.

Rabol L et al. 2010 Systematic review: 18 studies reviewedto determine outcomes of live,classroom-based, multi-professionalteam training

Health professionals Media: live Although most studies had weak designmethods, findings from the 18 studiesconcluded that team-based training led topositive participant evaluation, knowledgegain and behaviour change. However, theimpact on clinical outcomes was limited.

Technique: multiple

Frequency: single

Bluestoneet

al.Hum

anResources

forHealth

2013,11:51Page

17of

26http://w

ww.hum



Sulaiman N et al. 2010 Same study design as Liaw S et al. 2008for CPE intervention, but usedquestionnaires to determine any impacton completing written action plans orpatient outcomes

411 patient surveys frompatients of three arms utilizedin Liaw, S., et al. 2008 at baseline;341 at follow-up

See Liaw S et al. 2008 The interactive, small group workshops failedto translate into increased ownership ofwritten action plans, improved control ofasthma or improved quality of life, comparedto receiving guidelines alone or controlintervention.Country: Australia

Triola M et al. 2006 RCT to compare effectiveness of virtualpatients to live, standardized patientsfor improving clinical skills andknowledge

Health professionals Media: virtual patient vs live patient Improvements in diagnostic abilities wereequivalent in groups who experienced caseseither live or virtually. There was no subjectivedifference perceived by learners. Using virtualcases has the potential for cost efficiencies.

I = 23, C = 32 Technique: case-based


Intervention group received two live,standardized patient cases and two virtualpatient cases; control group received fourstandardized patient cases.

Turner M et al. 2006 Randomized, controlled, crossover trialto compare efficacy, studentpreference and cost of web-based,virtual patient vs live, standardizedpatient

2nd year medical students Media: virtual patient vs live patient There was no statistical difference in learningoutcomes between the web-based andstandardized patient; however, studentspreferred the standardized patient format.Start-up costs were comparable, but theongoing costs of the web-based format wereless expensive, suggesting that web-basedteaching may be a viable strategy.

I = 25, C = 24 Technique: case-based


Intervention group received web-basedinstruction for one topic, then standardizedpatient for another topic. This was reversedfor the second cohort, or control group,standardized patient first followed byweb-based instruction.

Wutoh R et al. 2004 Systematic review: 16 articles reviewedto determine the effect ofInternet-based CME interventions onphysician performance and healthcare outcomes

Physicians Media: Internet Results demonstrate that Internet-based CMEare just as effective in imparting knowledgeas traditional formats of CME. However, thereis a lack of quality studies to concludesignificant positive changes in practicebehaviour and additional studies are needed.

Technique: multiple


You J et al. 2009 Prospective, randomized study toinvestigate usefulness of video viamobile device as an instruction tool

Surgical residents Media: mobile videoconferencing/feedback The overall success rate for performing needlethoracocentesis was significantly higher forthe mobile phone video interventioncompared to the control group without aidedinstruction. Participants also rated the mobilephone intervention with significantly higherscores for instrument difficulty and proceduresatisfaction.

I = 24, C = 25 Technique: live, interactive with and withoutmobile POC feedback using video

Country: South KoreaFrequency: single

Both intervention groups had a didacticsession, performed a thoracentesis on amanikin while using video on a mobile phoneand received feedback from a live instructor;control group did not receive anyvideo-aided guidance.

C Control, CME Continuing medical education, CPR Cardiopulmonary resuscitation, EBM Evidence-based medicine, I Intervention, NR Not reported, PDA Personal digital assistant.POC Point-of-care, RCT Randomized controlled trial.

Bluestoneet

al.Hum

anResources

forHealth

2013,11:51Page

18of

26http://w

ww.hum


Table 7 Summary of articles focused on outcomes: knowledge, attitudes, types of skills, practice behaviour, clinical practice outcomes


Alvarez M and Agra Y 2006 Systematic review: 18 articles reviewed todetermine educational interventions inpalliative care and their impact onpractice behaviours

Physicians and other alliedhealth professionals

Practice behaviours Due to a lack of quality studies, there areinsufficient data to conclude about the impactof palliative care interventions on primary carephysician practice performance. Althoughimprovements in knowledge, some attitudesand provider satisfaction were demonstrated,there were no significant effects reported onpractice behaviours. Didactic education alonewas found to be ineffective. Interventionsinvolving multiple techniques, reminders andfeedback were found to be more effective atchanging behaviours.

Technique: multiple

Media: live

Frequency: both singleand multiple

Berkhof M et al. 2010 Systematic review: 12 systematic reviewsreviewed to determine effectiveeducational techniques to teachcommunication to physicians

Physicians Communication skills Sufficient evidence from 12 systematic reviewsto recommend training programmes last atleast 1 day, are learner-centred and focus onpracticing skills. The best training strategieswithin the programmes included role-play,feedback and small group discussions. Trainingprogrammes should include active,practice-oriented strategies. Oral presentationson communication skills, modeling and writteninformation should only be used as supportivestrategies.

Technique: multiple

Media: multiple


Bloom B 2005 Systematic review: 26 articles (all systemicreviews or meta-analyses) reviewed toexamine effectiveness of current CMEtools and techniques in changingphysician clinical practices andimproving patient health outcomes

Physicians Practice behaviours andclinical practice outcomes

Sufficient evidence to conclude that interactivetechniques (audit/feedback, academic detailing/outreach, reminders) are the most effectiveCME methods impacting practice outcomesand behaviours, while clinical guidelines andopinion leaders are less effective. Didacticpresentations and distributing printedinformation had little to no effect on physicianpractice.

Technique: multiple

Media: multiple


Bordage G et al. 2009a Systematic review: 29 articles reviewed todetermine if CME leads to an increase inphysician knowledge

Physicians and health professionals Knowledge Despite low quality of evidence presented inthe literature, there is sufficient evidence toconfirm an increase in physician knowledgewith the use of multimedia, multipleinstructional techniques and multipleexposures in CME.

Technique: multiple

Media: multiple


Davis D and Galbraith R 2009a Systematic review: 105 studies reviewedto determine impact of CME on practicebehaviours

Health professionals Practice behaviours Sufficient evidence to support the use of single,live or multimedia and multiple educationaltechniques as effective CME methods in changingphysician performance. Recommend multipleexposures over single exposures.

Technique: multiple

Media: live


Bluestoneet

al.Hum

anResources

forHealth

2013,11:51Page

19of

26http://w

ww.hum


Table 7 Summary of articles focused on outcomes: knowledge, attitudes, types of skills, practice behaviour, clinical practice outcomes (Continued)

Forsetlund L et al. 2009 Systematic review: 81 articles reviewed todetermine the effect of educationalmeetings on practice behaviours andclinical practice outcomes

More than 11,000 healthprofessionals

Practice behaviours andclinical practice outcomes

Sufficient evidence to conclude that educationalmeetings alone or combined with otherinterventions can have a small improvement onprofessional practice and health care outcomes,but no effect on changing complex behaviours.Previous reviews found that interactive workshopsresulted in moderate improvements, whereasdidactic sessions did not.

Technique: multiple

Media: live

Frequency: single

Gysels M et al. 2005 Systematic review: 16 articles reviewed toevaluate effective educational techniquesfor teaching communication skills

Health professionals Communication skills Sufficient evidence to recommendcommunication training programmes that arelearner-centred, carried out over a long periodof time, and combine didactic theoreticalcomponents with practical rehearsal andconstructive feedback.

Technique: multiple

Media: multiple


Hamilton R 2005 Systematic review: 24 articles reviewed todetermine how to enhance retention ofknowledge and skills during and afterresuscitation training

Health professionals Knowledge, skills Sufficient evidence to recommend in-hospitalsimulation to teach resuscitation training fornurses in clinical areas in addition to remedialtraining and the availability of resuscitationequipment for self-study. Video self-instructionhas been shown to improve competence inresuscitation.

Technique: multiple

Media: multiple


Marinopoulos S et al. 2007a Systematic review: from 68,000 citations,136 studies and nine systematic reviewswere identified and reviewed

Health professionals and alliedhealth professionals

Knowledge, skills, practicebehaviours and clinicalpractice outcomes

Firm conclusions are not possible due to theoverall low quality of the literature. Despitethis, the literature supported the concept thatCME was effective at the acquisition andretention of knowledge, attitudes, skills,behaviours and clinical outcomes. Commonthemes included that live media was moreeffective than print, multimedia was moreeffective than single media interventions,multiple exposures were more effective than asingle exposure, and simulation methods areeffective in the dissemination of psychomotorand procedural skills.

Technique: multiple

Media: live


Mansouri M and Lockyer J 2007 Meta-analysis: 31 studies reviewed todetermine the impact of CME onknowledge, skills and clinical practiceoutcomes

Mostly physicians Knowledge, skills, practicebehaviours and clinicalpractice outcomes

Sufficient information to conclude that theimpact of CME on physician performance andpatient outcome is small, but has a mediumeffect on knowledge and a larger effect whenthe interventions are interactive, use multiplemethods and are designed for a small groupof physicians from a single discipline.

Technique: multiple

Media: live


Bluestoneet

al.Hum

anResources

forHealth

2013,11:51Page

20of

26http://w

ww.hum



Mazmanian P et al. 2009a Systematic review: 37 articles reviewed todetermine the impact of CME on clinicalpractice outcomes

Physicians, nurse-practitioners,nurses, allied health professionals

Clinical practice outcomes Due to low quality of evidence, there is no firmconclusion on the impact of CME on clinicalpractice outcomes; however, multimedia,multiple techniques of instruction and multipleexposures to content are suggested to meetinstructional objectives intended to improveclinical outcomes.

Technique: multiple

Media: multiple


Moores L et al. 2009a Systematic review: 136 articles and ninesystematic reviews were reviewed toevaluate what makes CME effective

Health professionals General Significant evidence to support the use of CMEinterventions that use multimedia ininstruction, multiple instruction techniques andfrequency of exposure, to have a positive effecton knowledge, psychomotor skills, practiceperformance and clinical outcomes. The use ofprint media alone is not recommended.

Technique: multiple

Media: multiple


Nestel et al. 2011 Systematic review: 81 articles retrieved tosummarize the best evidence related touse of simulation for learning

Health professionals Psychomotor skills Sufficient evidence is available to conclude thatuse of simulation leads to improved knowledgeand skill. Studies with low-quality evidencesuggest a transfer of skills to the clinical setting.Instructional design and educational theory,contextualization, transferability, accessibility andscalability must all be considered in simulation-based education programmes.

Technique: multiple

Media: multiple


O’Neil K et al. 2009a Systematic review: from the 136 studiesidentified in the systematic review, 15articles, 12 addressing physicianapplication of knowledge and threeaddressing psychomotor skills, wereidentified and reviewed


Knowledge, psychomotor skills Sufficient evidence to support CME as effectivein improving physician application ofknowledge. Multiple exposures and longerdurations of CME are recommended to optimizeeducational outcomes. Quality of evidence doesnot address to psychomotor skill development.

Technique: multiple

Media: multiple


Rampatige R et al. 2009 Systematic review: 476 articles selectedfor inclusion. Section A relates to CPE ingeneral (sections B and C are not relevant);nine studies were reviewed to determineeffect of CME on practice behaviours andclinical practice outcomes

Health professionals Practice behaviours andclinical practice outcomes

Interactive and practice enabling strategies aremore useful than print-based and educationalmeetings. Multiple education efforts combinedwith good feedback/interaction betweeneducators and learners are most effective.Opinion leaders and outreach visits shown tobe effective.

Technique: multiple

Media: multiple


Raza A et al. 2009 (follow-upstudy to Coomarasamy A andKhan KS 2004)

Systematic review: Cochrane review of 36studies reviewed to determine evidenceto support effective CME

Health professionals General Evidence from 16 randomized trials supportinteractive and clinically integrated learningsessions and interactive classroom teaching assecond choice for an effective form of CME.Review demonstrated that interactiveworkshops improved knowledge and practicebehaviours.

Technique: multiple

Media: multiple


Bluestoneet

al.Hum

anResources

forHealth

2013,11:51Page

21of

26http://w

ww.hum



Satterlee W et al. 2008 Systematic review: nine articles reviewedto determine impact of CME on clinicalpractice outcomes

Health professionals Clinical practice outcomes Combined didactic presentations andinteractive workshops and combined didacticpresentations were more effective thantraditional didactic presentations alone. The useof multiple interventions over an extendedperiod increased effectiveness. Targetededucation should focus on changing abehaviour that is simple, since effect size isinversely proportional to the complexity of thebehaviour.

Technique: multiple

Media: multiple


Thomson O’Brien MA et al. 2001 Systematic review: 32 articles reviewed todetermine the effect of educationalmeetings on practice behaviours andclinical practice outcomes

Health professionals Practice behavioursand clinicalpractice outcomes

Moderate data quality suggests interactiveworkshops can result in moderately largechanges in professional practice. Didacticsessions alone are unlikely to changeprofessional practice.n = 2,995 Technique: multiple

Media: live

Frequency: single

Williams J et al. 2008 Systematic review: nine studies reviewedto evaluate if disaster training improvesknowledge and skills


Knowledge, skills Insufficient data quality exists to report on theimpact of disaster response traininginterventions on knowledge and skills. Datasuggest that both computer-based and liveinstruction may increase knowledge.

Technique: multiple

Media: multiple


aJHU EPC systematic review. CME Continuing medical education, JHU EPC Johns Hopkins University Evidence-Based Practice Center.

Bluestoneet

al.Hum

anResources

forHealth

2013,11:51Page

22of

26http://w

ww.hum



live instruction for addressing knowledge, while multiplerepetitive exposures leads to better knowledge gains than asingle exposure. Games can also contribute to knowledge ifdesigned as interactive learning experiences that stimulatehigher thinking through analysis, synthesis or evaluation.No studies or systematic reviews looked only at attitudes,

but CPE that includes clinical integration, simulations andfeedback may help address attitudes. The JHU EPC groupsystematic review evaluation of the short- and long-termeffects of CPE on physician attitudes reviewed 26 studiesand, despite the heterogeneity of the studies, identifiedtrends supporting the use of multimedia and multipleexposures for addressing attitudes [6].Several systematic reviews looked specifically at skills,

concluding that there is weak but sufficient evidence tosuggest that psychomotor skills can be addressed withCPE interventions that include simulations, practice withfeedback and/or clinical integration. ‘Dose-response’ orproviding sufficient practice and feedback was identified asimportant for skill-related outcomes. Other RCTs suggestclinically integrated education for supporting skill develop-ment. Choa et al. found that neither the audio mentoringvia mobile nor animated graphics via mobile resulted in thedesired psychomotor skills, reinforcing the need for practiceand feedback for psychomotor skill development identifiedin other studies [49].Two systematic reviews focused on communication skills

and found techniques that include behaviour modeling,practice and feedback, longer duration or more practice op-portunities were more effective [52,53]. Evidence suggeststhat development of communication skills requires inter-active techniques that include practice-oriented strategiesand feedback, and limit lecture and print-based materials tosupportive strategies only.Findings also suggest that simulation, PBL, multiple

exposures and clinically integrated CPE can improvecritical thinking skills. Mobile-based POC support wasfound to be more useful in the development of criticalthinking than print-based job aids.Several systematic reviews specifically looked at CPE,

practice behaviours and the behaviours of the provider.These studies found, despite reportedly weak evidence, thatinteractive techniques that involved feedback, interactionwith the educator, longer durations, multiple exposures,multimedia, multiple techniques and reminders mayinfluence practice behaviours.A targeted review of 37 articles from the JHU EPC review

on the impact of CPE on clinical practice outcomes drewno firm conclusions, but multiple exposures, multimediaand multiple techniques were recommended to improvepotential outcomes [6]. Interaction and feedback werefound to be more useful than print or educational meetings(systematic review of nine articles) [24], but print-basedunsolicited materials were not found to be effective [50].

The systematic review of live, classroom-based, multi-professional training conducted by Rabal et al. found‘the impact on clinical outcomes is limited’ [54].

DiscussionThe heterogeneity of study designs included in this reviewlimits the interpretations that can be drawn. However, thereis remarkable similarity between the information from stud-ies included in this review and similar discussions publishedin the educational psychology literature. We believe thatthere is sufficient evidence to support efforts to implementand evaluate the combinations of training techniques, fre-quency, settings and media included in this discussion.Avoid educational techniques that provide a passive

transfer of information, such as lecture and reading, andselect techniques that engage the learner in mentalprocessing, for example, case studies, simulation and otherinteractive strategies. This recommendation is reinforcedin educational psychology literature [55]. There is sufficientevidence to endorse the use of simulation as a preferrededucational technique, notably for psychomotor, com-munication or critical thinking skills. Given the lack ofevidence for didactic methods, selecting interactive, effect-ive educational techniques remains the critical point toconsider when designing CPE interventions.Self-directed learning was also found to be an effective

strategy, but requires the use of interactive techniquesthat engage the learner. Self-directed learning has theadditional advantage of allowing learners to study attheir own pace, select times convenient for them and tailorlearning to their specific needs.Limited evidence was found to support team-based

learning or the provision of training in work teams. Thereis a need for further study in this area, given the value ofengaging teams that are in the same place at the sametime in an in-service training intervention. This finding isespecially relevant for emergency skills that require thecollaboration and cooperation of a team.Repetitive exposure is supported in the literature. When

possible, replace single-event frequency with targeted,repetitive training that provides reinforcement of importantmessages, opportunities to practice skills and mechanismsfor fostering interaction. Recommendations drawn fromthe educational psychology literature that address the issueof cognitive overload [56] suggest targeting information toessentials and repetition.Select the setting based on its ability to deliver effective

educational techniques, be similar to the work environ-ment and allow for practice and feedback. In this time ofcrisis, workplace learning that reduces absenteeism andsupports individualized learning is critical. Conclusionsfrom literature in educational psychology reinforce theimportance of ‘situating’ learning to make the experienceas similar to the workplace as possible [57].


Certain common themes emerged from the many articlesthat commented on the role of media in CPE effectiveness.A number of systematic reviews suggest the use of multi-media in CPE. It is important to note that the studies thatfound similar knowledge outcomes between computer-based and live instruction stated that both utilized inter-active techniques, possibly indicating the effectiveness wasdue to the technique rather than the media through whichit was delivered. While the data on use of mobile technol-ogy to deliver CPE were limited, the study by Zurovac et al.indicated the potential power of mobile technology toimprove provider adherence to clinical protocols [29].Currently, there is unprecedented access to basic mobiletechnology and increasing access to lower-cost tablets andcomputers. The use of these devices to deliver effectivetechniques warrants exploration and evaluation, particularlyin low- and middle-income countries.CPE can positively impact desired learning outcomes

if effective techniques are used. There are, however, verylimited and weak data that directly link CPE to improvedclinical practice outcomes. There are also limited data thatlink CPE to improved clinical practice behaviours, whichmay influence the strength of the linkage to outcomes.

LimitationsThe following limitations apply to the methodology thatwe selected for this study. An integrative review of theliterature was selected because the majority of publishedstudies of education and training in low- and middle-resource countries did not meet the parameters requiredof a more rigorous systematic review or meta-analysis.The major limitation of integrative reviews is the potentialfor bias from their inclusion of non-peer-reviewed infor-mation or lower-quality studies. The inclusion of articlesrepresenting a range of rigor in their research designrestricts the degree of confidence that can be placed oninterpretations drawn by the authors of those articles,with the exception of original articles that explicitlydiscussed quality (such as systematic reviews). This reviewdid not make an additional attempt to reanalyse orcombine primary data.Therefore, for purpose of this article, we also graded

all articles and included only tier 1 articles in the analysis.This resulted in restriction of information on certain topicsfor this report, although a wider range of informationis available.We faced an additional limitation in that many articles

included in the review were neither fully transparent norconsistent with terminology definitions used in other re-ports. This is due in part to the fact that we went beyondthe bio-medical literature, to include studies conducted inthe education and educational psychology literature, as wasappropriate to the integrative review methodology. Certaintopics were underdeveloped in the literature, which limits

the interpretation that can be drawn on these topics. Othertopics are addressed in studies conducted using lower-tierresearch methodologies (for example observational and/orqualitative studies) that were not included in this article.In addition, the overwhelming majority of studies focusedon health professionals in developed or middle-incomecountries. There were very few articles of sufficient rigorconducted in low- and middle-income countries. Thislimits what we can say regarding the application of thesefindings among health workers of a lower educationallevel and in lower-resourced communities.

ConclusionsIn-service training has been and will remain a significantinvestment in developing and maintaining essential com-petencies required for optimal public health in all globalservice settings. Regrettably, in spite of major investments,we have limited evidence about the effectiveness of thetechniques commonly applied across countries, regardlessof level of resource.Nevertheless, all in-service training, wherever delivered,

must be evidence-based. As stated in Bloom’s systematic re-view, ‘Didactic techniques and providing printed materialsalone clustered in the range of no to low effects, whereasall interactive programmes exhibited mostly moderateto high beneficial effect. … The most commonly usedtechniques, thus, generally were found to have the leastbenefit’ [14]. The profusion of mobile technology andincreased access to technology present an opportunity todeliver in-service training in many new ways. Given currentgaps in high-quality evidence from low- and middle-income countries, the future educational research agendamust include well-constructed evaluations of effective,cost-effective and culturally appropriate combinations oftechnique, setting, frequency and media, developed forand tested among all levels of health workers in low- andmiddle-income countries.

AbbreviationsBEME: Best Evidence in Medical Education; CI: Confidence interval;CINAHL: Cumulative Index to Nursing and Allied Health Literature;CME: Continuing medical education; CPE: Continuing professional education;CPR: Cardiopulmonary resuscitation; EBM: Evidence-based medicine;JHU EPC: Johns Hopkins University Evidence-Based Practice Center;MeSH: Medical subject headings; OCEMB: Oxford Centre for Evidence-BasedMedicine; PBL: Problem-based learning; POC: Point-of-care; RCT: Randomizedcontrolled trial.

Competing interestsThe authors declare they have no competing interests.

Authors’ contributionsJB performed article reviews for inclusion, synthesized data and served asprimary author of the analysis and manuscript. PJ conceived the study,participated in its design and coordination, and provided significant inputinto the manuscript. JF provided guidance on the literature review process,grading and categorizing criteria, and quality review of selected articles, andparticipated actively as an author of the manuscript. CC and JBT contributedto writing of the manuscript. JA searched the literature, performed initial


review and coding, and contributed to selected sections of the manuscript.All authors read and approved the final manuscript.

AcknowledgmentsWe thank the Jhpiego Corporation for support for this research. We thankDana Lewison, Alisha Horowitz, Rachel Rivas D’Agostino and Trudy Conleyfor their support in editing and formatting the manuscript. We also thankSpyridon S Marinopoulos, MD, MBA, from the Johns Hopkins UniversitySchool of Medicine, for his initial input into the study and links to relevantresources. The findings, interpretations and conclusions expressed in thispaper are those of the authors and not necessarily those of the JhpiegoCorporation.

Author details1Jhpiego Corporation, 1615 Thames Street, Baltimore, MD 21231, USA.2Independent Consultant, San Diego, CA, USA. 3Research Assistant, Baltimore,MD, USA.

Received: 8 August 2012 Accepted: 2 May 2013Published: 1 October 2013

References1. Frenk J, Chen L, Bhutta ZA, Cohen J, Crisp N, Evans T, Fineberg H, Garcia P,

Ke Y, Kelley P, Kistnasamy B, Meleis A, Naylor D, Pablos-Mendez A, Reddy S,Scrimshaw S, Sepulveda J, Serwadda D, Zurayk H: Health professionals fora new century: transforming education to strengthen health systems inan interdependent world. Lancet 2010, 376(9756):1923–1958.

2. Wittemore R, Kanfl K: The integrative review: updated methodology.In J Adv Nursing 2005, 52(5):546–553.

3. Marinopoulos SS, Baumann MH: American College of Chest PhysiciansHealth and Science Policy Committee: Methods and definition of terms:effectiveness of continuing medical education: American College ofChest Physicians Evidence-Based Educational Guidelines. Chest 2009,135(3 Suppl):17S–28S.

4. Oxford Centre for Evidence-Based Medicine (OCEBM) Levels of EvidenceWorking Group: The Oxford 2011 Levels of Evidence. Oxford: Centre forEvidence Based Medicine; 2011. http://www.cebm.net/?O=1025.

5. Best Evidence in Medical Education (BEME): Appendix 1-BEME Coding Sheet;2011. http://www.medicalteacher.org/MEDTEACH_wip/supp%20files/BEME%204%20Figs%20&%20Appendices/BEME4_Appx1.pdf.

6. Marinopoulos SS, Dorman T, Ratanawongsa N, Wilson LM, Ashar BH,Magaziner JL, Miller RG, Thomas PA, Prokopowicz GP, Qayyum R, Bass EB:Effectiveness of continuing medical education. Evidence Report/TechnologyAssessment, No. 149, (Prepared by the Johns Hopkins Evidence-based PracticeCenter, under Contract No.290-02-0018) AHRQ Publication No. -7-E006. Agencyfor Healthcare Research and Quality: Rockville, MD; January 2007.

7. Triola M, Feldman H, Kalet AL, Zabar S, Kachur EK, Gillespie C, Anderson M,Griesser C, Lipkin M: A randomized trial of teaching clinical skills usingvirtual and live standardized patients. J Gen Intern Med 2006, 21(5):424–429.

8. Smits PB, De Buisonje CD, Verbeek JH, Van Dijk FJ, Metz JC, Ten Cate OJ:Problem-based learning versus lecture-based learning in postgraduatemedical education. Scand J Work Environ Health 2003, 29(4):280–287.

9. White M, Michaud G, Pachev G, Lirenman D, Kolenc A, FitzGerald JM:Randomized trial of problem-based versus didactic seminars fordisseminating evidence-based guidelines on asthma management toprimary care physicians. J ContinEduc Health Prof 2004, 24(4):237–243.

10. De Lorenzo RA, Abbott CA: Effectiveness of an adult-learning, self-directedmodel compared with traditional lecture-based teaching methods inout-of-hospital training. Acad Emerg Med 2004, 11(1):33–37.

11. Lin CF, Lu MS, Chung CC, Yang CM: A comparison of problem-basedlearning and conventional teaching in nursing ethics education.Nurs Ethics 2010, 17(3):373–382.

12. Reynolds A, Ayres-de-Campos D, Pereira-Cavaleiro A, Ferreira-Bastos L:Simulation for teaching normal delivery and shoulder dystocia tomidwives in training. Educ Health (Abingdon) 2010, 23(3):405.

13. Alvarez MP, Agra Y: Systematic review of educational interventions inpalliative care for primary care physicians. Palliat Med 2006, 20(7):673–683.

14. Bloom BS: Effects of continuing medical education on improvingphysician clinical care and patient health: a review of systematicreviews. Int J Technol Assess Health Care 2005, 21(3):380–385.

15. Satterlee WG, Eggers RG, Grimes DA: Effective medical education: insightsfrom the Cochrane Library. Obstet Gynecol Surv 2008, 63(5):329–333.

16. Herbert CP, Wright JM, Maclure M, Wakefield J, Dormuth C, Brett-MacLean P,Legare J, Premi J: Better Prescribing Project: a randomized controlled trialof the impact of case-based educational modules and personalprescribing feedback on prescribing for hypertension in primary care.Fam Pract 2004, 21(5):575–581.

17. Issenberg SB, McGaghie WC, Petrusa ER, Lee Gordon D, Scalese RJ: Featuresand uses of high-fidelity medical simulations that lead to effectivelearning: a BEME systematic review. Med Teach 2005, 27(1):10–28.

18. Raza A, Coomarasamy A, Khan KS: Best evidence continuous medicaleducation. Arch Gynecol Obstet 2009, 280(4):683–687.

19. Akl EA, Pretorius RW, Sackett K, Erdley WS, Bhoopathi PS, Alfarah Z,Schunemann HJ: The effect of educational games on medical students’learning outcomes: a systematic review: BEME Guide No 14. Med Teach2010, 32(1):16–27.

20. Laprise R, Thivierge R, Gosselin G, Bujas-Bobanovic M, Vandal S, Paquette D,Luneau M, Julien P, Goulet S, Desaulniers J, Maltais P: Improvedcardiovascular prevention using best CME practices: a randomized trial.J Contin Educ Health Prof 2009, 29(1):16–31.

21. Forsetlund L, Bjorndal A, Rashidian A, Jamtvedt G, O’Brien MA, Wolf F, DavisD, Odgaard-Jensen J, Oxman AD: Continuing education meetings andworkshops: effects on professional practice and health care outcomes.Cochrane Database Syst Rev 2009, 2, CD003030.

22. Mansouri M, Lockyer J: A meta-analysis of continuing medical educationeffectiveness. J Contin Educ Health Prof 2007, 27(1):6–15.

23. Perry M, Draskovic I, Lucassen P, Vernooij-Dassen M, Van Achterberg T,Rikkert MO: Effects of educational interventions on primary dementiacare: A systematic review. Int J Geriatr Psychiatr 2011, 26(1):1–11.

24. Rampatige R, Dunt D, Doyle C, Day S, Van Dort P: The effect of continuingprofessional education on health care outcomes: lessons for dementiacare. Int Psychogeriatr 2009, 21(Suppl 1):S34–S43.

25. Blaya JA, Fraser HS, Holt B: E-health technologies show promise indeveloping countries. Health Aff (Millwood) 2010, 29(2):244–251.

26. Leung GM, Johnston JM, Tin KY, Wong IO, Ho LM, Lam WW, Lam TH:Randomised controlled trial of clinical decision support tools to improvelearning of evidence-based medicine in medical students. BMJ 2003,327(7423):1,090.

27. You JS, Park S, Chung SP, Park JW: Usefulness of a mobile phone withvideo telephony in identifying the correct landmark for performingneedle thoracocentesis. Emerg Med J 2009, 26(3):177–179.

28. Yuan H, Williams BA, Fan L: A systematic review of selected evidence ondeveloping nursing students’ critical thinking through problem-basedlearning. Nurse Educ Today 2008, 28(6):657–663.

29. Zurovac D, Sudoi RK, Akhwale WS, Ndiritu M, Hamer DH, Rowe AK, SnowRW: The effect of mobile phone text-message reminders on Kenyanhealth workers’ adherence to malaria treatment guidelines: a clusterrandomised trial. Lancet 2011, 378(9793):795–803.

30. Murad MH, Coto-Yglesias F, Varkey P, Prokop LJ, Murad AL: Theeffectiveness of self-directed learning in health professions education:a systematic review. Med Educ 2010, 44(11):1,057–1,068.

31. Young JM, Ward J: Can distance learning improve smoking cessation advice infamily practice? A randomized trial. J Contin Educ Health Prof 2002, 22(2):84–93.

32. Harder BN: Use of simulation in teaching and learning in health sciences:a systematic review. J Nurs Educ 2010, 49(1):23–28.

33. McGaghie WC, Siddall VJ, Mazmanian PE, Myers J: American College ofChest Physicians Health and Science Policy Committee: Lessons forcontinuing medical education from simulation research inundergraduate and graduate medical education: effectiveness ofcontinuing medical education: American College of Chest PhysiciansEvidence-Based Educational Guidelines. Chest 2009, 135(3 Suppl):62S–68S.

34. Sturm LP, Windsor JA, Cosman PH, Cregan P, Hewett PJ, Maddern GJ:A systematic review of skills transfer after surgical simulation training.Ann Surg 2008, 248(2):166–179.

35. Steadman RH, Coates WC, Huang YM, Matevosian R, Larmon BR, McCulloughL, Ariel D: Simulation-based training is superior to problem-based learningfor the acquisition of critical assessment and management skills. Crit CareMed 2006, 34(1):151–157.

36. Lamb D: Could simulated emergency procedures practised in a staticenvironment improve the clinical performance of a Critical Care Air SupportTeam (CCAST)? A literature review. Intensive Crit Care Nurs 2007, 23(1):33–42.

http://www.cebm.net/?O=1025

http://www.medicalteacher.org/MEDTEACH_wip/supp%20files/BEME%204%20Figs%20&%20Appendices/BEME4_Appx1.pdf

http://www.medicalteacher.org/MEDTEACH_wip/supp%20files/BEME%204%20Figs%20&%20Appendices/BEME4_Appx1.pdf


37. Bruppacher HR, Alam SK, LeBlanc VR, Latter D, Naik VN, Savoldelli GL, MazerCD, Kurrek MM, Joo HS: Simulation-based training improves physicians’performance in patient care in high-stakes clinical setting of cardiacsurgery. Anesthesiology 2010, 112(4):985–992.

38. Daniels K, Arafeh J, Clark A, Waller S, Druzin M, Chueh J: Prospectiverandomized trial of simulation versus didactic teaching for obstetricalemergencies. Simul Healthc 2010, 5(1):40–45.

39. Ellis D, Crofts JF, Hunt LP, Read M, Fox R, James M: Hospital, simulationcenter, and teamwork training for eclampsia management: arandomized controlled trial. Obstet Gynecol 2008, 111(3):723–731.

40. Hamilton R: Nurse’s knowledge and skill retention followingcardiopulmonary resuscitation training: a review of the literature.J Adv Nurs 2005, 51(3):288–297.

41. Merien AE, Van de Ven J, Mol BW, Houterman S, Oei SG: Multidisciplinaryteam training in a simulation setting for acute obstetric emergencies:a systematic review. Obstet Gynecol 2010, 115(5):1,021–1,031.

42. Crofts JF, Ellis D, Draycott TJ, Winter C, Hunt LP, Akande VA: Change inknowledge of midwives and obstetricians following obstetric emergencytraining: a randomised controlled trial of local hospital, simulation centreand teamwork training. BJOG 2007, 114(12):1,534–1,541.

43. Kerfoot BP, Fu Y, Baker H, Connelly D, Ritchey ML, Genega EM: Online spacededucation generates transfer and improves long-term retention of diagnosticskills: a randomized controlled trial. J Am Coll Surg 2010, 211(3):331–337.

44. Kerfoot BP, Kearney MC, Connelly D, Ritchey ML: Interactive spacededucation to assess and improve knowledge of clinical practiceguidelines: a randomized controlled trial. Ann Surg 2009, 249(5):744–749.

45. Kerfoot BP, Baker HE, Koch MO, Connelly D, Joseph DB, Ritchey ML:Randomized, controlled trial of spaced education to urology residents inthe United States and Canada. J Urol 2007, 177(4):1,481–1,487.

46. Coomarasamy A, Khan KS: What is the evidence that postgraduateteaching in evidence based medicine changes anything? A systematicreview. BMJ 2004, 329(7473):1,017.

47. Harrington SS, Walker BL: The effects of computer-based training onimmediate and residual learning of nursing facility staff. J Contin EducNurs 2004, 35(4):154–163. quiz 186–7.

48. Curran VR, Fleet L: A review of evaluation outcomes of web-basedcontinuing medical education. Med Educ 2005, 39(6):561–567.

49. Choa M, Park I, Chung HS, Yoo SK, Shim H, Kim S: The effectiveness ofcardiopulmonary resuscitation instruction: animation versus dispatcherthrough a cellular phone. Resuscitation 2008, 77(1):87–94.

50. Farmer AP, Legare F, Turcot L, Grimshaw J, Harvey E, McGowan JL, Wolf F:Printed educational materials: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev 2008, 3, CD004398.

51. Liaw ST, Sulaiman ND, Barton CA, Chondros P, Harris CA, Sawyer S, DharmageSC: An interactive workshop plus locally adapted guidelines can improvegeneral practitioners asthma management and knowledge: a clusterrandomised trial in the Australian setting. BMC Fam Pract 2008, 9:22.

52. Gysels M, Richardson A, Higginson IJ: Communication training for healthprofessionals who care for patients with cancer: a systematic review oftraining methods. Support Care Canc 2005, 13(6):356–366.

53. Berkhof M, Van Rijssen HJ, Schellart AJ, Anema JR, Van der Beek AJ: Effectivetraining strategies for teaching communication skills to physicians: anoverview of systematic reviews. Patient Educ Couns 2011, 84(2):152–162.

54. Rabol LI, Ostergaard D, Mogensen T: Outcomes of classroom-based teamtraining interventions for multiprofessional hospital staff: A systematicreview. Qual Saf Health Care 2010, 19(6):e27.

55. Woolfolk AE: Educational Psychology. 11th edition. Upper Saddle River, NJ:Pearson/Merrill/Prentice Hall; 2009.

56. Mayer RE: Applying the science of learning to medical education.Med Educ 2010, 44(6):543–549.

57. Gott S, Lesgold A: Competence in the workplace: how cognitiveperformance models and situated instruction can accelerate skillacquisition. In Advances in Instructional Psychology, Volume 4. Edited byGlaser R. Hillsdale, NJ: Erlbaum; 2000.

doi:10.1186/1478-4491-11-51Cite this article as: Bluestone et al.: Effective in-service training designand delivery: evidence from an integrative literature review. HumanResources for Health 2013 11:51.

Submit your next manuscript to BioMed Centraland take full advantage of:

• Convenient online submission

• Thorough peer review

• No space constraints or color figure charges

• Immediate publication on acceptance

• Inclusion in PubMed, CAS, Scopus and Google Scholar

• Research which is freely available for redistribution

Submit your manuscript at www.biomedcentral.com/submit

effective in-service training design and delivery - human resources

Documents