using a delphi process to develop an effective train-the-trainers program to train health and social...

Journal of Traumatic StressJune 2012, 25, 337–343

BRIEF REPORT

Using a Delphi Process to Develop an Effective Train-the-TrainersProgram to Train Health and Social Care Professionals Throughout

Europe

Jennifer Pearce,1 Caryl Jones,1 Sinead Morrison,1 Miranda Olff,2 Susanne van Buschbach,2 Anke B.Witteveen,2 Richard Williams,3 Francisco Orengo-Garcıa,4 Dean Ajdukovic,5 A. Tamer Aker,6

Dag Nordanger,7 Brigitte Lueger-Schuster,8 and Jonathan I. Bisson9

1Department of Psychological Medicine, Cardiff University, Wales, United Kingdom2Department of Psychiatry, Academic Medical Center, University of Amsterdam, The Netherlands

3Professor of Mental Health Strategy, University of Glamorgan and Consultant Child and Adolescent Psychiatrist, Aneurin BevanHealth Board, Wales, United Kingdom

4Spanish Society of Psychotraumatology, Madrid, Spain5Department of Psychology, University of Zagreb, Zagreb, Croatia

6Department of Psychiatry & Centre for Psychological Trauma and Disaster Studies, Kocaeli University Medical School,Kocaeli, Turkey

7UNI Health, Centre for Child and Adolescent Mental Health–Western Norway, Bergen, Norway8Faculty of Psychology, University of Vienna, Vienna, Austria

9Cardiff University School of Medicine & Cardiff and Vale University Health Board, Wales, United Kingdom

Research has shown that developing a Train-the-Trainers (TTT) program is important if agencies are to implement guidelines, but the mosteffective way to deliver a TTT program remains unanswered. This article presents data from a 3-round Internet-based Delphi process,which was used to help develop consensus-based guidelines for a TTT programme to deliver to health and social care professionalsthroughout Europe a curriculum on traumatic stress. In Round 1, 74 experts rated the importance of statements relating to the TTT field andthen reassessed their scores in the light of others’ responses in subsequent rounds. Forty-one (67%) of 61 statements achieved consensus(defined as having a mean score >7 or < 3 on the 0–9 rating scales used and 70% of participants scoring 7 and above or 3 and below) forinclusion. Key TTT components included interactive and practical presentations, delivery to groups of 7–12 people over 2 days, externaland local expert facilitation, course manuals, refresher courses, and supervision. The Delphi process allowed a consensus to be achievedin an area in which there are limitations in the current evidence.

The effective dissemination and implementation of evidence-based guidelines is essential to deliver optimal psychosocial

Funding was from the European Union’s Executive Agency for Health andConsumers. We wish to thank all the participants in the Delphi process.

The statements agreed on through the Delphi process informed the developmentof a TTT manual that provides a step-by-step guide on how to organise andconduct TTT workshops orientated to the TENTS-TP curriculum. This manualis available from the authors on request.

Correspondence concerning this article should be addressed to Jonathan I.Bisson, Director of Research and Development, Cardiff University Schoolof Medicine & Cardiff and Vale University Health Board, 2nd Floor TB2,University Hospital of Wales, Heath Park Cardiff, CF14 4XW Wales, UnitedKingdom. E-mail: [email protected]

Copyright C© 2012 International Society for Traumatic Stress Studies. Viewthis article online at wileyonlinelibrary.comDOI: 10.1002/jts.21705

care following disasters and major incidents. Various trainingapproaches are used to implement guidelines from one-off di-dactic lectures and workshops to more complex attempts totransfer knowledge such as learning collaboratives. Previousresearch suggests that an interactive, multifaceted method mayprovide the most effective means of training professionals andimplementing guidelines (Prior, Guerin, & Grimmer-Somers,2008). Train-the-Trainer (TTT) refers to a program or coursethrough which practitioners receive training on a defined sub-ject and instruction on how to train, monitor, and superviseother professionals. A TTT approach to dissemination couldincorporate all of the identified components that are requiredto improve knowledge transfer, increase knowledge retention,and encourage compliance in a target population.

Despite studies that have used the TTT approach to trainhealth and social care professionals, there has been very little

337

338 Pearce et al.

Table 1Examples of Domains and Statements of Delphi Process

Domain Statement

Content of TENTS-TP TTT workshop The workshops should cover issues relating to human rights and rights-basedapproaches to humanitarian assistance.

Techniques to deliver TTT workshops The training should include case studies and scenarios.Course materials Trainers should be given a resource list for equipment that will be required to deliver

the subsequent workshops.Recruitment of TTT participants Participants on the TTT programme should have experience in training or teaching.Retention/commitment of TTT participants Managerial support should be obtained before professionals participate in the TTT

programme.Evaluation of trainers Trainers should be given written assessments in the training.Supervision, support, and website All trainers should have a named supervisor.

Note. TENTS-TP = The European Network for Traumatic Stress–Training and Practice; TTT = Train-the-Trainers.

research into the effectiveness of TTT programs. The authors,therefore, conducted a systematic review on the efficacy ofthe TTT approach (Pearce et al., 2011). Eighteen studies wereincluded in the review, 13 found that a TTT program helped toimprove clinical behaviours, increased knowledge, or resultedin better patient outcomes. It was, however, unclear how bestto deliver an effective TTT program.

To address this identified gap in the literature, we aimed toestablish a consensus, using a Delphi process, on the design ofan optimal TTT program to disseminate the European Networkfor Traumatic Stress–Training and Practice (TENTS-TP) cur-riculum (downloadable at http://www.tentsproject.eu/) to trainhealth and social care professionals.

Method

Delphi Process

The Delphi process was selected as a means of achieving con-sensus that has resulted in widely accepted outcomes, includingguidelines in the traumatic stress field (Bisson et al., 2010). Itrecognises the value of experts’ opinions, experience, and in-tuition when full scientific knowledge is lacking (Linstone &Turoff, 1975). A carefully selected group of experts answersurveys in two or more rounds. All participants who provide re-sponses to the first round are included in the subsequent rounds.After each round, a facilitator provides an anonymous summaryof the experts’ views and their comments, allowing all partici-pants to compare these with their own. The aim is that, duringthis iterative process, the range of answers decreases and thegroup converges towards the “correct” answer, without evi-dence of social conformism (Skulmoski, Hartman, & Krahn,2007).

Statement Development

Statements were developed based on components associatedwith positive outcomes in the systematic review of the efficacyof TTT programmes. These statements were then circulated to

the TENTS-TP group to gain agreement. Fifty-three statementswere created and categorised into domains (see Table 1 forexample statements in each domain).

A full list of the statements is available from the au-thors on request. The online survey tool Survey Monkey(www.surveymonkey.com) was used to present the statementswith clear instructions to indicate level of agreement or dis-agreement using a 9-point scale in which 1 = completely dis-agree, 9 = completely agree, and 5 = neutral. Participants werealso asked to provide comments regarding their score for eachstatement and suggest new statements if they felt importantissues were not covered. Representative comments from eachstatement were listed in subsequent rounds in order for par-ticipants to consider other raters’ perspectives. Some multiple-choice questions were also included.

In the second round, the survey comprised the same state-ments with summary statistics indicating the number and per-centage of participants who rated each score and the mean scorefor each statement. An example is given in Figure 1. Participantswere also sent a list of the full comments and their own scores.Some statements were amended slightly to improve compre-hension and eight new statements, suggested by participantsduring Round 1, were added.

Any statements that achieved a positive consensus (meanscore >7 or <3 and 70% of participants scoring 7 and aboveor 3 and below) were removed for Round 3 along with anystatements that were unlikely to achieve a consensus (thosewith a mean score of <6 and > 4). Statements that receivedscores between 6 and 7, and 3 and 4, were retained to determinewhether consensus could be achieved.

Participants

European Network for Traumatic Stress–Training and Practicepartners from nine different health and academic organisationsthroughout the European Union, all of whom are authors ofthis article, identified 219 potential participants through theirknowledge of individuals recognised as having expertise in the

Journal of Traumatic Stress DOI: 10.1002/jts. Published on behalf of the International Society for Traumatic Stress Studies.

Delphi Process 339

9.a) Participants on the TTT programme should be experienced trauma specialists

Round 1 comments:

We just would not get the numbers even if we thought that was a good criterion.

There is contradiction in this question - if they are experienced trauma specialist, what new can the TENTS-TP add to their expertise. Though, they may benefit from a trauma area they are not specialised in, but they are not likely to get involved in further training on such a topic because it is beyond their expertise.

Too limiting - include broader base of those who can use the knowledge

% 1 2 3 4 5 6 7 8 9

4.2 2.8 6.9 1.4 19.4 9.7 18.1 13.9 23.6

Rate (First round average = 6.42)

Additional comment:

Figure 1. A statement example from Round 2, which replicated the format from Round 1, but included summary statistics indicating the percentage of ratings foreach category, an overall mean for each statement, and sample comments from Round 1. Resized for illustrative purposes.

field of training and trauma. Participants included researchers,clinicians, emergency planners, trainers, and persons who hadsuffered from posttraumatic stress disorder. Ten of the 13 au-thors participated in the Delphi process. The aim was to invitekey people from countries throughout Europe and internation-ally recognised experts to participate in the survey. Care wastaken to identify a group of individuals considered represen-tative of all trainers of psychosocial care following traumaticevents.

Results

Round 1

Two-hundred nineteen participants were invited to take part, ofwhom 74 (34%) completed the first round of the survey. Thirty(57%) of the original 53 statements achieved good, positiveconsensus in Round 1 and eight new statements were formu-lated from respondents’ suggestions (see Table 2 for a list ofmeans and standard deviations for the statements that achievedconsensus in Round 1). Seven (13%) statements were amendedslightly to clarify their meaning as a result of participants’ com-ments.

Round 2

Round 2 was sent to the 74 people who had responded toRound 1 and 69 (93%) completed it. Thirty-six (59%) state-ments achieved positive consensus, 17 (27.9%) were in the 6–7range, and eight (13.1%) achieved a mean score between 4 and6 (see Table 3 for a list of means and standard deviations forstatements in Round 2). No statements achieved an averagerating of less than 4. One (12.5%) of the eight new statementsachieved positive consensus with a rating of 7.33 (SD = 1.44).

Table 2Means and Standard Deviations for Statements That AchievedConsensus in Round 1

Statement M SD1.c 7.04 1.941.e 7.53 1.811.h 7.39 1.452.a 8.34 0.962.c 7.37 1.612.d 7.50 1.542.e 8.34 0.872.f 7.96 1.672.g 7.08 2.282.h 7.97 1.532.m 8.08 1.242.n 8.10 1.494.a 8.37 1.304.b 7.42 1.994.c 7.10 2.144.d 7.62 1.504.e 7.59 1.499.b 8.18 1.039.c 7.19 1.789.d 7.23 1.739.e 7.71 1.7110.a 7.56 1.7610.b 7.51 1.6911.a 7.16 1.6211.d 7.76 1.4212.a 8.23 1.0512.b 8.31 1.05


340 Pearce et al.

Table 3Means and Standard Deviations for Statements that AchievedConsensus in Round 2

Statement M SD

1.a 5.15 2.401.b 6.68 2.101.c 7.40 1.051.d 6.78 1.711.e 8.11 1.231.f 6.33 2.541.g 6.01 1.741.h 5.68 2.051.I 7.32 1.422.a 8.64 0.752.b 7.58 1.652.c 8.01 1.282.d 8.01 1.122.e 6.94 2.112.f 8.33 1.222.g 8.02 1.562.h 7.31 1.902.I 8.23 1.302.j 7.41 1.842.k 6.26 2.202.l 6.73 1.962.m 6.23 2.482.n 7.97 1.522.o 8.57 0.854.a 8.75 0.724.b 8.10 1.354.c 7.93 1.514.d 7.46 1.954.e 8.32 1.049.a 6.45 2.059.b 4.69 2.499.c 8.55 0.879.d 7.62 1.609.e 7.55 1.529.f 8.04 1.459.g 5.83 1.559.h 4.04 2.0310.a 8.39 0.8810.b 8.09 1.4110.c 6.84 2.1611.a 7.25 1.5011.b 6.13 1.8811.c 6.68 1.5811.d 7.33 1.6611.e 8.12 1.2511.f 6.36 2.0111.g 6.94 1.5011.h 6.24 1.83

(Continued)

Table 3Continued

Statement M SD

11.I 4.25 1.9412.a 8.46 1.0212.b 8.67 0.7812.c 7.90 1.5412.e 5.55 1.8112.f 5.62 2.09

Round 3

Round 3 included 21 of the 61 Round 2 statements. It was sentto the 69 people who responded to Round 2 and 67 (97%) com-pleted it. Five of the 21 statements gained consensus. As a result,we deemed 41 (67%) of the 61 statements to be applicable todesigning the TENTS-TP TTT program.

The three statements with the highest average rating were“The training should include case studies and scenarios” (M =8.64, SD = 0.75); “A website should contain all course mate-rials” (M = 8.73, SD = 0.66), and “Trainers should be given acourse manual” (M = 8.75, SD = 0.72). The multiple-choicestatement with the highest percentage was “The TTT Programshould be delivered to health and social care professionals once,with additional refresher sessions” (96.7%). The three state-ments that obtained the lowest average ratings were “Trainingsessions should be tape recorded and later rated by a seniortrainer” (M = 3.73, SD = 1.6); “Participant places on the TTTprogram should be advertised on a first come first served basis”(M = 4.04, SD = 2.03), and “Participants on the TTT programshould be excluded if they have suffered a severe trauma withinthe last 12 months” (M = 4.15, SD = 2). A full list of theincluded and excluded items is presented in the Appendix.

Discussion

The Delphi process allowed us to develop a consensus on 41items that should be included in the TENTS-TP TTT programby an independent group of 74 experts from 20 different coun-tries. These items are likely to be important in the developmentof a practical and effective approach to TTT program delivery.Our participants were able to challenge the views of their peersin an anonymous and culturally sensitive manner, and it wasapparent that many participants’ views changed to a degree asa result of the process. There were also statements where theconsensus remained low, which suggests that participants didnot adjust their responses to those items to achieve conformity.

The results from this Delphi process support the theory ofa flexible, blended learning approach (Pereira et al., 2007;Prior et al., 2008). Consistent with previous work (Farmeret al., 2008; Sholomskas et al., 2005), experts felt that acombination of both interactive and didactic teaching stylesshould be incorporated into the TTT workshops, and that


Delphi Process 341

future trainers should be provided with printed materials, web-site support, and supervision.

Our study did not support some of the components thathave previously been suggested to produce positive effects in-cluding opinion leaders (Flodgren et al., 2011) and feedbackon strengths and weaknesses (Jamtvedt, Young, Kristoffersen,O’Brien, & Oxman, 2006). It is probable that there is no singleoptimal design for a TTT program. Rather we conclude thateach program should be flexible and use different training tech-niques depending on the topics and target audiences to whichthey are delivered.

The Delphi process cannot develop high-level evidencethough it does avoid several important problems that may beencountered with nonsystematic consensus development, forexample, failure to fully consider the limited evidence avail-able, bias towards certain views and dominance of certain per-sons that often occurs in face-to-face group discussions. Thehigh retention rate of participants across the three rounds wasa major positive feature of this study.

It is not possible to be sure that the experts were truly rep-resentative and the addition of a second sample of expertswould have strengthened the methodology. A consensus ap-proach to the development of guidelines should never replace agenuinely evidence-based approach or restrict efforts to achievethis. When the existing evidence-base is sparse and varied, how-ever, it is important to draw on the opinions and knowledge ofexperts in the field.

This Delphi process has helped to inform our designing ofguidelines for disseminating and implementing our curriculum.The findings are unlikely to be restricted to the traumatic stressfield and should be broadly applicable to TTT programs in otherareas. The final TENTS-TP TTT program has the potential to besuccessfully used in a wide range of countries with adaptationsto suit different cultures.

Further research is needed. The best blend of techniques touse when delivering a TTT program should be researched infield trials/randomised controlled trials. This Delphi process,along with previous research in the field has shown that, atpresent, the optimal format for a TTT program is a blendedlearning approach, but careful consideration of participant se-lection, training content, and methods is required and shouldbe employed when tailoring a program to fit the specific needsand preexisting knowledge/skills of the target audience.

ReferencesBisson, J. I., Tavakoly, B., Witteveen, A. B., Ajdukovic, D., Jehel, L., Johansen,

V., Olff, M. (2010). TENTS Guidelines: Development of post-disaster psy-chosocial care guidelines through a Delphi process. British Journal of Psy-chiatry, 196, 69–74. doi:10.1192/bjp.bp.109.066266

Farmer, A., Legare, F., Turcot, L., Grimshaw, J., Harvey, E., McGowan, J., &Wolf, F. M. (2008). Printed educational materials: Effects on professionalpractice and health care outcomes. Cochrane Database of Systematic Re-views, 3, CD004398. doi:10.1002/14651858.CD004398.pub2

Flodgren, G., Parmelli, E., Doumit, G., Gattellari, M., O’Brien, M., Grimshaw,J., & Eccles, M. P. (2011). Local opinion leaders: Effects on professionalpractice and health care outcomes. Cochrane Database of Systematic Re-views, 8, CD000125. doi:10.1002/14651858.CD000125.pub4

Jamtvedt, G., Young, J., Kristoffersen, D., O’Brien, M., & Oxman, A. (2006).Audit and feedback: Effects on professional practice and health care out-comes. Cochrane Database of Systematic Reviews, 2, CD000259. doi:10.1002/14651858.CD000259.pub2

Linstone, H., & Turoff, M. (1975). The Delphi method: Techniques and appli-cations. Reading, MA: Addison-Wesley.

Pearce, J., Mann, M. K., Jones, C., van Buschbach, S., Olff, M., Bisson,J. I. (In Press). The most effective way of delivering a Train-The-Trainersprogramme: A systematic review. The Journal of Continuing Education inthe Health Professions.

Pereira, J., Pleguezuelos, E., Merı, A., Molina-Ros, A., Molina-Tomas, M. C.,& Masdeu, C. (2007). Effectiveness of using blended learning strategiesfor teaching and learning human anatomy. Medical Care, 41, 189–195.doi:10.1111/j.1365-2929.2006.02672.x

Prior, M., Guerin, M., & Grimmer-Somers, K. (2008). The effectiveness ofclinical guideline implementation strategies—a synthesis of systematic re-view findings. Journal of Evaluation in Clinical Practice, 14, 888–897.doi:10.1111/j.1365-2753.2008.01014.x

Sholomskas, D., Syracuse-Siewert, G., Rounsaville, B., Ball, S., Nuro, K., &Carroll, K. (2005). We don’t train in vain: A dissemination trial of threestrategies of training clinicians in cognitive–behavioral therapy. Journalof Consulting and Clinical Psychology, 73, 106–115. doi:10.1037/0022-006X.73.1.106

Skulmoski, G., Hartman, F., & Krahn, J. (2007). The Delphi method for grad-uate research. Journal of Information Technology Education, 6, 1–21.

Appendix

Full List of Included and Excluded Items

Included items:.

1. Content of TENTS-TP TTT Programme Workshopsa. The workshops should introduce trainers to the cur-

riculum but not cover it in detailb. The workshops should focus on the content of the cur-

riculumc. The workshops should focus on how to deliver the

curriculumd. The workshops should combine an introduction to the

curriculum with information on how to deliver the cur-riculum

e. The workshops should include information on howand where to participate in relevant inter-agency co-ordination

2. Techniques to Deliver TENTS-TP TTT ProgrammeWorkshopsa. The training should include case studies and scenariosb. The training should involve didactic presentationsc. The training should involve video presentationsd. The training should be based on a set of TENTS-TP

agreed PowerPoint slidee. Clear guidelines should be given on how to present

PowerPoint slides, facilitate role-plays etcf. The training should involve interactive componentsg. The training should include practical demonstrations

and exercisesh. The training should include role-playsi. The training should involve group discussionsj. The training should help to improve motivation and

change attitudes if required


342 Pearce et al.

k. The workshops should include questions about thetraining and it’s implementation and general comments

l. The training should be delivered in the participants’local language or, when this is not possible, be providedwith translation

3. Delivery of TENTS-TP TTT Programmea. The TTT programme should be delivered by:b. TENTS-TP partner plus various local experts from the

country4. TENTS-TP TTT Course Material

a. Trainers should be given a course manual (includinglecture notes, slides, case studies and a copy of theguidelines)

b. Trainers should be given relevant pamphlets andbrochures from local resource

c. Trainers should be given additional educational mate-rials for the staff they will subsequently be training

d. Trainers should be given tools for the self-assessmentof performance as a trainer

e. Trainers should be given a resource list for equipmentthat will be required to deliver the subsequent work-shops

5. Participants on the TTT programme should be trainedover:a. 2 days

6. Length of Subsequent, Locally Delivered Traininga. The length of the subsequent locally delivered training

course should be 2 days7. Frequency of TENTS-TP TTT Programme Delivery

a. The TTT Programme should be delivered to healthand social care professionals once, with additional re-fresher sessions for trainers

8. Number of Participants on TENTS-TP TTT Coursesa. The training should be delivered in medium groups

(7-12 people)9. Recruitment of TENTS-TP TTT Programme Partici-

pantsa. Participants on the TTT programme should be cultur-

ally sensitive and have basic knowledge about localcultural attitudes and practices and systems of socialsupport

b. Participants on the TTT programme should have expe-rience of training or teaching

c. Participants on the TTT programme should cover dif-ferent geographical regions of the country

d. Participants on the TTT programme should includeboth health and social care professionals

10. Retention and Commitment of TENTS-TP TTT Pro-gramme Participantsa. There should be a requirement that any professional

who agrees to take part in the TTT programme shouldsubsequently deliver at least one workshop after theyhave been trained

b. Managerial support should be obtained before profes-sionals participate in the TTT programme

11. Evaluation of TENTS-TP TTT Trainersa. Trainers should compile a folder of activities (e.g. feed-

back reports, course evaluations etc.) to demonstratetheir continuing implementation of the guidance whichcan be approved by programme leaders

b. Participants should be given a pre and post trainingquestionnaire to assess knowledge acquisition and re-tention

c. Staff who are trained by the new trainers should beasked for feedback or tested on their level of under-standing of the content of the course

12. Supervision, Support and Websitea. E-support should be provided to trainers via a websiteb. A website should contain all course materialsc. All trainers should have a named supervisord. Supervision should occur as requirede. Update emails should be sent to trainers: Once a quarter

Excluded items:.

a. The workshops should cover the whole TENTS-TP cur-riculum in detail

b. Participants on the TTT programme should be local opin-ion leaders (i.e. health or social care professional who’sopinions particularly influence others)

c. Participant places on the TTT programme should be ad-vertised on a first come first served basis

d. Supervision should occur via phonee. Supervision should occur via emailf. The workshops should include a session whereby trainers

are given instruction on how to teach people how to copewith work-related problems (e.g. how to handle stress inan emergency situation)

g. The workshops should cover issues relating to humanrights and rights-based approaches to humanitarian assis-tance

h. The workshops should cover legal issues (e.g. access toassistance, compensation, family tracing etc).

i. The training should involve individual feedback onstrengths and weaknesses

j. The training should be problem-based (instead of tradi-tional lecture-based learning. Problem-based learning in-volves students/participants collaboratively solving prob-lems and reflecting on their experiences)

k. The training should include a session on preparing partic-ipants to deliver future workshops (e.g. preparing actionplans, discussing obstacles to delivery, public speakingand presentation skills etc)

l. Participants on the TTT programme should be experi-enced trauma specialists

m. Participants on the TTT programme should be excludedif they have suffered a severe trauma within the last 12months


Delphi Process 343

n. Permission from managers to support the trainer to deliversubsequent workshops should be sought before the pro-fessional agrees to participate in the programme Manage-rial support after the training should be confirmed beforeprofessionals participate in the TTT programme)

o. Participants on the TTT workshop should be given writ-ten assessments in the training (for example, be given awritten exam to test their knowledge at the end of thetraining)

p. Participants should be given practical assessments in thetraining (for example, be given practical scenarios tosolve)

q. When trained trainers cascade the training to otherhealth and social care professionals in their lo-cal areas, their teaching abilities should be peerreviewed

r. Senior trainers should attend some of thetrained trainers’ subsequently delivered work-shops in order to observe and support them whenneeded

s. A TENTS-TP accreditation scheme should be introducedthat requires central monitoring and incentivises trainers

t. Training sessions should be tape recorded and later ratedby a senior trainer


using a delphi process to develop an effective train-the-trainers program to train health and social...

Documents