Burden of proof: the evidence clinicians requirebefore implementing an intervention

Brian Allen1 & Natalie E. Armstrong2

1Center for Safe and Healthy Families, Primary Children’s Medical Center, 675 East 500 South, Suite 300, Salt Lake City, UT,84102, USA. E-mail: brian.allen@imail.org2Department of Psychology, Sam Houston State University, Huntsvile, TX, USA

Background: Greater implementation of evidence-based practice for children and adolescents is a currentemphasis in the mental health field; however, there is a need to understand how best to disseminate theseinterventions and convince community clinicians to use them. Method: A sample of 255 clinicians reported onthe likelihood that they would use an intervention given various types of evidence. Results: Case studies andclinical trials with an active or placebo control group scored as the most preferred types of evidence; however,more positive attitudes toward evidence-based practice predicted preferences for clinical trials, but were notrelated to case studies. Conclusion: Implementation of evidence-based practice may be improved by greaterdissemination of case studies demonstrating the use of these interventions in ‘real world’ settings. In addition,fostering a greater appreciation of research-derived interventions among clinicians appears necessary.

Key Practitioner Message:

● Clinicians prefer case studies and clinical trials with an active/placebo control group over other dissemina-tion methods when selecting an intervention

● Clinicians with more favorable attitudes toward evidence-based practice were more likely to prefer clinicaltrials

● Clinicians showed a preference for case studies regardless of their attitudes toward evidence-based practice

● Clinicians endorsing a cognitive-behavioral orientation were less likely to use an intervention based solelyon colleague endorsement than either eclectic or psychodynamic/humanistic clinicians

● Using case studies as a dissemination method may significantly increase the implementation of evidence-based practice in clinical settings

Keywords: Evidence-based practice; dissemination; implementation; child treatment

Introduction

In recent years, focused attention has sought to deter-mine effective ways for disseminating and implementinginterventions for children and adolescents that aresupported by the results of rigorous research, oftentimescalled evidence-based practice (EBP). These efforts haveincluded examinations of clinician attitudes towardEBP (Aarons et al., 2010), the impact of organizationalculture (Glisson et al., 2008), and the development andevaluation of numerous training models (for a review seeDecker, Jameson & Naugle, 2011). In short, researchsuggests that clinician attitudes toward evidence-basedpractice and utilization of empirically supported treat-ment protocols can be significantly improved with train-ing, especially when organizational-level change isincluded as a goal of the training program (Beidas & Ken-dall, 2010; Herschell, Kolko, Baumann &Davis, 2010).

Often neglected in the research, however, is the ques-tion of how to prompt clinician interest in seeking outtraining on evidence-based interventions. One possibleavenue is mandating training and usage of EBP by

governmental or agency administrations. Jensen-Doss,Hawley, Lopez and Osterberg (2009) recently surveyedcommunity clinicians in Texas who were working undera mandate by the state to implement EBP. They foundthat the clinicians on average held slightly negative viewsof EBP and believed that their colleagues did as well;however, clinicians with more positive attitudes towardEBP reported using the interventions more often. Evenworking under mandate and requiring training, itappears that a key factor for the implementation of EBPmay be improving clinician attitudes. In addition, Allen,Ghrargozloo & Johnson ( 2012) found that favorableattitudes toward EBP predicted clinician knowledge ofcurrently available EBP and a desire to obtain training inthem. These findings suggest that identifying ways toimprove the appeal of EBP to clinicians with less favor-able attitudes toward them may prompt greater trainingand utilization of EBP. However, researchers have gener-ally failed to find individual characteristics such asprofessional discipline, years of experience, or educationlevel (Master’s or doctoral) predict attitudes toward EBP(e.g. Aarons, 2004; Jensen-Doss et al., 2009).

© 2012 The Authors. Child and Adolescent Mental Health. © 2012 Association for Child and Adolescent Mental Health.Published by John Wiley & Sons Ltd, 9600 Garsington Road, Oxford OX4 2DQ, UK and 350 Main St, Malden, MA 02148, USA

Child and Adolescent Mental Health 19, No. 1, 2014, pp. 52–56 doi:10.1111/camh.12005

A recent study found that clinicians serving childsurvivors of maltreatment were generally unable toidentify the empirically supported treatments available,commonly endorsed having received training in inter-ventions not typically considered to be empirically sup-ported (e.g. nondirective play therapy, sandtraytherapy), and regularly utilized these interventions inpractice (Allen et al., 2012). Interestingly, however, clini-cians appeared acutely aware that the interventions theywere employing were not empirically supported. Giventhe wide popularity of interventions that traditionallyhave not been subjected to controlled empirical exami-nation, it seems likely that relatively few practicing clini-cians require rigorous research before deciding to seekout training in or when selecting interventions. Alterna-tively, and possibly complementary, clinicians may notbe trained to determine the methodological rigor of sci-entific studies; clinicians are often trained to rely on the-oretical understanding and techniques as opposed toseeking out the results of empirical research.

Identifying the types of research or evidence com-monly required by clinicians may provide insight as tothe avenues that may be most fruitful in increasingutilization of EBP, particularly among clinicians withnegative attitudes toward EBP. This study examines therigor of scientific evidence clinicians require beforedeciding to utilize an intervention by comparing clinicianratings of favorability among different forms of evidence.In addition, various clinician characteristics (e.g. theo-retical orientation, professional discipline) and attitudestoward EBP are examined to determine their relation-ship with clinician preferences.

Method

ParticipantsData for this study were obtained as part of the TreatmentAttitudes, Perceptions, and Practices for Neglected and AbusedChildren (TAPPNAC) project. TAPPNAC was a nationwide (US)online survey of clinicians who currently work with maltreatedchildren in a mental health setting. The recruitment of partici-pants occurred in three distinct phases. First, e-mails weresent by the National Children’s Alliance to executive directorsof Children’s Advocacy Centers (CACs) throughout the US.Recipients were asked to forward the e-mail and link to theonline survey to mental health clinicians within their respec-tive CACs. The best response rate estimate currently available,based on clinicians potentially receiving the e-mail and thenumber of clinicians responding, is 29%. The second phase ofrecruitment involved directly e-mailing members of the Ameri-can Professional Society on the Abuse of Children (APSAC)who identified themselves as clinicians on the APSAC memberdirectory. Phase 3 included sending e-mail invitations to clini-cians in private practice, community mental health centers,and other community agencies. During phases 2 and 3, a totalof 435 invitations were sent with a response per invitationratio of .26. The reader is referred to Allen and Johnson(2012) for more detailed information regarding the TAPPNACmethodology.

This study examined responses from the 255 clinicians forwhom complete data were available. The average respondentwas 44.4 years old (SD = 12.5) and had 13.2 years (SD = 9.9)experience as a mental health clinician. The majority werefemale (86%), identified their ethnicity as White/European-American (86%), and held a Master’s degree as their highestlevel of education (79%). The respondents represented diverseprofessional disciplines with 38% obtaining their degree incounseling, 36% in social work, and 21% in clinical psychology.In addition, diverse theoretical orientations were present with

42% endorsing a cognitive-behavioral orientation, 30% eclectic,and 22% humanistic or psychodynamic. Fifty-three percent ofrespondents were employed in a CAC and 82% of clinicians wereindependently licensed. These demographics did not differ fromthe full sample (n = 262).

MeasuresEvidence-Based Practices Attitudes Scale (EBPAS). TheEBPAS is a self-report measure that asks clinicians to describethe degree to which they agree with 15 different statementsrelated to the implementation of evidence-based practices(Aarons, 2004). Items are answered on a Likert-type scale withresponses ranging from 0 (Not at all) to 4 (To a very great extent).The items yield four scales: Appeal (willingness to use an inter-vention if it is intuitively appealing), Requirements (willingnessto use an intervention if required to do so), Openness (willing-ness to use new innovations and methods), and Divergence(perception of evidence-based practices as diverging from cur-rent practices). For complete psychometric information, thereader is referred to Aarons et al. (2010). Internal consistencyestimates (Cronbach’s a) from the current sample range from.64 (Divergence) to .90 (Requirements).

Evidence Requirement Questionnaire (A copy of the ques-tionnaire is available from the first author). Clinicianswere presented with a series of questions that asked about thelikelihood they would use an intervention given only a specifictype of evidence. Responses were provided using a Likert-typescale with the following anchors: 0 (Never), 1 (Probably Not),2 (Possibly), 3 (Probably), 4 (Certainly). Questions were designedto cover the range of possible evidence clinicians might require,including colleague recommendation, conference training, casestudies, and clinical trials of various types. In addition, ques-tions were structured to explicitly state that no other evidence isavailable beyond that provided in the question. Four questionsbegan with the stem ‘I would use an intervention/treatment ifthe only evidence of its effectiveness was…’ The endings of thesequestions were as follows: (a) ‘a clinical trial with a placebo oractive treatment control group,’ (b) ‘a clinical trial with a waitlistcontrol group,’ (c) ‘a clinical trial with no control group,’ and (d)‘case studies or case reports.’ The final two items stated ‘I woulduse a intervention/treatment if a colleague recommended it,even in the absence of empirical research supporting its effec-tiveness,’ and ‘I would use an intervention/treatment I learnedabout at a conference or training, even in the absence of empiri-cal research supporting its effectiveness.’ Although these itemscreate a hierarchy of methodological rigor, they were presentedto the respondent in random order.

Results

Preferred EvidenceA series of paired-samples t-tests were performed toexamine preferences among the various types of evi-dence examined. The results of all analyses, as well asdescriptive statistics and effect sizes (r2) are provided inTable 1. Although all mean ratings fell between 1.68 and2.05, corresponding to qualitative anchors between‘probably not’ and ‘possibly,’ clinicians reported beingmost likely to utilize an intervention when a case studywas available [M = 2.05, SD = .66, 95% confidence inter-val (CI): 1.97–2.13], followed by a clinical trial with anactive control group (M = 1.93, SD = .76, 95% CI: 1.83–2.03). Although the difference in preference for these twotechniques was not significant, it is worth noting thatcase studies were more preferred than all other types ofevidence. Also of note is the finding that learning aboutan intervention at a conference or training (M = 1.88,SD = .77, 95% CI: 1.78–1.98) carried more weight withclinicians than either clinical trials with no control group

© 2012 The Authors. Child and Adolescent Mental Health © 2012 Association for Child and Adolescent Mental Health.

doi:10.1111/camh.12005 Burden of proof 53

(M = 1.70, SD = .76, 95% CI: 1.60–1.80) or a waitlist(M = 1.68, SD = .73, 95% CI: 1.59–1.77), and was notsignificantly different than clinical trials with an activecontrol group. Colleague recommendation (M = 1.69,SD = .81, 95% CI: 1.59–1.79) was similar in influence asclinical trials with no control group or a waitlist.

Predictors of preferred evidenceNext, analyses were performed to determine if any train-ing characteristics or attitudes toward EBP predictedone’s likelihood to use an intervention given a particulartype of evidence. Prior to performing analyses, indepen-dent samples t-tests were performed to determine ifemployment in a CAC influenced scores; no significantresults were obtained, and therefore employment settingwas not controlled in future analyses. A series of 2 (Edu-cation level: Doctoral or Master’s) 9 3 (Discipline: SocialWork, Counseling, Clinical Psychology) 9 3 (Theoreticalorientation: CBT, Eclectic, Humanistic/Psychodynamic)ANOVAs were performed on each of the six types ofevidence. To minimize the possibility of reporting inaccu-rate results due to test-wise alpha inflation, only resultssignificant at the p = .01-level are reported. No interactioneffects were found, and only one main effect was identi-fied. Differences in the likelihood to use an interventiongiven only colleague recommendation were found basedon theoretical orientation [F(2,182) = 5.42, p = .005]. Posthoc analyses revealed that CBT clinicians (M = 1.48,SD = .78) were significantly less likely to use an interven-tion based solely on colleague recommendation thaneclectic (M = 1.83, SD = .91; t (183) = 2.81, p = .006,r2 = .04) or humanistic/psychodynamic clinicians(M = 1.89, SD = .68; t (162) = 3.50, p = .001, r2 = .07).

Finally, zero-order correlations were computedbetween each of the types of evidence and scales mea-suring attitudes toward EBP. Clinician experience wasalso included. All correlations are presented in Table 2.Interestingly, none of the measures of attitudes towardEBP correlated with the likelihood that clinicians woulduse an intervention based solely on case studies. Con-versely, clinical trials, irrespective of the identified con-trol group, were viewed more favorably by those scoringhigher on a measure of openness toward EBP (rs = .29–.30, all p < .01). This implies that those possessing lessopen attitudes may be less likely to use an interventionbased solely on these research techniques. It is worthpointing out that a greater perception of EBPs as diverg-ing from one’s own practices correlated with a greaterwillingness to use an intervention based on colleaguerecommendation (r = .18, p < .01) or learning about thetechnique at a conference or training (r = .19, p < .01).

However, openness to the requirements of EBP (r = .18,p < .01) and finding appeal in using different interven-tions (r = .18, p < .01) were also correlated with greaterinfluence of colleague recommendation.

Conclusions

This study sought to examine the types of evidenceclinicians require before deciding to utilize an interven-tion in practice. In addition, it attempted to identifyindividual characteristics and attitudes toward EBP thatmay predict preferences for different types of evidence.Clinicians reported their likelihood of using an interven-tion based on any one type of evidence as being between‘probably not’ and ‘possibly.’ These results suggest thatclinicians may not be overly swayed by any one type ofevidence, instead preferring multiple sources of support.However, the research methods most likely to promptintervention utilization, in the absence of other research,were case studies and clinical trials with active controlgroups. It appears also that clinicians find benefit inattending trainings or conferences and learning of newinterventions in these settings.

Interestingly, attitudes toward EBP did not predict theperceived benefit of case studies, suggesting that clini-cians are equally likely to utilize an intervention basedon case study evidence irrespective of their attitudestoward EBP. A different pattern emerged, however, whenexamining predictors of perceptions of clinical trials.Clinicians who were more open to features of EBPs (e.g.following a treatment manual, using interventions devel-oped by researchers) were more likely to use interven-tions based on clinical trials, regardless of the presenceor type of control group in the research. This last findingalso suggests that those less open to the features of EBPwere less persuaded by clinical trials. Level of education,theoretical orientation, professional discipline, andyears of clinical experience generally failed to predictclinician preferences of research evidence. The oneexception was that CBT clinicians were less likely to usean intervention based solely on colleague recommenda-tion than either eclectic or humanistic/psychodynamicclinicians.

These findings have significant implications for thedissemination of EBP for children. Researchers havesignificantly advanced our understanding of effectiveinterventions primarily through the implementation ofclinical trials, especially trials which compare the tar-get intervention to a control group that controls fornonspecific factors (e.g. passage of time, therapeuticrapport). However, it appears that these trials are most

Table 1. Paired-samples t-tests and descriptive statistics of types of evidence

Researchmethod1

t (r2)2

t (r2)3

t (r2)4

t (r2)5

t (r2)6

t (r2)M(SD)

1. Colleague recommended – 4.90* (.09) 7.43* (.18) 0.15 (<.01) 0.22 (<.01) 3.93* (.06) 1.69(.81)2. Conference/training – 3.58* (.05) 3.73* (.05) 3.85* (.06) .76 (<.01) 1.88(.77)3. Case studies/reports - 8.80*(.23) 8.74* (.23) 2.45 (.02) 2.05(.66)4. Trial: No control – – .69 (.01) 4.64* (.08) 1.70(.76)5. Trial: Waitlist control – 5.69* (.11) 1.68(.73)6. Trial: Active control – 1.93(.76)

Note: All analyses are t (254).*p < .001.

© 2012 The Authors. Child and Adolescent Mental Health © 2012 Association for Child and Adolescent Mental Health.

54 Brian Allen & Natalie E. Armstrong Child Adolesc Ment Health 2014; 19(1): 52–6

convincing to clinicians open to the use of researchdeveloped interventions or treatment manuals, suggest-ing that other methods of dissemination are necessary.Increasing dissemination of EBP must target thoseclinicians with attitudes less favorable to EBP. Thisstudy suggests that case studies are a potential avenuethrough which to achieve that goal. Case studies werethe most preferred form of evidence by the cliniciansand this status was not dependent on clinician atti-tudes toward evidence-based practices. Indeed, themostcommonly utilized interventions identified by Allen et al(2012), such as nondirective play therapy, sandtraytherapy, and art therapy, benefit from decades ofdetailed case reports available in journals and publishedvolumes. Although these methods lack the type of rigor-ous controlled research required to identify them as‘empirically-supported,’ they are nonetheless widely dis-seminated through case study literature.

Developers and researchers are encouraged toincrease the number of case studies published describ-ing the utilization of EBPs. Kazdin (1981) pointed out anumber of features that allow one to draw valid infer-ences from case studies, including using valid andreliable assessment measures and assessing differentdomains of functioning. It seems particularly likely thatresearchers conducting clinical trials have sufficientinformation from participants to publish detailed andrigorous case reports. Clinicians in practice are alsoencouraged to adopt similar practices and publish rigor-ous and detailed case studies. At current, however, itappears that such case studies are rarely given dueattention in prestigious journals. For instance, in theyear 2010, the Journal of Clinical Child and AdolescentPsychology published 79 articles, only two of which werecase studies. Editors should encourage submission ofcase studies meeting the types of rigorous standardspromoted by Kazdin, perhaps even publishing specialeditions devoted to case studies describing the use ofEBP in practice. Small excerpts of case demonstrationstypically are provided in published treatment manuals,but these examples often lack depth, fail to provide acomplete description of a case, or demonstrate how toovercome common clinical barriers. Given the preva-lence of interventions not typically considered EBP,which have relied primarily upon published volumesand case presentations for dissemination, it seems rea-sonable to assume that these avenues could promptwider dissemination and utilization of EBP in generalpractice as well.

Equally important as increasing the number of pub-lished case studies describing EBP is addressing howgraduate programs train future clinicians There is aneed for graduate programs to provide budding clini-cians with training in the methods most current or effec-tive for certain clinical issues, as attitudes towardcertain treatments often develop out of exposure to infor-mation about those treatments. It is logical to hypothe-size that by increasing exposure to evidence-basedpractice models during the training phase of a clinician’scareer, future clinicians will be more likely to utilize rig-orous empirical research when deciding whether or notto use a particular intervention in clinical practice. Oneway to provide this kind of exposure is to require theaddition or restructuring of graduate courses detailingpsychotherapy treatments and theories. Instead of pro-viding a course(s) that merely surveys the theories oftreatment and associated techniques, graduate pro-grams should strive to provide a detailed focus on empir-ically tested and validated treatment models to illustratethe importance of incorporating scientific rigor intodeciding what treatment(s) to use. In addition, graduateprograms could incorporate case studies that focus onEBP, which may appeal to developing clinicians much asthis study suggests that they do to current clinical pro-fessionals. It may be that a more fundamental shiftneeds to occur within graduate programs as to how EBPis viewed, so as to set the stage for change within futuregenerations of clinicians.

This study should be viewed in light of its limitations.All clinicians involved worked with survivors of childmaltreatment. Although no differences were detectedbetween CAC clinicians and those in other communitysettings, it may be possible that unique characteristicsof the child maltreatment field influenced the results. Inaddition, the structure of the questions asked cliniciansthe likelihood of using an intervention if only one type ofevidence was present. In actuality, numerous types ofevidence are usually available for a given interventionand may be considered when making treatment selec-tion decisions. Lastly, the sample examined may havelimited generalizability, as the overall response rate forthe survey is best estimated between 26% and 29% anda lack of diversity of respondents is apparent. Futurestudies should address these limitations and examinethe relative importance of different forms of evidence.For instance, having clinicians rank order preferredtypes of evidence may provide a clearer picture regardingthe approaches that are most favored.

Table 2. Correlations of attitudes toward EBP and types of evidence

1 2 3 4 5 6 7 8 9 10 11 M (SD): 95 CI

1. Years experience – �.16 �.02 �.17* .20* �.02 �.02 �.02 <.01 .02 �.02 13.2 (9.86): 12.0–14.42. EBPAS: Requirements – .29* .20* �.27* �.05 .02 .01 .08 .04 .10 2.74 (.83): 2.64–2.843. EBPAS: Appeal – .55* �.06 .18* .15 .09 .20* .14 .15 2.93 (.57): 2.85–3.014. EBPAS: Openness – –.20* .18* .09 .15 .29* .29* .30* 2.38 (.68): 2.30–2.465. EBPAS: Divergence – .18* .19* .08 .06 .05 �.03 1.17 (.64): 1.09–1.256. Colleague recommended – .69* .46* .45* .41* .23* 1.69 (.81): 1.59–1.797. Conference/training - .45* .46* .36* .15 1.88 (.77): 1.78–1.988. Case studies/reports – .60* .52* .45* 2.05 (.66): 1.97–2.139. Trial: No control – .81* .45* 1.70 (.76): 1.60–1.80

10. Trial: Waitlist control – .55* 1.68 (.73): 1.59–1.7711. Trial: Active control – 1.93 (.76): 1.83–2.03

*p < .01.

© 2012 The Authors. Child and Adolescent Mental Health © 2012 Association for Child and Adolescent Mental Health.

doi:10.1111/camh.12005 Burden of proof 55

Acknowledgement

The study received no external funding support. The authorshave declared that they have no competing or potential conflictsof interest.

References

Aarons, G.A. (2004). Mental health provider attitudes towardadoption of evidence-based practice: The Evidence-BasedPractice Attitude Scale (EBPAS). Mental Health ServicesResearch, 6, 61–74.

Aarons, G.A., Glisson, C., Hoagwood, K., Kelleher, K., Lands-verk, J., Cafri, G., & The Research Network on Youth MentalHealth. (2010). Psychometric properties and U.S. nationalnorms of the Evidence-Based Practice Attitude Scale (EB-PAS). Psychological Assessment, 22, 356–365.

Allen, B., & Gharagozloo, L., & Johnson, J.C. (2012). Clinicianknowledge and utilization of empirically supported treat-ments for maltreated children. Child Maltreatment, 17,11–21.

Allen, B., & Johnson, J.C. (2012). Utilization and implementa-tion of Trauma-Focused Cognitive Behavioral Therapy for thetreatment of maltreated children. Child Maltreatment, 17,80–85.

Beidas, R.S., & Kendall, P.C. (2010). Training therapists inevidence-based practice: A critical review of studies from

a systems-contextual perspective. Clinical Psychology, 17,1–30.

Decker, S.E., Jameson, M.T., & Naugle, A.E. (2011). Therapisttraining in empirically supported treatments: A review ofevaluation methods for short- and long-term outcomes.Administration and Policy in Mental Health, 38, 254–286.

Glisson, C., Schoenwald, S.K., Kelleher, K., Landsverk, J.,Hoagwood, K.E., Mayberg, S., . . . & The Research Network onYouth Mental Health. (2008). Therapist turnover and newprogram sustainability in mental health clinics as a functionof organizational culture, climate, and service structure.Administration and Policy in Mental Health, 35, 124–133.

Herschell, A.D., Kolko, D.J., Baumann, B.L., & Davis, A.C.(2010). The role of therapist training in the implementation ofpsychosocial treatments: A review and critique with recom-mendations. Clinical Psychology Review, 30, 448–466.

Jensen-Doss, A., Hawley, K.M., Lopez, M., & Osterberg, L.D.(2009). Using evidence-based treatments: The experiences ofyouth providers working under a mandate. ProfessionalPsychology: Research and Practice, 40, 417–424.

Kazdin, A.E. (1981). Drawing valid inferences from case studies.Journal of Consulting and Clinical Psychology, 49, 183–192.

Accepted for publication: 13 August 2012Published online: 12 October 2012

© 2012 The Authors. Child and Adolescent Mental Health © 2012 Association for Child and Adolescent Mental Health.

56 Brian Allen & Natalie E. Armstrong Child Adolesc Ment Health 2014; 19(1): 52–6