assessing organizational context for implementation research gregory a. aarons, ph.d. university of...
TRANSCRIPT
Assessing Organizational Context for Assessing Organizational Context for Implementation ResearchImplementation Research
Gregory A. Aarons, Ph.D.Gregory A. Aarons, Ph.D.
University of California, San DiegoUniversity of California, San Diego
Department of PsychiatryDepartment of Psychiatry
Presented at the Administration for Children and Families Conference: Improving Implementation Research Methods for Improving Implementation Research Methods for Behavioral and Social Science. Silver Spring, MD: Behavioral and Social Science. Silver Spring, MD: September 20-21, 2010
Overarching QuestionsOverarching Questions
What aspects of organizational context may What aspects of organizational context may be implicated in implementation be implicated in implementation effectiveness? effectiveness?
How do we better understand the process and How do we better understand the process and outcomes of implementation efforts? outcomes of implementation efforts?
Examples Examples – QuantitativeQuantitative– QualitativeQualitative– Mixed-MethodsMixed-Methods
Organizational Context of Social ServicesOrganizational Context of Social Services
Source: Glisson & Schoenwald (2005)Glisson & Schoenwald (2005)
Levels of ChangeLevels of Change
Larger System/ Environment
Organization
Group / Team
Individual
Reimbursement, legal, and regulatory policies are key
Structure and strategy are key
Cooperation, coordination, & shared knowledge are key
Knowledge, skill, and expertise are key
Source: Shortell, (2004) Medical Care Research and Review
Four Levels of Change for Assessing Performance Improvement
Assumptions about Change
Conceptual Model of Conceptual Model of Implementation ResearchImplementation Research
InterventionInterventionStrategiesStrategies
EvidenceEvidence BasedBased
PracticesPractices
ImplementationImplementationStrategiesStrategies
Systems Systems
EnvironmentEnvironment
OrganizationalOrganizational
Group/LearningGroup/Learning
SupervisionSupervision
ProvidersProviders
ConsumersConsumers
Implementation Implementation OutcomesOutcomes
FeasibilityFeasibility
FidelityFidelity
ReachReach
AcceptabilityAcceptability
SustainabilitySustainability
UptakeUptake
CostsCosts
Service Service OutcomesOutcomes
Efficiency Efficiency
SafetySafety
EffectivenessEffectiveness
EquityEquity
Patient-CenteredPatient-Centered
TimelinessTimeliness
Client Client OutcomesOutcomes
SymptomsSymptoms
Functioning Functioning
SatisfactionSatisfaction
Quality of LifeQuality of Life
OutcomesOutcomes
Implementation Research MethodsImplementation Research Methods
Source: Proctor, Landsverk, Aarons, Chambers, Glisson, Mittman (2009)Source: Proctor, Landsverk, Aarons, Chambers, Glisson, Mittman (2009)
*IOM Standards of Care*IOM Standards of Care
Conceptual Model of Implementation Conceptual Model of Implementation Phases and Context LevelsPhases and Context Levels
Source: Aarons, Hurlburt, & Horwitz,(In Review)
Organizational Culture and Climate
Organizational culture and climate are related but distinct constructs– Aarons & Sawitzky, 2006; Denison, 1996; Schneider, Ehrhart, &
Macey, 2010
32 definitions of organizational climate and 54 32 definitions of organizational climate and 54 definitions of organizational culture with some definitions of organizational culture with some overlap between these constructsoverlap between these constructs– Verbeke, Volgering, Hessels 1998Verbeke, Volgering, Hessels 1998
Organizational Culture
“…“…shared basic assumptions… considered valid and, the shared basic assumptions… considered valid and, the correct way to perceive, think, and feel in” an organization correct way to perceive, think, and feel in” an organization – Schein, 2004Schein, 2004
Behavioral norms and expectations for the ways in which Behavioral norms and expectations for the ways in which people behave in an organizationpeople behave in an organization– Cooke & Rousseau, 1988Cooke & Rousseau, 1988
““the way things are done here…”the way things are done here…”
Organizations with cultures that are more supportive of Organizations with cultures that are more supportive of employees and that are adaptable to changes in the employees and that are adaptable to changes in the environment are more effective environment are more effective – (Kotter & Heskett, 1992; Wilderom, glunk, & Maslowski, 2000)(Kotter & Heskett, 1992; Wilderom, glunk, & Maslowski, 2000)
Competing Values Framework
Source: Kalliath, Bluedorn, & Gillespie (1998) Ed & Psych MeasurementSource: Kalliath, Bluedorn, & Gillespie (1998) Ed & Psych Measurement
Organizational Culture Inventory
Source: human synergistics 1987-2007Source: human synergistics 1987-2007
Organizational Climate
Employees’ perceptions of the policies, practices, and procedures and the kinds of behaviors that get rewarded, supported, and expected in a setting– Schneider, 1990
Employee’s perceptions of the psychological impact Employee’s perceptions of the psychological impact of the work environment on his or her own well-of the work environment on his or her own well-being and functioning in the organization being and functioning in the organization – Glisson et al., 2007; Glisson and James, 2002Glisson et al., 2007; Glisson and James, 2002
Focused or strategic climates – e.g., Diversity climate, Service climate, Safety climate
Kuenzi & Schminke, 2009; Schneider et al., 2010
Organizational Social ContextOrganizational Social Context
Source: Glisson et al. 2007Source: Glisson et al. 2007Fig. 1 Confirmatory factor analysis (CFA) of organizational social context (OSC) Fig. 1 Confirmatory factor analysis (CFA) of organizational social context (OSC)
Quantitative AnalysesQuantitative Analyses
Relationship of Organizational Culture and Climate Relationship of Organizational Culture and Climate with Attitudes Toward Adopting EBPwith Attitudes Toward Adopting EBP
Source: Aarons & Sawitzky (2006) Source: Aarons & Sawitzky (2006)
Organizational Climate Partial Mediation of the Effect of Organizational Climate Partial Mediation of the Effect of Organizational Culture on Work Attitudes and TurnoverOrganizational Culture on Work Attitudes and Turnover
Source: Aarons & Sawitzky (2006) Source: Aarons & Sawitzky (2006)
Figure 1. Multigroup Clustered Path Analysis: Association of Transformational Leadership and Leader-Figure 1. Multigroup Clustered Path Analysis: Association of Transformational Leadership and Leader-Member Exchange with Team Climate for Innovation and Team Climate for Innovation with Staff Attitudes Member Exchange with Team Climate for Innovation and Team Climate for Innovation with Staff Attitudes Toward Innovation Adoption During Innovation Implementation compared to Services as Usual. Note: Toward Innovation Adoption During Innovation Implementation compared to Services as Usual. Note: N=140; Teams Implementing the SafeCare (n=85) / Teams Providing Services as Usual (n=55); N=140; Teams Implementing the SafeCare (n=85) / Teams Providing Services as Usual (n=55);
χχ22 (4)=1.105; (4)=1.105; pp=.894; CFI=1.000, TLI=1.037, RMSEA=0.000, SRMR=0.013; *p<.05, **p<.01, ***p<.001=.894; CFI=1.000, TLI=1.037, RMSEA=0.000, SRMR=0.013; *p<.05, **p<.01, ***p<.001
Leader MemberLeader MemberExchangeExchange
Attitudes Toward Attitudes Toward Adopting EBPAdopting EBP
TransformationalTransformationalLeadershipLeadership
.74**.74**/-.11/-.11
Team Climate for Team Climate for InnovationInnovation
-.09-.09/.89***/.89***
.29**.29**/.22/.22
.84***.84***/.91***/.91***
Source: Aarons & Sommerfeld (In Review) Source: Aarons & Sommerfeld (In Review)
Figure 3. Path model of partial mediation effects of agency type on organizational support for evidence-Figure 3. Path model of partial mediation effects of agency type on organizational support for evidence-based practice and attitudes toward evidence-based practice, effect of organizational support for based practice and attitudes toward evidence-based practice, effect of organizational support for evidence-based practice on provider attitudes toward evidence-based practice, and effect of organizational evidence-based practice on provider attitudes toward evidence-based practice, and effect of organizational support for evidence-based practice on provider use of evidence-based practice. support for evidence-based practice on provider use of evidence-based practice. N = 170; AIC = 2437.127, SBIC = 2436.638; *p<0.05, **p<0.01 (one-tailed)N = 170; AIC = 2437.127, SBIC = 2436.638; *p<0.05, **p<0.01 (one-tailed)
EBP UseEBP UseOrganizational Organizational
Support for Support for EBPEBP
AttitudesAttitudesTowardToward
EBPEBP
Agency TypeAgency Type
0.15*0.15* 0.16*0.16*
0.18*0.18*
0.09ns0.09ns
0.11**0.11**
Source: Aarons, Sommerfeld, & Walrath-Greene (2009)Source: Aarons, Sommerfeld, & Walrath-Greene (2009)
Qualitative AnalysesQualitative Analyses
Implementation of a Computerized Implementation of a Computerized HIV Clinical Reminder HIV Clinical Reminder (PIs, Goetz, Asch; VA QUERI)(PIs, Goetz, Asch; VA QUERI)
Screening rates of HIV in VA very lowScreening rates of HIV in VA very low
Question: What type of service improvement can improve Question: What type of service improvement can improve HIV testing with appropriate individualsHIV testing with appropriate individuals
Develop software to identify high-risk vets and remind Develop software to identify high-risk vets and remind providers to recommend HIV testproviders to recommend HIV test
– SoftwareSoftwareAssesses EMRAssesses EMRInitiates reminder – regardless of service (no wrong door)Initiates reminder – regardless of service (no wrong door)
Ethnographic process evaluationEthnographic process evaluation
Part of QUERI (Quality Enhancement Research Initiative)Part of QUERI (Quality Enhancement Research Initiative)
Sobo , Bowman, Aarons, Asch, Gifford (2008) Human Organization
Implementation of a Computerized Implementation of a Computerized HIV Clinical ReminderHIV Clinical Reminder
Sobo , Bowman, Aarons, Asch, Gifford (2008) Human Organization
Results:Results:Implementation of a Computerized Implementation of a Computerized
HIV Clinical Reminder HIV Clinical Reminder
Key organizational sub-culture differences in stakeholder Key organizational sub-culture differences in stakeholder agendas impacted implementation agendas impacted implementation – (e.g., physician, nursing, information technology, laboratory)(e.g., physician, nursing, information technology, laboratory)
Emergence of strategic communication processes that, Emergence of strategic communication processes that, despite their immediate utility, sometimes undermined despite their immediate utility, sometimes undermined progress and threatened long term relationshipsprogress and threatened long term relationships– focus on the local focus on the local – information reconfigurationsinformation reconfigurations– partiality partiality
Sobo , Bowman, Aarons, Asch, Gifford (2008) Human Organization
Mixed-MethodsMixed-MethodsCombines qualitative and quantitative approachesCombines qualitative and quantitative approaches
CollaborationCollaboration between quantitative and qualitative between quantitative and qualitative researchers during theresearchers during the study design phasestudy design phase
Open acknowledgement of the philosophicalOpen acknowledgement of the philosophical approaches brought to the study by various team approaches brought to the study by various team membersmembers
Shared willingnessShared willingness to negotiate emerging problemsto negotiate emerging problems
Should include mixing of design, analyses, and resultsShould include mixing of design, analyses, and results
Source: Willging et al., (2007)(2007)
Mixed-Methods Research Offers Several Mixed-Methods Research Offers Several Advantages over Single-Method ApproachesAdvantages over Single-Method Approaches
Combine the qualitative and quantitative Combine the qualitative and quantitative approaches into the research methodology of a approaches into the research methodology of a single study or multi-phased studysingle study or multi-phased study
Simultaneously answer confirmatory and Simultaneously answer confirmatory and exploratory questions, and therefore verify and exploratory questions, and therefore verify and generate theory in the same studygenerate theory in the same study– Teddlie & Tashakkori, 2003Teddlie & Tashakkori, 2003
Mixed-Methods Study of Statewide Mixed-Methods Study of Statewide EBP Implementation EBP Implementation (NIMH PI: Aarons)(NIMH PI: Aarons)
Implementation of SafeCare® in Oklahoma’s Implementation of SafeCare® in Oklahoma’s Statewide Children’s Services SystemStatewide Children’s Services System
Combines exploratory and confirmatory Combines exploratory and confirmatory approachesapproaches
Mixed MethodsMixed Methods– Equal quantitative and qualitative componentsEqual quantitative and qualitative components
Longitudinal at organization/team levelLongitudinal at organization/team level
Requires collaboration and ongoing relationship Requires collaboration and ongoing relationship building and maintenancebuilding and maintenance
Figure 1. Kaplan-Meier Survival Function Estimates (Retention Probability) by Study Condition. Note: SC/M = participating in SafeCare and fidelity monitoring; SC/Non = participating in SafeCare, but not fidelity monitoring; SAU/M = services as usual and receiving fidelity monitoring; and SAU/Non = services as usual and not receiving fidelity monitoring. N=153.
Source: Aarons, Sommerfeld, et al (2009), Journal of Consulting and Clinical Psychology
Annualized Turnover by Condition
Consultation
Yes No
Yes
SafeCare®
No
14.9% 33.4%
41.5% 37.6%
Implementation Outcomes Implementation Outcomes Effect of EBP Implementation on Staff RetentionEffect of EBP Implementation on Staff Retention
CCM perspective on EBP implementation CCM perspective on EBP implementation and turnoverand turnover
Having to learn new skills Having to learn new skills and dissatisfaction with SC and dissatisfaction with SC or with being monitored or with being monitored may have contributed to may have contributed to some of the older CCMs to some of the older CCMs to quit their jobs.quit their jobs.
Learning skills like SC were Learning skills like SC were motivations to stay with motivations to stay with current employers.current employers.
““And the CCM’s that I see having a problem And the CCM’s that I see having a problem adapting; actually the ones that have the adapting; actually the ones that have the trouble adapting were excellent case trouble adapting were excellent case managers, but they have a style that’s pretty managers, but they have a style that’s pretty free flowing and they just aren’t adjusting as free flowing and they just aren’t adjusting as well as you would like to see them.”well as you would like to see them.”
““I mean if they don’t all succeed and I never, I mean if they don’t all succeed and I never, ever am going to expect that they all ever am going to expect that they all succeed, because you have those that are succeed, because you have those that are not just to do it and work at it successfully. not just to do it and work at it successfully. But when you see the percentage of them But when you see the percentage of them that do succeed is so much higher than those that do succeed is so much higher than those that don’t, it really makes it worth it. And that don’t, it really makes it worth it. And that’s the whole goal for me with my families that’s the whole goal for me with my families is for them to succeed.”is for them to succeed.”
THEMETHEME QUOTEQUOTE
Agency/program director perspective on EBP Agency/program director perspective on EBP implementation and turnoverimplementation and turnover
Implementation of EBPs Implementation of EBPs helps to recruit and retain helps to recruit and retain new staff.new staff.
Learning new skills like SC Learning new skills like SC might inspire CCMs to seek might inspire CCMs to seek higher paying jobs higher paying jobs elsewhere.elsewhere.
“…“…its like any kind of change within, its like any kind of change within, you know, staff. There’s gonna be you know, staff. There’s gonna be some resistance. But I feel like, well, some resistance. But I feel like, well, as evidenced from our turnover. We as evidenced from our turnover. We have very little turnover here. And, you have very little turnover here. And, you know, if they weren’t happy, they know, if they weren’t happy, they wouldn’t stay.”wouldn’t stay.”
““It is helping recruit and retain good It is helping recruit and retain good staff to recognize that, wow, [agency] is staff to recognize that, wow, [agency] is the place where you can go and get the place where you can go and get trained in the latest evidence-based trained in the latest evidence-based practices and have good support, good practices and have good support, good supervision, and that sort of is part of supervision, and that sort of is part of our goal is to be recognized for that”our goal is to be recognized for that”
THEMETHEME QUOTEQUOTE
