![Page 1: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/1.jpg)
FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION SCIENCE TO SUPPORT ORGANIZATION-WIDE EFFORTS IN THE UTILIZATION AND UNDERSTANDING OF REPORT DATA
Presenters:Mollie Sternberg, MPHAmanda Gentz, B.A.Edgar Estrada, B.A.
Additional Authors:Erin Sullivan, B.A.Natalie Gallardo, B.A.Elisa Hernandez, B.A.Cameo Stanick, Ph. DJanine Quintero Psy. D.Debbie Manners, LCSW
menti.com83 49 53
![Page 2: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/2.jpg)
• Introduction & Background
• Learning Objectives
• Agency Background
• Who we are & Who’s in the room?
• Fostering a Data of Culture
• Explore: Framework and Resources
• Engage: Vignettes 1 and 2
• Small Group Work
• Questions
Agenda
2
![Page 3: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/3.jpg)
Learning Objectives
1. Identify and describe three foundational processes in data culture implementation.
2. Evaluate organization’s current culture of data for strengths, weaknesses, opportunities, and threats.
3. Assess a current report or data workflow for an area to incorporate at least one foundational process.
As a result of attending this session, participants will be able to:
3
![Page 4: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/4.jpg)
Wide Array of Mental Health Services:
Community-based mental health & child welfare organization serving
roughly 6,858 youth & families/year in Los Angeles County, CA,
• Wraparound
• Prevention and Early Intervention (PEI)
• Full Service Partnership (FSP)
• Therapeutic Behavioral Services (TBS)
• Outpatient
• Community-Based
• School-Based
• Residential (STRTP)
Hathaway-Sycamores
4
![Page 5: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/5.jpg)
Hathaway-Sycamores
5
![Page 6: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/6.jpg)
Introductions
6
![Page 7: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/7.jpg)
Source: https://episframework.com/what-is-epis Accessed: 3/25/2019
• Phases aligned with Implementation Process
• Considers internal and external contexts
7
![Page 8: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/8.jpg)
We asked:
What barriers to
utilizing data do you
experience?
8
![Page 9: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/9.jpg)
The Collaboration, Monitoring, and Internal Compliance Report Cycle
References for each step in the cycle
can be found on slide 38 9
![Page 10: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/10.jpg)
The Cycle: Repeat, Reuse, Recycle!
RecursiveAny stage of the process can be revisited at any time for changes and
improvement
Dynamic
Each stage of the process is interactive and subject to change over time
Responsive
Each stage of the process helps us prepare for future changes
InclusiveThe process looks for the input of different perspectives in the
organizational structure
10
![Page 11: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/11.jpg)
• Collaboration• Meet with and gather information from as many
individuals as practical
• Monitoring• Assess logistics of distribution
• Continue open feedback cycle
• Internal Compliance• Identify items for follow-up
• Initiate process improvement plans
• Reevaluate report’s ability to meet current needs
Accompanying Checklist
11
![Page 12: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/12.jpg)
Example 1
PEI Practice Council Reports
12
![Page 13: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/13.jpg)
PEI Reports: Consult Data Experts
• Met with Finance and Quality Management
• Determined what was already incorporated what new data was needed
Successes:• Able to ensure consistency in how finance metrics were reported across
departments
• Review what was already being done
Room for Improvements: • Gather stakeholders and data experts at the same time
• Be more specific when sharing data
• Ensure understanding of data from other parties including how metrics are defined and calculated
13
![Page 14: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/14.jpg)
PEI Reports: Meet with Stakeholders
• Met with each practice council lead• As collective group
• Followed up as needed individually
• Used a worksheet to record metric possibilities
Successes:• Targeted and actionable ideas on specific metrics
• Deadlines allowed for progress to be made on report creation
Room for Improvements: • Encourage even more conversation and collaboration
• Ensure understanding around the utilization of data
14
![Page 15: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/15.jpg)
PEI Reports: Create a Prototype
• Piloted a universal template
• Individualized template for each practice council
Successes:• Individual reports allowed for more targeted feedback
• Able to solve issues of joining multiple data sources before scaling
Room for Improvements: • Reflect time and resource intensive prototype process for each practice
council in timeline
• Involve data experts more
15
![Page 16: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/16.jpg)
PEI Reports: Pilot Prototype
• Shared pilot with goal of receiving feedback
• Discuss completeness of pilot and needed enhancements
Successes:• Revised vague metric titles and confusing graphics
• Solidified the report design
• Cleared up any discrepancies or confusion with the data
Room for Improvements: • Include data experts more
16
![Page 17: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/17.jpg)
PEI Reports: Monitor Distribution
• Solidify details around report distributed workflow• Frequency
• Recipients
• Method
Successes:• Uniform distribution allows for more transparency and consistency in
communication
• Email distribution is timely and resource efficient
Room for Improvements: • Consider balance between uniform distribution and timeliness as it applies to
utilization of data earlier in cycle
• Increase opportunities and methods for communication and feedback
17
![Page 18: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/18.jpg)
PEI Reports: Follow up with Stakeholders
• Leveraged an existing quarterly meeting time
• Create standing meeting items around feedback and needs/ current priorities
Successes:• Encouraging feedback helps to assure the report’s usefulness
• Following up on the data facilitates open discussion on key findings
Room for Improvements: • Limit introducing/ altering metrics if trend data is important
• Increase support around initial communications
18
![Page 19: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/19.jpg)
PEI Reports: Continually Monitor Data
• Routine review of workflow and data points
• Early identification of potential threats to data integrity
Successes:• Targets potential flaws in the data collection process
• Ensures completeness, accuracy, and consistency of data
Room for Improvements: • Early identification of a data expert for “red flag” identification
• Training of report analysts on data trends
19
![Page 20: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/20.jpg)
PEI Reports: Implement Process Improvement
• Guided discussion based on data points
• Work with stakeholders to identify areas to improve the programs/services
Successes:• Increased credibility for suggestions on improving the impact of funding
utilization
• Allowed for more targeted increase of PEI utilization
Room for Improvements: • Increase follow-up connections between the various departments to
close loop
20
![Page 21: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/21.jpg)
PEI Reports: Reevaluate Report• Reevaluated the data collection, analyzing, and reporting process
• Make continuous improvements in efficiency and accuracy of data
Successes:
• Discovered ways to make the data more approachable
• Joined different departments that work with same data but in different ways
• Solidified a common language around metrics and data to ensure data integrity
Room for Improvements:
• Increase opportunities for face-to-face communication to limit feedback and understanding delays in the processes
21
![Page 22: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/22.jpg)
22
PEI Reports:Repeat, Reuse, and Recycle
![Page 23: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/23.jpg)
Example 2
Personal Rights Check-in Tool and
Report
23
![Page 24: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/24.jpg)
Personal Rights Check-in: Consult Data Experts• Met with Residential Program leadership and staff
• Identified metrics, questions, and items of interest for the survey
Successes:• Flexibility to create survey items and accessible for staff
Room for Improvements: • Include more program staff voice in survey development and revisions
• Ensure that all program staff understand data utilization goal
24
![Page 25: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/25.jpg)
Personal Rights Check-in: Meet with Stakeholders• Met with program leadership and staff
• Discussed the survey data and value of the items
• Adjusted survey items based on feedback
Successes:• Collaborated with all program voices
• Able to address any concerns regarding survey questions
Room for Improvements: • Ensure that all program staff have appropriate training for survey
administration
• Identify incentives for survey completion by youth
25
![Page 26: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/26.jpg)
Personal Rights Check-in: Create a Prototype• Collaboration with stakeholders
• Identified best way to visually present the data create a visual report.
Successes:• Data easy to monitor and analyze on online platform
• Leveraged the resources and skills (i.e., data viz techniques) to develop visuals
• Multiple data experts were able to collaborate
Room for Improvements: • Identify methods to best automate visual report with current workflow
• Create instructions on why and how to best analyze the data to ease transition
26
![Page 27: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/27.jpg)
Personal Rights Check-in: Pilot Prototype• Test accessibility of survey in residential setting
• Test format of report to relevant program stakeholders
Successes:• Leverage existing technology resources
• Clear data metrics
• Gained program leadership buy-in
Room for Improvements: • Develop formal guidelines for reasoning behind items
• Identify alternatives to online survey format if needed
27
![Page 28: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/28.jpg)
Personal Rights Check-in: Monitor Distribution• Solidify details around report distributed workflow
• Frequency
• Recipients
• Method
Successes:• Able to disseminate report to relevant program stakeholders via email
• Determined frequency of reporting distribution as monthly and quarterly
Room for Improvements: • Create checklist to ensure all relevant stakeholders are included in report
distribution
• Create venue for research team and program staff to communicate about irregularities in the data in a timely manner
28
![Page 29: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/29.jpg)
Personal Rights Check-in: Follow up with Stakeholders• Established recurring “roundtable” meetings with program
leadership and staff
• Dedicate time to review and revise report
Successes:• Able to meet in person with all relevant program stakeholders
• Strong feedback created opportunities for meaningful revisions and future action steps
Room for Improvements: • Identify essential stakeholders that must be present to account for
scheduling difficulties
• Check for understanding among collaborators on how metrics are defined and calculated
29
![Page 30: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/30.jpg)
Personal Rights Check-in: Continually Monitor Data • Research team frequently monitors the data to ensure
completion and identifies inconsistencies.
Successes:• Able to track all data and related information
• House all data in one platform (i.e., Survey Monkey)
Room for Improvements: • Automate data analysis and provide guidelines on how to best
analyze data
• Improve timeliness of communication with program staff when irregularities arise
30
![Page 31: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/31.jpg)
Personal Rights Check-in: Implement Process Improvement
• Follow program leadership and staff suggestions for action items
• Track progress of change items
Successes:• Able to identify baselines for specific items
• Able to track changes over time on the items
Room for Improvements: • Create meetings with program leadership and staff to determine targets
and goals
• Align with other departments to ensure reporting consistency and presentation
31
![Page 32: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/32.jpg)
Personal Rights Check-in: Revaluate Report
• Schedule “roundtable” debriefs within the research team
Successes:
• Revisions made to items to minimize redundancy and maximize clarity
• Review items that were already implemented
Room for Improvements:
• Communicate reasons behind any survey change
• Train staff on workflow changes
32
![Page 33: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/33.jpg)
33
Personal Rights Check-in:Repeat, Reuse, and Recycle
![Page 34: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/34.jpg)
34
![Page 35: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/35.jpg)
Small Group DiscussionWhat can you bring back to your organization?
35
![Page 36: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/36.jpg)
Large Group Discussion
36
![Page 37: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/37.jpg)
Take Home/Summary
• Foundational processes in data culture implementation• Collaboration
• Consult Data Experts
• Meet with Stakeholders
• Create a Prototype
• Pilot Prototype
• Monitoring
• Monitor Distribution
• Follow Up with Stakeholders
• Internal Compliance
• Continually Monitor Data
• Implement Process Development
• Reevaluate Report
• Repeat, Reuse, Recycle!
37
![Page 38: FOSTERING A CULTURE OF DATA: USING IMPLEMENTATION … · Learning Objectives 1. Identify and describe three foundational processes in data culture implementation. 2. Evaluate organization’s](https://reader034.vdocuments.us/reader034/viewer/2022042218/5ec45d402e674518d25260b0/html5/thumbnails/38.jpg)
References• Aarons G, Hurlburt M, Horwitz S. Advancing a conceptual model of evidence-based practice
implementation in public service sectors. Adm Policy Ment Health Ment Health Serv Res. 2011;38.
• Arnold, D. H., & Harvey, E. A. (1998). Data monitoring: A hypothesis-testing approach for treatment-outcome research. Journal of Consulting and Clinical Psychology, 66(6), 1030–1035. https://doi-org.libproxy.chapman.edu/10.1037/0022-006X.66.6.1030
• Bruns, T., & Inefuku, H. (2016). Purposeful Metrics: Matching Institutional Repository Metrics to Purpose and Audience. In Callicott B., Scherer D., & Wesolek A. (Eds.), Making Institutional Repositories Work (pp. 213-234). West Lafayette, Indiana: Purdue University Press. doi:10.2307/j.ctt1wf4drg.21
• Langley, G. J., Moen, R. D., Nolan, K. M., Nolan, T. W., Norman, C. L., & Provost, L. P. (2009). The Improvement Guide: A Practical Approach to Enhancing Organizational Performance (2nd ed.). San Francisco, CA: Jossey-Bass. (174-175)
• Perrino, T., Howe, G., Sperling, A., Beardslee, W., Sandler, I., Shern, D., . . . Brown, C. (2013). Advancing Science Through Collaborative Data Sharing and Synthesis. Perspectives on Psychological Science, 8(4), 433-444. Retrieved from http://www.jstor.org.libproxy.chapman.edu/stable/44289915
• Powell, B. J., McMillen, J. C., Proctor, E. K., Carpenter, C. R., Griffey, R. T., Bunger, A. C., … York, J. L. (2012). A Compilation of Strategies for Implementing Clinical Innovations in Health and Mental Health. Medical Care Research and Review, 69(2), 123–157. https://doi.org/10.1177/1077558711430690
• Stringer, E. T. (2014). Action Research (Vol. 3). Thousand Oaks, CA: SAGE. • Managing Performance Measurement Data in Health Care. (2001). Oakbrook Terrace, IL: Joint
Commission on Accreditation of Healthcare Organizations.
38