1
Analytics-BasedEHRS
Implementation:
Improved Outcomes
Session 234, February 14, 2019 – 11:30 AM
John A. Rekart, Ph.D.Chief Psychologist, Quality Management and Informatics
Statewide Mental Health ProgramCalifornia Department of Corrections and Rehabilitations
2
©HIMSS 2018-2019S Suman Free use
3
MeIntroduction
YouAudience
Participation
EHRS Adoption FailureProblem
Formulation
Analytic/Evidence Based Project Management
Solutions
We Crushed ItOutcomes
Agenda
©HIMSS 2018-2019
4
• Demonstrate knowledge of how data will
improve success of electronic health
record system (EHRS) implementations
• Demonstrate an understanding of how
quality management tools can improve
training results for EHRS implementations
• Describe the metrics used to support a
data-based model of implementation over
a vendor-based cookie-cutter model
Learning Objectives
©HIMSS 2018-2019
5
California Correctional Health Care
Services– Statewide Mental Health
Program
122,901 Inmates
38,000 Mental Health
Patients
35 Institutions
1,300 Mental Health
Providers CDCR
©HIMSS 2018-2019
6
Profession
Audience Poll
Creative Commons
7
Electronic Healthcare Record
System
1) Audience Poll – EHRS
Creative Commons
8Implementation Goal
Audience Poll
Creative Commons
9
“In God we trust;
…all others must bring
data.”
W. Edwards Deming, physicist and quality
improvement pioneer©HIMSS 2018-2019Creative Commons
Industry EHRS Failure Rate
“50% of EMR system implementations result in failure.” International Journal of
Technology Assessment in Health Care, 1997
“Industry experts estimate that failure rates of EMR implementations range from 50-80%.” A Commonsense
Approach to EMRs, 2006
“. . . 50–80% EMR failure rate documented in the Milbank Quarterly and cited by the AMA.” 2010
Creative Commons ©HIMSS 2018-2019
More Bad News
“It is estimated that between 30-50% of EHR implementation projects fail. According to a research report by health IT research firm KLAS, by 2016, almost 50% of large hospitals will replace their current EHR.”
“Most of the EMR projects, like any large-scale enterprise-wide projects, take years to deploy and stabilize and cost as high as 3-6% of a hospital’s operating budget. And yet, 30-50% of such projects fail to deliver on the promise.”
Top 10 Reasons Why EMR Implementations Fail - Udai Kumar, May 12, 2016Creative Commons
Creative Commons
©HIMSS 2018-2019
Why they fail
Workflows Training
Adoption
&
Go Live
Poorly designedand implemented
Creative Commons©HIMSS 2018-2019
•Don’t get into stranger’s cars•Don’t meet people from the internet
• Literally summon strangers from the internet to get into their car
Carol Nichols, twitter.Creative Commons ©HIMSS 2018-2019
Data Liquidity • Data liquidity in health
information systems. ... Both had as their goals, implicit or explicit, to ensure the right dataare provided to the right person at the right time, which is one definition of "data liquidity."
• Data liquidity in health information systems. - NCBI - NIH
•https://www.ncbi.nlm.nih.gov/pubmed/21799328
Data AgentChange
Accuracy - Efficacy
Creative Commons
Better Data Liquidity Increased Process Influence
15
Data Entry and Database1. InfoPath2. SharePoint3. Access
Vendor Analytics Suite1. Discern Analytics – Front end2. LightsOn - System Analytics
Data Warehouse1. On_Demand Report2. Outlook Email Integration
Methods and Tools
©HIMSS 2018-2019
Problematic Workflow Issues
Reproduce Paper Based Workflows
Misunderstanding of EHR Capabilities
Competing and Conflicting Design Goals
Poor Integration of Interdisciplinary Objectives
Creative Commons ©HIMSS 2018-2019
Components of a Good Workflow
Increases Patient Safety
Streamlines Clinical Documentation
Increases Interoperability
Increases Provider Efficiency
Continuous Quality Improvement - OptimizationCreative Commons
18
TrainingApplication of Quality Management Principles and Practices to Training
Creative Commons©HIMSS 2018-2019
• Increased Training Hours• Divided the curriculum
• Basic EHRS training• Process Training
• Certification of Trainers• Audit and Feedback of
Training
• Data Liquidity
Training Improvements• To Meet New Build Complexity• Organize into Conceptual
Chunks
• Assure Quality of Instructors• Assist Chiefs of Mental Health
to Manage the Training Requirements
• Getting the Data to the Actors/Agents in real time
©HIMSS 2018-2019
20
Training Audits
©HIMSS 2018-2019
Training Audits
User Training
Train-The-Trainer users (TTT) trained
Super user trained (SUT)
Track training progress
eLearning
Online pre-training
Learning Management system will link to
database
Software Simulation
Self-Assessments
Web-based
Separated by classifications
Individual and Institutional summaries
Knowledge Checks
Web-based
Multiple Choice
End user and Process questions
©HIMSS 2018-2019
22
Knowledge Assessment
Ceiling Questions
Content Domains
Floor Questions
SuperUsers
Minimally
Competent
End Users
High
Opportunity
for Change
Knowledge Check - Design Strategy
©HIMSS 2018-2019
SharePointTeam Site
©HIMSS 2018-2019
Links to InfoPath Forms
Knowledge Check for Providers(Competency)
©HIMSS 2018-2019
InfoPath
25
Self Assessments
©HIMSS 2018-2019
26
EHRS Trainee Evaluation
©HIMSS 2018-2019
Super Users Train Mocks, Clinical Cutover, and RetrainingTraining Window
0%10%20%30%40%50%60%70%80%90%
100%
Staff Training Before Go-Live
Mental Health Provider Mental Health Scheduling Psychiatry
Training Completion
©HIMSS 2018-2019
TTTs
Tra
ined
Train-The-Trainer Certification Audit
• Determined by multiple data sources to a weighted algorithm, based on past successful• Self Report• Multiple Choice• Teach-backs
• TTT certification authorizes onsite trainings. Failure disqualifies them from conducting or assisting in any onsite EHRS trainings.
©HIMSS 2018-2019
The Ability to Monitor Progression of Training at the Institutional Sites by Class/Cohort
©HIMSS 2018-2019
Training Audits – Knowledge Checks
• Knowledge Checks Breakdown
• Grouped by MH Providers, MH Scheduling, and Psychiatry
• Top right corner indicates overall score, and each question has an institutional score.
• Institution used this audit for targeted booster training and ongoing training.
©HIMSS 2018-2019
Training Audits – Self Report
• Self-Report Breakdown
• Grouped by MH Providers, MH Scheduling, and Psychiatry
• Self-Report categories are color coded to easily identify trouble areas
©HIMSS 2018-2019
Executive OverviewTraining
©HIMSS 2018-2019
Training Composite ScoresThe knowledge checks were designed using test construction theory with the intention to discriminate the trainees into three groups: potential super-users, minimally competent users, and high opportunity for learning users.
©HIMSS 2018-2019
Data Integrity and Monitoring Alternative Test Taking Methodology
cheat/CHēt/
verb
gerund or present participle: cheating
1. act dishonestly or unfairly in order to gain an advantage, especially in a game or examination.
synonyms: swindle, defraud, deceive, trick, scam, dupe, hoodwink, double-cross, gull; rip off, con, fleece, shaft, hose, sting, bilk, diddle, rook, gyp, finagle, bamboozle, flimflam, put one over on, pull a fast one on, sucker, stiff, hornswoggle;
Creative Commons ©HIMSS 2018-2019
35
©HIMSS 2018-2019
36
Cutover
Another Training opportunity
Creative Commons ©HIMSS 2018-2019
37
1. Inpatient Medications
2. Levels of Care (Acuity)
3. Scheduling Appointments
4. Document Cutover
1. Initial Assessments
2. Suicide Risk and Self-harm Evaluations
3. Developmental Disability Program
Evaluations
4. Developmental Disability Program
Designation
Cutover Data
©HIMSS 2018-2019
38
1.Track required cutover requirements
2.Denominator is known
3.Develop required work projections to
meet cutover goals through frequent SQL
report updates
4.Get cutover analytics to providers and
local management
5.Escalate immediately when progress is
not meeting needed timelines
Cutover Analytics
©HIMSS 2018-2019
Clinical Cutover AuditRequires staff to input
PowerPlans for each non-GP patient.
The green line indicates a steady pace of cutover
with successful completion Friday before
Go-Live.
The White line is the institutions progress.
The Yellow line is an alert that suggests the pace of
cutover is suboptimal and may pose a risk for a
successful go live.
The Red line is an alert that suggests that the
pace of cutover is at risk of critical failure.
Requires staff to input PowerPlans for each non-
GP patient.
The green line indicates a steady pace of cutover
with successful completion Friday before
Go-Live.
The White line is the institutions progress.
The Yellow line is an alert that suggests the pace of
cutover is suboptimal and may pose a risk for a
successful go live.
The Red line is an alert that suggests that the
pace of cutover is at risk of critical failure.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
Minimum 57 115 172 230 287 344 402 459 516 574 631 689 746 803 861 918 976 103 109 114 120 126 132 137 143 149
Alert 344 387 430 473 516 560 603 646 689 732 775 818 861 904 947 990 103 107 111
High Risk 230 258 287 316 344 373 402 430 459 488 516 545 574 603 631 660 689 717 746
0
200
400
600
800
1000
1200
1400
Emai
l Au
dit
Sen
t
Progress Towards 100% Scheduling Cutover
©HIMSS 2018-2019
Progress of Document Cutover100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
CUT OVER WEEK 1
CUT OVER WEEK 2
CUT OVER WEEK 3
CUT OVER WEEK 4
Psychosocial Assessments
Suicide Risk Assessments
Developmental Disability
Assessments
Increased Provider Cutover = Better Prepared Provider
• An early institution had a difficult go live with providers feeling ill-prepared at go live.
• A review of the data showed that 10 (12%) users, completed 1,630 patients clinical cutover (over 80%) at the same institution.
©HIMSS 2018-2019
42
Go-Live/Adoption
Data Integrity – Workflow Adoption
Creative Commons©HIMSS 2018-2019
EHRS Go-Live/AdoptionData Entry Flags
• Data Entry Flags (Indicators)• Address critical faults, workflow deviations, and threats to data
integrity• Data is filtered and sent to stakeholders/project managers
©HIMSS 2018-2019
-
200
400
600
800
1,000
1,200
1,400
CCWF CMC CTF DVI SCC SVSP VSP©HIMSS 2018-2019
Region II: Data Entry Flags January 2017 To September
2017
143
14 -
50
100
150
200
250
300
350
400
January 2017 February 2017 March 2017 April 2017 May 2017 June 2017 July 2017 August 2017 September 2017
©HIMSS 2018-2019
Institution X Data Entry Flags September 2017
• MHI change during in-transit/temp departure 3• MHI not changed within 24 hours of DSH discharge 40• Inmate in CDCR over 30 days with no MHI entered since admission 3• MH referral resolved before it was received 90• Program/sub Program change during in-transit/temp departure 1• MH referral received date later than data entry date 1• MHCB Admission with wrong MHI 2
46
Creative Commons©HIMSS 2018-2019
Evidence Based Change Management
Evidence Based CM
Case Study CM
47
Adoption OutcomeMetrics
©HIMSS 2018-2019
Outcome
35/35Institutional Successful Go-lives
©HIMSS 2018-2019Creative Commons
8.3 million vendor transactions per
day
9,300 EHRS users per day
21 out of 344 of vendor clients ranked by transactions
Creative Commons
OutcomeSentinel Events
A Sentinel Event is defined by The Joint Commission (TJC) as any unanticipated event in a healthcare setting resulting in death or serious physical or psychological injury to a patient or patients, not related to the natural course of the patient's illness.25 percent
Excluding medication errors©HIMSS 2018-2019
decrease in sentinel events after go-live for Mental Health
40%
50%
60%
70%
80%
90%
100%
-6 -5 -4 -3 -2 -1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
Reference Mental Health Services
Scheduling and Access to Care
Go
-liv
e
©HIMSS 2018-2019
Pilot Institution Post Reboot Institution
Performance Reports
©HIMSS 2018-2019
“The EHRS has helped (the institution) do significantly better on their performance monitoring review (suicide prevention).”- Lindsay Hayes, National Expert on Suicide Prevention
©HIMSS 2018-2019
CDC_CA It has been awarded the 2018 Best of California Award as sponsored by
the Center for Digital Government:Best Application Serving an Agency’s
Business Needs Electronic Health Record System (EHRS),
California Correctional Health Care Services
2017 Statescoop 50- State IT Innovation of the Year
CCHCS Electronic Health Record System, California Correctional Health Care Services
2018 HIMSS Davies Community Award of Excellence©HIMSS 2018-2019
Outstanding Achievements
State IT
2018
55
Audience Poll – Continuous Quality Improvement
Feedback
Creative Commons
Questions?
Creative Commons©HIMSS 2018-2019
Contact Information
916-691-2640 office916-385-2965 Cell
[email protected] @drrekart
Please remember to complete the online session evaluation
Statewide Mental Health Program - California Correctional Health Care ServicesCalifornia Department of Corrections and Rehabilitation
Creative Commons