paul horstmeier and stuart gold...aug 30, 2018 · mike mastanduno, health catalyst simer sodhi...
TRANSCRIPT
Machine Learning Marketplace
Welcome Your Hosts
Paul Horstmeier and
Stuart Gold
Machine Learning Marketplace
Operationalizing Predictive Analytics Within Critical Care Environments
Dr. Craig Rusin, PhD
Assistant Professor, Baylor College of Medicine
Station A
33,366 Surgeries (2017)
800+ beds across 4 sites
3.7 Million patient encounters (2017)
One of the
Largest Children’s
Hospitals
in the World
Ranked #1 in Pediatric Cardiology (US News)
Now What?
We developed an algorithm to predict arrest in children with single ventricle physiology.
Web-based virtual patient monitor to watch for signs of arrest in single ventricle patients, built using Sickbay.
Meet Craig at Station A
Machine Learning Marketplace
A Natural Language Processing Toolkit for Healthcare:
Creating Happier Patients and Improving Healthcare Outcomes
Dr. Murtuza Ali Lakhani, DM
Senior Big Data Scientist, Sutter Health
Station B
Brief Background
All medical journeys are emotional in nature.
Emotional sense-making is a source of deep insight.
Can we do for patient experience what Nordstrom does for customer service?
Problem / Opportunity
Our patients go to great lengths to tell us how they feel.
The challenges in our ability to process them:
1. Scale / Volume
2. Bad Writing
3. Healthcare Vernacular
Machine Learning Approach
Fundamental Steps
1. Assemble comments from surveys, social media, and call centers.
2. Process them with RSentiment, CoreNLP, SentimentAnalysis. There are pros & cons to each.
3. Ensemble the results using a highly-tuned ML. Accuracy limited to low-to-mid 80%.
Needed Customizations
Dictionary-based language processing cannot handle healthcare parlance.
Supplemented with part-of-speech analysis and regular expressions, the solution delivers the desired result (> 95% accuracy).
Results and Lessons Learned
Value Delivered
1. Sentiment paints a highly accurate picture of
patient experience and is strongly related to
retention. Sentiment is a “leading indicator”
of business growth.
2. It decodes what is working and what is not
working in the system and enables us to act
quickly and precisely to address issues.
3. Allows benchmarking of affiliates based on
sentiment and has already become a part of
our performance systems.
4. It is a great resource not just for providers,
but also for ops teams to efficiently search
for relevant issues and trends –thus helping
improve outcomes and shaping org strategy.
Meet Ali at Station B
Machine Learning Marketplace
Machine Learning in the Real-World:Improving Accuracy of Readmission Risk Reporting
Enabling Service Line Reporting
Holly Burke, MHPA
Executive Director Clinical Innovation and Quality, Pulse Heart Institute, MultiCare
Dr. Needham Ward
Chief Medical Officer, Pulse Heart Institute, MultiCare
Station C
Coronary
Heart Failure
Heart Rhythm
Vascular
Structural Heart
Prevention
CardiologyCT Surgery
Vascular Surgery
MultiCare
Pulse Heart Institute
Pulse Heart Institute
Uniting cardiac, thoracic, and
vascular services into one
integrated entity, Pulse Heart
has a vision of becoming the
Pacific Northwest’s destination
for cardiovascular health. This
unique partnership has enabled
many advances in the quality
and cost of services provided.
Predicting Readmits
Total number of training cases: 69,000
Total number of input variables: 88
Final number of input variables used: 24
Number of daily predictions: ~150
Service Line Reporting
Total number of training cases: 3,401,457
Total number of input variables: 25
Final number of input variables used: 5
Service Line Accuracy: 98%
DRG assignment Accuracy: 94%
Results
• Restored service line reporting
• Avoided $1 million in labor costs
• Allowed for real time margin
optimization work
Meet Holly & Needham at Station C
Machine Learning Marketplace
No-Show Forecasting
Lixi Kong, MS
Senior Analyst, Dartmouth-Hitchcock
Station D
Problem / Opportunity
• Providers lose their valued time
and opportunities to see patients.
• Administrative staff have to
reschedule appointments.
• More than inconvenience, every
patient in no-show appointment status
is a percent of total potential revenue.
No Shows
Machine Learning Approach
Results and Lessons Learned
• Positive feedback from practice managers
and their team.
• Reduction in no-show rate for
departments utilizing the model.
• Challenges and plan for next steps.
Meet Lixi at Station D
Machine Learning Marketplace
Congratulations, It's a Model!
Now What?
(Technical debt involved in sustaining a production prediction model)
Andrew O. Johnson, PhD
Data Science Team, Mission Health System
Station E
A learning algorithm receives as input a training set S, sampled from an unknown distribution D and labeled by some target function f , and should output a predictor hs: X → Y… the goal of the algorithm is to find hsthat minimizes the error with respect to the unknown D and f.
Shalev-Shwartz, S., & Ben-David, S. (2016). Understanding machine learning: From theory to algorithms. New York: Cambridge University Press.
Variable Soup, by Craig Snodgrass: https://notsohumblepi.wordpress.com
What if these change over time?
Meet Andy at Station E
Machine Learning Marketplace
Claims-Based Machine Learning Applied to Opioid Use Disorder: Features Identified but Now What?
John Sanders, PhD
Chief Information Officer, Health Share of Oregon
Station F
Brief Background
Healthcare model: Health Share of Oregon is a Medicaid Coordinated Care Organization; our model is integration of service delivery across medical, behavioral and dental services, 320,000 members. We collaborate beyond the health system to generate better health outcomes in the communities we serve.
Mission: We partner with communities to achieve ongoing transformation, health equity and the best possible health for each individual.
Strategic focus: We use large integrated claims data sets to improve coordination of services and outcomes for members, particularly in early life health and behavioral health; for this machine learning effort, we focused on members with Opioid Use Disorders.
Problem / Opportunity
What was the problem/complication you were trying to solve?
Identification of members at risk of death and overdose from opioid prescribing.
How did you know this was a problem?
Oregon’s overdose and death rate are similar to nationwide numbers, and SUD is an especially high cost driver.
Why did your organization choose to prioritize this area of improvement?
Opioid use prevalence and cost; area of statewide focus.
Machine Learning Approach
How did you use machine learning?
Identified high opioid users and the
outcomes of interest (overdose and death);
explored potential time-based, demographic,
prescribing, clinical and utilization variables
linked to outcomes of interest.
Resources and time?
Health Catalyst data scientists, Health Share clinical & claims expertise, weekly virtual meetings.
What changed from your previous approach?
Initial outcomes of interest (OD and death) were not sufficiently prevalent in the data to refine the model; shifted to acute→ chronic prescribing.
Results and Lessons Learned
What did you learn?
Chronic use is associated with length of prescription (which we knew); seems to correlate with serious mental illness (which we did not know).
Engagement with clinical leaders across system is required; we shifted our thinking from descriptive toward predictive analytics.
What are your next steps?
Better use of trend features to create more nuanced patient profiles; explore SPMI link.
Apply to other data sets, populations, problems (how local is local?).
Refine how to explain and spread use of predictive analytics.
Meet John at Station F
Machine Learning Marketplace
Forecasting Inpatient Census For Operational Efficiency and Smarter Resource Allocation
Noah Geberer
Data Management & Analytics, Westchester Medical Center Health Network
Analytics Team: Mike Mastanduno, Health CatalystSimer Sodhi & Deborah Viola,Westchester Medical Center Health Network
Station G
Addressing Inpatient Surge
Scientific thought, then, is not momentary; it is
not a static instance; it is a process.
Jean Piaget (Psychologist)
• Persistent inpatient surges must be managed
• Smarter resource allocation increases revenue
• Appropriate staff loads lead to better patient care
Opportunity
• Predictive analytics are new, buy-in is hard
• Machine learning models require time to build and validate
Challenge
• Developed a machine learning model to forecast inpatient surge 7 days in advance
• Mixed in traditional analytics concurrently
• Iterated through business use, model development, user feedback
Approach
Mean
Rising volume
leads to historic
high
Success Through Ownership, Trust, and Action
https://englishharmony.com/wp-content/uploads/2014/04/speaking-english-like-lego-bricks.jpg
Implementation
Action Plan
Trust
Owned by
the Right
Champions
• Physician leadership, not
developers, took
ownership.
• Ensures long-term
survival of the work.
• 6 months of iteration through
business use, model
validation, and
operationalizing.
• Focused on consensus and
were willing to explore
alternatives.
• Shared decision-making and
required unanimous consent.
• Developed a trust between
stakeholders and analytics
team through data and
ability.
• Transparent plan, data,
model, assumptions, code,
and commitment.
• Made fixes and changes,
not excuses.
• On-the-ground
accountability comes from
senior staff.
• ML outputs put in context
to deliver the right
information at the right
time.
• Complementary action
plan gives direction and
motivates change.
Meet Noah at Station G
Machine Learning Marketplace
Clinical No-Show
Dane Hudelson
Director Enterprise Data & Analytics, Sanford Health
Station H
Q3 2017 Q4 2017 Q1 2018 Q2 2018 Q3 2018
July Aug Sept Oct Nov Dec Jan Feb Mar April May June July Aug Sept
Timeline
• Inception of Value Improvement Teams
• Management-level subject matter
experts tasked with optimizing quality
and financial metrics
• Clinical Team posed “How to Reduce No Shows”
• Enterprise Data & Analytics (EDA)
began model life cycle
• Information gathering
• Data analysis (3.4 Million
appointments per year)
• Evaluation of prior
published models
• Added a behavioral component
utilizing Item Response Theory
(IRT) commonly used in
psychometrics
• Improved c-statistic to .83 while
reducing variables to 17 and
decreasing extreme log-odds
• Began model development
• Initial proposed model:
logistic regression
• Evaluated 42 variables
• Reduced to 23 variables
• Enhanced regression to a
modified lasso logistic
• Used lasso to enhance feature
selection
• Reduction to 20
variables
• Model C-statistic of .789
• Created a pseudo-ensemble
applying Bayesian update
using Dirichlet Distribution
as posterior distribution
• Final model developed with
C-statistic of .801 • Began presenting model
outcomes to leadership
• Started planning for pilot
group and roll-out
• Pilot Began May 15th
• 30 clinics
• Model evaluating over
1000 patients per day
• Calling approximately 50 to
100 patients a day
• Evaluation of Pilot
• Pilot lasted 3 months with a
participation rate of 60% and
call rate of 70%
Results
PilotHigh-Risk No
Shows had a 20%no show rate after calling them and
confirming appointment
Clinics in 3 specialties
participating in the pilot had the
following changes in no show rates:
• Family Med: -7%
• Pediatrics: -1%
• ENT: +6%
Control
High Risk No Shows who didn’t receive a phone call had a 43% no-show rate
Control Clinics in 3 specialties had
the following changes in no show rates:
• Family Med: +6%
• Pediatrics: +12%
• ENT:
• +10%
Next StepsSecond Pilot
• In the Process of a Second pilot to test automated calling/text messaging
Analysis of Results
• Analyze the results from the two pilots:
• Determine optimal probabilities for different methods of intervention
Full-Scale Implementation
• Begin a continuous roll-out to include the remaining 380 clinics in the enterprise
Meet Dane at Station H
Machine Learning Marketplace
Using Machine Learning to Detect Errorsin Medical Data
Tommy Blanchard, PhD
Data Science Lead, Fresenius Medical Care North America
Station I
Model for Comorbidities
1 model
per
comorbidity
LabsPatient
treatment data
Demographics Rounding
notes
(keyword
search)
Other
comorbidities
Model for Comorbidities
Comorbid
Model
Myelodysplasia
Model
Chart review for
ANY comorbidity
GI Bleed
Model
GI
Pericarditis
Model
P M
Hemolytic
Anemia Model
HA
Sickle Cell
Anemia Model
SC
Probability of Comorbidities
Probability rank
Pro
ba
bili
ty
Meet Tommy at Station I
Machine Learning Marketplace
Risk Modeling for Falls in Value-Based Healthcare Using NLP and other Advanced Methods
Keegan Bailey
Strategy and Technology Leader, Acuitas Health
Dan Loman
Data Science Engineer, Acuitas Health
Francesca Romano
Data Science Engineer, Acuitas Health
Station J
#HASUMMIT18
Brief Background
Each year, more than one out of four people age 65+ fall in the U.S., but less than half tell their doctor.
One out of five falls cause a serious injury, including broken bones and head injuries, and 95% of hip fractures are caused by falling.
In 2015, the total medical costs for falls in the U.S. totaled more than $50 billion.
Problem / Opportunity
Current fall risk assessment methods require a face-to-face office visit and often only take a subset of important risk factors into account.
Can we identify patients age 65+ who are at risk for falls resulting in serious injuries based on EHR data and real time hospital activity alerts?
#HASUMMIT18
Machine Learning Approach
NLP techniques were applied to real time hospital activity alerts to retrospectively identify patients who had fallen.
A regularized XGBoost decision tree model was trained to classify patients at risk for falls based on falls identified with NLP.
The model was interpreted using SHAP to assess feature importance and interactions and using LIME to explain predictions.
Results and Lessons Learned
The initial model obtained a testing AUC of 0.7384. Of the high risk patients identified by the model who fell, over 58% were identified as low risk by traditional assessment methods.
Next steps:• Review misclassified patients for
improvements to NLP fall detection.
• Rerun model using a longer outcomes time period.
• Apply NLP to office notes and incorporate as features in model.
• Expand outcomes to falls documented in EHR.
Meet Keegan, Dan, & Francesca at Station J
Machine Learning Marketplace
Use the Healthcare Data Operating System (DOS™) to Turn Data Analysts Into Data Scientists
Imran Qureshi
Chief Software Development Officer, Health Catalyst
Station K
Native AI Support in DOS 18.2 (Sep 2018)
• DOS enables:
▪ Data analysts to do basic AI (with supervision from data scientists)
▪ Data scientists to spend less time on data wrangling and more on actual data science
Develop
model in
RStudio
Create
Features
in SAMD
Copy/Pas
te R code
in SAMD
Specify
variables to
load from
Feature
Tables
Upload
model
file(s)
Paste
sessionIn
fo for
package
list
Run SAMSee logs in
EdwConsole
DOS takes care of:
▪ Running hybrid SQL/R pipelines
▪ Piping data from full load or incremental load
▪ Saving output data frame
▪ Migration from Dev to Prod
▪ Dependency management
▪ Logging and monitoring
Feature
Engineering
using SQL
Use any entity in DOS as
a data source
Upload model files or
other files to DOS
DOS populates the destination entity
with the contents of this R data frame
Define packages and versions
and DOS can automatically
install the right versions
Target different R versions
Any R script can be pasted here
DOS manages dependencies automatically
R output is available in EDW Console
How DOS helps with the 80/20 dilemma
Define the use
case
Organize the
dataDevelop a model
Deploy the
model
Surface the insight
and guidance
• Curated data models that
are pre-populated from
various healthcare data
sources (EMRs, claims,
Billing etc.).
• Access to raw data from
200+ healthcare data
sources not just curated
data.
• Tools to make feature
engineering easy.
• Built-in NLP tools allow
access to 80% of useful
data in EMR (coming in
2019).
• Integrated
healthcare.ai library
that includes
common AI
algorithms optimized
for healthcare.
• Native AI in DOS
(with embedded R
and Python engines)
enabling you to mix
SQL and AI in the
same infrastructure.
• Production AI
pipelines can
be managed
by the same
people and
same tools as
ETL pipelines.
• EMR
Integration
enables
showing
insights in the
existing
workflows of
clinicians
(coming in
2019).
Health Catalyst Data Science Professional Services
• Health Catalyst
Data Science
Professional
Services can
help.
Meet Imran at Station K
Machine Learning Marketplace
Shaping the Future of Healthcare with Machine Learning: From Strategy to Operations
Justin Smith, PhD
Director of Data Analytics, Sanford Health
Station L
Data
Strategy Operations
RN
Turnover
HRC Recommendation
Engine
Meet Justin at Station L
The Walkabout Begins!
Session Feedback SurveyUse the “Feedback” tab for the session
On a scale of 1 to 5, how satisfied were you overall with this session?
1) Not at all satisfied
2) Somewhat satisfied
3) Moderately satisfied
4) Very satisfied
5) Extremely satisfied
What feedback or suggestions do you have?
Upcoming
Evening Activity Location
5:15 – 6:30 PM Closing Reception Grand America Garden Courtyard
Thursday Afternoon General Session Grand Ballroom
4:00 – 4:45 PM Kim Goodsell (Data-Empowered Patient)
4:45 – 5:00 PM Final Polls and Summit Winners
5:00 – 5:15 PM Closing Remarks (Dan Burton, CEO, Health Catalyst)