paul horstmeier and stuart gold...aug 30, 2018  · mike mastanduno, health catalyst simer sodhi...

Machine Learning Marketplace

Welcome Your Hosts

Paul Horstmeier and

Stuart Gold

Machine Learning Marketplace

Operationalizing Predictive Analytics Within Critical Care Environments

Dr. Craig Rusin, PhD

Assistant Professor, Baylor College of Medicine

Station A

33,366 Surgeries (2017)

800+ beds across 4 sites

3.7 Million patient encounters (2017)

One of the

Largest Children’s

Hospitals

in the World

Ranked #1 in Pediatric Cardiology (US News)

Now What?

We developed an algorithm to predict arrest in children with single ventricle physiology.

Web-based virtual patient monitor to watch for signs of arrest in single ventricle patients, built using Sickbay.

Meet Craig at Station A

Machine Learning Marketplace

A Natural Language Processing Toolkit for Healthcare:

Creating Happier Patients and Improving Healthcare Outcomes

Dr. Murtuza Ali Lakhani, DM

Senior Big Data Scientist, Sutter Health

Station B

Brief Background

All medical journeys are emotional in nature.

Emotional sense-making is a source of deep insight.

Can we do for patient experience what Nordstrom does for customer service?

Problem / Opportunity

Our patients go to great lengths to tell us how they feel.

The challenges in our ability to process them:

1. Scale / Volume

2. Bad Writing

3. Healthcare Vernacular

Machine Learning Approach

Fundamental Steps

1. Assemble comments from surveys, social media, and call centers.

2. Process them with RSentiment, CoreNLP, SentimentAnalysis. There are pros & cons to each.

3. Ensemble the results using a highly-tuned ML. Accuracy limited to low-to-mid 80%.

Needed Customizations

Dictionary-based language processing cannot handle healthcare parlance.

Supplemented with part-of-speech analysis and regular expressions, the solution delivers the desired result (> 95% accuracy).

Results and Lessons Learned

Value Delivered

1. Sentiment paints a highly accurate picture of

patient experience and is strongly related to

retention. Sentiment is a “leading indicator”

of business growth.

2. It decodes what is working and what is not

working in the system and enables us to act

quickly and precisely to address issues.

3. Allows benchmarking of affiliates based on

sentiment and has already become a part of

our performance systems.

4. It is a great resource not just for providers,

but also for ops teams to efficiently search

for relevant issues and trends –thus helping

improve outcomes and shaping org strategy.

Meet Ali at Station B

Machine Learning Marketplace

Machine Learning in the Real-World:Improving Accuracy of Readmission Risk Reporting

Enabling Service Line Reporting

Holly Burke, MHPA

Executive Director Clinical Innovation and Quality, Pulse Heart Institute, MultiCare

Dr. Needham Ward

Chief Medical Officer, Pulse Heart Institute, MultiCare

Station C

Coronary

Heart Failure

Heart Rhythm

Vascular

Structural Heart

Prevention

CardiologyCT Surgery

Vascular Surgery

MultiCare

Pulse Heart Institute

Pulse Heart Institute

Uniting cardiac, thoracic, and

vascular services into one

integrated entity, Pulse Heart

has a vision of becoming the

Pacific Northwest’s destination

for cardiovascular health. This

unique partnership has enabled

many advances in the quality

and cost of services provided.

Predicting Readmits

Total number of training cases: 69,000

Total number of input variables: 88

Final number of input variables used: 24

Number of daily predictions: ~150

Service Line Reporting

Total number of training cases: 3,401,457

Total number of input variables: 25

Final number of input variables used: 5

Service Line Accuracy: 98%

DRG assignment Accuracy: 94%

Results

• Restored service line reporting

• Avoided $1 million in labor costs

• Allowed for real time margin

optimization work

Meet Holly & Needham at Station C

Machine Learning Marketplace

No-Show Forecasting

Lixi Kong, MS

Senior Analyst, Dartmouth-Hitchcock

Station D

Problem / Opportunity

• Providers lose their valued time

and opportunities to see patients.

• Administrative staff have to

reschedule appointments.

• More than inconvenience, every

patient in no-show appointment status

is a percent of total potential revenue.

No Shows

Machine Learning Approach

Results and Lessons Learned

• Positive feedback from practice managers

and their team.

• Reduction in no-show rate for

departments utilizing the model.

• Challenges and plan for next steps.

Meet Lixi at Station D

Machine Learning Marketplace

Congratulations, It's a Model!

Now What?

(Technical debt involved in sustaining a production prediction model)

Andrew O. Johnson, PhD

Data Science Team, Mission Health System

Station E

A learning algorithm receives as input a training set S, sampled from an unknown distribution D and labeled by some target function f , and should output a predictor hs: X → Y… the goal of the algorithm is to find hsthat minimizes the error with respect to the unknown D and f.

Shalev-Shwartz, S., & Ben-David, S. (2016). Understanding machine learning: From theory to algorithms. New York: Cambridge University Press.

Variable Soup, by Craig Snodgrass: https://notsohumblepi.wordpress.com

What if these change over time?

Meet Andy at Station E

Machine Learning Marketplace

Claims-Based Machine Learning Applied to Opioid Use Disorder: Features Identified but Now What?

John Sanders, PhD

Chief Information Officer, Health Share of Oregon

Station F

Brief Background

Healthcare model: Health Share of Oregon is a Medicaid Coordinated Care Organization; our model is integration of service delivery across medical, behavioral and dental services, 320,000 members. We collaborate beyond the health system to generate better health outcomes in the communities we serve.

Mission: We partner with communities to achieve ongoing transformation, health equity and the best possible health for each individual.

Strategic focus: We use large integrated claims data sets to improve coordination of services and outcomes for members, particularly in early life health and behavioral health; for this machine learning effort, we focused on members with Opioid Use Disorders.

Problem / Opportunity

What was the problem/complication you were trying to solve?

Identification of members at risk of death and overdose from opioid prescribing.

How did you know this was a problem?

Oregon’s overdose and death rate are similar to nationwide numbers, and SUD is an especially high cost driver.

Why did your organization choose to prioritize this area of improvement?

Opioid use prevalence and cost; area of statewide focus.

Machine Learning Approach

How did you use machine learning?

Identified high opioid users and the

outcomes of interest (overdose and death);

explored potential time-based, demographic,

prescribing, clinical and utilization variables

linked to outcomes of interest.

Resources and time?

Health Catalyst data scientists, Health Share clinical & claims expertise, weekly virtual meetings.

What changed from your previous approach?

Initial outcomes of interest (OD and death) were not sufficiently prevalent in the data to refine the model; shifted to acute→ chronic prescribing.

Results and Lessons Learned

What did you learn?

Chronic use is associated with length of prescription (which we knew); seems to correlate with serious mental illness (which we did not know).

Engagement with clinical leaders across system is required; we shifted our thinking from descriptive toward predictive analytics.

What are your next steps?

Better use of trend features to create more nuanced patient profiles; explore SPMI link.

Apply to other data sets, populations, problems (how local is local?).

Refine how to explain and spread use of predictive analytics.

Meet John at Station F

Machine Learning Marketplace

Forecasting Inpatient Census For Operational Efficiency and Smarter Resource Allocation

Noah Geberer

Data Management & Analytics, Westchester Medical Center Health Network

Analytics Team: Mike Mastanduno, Health CatalystSimer Sodhi & Deborah Viola,Westchester Medical Center Health Network

Station G

Addressing Inpatient Surge

Scientific thought, then, is not momentary; it is

not a static instance; it is a process.

Jean Piaget (Psychologist)

• Persistent inpatient surges must be managed

• Smarter resource allocation increases revenue

• Appropriate staff loads lead to better patient care

Opportunity

• Predictive analytics are new, buy-in is hard

• Machine learning models require time to build and validate

Challenge

• Developed a machine learning model to forecast inpatient surge 7 days in advance

• Mixed in traditional analytics concurrently

• Iterated through business use, model development, user feedback

Approach

Mean

Rising volume

leads to historic

high

Success Through Ownership, Trust, and Action

https://englishharmony.com/wp-content/uploads/2014/04/speaking-english-like-lego-bricks.jpg

Implementation

Action Plan

Trust

Owned by

the Right

Champions

• Physician leadership, not

developers, took

ownership.

• Ensures long-term

survival of the work.

• 6 months of iteration through

business use, model

validation, and

operationalizing.

• Focused on consensus and

were willing to explore

alternatives.

• Shared decision-making and

required unanimous consent.

• Developed a trust between

stakeholders and analytics

team through data and

ability.

• Transparent plan, data,

model, assumptions, code,

and commitment.

• Made fixes and changes,

not excuses.

• On-the-ground

accountability comes from

senior staff.

• ML outputs put in context

to deliver the right

information at the right

time.

• Complementary action

plan gives direction and

motivates change.

Meet Noah at Station G

Machine Learning Marketplace

Clinical No-Show

Dane Hudelson

Director Enterprise Data & Analytics, Sanford Health

Station H

Q3 2017 Q4 2017 Q1 2018 Q2 2018 Q3 2018

July Aug Sept Oct Nov Dec Jan Feb Mar April May June July Aug Sept

Timeline

• Inception of Value Improvement Teams

• Management-level subject matter

experts tasked with optimizing quality

and financial metrics

• Clinical Team posed “How to Reduce No Shows”

• Enterprise Data & Analytics (EDA)

began model life cycle

• Information gathering

• Data analysis (3.4 Million

appointments per year)

• Evaluation of prior

published models

• Added a behavioral component

utilizing Item Response Theory

(IRT) commonly used in

psychometrics

• Improved c-statistic to .83 while

reducing variables to 17 and

decreasing extreme log-odds

• Began model development

• Initial proposed model:

logistic regression

• Evaluated 42 variables

• Reduced to 23 variables

• Enhanced regression to a

modified lasso logistic

• Used lasso to enhance feature

selection

• Reduction to 20

variables

• Model C-statistic of .789

• Created a pseudo-ensemble

applying Bayesian update

using Dirichlet Distribution

as posterior distribution

• Final model developed with

C-statistic of .801 • Began presenting model

outcomes to leadership

• Started planning for pilot

group and roll-out

• Pilot Began May 15th

• 30 clinics

• Model evaluating over

1000 patients per day

• Calling approximately 50 to

100 patients a day

• Evaluation of Pilot

• Pilot lasted 3 months with a

participation rate of 60% and

call rate of 70%

Results

PilotHigh-Risk No

Shows had a 20%no show rate after calling them and

confirming appointment

Clinics in 3 specialties

participating in the pilot had the

following changes in no show rates:

• Family Med: -7%

• Pediatrics: -1%

• ENT: +6%

Control

High Risk No Shows who didn’t receive a phone call had a 43% no-show rate

Control Clinics in 3 specialties had

the following changes in no show rates:

• Family Med: +6%

• Pediatrics: +12%

• ENT:

• +10%

Next StepsSecond Pilot

• In the Process of a Second pilot to test automated calling/text messaging

Analysis of Results

• Analyze the results from the two pilots:

• Determine optimal probabilities for different methods of intervention

Full-Scale Implementation

• Begin a continuous roll-out to include the remaining 380 clinics in the enterprise

Meet Dane at Station H

Machine Learning Marketplace

Using Machine Learning to Detect Errorsin Medical Data

Tommy Blanchard, PhD

Data Science Lead, Fresenius Medical Care North America

Station I

Model for Comorbidities

1 model

per

comorbidity

LabsPatient

treatment data

Demographics Rounding

notes

(keyword

search)

Other

comorbidities

Model for Comorbidities

Comorbid

Model

Myelodysplasia

Model

Chart review for

ANY comorbidity

GI Bleed

Model

GI

Pericarditis

Model

P M

Hemolytic

Anemia Model

HA

Sickle Cell

Anemia Model

SC

Probability of Comorbidities

Probability rank

Pro

ba

bili

ty

Meet Tommy at Station I

Machine Learning Marketplace

Risk Modeling for Falls in Value-Based Healthcare Using NLP and other Advanced Methods

Keegan Bailey

Strategy and Technology Leader, Acuitas Health

Dan Loman

Data Science Engineer, Acuitas Health

Francesca Romano

Data Science Engineer, Acuitas Health

Station J

#HASUMMIT18

Brief Background

Each year, more than one out of four people age 65+ fall in the U.S., but less than half tell their doctor.

One out of five falls cause a serious injury, including broken bones and head injuries, and 95% of hip fractures are caused by falling.

In 2015, the total medical costs for falls in the U.S. totaled more than $50 billion.

Problem / Opportunity

Current fall risk assessment methods require a face-to-face office visit and often only take a subset of important risk factors into account.

Can we identify patients age 65+ who are at risk for falls resulting in serious injuries based on EHR data and real time hospital activity alerts?

#HASUMMIT18

Machine Learning Approach

NLP techniques were applied to real time hospital activity alerts to retrospectively identify patients who had fallen.

A regularized XGBoost decision tree model was trained to classify patients at risk for falls based on falls identified with NLP.

The model was interpreted using SHAP to assess feature importance and interactions and using LIME to explain predictions.

Results and Lessons Learned

The initial model obtained a testing AUC of 0.7384. Of the high risk patients identified by the model who fell, over 58% were identified as low risk by traditional assessment methods.

Next steps:• Review misclassified patients for

improvements to NLP fall detection.

• Rerun model using a longer outcomes time period.

• Apply NLP to office notes and incorporate as features in model.

• Expand outcomes to falls documented in EHR.

Meet Keegan, Dan, & Francesca at Station J

Machine Learning Marketplace

Use the Healthcare Data Operating System (DOS™) to Turn Data Analysts Into Data Scientists

Imran Qureshi

Chief Software Development Officer, Health Catalyst

Station K

Native AI Support in DOS 18.2 (Sep 2018)

• DOS enables:

▪ Data analysts to do basic AI (with supervision from data scientists)

▪ Data scientists to spend less time on data wrangling and more on actual data science

Develop

model in

RStudio

Create

Features

in SAMD

Copy/Pas

te R code

in SAMD

Specify

variables to

load from

Feature

Tables

Upload

model

file(s)

Paste

sessionIn

fo for

package

list

Run SAMSee logs in

EdwConsole

DOS takes care of:

▪ Running hybrid SQL/R pipelines

▪ Piping data from full load or incremental load

▪ Saving output data frame

▪ Migration from Dev to Prod

▪ Dependency management

▪ Logging and monitoring

Feature

Engineering

using SQL

Use any entity in DOS as

a data source

Upload model files or

other files to DOS

DOS populates the destination entity

with the contents of this R data frame

Define packages and versions

and DOS can automatically

install the right versions

Target different R versions

Any R script can be pasted here

DOS manages dependencies automatically

R output is available in EDW Console

How DOS helps with the 80/20 dilemma

Define the use

case

Organize the

dataDevelop a model

Deploy the

model

Surface the insight

and guidance

• Curated data models that

are pre-populated from

various healthcare data

sources (EMRs, claims,

Billing etc.).

• Access to raw data from

200+ healthcare data

sources not just curated

data.

• Tools to make feature

engineering easy.

• Built-in NLP tools allow

access to 80% of useful

data in EMR (coming in

2019).

• Integrated

healthcare.ai library

that includes

common AI

algorithms optimized

for healthcare.

• Native AI in DOS

(with embedded R

and Python engines)

enabling you to mix

SQL and AI in the

same infrastructure.

• Production AI

pipelines can

be managed

by the same

people and

same tools as

ETL pipelines.

• EMR

Integration

enables

showing

insights in the

existing

workflows of

clinicians

(coming in

2019).

Health Catalyst Data Science Professional Services

• Health Catalyst

Data Science

Professional

Services can

help.

Meet Imran at Station K

Machine Learning Marketplace

Shaping the Future of Healthcare with Machine Learning: From Strategy to Operations

Justin Smith, PhD

Director of Data Analytics, Sanford Health

Station L

Data

Strategy Operations

RN

Turnover

HRC Recommendation

Engine

Meet Justin at Station L

The Walkabout Begins!

Session Feedback SurveyUse the “Feedback” tab for the session

On a scale of 1 to 5, how satisfied were you overall with this session?

1) Not at all satisfied

2) Somewhat satisfied

3) Moderately satisfied

4) Very satisfied

5) Extremely satisfied

What feedback or suggestions do you have?

Upcoming

Evening Activity Location

5:15 – 6:30 PM Closing Reception Grand America Garden Courtyard

Thursday Afternoon General Session Grand Ballroom

4:00 – 4:45 PM Kim Goodsell (Data-Empowered Patient)

4:45 – 5:00 PM Final Polls and Summit Winners

5:00 – 5:15 PM Closing Remarks (Dan Burton, CEO, Health Catalyst)