the decision machine - advisory · 2016 2030 2016 2030 2016 2030 2016 2030 2016 2030 australia2...
Post on 26-May-2020
56 Views
Preview:
TRANSCRIPT
Health Care IT Advisor
The Decision MachineAnalytics and the Rise of AI
Greg Kuhnen
Senior Research Director
kuhnengadvisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
2
Manage Your Audio
Use Telephone
If you select the ldquoTelephonerdquo option please
use the dial-in phone number and access
code provided on your GoTo panel
If you select the ldquoMic amp Speakersrdquo
option please be sure to check that your
speakersheadphones are connected
Use Microphone and Speakers
All attendees will be muted during the presentation
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
3
Manage Your GoTo Panel
Ask a Question
To ask a question please type it into the
ldquoQuestionsrdquo box on the GoTo panel and
press ldquoSendrdquo
Use the orange and white arrow to
minimize and maximize the GoTo panel
Use the monitor button to toggle
between Fullscreen and window mode
Minimize and Maximize Your Screen
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
4
2018 Health Care IT Advisor Virtual Summit
Beyond Meaningful Use and Operational Excellence
Virtual Summit Agenda
Source Health Care IT Advisor research and analysis
Final session September 6th at 300pm ET
Available on-demand to Advisory Board Members
The Decision Machine Analytics and the Rise of Artificial Intelligence
Available on-demand to Advisory Board Members
State of the Union
Available on-demand to Advisory Board Members
ITrsquos Role in the New Cost Control Mandate
Available on-demand to Advisory Board Members
Digital Health Systems The Innovation Journey Continues
Available on-demand to Advisory Board Members
The Era of the Connected Patient Patient-Generated Health Data
and Social Determinants of Health
See the 2018 Virtual Summit
Health Care IT Advisor
The Decision MachineAnalytics and the Rise of AI
Greg Kuhnen
Senior Research Director
kuhnengadvisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
6
Sources ldquoAnalyticsrdquo Wikipedia last modified March 5 2018
httpsenwikipediaorgwikiAnalytics English Oxford Living Dictionaries Oxford Oxford
University Press 2018 (also available at httpsenoxforddictionariescom) Health Care IT Advisor
research and analysis
Analytics
The discovery interpretation and communication of meaningful
patterns in data
ndashWikipedia
Artificial Intelligence (AI)
The theory and development of computer systems able to
perform tasks normally requiring human intelligencehellip
ndashOxford English Dictionary
Machine Learning (ML)
The field of study that gives computers the ability to learn
without being explicitly programmed
ndashArthur Samuel 1959
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP7
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
8
Health Economics An Unsustainable Trajectory
Analytics Can Be a Powerful Tool for Bending the Cost Curve
Sources Cottarelli C ldquoMacro-Fiscal Implications of Health Care Reform in Advanced and Emerging Economiesrdquo IMF
Fiscal Affairs Department December 2010 McKinsey amp Company ldquomHealth A new vision for healthcarerdquo 2010
Office of Economic Co-operation and development lsquoHealth Spendingrdquo Health Care IT Advisor research and analysis
1) Global Burden of Disease Health Financing Collaborator Network ldquoFuture and potential spending on
health 2015-40 development assistance for health and government prepaid private and out-of-
pocket health spending in 184 countriesrdquo The Lancet 389 No 10083 (2016) 2) According to the
Australian Institute of Health and Welfare (AIHW) calculations 3) New Zealand Ministry of Health
projections 4) GDP = Gross domestic product
Per Capita Health Care Spending1 Health Expenditures as Percentage of GDP4
2016 20302016 2030
2016 2030 2016 2030
2016 2030
Australia2
96144
2010 2030
New Zealand3
62 89
106159
110
180
FranceCanada
United States United Kingdom
172249
97135
$9237
$5234
$4751
$4576
$4032
$4050
$3749
$3096
$12448
$7799
$6437
$5926
$5606
$5496
$5002
$4245
United States
Netherlands
Belgium
Canada
Australia
New Zealand
UnitedKingdom
Spain
Health Expenditure per Capita 2014
Health Expenditure per Capita 2030
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
10
Technology Drives the Flood of Data
Smartphones and Genomes and Sensors Oh My
Source Health Care IT Advisor research and analysis
1) IoT = Internet of things
2) RTLS = Real-time locating system
3) ROI = Return on investment
Common Barriers to
Adoption of New Sources
bull Privacy security concerns
bull Immature technology
bull Lack of budget
bull Unclear benefit ROI3
bull Staff already overwhelmed
with data
A Sampling of New Data Sources
Integrated Partners
(eg social services
cross-continuum care)
Data from Outside
Organizations
(eg social)
Environmental Data
(eg IoT1 RTLS2
temperature air
quality)
Genomes Proteomes
Microbiomes
Metabolomes
Clinical Surveillance
(eg public health
outbreak reports)
Remote Patient Monitoring
(eg bedside monitors
remote patient monitoring
consumer wearables)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
11
Cognitive Overload and Clinician Burnout
Sources Millett D Exclusive ldquoThree GPs a day seeking help for burnout
official data showrdquo GP Jan 2018 Health Care IT Advisor research and
analysis
1) Obermeyer Z Lee T ldquoLost in Thought ndash The Limits of the Human Mind and
the Future of Medicinerdquo N Engl J Med 377 no 13 (2017)1209-1211
Tim
e R
eq
uir
ed
ldquoMedical thinking has become vastly more complex mirroring changes in
our patients our health care system and medical science The complexity
of medicine now exceeds the capacity of the human mind1rdquo
Ziad Obermeyer MD and Thomas Lee MD
Number of Considerations for Decision Making
Significant
opportunity for
AI and analytics
Augmented Intelligence Reduces Cognitive Burden
Reliance on heuristics rules of thumb ldquogood-enoughrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
12
Data Power the Decision Engine
Source Health Care IT Advisor research and analysis
Four Levels of Analytic Capabilities
Data
Inform
Recommend
Decision
Automation
Decision
Support
Descriptive
What happened
How many of our
patients were
readmitted last
month
What is likely to happen
Is this patient likely to
readmit What do we
expect seasonally
Prescriptive
Recommend or
decide the best action
What should we do
to address this
patientrsquos risks
PredictiveDiagnostic
Why did it happen
Rates spiked was there
a change in case mix
Demographics
Decide
Hum
an
Decis
ion
Analy
tics
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP13
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
14
AI Are We There Yet
And if Not When
Source Health Care IT Advisor research and analysis
We May Be at the Start of an Explosive
Growth in AI Capabilities
Incremental Growth
AI continues to make
small advances
delivering value in
niche applicationsAI Winters AI has
already gone through
past phases of hype and
troughs of disillusionment
Exponential Growth
AI rapidly gains
capability and becomes
a mainstream part of
most digital systems
Unpredictable Timing Some
advances never seem to arrive
(conversational systems)
while others take off
unexpectedly (smartphones)
60s 70s 80s 90s 2000 2010 2020 203050s 2040 2050
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
15
Fuel for the AI Fire
Source Health Care IT Advisor research and analysis
1) EMR = Electronic medical record
2) GPU = Graphics processing unit
3) TPU = Tensor processing unit
AlgorithmsData
bull Internet Everywhere
bull Broad EMR1 Adoption
bull Wearables
bull Real-Time Location
bull IoT Sensors
bull Genomics Proteomics
Microbiome
bull Automated Feature
Extraction
bull Cluster Compute
Platforms (eg
Spark Hadoop)
bull Deep Learning
bull Off-the-Shelf
Libraries for
Common Tasksbull Faster Processors
bull Specialised Processors (GPUs2 TPUs3)
bull Inexpensive Storage
bull Elastic Cloud Computing
Hardware
Combinatorial Advances Delivering
Radically Better Models
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
16
The AI Winter Is Thawing
Source Health Care IT Advisor research and analysis
1) NASA = National Aeronautics and Space Administration
Microsoft and Alibaba outscore humans in a Stanford reading comprehension test 2018
Google DeepMindrsquos AlphaZero convincingly leads at Chess Shogi Go 2017
Skype launches real-time voice translation 2014
Mass market voice assistants (Apple Siri Google Now Microsoft Cortana) 2012
IBM Watson conclusively beats Jeopardy game show champions 2011
NASArsquos1 Mars Rovers operate semi-autonomously 2004
IBMrsquos Deep Blue beats world chess champion Gary Kasparov 1997
1950 Alan Turing proposes the ldquoTuring Testrdquo for conversational systems
1956 The term ldquoArtificial Intelligencerdquo is coined
1970s First AI Winter
1981 Digital Equipment Corporation order
expert saves company $40M per year
Timeline of AI Successes
From Toys to Tools
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
17
DeepMindrsquos AlphaZero
Novice to Superhuman in Just Four Hours
DeepMindrsquos AlphaZero Uses lsquoZero Human Knowledgersquo
Sources Vincent J ldquoDeepMindrsquos AI became a superhuman chess player in a few hours just for funrdquo The
Verge December 6 2017 Gershgorn D ldquoDeepMind wants to find the next miracle materialmdashexperts just
donrsquot know theyrsquoll pull it offrdquo Quartz October 25 2017 Health Care IT Advisor research and analysis
By not using this human data by not using
human features or human expertise in any
fashion wersquove actually rem
oved the constraints of human knowledge Itrsquos
able to therefore create knowledge for itselfrdquo
bull AI product of Google
subsidiary DeepMind
bull Provided no prior
knowledge of the
game given only basic
game rules and an
objective (win)
bull Learns through self-play
at an accelerated pace
(lsquoreinforced learningrsquo)
searches 80 thousand
positions per second
bull Variants of AlphaZero also
defeated world champion
Go and Shogi systems
AlphaZero
outperformed chess
world-champion
Stockfish1 in just
4 hours (300k steps)
By not using this human data by not using human
features or human expertise in any fashion wersquove
actually removed the constraints of human knowledge
Itrsquos able to therefore create knowledge for itselfrdquo
David Silver Lead Programmer
DEEPMIND
1) Stockfish is the 2016 Top Chess Engine Championship (TCEC) world-champion
AlphaZero Chess
Performance
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
18
Need a Radiologist
Therersquos an App for That
Sources Kubota T ldquoStanford algorithm can diagnose pneumonia better than
radiologistsrdquo Stanford News Nov 2017 Health Care IT Advisor research and analysis
1) Stanford University Medical Center
2) NIH = National Institutes of Health
Case in Brief Stanford University1
CheXNet Algorithm
bull Algorithm developed by researchers at
Stanford University in the US can diagnose
14 common pathologies in chest x-rays
bull Trained on ChestX-ray14 a public data set
released by the NIH2 containing 112120
frontal-view chest x-ray images labelled with
the 14 possible pathologies
bull Outperforms previous models from the same
data set for all 14 conditions and diagnoses
pneumonia at an accuracy exceeding the
performance of four control radiologists
bull Produces heatmaps that visualize the areas
of an image most indicative of disease
Development of CheXNet
Sept 26 2017
ChestX-ray14 data set
released along with a
preliminary algorithm
that could detect the
labelled conditionsasymp One week later
CheXNet could
diagnose 10 of the
14 pathologies more
accurately than all
previous algorithmsasymp One month later
CheXNet surpassed
best published results
for all 14 pathologies
and outperformed
Stanford radiologists in
detecting pneumonialdquoI treat the model as a lazy resident Theyrsquore not dumb theyrsquoll
just use every trick they can find to avoid doing hard workrdquo
Dr Matthew Lungren RadiologistStanford University Medical Center
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
19
Lessons from Stanfordrsquos CheXNet
AI Is Effective on Narrow Tasks
Recent advances in AI have focused on
solving narrowly defined tasks Each
sub-problem (eg pneumonia) is a model
Handle ldquoNormalrdquo Explicitly
Identifying normal conditions versus
anomalies can help recognize when
human review may be needed
Explainable1 AI (XAI)
Models that can be interpreted at least
partially are easier to trust deploy and
govern Explainability is an area of rapid
progress in AI research
1) Also known as interpretable AI
Training Requires a Teacher or Referee
Machine learning needs to be told what
answers are correct or what actions produce
positive outcomes Well-curated data are
critical to the learning process
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
20
The AI Value Proposition
AI Excels at Decisions That Are Dull Detailed and Dear (Expensive)
Three Key Areas of Benefit
Dear
Provide speed capacity 24x7
availability and consistency
at a fraction of the cost
Although these are well
known benefits of automation
when you apply this to
cognitive and complex tasks
itrsquos a new ball game
Detailed
Evaluate a vastly broader
and deeper set of data to
improve decision quality
The complexity of health
care decision making is
growing due to new data
from genetics IoT devices
wearables better
interoperability and the
growing number of drugs
and treatments
Unlimited capacity
instant response
disruptive economics
Perform at super-human
levels exceeding experts
Serves as an assistant
(watches your back)
Dull
Free human decision makers
to focus on more challenging
ldquotop of licenserdquo activities
Reduce effort spent on
repetitive tasks process
large amounts of complex
information provide
continuous monitoring and
notifications and ensure
ldquostuffrdquo doesnrsquot ldquofall through
the cracksrdquo
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
21
Where to Start Where Wersquore Going
1) Narrow (Task Weak) AI focuses on narrowly defined problems such as predicting a patientrsquos risk of developing sepsis
2) General (Strong) AI attempts to match the flexibility and versatility of human intelligence
Narr
ow
(T
ask W
ea
k)
AI1
Clinical Decisions
Administrative Decisions
Ge
ne
ral (S
tro
ng
) A
I2
Full Diagnosis
and Treatment Plan
Clinical
Chatbots
Radiology
Interpretation
Acute
Clinical
Risk
Recruiting
Safety Risk Smart
Monitoring
Medication Dosing
Administrative
Chatbots
(Concierge)
Precision
Engagement
Precision
Medicine
Ambient
Documentation
Adaptive User
Interfaces
Worklist
Prioritization
Inevitable with Time
Todayrsquos Opportunity
(growing)
The Computer Will
See You Now
The Uber-ization of
Health Care
An AI Application Landscape
Capacity Staffing
Management
Population
Health Risk
The Self-Driving
HospitalInformation
Security
Targeted
Marketing
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
22
The Self-Driving Hospital
Combine Predictive Models with Front-Line Activation
Sources ldquoVirtual Air Traffic Control for Emergency Departmentsrdquo Qventus
httpsqventuscomcontentuploads201703QVE-ED-Solution-Briefpdf Health Care
IT Advisor research and analysis
1) EVS = Environmental services
2) LWBS = Left without being seen
3) LOS = Length of stay
Case in Brief
Mercy Hospital Fort Smith
bull 336-bed acute care hospital in Fort
Smith AR
bull Partnered with Qventus to improve
emergency department (ED) patient flow
and patient satisfaction
bull Analytics applies ML algorithms to
anticipate potential capacity shortfalls
process bottlenecks
bull Application sends real-time notifications
(nudges) to care teams via their mobile
phones to trigger specific interventions
bull Emergency department results
ndash LWBS2 rates dropped 30
ndash Door-to-doc time reduced by 20
ndash 24-minute reduction in average LOS3
ndash Annual case capacity increased by 6
See that a radiology order is delayed for a
patient within a few days of discharge and
notify radiology to prioritize the reading
Activate more EVS1 staff when high churn
of beds is anticipated
Send an early warning for difficult patient
placements at discharge
Prioritize inpatient transports depending
on expected bed needs
Sample ldquoDecision Recipesrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
23
Applicants Like You Liked This Job
Predictive Models Reduce Turnover Improve Engagement
Case in Brief MultiCare Health System
bull Non-profit system based in Tacoma WA with 16000 employees
bull Partnered with Arena a cloud-based HR solution to predict the
likelihood that a candidate will be retained in a role
bull Arenarsquos machine learning algorithms evaluate applicant
submissions digital interactions and externally sourced data on
the organization (eg employer review sites)
bull Achieved a 40 reduction in RN turnover at 180 days and a
28 reduction in overall turnover at 180 days
MultiCarersquos Hiring Process
Reduced recruiting
expenses
More predictable
staffing levels
More experienced
staff
Realized Benefits
Algorithms predict the likelihood the
applicant will be retained and
engaged in the target role and send
that information to the recruiter
Applicant completes
a 15-20 minute
assessment process
on Arenarsquos platform
Recruiters determine
whether or not to send
the candidate along to
hiring managers
Hiring managers use
Arenarsquos recommendation
as an optional factor in
the hiring decision
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
24
Accelerating Research
1) Yu KH et al ldquoPredicting non-small cell lung cancer prognosis by fully automated microscopic pathology image featuresrdquo Nature No 12474 (2016)
2) Rajwa B et al ldquoAutomated assessment of disease progression in acute myeloid leukemia by probabilistic analysis of flow cytometry datardquo IEEE Transactions on Biomedical Engineering 64 No 5 (2017) 1089-1098
3) Liu Y et al Detecting cancer metastases on gigapixel pathology images arXiv preprint arXiv1703 No 02442 (2017)
4) Rajpurkar P et al Cardiologist-level arrhythmia detection with convolutional neural networks arXiv preprint arXiv1707 No 01836 (2017)
5) Nielsen S et al A predictive model to identify Parkinson disease from administrative claims data Neurology 89 No 14 (2017) 1448-1456
6) Rajpurkar P et al CheXNet Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning arXiv preprint arXiv 1711 No 05225 (2017)
7) Poplin R et al Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning Nature Biomedical Engineering No 2 (2018)
8) ACC Machine learning techniques show promise for supporting medical decisions By quickly assessing risk new algorithms may help triage patients and inform clinical decisions ScienceDaily 2018
9) Oh J et al ldquoA Generalizable Data-Driven Approach to Predict Daily Risk of Clostridium difficile Infection at Two Large Academic Health Centersrdquo ICHE 39 No 4 (2018) 425-433
Nature1 ndash August 2016
ML algorithm can accurately
differentiate two types of lung
cancers and predict patient
survival time
IEEE2 ndash February 2017
ML model predicts relapse rates for
acute myelogenous leukemia with
90 accuracy
arXiv3 ndash March 2017
ML approach to imaging analytics
can identify metastasized breast
cancer with accuracy rates rivaling
those of pathologists
Neurology5 ndash Sept 2017
Predictive model to identify
Parkinsonrsquos disease from claims
data achieved 73-83 accuracy in
study results
arXiv4 ndash July 2017
ML algorithm can diagnose
heart arrhythmias using data
from wearables sensors with
cardiologist-level accuracy
arXiv6 ndash November 2017
Deep-learning algorithm
outperforms Stanford radiologists
at diagnosing pneumonia from
x-rays
American College of Cardiology8 ndash
Feb 2018
Model predicts heart attack
diagnosis among patients in the ED
with chest pain with ~90 accuracy
Nature7 ndash February 2018
ML algorithm can predict a
personrsquos cardiovascular risk
factors using eye scans
Infection Control Hospital
Epidemiology9 ndash April 2018
ML model predicts Clostridium
difficile infection (CDI) earlier
than current diagnostic methods
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
25
Driving the Decision Machine
1Acquire Data
bull Ensure all models have
a sponsor and regular
evaluation against goals
bull All models require
tune-ups as the
environment changes
Monitor Performance
and Revisit
42Train or Refine
the Model
bull The more the better
buthellip
bull High-quality well-
governed data pay
enormous dividends
bull Diverse cross-continuum
sources improve
predictions and
robustness of models
bull Decide what features
are important (eg
explainability)
bull Apply practical and
ethical constraints
bull Evaluate under
realistic conditions
(silent mode real-time
incomplete data)
bull Embed insights directly
into applications at key
decision points
bull Reengineer processes
as warranted
bull Consider the ldquo5 rightsrdquo
of decision support
3Incorporate Models
into Workflow
Hint The Analytics Is the Easy Part
Begin with
bull Explicit leadership agreement on outcome goals expected benefit mechanisms
bull Participation from empowered representatives of impacted process
bull External experiencemdashlearn from others
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
26
New Data Sources
Mainstream Systems Cover Only a Fraction of Predictive Power
Acquire Data
Sources Nash DB ldquoPopulation Health Why It Mattersrdquo Essentials in Population Health Childrenrsquos
Hospital Association Thomas Jefferson University October 2016 Health Care IT Advisor research and
analysis
1) RPM = Remote patient monitoring
EMR(s)
Lab Systems
Pharmacy Systems
Imaging
Medical Claims
Prescription Drug Claims
Billing and Supply Chain
Scheduling Systems
Medical
Care Data
10
Sensors RPM1
mHealth Apps
Patient-Reported
Outcomes
Genetics
Socioeconomic Data
Patient Questionnaires
Human
Biology
Data
20
Lifestyle
and
Behavior
Data
50
Social and
Environmental
Data
20
Data Sources by Influence on Health Outcomes
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
27
Big Data a Technology and a Philosophy
ldquoTherersquos Gold in Them Thar Hillsrdquo
Acquire Data
Sources Cloudera ldquoInova Translational Medicine Institute Partners with Cloudera to Advance Genome-Based Machine
Learning Initiatives and Save Livesrdquo PR Newswire June 2017 Health Care IT Advisor research and analysis
1) Originally coined by Doug Laney of the Meta Group
Assumes Value in the Data
Based on a premise that there are
valuable insights that can be distilled
from the mountains of data captured
by machines
Characterised by the 3 Vs1
bull Volume
bull Velocity
bull Variety
Places greater emphasis on
unstructured data
Free text images streams of sensor
data are all fair targets for analysis
Big data is the difference
between heart rates and
heart beats
Case in Brief Inova Health System
bull Inovarsquos Translational Medicine Institute
(ITMI) is assembling one of the worldrsquos
largest whole genome sequence
databases connected to patient
information in a health care system
bull Researchers use the Cloudera
machine learning platform to analyze
terabytes of clinical and genomic data
and identify genetic links to diseases
bull Insights are used to develop
personalized treatment plans for
patients in collaboration with the
treating physician
Resource Big Data in Health Care
Strengths challenges and pragmatic
guidelines for those considering the addition
of big data capabilities to their IT portfolio
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
30
How Machines Learn
Similar Methods Differing Goals
Build the Model
Source Advisory Board research and analysis
Input Data
bull The more the better
bull Labeled with key indicators
Adjust the Model
bull Automated trial and error
bull Cross-breeding successful
candidates
Repeat
Supervised Learning
Evaluate models against the ldquocorrectrdquo answers
provided with the data The learning process
focuses on minimizing errors
Unsupervised Learning
Evaluate models against an abstract algorithmic
goal Example Divide this population of diabetics
into a handful of subpopulations that have similar
future risk profiles
Reinforcement Learning
Evaluate models by giving a reward for
desirable outcomes and a penalty for negative
outcomes Models are trained to maximize their
total rewards over timeReady for Review
Evaluate Performance
bull Against what goal
bull Forces explicit
tradeoffs (eg cost
vs quality)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
32
Black Boxes Shades of Grey
Differing Views on the Importance of Explainability
Incorporate into Workflow
Mike Wall PharmD MBA
Chief Analytics Officer
The University of Chicago Medicine
The opaqueness of black-box models is
really scary When we canrsquot tell a
clinician why a patient wonrsquot do well on a
certain medication that can be a tough
thing to blindly trust from a clinical
perspective We want to reach a place
where we augment our practice
through AI rather than dictatingrdquo
Nigam Shah MBBS PhD
Associate Professor of Medicine
(Biomedical Informatics)
Stanford University
I am pushing to evaluate these models
via a similar set of mechanisms as RCT1
or AB testing We should evaluate
against the current status-quo and do a
beforeafter studyhellipWe have to convince
ourselves of the utility of these models
We should focus on utility before
explainabilityrdquo
Surgery is the
best option ()
Data
Black Box Model
1) RCT = Randomized controlled trial
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
33
Workflow Is Key to Adoption
Incorporate into Workflow
The Right Information
To the Right Person
In the Right Form
By the Right Channel
At the Right Time
Case in Brief Mayo Breast Oncology
Clinic Trials Recruitment
bull Up to 100 concurrent drug therapy clinical trials are
available to Mayo Clinic breast cancer patients making
it challenging to consistently identify candidates
bull New clinical trial matching technology developed in
collaboration with IBM Watson Health evaluates charts
to identify eligible patients
bull Launched in January 2016 as a point-of-care solution
for oncology providers initial adoption was low due to
time pressures and fell to zero by February
bull Rework by health systems engineers resulted in a July
2016 relaunch screening moved to occur in
advance of patient visits so clinicians know of trial
matches before opening a patientrsquos chart
bull New process enrolls significantly more patients in
clinical trials than the prior manual process success led
to additional coordinator funding
The Five Rights of
Clinical Decision Support
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
34
Health Carersquos Field Agents
Medical Devices Become Intelligent Partners
Incorporate into Workflow
1) EKG = Electrocardiogram
ME
DT
RO
NIC
MIN
IME
D 6
70G
Case in Brief Artificial Pancreas and Smart Infusion Pumpsmdash
Medtronic MiniMed Connect
bull SMARTGUARD mimics some functions of a healthy pancreas
measures and predicts glucose level drift and adjusts dosing
bull Insulin pump and continuous glucose monitoring can send data directly
to smartphone
bull Trials showed a 44 reduction in hypoglycemia and reduction in
average A1C values from 74 to 69
Case in Brief AliveCor KardiaMobile
bull Portable consumer-grade single-lead EKG1 retails for $100
bull Intended for self-capture of intermittent cardiac events such as atrial
fibrillation (AFIB)
bull Smartphone-based machine learning validates quality of trace
assesses possibility of AFIB classifies normal abnormal sinus rhythm
bull EKGs can be sent to a board-certified cardiologist for interpretation
for $19
Sources alivecorcom Medtroniccom Health Care IT Advisor research and
analysis
AliveC
orK
ard
iaM
obile
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
2
Manage Your Audio
Use Telephone
If you select the ldquoTelephonerdquo option please
use the dial-in phone number and access
code provided on your GoTo panel
If you select the ldquoMic amp Speakersrdquo
option please be sure to check that your
speakersheadphones are connected
Use Microphone and Speakers
All attendees will be muted during the presentation
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
3
Manage Your GoTo Panel
Ask a Question
To ask a question please type it into the
ldquoQuestionsrdquo box on the GoTo panel and
press ldquoSendrdquo
Use the orange and white arrow to
minimize and maximize the GoTo panel
Use the monitor button to toggle
between Fullscreen and window mode
Minimize and Maximize Your Screen
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
4
2018 Health Care IT Advisor Virtual Summit
Beyond Meaningful Use and Operational Excellence
Virtual Summit Agenda
Source Health Care IT Advisor research and analysis
Final session September 6th at 300pm ET
Available on-demand to Advisory Board Members
The Decision Machine Analytics and the Rise of Artificial Intelligence
Available on-demand to Advisory Board Members
State of the Union
Available on-demand to Advisory Board Members
ITrsquos Role in the New Cost Control Mandate
Available on-demand to Advisory Board Members
Digital Health Systems The Innovation Journey Continues
Available on-demand to Advisory Board Members
The Era of the Connected Patient Patient-Generated Health Data
and Social Determinants of Health
See the 2018 Virtual Summit
Health Care IT Advisor
The Decision MachineAnalytics and the Rise of AI
Greg Kuhnen
Senior Research Director
kuhnengadvisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
6
Sources ldquoAnalyticsrdquo Wikipedia last modified March 5 2018
httpsenwikipediaorgwikiAnalytics English Oxford Living Dictionaries Oxford Oxford
University Press 2018 (also available at httpsenoxforddictionariescom) Health Care IT Advisor
research and analysis
Analytics
The discovery interpretation and communication of meaningful
patterns in data
ndashWikipedia
Artificial Intelligence (AI)
The theory and development of computer systems able to
perform tasks normally requiring human intelligencehellip
ndashOxford English Dictionary
Machine Learning (ML)
The field of study that gives computers the ability to learn
without being explicitly programmed
ndashArthur Samuel 1959
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP7
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
8
Health Economics An Unsustainable Trajectory
Analytics Can Be a Powerful Tool for Bending the Cost Curve
Sources Cottarelli C ldquoMacro-Fiscal Implications of Health Care Reform in Advanced and Emerging Economiesrdquo IMF
Fiscal Affairs Department December 2010 McKinsey amp Company ldquomHealth A new vision for healthcarerdquo 2010
Office of Economic Co-operation and development lsquoHealth Spendingrdquo Health Care IT Advisor research and analysis
1) Global Burden of Disease Health Financing Collaborator Network ldquoFuture and potential spending on
health 2015-40 development assistance for health and government prepaid private and out-of-
pocket health spending in 184 countriesrdquo The Lancet 389 No 10083 (2016) 2) According to the
Australian Institute of Health and Welfare (AIHW) calculations 3) New Zealand Ministry of Health
projections 4) GDP = Gross domestic product
Per Capita Health Care Spending1 Health Expenditures as Percentage of GDP4
2016 20302016 2030
2016 2030 2016 2030
2016 2030
Australia2
96144
2010 2030
New Zealand3
62 89
106159
110
180
FranceCanada
United States United Kingdom
172249
97135
$9237
$5234
$4751
$4576
$4032
$4050
$3749
$3096
$12448
$7799
$6437
$5926
$5606
$5496
$5002
$4245
United States
Netherlands
Belgium
Canada
Australia
New Zealand
UnitedKingdom
Spain
Health Expenditure per Capita 2014
Health Expenditure per Capita 2030
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
10
Technology Drives the Flood of Data
Smartphones and Genomes and Sensors Oh My
Source Health Care IT Advisor research and analysis
1) IoT = Internet of things
2) RTLS = Real-time locating system
3) ROI = Return on investment
Common Barriers to
Adoption of New Sources
bull Privacy security concerns
bull Immature technology
bull Lack of budget
bull Unclear benefit ROI3
bull Staff already overwhelmed
with data
A Sampling of New Data Sources
Integrated Partners
(eg social services
cross-continuum care)
Data from Outside
Organizations
(eg social)
Environmental Data
(eg IoT1 RTLS2
temperature air
quality)
Genomes Proteomes
Microbiomes
Metabolomes
Clinical Surveillance
(eg public health
outbreak reports)
Remote Patient Monitoring
(eg bedside monitors
remote patient monitoring
consumer wearables)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
11
Cognitive Overload and Clinician Burnout
Sources Millett D Exclusive ldquoThree GPs a day seeking help for burnout
official data showrdquo GP Jan 2018 Health Care IT Advisor research and
analysis
1) Obermeyer Z Lee T ldquoLost in Thought ndash The Limits of the Human Mind and
the Future of Medicinerdquo N Engl J Med 377 no 13 (2017)1209-1211
Tim
e R
eq
uir
ed
ldquoMedical thinking has become vastly more complex mirroring changes in
our patients our health care system and medical science The complexity
of medicine now exceeds the capacity of the human mind1rdquo
Ziad Obermeyer MD and Thomas Lee MD
Number of Considerations for Decision Making
Significant
opportunity for
AI and analytics
Augmented Intelligence Reduces Cognitive Burden
Reliance on heuristics rules of thumb ldquogood-enoughrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
12
Data Power the Decision Engine
Source Health Care IT Advisor research and analysis
Four Levels of Analytic Capabilities
Data
Inform
Recommend
Decision
Automation
Decision
Support
Descriptive
What happened
How many of our
patients were
readmitted last
month
What is likely to happen
Is this patient likely to
readmit What do we
expect seasonally
Prescriptive
Recommend or
decide the best action
What should we do
to address this
patientrsquos risks
PredictiveDiagnostic
Why did it happen
Rates spiked was there
a change in case mix
Demographics
Decide
Hum
an
Decis
ion
Analy
tics
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP13
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
14
AI Are We There Yet
And if Not When
Source Health Care IT Advisor research and analysis
We May Be at the Start of an Explosive
Growth in AI Capabilities
Incremental Growth
AI continues to make
small advances
delivering value in
niche applicationsAI Winters AI has
already gone through
past phases of hype and
troughs of disillusionment
Exponential Growth
AI rapidly gains
capability and becomes
a mainstream part of
most digital systems
Unpredictable Timing Some
advances never seem to arrive
(conversational systems)
while others take off
unexpectedly (smartphones)
60s 70s 80s 90s 2000 2010 2020 203050s 2040 2050
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
15
Fuel for the AI Fire
Source Health Care IT Advisor research and analysis
1) EMR = Electronic medical record
2) GPU = Graphics processing unit
3) TPU = Tensor processing unit
AlgorithmsData
bull Internet Everywhere
bull Broad EMR1 Adoption
bull Wearables
bull Real-Time Location
bull IoT Sensors
bull Genomics Proteomics
Microbiome
bull Automated Feature
Extraction
bull Cluster Compute
Platforms (eg
Spark Hadoop)
bull Deep Learning
bull Off-the-Shelf
Libraries for
Common Tasksbull Faster Processors
bull Specialised Processors (GPUs2 TPUs3)
bull Inexpensive Storage
bull Elastic Cloud Computing
Hardware
Combinatorial Advances Delivering
Radically Better Models
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
16
The AI Winter Is Thawing
Source Health Care IT Advisor research and analysis
1) NASA = National Aeronautics and Space Administration
Microsoft and Alibaba outscore humans in a Stanford reading comprehension test 2018
Google DeepMindrsquos AlphaZero convincingly leads at Chess Shogi Go 2017
Skype launches real-time voice translation 2014
Mass market voice assistants (Apple Siri Google Now Microsoft Cortana) 2012
IBM Watson conclusively beats Jeopardy game show champions 2011
NASArsquos1 Mars Rovers operate semi-autonomously 2004
IBMrsquos Deep Blue beats world chess champion Gary Kasparov 1997
1950 Alan Turing proposes the ldquoTuring Testrdquo for conversational systems
1956 The term ldquoArtificial Intelligencerdquo is coined
1970s First AI Winter
1981 Digital Equipment Corporation order
expert saves company $40M per year
Timeline of AI Successes
From Toys to Tools
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
17
DeepMindrsquos AlphaZero
Novice to Superhuman in Just Four Hours
DeepMindrsquos AlphaZero Uses lsquoZero Human Knowledgersquo
Sources Vincent J ldquoDeepMindrsquos AI became a superhuman chess player in a few hours just for funrdquo The
Verge December 6 2017 Gershgorn D ldquoDeepMind wants to find the next miracle materialmdashexperts just
donrsquot know theyrsquoll pull it offrdquo Quartz October 25 2017 Health Care IT Advisor research and analysis
By not using this human data by not using
human features or human expertise in any
fashion wersquove actually rem
oved the constraints of human knowledge Itrsquos
able to therefore create knowledge for itselfrdquo
bull AI product of Google
subsidiary DeepMind
bull Provided no prior
knowledge of the
game given only basic
game rules and an
objective (win)
bull Learns through self-play
at an accelerated pace
(lsquoreinforced learningrsquo)
searches 80 thousand
positions per second
bull Variants of AlphaZero also
defeated world champion
Go and Shogi systems
AlphaZero
outperformed chess
world-champion
Stockfish1 in just
4 hours (300k steps)
By not using this human data by not using human
features or human expertise in any fashion wersquove
actually removed the constraints of human knowledge
Itrsquos able to therefore create knowledge for itselfrdquo
David Silver Lead Programmer
DEEPMIND
1) Stockfish is the 2016 Top Chess Engine Championship (TCEC) world-champion
AlphaZero Chess
Performance
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
18
Need a Radiologist
Therersquos an App for That
Sources Kubota T ldquoStanford algorithm can diagnose pneumonia better than
radiologistsrdquo Stanford News Nov 2017 Health Care IT Advisor research and analysis
1) Stanford University Medical Center
2) NIH = National Institutes of Health
Case in Brief Stanford University1
CheXNet Algorithm
bull Algorithm developed by researchers at
Stanford University in the US can diagnose
14 common pathologies in chest x-rays
bull Trained on ChestX-ray14 a public data set
released by the NIH2 containing 112120
frontal-view chest x-ray images labelled with
the 14 possible pathologies
bull Outperforms previous models from the same
data set for all 14 conditions and diagnoses
pneumonia at an accuracy exceeding the
performance of four control radiologists
bull Produces heatmaps that visualize the areas
of an image most indicative of disease
Development of CheXNet
Sept 26 2017
ChestX-ray14 data set
released along with a
preliminary algorithm
that could detect the
labelled conditionsasymp One week later
CheXNet could
diagnose 10 of the
14 pathologies more
accurately than all
previous algorithmsasymp One month later
CheXNet surpassed
best published results
for all 14 pathologies
and outperformed
Stanford radiologists in
detecting pneumonialdquoI treat the model as a lazy resident Theyrsquore not dumb theyrsquoll
just use every trick they can find to avoid doing hard workrdquo
Dr Matthew Lungren RadiologistStanford University Medical Center
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
19
Lessons from Stanfordrsquos CheXNet
AI Is Effective on Narrow Tasks
Recent advances in AI have focused on
solving narrowly defined tasks Each
sub-problem (eg pneumonia) is a model
Handle ldquoNormalrdquo Explicitly
Identifying normal conditions versus
anomalies can help recognize when
human review may be needed
Explainable1 AI (XAI)
Models that can be interpreted at least
partially are easier to trust deploy and
govern Explainability is an area of rapid
progress in AI research
1) Also known as interpretable AI
Training Requires a Teacher or Referee
Machine learning needs to be told what
answers are correct or what actions produce
positive outcomes Well-curated data are
critical to the learning process
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
20
The AI Value Proposition
AI Excels at Decisions That Are Dull Detailed and Dear (Expensive)
Three Key Areas of Benefit
Dear
Provide speed capacity 24x7
availability and consistency
at a fraction of the cost
Although these are well
known benefits of automation
when you apply this to
cognitive and complex tasks
itrsquos a new ball game
Detailed
Evaluate a vastly broader
and deeper set of data to
improve decision quality
The complexity of health
care decision making is
growing due to new data
from genetics IoT devices
wearables better
interoperability and the
growing number of drugs
and treatments
Unlimited capacity
instant response
disruptive economics
Perform at super-human
levels exceeding experts
Serves as an assistant
(watches your back)
Dull
Free human decision makers
to focus on more challenging
ldquotop of licenserdquo activities
Reduce effort spent on
repetitive tasks process
large amounts of complex
information provide
continuous monitoring and
notifications and ensure
ldquostuffrdquo doesnrsquot ldquofall through
the cracksrdquo
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
21
Where to Start Where Wersquore Going
1) Narrow (Task Weak) AI focuses on narrowly defined problems such as predicting a patientrsquos risk of developing sepsis
2) General (Strong) AI attempts to match the flexibility and versatility of human intelligence
Narr
ow
(T
ask W
ea
k)
AI1
Clinical Decisions
Administrative Decisions
Ge
ne
ral (S
tro
ng
) A
I2
Full Diagnosis
and Treatment Plan
Clinical
Chatbots
Radiology
Interpretation
Acute
Clinical
Risk
Recruiting
Safety Risk Smart
Monitoring
Medication Dosing
Administrative
Chatbots
(Concierge)
Precision
Engagement
Precision
Medicine
Ambient
Documentation
Adaptive User
Interfaces
Worklist
Prioritization
Inevitable with Time
Todayrsquos Opportunity
(growing)
The Computer Will
See You Now
The Uber-ization of
Health Care
An AI Application Landscape
Capacity Staffing
Management
Population
Health Risk
The Self-Driving
HospitalInformation
Security
Targeted
Marketing
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
22
The Self-Driving Hospital
Combine Predictive Models with Front-Line Activation
Sources ldquoVirtual Air Traffic Control for Emergency Departmentsrdquo Qventus
httpsqventuscomcontentuploads201703QVE-ED-Solution-Briefpdf Health Care
IT Advisor research and analysis
1) EVS = Environmental services
2) LWBS = Left without being seen
3) LOS = Length of stay
Case in Brief
Mercy Hospital Fort Smith
bull 336-bed acute care hospital in Fort
Smith AR
bull Partnered with Qventus to improve
emergency department (ED) patient flow
and patient satisfaction
bull Analytics applies ML algorithms to
anticipate potential capacity shortfalls
process bottlenecks
bull Application sends real-time notifications
(nudges) to care teams via their mobile
phones to trigger specific interventions
bull Emergency department results
ndash LWBS2 rates dropped 30
ndash Door-to-doc time reduced by 20
ndash 24-minute reduction in average LOS3
ndash Annual case capacity increased by 6
See that a radiology order is delayed for a
patient within a few days of discharge and
notify radiology to prioritize the reading
Activate more EVS1 staff when high churn
of beds is anticipated
Send an early warning for difficult patient
placements at discharge
Prioritize inpatient transports depending
on expected bed needs
Sample ldquoDecision Recipesrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
23
Applicants Like You Liked This Job
Predictive Models Reduce Turnover Improve Engagement
Case in Brief MultiCare Health System
bull Non-profit system based in Tacoma WA with 16000 employees
bull Partnered with Arena a cloud-based HR solution to predict the
likelihood that a candidate will be retained in a role
bull Arenarsquos machine learning algorithms evaluate applicant
submissions digital interactions and externally sourced data on
the organization (eg employer review sites)
bull Achieved a 40 reduction in RN turnover at 180 days and a
28 reduction in overall turnover at 180 days
MultiCarersquos Hiring Process
Reduced recruiting
expenses
More predictable
staffing levels
More experienced
staff
Realized Benefits
Algorithms predict the likelihood the
applicant will be retained and
engaged in the target role and send
that information to the recruiter
Applicant completes
a 15-20 minute
assessment process
on Arenarsquos platform
Recruiters determine
whether or not to send
the candidate along to
hiring managers
Hiring managers use
Arenarsquos recommendation
as an optional factor in
the hiring decision
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
24
Accelerating Research
1) Yu KH et al ldquoPredicting non-small cell lung cancer prognosis by fully automated microscopic pathology image featuresrdquo Nature No 12474 (2016)
2) Rajwa B et al ldquoAutomated assessment of disease progression in acute myeloid leukemia by probabilistic analysis of flow cytometry datardquo IEEE Transactions on Biomedical Engineering 64 No 5 (2017) 1089-1098
3) Liu Y et al Detecting cancer metastases on gigapixel pathology images arXiv preprint arXiv1703 No 02442 (2017)
4) Rajpurkar P et al Cardiologist-level arrhythmia detection with convolutional neural networks arXiv preprint arXiv1707 No 01836 (2017)
5) Nielsen S et al A predictive model to identify Parkinson disease from administrative claims data Neurology 89 No 14 (2017) 1448-1456
6) Rajpurkar P et al CheXNet Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning arXiv preprint arXiv 1711 No 05225 (2017)
7) Poplin R et al Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning Nature Biomedical Engineering No 2 (2018)
8) ACC Machine learning techniques show promise for supporting medical decisions By quickly assessing risk new algorithms may help triage patients and inform clinical decisions ScienceDaily 2018
9) Oh J et al ldquoA Generalizable Data-Driven Approach to Predict Daily Risk of Clostridium difficile Infection at Two Large Academic Health Centersrdquo ICHE 39 No 4 (2018) 425-433
Nature1 ndash August 2016
ML algorithm can accurately
differentiate two types of lung
cancers and predict patient
survival time
IEEE2 ndash February 2017
ML model predicts relapse rates for
acute myelogenous leukemia with
90 accuracy
arXiv3 ndash March 2017
ML approach to imaging analytics
can identify metastasized breast
cancer with accuracy rates rivaling
those of pathologists
Neurology5 ndash Sept 2017
Predictive model to identify
Parkinsonrsquos disease from claims
data achieved 73-83 accuracy in
study results
arXiv4 ndash July 2017
ML algorithm can diagnose
heart arrhythmias using data
from wearables sensors with
cardiologist-level accuracy
arXiv6 ndash November 2017
Deep-learning algorithm
outperforms Stanford radiologists
at diagnosing pneumonia from
x-rays
American College of Cardiology8 ndash
Feb 2018
Model predicts heart attack
diagnosis among patients in the ED
with chest pain with ~90 accuracy
Nature7 ndash February 2018
ML algorithm can predict a
personrsquos cardiovascular risk
factors using eye scans
Infection Control Hospital
Epidemiology9 ndash April 2018
ML model predicts Clostridium
difficile infection (CDI) earlier
than current diagnostic methods
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
25
Driving the Decision Machine
1Acquire Data
bull Ensure all models have
a sponsor and regular
evaluation against goals
bull All models require
tune-ups as the
environment changes
Monitor Performance
and Revisit
42Train or Refine
the Model
bull The more the better
buthellip
bull High-quality well-
governed data pay
enormous dividends
bull Diverse cross-continuum
sources improve
predictions and
robustness of models
bull Decide what features
are important (eg
explainability)
bull Apply practical and
ethical constraints
bull Evaluate under
realistic conditions
(silent mode real-time
incomplete data)
bull Embed insights directly
into applications at key
decision points
bull Reengineer processes
as warranted
bull Consider the ldquo5 rightsrdquo
of decision support
3Incorporate Models
into Workflow
Hint The Analytics Is the Easy Part
Begin with
bull Explicit leadership agreement on outcome goals expected benefit mechanisms
bull Participation from empowered representatives of impacted process
bull External experiencemdashlearn from others
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
26
New Data Sources
Mainstream Systems Cover Only a Fraction of Predictive Power
Acquire Data
Sources Nash DB ldquoPopulation Health Why It Mattersrdquo Essentials in Population Health Childrenrsquos
Hospital Association Thomas Jefferson University October 2016 Health Care IT Advisor research and
analysis
1) RPM = Remote patient monitoring
EMR(s)
Lab Systems
Pharmacy Systems
Imaging
Medical Claims
Prescription Drug Claims
Billing and Supply Chain
Scheduling Systems
Medical
Care Data
10
Sensors RPM1
mHealth Apps
Patient-Reported
Outcomes
Genetics
Socioeconomic Data
Patient Questionnaires
Human
Biology
Data
20
Lifestyle
and
Behavior
Data
50
Social and
Environmental
Data
20
Data Sources by Influence on Health Outcomes
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
27
Big Data a Technology and a Philosophy
ldquoTherersquos Gold in Them Thar Hillsrdquo
Acquire Data
Sources Cloudera ldquoInova Translational Medicine Institute Partners with Cloudera to Advance Genome-Based Machine
Learning Initiatives and Save Livesrdquo PR Newswire June 2017 Health Care IT Advisor research and analysis
1) Originally coined by Doug Laney of the Meta Group
Assumes Value in the Data
Based on a premise that there are
valuable insights that can be distilled
from the mountains of data captured
by machines
Characterised by the 3 Vs1
bull Volume
bull Velocity
bull Variety
Places greater emphasis on
unstructured data
Free text images streams of sensor
data are all fair targets for analysis
Big data is the difference
between heart rates and
heart beats
Case in Brief Inova Health System
bull Inovarsquos Translational Medicine Institute
(ITMI) is assembling one of the worldrsquos
largest whole genome sequence
databases connected to patient
information in a health care system
bull Researchers use the Cloudera
machine learning platform to analyze
terabytes of clinical and genomic data
and identify genetic links to diseases
bull Insights are used to develop
personalized treatment plans for
patients in collaboration with the
treating physician
Resource Big Data in Health Care
Strengths challenges and pragmatic
guidelines for those considering the addition
of big data capabilities to their IT portfolio
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
30
How Machines Learn
Similar Methods Differing Goals
Build the Model
Source Advisory Board research and analysis
Input Data
bull The more the better
bull Labeled with key indicators
Adjust the Model
bull Automated trial and error
bull Cross-breeding successful
candidates
Repeat
Supervised Learning
Evaluate models against the ldquocorrectrdquo answers
provided with the data The learning process
focuses on minimizing errors
Unsupervised Learning
Evaluate models against an abstract algorithmic
goal Example Divide this population of diabetics
into a handful of subpopulations that have similar
future risk profiles
Reinforcement Learning
Evaluate models by giving a reward for
desirable outcomes and a penalty for negative
outcomes Models are trained to maximize their
total rewards over timeReady for Review
Evaluate Performance
bull Against what goal
bull Forces explicit
tradeoffs (eg cost
vs quality)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
32
Black Boxes Shades of Grey
Differing Views on the Importance of Explainability
Incorporate into Workflow
Mike Wall PharmD MBA
Chief Analytics Officer
The University of Chicago Medicine
The opaqueness of black-box models is
really scary When we canrsquot tell a
clinician why a patient wonrsquot do well on a
certain medication that can be a tough
thing to blindly trust from a clinical
perspective We want to reach a place
where we augment our practice
through AI rather than dictatingrdquo
Nigam Shah MBBS PhD
Associate Professor of Medicine
(Biomedical Informatics)
Stanford University
I am pushing to evaluate these models
via a similar set of mechanisms as RCT1
or AB testing We should evaluate
against the current status-quo and do a
beforeafter studyhellipWe have to convince
ourselves of the utility of these models
We should focus on utility before
explainabilityrdquo
Surgery is the
best option ()
Data
Black Box Model
1) RCT = Randomized controlled trial
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
33
Workflow Is Key to Adoption
Incorporate into Workflow
The Right Information
To the Right Person
In the Right Form
By the Right Channel
At the Right Time
Case in Brief Mayo Breast Oncology
Clinic Trials Recruitment
bull Up to 100 concurrent drug therapy clinical trials are
available to Mayo Clinic breast cancer patients making
it challenging to consistently identify candidates
bull New clinical trial matching technology developed in
collaboration with IBM Watson Health evaluates charts
to identify eligible patients
bull Launched in January 2016 as a point-of-care solution
for oncology providers initial adoption was low due to
time pressures and fell to zero by February
bull Rework by health systems engineers resulted in a July
2016 relaunch screening moved to occur in
advance of patient visits so clinicians know of trial
matches before opening a patientrsquos chart
bull New process enrolls significantly more patients in
clinical trials than the prior manual process success led
to additional coordinator funding
The Five Rights of
Clinical Decision Support
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
34
Health Carersquos Field Agents
Medical Devices Become Intelligent Partners
Incorporate into Workflow
1) EKG = Electrocardiogram
ME
DT
RO
NIC
MIN
IME
D 6
70G
Case in Brief Artificial Pancreas and Smart Infusion Pumpsmdash
Medtronic MiniMed Connect
bull SMARTGUARD mimics some functions of a healthy pancreas
measures and predicts glucose level drift and adjusts dosing
bull Insulin pump and continuous glucose monitoring can send data directly
to smartphone
bull Trials showed a 44 reduction in hypoglycemia and reduction in
average A1C values from 74 to 69
Case in Brief AliveCor KardiaMobile
bull Portable consumer-grade single-lead EKG1 retails for $100
bull Intended for self-capture of intermittent cardiac events such as atrial
fibrillation (AFIB)
bull Smartphone-based machine learning validates quality of trace
assesses possibility of AFIB classifies normal abnormal sinus rhythm
bull EKGs can be sent to a board-certified cardiologist for interpretation
for $19
Sources alivecorcom Medtroniccom Health Care IT Advisor research and
analysis
AliveC
orK
ard
iaM
obile
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
3
Manage Your GoTo Panel
Ask a Question
To ask a question please type it into the
ldquoQuestionsrdquo box on the GoTo panel and
press ldquoSendrdquo
Use the orange and white arrow to
minimize and maximize the GoTo panel
Use the monitor button to toggle
between Fullscreen and window mode
Minimize and Maximize Your Screen
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
4
2018 Health Care IT Advisor Virtual Summit
Beyond Meaningful Use and Operational Excellence
Virtual Summit Agenda
Source Health Care IT Advisor research and analysis
Final session September 6th at 300pm ET
Available on-demand to Advisory Board Members
The Decision Machine Analytics and the Rise of Artificial Intelligence
Available on-demand to Advisory Board Members
State of the Union
Available on-demand to Advisory Board Members
ITrsquos Role in the New Cost Control Mandate
Available on-demand to Advisory Board Members
Digital Health Systems The Innovation Journey Continues
Available on-demand to Advisory Board Members
The Era of the Connected Patient Patient-Generated Health Data
and Social Determinants of Health
See the 2018 Virtual Summit
Health Care IT Advisor
The Decision MachineAnalytics and the Rise of AI
Greg Kuhnen
Senior Research Director
kuhnengadvisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
6
Sources ldquoAnalyticsrdquo Wikipedia last modified March 5 2018
httpsenwikipediaorgwikiAnalytics English Oxford Living Dictionaries Oxford Oxford
University Press 2018 (also available at httpsenoxforddictionariescom) Health Care IT Advisor
research and analysis
Analytics
The discovery interpretation and communication of meaningful
patterns in data
ndashWikipedia
Artificial Intelligence (AI)
The theory and development of computer systems able to
perform tasks normally requiring human intelligencehellip
ndashOxford English Dictionary
Machine Learning (ML)
The field of study that gives computers the ability to learn
without being explicitly programmed
ndashArthur Samuel 1959
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP7
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
8
Health Economics An Unsustainable Trajectory
Analytics Can Be a Powerful Tool for Bending the Cost Curve
Sources Cottarelli C ldquoMacro-Fiscal Implications of Health Care Reform in Advanced and Emerging Economiesrdquo IMF
Fiscal Affairs Department December 2010 McKinsey amp Company ldquomHealth A new vision for healthcarerdquo 2010
Office of Economic Co-operation and development lsquoHealth Spendingrdquo Health Care IT Advisor research and analysis
1) Global Burden of Disease Health Financing Collaborator Network ldquoFuture and potential spending on
health 2015-40 development assistance for health and government prepaid private and out-of-
pocket health spending in 184 countriesrdquo The Lancet 389 No 10083 (2016) 2) According to the
Australian Institute of Health and Welfare (AIHW) calculations 3) New Zealand Ministry of Health
projections 4) GDP = Gross domestic product
Per Capita Health Care Spending1 Health Expenditures as Percentage of GDP4
2016 20302016 2030
2016 2030 2016 2030
2016 2030
Australia2
96144
2010 2030
New Zealand3
62 89
106159
110
180
FranceCanada
United States United Kingdom
172249
97135
$9237
$5234
$4751
$4576
$4032
$4050
$3749
$3096
$12448
$7799
$6437
$5926
$5606
$5496
$5002
$4245
United States
Netherlands
Belgium
Canada
Australia
New Zealand
UnitedKingdom
Spain
Health Expenditure per Capita 2014
Health Expenditure per Capita 2030
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
10
Technology Drives the Flood of Data
Smartphones and Genomes and Sensors Oh My
Source Health Care IT Advisor research and analysis
1) IoT = Internet of things
2) RTLS = Real-time locating system
3) ROI = Return on investment
Common Barriers to
Adoption of New Sources
bull Privacy security concerns
bull Immature technology
bull Lack of budget
bull Unclear benefit ROI3
bull Staff already overwhelmed
with data
A Sampling of New Data Sources
Integrated Partners
(eg social services
cross-continuum care)
Data from Outside
Organizations
(eg social)
Environmental Data
(eg IoT1 RTLS2
temperature air
quality)
Genomes Proteomes
Microbiomes
Metabolomes
Clinical Surveillance
(eg public health
outbreak reports)
Remote Patient Monitoring
(eg bedside monitors
remote patient monitoring
consumer wearables)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
11
Cognitive Overload and Clinician Burnout
Sources Millett D Exclusive ldquoThree GPs a day seeking help for burnout
official data showrdquo GP Jan 2018 Health Care IT Advisor research and
analysis
1) Obermeyer Z Lee T ldquoLost in Thought ndash The Limits of the Human Mind and
the Future of Medicinerdquo N Engl J Med 377 no 13 (2017)1209-1211
Tim
e R
eq
uir
ed
ldquoMedical thinking has become vastly more complex mirroring changes in
our patients our health care system and medical science The complexity
of medicine now exceeds the capacity of the human mind1rdquo
Ziad Obermeyer MD and Thomas Lee MD
Number of Considerations for Decision Making
Significant
opportunity for
AI and analytics
Augmented Intelligence Reduces Cognitive Burden
Reliance on heuristics rules of thumb ldquogood-enoughrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
12
Data Power the Decision Engine
Source Health Care IT Advisor research and analysis
Four Levels of Analytic Capabilities
Data
Inform
Recommend
Decision
Automation
Decision
Support
Descriptive
What happened
How many of our
patients were
readmitted last
month
What is likely to happen
Is this patient likely to
readmit What do we
expect seasonally
Prescriptive
Recommend or
decide the best action
What should we do
to address this
patientrsquos risks
PredictiveDiagnostic
Why did it happen
Rates spiked was there
a change in case mix
Demographics
Decide
Hum
an
Decis
ion
Analy
tics
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP13
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
14
AI Are We There Yet
And if Not When
Source Health Care IT Advisor research and analysis
We May Be at the Start of an Explosive
Growth in AI Capabilities
Incremental Growth
AI continues to make
small advances
delivering value in
niche applicationsAI Winters AI has
already gone through
past phases of hype and
troughs of disillusionment
Exponential Growth
AI rapidly gains
capability and becomes
a mainstream part of
most digital systems
Unpredictable Timing Some
advances never seem to arrive
(conversational systems)
while others take off
unexpectedly (smartphones)
60s 70s 80s 90s 2000 2010 2020 203050s 2040 2050
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
15
Fuel for the AI Fire
Source Health Care IT Advisor research and analysis
1) EMR = Electronic medical record
2) GPU = Graphics processing unit
3) TPU = Tensor processing unit
AlgorithmsData
bull Internet Everywhere
bull Broad EMR1 Adoption
bull Wearables
bull Real-Time Location
bull IoT Sensors
bull Genomics Proteomics
Microbiome
bull Automated Feature
Extraction
bull Cluster Compute
Platforms (eg
Spark Hadoop)
bull Deep Learning
bull Off-the-Shelf
Libraries for
Common Tasksbull Faster Processors
bull Specialised Processors (GPUs2 TPUs3)
bull Inexpensive Storage
bull Elastic Cloud Computing
Hardware
Combinatorial Advances Delivering
Radically Better Models
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
16
The AI Winter Is Thawing
Source Health Care IT Advisor research and analysis
1) NASA = National Aeronautics and Space Administration
Microsoft and Alibaba outscore humans in a Stanford reading comprehension test 2018
Google DeepMindrsquos AlphaZero convincingly leads at Chess Shogi Go 2017
Skype launches real-time voice translation 2014
Mass market voice assistants (Apple Siri Google Now Microsoft Cortana) 2012
IBM Watson conclusively beats Jeopardy game show champions 2011
NASArsquos1 Mars Rovers operate semi-autonomously 2004
IBMrsquos Deep Blue beats world chess champion Gary Kasparov 1997
1950 Alan Turing proposes the ldquoTuring Testrdquo for conversational systems
1956 The term ldquoArtificial Intelligencerdquo is coined
1970s First AI Winter
1981 Digital Equipment Corporation order
expert saves company $40M per year
Timeline of AI Successes
From Toys to Tools
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
17
DeepMindrsquos AlphaZero
Novice to Superhuman in Just Four Hours
DeepMindrsquos AlphaZero Uses lsquoZero Human Knowledgersquo
Sources Vincent J ldquoDeepMindrsquos AI became a superhuman chess player in a few hours just for funrdquo The
Verge December 6 2017 Gershgorn D ldquoDeepMind wants to find the next miracle materialmdashexperts just
donrsquot know theyrsquoll pull it offrdquo Quartz October 25 2017 Health Care IT Advisor research and analysis
By not using this human data by not using
human features or human expertise in any
fashion wersquove actually rem
oved the constraints of human knowledge Itrsquos
able to therefore create knowledge for itselfrdquo
bull AI product of Google
subsidiary DeepMind
bull Provided no prior
knowledge of the
game given only basic
game rules and an
objective (win)
bull Learns through self-play
at an accelerated pace
(lsquoreinforced learningrsquo)
searches 80 thousand
positions per second
bull Variants of AlphaZero also
defeated world champion
Go and Shogi systems
AlphaZero
outperformed chess
world-champion
Stockfish1 in just
4 hours (300k steps)
By not using this human data by not using human
features or human expertise in any fashion wersquove
actually removed the constraints of human knowledge
Itrsquos able to therefore create knowledge for itselfrdquo
David Silver Lead Programmer
DEEPMIND
1) Stockfish is the 2016 Top Chess Engine Championship (TCEC) world-champion
AlphaZero Chess
Performance
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
18
Need a Radiologist
Therersquos an App for That
Sources Kubota T ldquoStanford algorithm can diagnose pneumonia better than
radiologistsrdquo Stanford News Nov 2017 Health Care IT Advisor research and analysis
1) Stanford University Medical Center
2) NIH = National Institutes of Health
Case in Brief Stanford University1
CheXNet Algorithm
bull Algorithm developed by researchers at
Stanford University in the US can diagnose
14 common pathologies in chest x-rays
bull Trained on ChestX-ray14 a public data set
released by the NIH2 containing 112120
frontal-view chest x-ray images labelled with
the 14 possible pathologies
bull Outperforms previous models from the same
data set for all 14 conditions and diagnoses
pneumonia at an accuracy exceeding the
performance of four control radiologists
bull Produces heatmaps that visualize the areas
of an image most indicative of disease
Development of CheXNet
Sept 26 2017
ChestX-ray14 data set
released along with a
preliminary algorithm
that could detect the
labelled conditionsasymp One week later
CheXNet could
diagnose 10 of the
14 pathologies more
accurately than all
previous algorithmsasymp One month later
CheXNet surpassed
best published results
for all 14 pathologies
and outperformed
Stanford radiologists in
detecting pneumonialdquoI treat the model as a lazy resident Theyrsquore not dumb theyrsquoll
just use every trick they can find to avoid doing hard workrdquo
Dr Matthew Lungren RadiologistStanford University Medical Center
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
19
Lessons from Stanfordrsquos CheXNet
AI Is Effective on Narrow Tasks
Recent advances in AI have focused on
solving narrowly defined tasks Each
sub-problem (eg pneumonia) is a model
Handle ldquoNormalrdquo Explicitly
Identifying normal conditions versus
anomalies can help recognize when
human review may be needed
Explainable1 AI (XAI)
Models that can be interpreted at least
partially are easier to trust deploy and
govern Explainability is an area of rapid
progress in AI research
1) Also known as interpretable AI
Training Requires a Teacher or Referee
Machine learning needs to be told what
answers are correct or what actions produce
positive outcomes Well-curated data are
critical to the learning process
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
20
The AI Value Proposition
AI Excels at Decisions That Are Dull Detailed and Dear (Expensive)
Three Key Areas of Benefit
Dear
Provide speed capacity 24x7
availability and consistency
at a fraction of the cost
Although these are well
known benefits of automation
when you apply this to
cognitive and complex tasks
itrsquos a new ball game
Detailed
Evaluate a vastly broader
and deeper set of data to
improve decision quality
The complexity of health
care decision making is
growing due to new data
from genetics IoT devices
wearables better
interoperability and the
growing number of drugs
and treatments
Unlimited capacity
instant response
disruptive economics
Perform at super-human
levels exceeding experts
Serves as an assistant
(watches your back)
Dull
Free human decision makers
to focus on more challenging
ldquotop of licenserdquo activities
Reduce effort spent on
repetitive tasks process
large amounts of complex
information provide
continuous monitoring and
notifications and ensure
ldquostuffrdquo doesnrsquot ldquofall through
the cracksrdquo
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
21
Where to Start Where Wersquore Going
1) Narrow (Task Weak) AI focuses on narrowly defined problems such as predicting a patientrsquos risk of developing sepsis
2) General (Strong) AI attempts to match the flexibility and versatility of human intelligence
Narr
ow
(T
ask W
ea
k)
AI1
Clinical Decisions
Administrative Decisions
Ge
ne
ral (S
tro
ng
) A
I2
Full Diagnosis
and Treatment Plan
Clinical
Chatbots
Radiology
Interpretation
Acute
Clinical
Risk
Recruiting
Safety Risk Smart
Monitoring
Medication Dosing
Administrative
Chatbots
(Concierge)
Precision
Engagement
Precision
Medicine
Ambient
Documentation
Adaptive User
Interfaces
Worklist
Prioritization
Inevitable with Time
Todayrsquos Opportunity
(growing)
The Computer Will
See You Now
The Uber-ization of
Health Care
An AI Application Landscape
Capacity Staffing
Management
Population
Health Risk
The Self-Driving
HospitalInformation
Security
Targeted
Marketing
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
22
The Self-Driving Hospital
Combine Predictive Models with Front-Line Activation
Sources ldquoVirtual Air Traffic Control for Emergency Departmentsrdquo Qventus
httpsqventuscomcontentuploads201703QVE-ED-Solution-Briefpdf Health Care
IT Advisor research and analysis
1) EVS = Environmental services
2) LWBS = Left without being seen
3) LOS = Length of stay
Case in Brief
Mercy Hospital Fort Smith
bull 336-bed acute care hospital in Fort
Smith AR
bull Partnered with Qventus to improve
emergency department (ED) patient flow
and patient satisfaction
bull Analytics applies ML algorithms to
anticipate potential capacity shortfalls
process bottlenecks
bull Application sends real-time notifications
(nudges) to care teams via their mobile
phones to trigger specific interventions
bull Emergency department results
ndash LWBS2 rates dropped 30
ndash Door-to-doc time reduced by 20
ndash 24-minute reduction in average LOS3
ndash Annual case capacity increased by 6
See that a radiology order is delayed for a
patient within a few days of discharge and
notify radiology to prioritize the reading
Activate more EVS1 staff when high churn
of beds is anticipated
Send an early warning for difficult patient
placements at discharge
Prioritize inpatient transports depending
on expected bed needs
Sample ldquoDecision Recipesrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
23
Applicants Like You Liked This Job
Predictive Models Reduce Turnover Improve Engagement
Case in Brief MultiCare Health System
bull Non-profit system based in Tacoma WA with 16000 employees
bull Partnered with Arena a cloud-based HR solution to predict the
likelihood that a candidate will be retained in a role
bull Arenarsquos machine learning algorithms evaluate applicant
submissions digital interactions and externally sourced data on
the organization (eg employer review sites)
bull Achieved a 40 reduction in RN turnover at 180 days and a
28 reduction in overall turnover at 180 days
MultiCarersquos Hiring Process
Reduced recruiting
expenses
More predictable
staffing levels
More experienced
staff
Realized Benefits
Algorithms predict the likelihood the
applicant will be retained and
engaged in the target role and send
that information to the recruiter
Applicant completes
a 15-20 minute
assessment process
on Arenarsquos platform
Recruiters determine
whether or not to send
the candidate along to
hiring managers
Hiring managers use
Arenarsquos recommendation
as an optional factor in
the hiring decision
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
24
Accelerating Research
1) Yu KH et al ldquoPredicting non-small cell lung cancer prognosis by fully automated microscopic pathology image featuresrdquo Nature No 12474 (2016)
2) Rajwa B et al ldquoAutomated assessment of disease progression in acute myeloid leukemia by probabilistic analysis of flow cytometry datardquo IEEE Transactions on Biomedical Engineering 64 No 5 (2017) 1089-1098
3) Liu Y et al Detecting cancer metastases on gigapixel pathology images arXiv preprint arXiv1703 No 02442 (2017)
4) Rajpurkar P et al Cardiologist-level arrhythmia detection with convolutional neural networks arXiv preprint arXiv1707 No 01836 (2017)
5) Nielsen S et al A predictive model to identify Parkinson disease from administrative claims data Neurology 89 No 14 (2017) 1448-1456
6) Rajpurkar P et al CheXNet Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning arXiv preprint arXiv 1711 No 05225 (2017)
7) Poplin R et al Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning Nature Biomedical Engineering No 2 (2018)
8) ACC Machine learning techniques show promise for supporting medical decisions By quickly assessing risk new algorithms may help triage patients and inform clinical decisions ScienceDaily 2018
9) Oh J et al ldquoA Generalizable Data-Driven Approach to Predict Daily Risk of Clostridium difficile Infection at Two Large Academic Health Centersrdquo ICHE 39 No 4 (2018) 425-433
Nature1 ndash August 2016
ML algorithm can accurately
differentiate two types of lung
cancers and predict patient
survival time
IEEE2 ndash February 2017
ML model predicts relapse rates for
acute myelogenous leukemia with
90 accuracy
arXiv3 ndash March 2017
ML approach to imaging analytics
can identify metastasized breast
cancer with accuracy rates rivaling
those of pathologists
Neurology5 ndash Sept 2017
Predictive model to identify
Parkinsonrsquos disease from claims
data achieved 73-83 accuracy in
study results
arXiv4 ndash July 2017
ML algorithm can diagnose
heart arrhythmias using data
from wearables sensors with
cardiologist-level accuracy
arXiv6 ndash November 2017
Deep-learning algorithm
outperforms Stanford radiologists
at diagnosing pneumonia from
x-rays
American College of Cardiology8 ndash
Feb 2018
Model predicts heart attack
diagnosis among patients in the ED
with chest pain with ~90 accuracy
Nature7 ndash February 2018
ML algorithm can predict a
personrsquos cardiovascular risk
factors using eye scans
Infection Control Hospital
Epidemiology9 ndash April 2018
ML model predicts Clostridium
difficile infection (CDI) earlier
than current diagnostic methods
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
25
Driving the Decision Machine
1Acquire Data
bull Ensure all models have
a sponsor and regular
evaluation against goals
bull All models require
tune-ups as the
environment changes
Monitor Performance
and Revisit
42Train or Refine
the Model
bull The more the better
buthellip
bull High-quality well-
governed data pay
enormous dividends
bull Diverse cross-continuum
sources improve
predictions and
robustness of models
bull Decide what features
are important (eg
explainability)
bull Apply practical and
ethical constraints
bull Evaluate under
realistic conditions
(silent mode real-time
incomplete data)
bull Embed insights directly
into applications at key
decision points
bull Reengineer processes
as warranted
bull Consider the ldquo5 rightsrdquo
of decision support
3Incorporate Models
into Workflow
Hint The Analytics Is the Easy Part
Begin with
bull Explicit leadership agreement on outcome goals expected benefit mechanisms
bull Participation from empowered representatives of impacted process
bull External experiencemdashlearn from others
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
26
New Data Sources
Mainstream Systems Cover Only a Fraction of Predictive Power
Acquire Data
Sources Nash DB ldquoPopulation Health Why It Mattersrdquo Essentials in Population Health Childrenrsquos
Hospital Association Thomas Jefferson University October 2016 Health Care IT Advisor research and
analysis
1) RPM = Remote patient monitoring
EMR(s)
Lab Systems
Pharmacy Systems
Imaging
Medical Claims
Prescription Drug Claims
Billing and Supply Chain
Scheduling Systems
Medical
Care Data
10
Sensors RPM1
mHealth Apps
Patient-Reported
Outcomes
Genetics
Socioeconomic Data
Patient Questionnaires
Human
Biology
Data
20
Lifestyle
and
Behavior
Data
50
Social and
Environmental
Data
20
Data Sources by Influence on Health Outcomes
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
27
Big Data a Technology and a Philosophy
ldquoTherersquos Gold in Them Thar Hillsrdquo
Acquire Data
Sources Cloudera ldquoInova Translational Medicine Institute Partners with Cloudera to Advance Genome-Based Machine
Learning Initiatives and Save Livesrdquo PR Newswire June 2017 Health Care IT Advisor research and analysis
1) Originally coined by Doug Laney of the Meta Group
Assumes Value in the Data
Based on a premise that there are
valuable insights that can be distilled
from the mountains of data captured
by machines
Characterised by the 3 Vs1
bull Volume
bull Velocity
bull Variety
Places greater emphasis on
unstructured data
Free text images streams of sensor
data are all fair targets for analysis
Big data is the difference
between heart rates and
heart beats
Case in Brief Inova Health System
bull Inovarsquos Translational Medicine Institute
(ITMI) is assembling one of the worldrsquos
largest whole genome sequence
databases connected to patient
information in a health care system
bull Researchers use the Cloudera
machine learning platform to analyze
terabytes of clinical and genomic data
and identify genetic links to diseases
bull Insights are used to develop
personalized treatment plans for
patients in collaboration with the
treating physician
Resource Big Data in Health Care
Strengths challenges and pragmatic
guidelines for those considering the addition
of big data capabilities to their IT portfolio
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
30
How Machines Learn
Similar Methods Differing Goals
Build the Model
Source Advisory Board research and analysis
Input Data
bull The more the better
bull Labeled with key indicators
Adjust the Model
bull Automated trial and error
bull Cross-breeding successful
candidates
Repeat
Supervised Learning
Evaluate models against the ldquocorrectrdquo answers
provided with the data The learning process
focuses on minimizing errors
Unsupervised Learning
Evaluate models against an abstract algorithmic
goal Example Divide this population of diabetics
into a handful of subpopulations that have similar
future risk profiles
Reinforcement Learning
Evaluate models by giving a reward for
desirable outcomes and a penalty for negative
outcomes Models are trained to maximize their
total rewards over timeReady for Review
Evaluate Performance
bull Against what goal
bull Forces explicit
tradeoffs (eg cost
vs quality)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
32
Black Boxes Shades of Grey
Differing Views on the Importance of Explainability
Incorporate into Workflow
Mike Wall PharmD MBA
Chief Analytics Officer
The University of Chicago Medicine
The opaqueness of black-box models is
really scary When we canrsquot tell a
clinician why a patient wonrsquot do well on a
certain medication that can be a tough
thing to blindly trust from a clinical
perspective We want to reach a place
where we augment our practice
through AI rather than dictatingrdquo
Nigam Shah MBBS PhD
Associate Professor of Medicine
(Biomedical Informatics)
Stanford University
I am pushing to evaluate these models
via a similar set of mechanisms as RCT1
or AB testing We should evaluate
against the current status-quo and do a
beforeafter studyhellipWe have to convince
ourselves of the utility of these models
We should focus on utility before
explainabilityrdquo
Surgery is the
best option ()
Data
Black Box Model
1) RCT = Randomized controlled trial
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
33
Workflow Is Key to Adoption
Incorporate into Workflow
The Right Information
To the Right Person
In the Right Form
By the Right Channel
At the Right Time
Case in Brief Mayo Breast Oncology
Clinic Trials Recruitment
bull Up to 100 concurrent drug therapy clinical trials are
available to Mayo Clinic breast cancer patients making
it challenging to consistently identify candidates
bull New clinical trial matching technology developed in
collaboration with IBM Watson Health evaluates charts
to identify eligible patients
bull Launched in January 2016 as a point-of-care solution
for oncology providers initial adoption was low due to
time pressures and fell to zero by February
bull Rework by health systems engineers resulted in a July
2016 relaunch screening moved to occur in
advance of patient visits so clinicians know of trial
matches before opening a patientrsquos chart
bull New process enrolls significantly more patients in
clinical trials than the prior manual process success led
to additional coordinator funding
The Five Rights of
Clinical Decision Support
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
34
Health Carersquos Field Agents
Medical Devices Become Intelligent Partners
Incorporate into Workflow
1) EKG = Electrocardiogram
ME
DT
RO
NIC
MIN
IME
D 6
70G
Case in Brief Artificial Pancreas and Smart Infusion Pumpsmdash
Medtronic MiniMed Connect
bull SMARTGUARD mimics some functions of a healthy pancreas
measures and predicts glucose level drift and adjusts dosing
bull Insulin pump and continuous glucose monitoring can send data directly
to smartphone
bull Trials showed a 44 reduction in hypoglycemia and reduction in
average A1C values from 74 to 69
Case in Brief AliveCor KardiaMobile
bull Portable consumer-grade single-lead EKG1 retails for $100
bull Intended for self-capture of intermittent cardiac events such as atrial
fibrillation (AFIB)
bull Smartphone-based machine learning validates quality of trace
assesses possibility of AFIB classifies normal abnormal sinus rhythm
bull EKGs can be sent to a board-certified cardiologist for interpretation
for $19
Sources alivecorcom Medtroniccom Health Care IT Advisor research and
analysis
AliveC
orK
ard
iaM
obile
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
4
2018 Health Care IT Advisor Virtual Summit
Beyond Meaningful Use and Operational Excellence
Virtual Summit Agenda
Source Health Care IT Advisor research and analysis
Final session September 6th at 300pm ET
Available on-demand to Advisory Board Members
The Decision Machine Analytics and the Rise of Artificial Intelligence
Available on-demand to Advisory Board Members
State of the Union
Available on-demand to Advisory Board Members
ITrsquos Role in the New Cost Control Mandate
Available on-demand to Advisory Board Members
Digital Health Systems The Innovation Journey Continues
Available on-demand to Advisory Board Members
The Era of the Connected Patient Patient-Generated Health Data
and Social Determinants of Health
See the 2018 Virtual Summit
Health Care IT Advisor
The Decision MachineAnalytics and the Rise of AI
Greg Kuhnen
Senior Research Director
kuhnengadvisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
6
Sources ldquoAnalyticsrdquo Wikipedia last modified March 5 2018
httpsenwikipediaorgwikiAnalytics English Oxford Living Dictionaries Oxford Oxford
University Press 2018 (also available at httpsenoxforddictionariescom) Health Care IT Advisor
research and analysis
Analytics
The discovery interpretation and communication of meaningful
patterns in data
ndashWikipedia
Artificial Intelligence (AI)
The theory and development of computer systems able to
perform tasks normally requiring human intelligencehellip
ndashOxford English Dictionary
Machine Learning (ML)
The field of study that gives computers the ability to learn
without being explicitly programmed
ndashArthur Samuel 1959
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP7
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
8
Health Economics An Unsustainable Trajectory
Analytics Can Be a Powerful Tool for Bending the Cost Curve
Sources Cottarelli C ldquoMacro-Fiscal Implications of Health Care Reform in Advanced and Emerging Economiesrdquo IMF
Fiscal Affairs Department December 2010 McKinsey amp Company ldquomHealth A new vision for healthcarerdquo 2010
Office of Economic Co-operation and development lsquoHealth Spendingrdquo Health Care IT Advisor research and analysis
1) Global Burden of Disease Health Financing Collaborator Network ldquoFuture and potential spending on
health 2015-40 development assistance for health and government prepaid private and out-of-
pocket health spending in 184 countriesrdquo The Lancet 389 No 10083 (2016) 2) According to the
Australian Institute of Health and Welfare (AIHW) calculations 3) New Zealand Ministry of Health
projections 4) GDP = Gross domestic product
Per Capita Health Care Spending1 Health Expenditures as Percentage of GDP4
2016 20302016 2030
2016 2030 2016 2030
2016 2030
Australia2
96144
2010 2030
New Zealand3
62 89
106159
110
180
FranceCanada
United States United Kingdom
172249
97135
$9237
$5234
$4751
$4576
$4032
$4050
$3749
$3096
$12448
$7799
$6437
$5926
$5606
$5496
$5002
$4245
United States
Netherlands
Belgium
Canada
Australia
New Zealand
UnitedKingdom
Spain
Health Expenditure per Capita 2014
Health Expenditure per Capita 2030
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
10
Technology Drives the Flood of Data
Smartphones and Genomes and Sensors Oh My
Source Health Care IT Advisor research and analysis
1) IoT = Internet of things
2) RTLS = Real-time locating system
3) ROI = Return on investment
Common Barriers to
Adoption of New Sources
bull Privacy security concerns
bull Immature technology
bull Lack of budget
bull Unclear benefit ROI3
bull Staff already overwhelmed
with data
A Sampling of New Data Sources
Integrated Partners
(eg social services
cross-continuum care)
Data from Outside
Organizations
(eg social)
Environmental Data
(eg IoT1 RTLS2
temperature air
quality)
Genomes Proteomes
Microbiomes
Metabolomes
Clinical Surveillance
(eg public health
outbreak reports)
Remote Patient Monitoring
(eg bedside monitors
remote patient monitoring
consumer wearables)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
11
Cognitive Overload and Clinician Burnout
Sources Millett D Exclusive ldquoThree GPs a day seeking help for burnout
official data showrdquo GP Jan 2018 Health Care IT Advisor research and
analysis
1) Obermeyer Z Lee T ldquoLost in Thought ndash The Limits of the Human Mind and
the Future of Medicinerdquo N Engl J Med 377 no 13 (2017)1209-1211
Tim
e R
eq
uir
ed
ldquoMedical thinking has become vastly more complex mirroring changes in
our patients our health care system and medical science The complexity
of medicine now exceeds the capacity of the human mind1rdquo
Ziad Obermeyer MD and Thomas Lee MD
Number of Considerations for Decision Making
Significant
opportunity for
AI and analytics
Augmented Intelligence Reduces Cognitive Burden
Reliance on heuristics rules of thumb ldquogood-enoughrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
12
Data Power the Decision Engine
Source Health Care IT Advisor research and analysis
Four Levels of Analytic Capabilities
Data
Inform
Recommend
Decision
Automation
Decision
Support
Descriptive
What happened
How many of our
patients were
readmitted last
month
What is likely to happen
Is this patient likely to
readmit What do we
expect seasonally
Prescriptive
Recommend or
decide the best action
What should we do
to address this
patientrsquos risks
PredictiveDiagnostic
Why did it happen
Rates spiked was there
a change in case mix
Demographics
Decide
Hum
an
Decis
ion
Analy
tics
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP13
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
14
AI Are We There Yet
And if Not When
Source Health Care IT Advisor research and analysis
We May Be at the Start of an Explosive
Growth in AI Capabilities
Incremental Growth
AI continues to make
small advances
delivering value in
niche applicationsAI Winters AI has
already gone through
past phases of hype and
troughs of disillusionment
Exponential Growth
AI rapidly gains
capability and becomes
a mainstream part of
most digital systems
Unpredictable Timing Some
advances never seem to arrive
(conversational systems)
while others take off
unexpectedly (smartphones)
60s 70s 80s 90s 2000 2010 2020 203050s 2040 2050
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
15
Fuel for the AI Fire
Source Health Care IT Advisor research and analysis
1) EMR = Electronic medical record
2) GPU = Graphics processing unit
3) TPU = Tensor processing unit
AlgorithmsData
bull Internet Everywhere
bull Broad EMR1 Adoption
bull Wearables
bull Real-Time Location
bull IoT Sensors
bull Genomics Proteomics
Microbiome
bull Automated Feature
Extraction
bull Cluster Compute
Platforms (eg
Spark Hadoop)
bull Deep Learning
bull Off-the-Shelf
Libraries for
Common Tasksbull Faster Processors
bull Specialised Processors (GPUs2 TPUs3)
bull Inexpensive Storage
bull Elastic Cloud Computing
Hardware
Combinatorial Advances Delivering
Radically Better Models
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
16
The AI Winter Is Thawing
Source Health Care IT Advisor research and analysis
1) NASA = National Aeronautics and Space Administration
Microsoft and Alibaba outscore humans in a Stanford reading comprehension test 2018
Google DeepMindrsquos AlphaZero convincingly leads at Chess Shogi Go 2017
Skype launches real-time voice translation 2014
Mass market voice assistants (Apple Siri Google Now Microsoft Cortana) 2012
IBM Watson conclusively beats Jeopardy game show champions 2011
NASArsquos1 Mars Rovers operate semi-autonomously 2004
IBMrsquos Deep Blue beats world chess champion Gary Kasparov 1997
1950 Alan Turing proposes the ldquoTuring Testrdquo for conversational systems
1956 The term ldquoArtificial Intelligencerdquo is coined
1970s First AI Winter
1981 Digital Equipment Corporation order
expert saves company $40M per year
Timeline of AI Successes
From Toys to Tools
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
17
DeepMindrsquos AlphaZero
Novice to Superhuman in Just Four Hours
DeepMindrsquos AlphaZero Uses lsquoZero Human Knowledgersquo
Sources Vincent J ldquoDeepMindrsquos AI became a superhuman chess player in a few hours just for funrdquo The
Verge December 6 2017 Gershgorn D ldquoDeepMind wants to find the next miracle materialmdashexperts just
donrsquot know theyrsquoll pull it offrdquo Quartz October 25 2017 Health Care IT Advisor research and analysis
By not using this human data by not using
human features or human expertise in any
fashion wersquove actually rem
oved the constraints of human knowledge Itrsquos
able to therefore create knowledge for itselfrdquo
bull AI product of Google
subsidiary DeepMind
bull Provided no prior
knowledge of the
game given only basic
game rules and an
objective (win)
bull Learns through self-play
at an accelerated pace
(lsquoreinforced learningrsquo)
searches 80 thousand
positions per second
bull Variants of AlphaZero also
defeated world champion
Go and Shogi systems
AlphaZero
outperformed chess
world-champion
Stockfish1 in just
4 hours (300k steps)
By not using this human data by not using human
features or human expertise in any fashion wersquove
actually removed the constraints of human knowledge
Itrsquos able to therefore create knowledge for itselfrdquo
David Silver Lead Programmer
DEEPMIND
1) Stockfish is the 2016 Top Chess Engine Championship (TCEC) world-champion
AlphaZero Chess
Performance
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
18
Need a Radiologist
Therersquos an App for That
Sources Kubota T ldquoStanford algorithm can diagnose pneumonia better than
radiologistsrdquo Stanford News Nov 2017 Health Care IT Advisor research and analysis
1) Stanford University Medical Center
2) NIH = National Institutes of Health
Case in Brief Stanford University1
CheXNet Algorithm
bull Algorithm developed by researchers at
Stanford University in the US can diagnose
14 common pathologies in chest x-rays
bull Trained on ChestX-ray14 a public data set
released by the NIH2 containing 112120
frontal-view chest x-ray images labelled with
the 14 possible pathologies
bull Outperforms previous models from the same
data set for all 14 conditions and diagnoses
pneumonia at an accuracy exceeding the
performance of four control radiologists
bull Produces heatmaps that visualize the areas
of an image most indicative of disease
Development of CheXNet
Sept 26 2017
ChestX-ray14 data set
released along with a
preliminary algorithm
that could detect the
labelled conditionsasymp One week later
CheXNet could
diagnose 10 of the
14 pathologies more
accurately than all
previous algorithmsasymp One month later
CheXNet surpassed
best published results
for all 14 pathologies
and outperformed
Stanford radiologists in
detecting pneumonialdquoI treat the model as a lazy resident Theyrsquore not dumb theyrsquoll
just use every trick they can find to avoid doing hard workrdquo
Dr Matthew Lungren RadiologistStanford University Medical Center
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
19
Lessons from Stanfordrsquos CheXNet
AI Is Effective on Narrow Tasks
Recent advances in AI have focused on
solving narrowly defined tasks Each
sub-problem (eg pneumonia) is a model
Handle ldquoNormalrdquo Explicitly
Identifying normal conditions versus
anomalies can help recognize when
human review may be needed
Explainable1 AI (XAI)
Models that can be interpreted at least
partially are easier to trust deploy and
govern Explainability is an area of rapid
progress in AI research
1) Also known as interpretable AI
Training Requires a Teacher or Referee
Machine learning needs to be told what
answers are correct or what actions produce
positive outcomes Well-curated data are
critical to the learning process
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
20
The AI Value Proposition
AI Excels at Decisions That Are Dull Detailed and Dear (Expensive)
Three Key Areas of Benefit
Dear
Provide speed capacity 24x7
availability and consistency
at a fraction of the cost
Although these are well
known benefits of automation
when you apply this to
cognitive and complex tasks
itrsquos a new ball game
Detailed
Evaluate a vastly broader
and deeper set of data to
improve decision quality
The complexity of health
care decision making is
growing due to new data
from genetics IoT devices
wearables better
interoperability and the
growing number of drugs
and treatments
Unlimited capacity
instant response
disruptive economics
Perform at super-human
levels exceeding experts
Serves as an assistant
(watches your back)
Dull
Free human decision makers
to focus on more challenging
ldquotop of licenserdquo activities
Reduce effort spent on
repetitive tasks process
large amounts of complex
information provide
continuous monitoring and
notifications and ensure
ldquostuffrdquo doesnrsquot ldquofall through
the cracksrdquo
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
21
Where to Start Where Wersquore Going
1) Narrow (Task Weak) AI focuses on narrowly defined problems such as predicting a patientrsquos risk of developing sepsis
2) General (Strong) AI attempts to match the flexibility and versatility of human intelligence
Narr
ow
(T
ask W
ea
k)
AI1
Clinical Decisions
Administrative Decisions
Ge
ne
ral (S
tro
ng
) A
I2
Full Diagnosis
and Treatment Plan
Clinical
Chatbots
Radiology
Interpretation
Acute
Clinical
Risk
Recruiting
Safety Risk Smart
Monitoring
Medication Dosing
Administrative
Chatbots
(Concierge)
Precision
Engagement
Precision
Medicine
Ambient
Documentation
Adaptive User
Interfaces
Worklist
Prioritization
Inevitable with Time
Todayrsquos Opportunity
(growing)
The Computer Will
See You Now
The Uber-ization of
Health Care
An AI Application Landscape
Capacity Staffing
Management
Population
Health Risk
The Self-Driving
HospitalInformation
Security
Targeted
Marketing
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
22
The Self-Driving Hospital
Combine Predictive Models with Front-Line Activation
Sources ldquoVirtual Air Traffic Control for Emergency Departmentsrdquo Qventus
httpsqventuscomcontentuploads201703QVE-ED-Solution-Briefpdf Health Care
IT Advisor research and analysis
1) EVS = Environmental services
2) LWBS = Left without being seen
3) LOS = Length of stay
Case in Brief
Mercy Hospital Fort Smith
bull 336-bed acute care hospital in Fort
Smith AR
bull Partnered with Qventus to improve
emergency department (ED) patient flow
and patient satisfaction
bull Analytics applies ML algorithms to
anticipate potential capacity shortfalls
process bottlenecks
bull Application sends real-time notifications
(nudges) to care teams via their mobile
phones to trigger specific interventions
bull Emergency department results
ndash LWBS2 rates dropped 30
ndash Door-to-doc time reduced by 20
ndash 24-minute reduction in average LOS3
ndash Annual case capacity increased by 6
See that a radiology order is delayed for a
patient within a few days of discharge and
notify radiology to prioritize the reading
Activate more EVS1 staff when high churn
of beds is anticipated
Send an early warning for difficult patient
placements at discharge
Prioritize inpatient transports depending
on expected bed needs
Sample ldquoDecision Recipesrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
23
Applicants Like You Liked This Job
Predictive Models Reduce Turnover Improve Engagement
Case in Brief MultiCare Health System
bull Non-profit system based in Tacoma WA with 16000 employees
bull Partnered with Arena a cloud-based HR solution to predict the
likelihood that a candidate will be retained in a role
bull Arenarsquos machine learning algorithms evaluate applicant
submissions digital interactions and externally sourced data on
the organization (eg employer review sites)
bull Achieved a 40 reduction in RN turnover at 180 days and a
28 reduction in overall turnover at 180 days
MultiCarersquos Hiring Process
Reduced recruiting
expenses
More predictable
staffing levels
More experienced
staff
Realized Benefits
Algorithms predict the likelihood the
applicant will be retained and
engaged in the target role and send
that information to the recruiter
Applicant completes
a 15-20 minute
assessment process
on Arenarsquos platform
Recruiters determine
whether or not to send
the candidate along to
hiring managers
Hiring managers use
Arenarsquos recommendation
as an optional factor in
the hiring decision
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
24
Accelerating Research
1) Yu KH et al ldquoPredicting non-small cell lung cancer prognosis by fully automated microscopic pathology image featuresrdquo Nature No 12474 (2016)
2) Rajwa B et al ldquoAutomated assessment of disease progression in acute myeloid leukemia by probabilistic analysis of flow cytometry datardquo IEEE Transactions on Biomedical Engineering 64 No 5 (2017) 1089-1098
3) Liu Y et al Detecting cancer metastases on gigapixel pathology images arXiv preprint arXiv1703 No 02442 (2017)
4) Rajpurkar P et al Cardiologist-level arrhythmia detection with convolutional neural networks arXiv preprint arXiv1707 No 01836 (2017)
5) Nielsen S et al A predictive model to identify Parkinson disease from administrative claims data Neurology 89 No 14 (2017) 1448-1456
6) Rajpurkar P et al CheXNet Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning arXiv preprint arXiv 1711 No 05225 (2017)
7) Poplin R et al Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning Nature Biomedical Engineering No 2 (2018)
8) ACC Machine learning techniques show promise for supporting medical decisions By quickly assessing risk new algorithms may help triage patients and inform clinical decisions ScienceDaily 2018
9) Oh J et al ldquoA Generalizable Data-Driven Approach to Predict Daily Risk of Clostridium difficile Infection at Two Large Academic Health Centersrdquo ICHE 39 No 4 (2018) 425-433
Nature1 ndash August 2016
ML algorithm can accurately
differentiate two types of lung
cancers and predict patient
survival time
IEEE2 ndash February 2017
ML model predicts relapse rates for
acute myelogenous leukemia with
90 accuracy
arXiv3 ndash March 2017
ML approach to imaging analytics
can identify metastasized breast
cancer with accuracy rates rivaling
those of pathologists
Neurology5 ndash Sept 2017
Predictive model to identify
Parkinsonrsquos disease from claims
data achieved 73-83 accuracy in
study results
arXiv4 ndash July 2017
ML algorithm can diagnose
heart arrhythmias using data
from wearables sensors with
cardiologist-level accuracy
arXiv6 ndash November 2017
Deep-learning algorithm
outperforms Stanford radiologists
at diagnosing pneumonia from
x-rays
American College of Cardiology8 ndash
Feb 2018
Model predicts heart attack
diagnosis among patients in the ED
with chest pain with ~90 accuracy
Nature7 ndash February 2018
ML algorithm can predict a
personrsquos cardiovascular risk
factors using eye scans
Infection Control Hospital
Epidemiology9 ndash April 2018
ML model predicts Clostridium
difficile infection (CDI) earlier
than current diagnostic methods
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
25
Driving the Decision Machine
1Acquire Data
bull Ensure all models have
a sponsor and regular
evaluation against goals
bull All models require
tune-ups as the
environment changes
Monitor Performance
and Revisit
42Train or Refine
the Model
bull The more the better
buthellip
bull High-quality well-
governed data pay
enormous dividends
bull Diverse cross-continuum
sources improve
predictions and
robustness of models
bull Decide what features
are important (eg
explainability)
bull Apply practical and
ethical constraints
bull Evaluate under
realistic conditions
(silent mode real-time
incomplete data)
bull Embed insights directly
into applications at key
decision points
bull Reengineer processes
as warranted
bull Consider the ldquo5 rightsrdquo
of decision support
3Incorporate Models
into Workflow
Hint The Analytics Is the Easy Part
Begin with
bull Explicit leadership agreement on outcome goals expected benefit mechanisms
bull Participation from empowered representatives of impacted process
bull External experiencemdashlearn from others
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
26
New Data Sources
Mainstream Systems Cover Only a Fraction of Predictive Power
Acquire Data
Sources Nash DB ldquoPopulation Health Why It Mattersrdquo Essentials in Population Health Childrenrsquos
Hospital Association Thomas Jefferson University October 2016 Health Care IT Advisor research and
analysis
1) RPM = Remote patient monitoring
EMR(s)
Lab Systems
Pharmacy Systems
Imaging
Medical Claims
Prescription Drug Claims
Billing and Supply Chain
Scheduling Systems
Medical
Care Data
10
Sensors RPM1
mHealth Apps
Patient-Reported
Outcomes
Genetics
Socioeconomic Data
Patient Questionnaires
Human
Biology
Data
20
Lifestyle
and
Behavior
Data
50
Social and
Environmental
Data
20
Data Sources by Influence on Health Outcomes
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
27
Big Data a Technology and a Philosophy
ldquoTherersquos Gold in Them Thar Hillsrdquo
Acquire Data
Sources Cloudera ldquoInova Translational Medicine Institute Partners with Cloudera to Advance Genome-Based Machine
Learning Initiatives and Save Livesrdquo PR Newswire June 2017 Health Care IT Advisor research and analysis
1) Originally coined by Doug Laney of the Meta Group
Assumes Value in the Data
Based on a premise that there are
valuable insights that can be distilled
from the mountains of data captured
by machines
Characterised by the 3 Vs1
bull Volume
bull Velocity
bull Variety
Places greater emphasis on
unstructured data
Free text images streams of sensor
data are all fair targets for analysis
Big data is the difference
between heart rates and
heart beats
Case in Brief Inova Health System
bull Inovarsquos Translational Medicine Institute
(ITMI) is assembling one of the worldrsquos
largest whole genome sequence
databases connected to patient
information in a health care system
bull Researchers use the Cloudera
machine learning platform to analyze
terabytes of clinical and genomic data
and identify genetic links to diseases
bull Insights are used to develop
personalized treatment plans for
patients in collaboration with the
treating physician
Resource Big Data in Health Care
Strengths challenges and pragmatic
guidelines for those considering the addition
of big data capabilities to their IT portfolio
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
30
How Machines Learn
Similar Methods Differing Goals
Build the Model
Source Advisory Board research and analysis
Input Data
bull The more the better
bull Labeled with key indicators
Adjust the Model
bull Automated trial and error
bull Cross-breeding successful
candidates
Repeat
Supervised Learning
Evaluate models against the ldquocorrectrdquo answers
provided with the data The learning process
focuses on minimizing errors
Unsupervised Learning
Evaluate models against an abstract algorithmic
goal Example Divide this population of diabetics
into a handful of subpopulations that have similar
future risk profiles
Reinforcement Learning
Evaluate models by giving a reward for
desirable outcomes and a penalty for negative
outcomes Models are trained to maximize their
total rewards over timeReady for Review
Evaluate Performance
bull Against what goal
bull Forces explicit
tradeoffs (eg cost
vs quality)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
32
Black Boxes Shades of Grey
Differing Views on the Importance of Explainability
Incorporate into Workflow
Mike Wall PharmD MBA
Chief Analytics Officer
The University of Chicago Medicine
The opaqueness of black-box models is
really scary When we canrsquot tell a
clinician why a patient wonrsquot do well on a
certain medication that can be a tough
thing to blindly trust from a clinical
perspective We want to reach a place
where we augment our practice
through AI rather than dictatingrdquo
Nigam Shah MBBS PhD
Associate Professor of Medicine
(Biomedical Informatics)
Stanford University
I am pushing to evaluate these models
via a similar set of mechanisms as RCT1
or AB testing We should evaluate
against the current status-quo and do a
beforeafter studyhellipWe have to convince
ourselves of the utility of these models
We should focus on utility before
explainabilityrdquo
Surgery is the
best option ()
Data
Black Box Model
1) RCT = Randomized controlled trial
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
33
Workflow Is Key to Adoption
Incorporate into Workflow
The Right Information
To the Right Person
In the Right Form
By the Right Channel
At the Right Time
Case in Brief Mayo Breast Oncology
Clinic Trials Recruitment
bull Up to 100 concurrent drug therapy clinical trials are
available to Mayo Clinic breast cancer patients making
it challenging to consistently identify candidates
bull New clinical trial matching technology developed in
collaboration with IBM Watson Health evaluates charts
to identify eligible patients
bull Launched in January 2016 as a point-of-care solution
for oncology providers initial adoption was low due to
time pressures and fell to zero by February
bull Rework by health systems engineers resulted in a July
2016 relaunch screening moved to occur in
advance of patient visits so clinicians know of trial
matches before opening a patientrsquos chart
bull New process enrolls significantly more patients in
clinical trials than the prior manual process success led
to additional coordinator funding
The Five Rights of
Clinical Decision Support
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
34
Health Carersquos Field Agents
Medical Devices Become Intelligent Partners
Incorporate into Workflow
1) EKG = Electrocardiogram
ME
DT
RO
NIC
MIN
IME
D 6
70G
Case in Brief Artificial Pancreas and Smart Infusion Pumpsmdash
Medtronic MiniMed Connect
bull SMARTGUARD mimics some functions of a healthy pancreas
measures and predicts glucose level drift and adjusts dosing
bull Insulin pump and continuous glucose monitoring can send data directly
to smartphone
bull Trials showed a 44 reduction in hypoglycemia and reduction in
average A1C values from 74 to 69
Case in Brief AliveCor KardiaMobile
bull Portable consumer-grade single-lead EKG1 retails for $100
bull Intended for self-capture of intermittent cardiac events such as atrial
fibrillation (AFIB)
bull Smartphone-based machine learning validates quality of trace
assesses possibility of AFIB classifies normal abnormal sinus rhythm
bull EKGs can be sent to a board-certified cardiologist for interpretation
for $19
Sources alivecorcom Medtroniccom Health Care IT Advisor research and
analysis
AliveC
orK
ard
iaM
obile
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
Health Care IT Advisor
The Decision MachineAnalytics and the Rise of AI
Greg Kuhnen
Senior Research Director
kuhnengadvisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
6
Sources ldquoAnalyticsrdquo Wikipedia last modified March 5 2018
httpsenwikipediaorgwikiAnalytics English Oxford Living Dictionaries Oxford Oxford
University Press 2018 (also available at httpsenoxforddictionariescom) Health Care IT Advisor
research and analysis
Analytics
The discovery interpretation and communication of meaningful
patterns in data
ndashWikipedia
Artificial Intelligence (AI)
The theory and development of computer systems able to
perform tasks normally requiring human intelligencehellip
ndashOxford English Dictionary
Machine Learning (ML)
The field of study that gives computers the ability to learn
without being explicitly programmed
ndashArthur Samuel 1959
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP7
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
8
Health Economics An Unsustainable Trajectory
Analytics Can Be a Powerful Tool for Bending the Cost Curve
Sources Cottarelli C ldquoMacro-Fiscal Implications of Health Care Reform in Advanced and Emerging Economiesrdquo IMF
Fiscal Affairs Department December 2010 McKinsey amp Company ldquomHealth A new vision for healthcarerdquo 2010
Office of Economic Co-operation and development lsquoHealth Spendingrdquo Health Care IT Advisor research and analysis
1) Global Burden of Disease Health Financing Collaborator Network ldquoFuture and potential spending on
health 2015-40 development assistance for health and government prepaid private and out-of-
pocket health spending in 184 countriesrdquo The Lancet 389 No 10083 (2016) 2) According to the
Australian Institute of Health and Welfare (AIHW) calculations 3) New Zealand Ministry of Health
projections 4) GDP = Gross domestic product
Per Capita Health Care Spending1 Health Expenditures as Percentage of GDP4
2016 20302016 2030
2016 2030 2016 2030
2016 2030
Australia2
96144
2010 2030
New Zealand3
62 89
106159
110
180
FranceCanada
United States United Kingdom
172249
97135
$9237
$5234
$4751
$4576
$4032
$4050
$3749
$3096
$12448
$7799
$6437
$5926
$5606
$5496
$5002
$4245
United States
Netherlands
Belgium
Canada
Australia
New Zealand
UnitedKingdom
Spain
Health Expenditure per Capita 2014
Health Expenditure per Capita 2030
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
10
Technology Drives the Flood of Data
Smartphones and Genomes and Sensors Oh My
Source Health Care IT Advisor research and analysis
1) IoT = Internet of things
2) RTLS = Real-time locating system
3) ROI = Return on investment
Common Barriers to
Adoption of New Sources
bull Privacy security concerns
bull Immature technology
bull Lack of budget
bull Unclear benefit ROI3
bull Staff already overwhelmed
with data
A Sampling of New Data Sources
Integrated Partners
(eg social services
cross-continuum care)
Data from Outside
Organizations
(eg social)
Environmental Data
(eg IoT1 RTLS2
temperature air
quality)
Genomes Proteomes
Microbiomes
Metabolomes
Clinical Surveillance
(eg public health
outbreak reports)
Remote Patient Monitoring
(eg bedside monitors
remote patient monitoring
consumer wearables)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
11
Cognitive Overload and Clinician Burnout
Sources Millett D Exclusive ldquoThree GPs a day seeking help for burnout
official data showrdquo GP Jan 2018 Health Care IT Advisor research and
analysis
1) Obermeyer Z Lee T ldquoLost in Thought ndash The Limits of the Human Mind and
the Future of Medicinerdquo N Engl J Med 377 no 13 (2017)1209-1211
Tim
e R
eq
uir
ed
ldquoMedical thinking has become vastly more complex mirroring changes in
our patients our health care system and medical science The complexity
of medicine now exceeds the capacity of the human mind1rdquo
Ziad Obermeyer MD and Thomas Lee MD
Number of Considerations for Decision Making
Significant
opportunity for
AI and analytics
Augmented Intelligence Reduces Cognitive Burden
Reliance on heuristics rules of thumb ldquogood-enoughrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
12
Data Power the Decision Engine
Source Health Care IT Advisor research and analysis
Four Levels of Analytic Capabilities
Data
Inform
Recommend
Decision
Automation
Decision
Support
Descriptive
What happened
How many of our
patients were
readmitted last
month
What is likely to happen
Is this patient likely to
readmit What do we
expect seasonally
Prescriptive
Recommend or
decide the best action
What should we do
to address this
patientrsquos risks
PredictiveDiagnostic
Why did it happen
Rates spiked was there
a change in case mix
Demographics
Decide
Hum
an
Decis
ion
Analy
tics
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP13
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
14
AI Are We There Yet
And if Not When
Source Health Care IT Advisor research and analysis
We May Be at the Start of an Explosive
Growth in AI Capabilities
Incremental Growth
AI continues to make
small advances
delivering value in
niche applicationsAI Winters AI has
already gone through
past phases of hype and
troughs of disillusionment
Exponential Growth
AI rapidly gains
capability and becomes
a mainstream part of
most digital systems
Unpredictable Timing Some
advances never seem to arrive
(conversational systems)
while others take off
unexpectedly (smartphones)
60s 70s 80s 90s 2000 2010 2020 203050s 2040 2050
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
15
Fuel for the AI Fire
Source Health Care IT Advisor research and analysis
1) EMR = Electronic medical record
2) GPU = Graphics processing unit
3) TPU = Tensor processing unit
AlgorithmsData
bull Internet Everywhere
bull Broad EMR1 Adoption
bull Wearables
bull Real-Time Location
bull IoT Sensors
bull Genomics Proteomics
Microbiome
bull Automated Feature
Extraction
bull Cluster Compute
Platforms (eg
Spark Hadoop)
bull Deep Learning
bull Off-the-Shelf
Libraries for
Common Tasksbull Faster Processors
bull Specialised Processors (GPUs2 TPUs3)
bull Inexpensive Storage
bull Elastic Cloud Computing
Hardware
Combinatorial Advances Delivering
Radically Better Models
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
16
The AI Winter Is Thawing
Source Health Care IT Advisor research and analysis
1) NASA = National Aeronautics and Space Administration
Microsoft and Alibaba outscore humans in a Stanford reading comprehension test 2018
Google DeepMindrsquos AlphaZero convincingly leads at Chess Shogi Go 2017
Skype launches real-time voice translation 2014
Mass market voice assistants (Apple Siri Google Now Microsoft Cortana) 2012
IBM Watson conclusively beats Jeopardy game show champions 2011
NASArsquos1 Mars Rovers operate semi-autonomously 2004
IBMrsquos Deep Blue beats world chess champion Gary Kasparov 1997
1950 Alan Turing proposes the ldquoTuring Testrdquo for conversational systems
1956 The term ldquoArtificial Intelligencerdquo is coined
1970s First AI Winter
1981 Digital Equipment Corporation order
expert saves company $40M per year
Timeline of AI Successes
From Toys to Tools
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
17
DeepMindrsquos AlphaZero
Novice to Superhuman in Just Four Hours
DeepMindrsquos AlphaZero Uses lsquoZero Human Knowledgersquo
Sources Vincent J ldquoDeepMindrsquos AI became a superhuman chess player in a few hours just for funrdquo The
Verge December 6 2017 Gershgorn D ldquoDeepMind wants to find the next miracle materialmdashexperts just
donrsquot know theyrsquoll pull it offrdquo Quartz October 25 2017 Health Care IT Advisor research and analysis
By not using this human data by not using
human features or human expertise in any
fashion wersquove actually rem
oved the constraints of human knowledge Itrsquos
able to therefore create knowledge for itselfrdquo
bull AI product of Google
subsidiary DeepMind
bull Provided no prior
knowledge of the
game given only basic
game rules and an
objective (win)
bull Learns through self-play
at an accelerated pace
(lsquoreinforced learningrsquo)
searches 80 thousand
positions per second
bull Variants of AlphaZero also
defeated world champion
Go and Shogi systems
AlphaZero
outperformed chess
world-champion
Stockfish1 in just
4 hours (300k steps)
By not using this human data by not using human
features or human expertise in any fashion wersquove
actually removed the constraints of human knowledge
Itrsquos able to therefore create knowledge for itselfrdquo
David Silver Lead Programmer
DEEPMIND
1) Stockfish is the 2016 Top Chess Engine Championship (TCEC) world-champion
AlphaZero Chess
Performance
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
18
Need a Radiologist
Therersquos an App for That
Sources Kubota T ldquoStanford algorithm can diagnose pneumonia better than
radiologistsrdquo Stanford News Nov 2017 Health Care IT Advisor research and analysis
1) Stanford University Medical Center
2) NIH = National Institutes of Health
Case in Brief Stanford University1
CheXNet Algorithm
bull Algorithm developed by researchers at
Stanford University in the US can diagnose
14 common pathologies in chest x-rays
bull Trained on ChestX-ray14 a public data set
released by the NIH2 containing 112120
frontal-view chest x-ray images labelled with
the 14 possible pathologies
bull Outperforms previous models from the same
data set for all 14 conditions and diagnoses
pneumonia at an accuracy exceeding the
performance of four control radiologists
bull Produces heatmaps that visualize the areas
of an image most indicative of disease
Development of CheXNet
Sept 26 2017
ChestX-ray14 data set
released along with a
preliminary algorithm
that could detect the
labelled conditionsasymp One week later
CheXNet could
diagnose 10 of the
14 pathologies more
accurately than all
previous algorithmsasymp One month later
CheXNet surpassed
best published results
for all 14 pathologies
and outperformed
Stanford radiologists in
detecting pneumonialdquoI treat the model as a lazy resident Theyrsquore not dumb theyrsquoll
just use every trick they can find to avoid doing hard workrdquo
Dr Matthew Lungren RadiologistStanford University Medical Center
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
19
Lessons from Stanfordrsquos CheXNet
AI Is Effective on Narrow Tasks
Recent advances in AI have focused on
solving narrowly defined tasks Each
sub-problem (eg pneumonia) is a model
Handle ldquoNormalrdquo Explicitly
Identifying normal conditions versus
anomalies can help recognize when
human review may be needed
Explainable1 AI (XAI)
Models that can be interpreted at least
partially are easier to trust deploy and
govern Explainability is an area of rapid
progress in AI research
1) Also known as interpretable AI
Training Requires a Teacher or Referee
Machine learning needs to be told what
answers are correct or what actions produce
positive outcomes Well-curated data are
critical to the learning process
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
20
The AI Value Proposition
AI Excels at Decisions That Are Dull Detailed and Dear (Expensive)
Three Key Areas of Benefit
Dear
Provide speed capacity 24x7
availability and consistency
at a fraction of the cost
Although these are well
known benefits of automation
when you apply this to
cognitive and complex tasks
itrsquos a new ball game
Detailed
Evaluate a vastly broader
and deeper set of data to
improve decision quality
The complexity of health
care decision making is
growing due to new data
from genetics IoT devices
wearables better
interoperability and the
growing number of drugs
and treatments
Unlimited capacity
instant response
disruptive economics
Perform at super-human
levels exceeding experts
Serves as an assistant
(watches your back)
Dull
Free human decision makers
to focus on more challenging
ldquotop of licenserdquo activities
Reduce effort spent on
repetitive tasks process
large amounts of complex
information provide
continuous monitoring and
notifications and ensure
ldquostuffrdquo doesnrsquot ldquofall through
the cracksrdquo
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
21
Where to Start Where Wersquore Going
1) Narrow (Task Weak) AI focuses on narrowly defined problems such as predicting a patientrsquos risk of developing sepsis
2) General (Strong) AI attempts to match the flexibility and versatility of human intelligence
Narr
ow
(T
ask W
ea
k)
AI1
Clinical Decisions
Administrative Decisions
Ge
ne
ral (S
tro
ng
) A
I2
Full Diagnosis
and Treatment Plan
Clinical
Chatbots
Radiology
Interpretation
Acute
Clinical
Risk
Recruiting
Safety Risk Smart
Monitoring
Medication Dosing
Administrative
Chatbots
(Concierge)
Precision
Engagement
Precision
Medicine
Ambient
Documentation
Adaptive User
Interfaces
Worklist
Prioritization
Inevitable with Time
Todayrsquos Opportunity
(growing)
The Computer Will
See You Now
The Uber-ization of
Health Care
An AI Application Landscape
Capacity Staffing
Management
Population
Health Risk
The Self-Driving
HospitalInformation
Security
Targeted
Marketing
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
22
The Self-Driving Hospital
Combine Predictive Models with Front-Line Activation
Sources ldquoVirtual Air Traffic Control for Emergency Departmentsrdquo Qventus
httpsqventuscomcontentuploads201703QVE-ED-Solution-Briefpdf Health Care
IT Advisor research and analysis
1) EVS = Environmental services
2) LWBS = Left without being seen
3) LOS = Length of stay
Case in Brief
Mercy Hospital Fort Smith
bull 336-bed acute care hospital in Fort
Smith AR
bull Partnered with Qventus to improve
emergency department (ED) patient flow
and patient satisfaction
bull Analytics applies ML algorithms to
anticipate potential capacity shortfalls
process bottlenecks
bull Application sends real-time notifications
(nudges) to care teams via their mobile
phones to trigger specific interventions
bull Emergency department results
ndash LWBS2 rates dropped 30
ndash Door-to-doc time reduced by 20
ndash 24-minute reduction in average LOS3
ndash Annual case capacity increased by 6
See that a radiology order is delayed for a
patient within a few days of discharge and
notify radiology to prioritize the reading
Activate more EVS1 staff when high churn
of beds is anticipated
Send an early warning for difficult patient
placements at discharge
Prioritize inpatient transports depending
on expected bed needs
Sample ldquoDecision Recipesrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
23
Applicants Like You Liked This Job
Predictive Models Reduce Turnover Improve Engagement
Case in Brief MultiCare Health System
bull Non-profit system based in Tacoma WA with 16000 employees
bull Partnered with Arena a cloud-based HR solution to predict the
likelihood that a candidate will be retained in a role
bull Arenarsquos machine learning algorithms evaluate applicant
submissions digital interactions and externally sourced data on
the organization (eg employer review sites)
bull Achieved a 40 reduction in RN turnover at 180 days and a
28 reduction in overall turnover at 180 days
MultiCarersquos Hiring Process
Reduced recruiting
expenses
More predictable
staffing levels
More experienced
staff
Realized Benefits
Algorithms predict the likelihood the
applicant will be retained and
engaged in the target role and send
that information to the recruiter
Applicant completes
a 15-20 minute
assessment process
on Arenarsquos platform
Recruiters determine
whether or not to send
the candidate along to
hiring managers
Hiring managers use
Arenarsquos recommendation
as an optional factor in
the hiring decision
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
24
Accelerating Research
1) Yu KH et al ldquoPredicting non-small cell lung cancer prognosis by fully automated microscopic pathology image featuresrdquo Nature No 12474 (2016)
2) Rajwa B et al ldquoAutomated assessment of disease progression in acute myeloid leukemia by probabilistic analysis of flow cytometry datardquo IEEE Transactions on Biomedical Engineering 64 No 5 (2017) 1089-1098
3) Liu Y et al Detecting cancer metastases on gigapixel pathology images arXiv preprint arXiv1703 No 02442 (2017)
4) Rajpurkar P et al Cardiologist-level arrhythmia detection with convolutional neural networks arXiv preprint arXiv1707 No 01836 (2017)
5) Nielsen S et al A predictive model to identify Parkinson disease from administrative claims data Neurology 89 No 14 (2017) 1448-1456
6) Rajpurkar P et al CheXNet Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning arXiv preprint arXiv 1711 No 05225 (2017)
7) Poplin R et al Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning Nature Biomedical Engineering No 2 (2018)
8) ACC Machine learning techniques show promise for supporting medical decisions By quickly assessing risk new algorithms may help triage patients and inform clinical decisions ScienceDaily 2018
9) Oh J et al ldquoA Generalizable Data-Driven Approach to Predict Daily Risk of Clostridium difficile Infection at Two Large Academic Health Centersrdquo ICHE 39 No 4 (2018) 425-433
Nature1 ndash August 2016
ML algorithm can accurately
differentiate two types of lung
cancers and predict patient
survival time
IEEE2 ndash February 2017
ML model predicts relapse rates for
acute myelogenous leukemia with
90 accuracy
arXiv3 ndash March 2017
ML approach to imaging analytics
can identify metastasized breast
cancer with accuracy rates rivaling
those of pathologists
Neurology5 ndash Sept 2017
Predictive model to identify
Parkinsonrsquos disease from claims
data achieved 73-83 accuracy in
study results
arXiv4 ndash July 2017
ML algorithm can diagnose
heart arrhythmias using data
from wearables sensors with
cardiologist-level accuracy
arXiv6 ndash November 2017
Deep-learning algorithm
outperforms Stanford radiologists
at diagnosing pneumonia from
x-rays
American College of Cardiology8 ndash
Feb 2018
Model predicts heart attack
diagnosis among patients in the ED
with chest pain with ~90 accuracy
Nature7 ndash February 2018
ML algorithm can predict a
personrsquos cardiovascular risk
factors using eye scans
Infection Control Hospital
Epidemiology9 ndash April 2018
ML model predicts Clostridium
difficile infection (CDI) earlier
than current diagnostic methods
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
25
Driving the Decision Machine
1Acquire Data
bull Ensure all models have
a sponsor and regular
evaluation against goals
bull All models require
tune-ups as the
environment changes
Monitor Performance
and Revisit
42Train or Refine
the Model
bull The more the better
buthellip
bull High-quality well-
governed data pay
enormous dividends
bull Diverse cross-continuum
sources improve
predictions and
robustness of models
bull Decide what features
are important (eg
explainability)
bull Apply practical and
ethical constraints
bull Evaluate under
realistic conditions
(silent mode real-time
incomplete data)
bull Embed insights directly
into applications at key
decision points
bull Reengineer processes
as warranted
bull Consider the ldquo5 rightsrdquo
of decision support
3Incorporate Models
into Workflow
Hint The Analytics Is the Easy Part
Begin with
bull Explicit leadership agreement on outcome goals expected benefit mechanisms
bull Participation from empowered representatives of impacted process
bull External experiencemdashlearn from others
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
26
New Data Sources
Mainstream Systems Cover Only a Fraction of Predictive Power
Acquire Data
Sources Nash DB ldquoPopulation Health Why It Mattersrdquo Essentials in Population Health Childrenrsquos
Hospital Association Thomas Jefferson University October 2016 Health Care IT Advisor research and
analysis
1) RPM = Remote patient monitoring
EMR(s)
Lab Systems
Pharmacy Systems
Imaging
Medical Claims
Prescription Drug Claims
Billing and Supply Chain
Scheduling Systems
Medical
Care Data
10
Sensors RPM1
mHealth Apps
Patient-Reported
Outcomes
Genetics
Socioeconomic Data
Patient Questionnaires
Human
Biology
Data
20
Lifestyle
and
Behavior
Data
50
Social and
Environmental
Data
20
Data Sources by Influence on Health Outcomes
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
27
Big Data a Technology and a Philosophy
ldquoTherersquos Gold in Them Thar Hillsrdquo
Acquire Data
Sources Cloudera ldquoInova Translational Medicine Institute Partners with Cloudera to Advance Genome-Based Machine
Learning Initiatives and Save Livesrdquo PR Newswire June 2017 Health Care IT Advisor research and analysis
1) Originally coined by Doug Laney of the Meta Group
Assumes Value in the Data
Based on a premise that there are
valuable insights that can be distilled
from the mountains of data captured
by machines
Characterised by the 3 Vs1
bull Volume
bull Velocity
bull Variety
Places greater emphasis on
unstructured data
Free text images streams of sensor
data are all fair targets for analysis
Big data is the difference
between heart rates and
heart beats
Case in Brief Inova Health System
bull Inovarsquos Translational Medicine Institute
(ITMI) is assembling one of the worldrsquos
largest whole genome sequence
databases connected to patient
information in a health care system
bull Researchers use the Cloudera
machine learning platform to analyze
terabytes of clinical and genomic data
and identify genetic links to diseases
bull Insights are used to develop
personalized treatment plans for
patients in collaboration with the
treating physician
Resource Big Data in Health Care
Strengths challenges and pragmatic
guidelines for those considering the addition
of big data capabilities to their IT portfolio
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
30
How Machines Learn
Similar Methods Differing Goals
Build the Model
Source Advisory Board research and analysis
Input Data
bull The more the better
bull Labeled with key indicators
Adjust the Model
bull Automated trial and error
bull Cross-breeding successful
candidates
Repeat
Supervised Learning
Evaluate models against the ldquocorrectrdquo answers
provided with the data The learning process
focuses on minimizing errors
Unsupervised Learning
Evaluate models against an abstract algorithmic
goal Example Divide this population of diabetics
into a handful of subpopulations that have similar
future risk profiles
Reinforcement Learning
Evaluate models by giving a reward for
desirable outcomes and a penalty for negative
outcomes Models are trained to maximize their
total rewards over timeReady for Review
Evaluate Performance
bull Against what goal
bull Forces explicit
tradeoffs (eg cost
vs quality)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
32
Black Boxes Shades of Grey
Differing Views on the Importance of Explainability
Incorporate into Workflow
Mike Wall PharmD MBA
Chief Analytics Officer
The University of Chicago Medicine
The opaqueness of black-box models is
really scary When we canrsquot tell a
clinician why a patient wonrsquot do well on a
certain medication that can be a tough
thing to blindly trust from a clinical
perspective We want to reach a place
where we augment our practice
through AI rather than dictatingrdquo
Nigam Shah MBBS PhD
Associate Professor of Medicine
(Biomedical Informatics)
Stanford University
I am pushing to evaluate these models
via a similar set of mechanisms as RCT1
or AB testing We should evaluate
against the current status-quo and do a
beforeafter studyhellipWe have to convince
ourselves of the utility of these models
We should focus on utility before
explainabilityrdquo
Surgery is the
best option ()
Data
Black Box Model
1) RCT = Randomized controlled trial
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
33
Workflow Is Key to Adoption
Incorporate into Workflow
The Right Information
To the Right Person
In the Right Form
By the Right Channel
At the Right Time
Case in Brief Mayo Breast Oncology
Clinic Trials Recruitment
bull Up to 100 concurrent drug therapy clinical trials are
available to Mayo Clinic breast cancer patients making
it challenging to consistently identify candidates
bull New clinical trial matching technology developed in
collaboration with IBM Watson Health evaluates charts
to identify eligible patients
bull Launched in January 2016 as a point-of-care solution
for oncology providers initial adoption was low due to
time pressures and fell to zero by February
bull Rework by health systems engineers resulted in a July
2016 relaunch screening moved to occur in
advance of patient visits so clinicians know of trial
matches before opening a patientrsquos chart
bull New process enrolls significantly more patients in
clinical trials than the prior manual process success led
to additional coordinator funding
The Five Rights of
Clinical Decision Support
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
34
Health Carersquos Field Agents
Medical Devices Become Intelligent Partners
Incorporate into Workflow
1) EKG = Electrocardiogram
ME
DT
RO
NIC
MIN
IME
D 6
70G
Case in Brief Artificial Pancreas and Smart Infusion Pumpsmdash
Medtronic MiniMed Connect
bull SMARTGUARD mimics some functions of a healthy pancreas
measures and predicts glucose level drift and adjusts dosing
bull Insulin pump and continuous glucose monitoring can send data directly
to smartphone
bull Trials showed a 44 reduction in hypoglycemia and reduction in
average A1C values from 74 to 69
Case in Brief AliveCor KardiaMobile
bull Portable consumer-grade single-lead EKG1 retails for $100
bull Intended for self-capture of intermittent cardiac events such as atrial
fibrillation (AFIB)
bull Smartphone-based machine learning validates quality of trace
assesses possibility of AFIB classifies normal abnormal sinus rhythm
bull EKGs can be sent to a board-certified cardiologist for interpretation
for $19
Sources alivecorcom Medtroniccom Health Care IT Advisor research and
analysis
AliveC
orK
ard
iaM
obile
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
6
Sources ldquoAnalyticsrdquo Wikipedia last modified March 5 2018
httpsenwikipediaorgwikiAnalytics English Oxford Living Dictionaries Oxford Oxford
University Press 2018 (also available at httpsenoxforddictionariescom) Health Care IT Advisor
research and analysis
Analytics
The discovery interpretation and communication of meaningful
patterns in data
ndashWikipedia
Artificial Intelligence (AI)
The theory and development of computer systems able to
perform tasks normally requiring human intelligencehellip
ndashOxford English Dictionary
Machine Learning (ML)
The field of study that gives computers the ability to learn
without being explicitly programmed
ndashArthur Samuel 1959
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP7
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
8
Health Economics An Unsustainable Trajectory
Analytics Can Be a Powerful Tool for Bending the Cost Curve
Sources Cottarelli C ldquoMacro-Fiscal Implications of Health Care Reform in Advanced and Emerging Economiesrdquo IMF
Fiscal Affairs Department December 2010 McKinsey amp Company ldquomHealth A new vision for healthcarerdquo 2010
Office of Economic Co-operation and development lsquoHealth Spendingrdquo Health Care IT Advisor research and analysis
1) Global Burden of Disease Health Financing Collaborator Network ldquoFuture and potential spending on
health 2015-40 development assistance for health and government prepaid private and out-of-
pocket health spending in 184 countriesrdquo The Lancet 389 No 10083 (2016) 2) According to the
Australian Institute of Health and Welfare (AIHW) calculations 3) New Zealand Ministry of Health
projections 4) GDP = Gross domestic product
Per Capita Health Care Spending1 Health Expenditures as Percentage of GDP4
2016 20302016 2030
2016 2030 2016 2030
2016 2030
Australia2
96144
2010 2030
New Zealand3
62 89
106159
110
180
FranceCanada
United States United Kingdom
172249
97135
$9237
$5234
$4751
$4576
$4032
$4050
$3749
$3096
$12448
$7799
$6437
$5926
$5606
$5496
$5002
$4245
United States
Netherlands
Belgium
Canada
Australia
New Zealand
UnitedKingdom
Spain
Health Expenditure per Capita 2014
Health Expenditure per Capita 2030
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
10
Technology Drives the Flood of Data
Smartphones and Genomes and Sensors Oh My
Source Health Care IT Advisor research and analysis
1) IoT = Internet of things
2) RTLS = Real-time locating system
3) ROI = Return on investment
Common Barriers to
Adoption of New Sources
bull Privacy security concerns
bull Immature technology
bull Lack of budget
bull Unclear benefit ROI3
bull Staff already overwhelmed
with data
A Sampling of New Data Sources
Integrated Partners
(eg social services
cross-continuum care)
Data from Outside
Organizations
(eg social)
Environmental Data
(eg IoT1 RTLS2
temperature air
quality)
Genomes Proteomes
Microbiomes
Metabolomes
Clinical Surveillance
(eg public health
outbreak reports)
Remote Patient Monitoring
(eg bedside monitors
remote patient monitoring
consumer wearables)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
11
Cognitive Overload and Clinician Burnout
Sources Millett D Exclusive ldquoThree GPs a day seeking help for burnout
official data showrdquo GP Jan 2018 Health Care IT Advisor research and
analysis
1) Obermeyer Z Lee T ldquoLost in Thought ndash The Limits of the Human Mind and
the Future of Medicinerdquo N Engl J Med 377 no 13 (2017)1209-1211
Tim
e R
eq
uir
ed
ldquoMedical thinking has become vastly more complex mirroring changes in
our patients our health care system and medical science The complexity
of medicine now exceeds the capacity of the human mind1rdquo
Ziad Obermeyer MD and Thomas Lee MD
Number of Considerations for Decision Making
Significant
opportunity for
AI and analytics
Augmented Intelligence Reduces Cognitive Burden
Reliance on heuristics rules of thumb ldquogood-enoughrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
12
Data Power the Decision Engine
Source Health Care IT Advisor research and analysis
Four Levels of Analytic Capabilities
Data
Inform
Recommend
Decision
Automation
Decision
Support
Descriptive
What happened
How many of our
patients were
readmitted last
month
What is likely to happen
Is this patient likely to
readmit What do we
expect seasonally
Prescriptive
Recommend or
decide the best action
What should we do
to address this
patientrsquos risks
PredictiveDiagnostic
Why did it happen
Rates spiked was there
a change in case mix
Demographics
Decide
Hum
an
Decis
ion
Analy
tics
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP13
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
14
AI Are We There Yet
And if Not When
Source Health Care IT Advisor research and analysis
We May Be at the Start of an Explosive
Growth in AI Capabilities
Incremental Growth
AI continues to make
small advances
delivering value in
niche applicationsAI Winters AI has
already gone through
past phases of hype and
troughs of disillusionment
Exponential Growth
AI rapidly gains
capability and becomes
a mainstream part of
most digital systems
Unpredictable Timing Some
advances never seem to arrive
(conversational systems)
while others take off
unexpectedly (smartphones)
60s 70s 80s 90s 2000 2010 2020 203050s 2040 2050
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
15
Fuel for the AI Fire
Source Health Care IT Advisor research and analysis
1) EMR = Electronic medical record
2) GPU = Graphics processing unit
3) TPU = Tensor processing unit
AlgorithmsData
bull Internet Everywhere
bull Broad EMR1 Adoption
bull Wearables
bull Real-Time Location
bull IoT Sensors
bull Genomics Proteomics
Microbiome
bull Automated Feature
Extraction
bull Cluster Compute
Platforms (eg
Spark Hadoop)
bull Deep Learning
bull Off-the-Shelf
Libraries for
Common Tasksbull Faster Processors
bull Specialised Processors (GPUs2 TPUs3)
bull Inexpensive Storage
bull Elastic Cloud Computing
Hardware
Combinatorial Advances Delivering
Radically Better Models
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
16
The AI Winter Is Thawing
Source Health Care IT Advisor research and analysis
1) NASA = National Aeronautics and Space Administration
Microsoft and Alibaba outscore humans in a Stanford reading comprehension test 2018
Google DeepMindrsquos AlphaZero convincingly leads at Chess Shogi Go 2017
Skype launches real-time voice translation 2014
Mass market voice assistants (Apple Siri Google Now Microsoft Cortana) 2012
IBM Watson conclusively beats Jeopardy game show champions 2011
NASArsquos1 Mars Rovers operate semi-autonomously 2004
IBMrsquos Deep Blue beats world chess champion Gary Kasparov 1997
1950 Alan Turing proposes the ldquoTuring Testrdquo for conversational systems
1956 The term ldquoArtificial Intelligencerdquo is coined
1970s First AI Winter
1981 Digital Equipment Corporation order
expert saves company $40M per year
Timeline of AI Successes
From Toys to Tools
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
17
DeepMindrsquos AlphaZero
Novice to Superhuman in Just Four Hours
DeepMindrsquos AlphaZero Uses lsquoZero Human Knowledgersquo
Sources Vincent J ldquoDeepMindrsquos AI became a superhuman chess player in a few hours just for funrdquo The
Verge December 6 2017 Gershgorn D ldquoDeepMind wants to find the next miracle materialmdashexperts just
donrsquot know theyrsquoll pull it offrdquo Quartz October 25 2017 Health Care IT Advisor research and analysis
By not using this human data by not using
human features or human expertise in any
fashion wersquove actually rem
oved the constraints of human knowledge Itrsquos
able to therefore create knowledge for itselfrdquo
bull AI product of Google
subsidiary DeepMind
bull Provided no prior
knowledge of the
game given only basic
game rules and an
objective (win)
bull Learns through self-play
at an accelerated pace
(lsquoreinforced learningrsquo)
searches 80 thousand
positions per second
bull Variants of AlphaZero also
defeated world champion
Go and Shogi systems
AlphaZero
outperformed chess
world-champion
Stockfish1 in just
4 hours (300k steps)
By not using this human data by not using human
features or human expertise in any fashion wersquove
actually removed the constraints of human knowledge
Itrsquos able to therefore create knowledge for itselfrdquo
David Silver Lead Programmer
DEEPMIND
1) Stockfish is the 2016 Top Chess Engine Championship (TCEC) world-champion
AlphaZero Chess
Performance
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
18
Need a Radiologist
Therersquos an App for That
Sources Kubota T ldquoStanford algorithm can diagnose pneumonia better than
radiologistsrdquo Stanford News Nov 2017 Health Care IT Advisor research and analysis
1) Stanford University Medical Center
2) NIH = National Institutes of Health
Case in Brief Stanford University1
CheXNet Algorithm
bull Algorithm developed by researchers at
Stanford University in the US can diagnose
14 common pathologies in chest x-rays
bull Trained on ChestX-ray14 a public data set
released by the NIH2 containing 112120
frontal-view chest x-ray images labelled with
the 14 possible pathologies
bull Outperforms previous models from the same
data set for all 14 conditions and diagnoses
pneumonia at an accuracy exceeding the
performance of four control radiologists
bull Produces heatmaps that visualize the areas
of an image most indicative of disease
Development of CheXNet
Sept 26 2017
ChestX-ray14 data set
released along with a
preliminary algorithm
that could detect the
labelled conditionsasymp One week later
CheXNet could
diagnose 10 of the
14 pathologies more
accurately than all
previous algorithmsasymp One month later
CheXNet surpassed
best published results
for all 14 pathologies
and outperformed
Stanford radiologists in
detecting pneumonialdquoI treat the model as a lazy resident Theyrsquore not dumb theyrsquoll
just use every trick they can find to avoid doing hard workrdquo
Dr Matthew Lungren RadiologistStanford University Medical Center
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
19
Lessons from Stanfordrsquos CheXNet
AI Is Effective on Narrow Tasks
Recent advances in AI have focused on
solving narrowly defined tasks Each
sub-problem (eg pneumonia) is a model
Handle ldquoNormalrdquo Explicitly
Identifying normal conditions versus
anomalies can help recognize when
human review may be needed
Explainable1 AI (XAI)
Models that can be interpreted at least
partially are easier to trust deploy and
govern Explainability is an area of rapid
progress in AI research
1) Also known as interpretable AI
Training Requires a Teacher or Referee
Machine learning needs to be told what
answers are correct or what actions produce
positive outcomes Well-curated data are
critical to the learning process
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
20
The AI Value Proposition
AI Excels at Decisions That Are Dull Detailed and Dear (Expensive)
Three Key Areas of Benefit
Dear
Provide speed capacity 24x7
availability and consistency
at a fraction of the cost
Although these are well
known benefits of automation
when you apply this to
cognitive and complex tasks
itrsquos a new ball game
Detailed
Evaluate a vastly broader
and deeper set of data to
improve decision quality
The complexity of health
care decision making is
growing due to new data
from genetics IoT devices
wearables better
interoperability and the
growing number of drugs
and treatments
Unlimited capacity
instant response
disruptive economics
Perform at super-human
levels exceeding experts
Serves as an assistant
(watches your back)
Dull
Free human decision makers
to focus on more challenging
ldquotop of licenserdquo activities
Reduce effort spent on
repetitive tasks process
large amounts of complex
information provide
continuous monitoring and
notifications and ensure
ldquostuffrdquo doesnrsquot ldquofall through
the cracksrdquo
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
21
Where to Start Where Wersquore Going
1) Narrow (Task Weak) AI focuses on narrowly defined problems such as predicting a patientrsquos risk of developing sepsis
2) General (Strong) AI attempts to match the flexibility and versatility of human intelligence
Narr
ow
(T
ask W
ea
k)
AI1
Clinical Decisions
Administrative Decisions
Ge
ne
ral (S
tro
ng
) A
I2
Full Diagnosis
and Treatment Plan
Clinical
Chatbots
Radiology
Interpretation
Acute
Clinical
Risk
Recruiting
Safety Risk Smart
Monitoring
Medication Dosing
Administrative
Chatbots
(Concierge)
Precision
Engagement
Precision
Medicine
Ambient
Documentation
Adaptive User
Interfaces
Worklist
Prioritization
Inevitable with Time
Todayrsquos Opportunity
(growing)
The Computer Will
See You Now
The Uber-ization of
Health Care
An AI Application Landscape
Capacity Staffing
Management
Population
Health Risk
The Self-Driving
HospitalInformation
Security
Targeted
Marketing
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
22
The Self-Driving Hospital
Combine Predictive Models with Front-Line Activation
Sources ldquoVirtual Air Traffic Control for Emergency Departmentsrdquo Qventus
httpsqventuscomcontentuploads201703QVE-ED-Solution-Briefpdf Health Care
IT Advisor research and analysis
1) EVS = Environmental services
2) LWBS = Left without being seen
3) LOS = Length of stay
Case in Brief
Mercy Hospital Fort Smith
bull 336-bed acute care hospital in Fort
Smith AR
bull Partnered with Qventus to improve
emergency department (ED) patient flow
and patient satisfaction
bull Analytics applies ML algorithms to
anticipate potential capacity shortfalls
process bottlenecks
bull Application sends real-time notifications
(nudges) to care teams via their mobile
phones to trigger specific interventions
bull Emergency department results
ndash LWBS2 rates dropped 30
ndash Door-to-doc time reduced by 20
ndash 24-minute reduction in average LOS3
ndash Annual case capacity increased by 6
See that a radiology order is delayed for a
patient within a few days of discharge and
notify radiology to prioritize the reading
Activate more EVS1 staff when high churn
of beds is anticipated
Send an early warning for difficult patient
placements at discharge
Prioritize inpatient transports depending
on expected bed needs
Sample ldquoDecision Recipesrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
23
Applicants Like You Liked This Job
Predictive Models Reduce Turnover Improve Engagement
Case in Brief MultiCare Health System
bull Non-profit system based in Tacoma WA with 16000 employees
bull Partnered with Arena a cloud-based HR solution to predict the
likelihood that a candidate will be retained in a role
bull Arenarsquos machine learning algorithms evaluate applicant
submissions digital interactions and externally sourced data on
the organization (eg employer review sites)
bull Achieved a 40 reduction in RN turnover at 180 days and a
28 reduction in overall turnover at 180 days
MultiCarersquos Hiring Process
Reduced recruiting
expenses
More predictable
staffing levels
More experienced
staff
Realized Benefits
Algorithms predict the likelihood the
applicant will be retained and
engaged in the target role and send
that information to the recruiter
Applicant completes
a 15-20 minute
assessment process
on Arenarsquos platform
Recruiters determine
whether or not to send
the candidate along to
hiring managers
Hiring managers use
Arenarsquos recommendation
as an optional factor in
the hiring decision
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
24
Accelerating Research
1) Yu KH et al ldquoPredicting non-small cell lung cancer prognosis by fully automated microscopic pathology image featuresrdquo Nature No 12474 (2016)
2) Rajwa B et al ldquoAutomated assessment of disease progression in acute myeloid leukemia by probabilistic analysis of flow cytometry datardquo IEEE Transactions on Biomedical Engineering 64 No 5 (2017) 1089-1098
3) Liu Y et al Detecting cancer metastases on gigapixel pathology images arXiv preprint arXiv1703 No 02442 (2017)
4) Rajpurkar P et al Cardiologist-level arrhythmia detection with convolutional neural networks arXiv preprint arXiv1707 No 01836 (2017)
5) Nielsen S et al A predictive model to identify Parkinson disease from administrative claims data Neurology 89 No 14 (2017) 1448-1456
6) Rajpurkar P et al CheXNet Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning arXiv preprint arXiv 1711 No 05225 (2017)
7) Poplin R et al Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning Nature Biomedical Engineering No 2 (2018)
8) ACC Machine learning techniques show promise for supporting medical decisions By quickly assessing risk new algorithms may help triage patients and inform clinical decisions ScienceDaily 2018
9) Oh J et al ldquoA Generalizable Data-Driven Approach to Predict Daily Risk of Clostridium difficile Infection at Two Large Academic Health Centersrdquo ICHE 39 No 4 (2018) 425-433
Nature1 ndash August 2016
ML algorithm can accurately
differentiate two types of lung
cancers and predict patient
survival time
IEEE2 ndash February 2017
ML model predicts relapse rates for
acute myelogenous leukemia with
90 accuracy
arXiv3 ndash March 2017
ML approach to imaging analytics
can identify metastasized breast
cancer with accuracy rates rivaling
those of pathologists
Neurology5 ndash Sept 2017
Predictive model to identify
Parkinsonrsquos disease from claims
data achieved 73-83 accuracy in
study results
arXiv4 ndash July 2017
ML algorithm can diagnose
heart arrhythmias using data
from wearables sensors with
cardiologist-level accuracy
arXiv6 ndash November 2017
Deep-learning algorithm
outperforms Stanford radiologists
at diagnosing pneumonia from
x-rays
American College of Cardiology8 ndash
Feb 2018
Model predicts heart attack
diagnosis among patients in the ED
with chest pain with ~90 accuracy
Nature7 ndash February 2018
ML algorithm can predict a
personrsquos cardiovascular risk
factors using eye scans
Infection Control Hospital
Epidemiology9 ndash April 2018
ML model predicts Clostridium
difficile infection (CDI) earlier
than current diagnostic methods
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
25
Driving the Decision Machine
1Acquire Data
bull Ensure all models have
a sponsor and regular
evaluation against goals
bull All models require
tune-ups as the
environment changes
Monitor Performance
and Revisit
42Train or Refine
the Model
bull The more the better
buthellip
bull High-quality well-
governed data pay
enormous dividends
bull Diverse cross-continuum
sources improve
predictions and
robustness of models
bull Decide what features
are important (eg
explainability)
bull Apply practical and
ethical constraints
bull Evaluate under
realistic conditions
(silent mode real-time
incomplete data)
bull Embed insights directly
into applications at key
decision points
bull Reengineer processes
as warranted
bull Consider the ldquo5 rightsrdquo
of decision support
3Incorporate Models
into Workflow
Hint The Analytics Is the Easy Part
Begin with
bull Explicit leadership agreement on outcome goals expected benefit mechanisms
bull Participation from empowered representatives of impacted process
bull External experiencemdashlearn from others
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
26
New Data Sources
Mainstream Systems Cover Only a Fraction of Predictive Power
Acquire Data
Sources Nash DB ldquoPopulation Health Why It Mattersrdquo Essentials in Population Health Childrenrsquos
Hospital Association Thomas Jefferson University October 2016 Health Care IT Advisor research and
analysis
1) RPM = Remote patient monitoring
EMR(s)
Lab Systems
Pharmacy Systems
Imaging
Medical Claims
Prescription Drug Claims
Billing and Supply Chain
Scheduling Systems
Medical
Care Data
10
Sensors RPM1
mHealth Apps
Patient-Reported
Outcomes
Genetics
Socioeconomic Data
Patient Questionnaires
Human
Biology
Data
20
Lifestyle
and
Behavior
Data
50
Social and
Environmental
Data
20
Data Sources by Influence on Health Outcomes
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
27
Big Data a Technology and a Philosophy
ldquoTherersquos Gold in Them Thar Hillsrdquo
Acquire Data
Sources Cloudera ldquoInova Translational Medicine Institute Partners with Cloudera to Advance Genome-Based Machine
Learning Initiatives and Save Livesrdquo PR Newswire June 2017 Health Care IT Advisor research and analysis
1) Originally coined by Doug Laney of the Meta Group
Assumes Value in the Data
Based on a premise that there are
valuable insights that can be distilled
from the mountains of data captured
by machines
Characterised by the 3 Vs1
bull Volume
bull Velocity
bull Variety
Places greater emphasis on
unstructured data
Free text images streams of sensor
data are all fair targets for analysis
Big data is the difference
between heart rates and
heart beats
Case in Brief Inova Health System
bull Inovarsquos Translational Medicine Institute
(ITMI) is assembling one of the worldrsquos
largest whole genome sequence
databases connected to patient
information in a health care system
bull Researchers use the Cloudera
machine learning platform to analyze
terabytes of clinical and genomic data
and identify genetic links to diseases
bull Insights are used to develop
personalized treatment plans for
patients in collaboration with the
treating physician
Resource Big Data in Health Care
Strengths challenges and pragmatic
guidelines for those considering the addition
of big data capabilities to their IT portfolio
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
30
How Machines Learn
Similar Methods Differing Goals
Build the Model
Source Advisory Board research and analysis
Input Data
bull The more the better
bull Labeled with key indicators
Adjust the Model
bull Automated trial and error
bull Cross-breeding successful
candidates
Repeat
Supervised Learning
Evaluate models against the ldquocorrectrdquo answers
provided with the data The learning process
focuses on minimizing errors
Unsupervised Learning
Evaluate models against an abstract algorithmic
goal Example Divide this population of diabetics
into a handful of subpopulations that have similar
future risk profiles
Reinforcement Learning
Evaluate models by giving a reward for
desirable outcomes and a penalty for negative
outcomes Models are trained to maximize their
total rewards over timeReady for Review
Evaluate Performance
bull Against what goal
bull Forces explicit
tradeoffs (eg cost
vs quality)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
32
Black Boxes Shades of Grey
Differing Views on the Importance of Explainability
Incorporate into Workflow
Mike Wall PharmD MBA
Chief Analytics Officer
The University of Chicago Medicine
The opaqueness of black-box models is
really scary When we canrsquot tell a
clinician why a patient wonrsquot do well on a
certain medication that can be a tough
thing to blindly trust from a clinical
perspective We want to reach a place
where we augment our practice
through AI rather than dictatingrdquo
Nigam Shah MBBS PhD
Associate Professor of Medicine
(Biomedical Informatics)
Stanford University
I am pushing to evaluate these models
via a similar set of mechanisms as RCT1
or AB testing We should evaluate
against the current status-quo and do a
beforeafter studyhellipWe have to convince
ourselves of the utility of these models
We should focus on utility before
explainabilityrdquo
Surgery is the
best option ()
Data
Black Box Model
1) RCT = Randomized controlled trial
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
33
Workflow Is Key to Adoption
Incorporate into Workflow
The Right Information
To the Right Person
In the Right Form
By the Right Channel
At the Right Time
Case in Brief Mayo Breast Oncology
Clinic Trials Recruitment
bull Up to 100 concurrent drug therapy clinical trials are
available to Mayo Clinic breast cancer patients making
it challenging to consistently identify candidates
bull New clinical trial matching technology developed in
collaboration with IBM Watson Health evaluates charts
to identify eligible patients
bull Launched in January 2016 as a point-of-care solution
for oncology providers initial adoption was low due to
time pressures and fell to zero by February
bull Rework by health systems engineers resulted in a July
2016 relaunch screening moved to occur in
advance of patient visits so clinicians know of trial
matches before opening a patientrsquos chart
bull New process enrolls significantly more patients in
clinical trials than the prior manual process success led
to additional coordinator funding
The Five Rights of
Clinical Decision Support
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
34
Health Carersquos Field Agents
Medical Devices Become Intelligent Partners
Incorporate into Workflow
1) EKG = Electrocardiogram
ME
DT
RO
NIC
MIN
IME
D 6
70G
Case in Brief Artificial Pancreas and Smart Infusion Pumpsmdash
Medtronic MiniMed Connect
bull SMARTGUARD mimics some functions of a healthy pancreas
measures and predicts glucose level drift and adjusts dosing
bull Insulin pump and continuous glucose monitoring can send data directly
to smartphone
bull Trials showed a 44 reduction in hypoglycemia and reduction in
average A1C values from 74 to 69
Case in Brief AliveCor KardiaMobile
bull Portable consumer-grade single-lead EKG1 retails for $100
bull Intended for self-capture of intermittent cardiac events such as atrial
fibrillation (AFIB)
bull Smartphone-based machine learning validates quality of trace
assesses possibility of AFIB classifies normal abnormal sinus rhythm
bull EKGs can be sent to a board-certified cardiologist for interpretation
for $19
Sources alivecorcom Medtroniccom Health Care IT Advisor research and
analysis
AliveC
orK
ard
iaM
obile
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP7
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
8
Health Economics An Unsustainable Trajectory
Analytics Can Be a Powerful Tool for Bending the Cost Curve
Sources Cottarelli C ldquoMacro-Fiscal Implications of Health Care Reform in Advanced and Emerging Economiesrdquo IMF
Fiscal Affairs Department December 2010 McKinsey amp Company ldquomHealth A new vision for healthcarerdquo 2010
Office of Economic Co-operation and development lsquoHealth Spendingrdquo Health Care IT Advisor research and analysis
1) Global Burden of Disease Health Financing Collaborator Network ldquoFuture and potential spending on
health 2015-40 development assistance for health and government prepaid private and out-of-
pocket health spending in 184 countriesrdquo The Lancet 389 No 10083 (2016) 2) According to the
Australian Institute of Health and Welfare (AIHW) calculations 3) New Zealand Ministry of Health
projections 4) GDP = Gross domestic product
Per Capita Health Care Spending1 Health Expenditures as Percentage of GDP4
2016 20302016 2030
2016 2030 2016 2030
2016 2030
Australia2
96144
2010 2030
New Zealand3
62 89
106159
110
180
FranceCanada
United States United Kingdom
172249
97135
$9237
$5234
$4751
$4576
$4032
$4050
$3749
$3096
$12448
$7799
$6437
$5926
$5606
$5496
$5002
$4245
United States
Netherlands
Belgium
Canada
Australia
New Zealand
UnitedKingdom
Spain
Health Expenditure per Capita 2014
Health Expenditure per Capita 2030
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
10
Technology Drives the Flood of Data
Smartphones and Genomes and Sensors Oh My
Source Health Care IT Advisor research and analysis
1) IoT = Internet of things
2) RTLS = Real-time locating system
3) ROI = Return on investment
Common Barriers to
Adoption of New Sources
bull Privacy security concerns
bull Immature technology
bull Lack of budget
bull Unclear benefit ROI3
bull Staff already overwhelmed
with data
A Sampling of New Data Sources
Integrated Partners
(eg social services
cross-continuum care)
Data from Outside
Organizations
(eg social)
Environmental Data
(eg IoT1 RTLS2
temperature air
quality)
Genomes Proteomes
Microbiomes
Metabolomes
Clinical Surveillance
(eg public health
outbreak reports)
Remote Patient Monitoring
(eg bedside monitors
remote patient monitoring
consumer wearables)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
11
Cognitive Overload and Clinician Burnout
Sources Millett D Exclusive ldquoThree GPs a day seeking help for burnout
official data showrdquo GP Jan 2018 Health Care IT Advisor research and
analysis
1) Obermeyer Z Lee T ldquoLost in Thought ndash The Limits of the Human Mind and
the Future of Medicinerdquo N Engl J Med 377 no 13 (2017)1209-1211
Tim
e R
eq
uir
ed
ldquoMedical thinking has become vastly more complex mirroring changes in
our patients our health care system and medical science The complexity
of medicine now exceeds the capacity of the human mind1rdquo
Ziad Obermeyer MD and Thomas Lee MD
Number of Considerations for Decision Making
Significant
opportunity for
AI and analytics
Augmented Intelligence Reduces Cognitive Burden
Reliance on heuristics rules of thumb ldquogood-enoughrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
12
Data Power the Decision Engine
Source Health Care IT Advisor research and analysis
Four Levels of Analytic Capabilities
Data
Inform
Recommend
Decision
Automation
Decision
Support
Descriptive
What happened
How many of our
patients were
readmitted last
month
What is likely to happen
Is this patient likely to
readmit What do we
expect seasonally
Prescriptive
Recommend or
decide the best action
What should we do
to address this
patientrsquos risks
PredictiveDiagnostic
Why did it happen
Rates spiked was there
a change in case mix
Demographics
Decide
Hum
an
Decis
ion
Analy
tics
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP13
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
14
AI Are We There Yet
And if Not When
Source Health Care IT Advisor research and analysis
We May Be at the Start of an Explosive
Growth in AI Capabilities
Incremental Growth
AI continues to make
small advances
delivering value in
niche applicationsAI Winters AI has
already gone through
past phases of hype and
troughs of disillusionment
Exponential Growth
AI rapidly gains
capability and becomes
a mainstream part of
most digital systems
Unpredictable Timing Some
advances never seem to arrive
(conversational systems)
while others take off
unexpectedly (smartphones)
60s 70s 80s 90s 2000 2010 2020 203050s 2040 2050
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
15
Fuel for the AI Fire
Source Health Care IT Advisor research and analysis
1) EMR = Electronic medical record
2) GPU = Graphics processing unit
3) TPU = Tensor processing unit
AlgorithmsData
bull Internet Everywhere
bull Broad EMR1 Adoption
bull Wearables
bull Real-Time Location
bull IoT Sensors
bull Genomics Proteomics
Microbiome
bull Automated Feature
Extraction
bull Cluster Compute
Platforms (eg
Spark Hadoop)
bull Deep Learning
bull Off-the-Shelf
Libraries for
Common Tasksbull Faster Processors
bull Specialised Processors (GPUs2 TPUs3)
bull Inexpensive Storage
bull Elastic Cloud Computing
Hardware
Combinatorial Advances Delivering
Radically Better Models
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
16
The AI Winter Is Thawing
Source Health Care IT Advisor research and analysis
1) NASA = National Aeronautics and Space Administration
Microsoft and Alibaba outscore humans in a Stanford reading comprehension test 2018
Google DeepMindrsquos AlphaZero convincingly leads at Chess Shogi Go 2017
Skype launches real-time voice translation 2014
Mass market voice assistants (Apple Siri Google Now Microsoft Cortana) 2012
IBM Watson conclusively beats Jeopardy game show champions 2011
NASArsquos1 Mars Rovers operate semi-autonomously 2004
IBMrsquos Deep Blue beats world chess champion Gary Kasparov 1997
1950 Alan Turing proposes the ldquoTuring Testrdquo for conversational systems
1956 The term ldquoArtificial Intelligencerdquo is coined
1970s First AI Winter
1981 Digital Equipment Corporation order
expert saves company $40M per year
Timeline of AI Successes
From Toys to Tools
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
17
DeepMindrsquos AlphaZero
Novice to Superhuman in Just Four Hours
DeepMindrsquos AlphaZero Uses lsquoZero Human Knowledgersquo
Sources Vincent J ldquoDeepMindrsquos AI became a superhuman chess player in a few hours just for funrdquo The
Verge December 6 2017 Gershgorn D ldquoDeepMind wants to find the next miracle materialmdashexperts just
donrsquot know theyrsquoll pull it offrdquo Quartz October 25 2017 Health Care IT Advisor research and analysis
By not using this human data by not using
human features or human expertise in any
fashion wersquove actually rem
oved the constraints of human knowledge Itrsquos
able to therefore create knowledge for itselfrdquo
bull AI product of Google
subsidiary DeepMind
bull Provided no prior
knowledge of the
game given only basic
game rules and an
objective (win)
bull Learns through self-play
at an accelerated pace
(lsquoreinforced learningrsquo)
searches 80 thousand
positions per second
bull Variants of AlphaZero also
defeated world champion
Go and Shogi systems
AlphaZero
outperformed chess
world-champion
Stockfish1 in just
4 hours (300k steps)
By not using this human data by not using human
features or human expertise in any fashion wersquove
actually removed the constraints of human knowledge
Itrsquos able to therefore create knowledge for itselfrdquo
David Silver Lead Programmer
DEEPMIND
1) Stockfish is the 2016 Top Chess Engine Championship (TCEC) world-champion
AlphaZero Chess
Performance
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
18
Need a Radiologist
Therersquos an App for That
Sources Kubota T ldquoStanford algorithm can diagnose pneumonia better than
radiologistsrdquo Stanford News Nov 2017 Health Care IT Advisor research and analysis
1) Stanford University Medical Center
2) NIH = National Institutes of Health
Case in Brief Stanford University1
CheXNet Algorithm
bull Algorithm developed by researchers at
Stanford University in the US can diagnose
14 common pathologies in chest x-rays
bull Trained on ChestX-ray14 a public data set
released by the NIH2 containing 112120
frontal-view chest x-ray images labelled with
the 14 possible pathologies
bull Outperforms previous models from the same
data set for all 14 conditions and diagnoses
pneumonia at an accuracy exceeding the
performance of four control radiologists
bull Produces heatmaps that visualize the areas
of an image most indicative of disease
Development of CheXNet
Sept 26 2017
ChestX-ray14 data set
released along with a
preliminary algorithm
that could detect the
labelled conditionsasymp One week later
CheXNet could
diagnose 10 of the
14 pathologies more
accurately than all
previous algorithmsasymp One month later
CheXNet surpassed
best published results
for all 14 pathologies
and outperformed
Stanford radiologists in
detecting pneumonialdquoI treat the model as a lazy resident Theyrsquore not dumb theyrsquoll
just use every trick they can find to avoid doing hard workrdquo
Dr Matthew Lungren RadiologistStanford University Medical Center
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
19
Lessons from Stanfordrsquos CheXNet
AI Is Effective on Narrow Tasks
Recent advances in AI have focused on
solving narrowly defined tasks Each
sub-problem (eg pneumonia) is a model
Handle ldquoNormalrdquo Explicitly
Identifying normal conditions versus
anomalies can help recognize when
human review may be needed
Explainable1 AI (XAI)
Models that can be interpreted at least
partially are easier to trust deploy and
govern Explainability is an area of rapid
progress in AI research
1) Also known as interpretable AI
Training Requires a Teacher or Referee
Machine learning needs to be told what
answers are correct or what actions produce
positive outcomes Well-curated data are
critical to the learning process
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
20
The AI Value Proposition
AI Excels at Decisions That Are Dull Detailed and Dear (Expensive)
Three Key Areas of Benefit
Dear
Provide speed capacity 24x7
availability and consistency
at a fraction of the cost
Although these are well
known benefits of automation
when you apply this to
cognitive and complex tasks
itrsquos a new ball game
Detailed
Evaluate a vastly broader
and deeper set of data to
improve decision quality
The complexity of health
care decision making is
growing due to new data
from genetics IoT devices
wearables better
interoperability and the
growing number of drugs
and treatments
Unlimited capacity
instant response
disruptive economics
Perform at super-human
levels exceeding experts
Serves as an assistant
(watches your back)
Dull
Free human decision makers
to focus on more challenging
ldquotop of licenserdquo activities
Reduce effort spent on
repetitive tasks process
large amounts of complex
information provide
continuous monitoring and
notifications and ensure
ldquostuffrdquo doesnrsquot ldquofall through
the cracksrdquo
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
21
Where to Start Where Wersquore Going
1) Narrow (Task Weak) AI focuses on narrowly defined problems such as predicting a patientrsquos risk of developing sepsis
2) General (Strong) AI attempts to match the flexibility and versatility of human intelligence
Narr
ow
(T
ask W
ea
k)
AI1
Clinical Decisions
Administrative Decisions
Ge
ne
ral (S
tro
ng
) A
I2
Full Diagnosis
and Treatment Plan
Clinical
Chatbots
Radiology
Interpretation
Acute
Clinical
Risk
Recruiting
Safety Risk Smart
Monitoring
Medication Dosing
Administrative
Chatbots
(Concierge)
Precision
Engagement
Precision
Medicine
Ambient
Documentation
Adaptive User
Interfaces
Worklist
Prioritization
Inevitable with Time
Todayrsquos Opportunity
(growing)
The Computer Will
See You Now
The Uber-ization of
Health Care
An AI Application Landscape
Capacity Staffing
Management
Population
Health Risk
The Self-Driving
HospitalInformation
Security
Targeted
Marketing
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
22
The Self-Driving Hospital
Combine Predictive Models with Front-Line Activation
Sources ldquoVirtual Air Traffic Control for Emergency Departmentsrdquo Qventus
httpsqventuscomcontentuploads201703QVE-ED-Solution-Briefpdf Health Care
IT Advisor research and analysis
1) EVS = Environmental services
2) LWBS = Left without being seen
3) LOS = Length of stay
Case in Brief
Mercy Hospital Fort Smith
bull 336-bed acute care hospital in Fort
Smith AR
bull Partnered with Qventus to improve
emergency department (ED) patient flow
and patient satisfaction
bull Analytics applies ML algorithms to
anticipate potential capacity shortfalls
process bottlenecks
bull Application sends real-time notifications
(nudges) to care teams via their mobile
phones to trigger specific interventions
bull Emergency department results
ndash LWBS2 rates dropped 30
ndash Door-to-doc time reduced by 20
ndash 24-minute reduction in average LOS3
ndash Annual case capacity increased by 6
See that a radiology order is delayed for a
patient within a few days of discharge and
notify radiology to prioritize the reading
Activate more EVS1 staff when high churn
of beds is anticipated
Send an early warning for difficult patient
placements at discharge
Prioritize inpatient transports depending
on expected bed needs
Sample ldquoDecision Recipesrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
23
Applicants Like You Liked This Job
Predictive Models Reduce Turnover Improve Engagement
Case in Brief MultiCare Health System
bull Non-profit system based in Tacoma WA with 16000 employees
bull Partnered with Arena a cloud-based HR solution to predict the
likelihood that a candidate will be retained in a role
bull Arenarsquos machine learning algorithms evaluate applicant
submissions digital interactions and externally sourced data on
the organization (eg employer review sites)
bull Achieved a 40 reduction in RN turnover at 180 days and a
28 reduction in overall turnover at 180 days
MultiCarersquos Hiring Process
Reduced recruiting
expenses
More predictable
staffing levels
More experienced
staff
Realized Benefits
Algorithms predict the likelihood the
applicant will be retained and
engaged in the target role and send
that information to the recruiter
Applicant completes
a 15-20 minute
assessment process
on Arenarsquos platform
Recruiters determine
whether or not to send
the candidate along to
hiring managers
Hiring managers use
Arenarsquos recommendation
as an optional factor in
the hiring decision
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
24
Accelerating Research
1) Yu KH et al ldquoPredicting non-small cell lung cancer prognosis by fully automated microscopic pathology image featuresrdquo Nature No 12474 (2016)
2) Rajwa B et al ldquoAutomated assessment of disease progression in acute myeloid leukemia by probabilistic analysis of flow cytometry datardquo IEEE Transactions on Biomedical Engineering 64 No 5 (2017) 1089-1098
3) Liu Y et al Detecting cancer metastases on gigapixel pathology images arXiv preprint arXiv1703 No 02442 (2017)
4) Rajpurkar P et al Cardiologist-level arrhythmia detection with convolutional neural networks arXiv preprint arXiv1707 No 01836 (2017)
5) Nielsen S et al A predictive model to identify Parkinson disease from administrative claims data Neurology 89 No 14 (2017) 1448-1456
6) Rajpurkar P et al CheXNet Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning arXiv preprint arXiv 1711 No 05225 (2017)
7) Poplin R et al Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning Nature Biomedical Engineering No 2 (2018)
8) ACC Machine learning techniques show promise for supporting medical decisions By quickly assessing risk new algorithms may help triage patients and inform clinical decisions ScienceDaily 2018
9) Oh J et al ldquoA Generalizable Data-Driven Approach to Predict Daily Risk of Clostridium difficile Infection at Two Large Academic Health Centersrdquo ICHE 39 No 4 (2018) 425-433
Nature1 ndash August 2016
ML algorithm can accurately
differentiate two types of lung
cancers and predict patient
survival time
IEEE2 ndash February 2017
ML model predicts relapse rates for
acute myelogenous leukemia with
90 accuracy
arXiv3 ndash March 2017
ML approach to imaging analytics
can identify metastasized breast
cancer with accuracy rates rivaling
those of pathologists
Neurology5 ndash Sept 2017
Predictive model to identify
Parkinsonrsquos disease from claims
data achieved 73-83 accuracy in
study results
arXiv4 ndash July 2017
ML algorithm can diagnose
heart arrhythmias using data
from wearables sensors with
cardiologist-level accuracy
arXiv6 ndash November 2017
Deep-learning algorithm
outperforms Stanford radiologists
at diagnosing pneumonia from
x-rays
American College of Cardiology8 ndash
Feb 2018
Model predicts heart attack
diagnosis among patients in the ED
with chest pain with ~90 accuracy
Nature7 ndash February 2018
ML algorithm can predict a
personrsquos cardiovascular risk
factors using eye scans
Infection Control Hospital
Epidemiology9 ndash April 2018
ML model predicts Clostridium
difficile infection (CDI) earlier
than current diagnostic methods
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
25
Driving the Decision Machine
1Acquire Data
bull Ensure all models have
a sponsor and regular
evaluation against goals
bull All models require
tune-ups as the
environment changes
Monitor Performance
and Revisit
42Train or Refine
the Model
bull The more the better
buthellip
bull High-quality well-
governed data pay
enormous dividends
bull Diverse cross-continuum
sources improve
predictions and
robustness of models
bull Decide what features
are important (eg
explainability)
bull Apply practical and
ethical constraints
bull Evaluate under
realistic conditions
(silent mode real-time
incomplete data)
bull Embed insights directly
into applications at key
decision points
bull Reengineer processes
as warranted
bull Consider the ldquo5 rightsrdquo
of decision support
3Incorporate Models
into Workflow
Hint The Analytics Is the Easy Part
Begin with
bull Explicit leadership agreement on outcome goals expected benefit mechanisms
bull Participation from empowered representatives of impacted process
bull External experiencemdashlearn from others
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
26
New Data Sources
Mainstream Systems Cover Only a Fraction of Predictive Power
Acquire Data
Sources Nash DB ldquoPopulation Health Why It Mattersrdquo Essentials in Population Health Childrenrsquos
Hospital Association Thomas Jefferson University October 2016 Health Care IT Advisor research and
analysis
1) RPM = Remote patient monitoring
EMR(s)
Lab Systems
Pharmacy Systems
Imaging
Medical Claims
Prescription Drug Claims
Billing and Supply Chain
Scheduling Systems
Medical
Care Data
10
Sensors RPM1
mHealth Apps
Patient-Reported
Outcomes
Genetics
Socioeconomic Data
Patient Questionnaires
Human
Biology
Data
20
Lifestyle
and
Behavior
Data
50
Social and
Environmental
Data
20
Data Sources by Influence on Health Outcomes
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
27
Big Data a Technology and a Philosophy
ldquoTherersquos Gold in Them Thar Hillsrdquo
Acquire Data
Sources Cloudera ldquoInova Translational Medicine Institute Partners with Cloudera to Advance Genome-Based Machine
Learning Initiatives and Save Livesrdquo PR Newswire June 2017 Health Care IT Advisor research and analysis
1) Originally coined by Doug Laney of the Meta Group
Assumes Value in the Data
Based on a premise that there are
valuable insights that can be distilled
from the mountains of data captured
by machines
Characterised by the 3 Vs1
bull Volume
bull Velocity
bull Variety
Places greater emphasis on
unstructured data
Free text images streams of sensor
data are all fair targets for analysis
Big data is the difference
between heart rates and
heart beats
Case in Brief Inova Health System
bull Inovarsquos Translational Medicine Institute
(ITMI) is assembling one of the worldrsquos
largest whole genome sequence
databases connected to patient
information in a health care system
bull Researchers use the Cloudera
machine learning platform to analyze
terabytes of clinical and genomic data
and identify genetic links to diseases
bull Insights are used to develop
personalized treatment plans for
patients in collaboration with the
treating physician
Resource Big Data in Health Care
Strengths challenges and pragmatic
guidelines for those considering the addition
of big data capabilities to their IT portfolio
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
30
How Machines Learn
Similar Methods Differing Goals
Build the Model
Source Advisory Board research and analysis
Input Data
bull The more the better
bull Labeled with key indicators
Adjust the Model
bull Automated trial and error
bull Cross-breeding successful
candidates
Repeat
Supervised Learning
Evaluate models against the ldquocorrectrdquo answers
provided with the data The learning process
focuses on minimizing errors
Unsupervised Learning
Evaluate models against an abstract algorithmic
goal Example Divide this population of diabetics
into a handful of subpopulations that have similar
future risk profiles
Reinforcement Learning
Evaluate models by giving a reward for
desirable outcomes and a penalty for negative
outcomes Models are trained to maximize their
total rewards over timeReady for Review
Evaluate Performance
bull Against what goal
bull Forces explicit
tradeoffs (eg cost
vs quality)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
32
Black Boxes Shades of Grey
Differing Views on the Importance of Explainability
Incorporate into Workflow
Mike Wall PharmD MBA
Chief Analytics Officer
The University of Chicago Medicine
The opaqueness of black-box models is
really scary When we canrsquot tell a
clinician why a patient wonrsquot do well on a
certain medication that can be a tough
thing to blindly trust from a clinical
perspective We want to reach a place
where we augment our practice
through AI rather than dictatingrdquo
Nigam Shah MBBS PhD
Associate Professor of Medicine
(Biomedical Informatics)
Stanford University
I am pushing to evaluate these models
via a similar set of mechanisms as RCT1
or AB testing We should evaluate
against the current status-quo and do a
beforeafter studyhellipWe have to convince
ourselves of the utility of these models
We should focus on utility before
explainabilityrdquo
Surgery is the
best option ()
Data
Black Box Model
1) RCT = Randomized controlled trial
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
33
Workflow Is Key to Adoption
Incorporate into Workflow
The Right Information
To the Right Person
In the Right Form
By the Right Channel
At the Right Time
Case in Brief Mayo Breast Oncology
Clinic Trials Recruitment
bull Up to 100 concurrent drug therapy clinical trials are
available to Mayo Clinic breast cancer patients making
it challenging to consistently identify candidates
bull New clinical trial matching technology developed in
collaboration with IBM Watson Health evaluates charts
to identify eligible patients
bull Launched in January 2016 as a point-of-care solution
for oncology providers initial adoption was low due to
time pressures and fell to zero by February
bull Rework by health systems engineers resulted in a July
2016 relaunch screening moved to occur in
advance of patient visits so clinicians know of trial
matches before opening a patientrsquos chart
bull New process enrolls significantly more patients in
clinical trials than the prior manual process success led
to additional coordinator funding
The Five Rights of
Clinical Decision Support
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
34
Health Carersquos Field Agents
Medical Devices Become Intelligent Partners
Incorporate into Workflow
1) EKG = Electrocardiogram
ME
DT
RO
NIC
MIN
IME
D 6
70G
Case in Brief Artificial Pancreas and Smart Infusion Pumpsmdash
Medtronic MiniMed Connect
bull SMARTGUARD mimics some functions of a healthy pancreas
measures and predicts glucose level drift and adjusts dosing
bull Insulin pump and continuous glucose monitoring can send data directly
to smartphone
bull Trials showed a 44 reduction in hypoglycemia and reduction in
average A1C values from 74 to 69
Case in Brief AliveCor KardiaMobile
bull Portable consumer-grade single-lead EKG1 retails for $100
bull Intended for self-capture of intermittent cardiac events such as atrial
fibrillation (AFIB)
bull Smartphone-based machine learning validates quality of trace
assesses possibility of AFIB classifies normal abnormal sinus rhythm
bull EKGs can be sent to a board-certified cardiologist for interpretation
for $19
Sources alivecorcom Medtroniccom Health Care IT Advisor research and
analysis
AliveC
orK
ard
iaM
obile
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
8
Health Economics An Unsustainable Trajectory
Analytics Can Be a Powerful Tool for Bending the Cost Curve
Sources Cottarelli C ldquoMacro-Fiscal Implications of Health Care Reform in Advanced and Emerging Economiesrdquo IMF
Fiscal Affairs Department December 2010 McKinsey amp Company ldquomHealth A new vision for healthcarerdquo 2010
Office of Economic Co-operation and development lsquoHealth Spendingrdquo Health Care IT Advisor research and analysis
1) Global Burden of Disease Health Financing Collaborator Network ldquoFuture and potential spending on
health 2015-40 development assistance for health and government prepaid private and out-of-
pocket health spending in 184 countriesrdquo The Lancet 389 No 10083 (2016) 2) According to the
Australian Institute of Health and Welfare (AIHW) calculations 3) New Zealand Ministry of Health
projections 4) GDP = Gross domestic product
Per Capita Health Care Spending1 Health Expenditures as Percentage of GDP4
2016 20302016 2030
2016 2030 2016 2030
2016 2030
Australia2
96144
2010 2030
New Zealand3
62 89
106159
110
180
FranceCanada
United States United Kingdom
172249
97135
$9237
$5234
$4751
$4576
$4032
$4050
$3749
$3096
$12448
$7799
$6437
$5926
$5606
$5496
$5002
$4245
United States
Netherlands
Belgium
Canada
Australia
New Zealand
UnitedKingdom
Spain
Health Expenditure per Capita 2014
Health Expenditure per Capita 2030
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
10
Technology Drives the Flood of Data
Smartphones and Genomes and Sensors Oh My
Source Health Care IT Advisor research and analysis
1) IoT = Internet of things
2) RTLS = Real-time locating system
3) ROI = Return on investment
Common Barriers to
Adoption of New Sources
bull Privacy security concerns
bull Immature technology
bull Lack of budget
bull Unclear benefit ROI3
bull Staff already overwhelmed
with data
A Sampling of New Data Sources
Integrated Partners
(eg social services
cross-continuum care)
Data from Outside
Organizations
(eg social)
Environmental Data
(eg IoT1 RTLS2
temperature air
quality)
Genomes Proteomes
Microbiomes
Metabolomes
Clinical Surveillance
(eg public health
outbreak reports)
Remote Patient Monitoring
(eg bedside monitors
remote patient monitoring
consumer wearables)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
11
Cognitive Overload and Clinician Burnout
Sources Millett D Exclusive ldquoThree GPs a day seeking help for burnout
official data showrdquo GP Jan 2018 Health Care IT Advisor research and
analysis
1) Obermeyer Z Lee T ldquoLost in Thought ndash The Limits of the Human Mind and
the Future of Medicinerdquo N Engl J Med 377 no 13 (2017)1209-1211
Tim
e R
eq
uir
ed
ldquoMedical thinking has become vastly more complex mirroring changes in
our patients our health care system and medical science The complexity
of medicine now exceeds the capacity of the human mind1rdquo
Ziad Obermeyer MD and Thomas Lee MD
Number of Considerations for Decision Making
Significant
opportunity for
AI and analytics
Augmented Intelligence Reduces Cognitive Burden
Reliance on heuristics rules of thumb ldquogood-enoughrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
12
Data Power the Decision Engine
Source Health Care IT Advisor research and analysis
Four Levels of Analytic Capabilities
Data
Inform
Recommend
Decision
Automation
Decision
Support
Descriptive
What happened
How many of our
patients were
readmitted last
month
What is likely to happen
Is this patient likely to
readmit What do we
expect seasonally
Prescriptive
Recommend or
decide the best action
What should we do
to address this
patientrsquos risks
PredictiveDiagnostic
Why did it happen
Rates spiked was there
a change in case mix
Demographics
Decide
Hum
an
Decis
ion
Analy
tics
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP13
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
14
AI Are We There Yet
And if Not When
Source Health Care IT Advisor research and analysis
We May Be at the Start of an Explosive
Growth in AI Capabilities
Incremental Growth
AI continues to make
small advances
delivering value in
niche applicationsAI Winters AI has
already gone through
past phases of hype and
troughs of disillusionment
Exponential Growth
AI rapidly gains
capability and becomes
a mainstream part of
most digital systems
Unpredictable Timing Some
advances never seem to arrive
(conversational systems)
while others take off
unexpectedly (smartphones)
60s 70s 80s 90s 2000 2010 2020 203050s 2040 2050
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
15
Fuel for the AI Fire
Source Health Care IT Advisor research and analysis
1) EMR = Electronic medical record
2) GPU = Graphics processing unit
3) TPU = Tensor processing unit
AlgorithmsData
bull Internet Everywhere
bull Broad EMR1 Adoption
bull Wearables
bull Real-Time Location
bull IoT Sensors
bull Genomics Proteomics
Microbiome
bull Automated Feature
Extraction
bull Cluster Compute
Platforms (eg
Spark Hadoop)
bull Deep Learning
bull Off-the-Shelf
Libraries for
Common Tasksbull Faster Processors
bull Specialised Processors (GPUs2 TPUs3)
bull Inexpensive Storage
bull Elastic Cloud Computing
Hardware
Combinatorial Advances Delivering
Radically Better Models
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
16
The AI Winter Is Thawing
Source Health Care IT Advisor research and analysis
1) NASA = National Aeronautics and Space Administration
Microsoft and Alibaba outscore humans in a Stanford reading comprehension test 2018
Google DeepMindrsquos AlphaZero convincingly leads at Chess Shogi Go 2017
Skype launches real-time voice translation 2014
Mass market voice assistants (Apple Siri Google Now Microsoft Cortana) 2012
IBM Watson conclusively beats Jeopardy game show champions 2011
NASArsquos1 Mars Rovers operate semi-autonomously 2004
IBMrsquos Deep Blue beats world chess champion Gary Kasparov 1997
1950 Alan Turing proposes the ldquoTuring Testrdquo for conversational systems
1956 The term ldquoArtificial Intelligencerdquo is coined
1970s First AI Winter
1981 Digital Equipment Corporation order
expert saves company $40M per year
Timeline of AI Successes
From Toys to Tools
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
17
DeepMindrsquos AlphaZero
Novice to Superhuman in Just Four Hours
DeepMindrsquos AlphaZero Uses lsquoZero Human Knowledgersquo
Sources Vincent J ldquoDeepMindrsquos AI became a superhuman chess player in a few hours just for funrdquo The
Verge December 6 2017 Gershgorn D ldquoDeepMind wants to find the next miracle materialmdashexperts just
donrsquot know theyrsquoll pull it offrdquo Quartz October 25 2017 Health Care IT Advisor research and analysis
By not using this human data by not using
human features or human expertise in any
fashion wersquove actually rem
oved the constraints of human knowledge Itrsquos
able to therefore create knowledge for itselfrdquo
bull AI product of Google
subsidiary DeepMind
bull Provided no prior
knowledge of the
game given only basic
game rules and an
objective (win)
bull Learns through self-play
at an accelerated pace
(lsquoreinforced learningrsquo)
searches 80 thousand
positions per second
bull Variants of AlphaZero also
defeated world champion
Go and Shogi systems
AlphaZero
outperformed chess
world-champion
Stockfish1 in just
4 hours (300k steps)
By not using this human data by not using human
features or human expertise in any fashion wersquove
actually removed the constraints of human knowledge
Itrsquos able to therefore create knowledge for itselfrdquo
David Silver Lead Programmer
DEEPMIND
1) Stockfish is the 2016 Top Chess Engine Championship (TCEC) world-champion
AlphaZero Chess
Performance
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
18
Need a Radiologist
Therersquos an App for That
Sources Kubota T ldquoStanford algorithm can diagnose pneumonia better than
radiologistsrdquo Stanford News Nov 2017 Health Care IT Advisor research and analysis
1) Stanford University Medical Center
2) NIH = National Institutes of Health
Case in Brief Stanford University1
CheXNet Algorithm
bull Algorithm developed by researchers at
Stanford University in the US can diagnose
14 common pathologies in chest x-rays
bull Trained on ChestX-ray14 a public data set
released by the NIH2 containing 112120
frontal-view chest x-ray images labelled with
the 14 possible pathologies
bull Outperforms previous models from the same
data set for all 14 conditions and diagnoses
pneumonia at an accuracy exceeding the
performance of four control radiologists
bull Produces heatmaps that visualize the areas
of an image most indicative of disease
Development of CheXNet
Sept 26 2017
ChestX-ray14 data set
released along with a
preliminary algorithm
that could detect the
labelled conditionsasymp One week later
CheXNet could
diagnose 10 of the
14 pathologies more
accurately than all
previous algorithmsasymp One month later
CheXNet surpassed
best published results
for all 14 pathologies
and outperformed
Stanford radiologists in
detecting pneumonialdquoI treat the model as a lazy resident Theyrsquore not dumb theyrsquoll
just use every trick they can find to avoid doing hard workrdquo
Dr Matthew Lungren RadiologistStanford University Medical Center
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
19
Lessons from Stanfordrsquos CheXNet
AI Is Effective on Narrow Tasks
Recent advances in AI have focused on
solving narrowly defined tasks Each
sub-problem (eg pneumonia) is a model
Handle ldquoNormalrdquo Explicitly
Identifying normal conditions versus
anomalies can help recognize when
human review may be needed
Explainable1 AI (XAI)
Models that can be interpreted at least
partially are easier to trust deploy and
govern Explainability is an area of rapid
progress in AI research
1) Also known as interpretable AI
Training Requires a Teacher or Referee
Machine learning needs to be told what
answers are correct or what actions produce
positive outcomes Well-curated data are
critical to the learning process
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
20
The AI Value Proposition
AI Excels at Decisions That Are Dull Detailed and Dear (Expensive)
Three Key Areas of Benefit
Dear
Provide speed capacity 24x7
availability and consistency
at a fraction of the cost
Although these are well
known benefits of automation
when you apply this to
cognitive and complex tasks
itrsquos a new ball game
Detailed
Evaluate a vastly broader
and deeper set of data to
improve decision quality
The complexity of health
care decision making is
growing due to new data
from genetics IoT devices
wearables better
interoperability and the
growing number of drugs
and treatments
Unlimited capacity
instant response
disruptive economics
Perform at super-human
levels exceeding experts
Serves as an assistant
(watches your back)
Dull
Free human decision makers
to focus on more challenging
ldquotop of licenserdquo activities
Reduce effort spent on
repetitive tasks process
large amounts of complex
information provide
continuous monitoring and
notifications and ensure
ldquostuffrdquo doesnrsquot ldquofall through
the cracksrdquo
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
21
Where to Start Where Wersquore Going
1) Narrow (Task Weak) AI focuses on narrowly defined problems such as predicting a patientrsquos risk of developing sepsis
2) General (Strong) AI attempts to match the flexibility and versatility of human intelligence
Narr
ow
(T
ask W
ea
k)
AI1
Clinical Decisions
Administrative Decisions
Ge
ne
ral (S
tro
ng
) A
I2
Full Diagnosis
and Treatment Plan
Clinical
Chatbots
Radiology
Interpretation
Acute
Clinical
Risk
Recruiting
Safety Risk Smart
Monitoring
Medication Dosing
Administrative
Chatbots
(Concierge)
Precision
Engagement
Precision
Medicine
Ambient
Documentation
Adaptive User
Interfaces
Worklist
Prioritization
Inevitable with Time
Todayrsquos Opportunity
(growing)
The Computer Will
See You Now
The Uber-ization of
Health Care
An AI Application Landscape
Capacity Staffing
Management
Population
Health Risk
The Self-Driving
HospitalInformation
Security
Targeted
Marketing
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
22
The Self-Driving Hospital
Combine Predictive Models with Front-Line Activation
Sources ldquoVirtual Air Traffic Control for Emergency Departmentsrdquo Qventus
httpsqventuscomcontentuploads201703QVE-ED-Solution-Briefpdf Health Care
IT Advisor research and analysis
1) EVS = Environmental services
2) LWBS = Left without being seen
3) LOS = Length of stay
Case in Brief
Mercy Hospital Fort Smith
bull 336-bed acute care hospital in Fort
Smith AR
bull Partnered with Qventus to improve
emergency department (ED) patient flow
and patient satisfaction
bull Analytics applies ML algorithms to
anticipate potential capacity shortfalls
process bottlenecks
bull Application sends real-time notifications
(nudges) to care teams via their mobile
phones to trigger specific interventions
bull Emergency department results
ndash LWBS2 rates dropped 30
ndash Door-to-doc time reduced by 20
ndash 24-minute reduction in average LOS3
ndash Annual case capacity increased by 6
See that a radiology order is delayed for a
patient within a few days of discharge and
notify radiology to prioritize the reading
Activate more EVS1 staff when high churn
of beds is anticipated
Send an early warning for difficult patient
placements at discharge
Prioritize inpatient transports depending
on expected bed needs
Sample ldquoDecision Recipesrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
23
Applicants Like You Liked This Job
Predictive Models Reduce Turnover Improve Engagement
Case in Brief MultiCare Health System
bull Non-profit system based in Tacoma WA with 16000 employees
bull Partnered with Arena a cloud-based HR solution to predict the
likelihood that a candidate will be retained in a role
bull Arenarsquos machine learning algorithms evaluate applicant
submissions digital interactions and externally sourced data on
the organization (eg employer review sites)
bull Achieved a 40 reduction in RN turnover at 180 days and a
28 reduction in overall turnover at 180 days
MultiCarersquos Hiring Process
Reduced recruiting
expenses
More predictable
staffing levels
More experienced
staff
Realized Benefits
Algorithms predict the likelihood the
applicant will be retained and
engaged in the target role and send
that information to the recruiter
Applicant completes
a 15-20 minute
assessment process
on Arenarsquos platform
Recruiters determine
whether or not to send
the candidate along to
hiring managers
Hiring managers use
Arenarsquos recommendation
as an optional factor in
the hiring decision
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
24
Accelerating Research
1) Yu KH et al ldquoPredicting non-small cell lung cancer prognosis by fully automated microscopic pathology image featuresrdquo Nature No 12474 (2016)
2) Rajwa B et al ldquoAutomated assessment of disease progression in acute myeloid leukemia by probabilistic analysis of flow cytometry datardquo IEEE Transactions on Biomedical Engineering 64 No 5 (2017) 1089-1098
3) Liu Y et al Detecting cancer metastases on gigapixel pathology images arXiv preprint arXiv1703 No 02442 (2017)
4) Rajpurkar P et al Cardiologist-level arrhythmia detection with convolutional neural networks arXiv preprint arXiv1707 No 01836 (2017)
5) Nielsen S et al A predictive model to identify Parkinson disease from administrative claims data Neurology 89 No 14 (2017) 1448-1456
6) Rajpurkar P et al CheXNet Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning arXiv preprint arXiv 1711 No 05225 (2017)
7) Poplin R et al Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning Nature Biomedical Engineering No 2 (2018)
8) ACC Machine learning techniques show promise for supporting medical decisions By quickly assessing risk new algorithms may help triage patients and inform clinical decisions ScienceDaily 2018
9) Oh J et al ldquoA Generalizable Data-Driven Approach to Predict Daily Risk of Clostridium difficile Infection at Two Large Academic Health Centersrdquo ICHE 39 No 4 (2018) 425-433
Nature1 ndash August 2016
ML algorithm can accurately
differentiate two types of lung
cancers and predict patient
survival time
IEEE2 ndash February 2017
ML model predicts relapse rates for
acute myelogenous leukemia with
90 accuracy
arXiv3 ndash March 2017
ML approach to imaging analytics
can identify metastasized breast
cancer with accuracy rates rivaling
those of pathologists
Neurology5 ndash Sept 2017
Predictive model to identify
Parkinsonrsquos disease from claims
data achieved 73-83 accuracy in
study results
arXiv4 ndash July 2017
ML algorithm can diagnose
heart arrhythmias using data
from wearables sensors with
cardiologist-level accuracy
arXiv6 ndash November 2017
Deep-learning algorithm
outperforms Stanford radiologists
at diagnosing pneumonia from
x-rays
American College of Cardiology8 ndash
Feb 2018
Model predicts heart attack
diagnosis among patients in the ED
with chest pain with ~90 accuracy
Nature7 ndash February 2018
ML algorithm can predict a
personrsquos cardiovascular risk
factors using eye scans
Infection Control Hospital
Epidemiology9 ndash April 2018
ML model predicts Clostridium
difficile infection (CDI) earlier
than current diagnostic methods
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
25
Driving the Decision Machine
1Acquire Data
bull Ensure all models have
a sponsor and regular
evaluation against goals
bull All models require
tune-ups as the
environment changes
Monitor Performance
and Revisit
42Train or Refine
the Model
bull The more the better
buthellip
bull High-quality well-
governed data pay
enormous dividends
bull Diverse cross-continuum
sources improve
predictions and
robustness of models
bull Decide what features
are important (eg
explainability)
bull Apply practical and
ethical constraints
bull Evaluate under
realistic conditions
(silent mode real-time
incomplete data)
bull Embed insights directly
into applications at key
decision points
bull Reengineer processes
as warranted
bull Consider the ldquo5 rightsrdquo
of decision support
3Incorporate Models
into Workflow
Hint The Analytics Is the Easy Part
Begin with
bull Explicit leadership agreement on outcome goals expected benefit mechanisms
bull Participation from empowered representatives of impacted process
bull External experiencemdashlearn from others
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
26
New Data Sources
Mainstream Systems Cover Only a Fraction of Predictive Power
Acquire Data
Sources Nash DB ldquoPopulation Health Why It Mattersrdquo Essentials in Population Health Childrenrsquos
Hospital Association Thomas Jefferson University October 2016 Health Care IT Advisor research and
analysis
1) RPM = Remote patient monitoring
EMR(s)
Lab Systems
Pharmacy Systems
Imaging
Medical Claims
Prescription Drug Claims
Billing and Supply Chain
Scheduling Systems
Medical
Care Data
10
Sensors RPM1
mHealth Apps
Patient-Reported
Outcomes
Genetics
Socioeconomic Data
Patient Questionnaires
Human
Biology
Data
20
Lifestyle
and
Behavior
Data
50
Social and
Environmental
Data
20
Data Sources by Influence on Health Outcomes
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
27
Big Data a Technology and a Philosophy
ldquoTherersquos Gold in Them Thar Hillsrdquo
Acquire Data
Sources Cloudera ldquoInova Translational Medicine Institute Partners with Cloudera to Advance Genome-Based Machine
Learning Initiatives and Save Livesrdquo PR Newswire June 2017 Health Care IT Advisor research and analysis
1) Originally coined by Doug Laney of the Meta Group
Assumes Value in the Data
Based on a premise that there are
valuable insights that can be distilled
from the mountains of data captured
by machines
Characterised by the 3 Vs1
bull Volume
bull Velocity
bull Variety
Places greater emphasis on
unstructured data
Free text images streams of sensor
data are all fair targets for analysis
Big data is the difference
between heart rates and
heart beats
Case in Brief Inova Health System
bull Inovarsquos Translational Medicine Institute
(ITMI) is assembling one of the worldrsquos
largest whole genome sequence
databases connected to patient
information in a health care system
bull Researchers use the Cloudera
machine learning platform to analyze
terabytes of clinical and genomic data
and identify genetic links to diseases
bull Insights are used to develop
personalized treatment plans for
patients in collaboration with the
treating physician
Resource Big Data in Health Care
Strengths challenges and pragmatic
guidelines for those considering the addition
of big data capabilities to their IT portfolio
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
30
How Machines Learn
Similar Methods Differing Goals
Build the Model
Source Advisory Board research and analysis
Input Data
bull The more the better
bull Labeled with key indicators
Adjust the Model
bull Automated trial and error
bull Cross-breeding successful
candidates
Repeat
Supervised Learning
Evaluate models against the ldquocorrectrdquo answers
provided with the data The learning process
focuses on minimizing errors
Unsupervised Learning
Evaluate models against an abstract algorithmic
goal Example Divide this population of diabetics
into a handful of subpopulations that have similar
future risk profiles
Reinforcement Learning
Evaluate models by giving a reward for
desirable outcomes and a penalty for negative
outcomes Models are trained to maximize their
total rewards over timeReady for Review
Evaluate Performance
bull Against what goal
bull Forces explicit
tradeoffs (eg cost
vs quality)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
32
Black Boxes Shades of Grey
Differing Views on the Importance of Explainability
Incorporate into Workflow
Mike Wall PharmD MBA
Chief Analytics Officer
The University of Chicago Medicine
The opaqueness of black-box models is
really scary When we canrsquot tell a
clinician why a patient wonrsquot do well on a
certain medication that can be a tough
thing to blindly trust from a clinical
perspective We want to reach a place
where we augment our practice
through AI rather than dictatingrdquo
Nigam Shah MBBS PhD
Associate Professor of Medicine
(Biomedical Informatics)
Stanford University
I am pushing to evaluate these models
via a similar set of mechanisms as RCT1
or AB testing We should evaluate
against the current status-quo and do a
beforeafter studyhellipWe have to convince
ourselves of the utility of these models
We should focus on utility before
explainabilityrdquo
Surgery is the
best option ()
Data
Black Box Model
1) RCT = Randomized controlled trial
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
33
Workflow Is Key to Adoption
Incorporate into Workflow
The Right Information
To the Right Person
In the Right Form
By the Right Channel
At the Right Time
Case in Brief Mayo Breast Oncology
Clinic Trials Recruitment
bull Up to 100 concurrent drug therapy clinical trials are
available to Mayo Clinic breast cancer patients making
it challenging to consistently identify candidates
bull New clinical trial matching technology developed in
collaboration with IBM Watson Health evaluates charts
to identify eligible patients
bull Launched in January 2016 as a point-of-care solution
for oncology providers initial adoption was low due to
time pressures and fell to zero by February
bull Rework by health systems engineers resulted in a July
2016 relaunch screening moved to occur in
advance of patient visits so clinicians know of trial
matches before opening a patientrsquos chart
bull New process enrolls significantly more patients in
clinical trials than the prior manual process success led
to additional coordinator funding
The Five Rights of
Clinical Decision Support
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
34
Health Carersquos Field Agents
Medical Devices Become Intelligent Partners
Incorporate into Workflow
1) EKG = Electrocardiogram
ME
DT
RO
NIC
MIN
IME
D 6
70G
Case in Brief Artificial Pancreas and Smart Infusion Pumpsmdash
Medtronic MiniMed Connect
bull SMARTGUARD mimics some functions of a healthy pancreas
measures and predicts glucose level drift and adjusts dosing
bull Insulin pump and continuous glucose monitoring can send data directly
to smartphone
bull Trials showed a 44 reduction in hypoglycemia and reduction in
average A1C values from 74 to 69
Case in Brief AliveCor KardiaMobile
bull Portable consumer-grade single-lead EKG1 retails for $100
bull Intended for self-capture of intermittent cardiac events such as atrial
fibrillation (AFIB)
bull Smartphone-based machine learning validates quality of trace
assesses possibility of AFIB classifies normal abnormal sinus rhythm
bull EKGs can be sent to a board-certified cardiologist for interpretation
for $19
Sources alivecorcom Medtroniccom Health Care IT Advisor research and
analysis
AliveC
orK
ard
iaM
obile
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
10
Technology Drives the Flood of Data
Smartphones and Genomes and Sensors Oh My
Source Health Care IT Advisor research and analysis
1) IoT = Internet of things
2) RTLS = Real-time locating system
3) ROI = Return on investment
Common Barriers to
Adoption of New Sources
bull Privacy security concerns
bull Immature technology
bull Lack of budget
bull Unclear benefit ROI3
bull Staff already overwhelmed
with data
A Sampling of New Data Sources
Integrated Partners
(eg social services
cross-continuum care)
Data from Outside
Organizations
(eg social)
Environmental Data
(eg IoT1 RTLS2
temperature air
quality)
Genomes Proteomes
Microbiomes
Metabolomes
Clinical Surveillance
(eg public health
outbreak reports)
Remote Patient Monitoring
(eg bedside monitors
remote patient monitoring
consumer wearables)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
11
Cognitive Overload and Clinician Burnout
Sources Millett D Exclusive ldquoThree GPs a day seeking help for burnout
official data showrdquo GP Jan 2018 Health Care IT Advisor research and
analysis
1) Obermeyer Z Lee T ldquoLost in Thought ndash The Limits of the Human Mind and
the Future of Medicinerdquo N Engl J Med 377 no 13 (2017)1209-1211
Tim
e R
eq
uir
ed
ldquoMedical thinking has become vastly more complex mirroring changes in
our patients our health care system and medical science The complexity
of medicine now exceeds the capacity of the human mind1rdquo
Ziad Obermeyer MD and Thomas Lee MD
Number of Considerations for Decision Making
Significant
opportunity for
AI and analytics
Augmented Intelligence Reduces Cognitive Burden
Reliance on heuristics rules of thumb ldquogood-enoughrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
12
Data Power the Decision Engine
Source Health Care IT Advisor research and analysis
Four Levels of Analytic Capabilities
Data
Inform
Recommend
Decision
Automation
Decision
Support
Descriptive
What happened
How many of our
patients were
readmitted last
month
What is likely to happen
Is this patient likely to
readmit What do we
expect seasonally
Prescriptive
Recommend or
decide the best action
What should we do
to address this
patientrsquos risks
PredictiveDiagnostic
Why did it happen
Rates spiked was there
a change in case mix
Demographics
Decide
Hum
an
Decis
ion
Analy
tics
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP13
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
14
AI Are We There Yet
And if Not When
Source Health Care IT Advisor research and analysis
We May Be at the Start of an Explosive
Growth in AI Capabilities
Incremental Growth
AI continues to make
small advances
delivering value in
niche applicationsAI Winters AI has
already gone through
past phases of hype and
troughs of disillusionment
Exponential Growth
AI rapidly gains
capability and becomes
a mainstream part of
most digital systems
Unpredictable Timing Some
advances never seem to arrive
(conversational systems)
while others take off
unexpectedly (smartphones)
60s 70s 80s 90s 2000 2010 2020 203050s 2040 2050
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
15
Fuel for the AI Fire
Source Health Care IT Advisor research and analysis
1) EMR = Electronic medical record
2) GPU = Graphics processing unit
3) TPU = Tensor processing unit
AlgorithmsData
bull Internet Everywhere
bull Broad EMR1 Adoption
bull Wearables
bull Real-Time Location
bull IoT Sensors
bull Genomics Proteomics
Microbiome
bull Automated Feature
Extraction
bull Cluster Compute
Platforms (eg
Spark Hadoop)
bull Deep Learning
bull Off-the-Shelf
Libraries for
Common Tasksbull Faster Processors
bull Specialised Processors (GPUs2 TPUs3)
bull Inexpensive Storage
bull Elastic Cloud Computing
Hardware
Combinatorial Advances Delivering
Radically Better Models
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
16
The AI Winter Is Thawing
Source Health Care IT Advisor research and analysis
1) NASA = National Aeronautics and Space Administration
Microsoft and Alibaba outscore humans in a Stanford reading comprehension test 2018
Google DeepMindrsquos AlphaZero convincingly leads at Chess Shogi Go 2017
Skype launches real-time voice translation 2014
Mass market voice assistants (Apple Siri Google Now Microsoft Cortana) 2012
IBM Watson conclusively beats Jeopardy game show champions 2011
NASArsquos1 Mars Rovers operate semi-autonomously 2004
IBMrsquos Deep Blue beats world chess champion Gary Kasparov 1997
1950 Alan Turing proposes the ldquoTuring Testrdquo for conversational systems
1956 The term ldquoArtificial Intelligencerdquo is coined
1970s First AI Winter
1981 Digital Equipment Corporation order
expert saves company $40M per year
Timeline of AI Successes
From Toys to Tools
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
17
DeepMindrsquos AlphaZero
Novice to Superhuman in Just Four Hours
DeepMindrsquos AlphaZero Uses lsquoZero Human Knowledgersquo
Sources Vincent J ldquoDeepMindrsquos AI became a superhuman chess player in a few hours just for funrdquo The
Verge December 6 2017 Gershgorn D ldquoDeepMind wants to find the next miracle materialmdashexperts just
donrsquot know theyrsquoll pull it offrdquo Quartz October 25 2017 Health Care IT Advisor research and analysis
By not using this human data by not using
human features or human expertise in any
fashion wersquove actually rem
oved the constraints of human knowledge Itrsquos
able to therefore create knowledge for itselfrdquo
bull AI product of Google
subsidiary DeepMind
bull Provided no prior
knowledge of the
game given only basic
game rules and an
objective (win)
bull Learns through self-play
at an accelerated pace
(lsquoreinforced learningrsquo)
searches 80 thousand
positions per second
bull Variants of AlphaZero also
defeated world champion
Go and Shogi systems
AlphaZero
outperformed chess
world-champion
Stockfish1 in just
4 hours (300k steps)
By not using this human data by not using human
features or human expertise in any fashion wersquove
actually removed the constraints of human knowledge
Itrsquos able to therefore create knowledge for itselfrdquo
David Silver Lead Programmer
DEEPMIND
1) Stockfish is the 2016 Top Chess Engine Championship (TCEC) world-champion
AlphaZero Chess
Performance
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
18
Need a Radiologist
Therersquos an App for That
Sources Kubota T ldquoStanford algorithm can diagnose pneumonia better than
radiologistsrdquo Stanford News Nov 2017 Health Care IT Advisor research and analysis
1) Stanford University Medical Center
2) NIH = National Institutes of Health
Case in Brief Stanford University1
CheXNet Algorithm
bull Algorithm developed by researchers at
Stanford University in the US can diagnose
14 common pathologies in chest x-rays
bull Trained on ChestX-ray14 a public data set
released by the NIH2 containing 112120
frontal-view chest x-ray images labelled with
the 14 possible pathologies
bull Outperforms previous models from the same
data set for all 14 conditions and diagnoses
pneumonia at an accuracy exceeding the
performance of four control radiologists
bull Produces heatmaps that visualize the areas
of an image most indicative of disease
Development of CheXNet
Sept 26 2017
ChestX-ray14 data set
released along with a
preliminary algorithm
that could detect the
labelled conditionsasymp One week later
CheXNet could
diagnose 10 of the
14 pathologies more
accurately than all
previous algorithmsasymp One month later
CheXNet surpassed
best published results
for all 14 pathologies
and outperformed
Stanford radiologists in
detecting pneumonialdquoI treat the model as a lazy resident Theyrsquore not dumb theyrsquoll
just use every trick they can find to avoid doing hard workrdquo
Dr Matthew Lungren RadiologistStanford University Medical Center
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
19
Lessons from Stanfordrsquos CheXNet
AI Is Effective on Narrow Tasks
Recent advances in AI have focused on
solving narrowly defined tasks Each
sub-problem (eg pneumonia) is a model
Handle ldquoNormalrdquo Explicitly
Identifying normal conditions versus
anomalies can help recognize when
human review may be needed
Explainable1 AI (XAI)
Models that can be interpreted at least
partially are easier to trust deploy and
govern Explainability is an area of rapid
progress in AI research
1) Also known as interpretable AI
Training Requires a Teacher or Referee
Machine learning needs to be told what
answers are correct or what actions produce
positive outcomes Well-curated data are
critical to the learning process
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
20
The AI Value Proposition
AI Excels at Decisions That Are Dull Detailed and Dear (Expensive)
Three Key Areas of Benefit
Dear
Provide speed capacity 24x7
availability and consistency
at a fraction of the cost
Although these are well
known benefits of automation
when you apply this to
cognitive and complex tasks
itrsquos a new ball game
Detailed
Evaluate a vastly broader
and deeper set of data to
improve decision quality
The complexity of health
care decision making is
growing due to new data
from genetics IoT devices
wearables better
interoperability and the
growing number of drugs
and treatments
Unlimited capacity
instant response
disruptive economics
Perform at super-human
levels exceeding experts
Serves as an assistant
(watches your back)
Dull
Free human decision makers
to focus on more challenging
ldquotop of licenserdquo activities
Reduce effort spent on
repetitive tasks process
large amounts of complex
information provide
continuous monitoring and
notifications and ensure
ldquostuffrdquo doesnrsquot ldquofall through
the cracksrdquo
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
21
Where to Start Where Wersquore Going
1) Narrow (Task Weak) AI focuses on narrowly defined problems such as predicting a patientrsquos risk of developing sepsis
2) General (Strong) AI attempts to match the flexibility and versatility of human intelligence
Narr
ow
(T
ask W
ea
k)
AI1
Clinical Decisions
Administrative Decisions
Ge
ne
ral (S
tro
ng
) A
I2
Full Diagnosis
and Treatment Plan
Clinical
Chatbots
Radiology
Interpretation
Acute
Clinical
Risk
Recruiting
Safety Risk Smart
Monitoring
Medication Dosing
Administrative
Chatbots
(Concierge)
Precision
Engagement
Precision
Medicine
Ambient
Documentation
Adaptive User
Interfaces
Worklist
Prioritization
Inevitable with Time
Todayrsquos Opportunity
(growing)
The Computer Will
See You Now
The Uber-ization of
Health Care
An AI Application Landscape
Capacity Staffing
Management
Population
Health Risk
The Self-Driving
HospitalInformation
Security
Targeted
Marketing
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
22
The Self-Driving Hospital
Combine Predictive Models with Front-Line Activation
Sources ldquoVirtual Air Traffic Control for Emergency Departmentsrdquo Qventus
httpsqventuscomcontentuploads201703QVE-ED-Solution-Briefpdf Health Care
IT Advisor research and analysis
1) EVS = Environmental services
2) LWBS = Left without being seen
3) LOS = Length of stay
Case in Brief
Mercy Hospital Fort Smith
bull 336-bed acute care hospital in Fort
Smith AR
bull Partnered with Qventus to improve
emergency department (ED) patient flow
and patient satisfaction
bull Analytics applies ML algorithms to
anticipate potential capacity shortfalls
process bottlenecks
bull Application sends real-time notifications
(nudges) to care teams via their mobile
phones to trigger specific interventions
bull Emergency department results
ndash LWBS2 rates dropped 30
ndash Door-to-doc time reduced by 20
ndash 24-minute reduction in average LOS3
ndash Annual case capacity increased by 6
See that a radiology order is delayed for a
patient within a few days of discharge and
notify radiology to prioritize the reading
Activate more EVS1 staff when high churn
of beds is anticipated
Send an early warning for difficult patient
placements at discharge
Prioritize inpatient transports depending
on expected bed needs
Sample ldquoDecision Recipesrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
23
Applicants Like You Liked This Job
Predictive Models Reduce Turnover Improve Engagement
Case in Brief MultiCare Health System
bull Non-profit system based in Tacoma WA with 16000 employees
bull Partnered with Arena a cloud-based HR solution to predict the
likelihood that a candidate will be retained in a role
bull Arenarsquos machine learning algorithms evaluate applicant
submissions digital interactions and externally sourced data on
the organization (eg employer review sites)
bull Achieved a 40 reduction in RN turnover at 180 days and a
28 reduction in overall turnover at 180 days
MultiCarersquos Hiring Process
Reduced recruiting
expenses
More predictable
staffing levels
More experienced
staff
Realized Benefits
Algorithms predict the likelihood the
applicant will be retained and
engaged in the target role and send
that information to the recruiter
Applicant completes
a 15-20 minute
assessment process
on Arenarsquos platform
Recruiters determine
whether or not to send
the candidate along to
hiring managers
Hiring managers use
Arenarsquos recommendation
as an optional factor in
the hiring decision
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
24
Accelerating Research
1) Yu KH et al ldquoPredicting non-small cell lung cancer prognosis by fully automated microscopic pathology image featuresrdquo Nature No 12474 (2016)
2) Rajwa B et al ldquoAutomated assessment of disease progression in acute myeloid leukemia by probabilistic analysis of flow cytometry datardquo IEEE Transactions on Biomedical Engineering 64 No 5 (2017) 1089-1098
3) Liu Y et al Detecting cancer metastases on gigapixel pathology images arXiv preprint arXiv1703 No 02442 (2017)
4) Rajpurkar P et al Cardiologist-level arrhythmia detection with convolutional neural networks arXiv preprint arXiv1707 No 01836 (2017)
5) Nielsen S et al A predictive model to identify Parkinson disease from administrative claims data Neurology 89 No 14 (2017) 1448-1456
6) Rajpurkar P et al CheXNet Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning arXiv preprint arXiv 1711 No 05225 (2017)
7) Poplin R et al Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning Nature Biomedical Engineering No 2 (2018)
8) ACC Machine learning techniques show promise for supporting medical decisions By quickly assessing risk new algorithms may help triage patients and inform clinical decisions ScienceDaily 2018
9) Oh J et al ldquoA Generalizable Data-Driven Approach to Predict Daily Risk of Clostridium difficile Infection at Two Large Academic Health Centersrdquo ICHE 39 No 4 (2018) 425-433
Nature1 ndash August 2016
ML algorithm can accurately
differentiate two types of lung
cancers and predict patient
survival time
IEEE2 ndash February 2017
ML model predicts relapse rates for
acute myelogenous leukemia with
90 accuracy
arXiv3 ndash March 2017
ML approach to imaging analytics
can identify metastasized breast
cancer with accuracy rates rivaling
those of pathologists
Neurology5 ndash Sept 2017
Predictive model to identify
Parkinsonrsquos disease from claims
data achieved 73-83 accuracy in
study results
arXiv4 ndash July 2017
ML algorithm can diagnose
heart arrhythmias using data
from wearables sensors with
cardiologist-level accuracy
arXiv6 ndash November 2017
Deep-learning algorithm
outperforms Stanford radiologists
at diagnosing pneumonia from
x-rays
American College of Cardiology8 ndash
Feb 2018
Model predicts heart attack
diagnosis among patients in the ED
with chest pain with ~90 accuracy
Nature7 ndash February 2018
ML algorithm can predict a
personrsquos cardiovascular risk
factors using eye scans
Infection Control Hospital
Epidemiology9 ndash April 2018
ML model predicts Clostridium
difficile infection (CDI) earlier
than current diagnostic methods
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
25
Driving the Decision Machine
1Acquire Data
bull Ensure all models have
a sponsor and regular
evaluation against goals
bull All models require
tune-ups as the
environment changes
Monitor Performance
and Revisit
42Train or Refine
the Model
bull The more the better
buthellip
bull High-quality well-
governed data pay
enormous dividends
bull Diverse cross-continuum
sources improve
predictions and
robustness of models
bull Decide what features
are important (eg
explainability)
bull Apply practical and
ethical constraints
bull Evaluate under
realistic conditions
(silent mode real-time
incomplete data)
bull Embed insights directly
into applications at key
decision points
bull Reengineer processes
as warranted
bull Consider the ldquo5 rightsrdquo
of decision support
3Incorporate Models
into Workflow
Hint The Analytics Is the Easy Part
Begin with
bull Explicit leadership agreement on outcome goals expected benefit mechanisms
bull Participation from empowered representatives of impacted process
bull External experiencemdashlearn from others
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
26
New Data Sources
Mainstream Systems Cover Only a Fraction of Predictive Power
Acquire Data
Sources Nash DB ldquoPopulation Health Why It Mattersrdquo Essentials in Population Health Childrenrsquos
Hospital Association Thomas Jefferson University October 2016 Health Care IT Advisor research and
analysis
1) RPM = Remote patient monitoring
EMR(s)
Lab Systems
Pharmacy Systems
Imaging
Medical Claims
Prescription Drug Claims
Billing and Supply Chain
Scheduling Systems
Medical
Care Data
10
Sensors RPM1
mHealth Apps
Patient-Reported
Outcomes
Genetics
Socioeconomic Data
Patient Questionnaires
Human
Biology
Data
20
Lifestyle
and
Behavior
Data
50
Social and
Environmental
Data
20
Data Sources by Influence on Health Outcomes
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
27
Big Data a Technology and a Philosophy
ldquoTherersquos Gold in Them Thar Hillsrdquo
Acquire Data
Sources Cloudera ldquoInova Translational Medicine Institute Partners with Cloudera to Advance Genome-Based Machine
Learning Initiatives and Save Livesrdquo PR Newswire June 2017 Health Care IT Advisor research and analysis
1) Originally coined by Doug Laney of the Meta Group
Assumes Value in the Data
Based on a premise that there are
valuable insights that can be distilled
from the mountains of data captured
by machines
Characterised by the 3 Vs1
bull Volume
bull Velocity
bull Variety
Places greater emphasis on
unstructured data
Free text images streams of sensor
data are all fair targets for analysis
Big data is the difference
between heart rates and
heart beats
Case in Brief Inova Health System
bull Inovarsquos Translational Medicine Institute
(ITMI) is assembling one of the worldrsquos
largest whole genome sequence
databases connected to patient
information in a health care system
bull Researchers use the Cloudera
machine learning platform to analyze
terabytes of clinical and genomic data
and identify genetic links to diseases
bull Insights are used to develop
personalized treatment plans for
patients in collaboration with the
treating physician
Resource Big Data in Health Care
Strengths challenges and pragmatic
guidelines for those considering the addition
of big data capabilities to their IT portfolio
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
30
How Machines Learn
Similar Methods Differing Goals
Build the Model
Source Advisory Board research and analysis
Input Data
bull The more the better
bull Labeled with key indicators
Adjust the Model
bull Automated trial and error
bull Cross-breeding successful
candidates
Repeat
Supervised Learning
Evaluate models against the ldquocorrectrdquo answers
provided with the data The learning process
focuses on minimizing errors
Unsupervised Learning
Evaluate models against an abstract algorithmic
goal Example Divide this population of diabetics
into a handful of subpopulations that have similar
future risk profiles
Reinforcement Learning
Evaluate models by giving a reward for
desirable outcomes and a penalty for negative
outcomes Models are trained to maximize their
total rewards over timeReady for Review
Evaluate Performance
bull Against what goal
bull Forces explicit
tradeoffs (eg cost
vs quality)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
32
Black Boxes Shades of Grey
Differing Views on the Importance of Explainability
Incorporate into Workflow
Mike Wall PharmD MBA
Chief Analytics Officer
The University of Chicago Medicine
The opaqueness of black-box models is
really scary When we canrsquot tell a
clinician why a patient wonrsquot do well on a
certain medication that can be a tough
thing to blindly trust from a clinical
perspective We want to reach a place
where we augment our practice
through AI rather than dictatingrdquo
Nigam Shah MBBS PhD
Associate Professor of Medicine
(Biomedical Informatics)
Stanford University
I am pushing to evaluate these models
via a similar set of mechanisms as RCT1
or AB testing We should evaluate
against the current status-quo and do a
beforeafter studyhellipWe have to convince
ourselves of the utility of these models
We should focus on utility before
explainabilityrdquo
Surgery is the
best option ()
Data
Black Box Model
1) RCT = Randomized controlled trial
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
33
Workflow Is Key to Adoption
Incorporate into Workflow
The Right Information
To the Right Person
In the Right Form
By the Right Channel
At the Right Time
Case in Brief Mayo Breast Oncology
Clinic Trials Recruitment
bull Up to 100 concurrent drug therapy clinical trials are
available to Mayo Clinic breast cancer patients making
it challenging to consistently identify candidates
bull New clinical trial matching technology developed in
collaboration with IBM Watson Health evaluates charts
to identify eligible patients
bull Launched in January 2016 as a point-of-care solution
for oncology providers initial adoption was low due to
time pressures and fell to zero by February
bull Rework by health systems engineers resulted in a July
2016 relaunch screening moved to occur in
advance of patient visits so clinicians know of trial
matches before opening a patientrsquos chart
bull New process enrolls significantly more patients in
clinical trials than the prior manual process success led
to additional coordinator funding
The Five Rights of
Clinical Decision Support
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
34
Health Carersquos Field Agents
Medical Devices Become Intelligent Partners
Incorporate into Workflow
1) EKG = Electrocardiogram
ME
DT
RO
NIC
MIN
IME
D 6
70G
Case in Brief Artificial Pancreas and Smart Infusion Pumpsmdash
Medtronic MiniMed Connect
bull SMARTGUARD mimics some functions of a healthy pancreas
measures and predicts glucose level drift and adjusts dosing
bull Insulin pump and continuous glucose monitoring can send data directly
to smartphone
bull Trials showed a 44 reduction in hypoglycemia and reduction in
average A1C values from 74 to 69
Case in Brief AliveCor KardiaMobile
bull Portable consumer-grade single-lead EKG1 retails for $100
bull Intended for self-capture of intermittent cardiac events such as atrial
fibrillation (AFIB)
bull Smartphone-based machine learning validates quality of trace
assesses possibility of AFIB classifies normal abnormal sinus rhythm
bull EKGs can be sent to a board-certified cardiologist for interpretation
for $19
Sources alivecorcom Medtroniccom Health Care IT Advisor research and
analysis
AliveC
orK
ard
iaM
obile
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
11
Cognitive Overload and Clinician Burnout
Sources Millett D Exclusive ldquoThree GPs a day seeking help for burnout
official data showrdquo GP Jan 2018 Health Care IT Advisor research and
analysis
1) Obermeyer Z Lee T ldquoLost in Thought ndash The Limits of the Human Mind and
the Future of Medicinerdquo N Engl J Med 377 no 13 (2017)1209-1211
Tim
e R
eq
uir
ed
ldquoMedical thinking has become vastly more complex mirroring changes in
our patients our health care system and medical science The complexity
of medicine now exceeds the capacity of the human mind1rdquo
Ziad Obermeyer MD and Thomas Lee MD
Number of Considerations for Decision Making
Significant
opportunity for
AI and analytics
Augmented Intelligence Reduces Cognitive Burden
Reliance on heuristics rules of thumb ldquogood-enoughrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
12
Data Power the Decision Engine
Source Health Care IT Advisor research and analysis
Four Levels of Analytic Capabilities
Data
Inform
Recommend
Decision
Automation
Decision
Support
Descriptive
What happened
How many of our
patients were
readmitted last
month
What is likely to happen
Is this patient likely to
readmit What do we
expect seasonally
Prescriptive
Recommend or
decide the best action
What should we do
to address this
patientrsquos risks
PredictiveDiagnostic
Why did it happen
Rates spiked was there
a change in case mix
Demographics
Decide
Hum
an
Decis
ion
Analy
tics
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP13
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
14
AI Are We There Yet
And if Not When
Source Health Care IT Advisor research and analysis
We May Be at the Start of an Explosive
Growth in AI Capabilities
Incremental Growth
AI continues to make
small advances
delivering value in
niche applicationsAI Winters AI has
already gone through
past phases of hype and
troughs of disillusionment
Exponential Growth
AI rapidly gains
capability and becomes
a mainstream part of
most digital systems
Unpredictable Timing Some
advances never seem to arrive
(conversational systems)
while others take off
unexpectedly (smartphones)
60s 70s 80s 90s 2000 2010 2020 203050s 2040 2050
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
15
Fuel for the AI Fire
Source Health Care IT Advisor research and analysis
1) EMR = Electronic medical record
2) GPU = Graphics processing unit
3) TPU = Tensor processing unit
AlgorithmsData
bull Internet Everywhere
bull Broad EMR1 Adoption
bull Wearables
bull Real-Time Location
bull IoT Sensors
bull Genomics Proteomics
Microbiome
bull Automated Feature
Extraction
bull Cluster Compute
Platforms (eg
Spark Hadoop)
bull Deep Learning
bull Off-the-Shelf
Libraries for
Common Tasksbull Faster Processors
bull Specialised Processors (GPUs2 TPUs3)
bull Inexpensive Storage
bull Elastic Cloud Computing
Hardware
Combinatorial Advances Delivering
Radically Better Models
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
16
The AI Winter Is Thawing
Source Health Care IT Advisor research and analysis
1) NASA = National Aeronautics and Space Administration
Microsoft and Alibaba outscore humans in a Stanford reading comprehension test 2018
Google DeepMindrsquos AlphaZero convincingly leads at Chess Shogi Go 2017
Skype launches real-time voice translation 2014
Mass market voice assistants (Apple Siri Google Now Microsoft Cortana) 2012
IBM Watson conclusively beats Jeopardy game show champions 2011
NASArsquos1 Mars Rovers operate semi-autonomously 2004
IBMrsquos Deep Blue beats world chess champion Gary Kasparov 1997
1950 Alan Turing proposes the ldquoTuring Testrdquo for conversational systems
1956 The term ldquoArtificial Intelligencerdquo is coined
1970s First AI Winter
1981 Digital Equipment Corporation order
expert saves company $40M per year
Timeline of AI Successes
From Toys to Tools
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
17
DeepMindrsquos AlphaZero
Novice to Superhuman in Just Four Hours
DeepMindrsquos AlphaZero Uses lsquoZero Human Knowledgersquo
Sources Vincent J ldquoDeepMindrsquos AI became a superhuman chess player in a few hours just for funrdquo The
Verge December 6 2017 Gershgorn D ldquoDeepMind wants to find the next miracle materialmdashexperts just
donrsquot know theyrsquoll pull it offrdquo Quartz October 25 2017 Health Care IT Advisor research and analysis
By not using this human data by not using
human features or human expertise in any
fashion wersquove actually rem
oved the constraints of human knowledge Itrsquos
able to therefore create knowledge for itselfrdquo
bull AI product of Google
subsidiary DeepMind
bull Provided no prior
knowledge of the
game given only basic
game rules and an
objective (win)
bull Learns through self-play
at an accelerated pace
(lsquoreinforced learningrsquo)
searches 80 thousand
positions per second
bull Variants of AlphaZero also
defeated world champion
Go and Shogi systems
AlphaZero
outperformed chess
world-champion
Stockfish1 in just
4 hours (300k steps)
By not using this human data by not using human
features or human expertise in any fashion wersquove
actually removed the constraints of human knowledge
Itrsquos able to therefore create knowledge for itselfrdquo
David Silver Lead Programmer
DEEPMIND
1) Stockfish is the 2016 Top Chess Engine Championship (TCEC) world-champion
AlphaZero Chess
Performance
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
18
Need a Radiologist
Therersquos an App for That
Sources Kubota T ldquoStanford algorithm can diagnose pneumonia better than
radiologistsrdquo Stanford News Nov 2017 Health Care IT Advisor research and analysis
1) Stanford University Medical Center
2) NIH = National Institutes of Health
Case in Brief Stanford University1
CheXNet Algorithm
bull Algorithm developed by researchers at
Stanford University in the US can diagnose
14 common pathologies in chest x-rays
bull Trained on ChestX-ray14 a public data set
released by the NIH2 containing 112120
frontal-view chest x-ray images labelled with
the 14 possible pathologies
bull Outperforms previous models from the same
data set for all 14 conditions and diagnoses
pneumonia at an accuracy exceeding the
performance of four control radiologists
bull Produces heatmaps that visualize the areas
of an image most indicative of disease
Development of CheXNet
Sept 26 2017
ChestX-ray14 data set
released along with a
preliminary algorithm
that could detect the
labelled conditionsasymp One week later
CheXNet could
diagnose 10 of the
14 pathologies more
accurately than all
previous algorithmsasymp One month later
CheXNet surpassed
best published results
for all 14 pathologies
and outperformed
Stanford radiologists in
detecting pneumonialdquoI treat the model as a lazy resident Theyrsquore not dumb theyrsquoll
just use every trick they can find to avoid doing hard workrdquo
Dr Matthew Lungren RadiologistStanford University Medical Center
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
19
Lessons from Stanfordrsquos CheXNet
AI Is Effective on Narrow Tasks
Recent advances in AI have focused on
solving narrowly defined tasks Each
sub-problem (eg pneumonia) is a model
Handle ldquoNormalrdquo Explicitly
Identifying normal conditions versus
anomalies can help recognize when
human review may be needed
Explainable1 AI (XAI)
Models that can be interpreted at least
partially are easier to trust deploy and
govern Explainability is an area of rapid
progress in AI research
1) Also known as interpretable AI
Training Requires a Teacher or Referee
Machine learning needs to be told what
answers are correct or what actions produce
positive outcomes Well-curated data are
critical to the learning process
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
20
The AI Value Proposition
AI Excels at Decisions That Are Dull Detailed and Dear (Expensive)
Three Key Areas of Benefit
Dear
Provide speed capacity 24x7
availability and consistency
at a fraction of the cost
Although these are well
known benefits of automation
when you apply this to
cognitive and complex tasks
itrsquos a new ball game
Detailed
Evaluate a vastly broader
and deeper set of data to
improve decision quality
The complexity of health
care decision making is
growing due to new data
from genetics IoT devices
wearables better
interoperability and the
growing number of drugs
and treatments
Unlimited capacity
instant response
disruptive economics
Perform at super-human
levels exceeding experts
Serves as an assistant
(watches your back)
Dull
Free human decision makers
to focus on more challenging
ldquotop of licenserdquo activities
Reduce effort spent on
repetitive tasks process
large amounts of complex
information provide
continuous monitoring and
notifications and ensure
ldquostuffrdquo doesnrsquot ldquofall through
the cracksrdquo
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
21
Where to Start Where Wersquore Going
1) Narrow (Task Weak) AI focuses on narrowly defined problems such as predicting a patientrsquos risk of developing sepsis
2) General (Strong) AI attempts to match the flexibility and versatility of human intelligence
Narr
ow
(T
ask W
ea
k)
AI1
Clinical Decisions
Administrative Decisions
Ge
ne
ral (S
tro
ng
) A
I2
Full Diagnosis
and Treatment Plan
Clinical
Chatbots
Radiology
Interpretation
Acute
Clinical
Risk
Recruiting
Safety Risk Smart
Monitoring
Medication Dosing
Administrative
Chatbots
(Concierge)
Precision
Engagement
Precision
Medicine
Ambient
Documentation
Adaptive User
Interfaces
Worklist
Prioritization
Inevitable with Time
Todayrsquos Opportunity
(growing)
The Computer Will
See You Now
The Uber-ization of
Health Care
An AI Application Landscape
Capacity Staffing
Management
Population
Health Risk
The Self-Driving
HospitalInformation
Security
Targeted
Marketing
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
22
The Self-Driving Hospital
Combine Predictive Models with Front-Line Activation
Sources ldquoVirtual Air Traffic Control for Emergency Departmentsrdquo Qventus
httpsqventuscomcontentuploads201703QVE-ED-Solution-Briefpdf Health Care
IT Advisor research and analysis
1) EVS = Environmental services
2) LWBS = Left without being seen
3) LOS = Length of stay
Case in Brief
Mercy Hospital Fort Smith
bull 336-bed acute care hospital in Fort
Smith AR
bull Partnered with Qventus to improve
emergency department (ED) patient flow
and patient satisfaction
bull Analytics applies ML algorithms to
anticipate potential capacity shortfalls
process bottlenecks
bull Application sends real-time notifications
(nudges) to care teams via their mobile
phones to trigger specific interventions
bull Emergency department results
ndash LWBS2 rates dropped 30
ndash Door-to-doc time reduced by 20
ndash 24-minute reduction in average LOS3
ndash Annual case capacity increased by 6
See that a radiology order is delayed for a
patient within a few days of discharge and
notify radiology to prioritize the reading
Activate more EVS1 staff when high churn
of beds is anticipated
Send an early warning for difficult patient
placements at discharge
Prioritize inpatient transports depending
on expected bed needs
Sample ldquoDecision Recipesrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
23
Applicants Like You Liked This Job
Predictive Models Reduce Turnover Improve Engagement
Case in Brief MultiCare Health System
bull Non-profit system based in Tacoma WA with 16000 employees
bull Partnered with Arena a cloud-based HR solution to predict the
likelihood that a candidate will be retained in a role
bull Arenarsquos machine learning algorithms evaluate applicant
submissions digital interactions and externally sourced data on
the organization (eg employer review sites)
bull Achieved a 40 reduction in RN turnover at 180 days and a
28 reduction in overall turnover at 180 days
MultiCarersquos Hiring Process
Reduced recruiting
expenses
More predictable
staffing levels
More experienced
staff
Realized Benefits
Algorithms predict the likelihood the
applicant will be retained and
engaged in the target role and send
that information to the recruiter
Applicant completes
a 15-20 minute
assessment process
on Arenarsquos platform
Recruiters determine
whether or not to send
the candidate along to
hiring managers
Hiring managers use
Arenarsquos recommendation
as an optional factor in
the hiring decision
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
24
Accelerating Research
1) Yu KH et al ldquoPredicting non-small cell lung cancer prognosis by fully automated microscopic pathology image featuresrdquo Nature No 12474 (2016)
2) Rajwa B et al ldquoAutomated assessment of disease progression in acute myeloid leukemia by probabilistic analysis of flow cytometry datardquo IEEE Transactions on Biomedical Engineering 64 No 5 (2017) 1089-1098
3) Liu Y et al Detecting cancer metastases on gigapixel pathology images arXiv preprint arXiv1703 No 02442 (2017)
4) Rajpurkar P et al Cardiologist-level arrhythmia detection with convolutional neural networks arXiv preprint arXiv1707 No 01836 (2017)
5) Nielsen S et al A predictive model to identify Parkinson disease from administrative claims data Neurology 89 No 14 (2017) 1448-1456
6) Rajpurkar P et al CheXNet Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning arXiv preprint arXiv 1711 No 05225 (2017)
7) Poplin R et al Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning Nature Biomedical Engineering No 2 (2018)
8) ACC Machine learning techniques show promise for supporting medical decisions By quickly assessing risk new algorithms may help triage patients and inform clinical decisions ScienceDaily 2018
9) Oh J et al ldquoA Generalizable Data-Driven Approach to Predict Daily Risk of Clostridium difficile Infection at Two Large Academic Health Centersrdquo ICHE 39 No 4 (2018) 425-433
Nature1 ndash August 2016
ML algorithm can accurately
differentiate two types of lung
cancers and predict patient
survival time
IEEE2 ndash February 2017
ML model predicts relapse rates for
acute myelogenous leukemia with
90 accuracy
arXiv3 ndash March 2017
ML approach to imaging analytics
can identify metastasized breast
cancer with accuracy rates rivaling
those of pathologists
Neurology5 ndash Sept 2017
Predictive model to identify
Parkinsonrsquos disease from claims
data achieved 73-83 accuracy in
study results
arXiv4 ndash July 2017
ML algorithm can diagnose
heart arrhythmias using data
from wearables sensors with
cardiologist-level accuracy
arXiv6 ndash November 2017
Deep-learning algorithm
outperforms Stanford radiologists
at diagnosing pneumonia from
x-rays
American College of Cardiology8 ndash
Feb 2018
Model predicts heart attack
diagnosis among patients in the ED
with chest pain with ~90 accuracy
Nature7 ndash February 2018
ML algorithm can predict a
personrsquos cardiovascular risk
factors using eye scans
Infection Control Hospital
Epidemiology9 ndash April 2018
ML model predicts Clostridium
difficile infection (CDI) earlier
than current diagnostic methods
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
25
Driving the Decision Machine
1Acquire Data
bull Ensure all models have
a sponsor and regular
evaluation against goals
bull All models require
tune-ups as the
environment changes
Monitor Performance
and Revisit
42Train or Refine
the Model
bull The more the better
buthellip
bull High-quality well-
governed data pay
enormous dividends
bull Diverse cross-continuum
sources improve
predictions and
robustness of models
bull Decide what features
are important (eg
explainability)
bull Apply practical and
ethical constraints
bull Evaluate under
realistic conditions
(silent mode real-time
incomplete data)
bull Embed insights directly
into applications at key
decision points
bull Reengineer processes
as warranted
bull Consider the ldquo5 rightsrdquo
of decision support
3Incorporate Models
into Workflow
Hint The Analytics Is the Easy Part
Begin with
bull Explicit leadership agreement on outcome goals expected benefit mechanisms
bull Participation from empowered representatives of impacted process
bull External experiencemdashlearn from others
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
26
New Data Sources
Mainstream Systems Cover Only a Fraction of Predictive Power
Acquire Data
Sources Nash DB ldquoPopulation Health Why It Mattersrdquo Essentials in Population Health Childrenrsquos
Hospital Association Thomas Jefferson University October 2016 Health Care IT Advisor research and
analysis
1) RPM = Remote patient monitoring
EMR(s)
Lab Systems
Pharmacy Systems
Imaging
Medical Claims
Prescription Drug Claims
Billing and Supply Chain
Scheduling Systems
Medical
Care Data
10
Sensors RPM1
mHealth Apps
Patient-Reported
Outcomes
Genetics
Socioeconomic Data
Patient Questionnaires
Human
Biology
Data
20
Lifestyle
and
Behavior
Data
50
Social and
Environmental
Data
20
Data Sources by Influence on Health Outcomes
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
27
Big Data a Technology and a Philosophy
ldquoTherersquos Gold in Them Thar Hillsrdquo
Acquire Data
Sources Cloudera ldquoInova Translational Medicine Institute Partners with Cloudera to Advance Genome-Based Machine
Learning Initiatives and Save Livesrdquo PR Newswire June 2017 Health Care IT Advisor research and analysis
1) Originally coined by Doug Laney of the Meta Group
Assumes Value in the Data
Based on a premise that there are
valuable insights that can be distilled
from the mountains of data captured
by machines
Characterised by the 3 Vs1
bull Volume
bull Velocity
bull Variety
Places greater emphasis on
unstructured data
Free text images streams of sensor
data are all fair targets for analysis
Big data is the difference
between heart rates and
heart beats
Case in Brief Inova Health System
bull Inovarsquos Translational Medicine Institute
(ITMI) is assembling one of the worldrsquos
largest whole genome sequence
databases connected to patient
information in a health care system
bull Researchers use the Cloudera
machine learning platform to analyze
terabytes of clinical and genomic data
and identify genetic links to diseases
bull Insights are used to develop
personalized treatment plans for
patients in collaboration with the
treating physician
Resource Big Data in Health Care
Strengths challenges and pragmatic
guidelines for those considering the addition
of big data capabilities to their IT portfolio
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
30
How Machines Learn
Similar Methods Differing Goals
Build the Model
Source Advisory Board research and analysis
Input Data
bull The more the better
bull Labeled with key indicators
Adjust the Model
bull Automated trial and error
bull Cross-breeding successful
candidates
Repeat
Supervised Learning
Evaluate models against the ldquocorrectrdquo answers
provided with the data The learning process
focuses on minimizing errors
Unsupervised Learning
Evaluate models against an abstract algorithmic
goal Example Divide this population of diabetics
into a handful of subpopulations that have similar
future risk profiles
Reinforcement Learning
Evaluate models by giving a reward for
desirable outcomes and a penalty for negative
outcomes Models are trained to maximize their
total rewards over timeReady for Review
Evaluate Performance
bull Against what goal
bull Forces explicit
tradeoffs (eg cost
vs quality)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
32
Black Boxes Shades of Grey
Differing Views on the Importance of Explainability
Incorporate into Workflow
Mike Wall PharmD MBA
Chief Analytics Officer
The University of Chicago Medicine
The opaqueness of black-box models is
really scary When we canrsquot tell a
clinician why a patient wonrsquot do well on a
certain medication that can be a tough
thing to blindly trust from a clinical
perspective We want to reach a place
where we augment our practice
through AI rather than dictatingrdquo
Nigam Shah MBBS PhD
Associate Professor of Medicine
(Biomedical Informatics)
Stanford University
I am pushing to evaluate these models
via a similar set of mechanisms as RCT1
or AB testing We should evaluate
against the current status-quo and do a
beforeafter studyhellipWe have to convince
ourselves of the utility of these models
We should focus on utility before
explainabilityrdquo
Surgery is the
best option ()
Data
Black Box Model
1) RCT = Randomized controlled trial
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
33
Workflow Is Key to Adoption
Incorporate into Workflow
The Right Information
To the Right Person
In the Right Form
By the Right Channel
At the Right Time
Case in Brief Mayo Breast Oncology
Clinic Trials Recruitment
bull Up to 100 concurrent drug therapy clinical trials are
available to Mayo Clinic breast cancer patients making
it challenging to consistently identify candidates
bull New clinical trial matching technology developed in
collaboration with IBM Watson Health evaluates charts
to identify eligible patients
bull Launched in January 2016 as a point-of-care solution
for oncology providers initial adoption was low due to
time pressures and fell to zero by February
bull Rework by health systems engineers resulted in a July
2016 relaunch screening moved to occur in
advance of patient visits so clinicians know of trial
matches before opening a patientrsquos chart
bull New process enrolls significantly more patients in
clinical trials than the prior manual process success led
to additional coordinator funding
The Five Rights of
Clinical Decision Support
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
34
Health Carersquos Field Agents
Medical Devices Become Intelligent Partners
Incorporate into Workflow
1) EKG = Electrocardiogram
ME
DT
RO
NIC
MIN
IME
D 6
70G
Case in Brief Artificial Pancreas and Smart Infusion Pumpsmdash
Medtronic MiniMed Connect
bull SMARTGUARD mimics some functions of a healthy pancreas
measures and predicts glucose level drift and adjusts dosing
bull Insulin pump and continuous glucose monitoring can send data directly
to smartphone
bull Trials showed a 44 reduction in hypoglycemia and reduction in
average A1C values from 74 to 69
Case in Brief AliveCor KardiaMobile
bull Portable consumer-grade single-lead EKG1 retails for $100
bull Intended for self-capture of intermittent cardiac events such as atrial
fibrillation (AFIB)
bull Smartphone-based machine learning validates quality of trace
assesses possibility of AFIB classifies normal abnormal sinus rhythm
bull EKGs can be sent to a board-certified cardiologist for interpretation
for $19
Sources alivecorcom Medtroniccom Health Care IT Advisor research and
analysis
AliveC
orK
ard
iaM
obile
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
12
Data Power the Decision Engine
Source Health Care IT Advisor research and analysis
Four Levels of Analytic Capabilities
Data
Inform
Recommend
Decision
Automation
Decision
Support
Descriptive
What happened
How many of our
patients were
readmitted last
month
What is likely to happen
Is this patient likely to
readmit What do we
expect seasonally
Prescriptive
Recommend or
decide the best action
What should we do
to address this
patientrsquos risks
PredictiveDiagnostic
Why did it happen
Rates spiked was there
a change in case mix
Demographics
Decide
Hum
an
Decis
ion
Analy
tics
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP13
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
14
AI Are We There Yet
And if Not When
Source Health Care IT Advisor research and analysis
We May Be at the Start of an Explosive
Growth in AI Capabilities
Incremental Growth
AI continues to make
small advances
delivering value in
niche applicationsAI Winters AI has
already gone through
past phases of hype and
troughs of disillusionment
Exponential Growth
AI rapidly gains
capability and becomes
a mainstream part of
most digital systems
Unpredictable Timing Some
advances never seem to arrive
(conversational systems)
while others take off
unexpectedly (smartphones)
60s 70s 80s 90s 2000 2010 2020 203050s 2040 2050
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
15
Fuel for the AI Fire
Source Health Care IT Advisor research and analysis
1) EMR = Electronic medical record
2) GPU = Graphics processing unit
3) TPU = Tensor processing unit
AlgorithmsData
bull Internet Everywhere
bull Broad EMR1 Adoption
bull Wearables
bull Real-Time Location
bull IoT Sensors
bull Genomics Proteomics
Microbiome
bull Automated Feature
Extraction
bull Cluster Compute
Platforms (eg
Spark Hadoop)
bull Deep Learning
bull Off-the-Shelf
Libraries for
Common Tasksbull Faster Processors
bull Specialised Processors (GPUs2 TPUs3)
bull Inexpensive Storage
bull Elastic Cloud Computing
Hardware
Combinatorial Advances Delivering
Radically Better Models
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
16
The AI Winter Is Thawing
Source Health Care IT Advisor research and analysis
1) NASA = National Aeronautics and Space Administration
Microsoft and Alibaba outscore humans in a Stanford reading comprehension test 2018
Google DeepMindrsquos AlphaZero convincingly leads at Chess Shogi Go 2017
Skype launches real-time voice translation 2014
Mass market voice assistants (Apple Siri Google Now Microsoft Cortana) 2012
IBM Watson conclusively beats Jeopardy game show champions 2011
NASArsquos1 Mars Rovers operate semi-autonomously 2004
IBMrsquos Deep Blue beats world chess champion Gary Kasparov 1997
1950 Alan Turing proposes the ldquoTuring Testrdquo for conversational systems
1956 The term ldquoArtificial Intelligencerdquo is coined
1970s First AI Winter
1981 Digital Equipment Corporation order
expert saves company $40M per year
Timeline of AI Successes
From Toys to Tools
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
17
DeepMindrsquos AlphaZero
Novice to Superhuman in Just Four Hours
DeepMindrsquos AlphaZero Uses lsquoZero Human Knowledgersquo
Sources Vincent J ldquoDeepMindrsquos AI became a superhuman chess player in a few hours just for funrdquo The
Verge December 6 2017 Gershgorn D ldquoDeepMind wants to find the next miracle materialmdashexperts just
donrsquot know theyrsquoll pull it offrdquo Quartz October 25 2017 Health Care IT Advisor research and analysis
By not using this human data by not using
human features or human expertise in any
fashion wersquove actually rem
oved the constraints of human knowledge Itrsquos
able to therefore create knowledge for itselfrdquo
bull AI product of Google
subsidiary DeepMind
bull Provided no prior
knowledge of the
game given only basic
game rules and an
objective (win)
bull Learns through self-play
at an accelerated pace
(lsquoreinforced learningrsquo)
searches 80 thousand
positions per second
bull Variants of AlphaZero also
defeated world champion
Go and Shogi systems
AlphaZero
outperformed chess
world-champion
Stockfish1 in just
4 hours (300k steps)
By not using this human data by not using human
features or human expertise in any fashion wersquove
actually removed the constraints of human knowledge
Itrsquos able to therefore create knowledge for itselfrdquo
David Silver Lead Programmer
DEEPMIND
1) Stockfish is the 2016 Top Chess Engine Championship (TCEC) world-champion
AlphaZero Chess
Performance
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
18
Need a Radiologist
Therersquos an App for That
Sources Kubota T ldquoStanford algorithm can diagnose pneumonia better than
radiologistsrdquo Stanford News Nov 2017 Health Care IT Advisor research and analysis
1) Stanford University Medical Center
2) NIH = National Institutes of Health
Case in Brief Stanford University1
CheXNet Algorithm
bull Algorithm developed by researchers at
Stanford University in the US can diagnose
14 common pathologies in chest x-rays
bull Trained on ChestX-ray14 a public data set
released by the NIH2 containing 112120
frontal-view chest x-ray images labelled with
the 14 possible pathologies
bull Outperforms previous models from the same
data set for all 14 conditions and diagnoses
pneumonia at an accuracy exceeding the
performance of four control radiologists
bull Produces heatmaps that visualize the areas
of an image most indicative of disease
Development of CheXNet
Sept 26 2017
ChestX-ray14 data set
released along with a
preliminary algorithm
that could detect the
labelled conditionsasymp One week later
CheXNet could
diagnose 10 of the
14 pathologies more
accurately than all
previous algorithmsasymp One month later
CheXNet surpassed
best published results
for all 14 pathologies
and outperformed
Stanford radiologists in
detecting pneumonialdquoI treat the model as a lazy resident Theyrsquore not dumb theyrsquoll
just use every trick they can find to avoid doing hard workrdquo
Dr Matthew Lungren RadiologistStanford University Medical Center
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
19
Lessons from Stanfordrsquos CheXNet
AI Is Effective on Narrow Tasks
Recent advances in AI have focused on
solving narrowly defined tasks Each
sub-problem (eg pneumonia) is a model
Handle ldquoNormalrdquo Explicitly
Identifying normal conditions versus
anomalies can help recognize when
human review may be needed
Explainable1 AI (XAI)
Models that can be interpreted at least
partially are easier to trust deploy and
govern Explainability is an area of rapid
progress in AI research
1) Also known as interpretable AI
Training Requires a Teacher or Referee
Machine learning needs to be told what
answers are correct or what actions produce
positive outcomes Well-curated data are
critical to the learning process
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
20
The AI Value Proposition
AI Excels at Decisions That Are Dull Detailed and Dear (Expensive)
Three Key Areas of Benefit
Dear
Provide speed capacity 24x7
availability and consistency
at a fraction of the cost
Although these are well
known benefits of automation
when you apply this to
cognitive and complex tasks
itrsquos a new ball game
Detailed
Evaluate a vastly broader
and deeper set of data to
improve decision quality
The complexity of health
care decision making is
growing due to new data
from genetics IoT devices
wearables better
interoperability and the
growing number of drugs
and treatments
Unlimited capacity
instant response
disruptive economics
Perform at super-human
levels exceeding experts
Serves as an assistant
(watches your back)
Dull
Free human decision makers
to focus on more challenging
ldquotop of licenserdquo activities
Reduce effort spent on
repetitive tasks process
large amounts of complex
information provide
continuous monitoring and
notifications and ensure
ldquostuffrdquo doesnrsquot ldquofall through
the cracksrdquo
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
21
Where to Start Where Wersquore Going
1) Narrow (Task Weak) AI focuses on narrowly defined problems such as predicting a patientrsquos risk of developing sepsis
2) General (Strong) AI attempts to match the flexibility and versatility of human intelligence
Narr
ow
(T
ask W
ea
k)
AI1
Clinical Decisions
Administrative Decisions
Ge
ne
ral (S
tro
ng
) A
I2
Full Diagnosis
and Treatment Plan
Clinical
Chatbots
Radiology
Interpretation
Acute
Clinical
Risk
Recruiting
Safety Risk Smart
Monitoring
Medication Dosing
Administrative
Chatbots
(Concierge)
Precision
Engagement
Precision
Medicine
Ambient
Documentation
Adaptive User
Interfaces
Worklist
Prioritization
Inevitable with Time
Todayrsquos Opportunity
(growing)
The Computer Will
See You Now
The Uber-ization of
Health Care
An AI Application Landscape
Capacity Staffing
Management
Population
Health Risk
The Self-Driving
HospitalInformation
Security
Targeted
Marketing
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
22
The Self-Driving Hospital
Combine Predictive Models with Front-Line Activation
Sources ldquoVirtual Air Traffic Control for Emergency Departmentsrdquo Qventus
httpsqventuscomcontentuploads201703QVE-ED-Solution-Briefpdf Health Care
IT Advisor research and analysis
1) EVS = Environmental services
2) LWBS = Left without being seen
3) LOS = Length of stay
Case in Brief
Mercy Hospital Fort Smith
bull 336-bed acute care hospital in Fort
Smith AR
bull Partnered with Qventus to improve
emergency department (ED) patient flow
and patient satisfaction
bull Analytics applies ML algorithms to
anticipate potential capacity shortfalls
process bottlenecks
bull Application sends real-time notifications
(nudges) to care teams via their mobile
phones to trigger specific interventions
bull Emergency department results
ndash LWBS2 rates dropped 30
ndash Door-to-doc time reduced by 20
ndash 24-minute reduction in average LOS3
ndash Annual case capacity increased by 6
See that a radiology order is delayed for a
patient within a few days of discharge and
notify radiology to prioritize the reading
Activate more EVS1 staff when high churn
of beds is anticipated
Send an early warning for difficult patient
placements at discharge
Prioritize inpatient transports depending
on expected bed needs
Sample ldquoDecision Recipesrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
23
Applicants Like You Liked This Job
Predictive Models Reduce Turnover Improve Engagement
Case in Brief MultiCare Health System
bull Non-profit system based in Tacoma WA with 16000 employees
bull Partnered with Arena a cloud-based HR solution to predict the
likelihood that a candidate will be retained in a role
bull Arenarsquos machine learning algorithms evaluate applicant
submissions digital interactions and externally sourced data on
the organization (eg employer review sites)
bull Achieved a 40 reduction in RN turnover at 180 days and a
28 reduction in overall turnover at 180 days
MultiCarersquos Hiring Process
Reduced recruiting
expenses
More predictable
staffing levels
More experienced
staff
Realized Benefits
Algorithms predict the likelihood the
applicant will be retained and
engaged in the target role and send
that information to the recruiter
Applicant completes
a 15-20 minute
assessment process
on Arenarsquos platform
Recruiters determine
whether or not to send
the candidate along to
hiring managers
Hiring managers use
Arenarsquos recommendation
as an optional factor in
the hiring decision
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
24
Accelerating Research
1) Yu KH et al ldquoPredicting non-small cell lung cancer prognosis by fully automated microscopic pathology image featuresrdquo Nature No 12474 (2016)
2) Rajwa B et al ldquoAutomated assessment of disease progression in acute myeloid leukemia by probabilistic analysis of flow cytometry datardquo IEEE Transactions on Biomedical Engineering 64 No 5 (2017) 1089-1098
3) Liu Y et al Detecting cancer metastases on gigapixel pathology images arXiv preprint arXiv1703 No 02442 (2017)
4) Rajpurkar P et al Cardiologist-level arrhythmia detection with convolutional neural networks arXiv preprint arXiv1707 No 01836 (2017)
5) Nielsen S et al A predictive model to identify Parkinson disease from administrative claims data Neurology 89 No 14 (2017) 1448-1456
6) Rajpurkar P et al CheXNet Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning arXiv preprint arXiv 1711 No 05225 (2017)
7) Poplin R et al Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning Nature Biomedical Engineering No 2 (2018)
8) ACC Machine learning techniques show promise for supporting medical decisions By quickly assessing risk new algorithms may help triage patients and inform clinical decisions ScienceDaily 2018
9) Oh J et al ldquoA Generalizable Data-Driven Approach to Predict Daily Risk of Clostridium difficile Infection at Two Large Academic Health Centersrdquo ICHE 39 No 4 (2018) 425-433
Nature1 ndash August 2016
ML algorithm can accurately
differentiate two types of lung
cancers and predict patient
survival time
IEEE2 ndash February 2017
ML model predicts relapse rates for
acute myelogenous leukemia with
90 accuracy
arXiv3 ndash March 2017
ML approach to imaging analytics
can identify metastasized breast
cancer with accuracy rates rivaling
those of pathologists
Neurology5 ndash Sept 2017
Predictive model to identify
Parkinsonrsquos disease from claims
data achieved 73-83 accuracy in
study results
arXiv4 ndash July 2017
ML algorithm can diagnose
heart arrhythmias using data
from wearables sensors with
cardiologist-level accuracy
arXiv6 ndash November 2017
Deep-learning algorithm
outperforms Stanford radiologists
at diagnosing pneumonia from
x-rays
American College of Cardiology8 ndash
Feb 2018
Model predicts heart attack
diagnosis among patients in the ED
with chest pain with ~90 accuracy
Nature7 ndash February 2018
ML algorithm can predict a
personrsquos cardiovascular risk
factors using eye scans
Infection Control Hospital
Epidemiology9 ndash April 2018
ML model predicts Clostridium
difficile infection (CDI) earlier
than current diagnostic methods
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
25
Driving the Decision Machine
1Acquire Data
bull Ensure all models have
a sponsor and regular
evaluation against goals
bull All models require
tune-ups as the
environment changes
Monitor Performance
and Revisit
42Train or Refine
the Model
bull The more the better
buthellip
bull High-quality well-
governed data pay
enormous dividends
bull Diverse cross-continuum
sources improve
predictions and
robustness of models
bull Decide what features
are important (eg
explainability)
bull Apply practical and
ethical constraints
bull Evaluate under
realistic conditions
(silent mode real-time
incomplete data)
bull Embed insights directly
into applications at key
decision points
bull Reengineer processes
as warranted
bull Consider the ldquo5 rightsrdquo
of decision support
3Incorporate Models
into Workflow
Hint The Analytics Is the Easy Part
Begin with
bull Explicit leadership agreement on outcome goals expected benefit mechanisms
bull Participation from empowered representatives of impacted process
bull External experiencemdashlearn from others
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
26
New Data Sources
Mainstream Systems Cover Only a Fraction of Predictive Power
Acquire Data
Sources Nash DB ldquoPopulation Health Why It Mattersrdquo Essentials in Population Health Childrenrsquos
Hospital Association Thomas Jefferson University October 2016 Health Care IT Advisor research and
analysis
1) RPM = Remote patient monitoring
EMR(s)
Lab Systems
Pharmacy Systems
Imaging
Medical Claims
Prescription Drug Claims
Billing and Supply Chain
Scheduling Systems
Medical
Care Data
10
Sensors RPM1
mHealth Apps
Patient-Reported
Outcomes
Genetics
Socioeconomic Data
Patient Questionnaires
Human
Biology
Data
20
Lifestyle
and
Behavior
Data
50
Social and
Environmental
Data
20
Data Sources by Influence on Health Outcomes
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
27
Big Data a Technology and a Philosophy
ldquoTherersquos Gold in Them Thar Hillsrdquo
Acquire Data
Sources Cloudera ldquoInova Translational Medicine Institute Partners with Cloudera to Advance Genome-Based Machine
Learning Initiatives and Save Livesrdquo PR Newswire June 2017 Health Care IT Advisor research and analysis
1) Originally coined by Doug Laney of the Meta Group
Assumes Value in the Data
Based on a premise that there are
valuable insights that can be distilled
from the mountains of data captured
by machines
Characterised by the 3 Vs1
bull Volume
bull Velocity
bull Variety
Places greater emphasis on
unstructured data
Free text images streams of sensor
data are all fair targets for analysis
Big data is the difference
between heart rates and
heart beats
Case in Brief Inova Health System
bull Inovarsquos Translational Medicine Institute
(ITMI) is assembling one of the worldrsquos
largest whole genome sequence
databases connected to patient
information in a health care system
bull Researchers use the Cloudera
machine learning platform to analyze
terabytes of clinical and genomic data
and identify genetic links to diseases
bull Insights are used to develop
personalized treatment plans for
patients in collaboration with the
treating physician
Resource Big Data in Health Care
Strengths challenges and pragmatic
guidelines for those considering the addition
of big data capabilities to their IT portfolio
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
30
How Machines Learn
Similar Methods Differing Goals
Build the Model
Source Advisory Board research and analysis
Input Data
bull The more the better
bull Labeled with key indicators
Adjust the Model
bull Automated trial and error
bull Cross-breeding successful
candidates
Repeat
Supervised Learning
Evaluate models against the ldquocorrectrdquo answers
provided with the data The learning process
focuses on minimizing errors
Unsupervised Learning
Evaluate models against an abstract algorithmic
goal Example Divide this population of diabetics
into a handful of subpopulations that have similar
future risk profiles
Reinforcement Learning
Evaluate models by giving a reward for
desirable outcomes and a penalty for negative
outcomes Models are trained to maximize their
total rewards over timeReady for Review
Evaluate Performance
bull Against what goal
bull Forces explicit
tradeoffs (eg cost
vs quality)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
32
Black Boxes Shades of Grey
Differing Views on the Importance of Explainability
Incorporate into Workflow
Mike Wall PharmD MBA
Chief Analytics Officer
The University of Chicago Medicine
The opaqueness of black-box models is
really scary When we canrsquot tell a
clinician why a patient wonrsquot do well on a
certain medication that can be a tough
thing to blindly trust from a clinical
perspective We want to reach a place
where we augment our practice
through AI rather than dictatingrdquo
Nigam Shah MBBS PhD
Associate Professor of Medicine
(Biomedical Informatics)
Stanford University
I am pushing to evaluate these models
via a similar set of mechanisms as RCT1
or AB testing We should evaluate
against the current status-quo and do a
beforeafter studyhellipWe have to convince
ourselves of the utility of these models
We should focus on utility before
explainabilityrdquo
Surgery is the
best option ()
Data
Black Box Model
1) RCT = Randomized controlled trial
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
33
Workflow Is Key to Adoption
Incorporate into Workflow
The Right Information
To the Right Person
In the Right Form
By the Right Channel
At the Right Time
Case in Brief Mayo Breast Oncology
Clinic Trials Recruitment
bull Up to 100 concurrent drug therapy clinical trials are
available to Mayo Clinic breast cancer patients making
it challenging to consistently identify candidates
bull New clinical trial matching technology developed in
collaboration with IBM Watson Health evaluates charts
to identify eligible patients
bull Launched in January 2016 as a point-of-care solution
for oncology providers initial adoption was low due to
time pressures and fell to zero by February
bull Rework by health systems engineers resulted in a July
2016 relaunch screening moved to occur in
advance of patient visits so clinicians know of trial
matches before opening a patientrsquos chart
bull New process enrolls significantly more patients in
clinical trials than the prior manual process success led
to additional coordinator funding
The Five Rights of
Clinical Decision Support
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
34
Health Carersquos Field Agents
Medical Devices Become Intelligent Partners
Incorporate into Workflow
1) EKG = Electrocardiogram
ME
DT
RO
NIC
MIN
IME
D 6
70G
Case in Brief Artificial Pancreas and Smart Infusion Pumpsmdash
Medtronic MiniMed Connect
bull SMARTGUARD mimics some functions of a healthy pancreas
measures and predicts glucose level drift and adjusts dosing
bull Insulin pump and continuous glucose monitoring can send data directly
to smartphone
bull Trials showed a 44 reduction in hypoglycemia and reduction in
average A1C values from 74 to 69
Case in Brief AliveCor KardiaMobile
bull Portable consumer-grade single-lead EKG1 retails for $100
bull Intended for self-capture of intermittent cardiac events such as atrial
fibrillation (AFIB)
bull Smartphone-based machine learning validates quality of trace
assesses possibility of AFIB classifies normal abnormal sinus rhythm
bull EKGs can be sent to a board-certified cardiologist for interpretation
for $19
Sources alivecorcom Medtroniccom Health Care IT Advisor research and
analysis
AliveC
orK
ard
iaM
obile
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP13
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
14
AI Are We There Yet
And if Not When
Source Health Care IT Advisor research and analysis
We May Be at the Start of an Explosive
Growth in AI Capabilities
Incremental Growth
AI continues to make
small advances
delivering value in
niche applicationsAI Winters AI has
already gone through
past phases of hype and
troughs of disillusionment
Exponential Growth
AI rapidly gains
capability and becomes
a mainstream part of
most digital systems
Unpredictable Timing Some
advances never seem to arrive
(conversational systems)
while others take off
unexpectedly (smartphones)
60s 70s 80s 90s 2000 2010 2020 203050s 2040 2050
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
15
Fuel for the AI Fire
Source Health Care IT Advisor research and analysis
1) EMR = Electronic medical record
2) GPU = Graphics processing unit
3) TPU = Tensor processing unit
AlgorithmsData
bull Internet Everywhere
bull Broad EMR1 Adoption
bull Wearables
bull Real-Time Location
bull IoT Sensors
bull Genomics Proteomics
Microbiome
bull Automated Feature
Extraction
bull Cluster Compute
Platforms (eg
Spark Hadoop)
bull Deep Learning
bull Off-the-Shelf
Libraries for
Common Tasksbull Faster Processors
bull Specialised Processors (GPUs2 TPUs3)
bull Inexpensive Storage
bull Elastic Cloud Computing
Hardware
Combinatorial Advances Delivering
Radically Better Models
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
16
The AI Winter Is Thawing
Source Health Care IT Advisor research and analysis
1) NASA = National Aeronautics and Space Administration
Microsoft and Alibaba outscore humans in a Stanford reading comprehension test 2018
Google DeepMindrsquos AlphaZero convincingly leads at Chess Shogi Go 2017
Skype launches real-time voice translation 2014
Mass market voice assistants (Apple Siri Google Now Microsoft Cortana) 2012
IBM Watson conclusively beats Jeopardy game show champions 2011
NASArsquos1 Mars Rovers operate semi-autonomously 2004
IBMrsquos Deep Blue beats world chess champion Gary Kasparov 1997
1950 Alan Turing proposes the ldquoTuring Testrdquo for conversational systems
1956 The term ldquoArtificial Intelligencerdquo is coined
1970s First AI Winter
1981 Digital Equipment Corporation order
expert saves company $40M per year
Timeline of AI Successes
From Toys to Tools
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
17
DeepMindrsquos AlphaZero
Novice to Superhuman in Just Four Hours
DeepMindrsquos AlphaZero Uses lsquoZero Human Knowledgersquo
Sources Vincent J ldquoDeepMindrsquos AI became a superhuman chess player in a few hours just for funrdquo The
Verge December 6 2017 Gershgorn D ldquoDeepMind wants to find the next miracle materialmdashexperts just
donrsquot know theyrsquoll pull it offrdquo Quartz October 25 2017 Health Care IT Advisor research and analysis
By not using this human data by not using
human features or human expertise in any
fashion wersquove actually rem
oved the constraints of human knowledge Itrsquos
able to therefore create knowledge for itselfrdquo
bull AI product of Google
subsidiary DeepMind
bull Provided no prior
knowledge of the
game given only basic
game rules and an
objective (win)
bull Learns through self-play
at an accelerated pace
(lsquoreinforced learningrsquo)
searches 80 thousand
positions per second
bull Variants of AlphaZero also
defeated world champion
Go and Shogi systems
AlphaZero
outperformed chess
world-champion
Stockfish1 in just
4 hours (300k steps)
By not using this human data by not using human
features or human expertise in any fashion wersquove
actually removed the constraints of human knowledge
Itrsquos able to therefore create knowledge for itselfrdquo
David Silver Lead Programmer
DEEPMIND
1) Stockfish is the 2016 Top Chess Engine Championship (TCEC) world-champion
AlphaZero Chess
Performance
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
18
Need a Radiologist
Therersquos an App for That
Sources Kubota T ldquoStanford algorithm can diagnose pneumonia better than
radiologistsrdquo Stanford News Nov 2017 Health Care IT Advisor research and analysis
1) Stanford University Medical Center
2) NIH = National Institutes of Health
Case in Brief Stanford University1
CheXNet Algorithm
bull Algorithm developed by researchers at
Stanford University in the US can diagnose
14 common pathologies in chest x-rays
bull Trained on ChestX-ray14 a public data set
released by the NIH2 containing 112120
frontal-view chest x-ray images labelled with
the 14 possible pathologies
bull Outperforms previous models from the same
data set for all 14 conditions and diagnoses
pneumonia at an accuracy exceeding the
performance of four control radiologists
bull Produces heatmaps that visualize the areas
of an image most indicative of disease
Development of CheXNet
Sept 26 2017
ChestX-ray14 data set
released along with a
preliminary algorithm
that could detect the
labelled conditionsasymp One week later
CheXNet could
diagnose 10 of the
14 pathologies more
accurately than all
previous algorithmsasymp One month later
CheXNet surpassed
best published results
for all 14 pathologies
and outperformed
Stanford radiologists in
detecting pneumonialdquoI treat the model as a lazy resident Theyrsquore not dumb theyrsquoll
just use every trick they can find to avoid doing hard workrdquo
Dr Matthew Lungren RadiologistStanford University Medical Center
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
19
Lessons from Stanfordrsquos CheXNet
AI Is Effective on Narrow Tasks
Recent advances in AI have focused on
solving narrowly defined tasks Each
sub-problem (eg pneumonia) is a model
Handle ldquoNormalrdquo Explicitly
Identifying normal conditions versus
anomalies can help recognize when
human review may be needed
Explainable1 AI (XAI)
Models that can be interpreted at least
partially are easier to trust deploy and
govern Explainability is an area of rapid
progress in AI research
1) Also known as interpretable AI
Training Requires a Teacher or Referee
Machine learning needs to be told what
answers are correct or what actions produce
positive outcomes Well-curated data are
critical to the learning process
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
20
The AI Value Proposition
AI Excels at Decisions That Are Dull Detailed and Dear (Expensive)
Three Key Areas of Benefit
Dear
Provide speed capacity 24x7
availability and consistency
at a fraction of the cost
Although these are well
known benefits of automation
when you apply this to
cognitive and complex tasks
itrsquos a new ball game
Detailed
Evaluate a vastly broader
and deeper set of data to
improve decision quality
The complexity of health
care decision making is
growing due to new data
from genetics IoT devices
wearables better
interoperability and the
growing number of drugs
and treatments
Unlimited capacity
instant response
disruptive economics
Perform at super-human
levels exceeding experts
Serves as an assistant
(watches your back)
Dull
Free human decision makers
to focus on more challenging
ldquotop of licenserdquo activities
Reduce effort spent on
repetitive tasks process
large amounts of complex
information provide
continuous monitoring and
notifications and ensure
ldquostuffrdquo doesnrsquot ldquofall through
the cracksrdquo
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
21
Where to Start Where Wersquore Going
1) Narrow (Task Weak) AI focuses on narrowly defined problems such as predicting a patientrsquos risk of developing sepsis
2) General (Strong) AI attempts to match the flexibility and versatility of human intelligence
Narr
ow
(T
ask W
ea
k)
AI1
Clinical Decisions
Administrative Decisions
Ge
ne
ral (S
tro
ng
) A
I2
Full Diagnosis
and Treatment Plan
Clinical
Chatbots
Radiology
Interpretation
Acute
Clinical
Risk
Recruiting
Safety Risk Smart
Monitoring
Medication Dosing
Administrative
Chatbots
(Concierge)
Precision
Engagement
Precision
Medicine
Ambient
Documentation
Adaptive User
Interfaces
Worklist
Prioritization
Inevitable with Time
Todayrsquos Opportunity
(growing)
The Computer Will
See You Now
The Uber-ization of
Health Care
An AI Application Landscape
Capacity Staffing
Management
Population
Health Risk
The Self-Driving
HospitalInformation
Security
Targeted
Marketing
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
22
The Self-Driving Hospital
Combine Predictive Models with Front-Line Activation
Sources ldquoVirtual Air Traffic Control for Emergency Departmentsrdquo Qventus
httpsqventuscomcontentuploads201703QVE-ED-Solution-Briefpdf Health Care
IT Advisor research and analysis
1) EVS = Environmental services
2) LWBS = Left without being seen
3) LOS = Length of stay
Case in Brief
Mercy Hospital Fort Smith
bull 336-bed acute care hospital in Fort
Smith AR
bull Partnered with Qventus to improve
emergency department (ED) patient flow
and patient satisfaction
bull Analytics applies ML algorithms to
anticipate potential capacity shortfalls
process bottlenecks
bull Application sends real-time notifications
(nudges) to care teams via their mobile
phones to trigger specific interventions
bull Emergency department results
ndash LWBS2 rates dropped 30
ndash Door-to-doc time reduced by 20
ndash 24-minute reduction in average LOS3
ndash Annual case capacity increased by 6
See that a radiology order is delayed for a
patient within a few days of discharge and
notify radiology to prioritize the reading
Activate more EVS1 staff when high churn
of beds is anticipated
Send an early warning for difficult patient
placements at discharge
Prioritize inpatient transports depending
on expected bed needs
Sample ldquoDecision Recipesrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
23
Applicants Like You Liked This Job
Predictive Models Reduce Turnover Improve Engagement
Case in Brief MultiCare Health System
bull Non-profit system based in Tacoma WA with 16000 employees
bull Partnered with Arena a cloud-based HR solution to predict the
likelihood that a candidate will be retained in a role
bull Arenarsquos machine learning algorithms evaluate applicant
submissions digital interactions and externally sourced data on
the organization (eg employer review sites)
bull Achieved a 40 reduction in RN turnover at 180 days and a
28 reduction in overall turnover at 180 days
MultiCarersquos Hiring Process
Reduced recruiting
expenses
More predictable
staffing levels
More experienced
staff
Realized Benefits
Algorithms predict the likelihood the
applicant will be retained and
engaged in the target role and send
that information to the recruiter
Applicant completes
a 15-20 minute
assessment process
on Arenarsquos platform
Recruiters determine
whether or not to send
the candidate along to
hiring managers
Hiring managers use
Arenarsquos recommendation
as an optional factor in
the hiring decision
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
24
Accelerating Research
1) Yu KH et al ldquoPredicting non-small cell lung cancer prognosis by fully automated microscopic pathology image featuresrdquo Nature No 12474 (2016)
2) Rajwa B et al ldquoAutomated assessment of disease progression in acute myeloid leukemia by probabilistic analysis of flow cytometry datardquo IEEE Transactions on Biomedical Engineering 64 No 5 (2017) 1089-1098
3) Liu Y et al Detecting cancer metastases on gigapixel pathology images arXiv preprint arXiv1703 No 02442 (2017)
4) Rajpurkar P et al Cardiologist-level arrhythmia detection with convolutional neural networks arXiv preprint arXiv1707 No 01836 (2017)
5) Nielsen S et al A predictive model to identify Parkinson disease from administrative claims data Neurology 89 No 14 (2017) 1448-1456
6) Rajpurkar P et al CheXNet Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning arXiv preprint arXiv 1711 No 05225 (2017)
7) Poplin R et al Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning Nature Biomedical Engineering No 2 (2018)
8) ACC Machine learning techniques show promise for supporting medical decisions By quickly assessing risk new algorithms may help triage patients and inform clinical decisions ScienceDaily 2018
9) Oh J et al ldquoA Generalizable Data-Driven Approach to Predict Daily Risk of Clostridium difficile Infection at Two Large Academic Health Centersrdquo ICHE 39 No 4 (2018) 425-433
Nature1 ndash August 2016
ML algorithm can accurately
differentiate two types of lung
cancers and predict patient
survival time
IEEE2 ndash February 2017
ML model predicts relapse rates for
acute myelogenous leukemia with
90 accuracy
arXiv3 ndash March 2017
ML approach to imaging analytics
can identify metastasized breast
cancer with accuracy rates rivaling
those of pathologists
Neurology5 ndash Sept 2017
Predictive model to identify
Parkinsonrsquos disease from claims
data achieved 73-83 accuracy in
study results
arXiv4 ndash July 2017
ML algorithm can diagnose
heart arrhythmias using data
from wearables sensors with
cardiologist-level accuracy
arXiv6 ndash November 2017
Deep-learning algorithm
outperforms Stanford radiologists
at diagnosing pneumonia from
x-rays
American College of Cardiology8 ndash
Feb 2018
Model predicts heart attack
diagnosis among patients in the ED
with chest pain with ~90 accuracy
Nature7 ndash February 2018
ML algorithm can predict a
personrsquos cardiovascular risk
factors using eye scans
Infection Control Hospital
Epidemiology9 ndash April 2018
ML model predicts Clostridium
difficile infection (CDI) earlier
than current diagnostic methods
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
25
Driving the Decision Machine
1Acquire Data
bull Ensure all models have
a sponsor and regular
evaluation against goals
bull All models require
tune-ups as the
environment changes
Monitor Performance
and Revisit
42Train or Refine
the Model
bull The more the better
buthellip
bull High-quality well-
governed data pay
enormous dividends
bull Diverse cross-continuum
sources improve
predictions and
robustness of models
bull Decide what features
are important (eg
explainability)
bull Apply practical and
ethical constraints
bull Evaluate under
realistic conditions
(silent mode real-time
incomplete data)
bull Embed insights directly
into applications at key
decision points
bull Reengineer processes
as warranted
bull Consider the ldquo5 rightsrdquo
of decision support
3Incorporate Models
into Workflow
Hint The Analytics Is the Easy Part
Begin with
bull Explicit leadership agreement on outcome goals expected benefit mechanisms
bull Participation from empowered representatives of impacted process
bull External experiencemdashlearn from others
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
26
New Data Sources
Mainstream Systems Cover Only a Fraction of Predictive Power
Acquire Data
Sources Nash DB ldquoPopulation Health Why It Mattersrdquo Essentials in Population Health Childrenrsquos
Hospital Association Thomas Jefferson University October 2016 Health Care IT Advisor research and
analysis
1) RPM = Remote patient monitoring
EMR(s)
Lab Systems
Pharmacy Systems
Imaging
Medical Claims
Prescription Drug Claims
Billing and Supply Chain
Scheduling Systems
Medical
Care Data
10
Sensors RPM1
mHealth Apps
Patient-Reported
Outcomes
Genetics
Socioeconomic Data
Patient Questionnaires
Human
Biology
Data
20
Lifestyle
and
Behavior
Data
50
Social and
Environmental
Data
20
Data Sources by Influence on Health Outcomes
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
27
Big Data a Technology and a Philosophy
ldquoTherersquos Gold in Them Thar Hillsrdquo
Acquire Data
Sources Cloudera ldquoInova Translational Medicine Institute Partners with Cloudera to Advance Genome-Based Machine
Learning Initiatives and Save Livesrdquo PR Newswire June 2017 Health Care IT Advisor research and analysis
1) Originally coined by Doug Laney of the Meta Group
Assumes Value in the Data
Based on a premise that there are
valuable insights that can be distilled
from the mountains of data captured
by machines
Characterised by the 3 Vs1
bull Volume
bull Velocity
bull Variety
Places greater emphasis on
unstructured data
Free text images streams of sensor
data are all fair targets for analysis
Big data is the difference
between heart rates and
heart beats
Case in Brief Inova Health System
bull Inovarsquos Translational Medicine Institute
(ITMI) is assembling one of the worldrsquos
largest whole genome sequence
databases connected to patient
information in a health care system
bull Researchers use the Cloudera
machine learning platform to analyze
terabytes of clinical and genomic data
and identify genetic links to diseases
bull Insights are used to develop
personalized treatment plans for
patients in collaboration with the
treating physician
Resource Big Data in Health Care
Strengths challenges and pragmatic
guidelines for those considering the addition
of big data capabilities to their IT portfolio
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
30
How Machines Learn
Similar Methods Differing Goals
Build the Model
Source Advisory Board research and analysis
Input Data
bull The more the better
bull Labeled with key indicators
Adjust the Model
bull Automated trial and error
bull Cross-breeding successful
candidates
Repeat
Supervised Learning
Evaluate models against the ldquocorrectrdquo answers
provided with the data The learning process
focuses on minimizing errors
Unsupervised Learning
Evaluate models against an abstract algorithmic
goal Example Divide this population of diabetics
into a handful of subpopulations that have similar
future risk profiles
Reinforcement Learning
Evaluate models by giving a reward for
desirable outcomes and a penalty for negative
outcomes Models are trained to maximize their
total rewards over timeReady for Review
Evaluate Performance
bull Against what goal
bull Forces explicit
tradeoffs (eg cost
vs quality)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
32
Black Boxes Shades of Grey
Differing Views on the Importance of Explainability
Incorporate into Workflow
Mike Wall PharmD MBA
Chief Analytics Officer
The University of Chicago Medicine
The opaqueness of black-box models is
really scary When we canrsquot tell a
clinician why a patient wonrsquot do well on a
certain medication that can be a tough
thing to blindly trust from a clinical
perspective We want to reach a place
where we augment our practice
through AI rather than dictatingrdquo
Nigam Shah MBBS PhD
Associate Professor of Medicine
(Biomedical Informatics)
Stanford University
I am pushing to evaluate these models
via a similar set of mechanisms as RCT1
or AB testing We should evaluate
against the current status-quo and do a
beforeafter studyhellipWe have to convince
ourselves of the utility of these models
We should focus on utility before
explainabilityrdquo
Surgery is the
best option ()
Data
Black Box Model
1) RCT = Randomized controlled trial
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
33
Workflow Is Key to Adoption
Incorporate into Workflow
The Right Information
To the Right Person
In the Right Form
By the Right Channel
At the Right Time
Case in Brief Mayo Breast Oncology
Clinic Trials Recruitment
bull Up to 100 concurrent drug therapy clinical trials are
available to Mayo Clinic breast cancer patients making
it challenging to consistently identify candidates
bull New clinical trial matching technology developed in
collaboration with IBM Watson Health evaluates charts
to identify eligible patients
bull Launched in January 2016 as a point-of-care solution
for oncology providers initial adoption was low due to
time pressures and fell to zero by February
bull Rework by health systems engineers resulted in a July
2016 relaunch screening moved to occur in
advance of patient visits so clinicians know of trial
matches before opening a patientrsquos chart
bull New process enrolls significantly more patients in
clinical trials than the prior manual process success led
to additional coordinator funding
The Five Rights of
Clinical Decision Support
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
34
Health Carersquos Field Agents
Medical Devices Become Intelligent Partners
Incorporate into Workflow
1) EKG = Electrocardiogram
ME
DT
RO
NIC
MIN
IME
D 6
70G
Case in Brief Artificial Pancreas and Smart Infusion Pumpsmdash
Medtronic MiniMed Connect
bull SMARTGUARD mimics some functions of a healthy pancreas
measures and predicts glucose level drift and adjusts dosing
bull Insulin pump and continuous glucose monitoring can send data directly
to smartphone
bull Trials showed a 44 reduction in hypoglycemia and reduction in
average A1C values from 74 to 69
Case in Brief AliveCor KardiaMobile
bull Portable consumer-grade single-lead EKG1 retails for $100
bull Intended for self-capture of intermittent cardiac events such as atrial
fibrillation (AFIB)
bull Smartphone-based machine learning validates quality of trace
assesses possibility of AFIB classifies normal abnormal sinus rhythm
bull EKGs can be sent to a board-certified cardiologist for interpretation
for $19
Sources alivecorcom Medtroniccom Health Care IT Advisor research and
analysis
AliveC
orK
ard
iaM
obile
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
14
AI Are We There Yet
And if Not When
Source Health Care IT Advisor research and analysis
We May Be at the Start of an Explosive
Growth in AI Capabilities
Incremental Growth
AI continues to make
small advances
delivering value in
niche applicationsAI Winters AI has
already gone through
past phases of hype and
troughs of disillusionment
Exponential Growth
AI rapidly gains
capability and becomes
a mainstream part of
most digital systems
Unpredictable Timing Some
advances never seem to arrive
(conversational systems)
while others take off
unexpectedly (smartphones)
60s 70s 80s 90s 2000 2010 2020 203050s 2040 2050
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
15
Fuel for the AI Fire
Source Health Care IT Advisor research and analysis
1) EMR = Electronic medical record
2) GPU = Graphics processing unit
3) TPU = Tensor processing unit
AlgorithmsData
bull Internet Everywhere
bull Broad EMR1 Adoption
bull Wearables
bull Real-Time Location
bull IoT Sensors
bull Genomics Proteomics
Microbiome
bull Automated Feature
Extraction
bull Cluster Compute
Platforms (eg
Spark Hadoop)
bull Deep Learning
bull Off-the-Shelf
Libraries for
Common Tasksbull Faster Processors
bull Specialised Processors (GPUs2 TPUs3)
bull Inexpensive Storage
bull Elastic Cloud Computing
Hardware
Combinatorial Advances Delivering
Radically Better Models
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
16
The AI Winter Is Thawing
Source Health Care IT Advisor research and analysis
1) NASA = National Aeronautics and Space Administration
Microsoft and Alibaba outscore humans in a Stanford reading comprehension test 2018
Google DeepMindrsquos AlphaZero convincingly leads at Chess Shogi Go 2017
Skype launches real-time voice translation 2014
Mass market voice assistants (Apple Siri Google Now Microsoft Cortana) 2012
IBM Watson conclusively beats Jeopardy game show champions 2011
NASArsquos1 Mars Rovers operate semi-autonomously 2004
IBMrsquos Deep Blue beats world chess champion Gary Kasparov 1997
1950 Alan Turing proposes the ldquoTuring Testrdquo for conversational systems
1956 The term ldquoArtificial Intelligencerdquo is coined
1970s First AI Winter
1981 Digital Equipment Corporation order
expert saves company $40M per year
Timeline of AI Successes
From Toys to Tools
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
17
DeepMindrsquos AlphaZero
Novice to Superhuman in Just Four Hours
DeepMindrsquos AlphaZero Uses lsquoZero Human Knowledgersquo
Sources Vincent J ldquoDeepMindrsquos AI became a superhuman chess player in a few hours just for funrdquo The
Verge December 6 2017 Gershgorn D ldquoDeepMind wants to find the next miracle materialmdashexperts just
donrsquot know theyrsquoll pull it offrdquo Quartz October 25 2017 Health Care IT Advisor research and analysis
By not using this human data by not using
human features or human expertise in any
fashion wersquove actually rem
oved the constraints of human knowledge Itrsquos
able to therefore create knowledge for itselfrdquo
bull AI product of Google
subsidiary DeepMind
bull Provided no prior
knowledge of the
game given only basic
game rules and an
objective (win)
bull Learns through self-play
at an accelerated pace
(lsquoreinforced learningrsquo)
searches 80 thousand
positions per second
bull Variants of AlphaZero also
defeated world champion
Go and Shogi systems
AlphaZero
outperformed chess
world-champion
Stockfish1 in just
4 hours (300k steps)
By not using this human data by not using human
features or human expertise in any fashion wersquove
actually removed the constraints of human knowledge
Itrsquos able to therefore create knowledge for itselfrdquo
David Silver Lead Programmer
DEEPMIND
1) Stockfish is the 2016 Top Chess Engine Championship (TCEC) world-champion
AlphaZero Chess
Performance
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
18
Need a Radiologist
Therersquos an App for That
Sources Kubota T ldquoStanford algorithm can diagnose pneumonia better than
radiologistsrdquo Stanford News Nov 2017 Health Care IT Advisor research and analysis
1) Stanford University Medical Center
2) NIH = National Institutes of Health
Case in Brief Stanford University1
CheXNet Algorithm
bull Algorithm developed by researchers at
Stanford University in the US can diagnose
14 common pathologies in chest x-rays
bull Trained on ChestX-ray14 a public data set
released by the NIH2 containing 112120
frontal-view chest x-ray images labelled with
the 14 possible pathologies
bull Outperforms previous models from the same
data set for all 14 conditions and diagnoses
pneumonia at an accuracy exceeding the
performance of four control radiologists
bull Produces heatmaps that visualize the areas
of an image most indicative of disease
Development of CheXNet
Sept 26 2017
ChestX-ray14 data set
released along with a
preliminary algorithm
that could detect the
labelled conditionsasymp One week later
CheXNet could
diagnose 10 of the
14 pathologies more
accurately than all
previous algorithmsasymp One month later
CheXNet surpassed
best published results
for all 14 pathologies
and outperformed
Stanford radiologists in
detecting pneumonialdquoI treat the model as a lazy resident Theyrsquore not dumb theyrsquoll
just use every trick they can find to avoid doing hard workrdquo
Dr Matthew Lungren RadiologistStanford University Medical Center
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
19
Lessons from Stanfordrsquos CheXNet
AI Is Effective on Narrow Tasks
Recent advances in AI have focused on
solving narrowly defined tasks Each
sub-problem (eg pneumonia) is a model
Handle ldquoNormalrdquo Explicitly
Identifying normal conditions versus
anomalies can help recognize when
human review may be needed
Explainable1 AI (XAI)
Models that can be interpreted at least
partially are easier to trust deploy and
govern Explainability is an area of rapid
progress in AI research
1) Also known as interpretable AI
Training Requires a Teacher or Referee
Machine learning needs to be told what
answers are correct or what actions produce
positive outcomes Well-curated data are
critical to the learning process
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
20
The AI Value Proposition
AI Excels at Decisions That Are Dull Detailed and Dear (Expensive)
Three Key Areas of Benefit
Dear
Provide speed capacity 24x7
availability and consistency
at a fraction of the cost
Although these are well
known benefits of automation
when you apply this to
cognitive and complex tasks
itrsquos a new ball game
Detailed
Evaluate a vastly broader
and deeper set of data to
improve decision quality
The complexity of health
care decision making is
growing due to new data
from genetics IoT devices
wearables better
interoperability and the
growing number of drugs
and treatments
Unlimited capacity
instant response
disruptive economics
Perform at super-human
levels exceeding experts
Serves as an assistant
(watches your back)
Dull
Free human decision makers
to focus on more challenging
ldquotop of licenserdquo activities
Reduce effort spent on
repetitive tasks process
large amounts of complex
information provide
continuous monitoring and
notifications and ensure
ldquostuffrdquo doesnrsquot ldquofall through
the cracksrdquo
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
21
Where to Start Where Wersquore Going
1) Narrow (Task Weak) AI focuses on narrowly defined problems such as predicting a patientrsquos risk of developing sepsis
2) General (Strong) AI attempts to match the flexibility and versatility of human intelligence
Narr
ow
(T
ask W
ea
k)
AI1
Clinical Decisions
Administrative Decisions
Ge
ne
ral (S
tro
ng
) A
I2
Full Diagnosis
and Treatment Plan
Clinical
Chatbots
Radiology
Interpretation
Acute
Clinical
Risk
Recruiting
Safety Risk Smart
Monitoring
Medication Dosing
Administrative
Chatbots
(Concierge)
Precision
Engagement
Precision
Medicine
Ambient
Documentation
Adaptive User
Interfaces
Worklist
Prioritization
Inevitable with Time
Todayrsquos Opportunity
(growing)
The Computer Will
See You Now
The Uber-ization of
Health Care
An AI Application Landscape
Capacity Staffing
Management
Population
Health Risk
The Self-Driving
HospitalInformation
Security
Targeted
Marketing
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
22
The Self-Driving Hospital
Combine Predictive Models with Front-Line Activation
Sources ldquoVirtual Air Traffic Control for Emergency Departmentsrdquo Qventus
httpsqventuscomcontentuploads201703QVE-ED-Solution-Briefpdf Health Care
IT Advisor research and analysis
1) EVS = Environmental services
2) LWBS = Left without being seen
3) LOS = Length of stay
Case in Brief
Mercy Hospital Fort Smith
bull 336-bed acute care hospital in Fort
Smith AR
bull Partnered with Qventus to improve
emergency department (ED) patient flow
and patient satisfaction
bull Analytics applies ML algorithms to
anticipate potential capacity shortfalls
process bottlenecks
bull Application sends real-time notifications
(nudges) to care teams via their mobile
phones to trigger specific interventions
bull Emergency department results
ndash LWBS2 rates dropped 30
ndash Door-to-doc time reduced by 20
ndash 24-minute reduction in average LOS3
ndash Annual case capacity increased by 6
See that a radiology order is delayed for a
patient within a few days of discharge and
notify radiology to prioritize the reading
Activate more EVS1 staff when high churn
of beds is anticipated
Send an early warning for difficult patient
placements at discharge
Prioritize inpatient transports depending
on expected bed needs
Sample ldquoDecision Recipesrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
23
Applicants Like You Liked This Job
Predictive Models Reduce Turnover Improve Engagement
Case in Brief MultiCare Health System
bull Non-profit system based in Tacoma WA with 16000 employees
bull Partnered with Arena a cloud-based HR solution to predict the
likelihood that a candidate will be retained in a role
bull Arenarsquos machine learning algorithms evaluate applicant
submissions digital interactions and externally sourced data on
the organization (eg employer review sites)
bull Achieved a 40 reduction in RN turnover at 180 days and a
28 reduction in overall turnover at 180 days
MultiCarersquos Hiring Process
Reduced recruiting
expenses
More predictable
staffing levels
More experienced
staff
Realized Benefits
Algorithms predict the likelihood the
applicant will be retained and
engaged in the target role and send
that information to the recruiter
Applicant completes
a 15-20 minute
assessment process
on Arenarsquos platform
Recruiters determine
whether or not to send
the candidate along to
hiring managers
Hiring managers use
Arenarsquos recommendation
as an optional factor in
the hiring decision
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
24
Accelerating Research
1) Yu KH et al ldquoPredicting non-small cell lung cancer prognosis by fully automated microscopic pathology image featuresrdquo Nature No 12474 (2016)
2) Rajwa B et al ldquoAutomated assessment of disease progression in acute myeloid leukemia by probabilistic analysis of flow cytometry datardquo IEEE Transactions on Biomedical Engineering 64 No 5 (2017) 1089-1098
3) Liu Y et al Detecting cancer metastases on gigapixel pathology images arXiv preprint arXiv1703 No 02442 (2017)
4) Rajpurkar P et al Cardiologist-level arrhythmia detection with convolutional neural networks arXiv preprint arXiv1707 No 01836 (2017)
5) Nielsen S et al A predictive model to identify Parkinson disease from administrative claims data Neurology 89 No 14 (2017) 1448-1456
6) Rajpurkar P et al CheXNet Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning arXiv preprint arXiv 1711 No 05225 (2017)
7) Poplin R et al Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning Nature Biomedical Engineering No 2 (2018)
8) ACC Machine learning techniques show promise for supporting medical decisions By quickly assessing risk new algorithms may help triage patients and inform clinical decisions ScienceDaily 2018
9) Oh J et al ldquoA Generalizable Data-Driven Approach to Predict Daily Risk of Clostridium difficile Infection at Two Large Academic Health Centersrdquo ICHE 39 No 4 (2018) 425-433
Nature1 ndash August 2016
ML algorithm can accurately
differentiate two types of lung
cancers and predict patient
survival time
IEEE2 ndash February 2017
ML model predicts relapse rates for
acute myelogenous leukemia with
90 accuracy
arXiv3 ndash March 2017
ML approach to imaging analytics
can identify metastasized breast
cancer with accuracy rates rivaling
those of pathologists
Neurology5 ndash Sept 2017
Predictive model to identify
Parkinsonrsquos disease from claims
data achieved 73-83 accuracy in
study results
arXiv4 ndash July 2017
ML algorithm can diagnose
heart arrhythmias using data
from wearables sensors with
cardiologist-level accuracy
arXiv6 ndash November 2017
Deep-learning algorithm
outperforms Stanford radiologists
at diagnosing pneumonia from
x-rays
American College of Cardiology8 ndash
Feb 2018
Model predicts heart attack
diagnosis among patients in the ED
with chest pain with ~90 accuracy
Nature7 ndash February 2018
ML algorithm can predict a
personrsquos cardiovascular risk
factors using eye scans
Infection Control Hospital
Epidemiology9 ndash April 2018
ML model predicts Clostridium
difficile infection (CDI) earlier
than current diagnostic methods
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
25
Driving the Decision Machine
1Acquire Data
bull Ensure all models have
a sponsor and regular
evaluation against goals
bull All models require
tune-ups as the
environment changes
Monitor Performance
and Revisit
42Train or Refine
the Model
bull The more the better
buthellip
bull High-quality well-
governed data pay
enormous dividends
bull Diverse cross-continuum
sources improve
predictions and
robustness of models
bull Decide what features
are important (eg
explainability)
bull Apply practical and
ethical constraints
bull Evaluate under
realistic conditions
(silent mode real-time
incomplete data)
bull Embed insights directly
into applications at key
decision points
bull Reengineer processes
as warranted
bull Consider the ldquo5 rightsrdquo
of decision support
3Incorporate Models
into Workflow
Hint The Analytics Is the Easy Part
Begin with
bull Explicit leadership agreement on outcome goals expected benefit mechanisms
bull Participation from empowered representatives of impacted process
bull External experiencemdashlearn from others
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
26
New Data Sources
Mainstream Systems Cover Only a Fraction of Predictive Power
Acquire Data
Sources Nash DB ldquoPopulation Health Why It Mattersrdquo Essentials in Population Health Childrenrsquos
Hospital Association Thomas Jefferson University October 2016 Health Care IT Advisor research and
analysis
1) RPM = Remote patient monitoring
EMR(s)
Lab Systems
Pharmacy Systems
Imaging
Medical Claims
Prescription Drug Claims
Billing and Supply Chain
Scheduling Systems
Medical
Care Data
10
Sensors RPM1
mHealth Apps
Patient-Reported
Outcomes
Genetics
Socioeconomic Data
Patient Questionnaires
Human
Biology
Data
20
Lifestyle
and
Behavior
Data
50
Social and
Environmental
Data
20
Data Sources by Influence on Health Outcomes
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
27
Big Data a Technology and a Philosophy
ldquoTherersquos Gold in Them Thar Hillsrdquo
Acquire Data
Sources Cloudera ldquoInova Translational Medicine Institute Partners with Cloudera to Advance Genome-Based Machine
Learning Initiatives and Save Livesrdquo PR Newswire June 2017 Health Care IT Advisor research and analysis
1) Originally coined by Doug Laney of the Meta Group
Assumes Value in the Data
Based on a premise that there are
valuable insights that can be distilled
from the mountains of data captured
by machines
Characterised by the 3 Vs1
bull Volume
bull Velocity
bull Variety
Places greater emphasis on
unstructured data
Free text images streams of sensor
data are all fair targets for analysis
Big data is the difference
between heart rates and
heart beats
Case in Brief Inova Health System
bull Inovarsquos Translational Medicine Institute
(ITMI) is assembling one of the worldrsquos
largest whole genome sequence
databases connected to patient
information in a health care system
bull Researchers use the Cloudera
machine learning platform to analyze
terabytes of clinical and genomic data
and identify genetic links to diseases
bull Insights are used to develop
personalized treatment plans for
patients in collaboration with the
treating physician
Resource Big Data in Health Care
Strengths challenges and pragmatic
guidelines for those considering the addition
of big data capabilities to their IT portfolio
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
30
How Machines Learn
Similar Methods Differing Goals
Build the Model
Source Advisory Board research and analysis
Input Data
bull The more the better
bull Labeled with key indicators
Adjust the Model
bull Automated trial and error
bull Cross-breeding successful
candidates
Repeat
Supervised Learning
Evaluate models against the ldquocorrectrdquo answers
provided with the data The learning process
focuses on minimizing errors
Unsupervised Learning
Evaluate models against an abstract algorithmic
goal Example Divide this population of diabetics
into a handful of subpopulations that have similar
future risk profiles
Reinforcement Learning
Evaluate models by giving a reward for
desirable outcomes and a penalty for negative
outcomes Models are trained to maximize their
total rewards over timeReady for Review
Evaluate Performance
bull Against what goal
bull Forces explicit
tradeoffs (eg cost
vs quality)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
32
Black Boxes Shades of Grey
Differing Views on the Importance of Explainability
Incorporate into Workflow
Mike Wall PharmD MBA
Chief Analytics Officer
The University of Chicago Medicine
The opaqueness of black-box models is
really scary When we canrsquot tell a
clinician why a patient wonrsquot do well on a
certain medication that can be a tough
thing to blindly trust from a clinical
perspective We want to reach a place
where we augment our practice
through AI rather than dictatingrdquo
Nigam Shah MBBS PhD
Associate Professor of Medicine
(Biomedical Informatics)
Stanford University
I am pushing to evaluate these models
via a similar set of mechanisms as RCT1
or AB testing We should evaluate
against the current status-quo and do a
beforeafter studyhellipWe have to convince
ourselves of the utility of these models
We should focus on utility before
explainabilityrdquo
Surgery is the
best option ()
Data
Black Box Model
1) RCT = Randomized controlled trial
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
33
Workflow Is Key to Adoption
Incorporate into Workflow
The Right Information
To the Right Person
In the Right Form
By the Right Channel
At the Right Time
Case in Brief Mayo Breast Oncology
Clinic Trials Recruitment
bull Up to 100 concurrent drug therapy clinical trials are
available to Mayo Clinic breast cancer patients making
it challenging to consistently identify candidates
bull New clinical trial matching technology developed in
collaboration with IBM Watson Health evaluates charts
to identify eligible patients
bull Launched in January 2016 as a point-of-care solution
for oncology providers initial adoption was low due to
time pressures and fell to zero by February
bull Rework by health systems engineers resulted in a July
2016 relaunch screening moved to occur in
advance of patient visits so clinicians know of trial
matches before opening a patientrsquos chart
bull New process enrolls significantly more patients in
clinical trials than the prior manual process success led
to additional coordinator funding
The Five Rights of
Clinical Decision Support
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
34
Health Carersquos Field Agents
Medical Devices Become Intelligent Partners
Incorporate into Workflow
1) EKG = Electrocardiogram
ME
DT
RO
NIC
MIN
IME
D 6
70G
Case in Brief Artificial Pancreas and Smart Infusion Pumpsmdash
Medtronic MiniMed Connect
bull SMARTGUARD mimics some functions of a healthy pancreas
measures and predicts glucose level drift and adjusts dosing
bull Insulin pump and continuous glucose monitoring can send data directly
to smartphone
bull Trials showed a 44 reduction in hypoglycemia and reduction in
average A1C values from 74 to 69
Case in Brief AliveCor KardiaMobile
bull Portable consumer-grade single-lead EKG1 retails for $100
bull Intended for self-capture of intermittent cardiac events such as atrial
fibrillation (AFIB)
bull Smartphone-based machine learning validates quality of trace
assesses possibility of AFIB classifies normal abnormal sinus rhythm
bull EKGs can be sent to a board-certified cardiologist for interpretation
for $19
Sources alivecorcom Medtroniccom Health Care IT Advisor research and
analysis
AliveC
orK
ard
iaM
obile
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
15
Fuel for the AI Fire
Source Health Care IT Advisor research and analysis
1) EMR = Electronic medical record
2) GPU = Graphics processing unit
3) TPU = Tensor processing unit
AlgorithmsData
bull Internet Everywhere
bull Broad EMR1 Adoption
bull Wearables
bull Real-Time Location
bull IoT Sensors
bull Genomics Proteomics
Microbiome
bull Automated Feature
Extraction
bull Cluster Compute
Platforms (eg
Spark Hadoop)
bull Deep Learning
bull Off-the-Shelf
Libraries for
Common Tasksbull Faster Processors
bull Specialised Processors (GPUs2 TPUs3)
bull Inexpensive Storage
bull Elastic Cloud Computing
Hardware
Combinatorial Advances Delivering
Radically Better Models
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
16
The AI Winter Is Thawing
Source Health Care IT Advisor research and analysis
1) NASA = National Aeronautics and Space Administration
Microsoft and Alibaba outscore humans in a Stanford reading comprehension test 2018
Google DeepMindrsquos AlphaZero convincingly leads at Chess Shogi Go 2017
Skype launches real-time voice translation 2014
Mass market voice assistants (Apple Siri Google Now Microsoft Cortana) 2012
IBM Watson conclusively beats Jeopardy game show champions 2011
NASArsquos1 Mars Rovers operate semi-autonomously 2004
IBMrsquos Deep Blue beats world chess champion Gary Kasparov 1997
1950 Alan Turing proposes the ldquoTuring Testrdquo for conversational systems
1956 The term ldquoArtificial Intelligencerdquo is coined
1970s First AI Winter
1981 Digital Equipment Corporation order
expert saves company $40M per year
Timeline of AI Successes
From Toys to Tools
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
17
DeepMindrsquos AlphaZero
Novice to Superhuman in Just Four Hours
DeepMindrsquos AlphaZero Uses lsquoZero Human Knowledgersquo
Sources Vincent J ldquoDeepMindrsquos AI became a superhuman chess player in a few hours just for funrdquo The
Verge December 6 2017 Gershgorn D ldquoDeepMind wants to find the next miracle materialmdashexperts just
donrsquot know theyrsquoll pull it offrdquo Quartz October 25 2017 Health Care IT Advisor research and analysis
By not using this human data by not using
human features or human expertise in any
fashion wersquove actually rem
oved the constraints of human knowledge Itrsquos
able to therefore create knowledge for itselfrdquo
bull AI product of Google
subsidiary DeepMind
bull Provided no prior
knowledge of the
game given only basic
game rules and an
objective (win)
bull Learns through self-play
at an accelerated pace
(lsquoreinforced learningrsquo)
searches 80 thousand
positions per second
bull Variants of AlphaZero also
defeated world champion
Go and Shogi systems
AlphaZero
outperformed chess
world-champion
Stockfish1 in just
4 hours (300k steps)
By not using this human data by not using human
features or human expertise in any fashion wersquove
actually removed the constraints of human knowledge
Itrsquos able to therefore create knowledge for itselfrdquo
David Silver Lead Programmer
DEEPMIND
1) Stockfish is the 2016 Top Chess Engine Championship (TCEC) world-champion
AlphaZero Chess
Performance
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
18
Need a Radiologist
Therersquos an App for That
Sources Kubota T ldquoStanford algorithm can diagnose pneumonia better than
radiologistsrdquo Stanford News Nov 2017 Health Care IT Advisor research and analysis
1) Stanford University Medical Center
2) NIH = National Institutes of Health
Case in Brief Stanford University1
CheXNet Algorithm
bull Algorithm developed by researchers at
Stanford University in the US can diagnose
14 common pathologies in chest x-rays
bull Trained on ChestX-ray14 a public data set
released by the NIH2 containing 112120
frontal-view chest x-ray images labelled with
the 14 possible pathologies
bull Outperforms previous models from the same
data set for all 14 conditions and diagnoses
pneumonia at an accuracy exceeding the
performance of four control radiologists
bull Produces heatmaps that visualize the areas
of an image most indicative of disease
Development of CheXNet
Sept 26 2017
ChestX-ray14 data set
released along with a
preliminary algorithm
that could detect the
labelled conditionsasymp One week later
CheXNet could
diagnose 10 of the
14 pathologies more
accurately than all
previous algorithmsasymp One month later
CheXNet surpassed
best published results
for all 14 pathologies
and outperformed
Stanford radiologists in
detecting pneumonialdquoI treat the model as a lazy resident Theyrsquore not dumb theyrsquoll
just use every trick they can find to avoid doing hard workrdquo
Dr Matthew Lungren RadiologistStanford University Medical Center
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
19
Lessons from Stanfordrsquos CheXNet
AI Is Effective on Narrow Tasks
Recent advances in AI have focused on
solving narrowly defined tasks Each
sub-problem (eg pneumonia) is a model
Handle ldquoNormalrdquo Explicitly
Identifying normal conditions versus
anomalies can help recognize when
human review may be needed
Explainable1 AI (XAI)
Models that can be interpreted at least
partially are easier to trust deploy and
govern Explainability is an area of rapid
progress in AI research
1) Also known as interpretable AI
Training Requires a Teacher or Referee
Machine learning needs to be told what
answers are correct or what actions produce
positive outcomes Well-curated data are
critical to the learning process
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
20
The AI Value Proposition
AI Excels at Decisions That Are Dull Detailed and Dear (Expensive)
Three Key Areas of Benefit
Dear
Provide speed capacity 24x7
availability and consistency
at a fraction of the cost
Although these are well
known benefits of automation
when you apply this to
cognitive and complex tasks
itrsquos a new ball game
Detailed
Evaluate a vastly broader
and deeper set of data to
improve decision quality
The complexity of health
care decision making is
growing due to new data
from genetics IoT devices
wearables better
interoperability and the
growing number of drugs
and treatments
Unlimited capacity
instant response
disruptive economics
Perform at super-human
levels exceeding experts
Serves as an assistant
(watches your back)
Dull
Free human decision makers
to focus on more challenging
ldquotop of licenserdquo activities
Reduce effort spent on
repetitive tasks process
large amounts of complex
information provide
continuous monitoring and
notifications and ensure
ldquostuffrdquo doesnrsquot ldquofall through
the cracksrdquo
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
21
Where to Start Where Wersquore Going
1) Narrow (Task Weak) AI focuses on narrowly defined problems such as predicting a patientrsquos risk of developing sepsis
2) General (Strong) AI attempts to match the flexibility and versatility of human intelligence
Narr
ow
(T
ask W
ea
k)
AI1
Clinical Decisions
Administrative Decisions
Ge
ne
ral (S
tro
ng
) A
I2
Full Diagnosis
and Treatment Plan
Clinical
Chatbots
Radiology
Interpretation
Acute
Clinical
Risk
Recruiting
Safety Risk Smart
Monitoring
Medication Dosing
Administrative
Chatbots
(Concierge)
Precision
Engagement
Precision
Medicine
Ambient
Documentation
Adaptive User
Interfaces
Worklist
Prioritization
Inevitable with Time
Todayrsquos Opportunity
(growing)
The Computer Will
See You Now
The Uber-ization of
Health Care
An AI Application Landscape
Capacity Staffing
Management
Population
Health Risk
The Self-Driving
HospitalInformation
Security
Targeted
Marketing
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
22
The Self-Driving Hospital
Combine Predictive Models with Front-Line Activation
Sources ldquoVirtual Air Traffic Control for Emergency Departmentsrdquo Qventus
httpsqventuscomcontentuploads201703QVE-ED-Solution-Briefpdf Health Care
IT Advisor research and analysis
1) EVS = Environmental services
2) LWBS = Left without being seen
3) LOS = Length of stay
Case in Brief
Mercy Hospital Fort Smith
bull 336-bed acute care hospital in Fort
Smith AR
bull Partnered with Qventus to improve
emergency department (ED) patient flow
and patient satisfaction
bull Analytics applies ML algorithms to
anticipate potential capacity shortfalls
process bottlenecks
bull Application sends real-time notifications
(nudges) to care teams via their mobile
phones to trigger specific interventions
bull Emergency department results
ndash LWBS2 rates dropped 30
ndash Door-to-doc time reduced by 20
ndash 24-minute reduction in average LOS3
ndash Annual case capacity increased by 6
See that a radiology order is delayed for a
patient within a few days of discharge and
notify radiology to prioritize the reading
Activate more EVS1 staff when high churn
of beds is anticipated
Send an early warning for difficult patient
placements at discharge
Prioritize inpatient transports depending
on expected bed needs
Sample ldquoDecision Recipesrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
23
Applicants Like You Liked This Job
Predictive Models Reduce Turnover Improve Engagement
Case in Brief MultiCare Health System
bull Non-profit system based in Tacoma WA with 16000 employees
bull Partnered with Arena a cloud-based HR solution to predict the
likelihood that a candidate will be retained in a role
bull Arenarsquos machine learning algorithms evaluate applicant
submissions digital interactions and externally sourced data on
the organization (eg employer review sites)
bull Achieved a 40 reduction in RN turnover at 180 days and a
28 reduction in overall turnover at 180 days
MultiCarersquos Hiring Process
Reduced recruiting
expenses
More predictable
staffing levels
More experienced
staff
Realized Benefits
Algorithms predict the likelihood the
applicant will be retained and
engaged in the target role and send
that information to the recruiter
Applicant completes
a 15-20 minute
assessment process
on Arenarsquos platform
Recruiters determine
whether or not to send
the candidate along to
hiring managers
Hiring managers use
Arenarsquos recommendation
as an optional factor in
the hiring decision
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
24
Accelerating Research
1) Yu KH et al ldquoPredicting non-small cell lung cancer prognosis by fully automated microscopic pathology image featuresrdquo Nature No 12474 (2016)
2) Rajwa B et al ldquoAutomated assessment of disease progression in acute myeloid leukemia by probabilistic analysis of flow cytometry datardquo IEEE Transactions on Biomedical Engineering 64 No 5 (2017) 1089-1098
3) Liu Y et al Detecting cancer metastases on gigapixel pathology images arXiv preprint arXiv1703 No 02442 (2017)
4) Rajpurkar P et al Cardiologist-level arrhythmia detection with convolutional neural networks arXiv preprint arXiv1707 No 01836 (2017)
5) Nielsen S et al A predictive model to identify Parkinson disease from administrative claims data Neurology 89 No 14 (2017) 1448-1456
6) Rajpurkar P et al CheXNet Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning arXiv preprint arXiv 1711 No 05225 (2017)
7) Poplin R et al Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning Nature Biomedical Engineering No 2 (2018)
8) ACC Machine learning techniques show promise for supporting medical decisions By quickly assessing risk new algorithms may help triage patients and inform clinical decisions ScienceDaily 2018
9) Oh J et al ldquoA Generalizable Data-Driven Approach to Predict Daily Risk of Clostridium difficile Infection at Two Large Academic Health Centersrdquo ICHE 39 No 4 (2018) 425-433
Nature1 ndash August 2016
ML algorithm can accurately
differentiate two types of lung
cancers and predict patient
survival time
IEEE2 ndash February 2017
ML model predicts relapse rates for
acute myelogenous leukemia with
90 accuracy
arXiv3 ndash March 2017
ML approach to imaging analytics
can identify metastasized breast
cancer with accuracy rates rivaling
those of pathologists
Neurology5 ndash Sept 2017
Predictive model to identify
Parkinsonrsquos disease from claims
data achieved 73-83 accuracy in
study results
arXiv4 ndash July 2017
ML algorithm can diagnose
heart arrhythmias using data
from wearables sensors with
cardiologist-level accuracy
arXiv6 ndash November 2017
Deep-learning algorithm
outperforms Stanford radiologists
at diagnosing pneumonia from
x-rays
American College of Cardiology8 ndash
Feb 2018
Model predicts heart attack
diagnosis among patients in the ED
with chest pain with ~90 accuracy
Nature7 ndash February 2018
ML algorithm can predict a
personrsquos cardiovascular risk
factors using eye scans
Infection Control Hospital
Epidemiology9 ndash April 2018
ML model predicts Clostridium
difficile infection (CDI) earlier
than current diagnostic methods
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
25
Driving the Decision Machine
1Acquire Data
bull Ensure all models have
a sponsor and regular
evaluation against goals
bull All models require
tune-ups as the
environment changes
Monitor Performance
and Revisit
42Train or Refine
the Model
bull The more the better
buthellip
bull High-quality well-
governed data pay
enormous dividends
bull Diverse cross-continuum
sources improve
predictions and
robustness of models
bull Decide what features
are important (eg
explainability)
bull Apply practical and
ethical constraints
bull Evaluate under
realistic conditions
(silent mode real-time
incomplete data)
bull Embed insights directly
into applications at key
decision points
bull Reengineer processes
as warranted
bull Consider the ldquo5 rightsrdquo
of decision support
3Incorporate Models
into Workflow
Hint The Analytics Is the Easy Part
Begin with
bull Explicit leadership agreement on outcome goals expected benefit mechanisms
bull Participation from empowered representatives of impacted process
bull External experiencemdashlearn from others
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
26
New Data Sources
Mainstream Systems Cover Only a Fraction of Predictive Power
Acquire Data
Sources Nash DB ldquoPopulation Health Why It Mattersrdquo Essentials in Population Health Childrenrsquos
Hospital Association Thomas Jefferson University October 2016 Health Care IT Advisor research and
analysis
1) RPM = Remote patient monitoring
EMR(s)
Lab Systems
Pharmacy Systems
Imaging
Medical Claims
Prescription Drug Claims
Billing and Supply Chain
Scheduling Systems
Medical
Care Data
10
Sensors RPM1
mHealth Apps
Patient-Reported
Outcomes
Genetics
Socioeconomic Data
Patient Questionnaires
Human
Biology
Data
20
Lifestyle
and
Behavior
Data
50
Social and
Environmental
Data
20
Data Sources by Influence on Health Outcomes
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
27
Big Data a Technology and a Philosophy
ldquoTherersquos Gold in Them Thar Hillsrdquo
Acquire Data
Sources Cloudera ldquoInova Translational Medicine Institute Partners with Cloudera to Advance Genome-Based Machine
Learning Initiatives and Save Livesrdquo PR Newswire June 2017 Health Care IT Advisor research and analysis
1) Originally coined by Doug Laney of the Meta Group
Assumes Value in the Data
Based on a premise that there are
valuable insights that can be distilled
from the mountains of data captured
by machines
Characterised by the 3 Vs1
bull Volume
bull Velocity
bull Variety
Places greater emphasis on
unstructured data
Free text images streams of sensor
data are all fair targets for analysis
Big data is the difference
between heart rates and
heart beats
Case in Brief Inova Health System
bull Inovarsquos Translational Medicine Institute
(ITMI) is assembling one of the worldrsquos
largest whole genome sequence
databases connected to patient
information in a health care system
bull Researchers use the Cloudera
machine learning platform to analyze
terabytes of clinical and genomic data
and identify genetic links to diseases
bull Insights are used to develop
personalized treatment plans for
patients in collaboration with the
treating physician
Resource Big Data in Health Care
Strengths challenges and pragmatic
guidelines for those considering the addition
of big data capabilities to their IT portfolio
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
30
How Machines Learn
Similar Methods Differing Goals
Build the Model
Source Advisory Board research and analysis
Input Data
bull The more the better
bull Labeled with key indicators
Adjust the Model
bull Automated trial and error
bull Cross-breeding successful
candidates
Repeat
Supervised Learning
Evaluate models against the ldquocorrectrdquo answers
provided with the data The learning process
focuses on minimizing errors
Unsupervised Learning
Evaluate models against an abstract algorithmic
goal Example Divide this population of diabetics
into a handful of subpopulations that have similar
future risk profiles
Reinforcement Learning
Evaluate models by giving a reward for
desirable outcomes and a penalty for negative
outcomes Models are trained to maximize their
total rewards over timeReady for Review
Evaluate Performance
bull Against what goal
bull Forces explicit
tradeoffs (eg cost
vs quality)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
32
Black Boxes Shades of Grey
Differing Views on the Importance of Explainability
Incorporate into Workflow
Mike Wall PharmD MBA
Chief Analytics Officer
The University of Chicago Medicine
The opaqueness of black-box models is
really scary When we canrsquot tell a
clinician why a patient wonrsquot do well on a
certain medication that can be a tough
thing to blindly trust from a clinical
perspective We want to reach a place
where we augment our practice
through AI rather than dictatingrdquo
Nigam Shah MBBS PhD
Associate Professor of Medicine
(Biomedical Informatics)
Stanford University
I am pushing to evaluate these models
via a similar set of mechanisms as RCT1
or AB testing We should evaluate
against the current status-quo and do a
beforeafter studyhellipWe have to convince
ourselves of the utility of these models
We should focus on utility before
explainabilityrdquo
Surgery is the
best option ()
Data
Black Box Model
1) RCT = Randomized controlled trial
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
33
Workflow Is Key to Adoption
Incorporate into Workflow
The Right Information
To the Right Person
In the Right Form
By the Right Channel
At the Right Time
Case in Brief Mayo Breast Oncology
Clinic Trials Recruitment
bull Up to 100 concurrent drug therapy clinical trials are
available to Mayo Clinic breast cancer patients making
it challenging to consistently identify candidates
bull New clinical trial matching technology developed in
collaboration with IBM Watson Health evaluates charts
to identify eligible patients
bull Launched in January 2016 as a point-of-care solution
for oncology providers initial adoption was low due to
time pressures and fell to zero by February
bull Rework by health systems engineers resulted in a July
2016 relaunch screening moved to occur in
advance of patient visits so clinicians know of trial
matches before opening a patientrsquos chart
bull New process enrolls significantly more patients in
clinical trials than the prior manual process success led
to additional coordinator funding
The Five Rights of
Clinical Decision Support
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
34
Health Carersquos Field Agents
Medical Devices Become Intelligent Partners
Incorporate into Workflow
1) EKG = Electrocardiogram
ME
DT
RO
NIC
MIN
IME
D 6
70G
Case in Brief Artificial Pancreas and Smart Infusion Pumpsmdash
Medtronic MiniMed Connect
bull SMARTGUARD mimics some functions of a healthy pancreas
measures and predicts glucose level drift and adjusts dosing
bull Insulin pump and continuous glucose monitoring can send data directly
to smartphone
bull Trials showed a 44 reduction in hypoglycemia and reduction in
average A1C values from 74 to 69
Case in Brief AliveCor KardiaMobile
bull Portable consumer-grade single-lead EKG1 retails for $100
bull Intended for self-capture of intermittent cardiac events such as atrial
fibrillation (AFIB)
bull Smartphone-based machine learning validates quality of trace
assesses possibility of AFIB classifies normal abnormal sinus rhythm
bull EKGs can be sent to a board-certified cardiologist for interpretation
for $19
Sources alivecorcom Medtroniccom Health Care IT Advisor research and
analysis
AliveC
orK
ard
iaM
obile
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
16
The AI Winter Is Thawing
Source Health Care IT Advisor research and analysis
1) NASA = National Aeronautics and Space Administration
Microsoft and Alibaba outscore humans in a Stanford reading comprehension test 2018
Google DeepMindrsquos AlphaZero convincingly leads at Chess Shogi Go 2017
Skype launches real-time voice translation 2014
Mass market voice assistants (Apple Siri Google Now Microsoft Cortana) 2012
IBM Watson conclusively beats Jeopardy game show champions 2011
NASArsquos1 Mars Rovers operate semi-autonomously 2004
IBMrsquos Deep Blue beats world chess champion Gary Kasparov 1997
1950 Alan Turing proposes the ldquoTuring Testrdquo for conversational systems
1956 The term ldquoArtificial Intelligencerdquo is coined
1970s First AI Winter
1981 Digital Equipment Corporation order
expert saves company $40M per year
Timeline of AI Successes
From Toys to Tools
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
17
DeepMindrsquos AlphaZero
Novice to Superhuman in Just Four Hours
DeepMindrsquos AlphaZero Uses lsquoZero Human Knowledgersquo
Sources Vincent J ldquoDeepMindrsquos AI became a superhuman chess player in a few hours just for funrdquo The
Verge December 6 2017 Gershgorn D ldquoDeepMind wants to find the next miracle materialmdashexperts just
donrsquot know theyrsquoll pull it offrdquo Quartz October 25 2017 Health Care IT Advisor research and analysis
By not using this human data by not using
human features or human expertise in any
fashion wersquove actually rem
oved the constraints of human knowledge Itrsquos
able to therefore create knowledge for itselfrdquo
bull AI product of Google
subsidiary DeepMind
bull Provided no prior
knowledge of the
game given only basic
game rules and an
objective (win)
bull Learns through self-play
at an accelerated pace
(lsquoreinforced learningrsquo)
searches 80 thousand
positions per second
bull Variants of AlphaZero also
defeated world champion
Go and Shogi systems
AlphaZero
outperformed chess
world-champion
Stockfish1 in just
4 hours (300k steps)
By not using this human data by not using human
features or human expertise in any fashion wersquove
actually removed the constraints of human knowledge
Itrsquos able to therefore create knowledge for itselfrdquo
David Silver Lead Programmer
DEEPMIND
1) Stockfish is the 2016 Top Chess Engine Championship (TCEC) world-champion
AlphaZero Chess
Performance
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
18
Need a Radiologist
Therersquos an App for That
Sources Kubota T ldquoStanford algorithm can diagnose pneumonia better than
radiologistsrdquo Stanford News Nov 2017 Health Care IT Advisor research and analysis
1) Stanford University Medical Center
2) NIH = National Institutes of Health
Case in Brief Stanford University1
CheXNet Algorithm
bull Algorithm developed by researchers at
Stanford University in the US can diagnose
14 common pathologies in chest x-rays
bull Trained on ChestX-ray14 a public data set
released by the NIH2 containing 112120
frontal-view chest x-ray images labelled with
the 14 possible pathologies
bull Outperforms previous models from the same
data set for all 14 conditions and diagnoses
pneumonia at an accuracy exceeding the
performance of four control radiologists
bull Produces heatmaps that visualize the areas
of an image most indicative of disease
Development of CheXNet
Sept 26 2017
ChestX-ray14 data set
released along with a
preliminary algorithm
that could detect the
labelled conditionsasymp One week later
CheXNet could
diagnose 10 of the
14 pathologies more
accurately than all
previous algorithmsasymp One month later
CheXNet surpassed
best published results
for all 14 pathologies
and outperformed
Stanford radiologists in
detecting pneumonialdquoI treat the model as a lazy resident Theyrsquore not dumb theyrsquoll
just use every trick they can find to avoid doing hard workrdquo
Dr Matthew Lungren RadiologistStanford University Medical Center
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
19
Lessons from Stanfordrsquos CheXNet
AI Is Effective on Narrow Tasks
Recent advances in AI have focused on
solving narrowly defined tasks Each
sub-problem (eg pneumonia) is a model
Handle ldquoNormalrdquo Explicitly
Identifying normal conditions versus
anomalies can help recognize when
human review may be needed
Explainable1 AI (XAI)
Models that can be interpreted at least
partially are easier to trust deploy and
govern Explainability is an area of rapid
progress in AI research
1) Also known as interpretable AI
Training Requires a Teacher or Referee
Machine learning needs to be told what
answers are correct or what actions produce
positive outcomes Well-curated data are
critical to the learning process
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
20
The AI Value Proposition
AI Excels at Decisions That Are Dull Detailed and Dear (Expensive)
Three Key Areas of Benefit
Dear
Provide speed capacity 24x7
availability and consistency
at a fraction of the cost
Although these are well
known benefits of automation
when you apply this to
cognitive and complex tasks
itrsquos a new ball game
Detailed
Evaluate a vastly broader
and deeper set of data to
improve decision quality
The complexity of health
care decision making is
growing due to new data
from genetics IoT devices
wearables better
interoperability and the
growing number of drugs
and treatments
Unlimited capacity
instant response
disruptive economics
Perform at super-human
levels exceeding experts
Serves as an assistant
(watches your back)
Dull
Free human decision makers
to focus on more challenging
ldquotop of licenserdquo activities
Reduce effort spent on
repetitive tasks process
large amounts of complex
information provide
continuous monitoring and
notifications and ensure
ldquostuffrdquo doesnrsquot ldquofall through
the cracksrdquo
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
21
Where to Start Where Wersquore Going
1) Narrow (Task Weak) AI focuses on narrowly defined problems such as predicting a patientrsquos risk of developing sepsis
2) General (Strong) AI attempts to match the flexibility and versatility of human intelligence
Narr
ow
(T
ask W
ea
k)
AI1
Clinical Decisions
Administrative Decisions
Ge
ne
ral (S
tro
ng
) A
I2
Full Diagnosis
and Treatment Plan
Clinical
Chatbots
Radiology
Interpretation
Acute
Clinical
Risk
Recruiting
Safety Risk Smart
Monitoring
Medication Dosing
Administrative
Chatbots
(Concierge)
Precision
Engagement
Precision
Medicine
Ambient
Documentation
Adaptive User
Interfaces
Worklist
Prioritization
Inevitable with Time
Todayrsquos Opportunity
(growing)
The Computer Will
See You Now
The Uber-ization of
Health Care
An AI Application Landscape
Capacity Staffing
Management
Population
Health Risk
The Self-Driving
HospitalInformation
Security
Targeted
Marketing
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
22
The Self-Driving Hospital
Combine Predictive Models with Front-Line Activation
Sources ldquoVirtual Air Traffic Control for Emergency Departmentsrdquo Qventus
httpsqventuscomcontentuploads201703QVE-ED-Solution-Briefpdf Health Care
IT Advisor research and analysis
1) EVS = Environmental services
2) LWBS = Left without being seen
3) LOS = Length of stay
Case in Brief
Mercy Hospital Fort Smith
bull 336-bed acute care hospital in Fort
Smith AR
bull Partnered with Qventus to improve
emergency department (ED) patient flow
and patient satisfaction
bull Analytics applies ML algorithms to
anticipate potential capacity shortfalls
process bottlenecks
bull Application sends real-time notifications
(nudges) to care teams via their mobile
phones to trigger specific interventions
bull Emergency department results
ndash LWBS2 rates dropped 30
ndash Door-to-doc time reduced by 20
ndash 24-minute reduction in average LOS3
ndash Annual case capacity increased by 6
See that a radiology order is delayed for a
patient within a few days of discharge and
notify radiology to prioritize the reading
Activate more EVS1 staff when high churn
of beds is anticipated
Send an early warning for difficult patient
placements at discharge
Prioritize inpatient transports depending
on expected bed needs
Sample ldquoDecision Recipesrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
23
Applicants Like You Liked This Job
Predictive Models Reduce Turnover Improve Engagement
Case in Brief MultiCare Health System
bull Non-profit system based in Tacoma WA with 16000 employees
bull Partnered with Arena a cloud-based HR solution to predict the
likelihood that a candidate will be retained in a role
bull Arenarsquos machine learning algorithms evaluate applicant
submissions digital interactions and externally sourced data on
the organization (eg employer review sites)
bull Achieved a 40 reduction in RN turnover at 180 days and a
28 reduction in overall turnover at 180 days
MultiCarersquos Hiring Process
Reduced recruiting
expenses
More predictable
staffing levels
More experienced
staff
Realized Benefits
Algorithms predict the likelihood the
applicant will be retained and
engaged in the target role and send
that information to the recruiter
Applicant completes
a 15-20 minute
assessment process
on Arenarsquos platform
Recruiters determine
whether or not to send
the candidate along to
hiring managers
Hiring managers use
Arenarsquos recommendation
as an optional factor in
the hiring decision
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
24
Accelerating Research
1) Yu KH et al ldquoPredicting non-small cell lung cancer prognosis by fully automated microscopic pathology image featuresrdquo Nature No 12474 (2016)
2) Rajwa B et al ldquoAutomated assessment of disease progression in acute myeloid leukemia by probabilistic analysis of flow cytometry datardquo IEEE Transactions on Biomedical Engineering 64 No 5 (2017) 1089-1098
3) Liu Y et al Detecting cancer metastases on gigapixel pathology images arXiv preprint arXiv1703 No 02442 (2017)
4) Rajpurkar P et al Cardiologist-level arrhythmia detection with convolutional neural networks arXiv preprint arXiv1707 No 01836 (2017)
5) Nielsen S et al A predictive model to identify Parkinson disease from administrative claims data Neurology 89 No 14 (2017) 1448-1456
6) Rajpurkar P et al CheXNet Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning arXiv preprint arXiv 1711 No 05225 (2017)
7) Poplin R et al Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning Nature Biomedical Engineering No 2 (2018)
8) ACC Machine learning techniques show promise for supporting medical decisions By quickly assessing risk new algorithms may help triage patients and inform clinical decisions ScienceDaily 2018
9) Oh J et al ldquoA Generalizable Data-Driven Approach to Predict Daily Risk of Clostridium difficile Infection at Two Large Academic Health Centersrdquo ICHE 39 No 4 (2018) 425-433
Nature1 ndash August 2016
ML algorithm can accurately
differentiate two types of lung
cancers and predict patient
survival time
IEEE2 ndash February 2017
ML model predicts relapse rates for
acute myelogenous leukemia with
90 accuracy
arXiv3 ndash March 2017
ML approach to imaging analytics
can identify metastasized breast
cancer with accuracy rates rivaling
those of pathologists
Neurology5 ndash Sept 2017
Predictive model to identify
Parkinsonrsquos disease from claims
data achieved 73-83 accuracy in
study results
arXiv4 ndash July 2017
ML algorithm can diagnose
heart arrhythmias using data
from wearables sensors with
cardiologist-level accuracy
arXiv6 ndash November 2017
Deep-learning algorithm
outperforms Stanford radiologists
at diagnosing pneumonia from
x-rays
American College of Cardiology8 ndash
Feb 2018
Model predicts heart attack
diagnosis among patients in the ED
with chest pain with ~90 accuracy
Nature7 ndash February 2018
ML algorithm can predict a
personrsquos cardiovascular risk
factors using eye scans
Infection Control Hospital
Epidemiology9 ndash April 2018
ML model predicts Clostridium
difficile infection (CDI) earlier
than current diagnostic methods
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
25
Driving the Decision Machine
1Acquire Data
bull Ensure all models have
a sponsor and regular
evaluation against goals
bull All models require
tune-ups as the
environment changes
Monitor Performance
and Revisit
42Train or Refine
the Model
bull The more the better
buthellip
bull High-quality well-
governed data pay
enormous dividends
bull Diverse cross-continuum
sources improve
predictions and
robustness of models
bull Decide what features
are important (eg
explainability)
bull Apply practical and
ethical constraints
bull Evaluate under
realistic conditions
(silent mode real-time
incomplete data)
bull Embed insights directly
into applications at key
decision points
bull Reengineer processes
as warranted
bull Consider the ldquo5 rightsrdquo
of decision support
3Incorporate Models
into Workflow
Hint The Analytics Is the Easy Part
Begin with
bull Explicit leadership agreement on outcome goals expected benefit mechanisms
bull Participation from empowered representatives of impacted process
bull External experiencemdashlearn from others
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
26
New Data Sources
Mainstream Systems Cover Only a Fraction of Predictive Power
Acquire Data
Sources Nash DB ldquoPopulation Health Why It Mattersrdquo Essentials in Population Health Childrenrsquos
Hospital Association Thomas Jefferson University October 2016 Health Care IT Advisor research and
analysis
1) RPM = Remote patient monitoring
EMR(s)
Lab Systems
Pharmacy Systems
Imaging
Medical Claims
Prescription Drug Claims
Billing and Supply Chain
Scheduling Systems
Medical
Care Data
10
Sensors RPM1
mHealth Apps
Patient-Reported
Outcomes
Genetics
Socioeconomic Data
Patient Questionnaires
Human
Biology
Data
20
Lifestyle
and
Behavior
Data
50
Social and
Environmental
Data
20
Data Sources by Influence on Health Outcomes
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
27
Big Data a Technology and a Philosophy
ldquoTherersquos Gold in Them Thar Hillsrdquo
Acquire Data
Sources Cloudera ldquoInova Translational Medicine Institute Partners with Cloudera to Advance Genome-Based Machine
Learning Initiatives and Save Livesrdquo PR Newswire June 2017 Health Care IT Advisor research and analysis
1) Originally coined by Doug Laney of the Meta Group
Assumes Value in the Data
Based on a premise that there are
valuable insights that can be distilled
from the mountains of data captured
by machines
Characterised by the 3 Vs1
bull Volume
bull Velocity
bull Variety
Places greater emphasis on
unstructured data
Free text images streams of sensor
data are all fair targets for analysis
Big data is the difference
between heart rates and
heart beats
Case in Brief Inova Health System
bull Inovarsquos Translational Medicine Institute
(ITMI) is assembling one of the worldrsquos
largest whole genome sequence
databases connected to patient
information in a health care system
bull Researchers use the Cloudera
machine learning platform to analyze
terabytes of clinical and genomic data
and identify genetic links to diseases
bull Insights are used to develop
personalized treatment plans for
patients in collaboration with the
treating physician
Resource Big Data in Health Care
Strengths challenges and pragmatic
guidelines for those considering the addition
of big data capabilities to their IT portfolio
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
30
How Machines Learn
Similar Methods Differing Goals
Build the Model
Source Advisory Board research and analysis
Input Data
bull The more the better
bull Labeled with key indicators
Adjust the Model
bull Automated trial and error
bull Cross-breeding successful
candidates
Repeat
Supervised Learning
Evaluate models against the ldquocorrectrdquo answers
provided with the data The learning process
focuses on minimizing errors
Unsupervised Learning
Evaluate models against an abstract algorithmic
goal Example Divide this population of diabetics
into a handful of subpopulations that have similar
future risk profiles
Reinforcement Learning
Evaluate models by giving a reward for
desirable outcomes and a penalty for negative
outcomes Models are trained to maximize their
total rewards over timeReady for Review
Evaluate Performance
bull Against what goal
bull Forces explicit
tradeoffs (eg cost
vs quality)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
32
Black Boxes Shades of Grey
Differing Views on the Importance of Explainability
Incorporate into Workflow
Mike Wall PharmD MBA
Chief Analytics Officer
The University of Chicago Medicine
The opaqueness of black-box models is
really scary When we canrsquot tell a
clinician why a patient wonrsquot do well on a
certain medication that can be a tough
thing to blindly trust from a clinical
perspective We want to reach a place
where we augment our practice
through AI rather than dictatingrdquo
Nigam Shah MBBS PhD
Associate Professor of Medicine
(Biomedical Informatics)
Stanford University
I am pushing to evaluate these models
via a similar set of mechanisms as RCT1
or AB testing We should evaluate
against the current status-quo and do a
beforeafter studyhellipWe have to convince
ourselves of the utility of these models
We should focus on utility before
explainabilityrdquo
Surgery is the
best option ()
Data
Black Box Model
1) RCT = Randomized controlled trial
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
33
Workflow Is Key to Adoption
Incorporate into Workflow
The Right Information
To the Right Person
In the Right Form
By the Right Channel
At the Right Time
Case in Brief Mayo Breast Oncology
Clinic Trials Recruitment
bull Up to 100 concurrent drug therapy clinical trials are
available to Mayo Clinic breast cancer patients making
it challenging to consistently identify candidates
bull New clinical trial matching technology developed in
collaboration with IBM Watson Health evaluates charts
to identify eligible patients
bull Launched in January 2016 as a point-of-care solution
for oncology providers initial adoption was low due to
time pressures and fell to zero by February
bull Rework by health systems engineers resulted in a July
2016 relaunch screening moved to occur in
advance of patient visits so clinicians know of trial
matches before opening a patientrsquos chart
bull New process enrolls significantly more patients in
clinical trials than the prior manual process success led
to additional coordinator funding
The Five Rights of
Clinical Decision Support
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
34
Health Carersquos Field Agents
Medical Devices Become Intelligent Partners
Incorporate into Workflow
1) EKG = Electrocardiogram
ME
DT
RO
NIC
MIN
IME
D 6
70G
Case in Brief Artificial Pancreas and Smart Infusion Pumpsmdash
Medtronic MiniMed Connect
bull SMARTGUARD mimics some functions of a healthy pancreas
measures and predicts glucose level drift and adjusts dosing
bull Insulin pump and continuous glucose monitoring can send data directly
to smartphone
bull Trials showed a 44 reduction in hypoglycemia and reduction in
average A1C values from 74 to 69
Case in Brief AliveCor KardiaMobile
bull Portable consumer-grade single-lead EKG1 retails for $100
bull Intended for self-capture of intermittent cardiac events such as atrial
fibrillation (AFIB)
bull Smartphone-based machine learning validates quality of trace
assesses possibility of AFIB classifies normal abnormal sinus rhythm
bull EKGs can be sent to a board-certified cardiologist for interpretation
for $19
Sources alivecorcom Medtroniccom Health Care IT Advisor research and
analysis
AliveC
orK
ard
iaM
obile
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
17
DeepMindrsquos AlphaZero
Novice to Superhuman in Just Four Hours
DeepMindrsquos AlphaZero Uses lsquoZero Human Knowledgersquo
Sources Vincent J ldquoDeepMindrsquos AI became a superhuman chess player in a few hours just for funrdquo The
Verge December 6 2017 Gershgorn D ldquoDeepMind wants to find the next miracle materialmdashexperts just
donrsquot know theyrsquoll pull it offrdquo Quartz October 25 2017 Health Care IT Advisor research and analysis
By not using this human data by not using
human features or human expertise in any
fashion wersquove actually rem
oved the constraints of human knowledge Itrsquos
able to therefore create knowledge for itselfrdquo
bull AI product of Google
subsidiary DeepMind
bull Provided no prior
knowledge of the
game given only basic
game rules and an
objective (win)
bull Learns through self-play
at an accelerated pace
(lsquoreinforced learningrsquo)
searches 80 thousand
positions per second
bull Variants of AlphaZero also
defeated world champion
Go and Shogi systems
AlphaZero
outperformed chess
world-champion
Stockfish1 in just
4 hours (300k steps)
By not using this human data by not using human
features or human expertise in any fashion wersquove
actually removed the constraints of human knowledge
Itrsquos able to therefore create knowledge for itselfrdquo
David Silver Lead Programmer
DEEPMIND
1) Stockfish is the 2016 Top Chess Engine Championship (TCEC) world-champion
AlphaZero Chess
Performance
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
18
Need a Radiologist
Therersquos an App for That
Sources Kubota T ldquoStanford algorithm can diagnose pneumonia better than
radiologistsrdquo Stanford News Nov 2017 Health Care IT Advisor research and analysis
1) Stanford University Medical Center
2) NIH = National Institutes of Health
Case in Brief Stanford University1
CheXNet Algorithm
bull Algorithm developed by researchers at
Stanford University in the US can diagnose
14 common pathologies in chest x-rays
bull Trained on ChestX-ray14 a public data set
released by the NIH2 containing 112120
frontal-view chest x-ray images labelled with
the 14 possible pathologies
bull Outperforms previous models from the same
data set for all 14 conditions and diagnoses
pneumonia at an accuracy exceeding the
performance of four control radiologists
bull Produces heatmaps that visualize the areas
of an image most indicative of disease
Development of CheXNet
Sept 26 2017
ChestX-ray14 data set
released along with a
preliminary algorithm
that could detect the
labelled conditionsasymp One week later
CheXNet could
diagnose 10 of the
14 pathologies more
accurately than all
previous algorithmsasymp One month later
CheXNet surpassed
best published results
for all 14 pathologies
and outperformed
Stanford radiologists in
detecting pneumonialdquoI treat the model as a lazy resident Theyrsquore not dumb theyrsquoll
just use every trick they can find to avoid doing hard workrdquo
Dr Matthew Lungren RadiologistStanford University Medical Center
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
19
Lessons from Stanfordrsquos CheXNet
AI Is Effective on Narrow Tasks
Recent advances in AI have focused on
solving narrowly defined tasks Each
sub-problem (eg pneumonia) is a model
Handle ldquoNormalrdquo Explicitly
Identifying normal conditions versus
anomalies can help recognize when
human review may be needed
Explainable1 AI (XAI)
Models that can be interpreted at least
partially are easier to trust deploy and
govern Explainability is an area of rapid
progress in AI research
1) Also known as interpretable AI
Training Requires a Teacher or Referee
Machine learning needs to be told what
answers are correct or what actions produce
positive outcomes Well-curated data are
critical to the learning process
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
20
The AI Value Proposition
AI Excels at Decisions That Are Dull Detailed and Dear (Expensive)
Three Key Areas of Benefit
Dear
Provide speed capacity 24x7
availability and consistency
at a fraction of the cost
Although these are well
known benefits of automation
when you apply this to
cognitive and complex tasks
itrsquos a new ball game
Detailed
Evaluate a vastly broader
and deeper set of data to
improve decision quality
The complexity of health
care decision making is
growing due to new data
from genetics IoT devices
wearables better
interoperability and the
growing number of drugs
and treatments
Unlimited capacity
instant response
disruptive economics
Perform at super-human
levels exceeding experts
Serves as an assistant
(watches your back)
Dull
Free human decision makers
to focus on more challenging
ldquotop of licenserdquo activities
Reduce effort spent on
repetitive tasks process
large amounts of complex
information provide
continuous monitoring and
notifications and ensure
ldquostuffrdquo doesnrsquot ldquofall through
the cracksrdquo
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
21
Where to Start Where Wersquore Going
1) Narrow (Task Weak) AI focuses on narrowly defined problems such as predicting a patientrsquos risk of developing sepsis
2) General (Strong) AI attempts to match the flexibility and versatility of human intelligence
Narr
ow
(T
ask W
ea
k)
AI1
Clinical Decisions
Administrative Decisions
Ge
ne
ral (S
tro
ng
) A
I2
Full Diagnosis
and Treatment Plan
Clinical
Chatbots
Radiology
Interpretation
Acute
Clinical
Risk
Recruiting
Safety Risk Smart
Monitoring
Medication Dosing
Administrative
Chatbots
(Concierge)
Precision
Engagement
Precision
Medicine
Ambient
Documentation
Adaptive User
Interfaces
Worklist
Prioritization
Inevitable with Time
Todayrsquos Opportunity
(growing)
The Computer Will
See You Now
The Uber-ization of
Health Care
An AI Application Landscape
Capacity Staffing
Management
Population
Health Risk
The Self-Driving
HospitalInformation
Security
Targeted
Marketing
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
22
The Self-Driving Hospital
Combine Predictive Models with Front-Line Activation
Sources ldquoVirtual Air Traffic Control for Emergency Departmentsrdquo Qventus
httpsqventuscomcontentuploads201703QVE-ED-Solution-Briefpdf Health Care
IT Advisor research and analysis
1) EVS = Environmental services
2) LWBS = Left without being seen
3) LOS = Length of stay
Case in Brief
Mercy Hospital Fort Smith
bull 336-bed acute care hospital in Fort
Smith AR
bull Partnered with Qventus to improve
emergency department (ED) patient flow
and patient satisfaction
bull Analytics applies ML algorithms to
anticipate potential capacity shortfalls
process bottlenecks
bull Application sends real-time notifications
(nudges) to care teams via their mobile
phones to trigger specific interventions
bull Emergency department results
ndash LWBS2 rates dropped 30
ndash Door-to-doc time reduced by 20
ndash 24-minute reduction in average LOS3
ndash Annual case capacity increased by 6
See that a radiology order is delayed for a
patient within a few days of discharge and
notify radiology to prioritize the reading
Activate more EVS1 staff when high churn
of beds is anticipated
Send an early warning for difficult patient
placements at discharge
Prioritize inpatient transports depending
on expected bed needs
Sample ldquoDecision Recipesrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
23
Applicants Like You Liked This Job
Predictive Models Reduce Turnover Improve Engagement
Case in Brief MultiCare Health System
bull Non-profit system based in Tacoma WA with 16000 employees
bull Partnered with Arena a cloud-based HR solution to predict the
likelihood that a candidate will be retained in a role
bull Arenarsquos machine learning algorithms evaluate applicant
submissions digital interactions and externally sourced data on
the organization (eg employer review sites)
bull Achieved a 40 reduction in RN turnover at 180 days and a
28 reduction in overall turnover at 180 days
MultiCarersquos Hiring Process
Reduced recruiting
expenses
More predictable
staffing levels
More experienced
staff
Realized Benefits
Algorithms predict the likelihood the
applicant will be retained and
engaged in the target role and send
that information to the recruiter
Applicant completes
a 15-20 minute
assessment process
on Arenarsquos platform
Recruiters determine
whether or not to send
the candidate along to
hiring managers
Hiring managers use
Arenarsquos recommendation
as an optional factor in
the hiring decision
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
24
Accelerating Research
1) Yu KH et al ldquoPredicting non-small cell lung cancer prognosis by fully automated microscopic pathology image featuresrdquo Nature No 12474 (2016)
2) Rajwa B et al ldquoAutomated assessment of disease progression in acute myeloid leukemia by probabilistic analysis of flow cytometry datardquo IEEE Transactions on Biomedical Engineering 64 No 5 (2017) 1089-1098
3) Liu Y et al Detecting cancer metastases on gigapixel pathology images arXiv preprint arXiv1703 No 02442 (2017)
4) Rajpurkar P et al Cardiologist-level arrhythmia detection with convolutional neural networks arXiv preprint arXiv1707 No 01836 (2017)
5) Nielsen S et al A predictive model to identify Parkinson disease from administrative claims data Neurology 89 No 14 (2017) 1448-1456
6) Rajpurkar P et al CheXNet Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning arXiv preprint arXiv 1711 No 05225 (2017)
7) Poplin R et al Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning Nature Biomedical Engineering No 2 (2018)
8) ACC Machine learning techniques show promise for supporting medical decisions By quickly assessing risk new algorithms may help triage patients and inform clinical decisions ScienceDaily 2018
9) Oh J et al ldquoA Generalizable Data-Driven Approach to Predict Daily Risk of Clostridium difficile Infection at Two Large Academic Health Centersrdquo ICHE 39 No 4 (2018) 425-433
Nature1 ndash August 2016
ML algorithm can accurately
differentiate two types of lung
cancers and predict patient
survival time
IEEE2 ndash February 2017
ML model predicts relapse rates for
acute myelogenous leukemia with
90 accuracy
arXiv3 ndash March 2017
ML approach to imaging analytics
can identify metastasized breast
cancer with accuracy rates rivaling
those of pathologists
Neurology5 ndash Sept 2017
Predictive model to identify
Parkinsonrsquos disease from claims
data achieved 73-83 accuracy in
study results
arXiv4 ndash July 2017
ML algorithm can diagnose
heart arrhythmias using data
from wearables sensors with
cardiologist-level accuracy
arXiv6 ndash November 2017
Deep-learning algorithm
outperforms Stanford radiologists
at diagnosing pneumonia from
x-rays
American College of Cardiology8 ndash
Feb 2018
Model predicts heart attack
diagnosis among patients in the ED
with chest pain with ~90 accuracy
Nature7 ndash February 2018
ML algorithm can predict a
personrsquos cardiovascular risk
factors using eye scans
Infection Control Hospital
Epidemiology9 ndash April 2018
ML model predicts Clostridium
difficile infection (CDI) earlier
than current diagnostic methods
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
25
Driving the Decision Machine
1Acquire Data
bull Ensure all models have
a sponsor and regular
evaluation against goals
bull All models require
tune-ups as the
environment changes
Monitor Performance
and Revisit
42Train or Refine
the Model
bull The more the better
buthellip
bull High-quality well-
governed data pay
enormous dividends
bull Diverse cross-continuum
sources improve
predictions and
robustness of models
bull Decide what features
are important (eg
explainability)
bull Apply practical and
ethical constraints
bull Evaluate under
realistic conditions
(silent mode real-time
incomplete data)
bull Embed insights directly
into applications at key
decision points
bull Reengineer processes
as warranted
bull Consider the ldquo5 rightsrdquo
of decision support
3Incorporate Models
into Workflow
Hint The Analytics Is the Easy Part
Begin with
bull Explicit leadership agreement on outcome goals expected benefit mechanisms
bull Participation from empowered representatives of impacted process
bull External experiencemdashlearn from others
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
26
New Data Sources
Mainstream Systems Cover Only a Fraction of Predictive Power
Acquire Data
Sources Nash DB ldquoPopulation Health Why It Mattersrdquo Essentials in Population Health Childrenrsquos
Hospital Association Thomas Jefferson University October 2016 Health Care IT Advisor research and
analysis
1) RPM = Remote patient monitoring
EMR(s)
Lab Systems
Pharmacy Systems
Imaging
Medical Claims
Prescription Drug Claims
Billing and Supply Chain
Scheduling Systems
Medical
Care Data
10
Sensors RPM1
mHealth Apps
Patient-Reported
Outcomes
Genetics
Socioeconomic Data
Patient Questionnaires
Human
Biology
Data
20
Lifestyle
and
Behavior
Data
50
Social and
Environmental
Data
20
Data Sources by Influence on Health Outcomes
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
27
Big Data a Technology and a Philosophy
ldquoTherersquos Gold in Them Thar Hillsrdquo
Acquire Data
Sources Cloudera ldquoInova Translational Medicine Institute Partners with Cloudera to Advance Genome-Based Machine
Learning Initiatives and Save Livesrdquo PR Newswire June 2017 Health Care IT Advisor research and analysis
1) Originally coined by Doug Laney of the Meta Group
Assumes Value in the Data
Based on a premise that there are
valuable insights that can be distilled
from the mountains of data captured
by machines
Characterised by the 3 Vs1
bull Volume
bull Velocity
bull Variety
Places greater emphasis on
unstructured data
Free text images streams of sensor
data are all fair targets for analysis
Big data is the difference
between heart rates and
heart beats
Case in Brief Inova Health System
bull Inovarsquos Translational Medicine Institute
(ITMI) is assembling one of the worldrsquos
largest whole genome sequence
databases connected to patient
information in a health care system
bull Researchers use the Cloudera
machine learning platform to analyze
terabytes of clinical and genomic data
and identify genetic links to diseases
bull Insights are used to develop
personalized treatment plans for
patients in collaboration with the
treating physician
Resource Big Data in Health Care
Strengths challenges and pragmatic
guidelines for those considering the addition
of big data capabilities to their IT portfolio
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
30
How Machines Learn
Similar Methods Differing Goals
Build the Model
Source Advisory Board research and analysis
Input Data
bull The more the better
bull Labeled with key indicators
Adjust the Model
bull Automated trial and error
bull Cross-breeding successful
candidates
Repeat
Supervised Learning
Evaluate models against the ldquocorrectrdquo answers
provided with the data The learning process
focuses on minimizing errors
Unsupervised Learning
Evaluate models against an abstract algorithmic
goal Example Divide this population of diabetics
into a handful of subpopulations that have similar
future risk profiles
Reinforcement Learning
Evaluate models by giving a reward for
desirable outcomes and a penalty for negative
outcomes Models are trained to maximize their
total rewards over timeReady for Review
Evaluate Performance
bull Against what goal
bull Forces explicit
tradeoffs (eg cost
vs quality)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
32
Black Boxes Shades of Grey
Differing Views on the Importance of Explainability
Incorporate into Workflow
Mike Wall PharmD MBA
Chief Analytics Officer
The University of Chicago Medicine
The opaqueness of black-box models is
really scary When we canrsquot tell a
clinician why a patient wonrsquot do well on a
certain medication that can be a tough
thing to blindly trust from a clinical
perspective We want to reach a place
where we augment our practice
through AI rather than dictatingrdquo
Nigam Shah MBBS PhD
Associate Professor of Medicine
(Biomedical Informatics)
Stanford University
I am pushing to evaluate these models
via a similar set of mechanisms as RCT1
or AB testing We should evaluate
against the current status-quo and do a
beforeafter studyhellipWe have to convince
ourselves of the utility of these models
We should focus on utility before
explainabilityrdquo
Surgery is the
best option ()
Data
Black Box Model
1) RCT = Randomized controlled trial
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
33
Workflow Is Key to Adoption
Incorporate into Workflow
The Right Information
To the Right Person
In the Right Form
By the Right Channel
At the Right Time
Case in Brief Mayo Breast Oncology
Clinic Trials Recruitment
bull Up to 100 concurrent drug therapy clinical trials are
available to Mayo Clinic breast cancer patients making
it challenging to consistently identify candidates
bull New clinical trial matching technology developed in
collaboration with IBM Watson Health evaluates charts
to identify eligible patients
bull Launched in January 2016 as a point-of-care solution
for oncology providers initial adoption was low due to
time pressures and fell to zero by February
bull Rework by health systems engineers resulted in a July
2016 relaunch screening moved to occur in
advance of patient visits so clinicians know of trial
matches before opening a patientrsquos chart
bull New process enrolls significantly more patients in
clinical trials than the prior manual process success led
to additional coordinator funding
The Five Rights of
Clinical Decision Support
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
34
Health Carersquos Field Agents
Medical Devices Become Intelligent Partners
Incorporate into Workflow
1) EKG = Electrocardiogram
ME
DT
RO
NIC
MIN
IME
D 6
70G
Case in Brief Artificial Pancreas and Smart Infusion Pumpsmdash
Medtronic MiniMed Connect
bull SMARTGUARD mimics some functions of a healthy pancreas
measures and predicts glucose level drift and adjusts dosing
bull Insulin pump and continuous glucose monitoring can send data directly
to smartphone
bull Trials showed a 44 reduction in hypoglycemia and reduction in
average A1C values from 74 to 69
Case in Brief AliveCor KardiaMobile
bull Portable consumer-grade single-lead EKG1 retails for $100
bull Intended for self-capture of intermittent cardiac events such as atrial
fibrillation (AFIB)
bull Smartphone-based machine learning validates quality of trace
assesses possibility of AFIB classifies normal abnormal sinus rhythm
bull EKGs can be sent to a board-certified cardiologist for interpretation
for $19
Sources alivecorcom Medtroniccom Health Care IT Advisor research and
analysis
AliveC
orK
ard
iaM
obile
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
18
Need a Radiologist
Therersquos an App for That
Sources Kubota T ldquoStanford algorithm can diagnose pneumonia better than
radiologistsrdquo Stanford News Nov 2017 Health Care IT Advisor research and analysis
1) Stanford University Medical Center
2) NIH = National Institutes of Health
Case in Brief Stanford University1
CheXNet Algorithm
bull Algorithm developed by researchers at
Stanford University in the US can diagnose
14 common pathologies in chest x-rays
bull Trained on ChestX-ray14 a public data set
released by the NIH2 containing 112120
frontal-view chest x-ray images labelled with
the 14 possible pathologies
bull Outperforms previous models from the same
data set for all 14 conditions and diagnoses
pneumonia at an accuracy exceeding the
performance of four control radiologists
bull Produces heatmaps that visualize the areas
of an image most indicative of disease
Development of CheXNet
Sept 26 2017
ChestX-ray14 data set
released along with a
preliminary algorithm
that could detect the
labelled conditionsasymp One week later
CheXNet could
diagnose 10 of the
14 pathologies more
accurately than all
previous algorithmsasymp One month later
CheXNet surpassed
best published results
for all 14 pathologies
and outperformed
Stanford radiologists in
detecting pneumonialdquoI treat the model as a lazy resident Theyrsquore not dumb theyrsquoll
just use every trick they can find to avoid doing hard workrdquo
Dr Matthew Lungren RadiologistStanford University Medical Center
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
19
Lessons from Stanfordrsquos CheXNet
AI Is Effective on Narrow Tasks
Recent advances in AI have focused on
solving narrowly defined tasks Each
sub-problem (eg pneumonia) is a model
Handle ldquoNormalrdquo Explicitly
Identifying normal conditions versus
anomalies can help recognize when
human review may be needed
Explainable1 AI (XAI)
Models that can be interpreted at least
partially are easier to trust deploy and
govern Explainability is an area of rapid
progress in AI research
1) Also known as interpretable AI
Training Requires a Teacher or Referee
Machine learning needs to be told what
answers are correct or what actions produce
positive outcomes Well-curated data are
critical to the learning process
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
20
The AI Value Proposition
AI Excels at Decisions That Are Dull Detailed and Dear (Expensive)
Three Key Areas of Benefit
Dear
Provide speed capacity 24x7
availability and consistency
at a fraction of the cost
Although these are well
known benefits of automation
when you apply this to
cognitive and complex tasks
itrsquos a new ball game
Detailed
Evaluate a vastly broader
and deeper set of data to
improve decision quality
The complexity of health
care decision making is
growing due to new data
from genetics IoT devices
wearables better
interoperability and the
growing number of drugs
and treatments
Unlimited capacity
instant response
disruptive economics
Perform at super-human
levels exceeding experts
Serves as an assistant
(watches your back)
Dull
Free human decision makers
to focus on more challenging
ldquotop of licenserdquo activities
Reduce effort spent on
repetitive tasks process
large amounts of complex
information provide
continuous monitoring and
notifications and ensure
ldquostuffrdquo doesnrsquot ldquofall through
the cracksrdquo
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
21
Where to Start Where Wersquore Going
1) Narrow (Task Weak) AI focuses on narrowly defined problems such as predicting a patientrsquos risk of developing sepsis
2) General (Strong) AI attempts to match the flexibility and versatility of human intelligence
Narr
ow
(T
ask W
ea
k)
AI1
Clinical Decisions
Administrative Decisions
Ge
ne
ral (S
tro
ng
) A
I2
Full Diagnosis
and Treatment Plan
Clinical
Chatbots
Radiology
Interpretation
Acute
Clinical
Risk
Recruiting
Safety Risk Smart
Monitoring
Medication Dosing
Administrative
Chatbots
(Concierge)
Precision
Engagement
Precision
Medicine
Ambient
Documentation
Adaptive User
Interfaces
Worklist
Prioritization
Inevitable with Time
Todayrsquos Opportunity
(growing)
The Computer Will
See You Now
The Uber-ization of
Health Care
An AI Application Landscape
Capacity Staffing
Management
Population
Health Risk
The Self-Driving
HospitalInformation
Security
Targeted
Marketing
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
22
The Self-Driving Hospital
Combine Predictive Models with Front-Line Activation
Sources ldquoVirtual Air Traffic Control for Emergency Departmentsrdquo Qventus
httpsqventuscomcontentuploads201703QVE-ED-Solution-Briefpdf Health Care
IT Advisor research and analysis
1) EVS = Environmental services
2) LWBS = Left without being seen
3) LOS = Length of stay
Case in Brief
Mercy Hospital Fort Smith
bull 336-bed acute care hospital in Fort
Smith AR
bull Partnered with Qventus to improve
emergency department (ED) patient flow
and patient satisfaction
bull Analytics applies ML algorithms to
anticipate potential capacity shortfalls
process bottlenecks
bull Application sends real-time notifications
(nudges) to care teams via their mobile
phones to trigger specific interventions
bull Emergency department results
ndash LWBS2 rates dropped 30
ndash Door-to-doc time reduced by 20
ndash 24-minute reduction in average LOS3
ndash Annual case capacity increased by 6
See that a radiology order is delayed for a
patient within a few days of discharge and
notify radiology to prioritize the reading
Activate more EVS1 staff when high churn
of beds is anticipated
Send an early warning for difficult patient
placements at discharge
Prioritize inpatient transports depending
on expected bed needs
Sample ldquoDecision Recipesrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
23
Applicants Like You Liked This Job
Predictive Models Reduce Turnover Improve Engagement
Case in Brief MultiCare Health System
bull Non-profit system based in Tacoma WA with 16000 employees
bull Partnered with Arena a cloud-based HR solution to predict the
likelihood that a candidate will be retained in a role
bull Arenarsquos machine learning algorithms evaluate applicant
submissions digital interactions and externally sourced data on
the organization (eg employer review sites)
bull Achieved a 40 reduction in RN turnover at 180 days and a
28 reduction in overall turnover at 180 days
MultiCarersquos Hiring Process
Reduced recruiting
expenses
More predictable
staffing levels
More experienced
staff
Realized Benefits
Algorithms predict the likelihood the
applicant will be retained and
engaged in the target role and send
that information to the recruiter
Applicant completes
a 15-20 minute
assessment process
on Arenarsquos platform
Recruiters determine
whether or not to send
the candidate along to
hiring managers
Hiring managers use
Arenarsquos recommendation
as an optional factor in
the hiring decision
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
24
Accelerating Research
1) Yu KH et al ldquoPredicting non-small cell lung cancer prognosis by fully automated microscopic pathology image featuresrdquo Nature No 12474 (2016)
2) Rajwa B et al ldquoAutomated assessment of disease progression in acute myeloid leukemia by probabilistic analysis of flow cytometry datardquo IEEE Transactions on Biomedical Engineering 64 No 5 (2017) 1089-1098
3) Liu Y et al Detecting cancer metastases on gigapixel pathology images arXiv preprint arXiv1703 No 02442 (2017)
4) Rajpurkar P et al Cardiologist-level arrhythmia detection with convolutional neural networks arXiv preprint arXiv1707 No 01836 (2017)
5) Nielsen S et al A predictive model to identify Parkinson disease from administrative claims data Neurology 89 No 14 (2017) 1448-1456
6) Rajpurkar P et al CheXNet Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning arXiv preprint arXiv 1711 No 05225 (2017)
7) Poplin R et al Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning Nature Biomedical Engineering No 2 (2018)
8) ACC Machine learning techniques show promise for supporting medical decisions By quickly assessing risk new algorithms may help triage patients and inform clinical decisions ScienceDaily 2018
9) Oh J et al ldquoA Generalizable Data-Driven Approach to Predict Daily Risk of Clostridium difficile Infection at Two Large Academic Health Centersrdquo ICHE 39 No 4 (2018) 425-433
Nature1 ndash August 2016
ML algorithm can accurately
differentiate two types of lung
cancers and predict patient
survival time
IEEE2 ndash February 2017
ML model predicts relapse rates for
acute myelogenous leukemia with
90 accuracy
arXiv3 ndash March 2017
ML approach to imaging analytics
can identify metastasized breast
cancer with accuracy rates rivaling
those of pathologists
Neurology5 ndash Sept 2017
Predictive model to identify
Parkinsonrsquos disease from claims
data achieved 73-83 accuracy in
study results
arXiv4 ndash July 2017
ML algorithm can diagnose
heart arrhythmias using data
from wearables sensors with
cardiologist-level accuracy
arXiv6 ndash November 2017
Deep-learning algorithm
outperforms Stanford radiologists
at diagnosing pneumonia from
x-rays
American College of Cardiology8 ndash
Feb 2018
Model predicts heart attack
diagnosis among patients in the ED
with chest pain with ~90 accuracy
Nature7 ndash February 2018
ML algorithm can predict a
personrsquos cardiovascular risk
factors using eye scans
Infection Control Hospital
Epidemiology9 ndash April 2018
ML model predicts Clostridium
difficile infection (CDI) earlier
than current diagnostic methods
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
25
Driving the Decision Machine
1Acquire Data
bull Ensure all models have
a sponsor and regular
evaluation against goals
bull All models require
tune-ups as the
environment changes
Monitor Performance
and Revisit
42Train or Refine
the Model
bull The more the better
buthellip
bull High-quality well-
governed data pay
enormous dividends
bull Diverse cross-continuum
sources improve
predictions and
robustness of models
bull Decide what features
are important (eg
explainability)
bull Apply practical and
ethical constraints
bull Evaluate under
realistic conditions
(silent mode real-time
incomplete data)
bull Embed insights directly
into applications at key
decision points
bull Reengineer processes
as warranted
bull Consider the ldquo5 rightsrdquo
of decision support
3Incorporate Models
into Workflow
Hint The Analytics Is the Easy Part
Begin with
bull Explicit leadership agreement on outcome goals expected benefit mechanisms
bull Participation from empowered representatives of impacted process
bull External experiencemdashlearn from others
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
26
New Data Sources
Mainstream Systems Cover Only a Fraction of Predictive Power
Acquire Data
Sources Nash DB ldquoPopulation Health Why It Mattersrdquo Essentials in Population Health Childrenrsquos
Hospital Association Thomas Jefferson University October 2016 Health Care IT Advisor research and
analysis
1) RPM = Remote patient monitoring
EMR(s)
Lab Systems
Pharmacy Systems
Imaging
Medical Claims
Prescription Drug Claims
Billing and Supply Chain
Scheduling Systems
Medical
Care Data
10
Sensors RPM1
mHealth Apps
Patient-Reported
Outcomes
Genetics
Socioeconomic Data
Patient Questionnaires
Human
Biology
Data
20
Lifestyle
and
Behavior
Data
50
Social and
Environmental
Data
20
Data Sources by Influence on Health Outcomes
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
27
Big Data a Technology and a Philosophy
ldquoTherersquos Gold in Them Thar Hillsrdquo
Acquire Data
Sources Cloudera ldquoInova Translational Medicine Institute Partners with Cloudera to Advance Genome-Based Machine
Learning Initiatives and Save Livesrdquo PR Newswire June 2017 Health Care IT Advisor research and analysis
1) Originally coined by Doug Laney of the Meta Group
Assumes Value in the Data
Based on a premise that there are
valuable insights that can be distilled
from the mountains of data captured
by machines
Characterised by the 3 Vs1
bull Volume
bull Velocity
bull Variety
Places greater emphasis on
unstructured data
Free text images streams of sensor
data are all fair targets for analysis
Big data is the difference
between heart rates and
heart beats
Case in Brief Inova Health System
bull Inovarsquos Translational Medicine Institute
(ITMI) is assembling one of the worldrsquos
largest whole genome sequence
databases connected to patient
information in a health care system
bull Researchers use the Cloudera
machine learning platform to analyze
terabytes of clinical and genomic data
and identify genetic links to diseases
bull Insights are used to develop
personalized treatment plans for
patients in collaboration with the
treating physician
Resource Big Data in Health Care
Strengths challenges and pragmatic
guidelines for those considering the addition
of big data capabilities to their IT portfolio
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
30
How Machines Learn
Similar Methods Differing Goals
Build the Model
Source Advisory Board research and analysis
Input Data
bull The more the better
bull Labeled with key indicators
Adjust the Model
bull Automated trial and error
bull Cross-breeding successful
candidates
Repeat
Supervised Learning
Evaluate models against the ldquocorrectrdquo answers
provided with the data The learning process
focuses on minimizing errors
Unsupervised Learning
Evaluate models against an abstract algorithmic
goal Example Divide this population of diabetics
into a handful of subpopulations that have similar
future risk profiles
Reinforcement Learning
Evaluate models by giving a reward for
desirable outcomes and a penalty for negative
outcomes Models are trained to maximize their
total rewards over timeReady for Review
Evaluate Performance
bull Against what goal
bull Forces explicit
tradeoffs (eg cost
vs quality)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
32
Black Boxes Shades of Grey
Differing Views on the Importance of Explainability
Incorporate into Workflow
Mike Wall PharmD MBA
Chief Analytics Officer
The University of Chicago Medicine
The opaqueness of black-box models is
really scary When we canrsquot tell a
clinician why a patient wonrsquot do well on a
certain medication that can be a tough
thing to blindly trust from a clinical
perspective We want to reach a place
where we augment our practice
through AI rather than dictatingrdquo
Nigam Shah MBBS PhD
Associate Professor of Medicine
(Biomedical Informatics)
Stanford University
I am pushing to evaluate these models
via a similar set of mechanisms as RCT1
or AB testing We should evaluate
against the current status-quo and do a
beforeafter studyhellipWe have to convince
ourselves of the utility of these models
We should focus on utility before
explainabilityrdquo
Surgery is the
best option ()
Data
Black Box Model
1) RCT = Randomized controlled trial
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
33
Workflow Is Key to Adoption
Incorporate into Workflow
The Right Information
To the Right Person
In the Right Form
By the Right Channel
At the Right Time
Case in Brief Mayo Breast Oncology
Clinic Trials Recruitment
bull Up to 100 concurrent drug therapy clinical trials are
available to Mayo Clinic breast cancer patients making
it challenging to consistently identify candidates
bull New clinical trial matching technology developed in
collaboration with IBM Watson Health evaluates charts
to identify eligible patients
bull Launched in January 2016 as a point-of-care solution
for oncology providers initial adoption was low due to
time pressures and fell to zero by February
bull Rework by health systems engineers resulted in a July
2016 relaunch screening moved to occur in
advance of patient visits so clinicians know of trial
matches before opening a patientrsquos chart
bull New process enrolls significantly more patients in
clinical trials than the prior manual process success led
to additional coordinator funding
The Five Rights of
Clinical Decision Support
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
34
Health Carersquos Field Agents
Medical Devices Become Intelligent Partners
Incorporate into Workflow
1) EKG = Electrocardiogram
ME
DT
RO
NIC
MIN
IME
D 6
70G
Case in Brief Artificial Pancreas and Smart Infusion Pumpsmdash
Medtronic MiniMed Connect
bull SMARTGUARD mimics some functions of a healthy pancreas
measures and predicts glucose level drift and adjusts dosing
bull Insulin pump and continuous glucose monitoring can send data directly
to smartphone
bull Trials showed a 44 reduction in hypoglycemia and reduction in
average A1C values from 74 to 69
Case in Brief AliveCor KardiaMobile
bull Portable consumer-grade single-lead EKG1 retails for $100
bull Intended for self-capture of intermittent cardiac events such as atrial
fibrillation (AFIB)
bull Smartphone-based machine learning validates quality of trace
assesses possibility of AFIB classifies normal abnormal sinus rhythm
bull EKGs can be sent to a board-certified cardiologist for interpretation
for $19
Sources alivecorcom Medtroniccom Health Care IT Advisor research and
analysis
AliveC
orK
ard
iaM
obile
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
19
Lessons from Stanfordrsquos CheXNet
AI Is Effective on Narrow Tasks
Recent advances in AI have focused on
solving narrowly defined tasks Each
sub-problem (eg pneumonia) is a model
Handle ldquoNormalrdquo Explicitly
Identifying normal conditions versus
anomalies can help recognize when
human review may be needed
Explainable1 AI (XAI)
Models that can be interpreted at least
partially are easier to trust deploy and
govern Explainability is an area of rapid
progress in AI research
1) Also known as interpretable AI
Training Requires a Teacher or Referee
Machine learning needs to be told what
answers are correct or what actions produce
positive outcomes Well-curated data are
critical to the learning process
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
20
The AI Value Proposition
AI Excels at Decisions That Are Dull Detailed and Dear (Expensive)
Three Key Areas of Benefit
Dear
Provide speed capacity 24x7
availability and consistency
at a fraction of the cost
Although these are well
known benefits of automation
when you apply this to
cognitive and complex tasks
itrsquos a new ball game
Detailed
Evaluate a vastly broader
and deeper set of data to
improve decision quality
The complexity of health
care decision making is
growing due to new data
from genetics IoT devices
wearables better
interoperability and the
growing number of drugs
and treatments
Unlimited capacity
instant response
disruptive economics
Perform at super-human
levels exceeding experts
Serves as an assistant
(watches your back)
Dull
Free human decision makers
to focus on more challenging
ldquotop of licenserdquo activities
Reduce effort spent on
repetitive tasks process
large amounts of complex
information provide
continuous monitoring and
notifications and ensure
ldquostuffrdquo doesnrsquot ldquofall through
the cracksrdquo
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
21
Where to Start Where Wersquore Going
1) Narrow (Task Weak) AI focuses on narrowly defined problems such as predicting a patientrsquos risk of developing sepsis
2) General (Strong) AI attempts to match the flexibility and versatility of human intelligence
Narr
ow
(T
ask W
ea
k)
AI1
Clinical Decisions
Administrative Decisions
Ge
ne
ral (S
tro
ng
) A
I2
Full Diagnosis
and Treatment Plan
Clinical
Chatbots
Radiology
Interpretation
Acute
Clinical
Risk
Recruiting
Safety Risk Smart
Monitoring
Medication Dosing
Administrative
Chatbots
(Concierge)
Precision
Engagement
Precision
Medicine
Ambient
Documentation
Adaptive User
Interfaces
Worklist
Prioritization
Inevitable with Time
Todayrsquos Opportunity
(growing)
The Computer Will
See You Now
The Uber-ization of
Health Care
An AI Application Landscape
Capacity Staffing
Management
Population
Health Risk
The Self-Driving
HospitalInformation
Security
Targeted
Marketing
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
22
The Self-Driving Hospital
Combine Predictive Models with Front-Line Activation
Sources ldquoVirtual Air Traffic Control for Emergency Departmentsrdquo Qventus
httpsqventuscomcontentuploads201703QVE-ED-Solution-Briefpdf Health Care
IT Advisor research and analysis
1) EVS = Environmental services
2) LWBS = Left without being seen
3) LOS = Length of stay
Case in Brief
Mercy Hospital Fort Smith
bull 336-bed acute care hospital in Fort
Smith AR
bull Partnered with Qventus to improve
emergency department (ED) patient flow
and patient satisfaction
bull Analytics applies ML algorithms to
anticipate potential capacity shortfalls
process bottlenecks
bull Application sends real-time notifications
(nudges) to care teams via their mobile
phones to trigger specific interventions
bull Emergency department results
ndash LWBS2 rates dropped 30
ndash Door-to-doc time reduced by 20
ndash 24-minute reduction in average LOS3
ndash Annual case capacity increased by 6
See that a radiology order is delayed for a
patient within a few days of discharge and
notify radiology to prioritize the reading
Activate more EVS1 staff when high churn
of beds is anticipated
Send an early warning for difficult patient
placements at discharge
Prioritize inpatient transports depending
on expected bed needs
Sample ldquoDecision Recipesrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
23
Applicants Like You Liked This Job
Predictive Models Reduce Turnover Improve Engagement
Case in Brief MultiCare Health System
bull Non-profit system based in Tacoma WA with 16000 employees
bull Partnered with Arena a cloud-based HR solution to predict the
likelihood that a candidate will be retained in a role
bull Arenarsquos machine learning algorithms evaluate applicant
submissions digital interactions and externally sourced data on
the organization (eg employer review sites)
bull Achieved a 40 reduction in RN turnover at 180 days and a
28 reduction in overall turnover at 180 days
MultiCarersquos Hiring Process
Reduced recruiting
expenses
More predictable
staffing levels
More experienced
staff
Realized Benefits
Algorithms predict the likelihood the
applicant will be retained and
engaged in the target role and send
that information to the recruiter
Applicant completes
a 15-20 minute
assessment process
on Arenarsquos platform
Recruiters determine
whether or not to send
the candidate along to
hiring managers
Hiring managers use
Arenarsquos recommendation
as an optional factor in
the hiring decision
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
24
Accelerating Research
1) Yu KH et al ldquoPredicting non-small cell lung cancer prognosis by fully automated microscopic pathology image featuresrdquo Nature No 12474 (2016)
2) Rajwa B et al ldquoAutomated assessment of disease progression in acute myeloid leukemia by probabilistic analysis of flow cytometry datardquo IEEE Transactions on Biomedical Engineering 64 No 5 (2017) 1089-1098
3) Liu Y et al Detecting cancer metastases on gigapixel pathology images arXiv preprint arXiv1703 No 02442 (2017)
4) Rajpurkar P et al Cardiologist-level arrhythmia detection with convolutional neural networks arXiv preprint arXiv1707 No 01836 (2017)
5) Nielsen S et al A predictive model to identify Parkinson disease from administrative claims data Neurology 89 No 14 (2017) 1448-1456
6) Rajpurkar P et al CheXNet Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning arXiv preprint arXiv 1711 No 05225 (2017)
7) Poplin R et al Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning Nature Biomedical Engineering No 2 (2018)
8) ACC Machine learning techniques show promise for supporting medical decisions By quickly assessing risk new algorithms may help triage patients and inform clinical decisions ScienceDaily 2018
9) Oh J et al ldquoA Generalizable Data-Driven Approach to Predict Daily Risk of Clostridium difficile Infection at Two Large Academic Health Centersrdquo ICHE 39 No 4 (2018) 425-433
Nature1 ndash August 2016
ML algorithm can accurately
differentiate two types of lung
cancers and predict patient
survival time
IEEE2 ndash February 2017
ML model predicts relapse rates for
acute myelogenous leukemia with
90 accuracy
arXiv3 ndash March 2017
ML approach to imaging analytics
can identify metastasized breast
cancer with accuracy rates rivaling
those of pathologists
Neurology5 ndash Sept 2017
Predictive model to identify
Parkinsonrsquos disease from claims
data achieved 73-83 accuracy in
study results
arXiv4 ndash July 2017
ML algorithm can diagnose
heart arrhythmias using data
from wearables sensors with
cardiologist-level accuracy
arXiv6 ndash November 2017
Deep-learning algorithm
outperforms Stanford radiologists
at diagnosing pneumonia from
x-rays
American College of Cardiology8 ndash
Feb 2018
Model predicts heart attack
diagnosis among patients in the ED
with chest pain with ~90 accuracy
Nature7 ndash February 2018
ML algorithm can predict a
personrsquos cardiovascular risk
factors using eye scans
Infection Control Hospital
Epidemiology9 ndash April 2018
ML model predicts Clostridium
difficile infection (CDI) earlier
than current diagnostic methods
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
25
Driving the Decision Machine
1Acquire Data
bull Ensure all models have
a sponsor and regular
evaluation against goals
bull All models require
tune-ups as the
environment changes
Monitor Performance
and Revisit
42Train or Refine
the Model
bull The more the better
buthellip
bull High-quality well-
governed data pay
enormous dividends
bull Diverse cross-continuum
sources improve
predictions and
robustness of models
bull Decide what features
are important (eg
explainability)
bull Apply practical and
ethical constraints
bull Evaluate under
realistic conditions
(silent mode real-time
incomplete data)
bull Embed insights directly
into applications at key
decision points
bull Reengineer processes
as warranted
bull Consider the ldquo5 rightsrdquo
of decision support
3Incorporate Models
into Workflow
Hint The Analytics Is the Easy Part
Begin with
bull Explicit leadership agreement on outcome goals expected benefit mechanisms
bull Participation from empowered representatives of impacted process
bull External experiencemdashlearn from others
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
26
New Data Sources
Mainstream Systems Cover Only a Fraction of Predictive Power
Acquire Data
Sources Nash DB ldquoPopulation Health Why It Mattersrdquo Essentials in Population Health Childrenrsquos
Hospital Association Thomas Jefferson University October 2016 Health Care IT Advisor research and
analysis
1) RPM = Remote patient monitoring
EMR(s)
Lab Systems
Pharmacy Systems
Imaging
Medical Claims
Prescription Drug Claims
Billing and Supply Chain
Scheduling Systems
Medical
Care Data
10
Sensors RPM1
mHealth Apps
Patient-Reported
Outcomes
Genetics
Socioeconomic Data
Patient Questionnaires
Human
Biology
Data
20
Lifestyle
and
Behavior
Data
50
Social and
Environmental
Data
20
Data Sources by Influence on Health Outcomes
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
27
Big Data a Technology and a Philosophy
ldquoTherersquos Gold in Them Thar Hillsrdquo
Acquire Data
Sources Cloudera ldquoInova Translational Medicine Institute Partners with Cloudera to Advance Genome-Based Machine
Learning Initiatives and Save Livesrdquo PR Newswire June 2017 Health Care IT Advisor research and analysis
1) Originally coined by Doug Laney of the Meta Group
Assumes Value in the Data
Based on a premise that there are
valuable insights that can be distilled
from the mountains of data captured
by machines
Characterised by the 3 Vs1
bull Volume
bull Velocity
bull Variety
Places greater emphasis on
unstructured data
Free text images streams of sensor
data are all fair targets for analysis
Big data is the difference
between heart rates and
heart beats
Case in Brief Inova Health System
bull Inovarsquos Translational Medicine Institute
(ITMI) is assembling one of the worldrsquos
largest whole genome sequence
databases connected to patient
information in a health care system
bull Researchers use the Cloudera
machine learning platform to analyze
terabytes of clinical and genomic data
and identify genetic links to diseases
bull Insights are used to develop
personalized treatment plans for
patients in collaboration with the
treating physician
Resource Big Data in Health Care
Strengths challenges and pragmatic
guidelines for those considering the addition
of big data capabilities to their IT portfolio
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
30
How Machines Learn
Similar Methods Differing Goals
Build the Model
Source Advisory Board research and analysis
Input Data
bull The more the better
bull Labeled with key indicators
Adjust the Model
bull Automated trial and error
bull Cross-breeding successful
candidates
Repeat
Supervised Learning
Evaluate models against the ldquocorrectrdquo answers
provided with the data The learning process
focuses on minimizing errors
Unsupervised Learning
Evaluate models against an abstract algorithmic
goal Example Divide this population of diabetics
into a handful of subpopulations that have similar
future risk profiles
Reinforcement Learning
Evaluate models by giving a reward for
desirable outcomes and a penalty for negative
outcomes Models are trained to maximize their
total rewards over timeReady for Review
Evaluate Performance
bull Against what goal
bull Forces explicit
tradeoffs (eg cost
vs quality)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
32
Black Boxes Shades of Grey
Differing Views on the Importance of Explainability
Incorporate into Workflow
Mike Wall PharmD MBA
Chief Analytics Officer
The University of Chicago Medicine
The opaqueness of black-box models is
really scary When we canrsquot tell a
clinician why a patient wonrsquot do well on a
certain medication that can be a tough
thing to blindly trust from a clinical
perspective We want to reach a place
where we augment our practice
through AI rather than dictatingrdquo
Nigam Shah MBBS PhD
Associate Professor of Medicine
(Biomedical Informatics)
Stanford University
I am pushing to evaluate these models
via a similar set of mechanisms as RCT1
or AB testing We should evaluate
against the current status-quo and do a
beforeafter studyhellipWe have to convince
ourselves of the utility of these models
We should focus on utility before
explainabilityrdquo
Surgery is the
best option ()
Data
Black Box Model
1) RCT = Randomized controlled trial
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
33
Workflow Is Key to Adoption
Incorporate into Workflow
The Right Information
To the Right Person
In the Right Form
By the Right Channel
At the Right Time
Case in Brief Mayo Breast Oncology
Clinic Trials Recruitment
bull Up to 100 concurrent drug therapy clinical trials are
available to Mayo Clinic breast cancer patients making
it challenging to consistently identify candidates
bull New clinical trial matching technology developed in
collaboration with IBM Watson Health evaluates charts
to identify eligible patients
bull Launched in January 2016 as a point-of-care solution
for oncology providers initial adoption was low due to
time pressures and fell to zero by February
bull Rework by health systems engineers resulted in a July
2016 relaunch screening moved to occur in
advance of patient visits so clinicians know of trial
matches before opening a patientrsquos chart
bull New process enrolls significantly more patients in
clinical trials than the prior manual process success led
to additional coordinator funding
The Five Rights of
Clinical Decision Support
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
34
Health Carersquos Field Agents
Medical Devices Become Intelligent Partners
Incorporate into Workflow
1) EKG = Electrocardiogram
ME
DT
RO
NIC
MIN
IME
D 6
70G
Case in Brief Artificial Pancreas and Smart Infusion Pumpsmdash
Medtronic MiniMed Connect
bull SMARTGUARD mimics some functions of a healthy pancreas
measures and predicts glucose level drift and adjusts dosing
bull Insulin pump and continuous glucose monitoring can send data directly
to smartphone
bull Trials showed a 44 reduction in hypoglycemia and reduction in
average A1C values from 74 to 69
Case in Brief AliveCor KardiaMobile
bull Portable consumer-grade single-lead EKG1 retails for $100
bull Intended for self-capture of intermittent cardiac events such as atrial
fibrillation (AFIB)
bull Smartphone-based machine learning validates quality of trace
assesses possibility of AFIB classifies normal abnormal sinus rhythm
bull EKGs can be sent to a board-certified cardiologist for interpretation
for $19
Sources alivecorcom Medtroniccom Health Care IT Advisor research and
analysis
AliveC
orK
ard
iaM
obile
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
20
The AI Value Proposition
AI Excels at Decisions That Are Dull Detailed and Dear (Expensive)
Three Key Areas of Benefit
Dear
Provide speed capacity 24x7
availability and consistency
at a fraction of the cost
Although these are well
known benefits of automation
when you apply this to
cognitive and complex tasks
itrsquos a new ball game
Detailed
Evaluate a vastly broader
and deeper set of data to
improve decision quality
The complexity of health
care decision making is
growing due to new data
from genetics IoT devices
wearables better
interoperability and the
growing number of drugs
and treatments
Unlimited capacity
instant response
disruptive economics
Perform at super-human
levels exceeding experts
Serves as an assistant
(watches your back)
Dull
Free human decision makers
to focus on more challenging
ldquotop of licenserdquo activities
Reduce effort spent on
repetitive tasks process
large amounts of complex
information provide
continuous monitoring and
notifications and ensure
ldquostuffrdquo doesnrsquot ldquofall through
the cracksrdquo
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
21
Where to Start Where Wersquore Going
1) Narrow (Task Weak) AI focuses on narrowly defined problems such as predicting a patientrsquos risk of developing sepsis
2) General (Strong) AI attempts to match the flexibility and versatility of human intelligence
Narr
ow
(T
ask W
ea
k)
AI1
Clinical Decisions
Administrative Decisions
Ge
ne
ral (S
tro
ng
) A
I2
Full Diagnosis
and Treatment Plan
Clinical
Chatbots
Radiology
Interpretation
Acute
Clinical
Risk
Recruiting
Safety Risk Smart
Monitoring
Medication Dosing
Administrative
Chatbots
(Concierge)
Precision
Engagement
Precision
Medicine
Ambient
Documentation
Adaptive User
Interfaces
Worklist
Prioritization
Inevitable with Time
Todayrsquos Opportunity
(growing)
The Computer Will
See You Now
The Uber-ization of
Health Care
An AI Application Landscape
Capacity Staffing
Management
Population
Health Risk
The Self-Driving
HospitalInformation
Security
Targeted
Marketing
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
22
The Self-Driving Hospital
Combine Predictive Models with Front-Line Activation
Sources ldquoVirtual Air Traffic Control for Emergency Departmentsrdquo Qventus
httpsqventuscomcontentuploads201703QVE-ED-Solution-Briefpdf Health Care
IT Advisor research and analysis
1) EVS = Environmental services
2) LWBS = Left without being seen
3) LOS = Length of stay
Case in Brief
Mercy Hospital Fort Smith
bull 336-bed acute care hospital in Fort
Smith AR
bull Partnered with Qventus to improve
emergency department (ED) patient flow
and patient satisfaction
bull Analytics applies ML algorithms to
anticipate potential capacity shortfalls
process bottlenecks
bull Application sends real-time notifications
(nudges) to care teams via their mobile
phones to trigger specific interventions
bull Emergency department results
ndash LWBS2 rates dropped 30
ndash Door-to-doc time reduced by 20
ndash 24-minute reduction in average LOS3
ndash Annual case capacity increased by 6
See that a radiology order is delayed for a
patient within a few days of discharge and
notify radiology to prioritize the reading
Activate more EVS1 staff when high churn
of beds is anticipated
Send an early warning for difficult patient
placements at discharge
Prioritize inpatient transports depending
on expected bed needs
Sample ldquoDecision Recipesrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
23
Applicants Like You Liked This Job
Predictive Models Reduce Turnover Improve Engagement
Case in Brief MultiCare Health System
bull Non-profit system based in Tacoma WA with 16000 employees
bull Partnered with Arena a cloud-based HR solution to predict the
likelihood that a candidate will be retained in a role
bull Arenarsquos machine learning algorithms evaluate applicant
submissions digital interactions and externally sourced data on
the organization (eg employer review sites)
bull Achieved a 40 reduction in RN turnover at 180 days and a
28 reduction in overall turnover at 180 days
MultiCarersquos Hiring Process
Reduced recruiting
expenses
More predictable
staffing levels
More experienced
staff
Realized Benefits
Algorithms predict the likelihood the
applicant will be retained and
engaged in the target role and send
that information to the recruiter
Applicant completes
a 15-20 minute
assessment process
on Arenarsquos platform
Recruiters determine
whether or not to send
the candidate along to
hiring managers
Hiring managers use
Arenarsquos recommendation
as an optional factor in
the hiring decision
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
24
Accelerating Research
1) Yu KH et al ldquoPredicting non-small cell lung cancer prognosis by fully automated microscopic pathology image featuresrdquo Nature No 12474 (2016)
2) Rajwa B et al ldquoAutomated assessment of disease progression in acute myeloid leukemia by probabilistic analysis of flow cytometry datardquo IEEE Transactions on Biomedical Engineering 64 No 5 (2017) 1089-1098
3) Liu Y et al Detecting cancer metastases on gigapixel pathology images arXiv preprint arXiv1703 No 02442 (2017)
4) Rajpurkar P et al Cardiologist-level arrhythmia detection with convolutional neural networks arXiv preprint arXiv1707 No 01836 (2017)
5) Nielsen S et al A predictive model to identify Parkinson disease from administrative claims data Neurology 89 No 14 (2017) 1448-1456
6) Rajpurkar P et al CheXNet Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning arXiv preprint arXiv 1711 No 05225 (2017)
7) Poplin R et al Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning Nature Biomedical Engineering No 2 (2018)
8) ACC Machine learning techniques show promise for supporting medical decisions By quickly assessing risk new algorithms may help triage patients and inform clinical decisions ScienceDaily 2018
9) Oh J et al ldquoA Generalizable Data-Driven Approach to Predict Daily Risk of Clostridium difficile Infection at Two Large Academic Health Centersrdquo ICHE 39 No 4 (2018) 425-433
Nature1 ndash August 2016
ML algorithm can accurately
differentiate two types of lung
cancers and predict patient
survival time
IEEE2 ndash February 2017
ML model predicts relapse rates for
acute myelogenous leukemia with
90 accuracy
arXiv3 ndash March 2017
ML approach to imaging analytics
can identify metastasized breast
cancer with accuracy rates rivaling
those of pathologists
Neurology5 ndash Sept 2017
Predictive model to identify
Parkinsonrsquos disease from claims
data achieved 73-83 accuracy in
study results
arXiv4 ndash July 2017
ML algorithm can diagnose
heart arrhythmias using data
from wearables sensors with
cardiologist-level accuracy
arXiv6 ndash November 2017
Deep-learning algorithm
outperforms Stanford radiologists
at diagnosing pneumonia from
x-rays
American College of Cardiology8 ndash
Feb 2018
Model predicts heart attack
diagnosis among patients in the ED
with chest pain with ~90 accuracy
Nature7 ndash February 2018
ML algorithm can predict a
personrsquos cardiovascular risk
factors using eye scans
Infection Control Hospital
Epidemiology9 ndash April 2018
ML model predicts Clostridium
difficile infection (CDI) earlier
than current diagnostic methods
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
25
Driving the Decision Machine
1Acquire Data
bull Ensure all models have
a sponsor and regular
evaluation against goals
bull All models require
tune-ups as the
environment changes
Monitor Performance
and Revisit
42Train or Refine
the Model
bull The more the better
buthellip
bull High-quality well-
governed data pay
enormous dividends
bull Diverse cross-continuum
sources improve
predictions and
robustness of models
bull Decide what features
are important (eg
explainability)
bull Apply practical and
ethical constraints
bull Evaluate under
realistic conditions
(silent mode real-time
incomplete data)
bull Embed insights directly
into applications at key
decision points
bull Reengineer processes
as warranted
bull Consider the ldquo5 rightsrdquo
of decision support
3Incorporate Models
into Workflow
Hint The Analytics Is the Easy Part
Begin with
bull Explicit leadership agreement on outcome goals expected benefit mechanisms
bull Participation from empowered representatives of impacted process
bull External experiencemdashlearn from others
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
26
New Data Sources
Mainstream Systems Cover Only a Fraction of Predictive Power
Acquire Data
Sources Nash DB ldquoPopulation Health Why It Mattersrdquo Essentials in Population Health Childrenrsquos
Hospital Association Thomas Jefferson University October 2016 Health Care IT Advisor research and
analysis
1) RPM = Remote patient monitoring
EMR(s)
Lab Systems
Pharmacy Systems
Imaging
Medical Claims
Prescription Drug Claims
Billing and Supply Chain
Scheduling Systems
Medical
Care Data
10
Sensors RPM1
mHealth Apps
Patient-Reported
Outcomes
Genetics
Socioeconomic Data
Patient Questionnaires
Human
Biology
Data
20
Lifestyle
and
Behavior
Data
50
Social and
Environmental
Data
20
Data Sources by Influence on Health Outcomes
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
27
Big Data a Technology and a Philosophy
ldquoTherersquos Gold in Them Thar Hillsrdquo
Acquire Data
Sources Cloudera ldquoInova Translational Medicine Institute Partners with Cloudera to Advance Genome-Based Machine
Learning Initiatives and Save Livesrdquo PR Newswire June 2017 Health Care IT Advisor research and analysis
1) Originally coined by Doug Laney of the Meta Group
Assumes Value in the Data
Based on a premise that there are
valuable insights that can be distilled
from the mountains of data captured
by machines
Characterised by the 3 Vs1
bull Volume
bull Velocity
bull Variety
Places greater emphasis on
unstructured data
Free text images streams of sensor
data are all fair targets for analysis
Big data is the difference
between heart rates and
heart beats
Case in Brief Inova Health System
bull Inovarsquos Translational Medicine Institute
(ITMI) is assembling one of the worldrsquos
largest whole genome sequence
databases connected to patient
information in a health care system
bull Researchers use the Cloudera
machine learning platform to analyze
terabytes of clinical and genomic data
and identify genetic links to diseases
bull Insights are used to develop
personalized treatment plans for
patients in collaboration with the
treating physician
Resource Big Data in Health Care
Strengths challenges and pragmatic
guidelines for those considering the addition
of big data capabilities to their IT portfolio
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
30
How Machines Learn
Similar Methods Differing Goals
Build the Model
Source Advisory Board research and analysis
Input Data
bull The more the better
bull Labeled with key indicators
Adjust the Model
bull Automated trial and error
bull Cross-breeding successful
candidates
Repeat
Supervised Learning
Evaluate models against the ldquocorrectrdquo answers
provided with the data The learning process
focuses on minimizing errors
Unsupervised Learning
Evaluate models against an abstract algorithmic
goal Example Divide this population of diabetics
into a handful of subpopulations that have similar
future risk profiles
Reinforcement Learning
Evaluate models by giving a reward for
desirable outcomes and a penalty for negative
outcomes Models are trained to maximize their
total rewards over timeReady for Review
Evaluate Performance
bull Against what goal
bull Forces explicit
tradeoffs (eg cost
vs quality)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
32
Black Boxes Shades of Grey
Differing Views on the Importance of Explainability
Incorporate into Workflow
Mike Wall PharmD MBA
Chief Analytics Officer
The University of Chicago Medicine
The opaqueness of black-box models is
really scary When we canrsquot tell a
clinician why a patient wonrsquot do well on a
certain medication that can be a tough
thing to blindly trust from a clinical
perspective We want to reach a place
where we augment our practice
through AI rather than dictatingrdquo
Nigam Shah MBBS PhD
Associate Professor of Medicine
(Biomedical Informatics)
Stanford University
I am pushing to evaluate these models
via a similar set of mechanisms as RCT1
or AB testing We should evaluate
against the current status-quo and do a
beforeafter studyhellipWe have to convince
ourselves of the utility of these models
We should focus on utility before
explainabilityrdquo
Surgery is the
best option ()
Data
Black Box Model
1) RCT = Randomized controlled trial
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
33
Workflow Is Key to Adoption
Incorporate into Workflow
The Right Information
To the Right Person
In the Right Form
By the Right Channel
At the Right Time
Case in Brief Mayo Breast Oncology
Clinic Trials Recruitment
bull Up to 100 concurrent drug therapy clinical trials are
available to Mayo Clinic breast cancer patients making
it challenging to consistently identify candidates
bull New clinical trial matching technology developed in
collaboration with IBM Watson Health evaluates charts
to identify eligible patients
bull Launched in January 2016 as a point-of-care solution
for oncology providers initial adoption was low due to
time pressures and fell to zero by February
bull Rework by health systems engineers resulted in a July
2016 relaunch screening moved to occur in
advance of patient visits so clinicians know of trial
matches before opening a patientrsquos chart
bull New process enrolls significantly more patients in
clinical trials than the prior manual process success led
to additional coordinator funding
The Five Rights of
Clinical Decision Support
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
34
Health Carersquos Field Agents
Medical Devices Become Intelligent Partners
Incorporate into Workflow
1) EKG = Electrocardiogram
ME
DT
RO
NIC
MIN
IME
D 6
70G
Case in Brief Artificial Pancreas and Smart Infusion Pumpsmdash
Medtronic MiniMed Connect
bull SMARTGUARD mimics some functions of a healthy pancreas
measures and predicts glucose level drift and adjusts dosing
bull Insulin pump and continuous glucose monitoring can send data directly
to smartphone
bull Trials showed a 44 reduction in hypoglycemia and reduction in
average A1C values from 74 to 69
Case in Brief AliveCor KardiaMobile
bull Portable consumer-grade single-lead EKG1 retails for $100
bull Intended for self-capture of intermittent cardiac events such as atrial
fibrillation (AFIB)
bull Smartphone-based machine learning validates quality of trace
assesses possibility of AFIB classifies normal abnormal sinus rhythm
bull EKGs can be sent to a board-certified cardiologist for interpretation
for $19
Sources alivecorcom Medtroniccom Health Care IT Advisor research and
analysis
AliveC
orK
ard
iaM
obile
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
21
Where to Start Where Wersquore Going
1) Narrow (Task Weak) AI focuses on narrowly defined problems such as predicting a patientrsquos risk of developing sepsis
2) General (Strong) AI attempts to match the flexibility and versatility of human intelligence
Narr
ow
(T
ask W
ea
k)
AI1
Clinical Decisions
Administrative Decisions
Ge
ne
ral (S
tro
ng
) A
I2
Full Diagnosis
and Treatment Plan
Clinical
Chatbots
Radiology
Interpretation
Acute
Clinical
Risk
Recruiting
Safety Risk Smart
Monitoring
Medication Dosing
Administrative
Chatbots
(Concierge)
Precision
Engagement
Precision
Medicine
Ambient
Documentation
Adaptive User
Interfaces
Worklist
Prioritization
Inevitable with Time
Todayrsquos Opportunity
(growing)
The Computer Will
See You Now
The Uber-ization of
Health Care
An AI Application Landscape
Capacity Staffing
Management
Population
Health Risk
The Self-Driving
HospitalInformation
Security
Targeted
Marketing
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
22
The Self-Driving Hospital
Combine Predictive Models with Front-Line Activation
Sources ldquoVirtual Air Traffic Control for Emergency Departmentsrdquo Qventus
httpsqventuscomcontentuploads201703QVE-ED-Solution-Briefpdf Health Care
IT Advisor research and analysis
1) EVS = Environmental services
2) LWBS = Left without being seen
3) LOS = Length of stay
Case in Brief
Mercy Hospital Fort Smith
bull 336-bed acute care hospital in Fort
Smith AR
bull Partnered with Qventus to improve
emergency department (ED) patient flow
and patient satisfaction
bull Analytics applies ML algorithms to
anticipate potential capacity shortfalls
process bottlenecks
bull Application sends real-time notifications
(nudges) to care teams via their mobile
phones to trigger specific interventions
bull Emergency department results
ndash LWBS2 rates dropped 30
ndash Door-to-doc time reduced by 20
ndash 24-minute reduction in average LOS3
ndash Annual case capacity increased by 6
See that a radiology order is delayed for a
patient within a few days of discharge and
notify radiology to prioritize the reading
Activate more EVS1 staff when high churn
of beds is anticipated
Send an early warning for difficult patient
placements at discharge
Prioritize inpatient transports depending
on expected bed needs
Sample ldquoDecision Recipesrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
23
Applicants Like You Liked This Job
Predictive Models Reduce Turnover Improve Engagement
Case in Brief MultiCare Health System
bull Non-profit system based in Tacoma WA with 16000 employees
bull Partnered with Arena a cloud-based HR solution to predict the
likelihood that a candidate will be retained in a role
bull Arenarsquos machine learning algorithms evaluate applicant
submissions digital interactions and externally sourced data on
the organization (eg employer review sites)
bull Achieved a 40 reduction in RN turnover at 180 days and a
28 reduction in overall turnover at 180 days
MultiCarersquos Hiring Process
Reduced recruiting
expenses
More predictable
staffing levels
More experienced
staff
Realized Benefits
Algorithms predict the likelihood the
applicant will be retained and
engaged in the target role and send
that information to the recruiter
Applicant completes
a 15-20 minute
assessment process
on Arenarsquos platform
Recruiters determine
whether or not to send
the candidate along to
hiring managers
Hiring managers use
Arenarsquos recommendation
as an optional factor in
the hiring decision
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
24
Accelerating Research
1) Yu KH et al ldquoPredicting non-small cell lung cancer prognosis by fully automated microscopic pathology image featuresrdquo Nature No 12474 (2016)
2) Rajwa B et al ldquoAutomated assessment of disease progression in acute myeloid leukemia by probabilistic analysis of flow cytometry datardquo IEEE Transactions on Biomedical Engineering 64 No 5 (2017) 1089-1098
3) Liu Y et al Detecting cancer metastases on gigapixel pathology images arXiv preprint arXiv1703 No 02442 (2017)
4) Rajpurkar P et al Cardiologist-level arrhythmia detection with convolutional neural networks arXiv preprint arXiv1707 No 01836 (2017)
5) Nielsen S et al A predictive model to identify Parkinson disease from administrative claims data Neurology 89 No 14 (2017) 1448-1456
6) Rajpurkar P et al CheXNet Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning arXiv preprint arXiv 1711 No 05225 (2017)
7) Poplin R et al Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning Nature Biomedical Engineering No 2 (2018)
8) ACC Machine learning techniques show promise for supporting medical decisions By quickly assessing risk new algorithms may help triage patients and inform clinical decisions ScienceDaily 2018
9) Oh J et al ldquoA Generalizable Data-Driven Approach to Predict Daily Risk of Clostridium difficile Infection at Two Large Academic Health Centersrdquo ICHE 39 No 4 (2018) 425-433
Nature1 ndash August 2016
ML algorithm can accurately
differentiate two types of lung
cancers and predict patient
survival time
IEEE2 ndash February 2017
ML model predicts relapse rates for
acute myelogenous leukemia with
90 accuracy
arXiv3 ndash March 2017
ML approach to imaging analytics
can identify metastasized breast
cancer with accuracy rates rivaling
those of pathologists
Neurology5 ndash Sept 2017
Predictive model to identify
Parkinsonrsquos disease from claims
data achieved 73-83 accuracy in
study results
arXiv4 ndash July 2017
ML algorithm can diagnose
heart arrhythmias using data
from wearables sensors with
cardiologist-level accuracy
arXiv6 ndash November 2017
Deep-learning algorithm
outperforms Stanford radiologists
at diagnosing pneumonia from
x-rays
American College of Cardiology8 ndash
Feb 2018
Model predicts heart attack
diagnosis among patients in the ED
with chest pain with ~90 accuracy
Nature7 ndash February 2018
ML algorithm can predict a
personrsquos cardiovascular risk
factors using eye scans
Infection Control Hospital
Epidemiology9 ndash April 2018
ML model predicts Clostridium
difficile infection (CDI) earlier
than current diagnostic methods
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
25
Driving the Decision Machine
1Acquire Data
bull Ensure all models have
a sponsor and regular
evaluation against goals
bull All models require
tune-ups as the
environment changes
Monitor Performance
and Revisit
42Train or Refine
the Model
bull The more the better
buthellip
bull High-quality well-
governed data pay
enormous dividends
bull Diverse cross-continuum
sources improve
predictions and
robustness of models
bull Decide what features
are important (eg
explainability)
bull Apply practical and
ethical constraints
bull Evaluate under
realistic conditions
(silent mode real-time
incomplete data)
bull Embed insights directly
into applications at key
decision points
bull Reengineer processes
as warranted
bull Consider the ldquo5 rightsrdquo
of decision support
3Incorporate Models
into Workflow
Hint The Analytics Is the Easy Part
Begin with
bull Explicit leadership agreement on outcome goals expected benefit mechanisms
bull Participation from empowered representatives of impacted process
bull External experiencemdashlearn from others
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
26
New Data Sources
Mainstream Systems Cover Only a Fraction of Predictive Power
Acquire Data
Sources Nash DB ldquoPopulation Health Why It Mattersrdquo Essentials in Population Health Childrenrsquos
Hospital Association Thomas Jefferson University October 2016 Health Care IT Advisor research and
analysis
1) RPM = Remote patient monitoring
EMR(s)
Lab Systems
Pharmacy Systems
Imaging
Medical Claims
Prescription Drug Claims
Billing and Supply Chain
Scheduling Systems
Medical
Care Data
10
Sensors RPM1
mHealth Apps
Patient-Reported
Outcomes
Genetics
Socioeconomic Data
Patient Questionnaires
Human
Biology
Data
20
Lifestyle
and
Behavior
Data
50
Social and
Environmental
Data
20
Data Sources by Influence on Health Outcomes
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
27
Big Data a Technology and a Philosophy
ldquoTherersquos Gold in Them Thar Hillsrdquo
Acquire Data
Sources Cloudera ldquoInova Translational Medicine Institute Partners with Cloudera to Advance Genome-Based Machine
Learning Initiatives and Save Livesrdquo PR Newswire June 2017 Health Care IT Advisor research and analysis
1) Originally coined by Doug Laney of the Meta Group
Assumes Value in the Data
Based on a premise that there are
valuable insights that can be distilled
from the mountains of data captured
by machines
Characterised by the 3 Vs1
bull Volume
bull Velocity
bull Variety
Places greater emphasis on
unstructured data
Free text images streams of sensor
data are all fair targets for analysis
Big data is the difference
between heart rates and
heart beats
Case in Brief Inova Health System
bull Inovarsquos Translational Medicine Institute
(ITMI) is assembling one of the worldrsquos
largest whole genome sequence
databases connected to patient
information in a health care system
bull Researchers use the Cloudera
machine learning platform to analyze
terabytes of clinical and genomic data
and identify genetic links to diseases
bull Insights are used to develop
personalized treatment plans for
patients in collaboration with the
treating physician
Resource Big Data in Health Care
Strengths challenges and pragmatic
guidelines for those considering the addition
of big data capabilities to their IT portfolio
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
30
How Machines Learn
Similar Methods Differing Goals
Build the Model
Source Advisory Board research and analysis
Input Data
bull The more the better
bull Labeled with key indicators
Adjust the Model
bull Automated trial and error
bull Cross-breeding successful
candidates
Repeat
Supervised Learning
Evaluate models against the ldquocorrectrdquo answers
provided with the data The learning process
focuses on minimizing errors
Unsupervised Learning
Evaluate models against an abstract algorithmic
goal Example Divide this population of diabetics
into a handful of subpopulations that have similar
future risk profiles
Reinforcement Learning
Evaluate models by giving a reward for
desirable outcomes and a penalty for negative
outcomes Models are trained to maximize their
total rewards over timeReady for Review
Evaluate Performance
bull Against what goal
bull Forces explicit
tradeoffs (eg cost
vs quality)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
32
Black Boxes Shades of Grey
Differing Views on the Importance of Explainability
Incorporate into Workflow
Mike Wall PharmD MBA
Chief Analytics Officer
The University of Chicago Medicine
The opaqueness of black-box models is
really scary When we canrsquot tell a
clinician why a patient wonrsquot do well on a
certain medication that can be a tough
thing to blindly trust from a clinical
perspective We want to reach a place
where we augment our practice
through AI rather than dictatingrdquo
Nigam Shah MBBS PhD
Associate Professor of Medicine
(Biomedical Informatics)
Stanford University
I am pushing to evaluate these models
via a similar set of mechanisms as RCT1
or AB testing We should evaluate
against the current status-quo and do a
beforeafter studyhellipWe have to convince
ourselves of the utility of these models
We should focus on utility before
explainabilityrdquo
Surgery is the
best option ()
Data
Black Box Model
1) RCT = Randomized controlled trial
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
33
Workflow Is Key to Adoption
Incorporate into Workflow
The Right Information
To the Right Person
In the Right Form
By the Right Channel
At the Right Time
Case in Brief Mayo Breast Oncology
Clinic Trials Recruitment
bull Up to 100 concurrent drug therapy clinical trials are
available to Mayo Clinic breast cancer patients making
it challenging to consistently identify candidates
bull New clinical trial matching technology developed in
collaboration with IBM Watson Health evaluates charts
to identify eligible patients
bull Launched in January 2016 as a point-of-care solution
for oncology providers initial adoption was low due to
time pressures and fell to zero by February
bull Rework by health systems engineers resulted in a July
2016 relaunch screening moved to occur in
advance of patient visits so clinicians know of trial
matches before opening a patientrsquos chart
bull New process enrolls significantly more patients in
clinical trials than the prior manual process success led
to additional coordinator funding
The Five Rights of
Clinical Decision Support
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
34
Health Carersquos Field Agents
Medical Devices Become Intelligent Partners
Incorporate into Workflow
1) EKG = Electrocardiogram
ME
DT
RO
NIC
MIN
IME
D 6
70G
Case in Brief Artificial Pancreas and Smart Infusion Pumpsmdash
Medtronic MiniMed Connect
bull SMARTGUARD mimics some functions of a healthy pancreas
measures and predicts glucose level drift and adjusts dosing
bull Insulin pump and continuous glucose monitoring can send data directly
to smartphone
bull Trials showed a 44 reduction in hypoglycemia and reduction in
average A1C values from 74 to 69
Case in Brief AliveCor KardiaMobile
bull Portable consumer-grade single-lead EKG1 retails for $100
bull Intended for self-capture of intermittent cardiac events such as atrial
fibrillation (AFIB)
bull Smartphone-based machine learning validates quality of trace
assesses possibility of AFIB classifies normal abnormal sinus rhythm
bull EKGs can be sent to a board-certified cardiologist for interpretation
for $19
Sources alivecorcom Medtroniccom Health Care IT Advisor research and
analysis
AliveC
orK
ard
iaM
obile
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
22
The Self-Driving Hospital
Combine Predictive Models with Front-Line Activation
Sources ldquoVirtual Air Traffic Control for Emergency Departmentsrdquo Qventus
httpsqventuscomcontentuploads201703QVE-ED-Solution-Briefpdf Health Care
IT Advisor research and analysis
1) EVS = Environmental services
2) LWBS = Left without being seen
3) LOS = Length of stay
Case in Brief
Mercy Hospital Fort Smith
bull 336-bed acute care hospital in Fort
Smith AR
bull Partnered with Qventus to improve
emergency department (ED) patient flow
and patient satisfaction
bull Analytics applies ML algorithms to
anticipate potential capacity shortfalls
process bottlenecks
bull Application sends real-time notifications
(nudges) to care teams via their mobile
phones to trigger specific interventions
bull Emergency department results
ndash LWBS2 rates dropped 30
ndash Door-to-doc time reduced by 20
ndash 24-minute reduction in average LOS3
ndash Annual case capacity increased by 6
See that a radiology order is delayed for a
patient within a few days of discharge and
notify radiology to prioritize the reading
Activate more EVS1 staff when high churn
of beds is anticipated
Send an early warning for difficult patient
placements at discharge
Prioritize inpatient transports depending
on expected bed needs
Sample ldquoDecision Recipesrdquo
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
23
Applicants Like You Liked This Job
Predictive Models Reduce Turnover Improve Engagement
Case in Brief MultiCare Health System
bull Non-profit system based in Tacoma WA with 16000 employees
bull Partnered with Arena a cloud-based HR solution to predict the
likelihood that a candidate will be retained in a role
bull Arenarsquos machine learning algorithms evaluate applicant
submissions digital interactions and externally sourced data on
the organization (eg employer review sites)
bull Achieved a 40 reduction in RN turnover at 180 days and a
28 reduction in overall turnover at 180 days
MultiCarersquos Hiring Process
Reduced recruiting
expenses
More predictable
staffing levels
More experienced
staff
Realized Benefits
Algorithms predict the likelihood the
applicant will be retained and
engaged in the target role and send
that information to the recruiter
Applicant completes
a 15-20 minute
assessment process
on Arenarsquos platform
Recruiters determine
whether or not to send
the candidate along to
hiring managers
Hiring managers use
Arenarsquos recommendation
as an optional factor in
the hiring decision
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
24
Accelerating Research
1) Yu KH et al ldquoPredicting non-small cell lung cancer prognosis by fully automated microscopic pathology image featuresrdquo Nature No 12474 (2016)
2) Rajwa B et al ldquoAutomated assessment of disease progression in acute myeloid leukemia by probabilistic analysis of flow cytometry datardquo IEEE Transactions on Biomedical Engineering 64 No 5 (2017) 1089-1098
3) Liu Y et al Detecting cancer metastases on gigapixel pathology images arXiv preprint arXiv1703 No 02442 (2017)
4) Rajpurkar P et al Cardiologist-level arrhythmia detection with convolutional neural networks arXiv preprint arXiv1707 No 01836 (2017)
5) Nielsen S et al A predictive model to identify Parkinson disease from administrative claims data Neurology 89 No 14 (2017) 1448-1456
6) Rajpurkar P et al CheXNet Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning arXiv preprint arXiv 1711 No 05225 (2017)
7) Poplin R et al Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning Nature Biomedical Engineering No 2 (2018)
8) ACC Machine learning techniques show promise for supporting medical decisions By quickly assessing risk new algorithms may help triage patients and inform clinical decisions ScienceDaily 2018
9) Oh J et al ldquoA Generalizable Data-Driven Approach to Predict Daily Risk of Clostridium difficile Infection at Two Large Academic Health Centersrdquo ICHE 39 No 4 (2018) 425-433
Nature1 ndash August 2016
ML algorithm can accurately
differentiate two types of lung
cancers and predict patient
survival time
IEEE2 ndash February 2017
ML model predicts relapse rates for
acute myelogenous leukemia with
90 accuracy
arXiv3 ndash March 2017
ML approach to imaging analytics
can identify metastasized breast
cancer with accuracy rates rivaling
those of pathologists
Neurology5 ndash Sept 2017
Predictive model to identify
Parkinsonrsquos disease from claims
data achieved 73-83 accuracy in
study results
arXiv4 ndash July 2017
ML algorithm can diagnose
heart arrhythmias using data
from wearables sensors with
cardiologist-level accuracy
arXiv6 ndash November 2017
Deep-learning algorithm
outperforms Stanford radiologists
at diagnosing pneumonia from
x-rays
American College of Cardiology8 ndash
Feb 2018
Model predicts heart attack
diagnosis among patients in the ED
with chest pain with ~90 accuracy
Nature7 ndash February 2018
ML algorithm can predict a
personrsquos cardiovascular risk
factors using eye scans
Infection Control Hospital
Epidemiology9 ndash April 2018
ML model predicts Clostridium
difficile infection (CDI) earlier
than current diagnostic methods
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
25
Driving the Decision Machine
1Acquire Data
bull Ensure all models have
a sponsor and regular
evaluation against goals
bull All models require
tune-ups as the
environment changes
Monitor Performance
and Revisit
42Train or Refine
the Model
bull The more the better
buthellip
bull High-quality well-
governed data pay
enormous dividends
bull Diverse cross-continuum
sources improve
predictions and
robustness of models
bull Decide what features
are important (eg
explainability)
bull Apply practical and
ethical constraints
bull Evaluate under
realistic conditions
(silent mode real-time
incomplete data)
bull Embed insights directly
into applications at key
decision points
bull Reengineer processes
as warranted
bull Consider the ldquo5 rightsrdquo
of decision support
3Incorporate Models
into Workflow
Hint The Analytics Is the Easy Part
Begin with
bull Explicit leadership agreement on outcome goals expected benefit mechanisms
bull Participation from empowered representatives of impacted process
bull External experiencemdashlearn from others
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
26
New Data Sources
Mainstream Systems Cover Only a Fraction of Predictive Power
Acquire Data
Sources Nash DB ldquoPopulation Health Why It Mattersrdquo Essentials in Population Health Childrenrsquos
Hospital Association Thomas Jefferson University October 2016 Health Care IT Advisor research and
analysis
1) RPM = Remote patient monitoring
EMR(s)
Lab Systems
Pharmacy Systems
Imaging
Medical Claims
Prescription Drug Claims
Billing and Supply Chain
Scheduling Systems
Medical
Care Data
10
Sensors RPM1
mHealth Apps
Patient-Reported
Outcomes
Genetics
Socioeconomic Data
Patient Questionnaires
Human
Biology
Data
20
Lifestyle
and
Behavior
Data
50
Social and
Environmental
Data
20
Data Sources by Influence on Health Outcomes
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
27
Big Data a Technology and a Philosophy
ldquoTherersquos Gold in Them Thar Hillsrdquo
Acquire Data
Sources Cloudera ldquoInova Translational Medicine Institute Partners with Cloudera to Advance Genome-Based Machine
Learning Initiatives and Save Livesrdquo PR Newswire June 2017 Health Care IT Advisor research and analysis
1) Originally coined by Doug Laney of the Meta Group
Assumes Value in the Data
Based on a premise that there are
valuable insights that can be distilled
from the mountains of data captured
by machines
Characterised by the 3 Vs1
bull Volume
bull Velocity
bull Variety
Places greater emphasis on
unstructured data
Free text images streams of sensor
data are all fair targets for analysis
Big data is the difference
between heart rates and
heart beats
Case in Brief Inova Health System
bull Inovarsquos Translational Medicine Institute
(ITMI) is assembling one of the worldrsquos
largest whole genome sequence
databases connected to patient
information in a health care system
bull Researchers use the Cloudera
machine learning platform to analyze
terabytes of clinical and genomic data
and identify genetic links to diseases
bull Insights are used to develop
personalized treatment plans for
patients in collaboration with the
treating physician
Resource Big Data in Health Care
Strengths challenges and pragmatic
guidelines for those considering the addition
of big data capabilities to their IT portfolio
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
30
How Machines Learn
Similar Methods Differing Goals
Build the Model
Source Advisory Board research and analysis
Input Data
bull The more the better
bull Labeled with key indicators
Adjust the Model
bull Automated trial and error
bull Cross-breeding successful
candidates
Repeat
Supervised Learning
Evaluate models against the ldquocorrectrdquo answers
provided with the data The learning process
focuses on minimizing errors
Unsupervised Learning
Evaluate models against an abstract algorithmic
goal Example Divide this population of diabetics
into a handful of subpopulations that have similar
future risk profiles
Reinforcement Learning
Evaluate models by giving a reward for
desirable outcomes and a penalty for negative
outcomes Models are trained to maximize their
total rewards over timeReady for Review
Evaluate Performance
bull Against what goal
bull Forces explicit
tradeoffs (eg cost
vs quality)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
32
Black Boxes Shades of Grey
Differing Views on the Importance of Explainability
Incorporate into Workflow
Mike Wall PharmD MBA
Chief Analytics Officer
The University of Chicago Medicine
The opaqueness of black-box models is
really scary When we canrsquot tell a
clinician why a patient wonrsquot do well on a
certain medication that can be a tough
thing to blindly trust from a clinical
perspective We want to reach a place
where we augment our practice
through AI rather than dictatingrdquo
Nigam Shah MBBS PhD
Associate Professor of Medicine
(Biomedical Informatics)
Stanford University
I am pushing to evaluate these models
via a similar set of mechanisms as RCT1
or AB testing We should evaluate
against the current status-quo and do a
beforeafter studyhellipWe have to convince
ourselves of the utility of these models
We should focus on utility before
explainabilityrdquo
Surgery is the
best option ()
Data
Black Box Model
1) RCT = Randomized controlled trial
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
33
Workflow Is Key to Adoption
Incorporate into Workflow
The Right Information
To the Right Person
In the Right Form
By the Right Channel
At the Right Time
Case in Brief Mayo Breast Oncology
Clinic Trials Recruitment
bull Up to 100 concurrent drug therapy clinical trials are
available to Mayo Clinic breast cancer patients making
it challenging to consistently identify candidates
bull New clinical trial matching technology developed in
collaboration with IBM Watson Health evaluates charts
to identify eligible patients
bull Launched in January 2016 as a point-of-care solution
for oncology providers initial adoption was low due to
time pressures and fell to zero by February
bull Rework by health systems engineers resulted in a July
2016 relaunch screening moved to occur in
advance of patient visits so clinicians know of trial
matches before opening a patientrsquos chart
bull New process enrolls significantly more patients in
clinical trials than the prior manual process success led
to additional coordinator funding
The Five Rights of
Clinical Decision Support
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
34
Health Carersquos Field Agents
Medical Devices Become Intelligent Partners
Incorporate into Workflow
1) EKG = Electrocardiogram
ME
DT
RO
NIC
MIN
IME
D 6
70G
Case in Brief Artificial Pancreas and Smart Infusion Pumpsmdash
Medtronic MiniMed Connect
bull SMARTGUARD mimics some functions of a healthy pancreas
measures and predicts glucose level drift and adjusts dosing
bull Insulin pump and continuous glucose monitoring can send data directly
to smartphone
bull Trials showed a 44 reduction in hypoglycemia and reduction in
average A1C values from 74 to 69
Case in Brief AliveCor KardiaMobile
bull Portable consumer-grade single-lead EKG1 retails for $100
bull Intended for self-capture of intermittent cardiac events such as atrial
fibrillation (AFIB)
bull Smartphone-based machine learning validates quality of trace
assesses possibility of AFIB classifies normal abnormal sinus rhythm
bull EKGs can be sent to a board-certified cardiologist for interpretation
for $19
Sources alivecorcom Medtroniccom Health Care IT Advisor research and
analysis
AliveC
orK
ard
iaM
obile
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
23
Applicants Like You Liked This Job
Predictive Models Reduce Turnover Improve Engagement
Case in Brief MultiCare Health System
bull Non-profit system based in Tacoma WA with 16000 employees
bull Partnered with Arena a cloud-based HR solution to predict the
likelihood that a candidate will be retained in a role
bull Arenarsquos machine learning algorithms evaluate applicant
submissions digital interactions and externally sourced data on
the organization (eg employer review sites)
bull Achieved a 40 reduction in RN turnover at 180 days and a
28 reduction in overall turnover at 180 days
MultiCarersquos Hiring Process
Reduced recruiting
expenses
More predictable
staffing levels
More experienced
staff
Realized Benefits
Algorithms predict the likelihood the
applicant will be retained and
engaged in the target role and send
that information to the recruiter
Applicant completes
a 15-20 minute
assessment process
on Arenarsquos platform
Recruiters determine
whether or not to send
the candidate along to
hiring managers
Hiring managers use
Arenarsquos recommendation
as an optional factor in
the hiring decision
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
24
Accelerating Research
1) Yu KH et al ldquoPredicting non-small cell lung cancer prognosis by fully automated microscopic pathology image featuresrdquo Nature No 12474 (2016)
2) Rajwa B et al ldquoAutomated assessment of disease progression in acute myeloid leukemia by probabilistic analysis of flow cytometry datardquo IEEE Transactions on Biomedical Engineering 64 No 5 (2017) 1089-1098
3) Liu Y et al Detecting cancer metastases on gigapixel pathology images arXiv preprint arXiv1703 No 02442 (2017)
4) Rajpurkar P et al Cardiologist-level arrhythmia detection with convolutional neural networks arXiv preprint arXiv1707 No 01836 (2017)
5) Nielsen S et al A predictive model to identify Parkinson disease from administrative claims data Neurology 89 No 14 (2017) 1448-1456
6) Rajpurkar P et al CheXNet Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning arXiv preprint arXiv 1711 No 05225 (2017)
7) Poplin R et al Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning Nature Biomedical Engineering No 2 (2018)
8) ACC Machine learning techniques show promise for supporting medical decisions By quickly assessing risk new algorithms may help triage patients and inform clinical decisions ScienceDaily 2018
9) Oh J et al ldquoA Generalizable Data-Driven Approach to Predict Daily Risk of Clostridium difficile Infection at Two Large Academic Health Centersrdquo ICHE 39 No 4 (2018) 425-433
Nature1 ndash August 2016
ML algorithm can accurately
differentiate two types of lung
cancers and predict patient
survival time
IEEE2 ndash February 2017
ML model predicts relapse rates for
acute myelogenous leukemia with
90 accuracy
arXiv3 ndash March 2017
ML approach to imaging analytics
can identify metastasized breast
cancer with accuracy rates rivaling
those of pathologists
Neurology5 ndash Sept 2017
Predictive model to identify
Parkinsonrsquos disease from claims
data achieved 73-83 accuracy in
study results
arXiv4 ndash July 2017
ML algorithm can diagnose
heart arrhythmias using data
from wearables sensors with
cardiologist-level accuracy
arXiv6 ndash November 2017
Deep-learning algorithm
outperforms Stanford radiologists
at diagnosing pneumonia from
x-rays
American College of Cardiology8 ndash
Feb 2018
Model predicts heart attack
diagnosis among patients in the ED
with chest pain with ~90 accuracy
Nature7 ndash February 2018
ML algorithm can predict a
personrsquos cardiovascular risk
factors using eye scans
Infection Control Hospital
Epidemiology9 ndash April 2018
ML model predicts Clostridium
difficile infection (CDI) earlier
than current diagnostic methods
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
25
Driving the Decision Machine
1Acquire Data
bull Ensure all models have
a sponsor and regular
evaluation against goals
bull All models require
tune-ups as the
environment changes
Monitor Performance
and Revisit
42Train or Refine
the Model
bull The more the better
buthellip
bull High-quality well-
governed data pay
enormous dividends
bull Diverse cross-continuum
sources improve
predictions and
robustness of models
bull Decide what features
are important (eg
explainability)
bull Apply practical and
ethical constraints
bull Evaluate under
realistic conditions
(silent mode real-time
incomplete data)
bull Embed insights directly
into applications at key
decision points
bull Reengineer processes
as warranted
bull Consider the ldquo5 rightsrdquo
of decision support
3Incorporate Models
into Workflow
Hint The Analytics Is the Easy Part
Begin with
bull Explicit leadership agreement on outcome goals expected benefit mechanisms
bull Participation from empowered representatives of impacted process
bull External experiencemdashlearn from others
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
26
New Data Sources
Mainstream Systems Cover Only a Fraction of Predictive Power
Acquire Data
Sources Nash DB ldquoPopulation Health Why It Mattersrdquo Essentials in Population Health Childrenrsquos
Hospital Association Thomas Jefferson University October 2016 Health Care IT Advisor research and
analysis
1) RPM = Remote patient monitoring
EMR(s)
Lab Systems
Pharmacy Systems
Imaging
Medical Claims
Prescription Drug Claims
Billing and Supply Chain
Scheduling Systems
Medical
Care Data
10
Sensors RPM1
mHealth Apps
Patient-Reported
Outcomes
Genetics
Socioeconomic Data
Patient Questionnaires
Human
Biology
Data
20
Lifestyle
and
Behavior
Data
50
Social and
Environmental
Data
20
Data Sources by Influence on Health Outcomes
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
27
Big Data a Technology and a Philosophy
ldquoTherersquos Gold in Them Thar Hillsrdquo
Acquire Data
Sources Cloudera ldquoInova Translational Medicine Institute Partners with Cloudera to Advance Genome-Based Machine
Learning Initiatives and Save Livesrdquo PR Newswire June 2017 Health Care IT Advisor research and analysis
1) Originally coined by Doug Laney of the Meta Group
Assumes Value in the Data
Based on a premise that there are
valuable insights that can be distilled
from the mountains of data captured
by machines
Characterised by the 3 Vs1
bull Volume
bull Velocity
bull Variety
Places greater emphasis on
unstructured data
Free text images streams of sensor
data are all fair targets for analysis
Big data is the difference
between heart rates and
heart beats
Case in Brief Inova Health System
bull Inovarsquos Translational Medicine Institute
(ITMI) is assembling one of the worldrsquos
largest whole genome sequence
databases connected to patient
information in a health care system
bull Researchers use the Cloudera
machine learning platform to analyze
terabytes of clinical and genomic data
and identify genetic links to diseases
bull Insights are used to develop
personalized treatment plans for
patients in collaboration with the
treating physician
Resource Big Data in Health Care
Strengths challenges and pragmatic
guidelines for those considering the addition
of big data capabilities to their IT portfolio
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
30
How Machines Learn
Similar Methods Differing Goals
Build the Model
Source Advisory Board research and analysis
Input Data
bull The more the better
bull Labeled with key indicators
Adjust the Model
bull Automated trial and error
bull Cross-breeding successful
candidates
Repeat
Supervised Learning
Evaluate models against the ldquocorrectrdquo answers
provided with the data The learning process
focuses on minimizing errors
Unsupervised Learning
Evaluate models against an abstract algorithmic
goal Example Divide this population of diabetics
into a handful of subpopulations that have similar
future risk profiles
Reinforcement Learning
Evaluate models by giving a reward for
desirable outcomes and a penalty for negative
outcomes Models are trained to maximize their
total rewards over timeReady for Review
Evaluate Performance
bull Against what goal
bull Forces explicit
tradeoffs (eg cost
vs quality)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
32
Black Boxes Shades of Grey
Differing Views on the Importance of Explainability
Incorporate into Workflow
Mike Wall PharmD MBA
Chief Analytics Officer
The University of Chicago Medicine
The opaqueness of black-box models is
really scary When we canrsquot tell a
clinician why a patient wonrsquot do well on a
certain medication that can be a tough
thing to blindly trust from a clinical
perspective We want to reach a place
where we augment our practice
through AI rather than dictatingrdquo
Nigam Shah MBBS PhD
Associate Professor of Medicine
(Biomedical Informatics)
Stanford University
I am pushing to evaluate these models
via a similar set of mechanisms as RCT1
or AB testing We should evaluate
against the current status-quo and do a
beforeafter studyhellipWe have to convince
ourselves of the utility of these models
We should focus on utility before
explainabilityrdquo
Surgery is the
best option ()
Data
Black Box Model
1) RCT = Randomized controlled trial
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
33
Workflow Is Key to Adoption
Incorporate into Workflow
The Right Information
To the Right Person
In the Right Form
By the Right Channel
At the Right Time
Case in Brief Mayo Breast Oncology
Clinic Trials Recruitment
bull Up to 100 concurrent drug therapy clinical trials are
available to Mayo Clinic breast cancer patients making
it challenging to consistently identify candidates
bull New clinical trial matching technology developed in
collaboration with IBM Watson Health evaluates charts
to identify eligible patients
bull Launched in January 2016 as a point-of-care solution
for oncology providers initial adoption was low due to
time pressures and fell to zero by February
bull Rework by health systems engineers resulted in a July
2016 relaunch screening moved to occur in
advance of patient visits so clinicians know of trial
matches before opening a patientrsquos chart
bull New process enrolls significantly more patients in
clinical trials than the prior manual process success led
to additional coordinator funding
The Five Rights of
Clinical Decision Support
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
34
Health Carersquos Field Agents
Medical Devices Become Intelligent Partners
Incorporate into Workflow
1) EKG = Electrocardiogram
ME
DT
RO
NIC
MIN
IME
D 6
70G
Case in Brief Artificial Pancreas and Smart Infusion Pumpsmdash
Medtronic MiniMed Connect
bull SMARTGUARD mimics some functions of a healthy pancreas
measures and predicts glucose level drift and adjusts dosing
bull Insulin pump and continuous glucose monitoring can send data directly
to smartphone
bull Trials showed a 44 reduction in hypoglycemia and reduction in
average A1C values from 74 to 69
Case in Brief AliveCor KardiaMobile
bull Portable consumer-grade single-lead EKG1 retails for $100
bull Intended for self-capture of intermittent cardiac events such as atrial
fibrillation (AFIB)
bull Smartphone-based machine learning validates quality of trace
assesses possibility of AFIB classifies normal abnormal sinus rhythm
bull EKGs can be sent to a board-certified cardiologist for interpretation
for $19
Sources alivecorcom Medtroniccom Health Care IT Advisor research and
analysis
AliveC
orK
ard
iaM
obile
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
24
Accelerating Research
1) Yu KH et al ldquoPredicting non-small cell lung cancer prognosis by fully automated microscopic pathology image featuresrdquo Nature No 12474 (2016)
2) Rajwa B et al ldquoAutomated assessment of disease progression in acute myeloid leukemia by probabilistic analysis of flow cytometry datardquo IEEE Transactions on Biomedical Engineering 64 No 5 (2017) 1089-1098
3) Liu Y et al Detecting cancer metastases on gigapixel pathology images arXiv preprint arXiv1703 No 02442 (2017)
4) Rajpurkar P et al Cardiologist-level arrhythmia detection with convolutional neural networks arXiv preprint arXiv1707 No 01836 (2017)
5) Nielsen S et al A predictive model to identify Parkinson disease from administrative claims data Neurology 89 No 14 (2017) 1448-1456
6) Rajpurkar P et al CheXNet Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning arXiv preprint arXiv 1711 No 05225 (2017)
7) Poplin R et al Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning Nature Biomedical Engineering No 2 (2018)
8) ACC Machine learning techniques show promise for supporting medical decisions By quickly assessing risk new algorithms may help triage patients and inform clinical decisions ScienceDaily 2018
9) Oh J et al ldquoA Generalizable Data-Driven Approach to Predict Daily Risk of Clostridium difficile Infection at Two Large Academic Health Centersrdquo ICHE 39 No 4 (2018) 425-433
Nature1 ndash August 2016
ML algorithm can accurately
differentiate two types of lung
cancers and predict patient
survival time
IEEE2 ndash February 2017
ML model predicts relapse rates for
acute myelogenous leukemia with
90 accuracy
arXiv3 ndash March 2017
ML approach to imaging analytics
can identify metastasized breast
cancer with accuracy rates rivaling
those of pathologists
Neurology5 ndash Sept 2017
Predictive model to identify
Parkinsonrsquos disease from claims
data achieved 73-83 accuracy in
study results
arXiv4 ndash July 2017
ML algorithm can diagnose
heart arrhythmias using data
from wearables sensors with
cardiologist-level accuracy
arXiv6 ndash November 2017
Deep-learning algorithm
outperforms Stanford radiologists
at diagnosing pneumonia from
x-rays
American College of Cardiology8 ndash
Feb 2018
Model predicts heart attack
diagnosis among patients in the ED
with chest pain with ~90 accuracy
Nature7 ndash February 2018
ML algorithm can predict a
personrsquos cardiovascular risk
factors using eye scans
Infection Control Hospital
Epidemiology9 ndash April 2018
ML model predicts Clostridium
difficile infection (CDI) earlier
than current diagnostic methods
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
25
Driving the Decision Machine
1Acquire Data
bull Ensure all models have
a sponsor and regular
evaluation against goals
bull All models require
tune-ups as the
environment changes
Monitor Performance
and Revisit
42Train or Refine
the Model
bull The more the better
buthellip
bull High-quality well-
governed data pay
enormous dividends
bull Diverse cross-continuum
sources improve
predictions and
robustness of models
bull Decide what features
are important (eg
explainability)
bull Apply practical and
ethical constraints
bull Evaluate under
realistic conditions
(silent mode real-time
incomplete data)
bull Embed insights directly
into applications at key
decision points
bull Reengineer processes
as warranted
bull Consider the ldquo5 rightsrdquo
of decision support
3Incorporate Models
into Workflow
Hint The Analytics Is the Easy Part
Begin with
bull Explicit leadership agreement on outcome goals expected benefit mechanisms
bull Participation from empowered representatives of impacted process
bull External experiencemdashlearn from others
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
26
New Data Sources
Mainstream Systems Cover Only a Fraction of Predictive Power
Acquire Data
Sources Nash DB ldquoPopulation Health Why It Mattersrdquo Essentials in Population Health Childrenrsquos
Hospital Association Thomas Jefferson University October 2016 Health Care IT Advisor research and
analysis
1) RPM = Remote patient monitoring
EMR(s)
Lab Systems
Pharmacy Systems
Imaging
Medical Claims
Prescription Drug Claims
Billing and Supply Chain
Scheduling Systems
Medical
Care Data
10
Sensors RPM1
mHealth Apps
Patient-Reported
Outcomes
Genetics
Socioeconomic Data
Patient Questionnaires
Human
Biology
Data
20
Lifestyle
and
Behavior
Data
50
Social and
Environmental
Data
20
Data Sources by Influence on Health Outcomes
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
27
Big Data a Technology and a Philosophy
ldquoTherersquos Gold in Them Thar Hillsrdquo
Acquire Data
Sources Cloudera ldquoInova Translational Medicine Institute Partners with Cloudera to Advance Genome-Based Machine
Learning Initiatives and Save Livesrdquo PR Newswire June 2017 Health Care IT Advisor research and analysis
1) Originally coined by Doug Laney of the Meta Group
Assumes Value in the Data
Based on a premise that there are
valuable insights that can be distilled
from the mountains of data captured
by machines
Characterised by the 3 Vs1
bull Volume
bull Velocity
bull Variety
Places greater emphasis on
unstructured data
Free text images streams of sensor
data are all fair targets for analysis
Big data is the difference
between heart rates and
heart beats
Case in Brief Inova Health System
bull Inovarsquos Translational Medicine Institute
(ITMI) is assembling one of the worldrsquos
largest whole genome sequence
databases connected to patient
information in a health care system
bull Researchers use the Cloudera
machine learning platform to analyze
terabytes of clinical and genomic data
and identify genetic links to diseases
bull Insights are used to develop
personalized treatment plans for
patients in collaboration with the
treating physician
Resource Big Data in Health Care
Strengths challenges and pragmatic
guidelines for those considering the addition
of big data capabilities to their IT portfolio
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
30
How Machines Learn
Similar Methods Differing Goals
Build the Model
Source Advisory Board research and analysis
Input Data
bull The more the better
bull Labeled with key indicators
Adjust the Model
bull Automated trial and error
bull Cross-breeding successful
candidates
Repeat
Supervised Learning
Evaluate models against the ldquocorrectrdquo answers
provided with the data The learning process
focuses on minimizing errors
Unsupervised Learning
Evaluate models against an abstract algorithmic
goal Example Divide this population of diabetics
into a handful of subpopulations that have similar
future risk profiles
Reinforcement Learning
Evaluate models by giving a reward for
desirable outcomes and a penalty for negative
outcomes Models are trained to maximize their
total rewards over timeReady for Review
Evaluate Performance
bull Against what goal
bull Forces explicit
tradeoffs (eg cost
vs quality)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
32
Black Boxes Shades of Grey
Differing Views on the Importance of Explainability
Incorporate into Workflow
Mike Wall PharmD MBA
Chief Analytics Officer
The University of Chicago Medicine
The opaqueness of black-box models is
really scary When we canrsquot tell a
clinician why a patient wonrsquot do well on a
certain medication that can be a tough
thing to blindly trust from a clinical
perspective We want to reach a place
where we augment our practice
through AI rather than dictatingrdquo
Nigam Shah MBBS PhD
Associate Professor of Medicine
(Biomedical Informatics)
Stanford University
I am pushing to evaluate these models
via a similar set of mechanisms as RCT1
or AB testing We should evaluate
against the current status-quo and do a
beforeafter studyhellipWe have to convince
ourselves of the utility of these models
We should focus on utility before
explainabilityrdquo
Surgery is the
best option ()
Data
Black Box Model
1) RCT = Randomized controlled trial
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
33
Workflow Is Key to Adoption
Incorporate into Workflow
The Right Information
To the Right Person
In the Right Form
By the Right Channel
At the Right Time
Case in Brief Mayo Breast Oncology
Clinic Trials Recruitment
bull Up to 100 concurrent drug therapy clinical trials are
available to Mayo Clinic breast cancer patients making
it challenging to consistently identify candidates
bull New clinical trial matching technology developed in
collaboration with IBM Watson Health evaluates charts
to identify eligible patients
bull Launched in January 2016 as a point-of-care solution
for oncology providers initial adoption was low due to
time pressures and fell to zero by February
bull Rework by health systems engineers resulted in a July
2016 relaunch screening moved to occur in
advance of patient visits so clinicians know of trial
matches before opening a patientrsquos chart
bull New process enrolls significantly more patients in
clinical trials than the prior manual process success led
to additional coordinator funding
The Five Rights of
Clinical Decision Support
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
34
Health Carersquos Field Agents
Medical Devices Become Intelligent Partners
Incorporate into Workflow
1) EKG = Electrocardiogram
ME
DT
RO
NIC
MIN
IME
D 6
70G
Case in Brief Artificial Pancreas and Smart Infusion Pumpsmdash
Medtronic MiniMed Connect
bull SMARTGUARD mimics some functions of a healthy pancreas
measures and predicts glucose level drift and adjusts dosing
bull Insulin pump and continuous glucose monitoring can send data directly
to smartphone
bull Trials showed a 44 reduction in hypoglycemia and reduction in
average A1C values from 74 to 69
Case in Brief AliveCor KardiaMobile
bull Portable consumer-grade single-lead EKG1 retails for $100
bull Intended for self-capture of intermittent cardiac events such as atrial
fibrillation (AFIB)
bull Smartphone-based machine learning validates quality of trace
assesses possibility of AFIB classifies normal abnormal sinus rhythm
bull EKGs can be sent to a board-certified cardiologist for interpretation
for $19
Sources alivecorcom Medtroniccom Health Care IT Advisor research and
analysis
AliveC
orK
ard
iaM
obile
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
25
Driving the Decision Machine
1Acquire Data
bull Ensure all models have
a sponsor and regular
evaluation against goals
bull All models require
tune-ups as the
environment changes
Monitor Performance
and Revisit
42Train or Refine
the Model
bull The more the better
buthellip
bull High-quality well-
governed data pay
enormous dividends
bull Diverse cross-continuum
sources improve
predictions and
robustness of models
bull Decide what features
are important (eg
explainability)
bull Apply practical and
ethical constraints
bull Evaluate under
realistic conditions
(silent mode real-time
incomplete data)
bull Embed insights directly
into applications at key
decision points
bull Reengineer processes
as warranted
bull Consider the ldquo5 rightsrdquo
of decision support
3Incorporate Models
into Workflow
Hint The Analytics Is the Easy Part
Begin with
bull Explicit leadership agreement on outcome goals expected benefit mechanisms
bull Participation from empowered representatives of impacted process
bull External experiencemdashlearn from others
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
26
New Data Sources
Mainstream Systems Cover Only a Fraction of Predictive Power
Acquire Data
Sources Nash DB ldquoPopulation Health Why It Mattersrdquo Essentials in Population Health Childrenrsquos
Hospital Association Thomas Jefferson University October 2016 Health Care IT Advisor research and
analysis
1) RPM = Remote patient monitoring
EMR(s)
Lab Systems
Pharmacy Systems
Imaging
Medical Claims
Prescription Drug Claims
Billing and Supply Chain
Scheduling Systems
Medical
Care Data
10
Sensors RPM1
mHealth Apps
Patient-Reported
Outcomes
Genetics
Socioeconomic Data
Patient Questionnaires
Human
Biology
Data
20
Lifestyle
and
Behavior
Data
50
Social and
Environmental
Data
20
Data Sources by Influence on Health Outcomes
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
27
Big Data a Technology and a Philosophy
ldquoTherersquos Gold in Them Thar Hillsrdquo
Acquire Data
Sources Cloudera ldquoInova Translational Medicine Institute Partners with Cloudera to Advance Genome-Based Machine
Learning Initiatives and Save Livesrdquo PR Newswire June 2017 Health Care IT Advisor research and analysis
1) Originally coined by Doug Laney of the Meta Group
Assumes Value in the Data
Based on a premise that there are
valuable insights that can be distilled
from the mountains of data captured
by machines
Characterised by the 3 Vs1
bull Volume
bull Velocity
bull Variety
Places greater emphasis on
unstructured data
Free text images streams of sensor
data are all fair targets for analysis
Big data is the difference
between heart rates and
heart beats
Case in Brief Inova Health System
bull Inovarsquos Translational Medicine Institute
(ITMI) is assembling one of the worldrsquos
largest whole genome sequence
databases connected to patient
information in a health care system
bull Researchers use the Cloudera
machine learning platform to analyze
terabytes of clinical and genomic data
and identify genetic links to diseases
bull Insights are used to develop
personalized treatment plans for
patients in collaboration with the
treating physician
Resource Big Data in Health Care
Strengths challenges and pragmatic
guidelines for those considering the addition
of big data capabilities to their IT portfolio
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
30
How Machines Learn
Similar Methods Differing Goals
Build the Model
Source Advisory Board research and analysis
Input Data
bull The more the better
bull Labeled with key indicators
Adjust the Model
bull Automated trial and error
bull Cross-breeding successful
candidates
Repeat
Supervised Learning
Evaluate models against the ldquocorrectrdquo answers
provided with the data The learning process
focuses on minimizing errors
Unsupervised Learning
Evaluate models against an abstract algorithmic
goal Example Divide this population of diabetics
into a handful of subpopulations that have similar
future risk profiles
Reinforcement Learning
Evaluate models by giving a reward for
desirable outcomes and a penalty for negative
outcomes Models are trained to maximize their
total rewards over timeReady for Review
Evaluate Performance
bull Against what goal
bull Forces explicit
tradeoffs (eg cost
vs quality)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
32
Black Boxes Shades of Grey
Differing Views on the Importance of Explainability
Incorporate into Workflow
Mike Wall PharmD MBA
Chief Analytics Officer
The University of Chicago Medicine
The opaqueness of black-box models is
really scary When we canrsquot tell a
clinician why a patient wonrsquot do well on a
certain medication that can be a tough
thing to blindly trust from a clinical
perspective We want to reach a place
where we augment our practice
through AI rather than dictatingrdquo
Nigam Shah MBBS PhD
Associate Professor of Medicine
(Biomedical Informatics)
Stanford University
I am pushing to evaluate these models
via a similar set of mechanisms as RCT1
or AB testing We should evaluate
against the current status-quo and do a
beforeafter studyhellipWe have to convince
ourselves of the utility of these models
We should focus on utility before
explainabilityrdquo
Surgery is the
best option ()
Data
Black Box Model
1) RCT = Randomized controlled trial
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
33
Workflow Is Key to Adoption
Incorporate into Workflow
The Right Information
To the Right Person
In the Right Form
By the Right Channel
At the Right Time
Case in Brief Mayo Breast Oncology
Clinic Trials Recruitment
bull Up to 100 concurrent drug therapy clinical trials are
available to Mayo Clinic breast cancer patients making
it challenging to consistently identify candidates
bull New clinical trial matching technology developed in
collaboration with IBM Watson Health evaluates charts
to identify eligible patients
bull Launched in January 2016 as a point-of-care solution
for oncology providers initial adoption was low due to
time pressures and fell to zero by February
bull Rework by health systems engineers resulted in a July
2016 relaunch screening moved to occur in
advance of patient visits so clinicians know of trial
matches before opening a patientrsquos chart
bull New process enrolls significantly more patients in
clinical trials than the prior manual process success led
to additional coordinator funding
The Five Rights of
Clinical Decision Support
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
34
Health Carersquos Field Agents
Medical Devices Become Intelligent Partners
Incorporate into Workflow
1) EKG = Electrocardiogram
ME
DT
RO
NIC
MIN
IME
D 6
70G
Case in Brief Artificial Pancreas and Smart Infusion Pumpsmdash
Medtronic MiniMed Connect
bull SMARTGUARD mimics some functions of a healthy pancreas
measures and predicts glucose level drift and adjusts dosing
bull Insulin pump and continuous glucose monitoring can send data directly
to smartphone
bull Trials showed a 44 reduction in hypoglycemia and reduction in
average A1C values from 74 to 69
Case in Brief AliveCor KardiaMobile
bull Portable consumer-grade single-lead EKG1 retails for $100
bull Intended for self-capture of intermittent cardiac events such as atrial
fibrillation (AFIB)
bull Smartphone-based machine learning validates quality of trace
assesses possibility of AFIB classifies normal abnormal sinus rhythm
bull EKGs can be sent to a board-certified cardiologist for interpretation
for $19
Sources alivecorcom Medtroniccom Health Care IT Advisor research and
analysis
AliveC
orK
ard
iaM
obile
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
26
New Data Sources
Mainstream Systems Cover Only a Fraction of Predictive Power
Acquire Data
Sources Nash DB ldquoPopulation Health Why It Mattersrdquo Essentials in Population Health Childrenrsquos
Hospital Association Thomas Jefferson University October 2016 Health Care IT Advisor research and
analysis
1) RPM = Remote patient monitoring
EMR(s)
Lab Systems
Pharmacy Systems
Imaging
Medical Claims
Prescription Drug Claims
Billing and Supply Chain
Scheduling Systems
Medical
Care Data
10
Sensors RPM1
mHealth Apps
Patient-Reported
Outcomes
Genetics
Socioeconomic Data
Patient Questionnaires
Human
Biology
Data
20
Lifestyle
and
Behavior
Data
50
Social and
Environmental
Data
20
Data Sources by Influence on Health Outcomes
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
27
Big Data a Technology and a Philosophy
ldquoTherersquos Gold in Them Thar Hillsrdquo
Acquire Data
Sources Cloudera ldquoInova Translational Medicine Institute Partners with Cloudera to Advance Genome-Based Machine
Learning Initiatives and Save Livesrdquo PR Newswire June 2017 Health Care IT Advisor research and analysis
1) Originally coined by Doug Laney of the Meta Group
Assumes Value in the Data
Based on a premise that there are
valuable insights that can be distilled
from the mountains of data captured
by machines
Characterised by the 3 Vs1
bull Volume
bull Velocity
bull Variety
Places greater emphasis on
unstructured data
Free text images streams of sensor
data are all fair targets for analysis
Big data is the difference
between heart rates and
heart beats
Case in Brief Inova Health System
bull Inovarsquos Translational Medicine Institute
(ITMI) is assembling one of the worldrsquos
largest whole genome sequence
databases connected to patient
information in a health care system
bull Researchers use the Cloudera
machine learning platform to analyze
terabytes of clinical and genomic data
and identify genetic links to diseases
bull Insights are used to develop
personalized treatment plans for
patients in collaboration with the
treating physician
Resource Big Data in Health Care
Strengths challenges and pragmatic
guidelines for those considering the addition
of big data capabilities to their IT portfolio
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
30
How Machines Learn
Similar Methods Differing Goals
Build the Model
Source Advisory Board research and analysis
Input Data
bull The more the better
bull Labeled with key indicators
Adjust the Model
bull Automated trial and error
bull Cross-breeding successful
candidates
Repeat
Supervised Learning
Evaluate models against the ldquocorrectrdquo answers
provided with the data The learning process
focuses on minimizing errors
Unsupervised Learning
Evaluate models against an abstract algorithmic
goal Example Divide this population of diabetics
into a handful of subpopulations that have similar
future risk profiles
Reinforcement Learning
Evaluate models by giving a reward for
desirable outcomes and a penalty for negative
outcomes Models are trained to maximize their
total rewards over timeReady for Review
Evaluate Performance
bull Against what goal
bull Forces explicit
tradeoffs (eg cost
vs quality)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
32
Black Boxes Shades of Grey
Differing Views on the Importance of Explainability
Incorporate into Workflow
Mike Wall PharmD MBA
Chief Analytics Officer
The University of Chicago Medicine
The opaqueness of black-box models is
really scary When we canrsquot tell a
clinician why a patient wonrsquot do well on a
certain medication that can be a tough
thing to blindly trust from a clinical
perspective We want to reach a place
where we augment our practice
through AI rather than dictatingrdquo
Nigam Shah MBBS PhD
Associate Professor of Medicine
(Biomedical Informatics)
Stanford University
I am pushing to evaluate these models
via a similar set of mechanisms as RCT1
or AB testing We should evaluate
against the current status-quo and do a
beforeafter studyhellipWe have to convince
ourselves of the utility of these models
We should focus on utility before
explainabilityrdquo
Surgery is the
best option ()
Data
Black Box Model
1) RCT = Randomized controlled trial
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
33
Workflow Is Key to Adoption
Incorporate into Workflow
The Right Information
To the Right Person
In the Right Form
By the Right Channel
At the Right Time
Case in Brief Mayo Breast Oncology
Clinic Trials Recruitment
bull Up to 100 concurrent drug therapy clinical trials are
available to Mayo Clinic breast cancer patients making
it challenging to consistently identify candidates
bull New clinical trial matching technology developed in
collaboration with IBM Watson Health evaluates charts
to identify eligible patients
bull Launched in January 2016 as a point-of-care solution
for oncology providers initial adoption was low due to
time pressures and fell to zero by February
bull Rework by health systems engineers resulted in a July
2016 relaunch screening moved to occur in
advance of patient visits so clinicians know of trial
matches before opening a patientrsquos chart
bull New process enrolls significantly more patients in
clinical trials than the prior manual process success led
to additional coordinator funding
The Five Rights of
Clinical Decision Support
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
34
Health Carersquos Field Agents
Medical Devices Become Intelligent Partners
Incorporate into Workflow
1) EKG = Electrocardiogram
ME
DT
RO
NIC
MIN
IME
D 6
70G
Case in Brief Artificial Pancreas and Smart Infusion Pumpsmdash
Medtronic MiniMed Connect
bull SMARTGUARD mimics some functions of a healthy pancreas
measures and predicts glucose level drift and adjusts dosing
bull Insulin pump and continuous glucose monitoring can send data directly
to smartphone
bull Trials showed a 44 reduction in hypoglycemia and reduction in
average A1C values from 74 to 69
Case in Brief AliveCor KardiaMobile
bull Portable consumer-grade single-lead EKG1 retails for $100
bull Intended for self-capture of intermittent cardiac events such as atrial
fibrillation (AFIB)
bull Smartphone-based machine learning validates quality of trace
assesses possibility of AFIB classifies normal abnormal sinus rhythm
bull EKGs can be sent to a board-certified cardiologist for interpretation
for $19
Sources alivecorcom Medtroniccom Health Care IT Advisor research and
analysis
AliveC
orK
ard
iaM
obile
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
27
Big Data a Technology and a Philosophy
ldquoTherersquos Gold in Them Thar Hillsrdquo
Acquire Data
Sources Cloudera ldquoInova Translational Medicine Institute Partners with Cloudera to Advance Genome-Based Machine
Learning Initiatives and Save Livesrdquo PR Newswire June 2017 Health Care IT Advisor research and analysis
1) Originally coined by Doug Laney of the Meta Group
Assumes Value in the Data
Based on a premise that there are
valuable insights that can be distilled
from the mountains of data captured
by machines
Characterised by the 3 Vs1
bull Volume
bull Velocity
bull Variety
Places greater emphasis on
unstructured data
Free text images streams of sensor
data are all fair targets for analysis
Big data is the difference
between heart rates and
heart beats
Case in Brief Inova Health System
bull Inovarsquos Translational Medicine Institute
(ITMI) is assembling one of the worldrsquos
largest whole genome sequence
databases connected to patient
information in a health care system
bull Researchers use the Cloudera
machine learning platform to analyze
terabytes of clinical and genomic data
and identify genetic links to diseases
bull Insights are used to develop
personalized treatment plans for
patients in collaboration with the
treating physician
Resource Big Data in Health Care
Strengths challenges and pragmatic
guidelines for those considering the addition
of big data capabilities to their IT portfolio
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
30
How Machines Learn
Similar Methods Differing Goals
Build the Model
Source Advisory Board research and analysis
Input Data
bull The more the better
bull Labeled with key indicators
Adjust the Model
bull Automated trial and error
bull Cross-breeding successful
candidates
Repeat
Supervised Learning
Evaluate models against the ldquocorrectrdquo answers
provided with the data The learning process
focuses on minimizing errors
Unsupervised Learning
Evaluate models against an abstract algorithmic
goal Example Divide this population of diabetics
into a handful of subpopulations that have similar
future risk profiles
Reinforcement Learning
Evaluate models by giving a reward for
desirable outcomes and a penalty for negative
outcomes Models are trained to maximize their
total rewards over timeReady for Review
Evaluate Performance
bull Against what goal
bull Forces explicit
tradeoffs (eg cost
vs quality)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
32
Black Boxes Shades of Grey
Differing Views on the Importance of Explainability
Incorporate into Workflow
Mike Wall PharmD MBA
Chief Analytics Officer
The University of Chicago Medicine
The opaqueness of black-box models is
really scary When we canrsquot tell a
clinician why a patient wonrsquot do well on a
certain medication that can be a tough
thing to blindly trust from a clinical
perspective We want to reach a place
where we augment our practice
through AI rather than dictatingrdquo
Nigam Shah MBBS PhD
Associate Professor of Medicine
(Biomedical Informatics)
Stanford University
I am pushing to evaluate these models
via a similar set of mechanisms as RCT1
or AB testing We should evaluate
against the current status-quo and do a
beforeafter studyhellipWe have to convince
ourselves of the utility of these models
We should focus on utility before
explainabilityrdquo
Surgery is the
best option ()
Data
Black Box Model
1) RCT = Randomized controlled trial
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
33
Workflow Is Key to Adoption
Incorporate into Workflow
The Right Information
To the Right Person
In the Right Form
By the Right Channel
At the Right Time
Case in Brief Mayo Breast Oncology
Clinic Trials Recruitment
bull Up to 100 concurrent drug therapy clinical trials are
available to Mayo Clinic breast cancer patients making
it challenging to consistently identify candidates
bull New clinical trial matching technology developed in
collaboration with IBM Watson Health evaluates charts
to identify eligible patients
bull Launched in January 2016 as a point-of-care solution
for oncology providers initial adoption was low due to
time pressures and fell to zero by February
bull Rework by health systems engineers resulted in a July
2016 relaunch screening moved to occur in
advance of patient visits so clinicians know of trial
matches before opening a patientrsquos chart
bull New process enrolls significantly more patients in
clinical trials than the prior manual process success led
to additional coordinator funding
The Five Rights of
Clinical Decision Support
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
34
Health Carersquos Field Agents
Medical Devices Become Intelligent Partners
Incorporate into Workflow
1) EKG = Electrocardiogram
ME
DT
RO
NIC
MIN
IME
D 6
70G
Case in Brief Artificial Pancreas and Smart Infusion Pumpsmdash
Medtronic MiniMed Connect
bull SMARTGUARD mimics some functions of a healthy pancreas
measures and predicts glucose level drift and adjusts dosing
bull Insulin pump and continuous glucose monitoring can send data directly
to smartphone
bull Trials showed a 44 reduction in hypoglycemia and reduction in
average A1C values from 74 to 69
Case in Brief AliveCor KardiaMobile
bull Portable consumer-grade single-lead EKG1 retails for $100
bull Intended for self-capture of intermittent cardiac events such as atrial
fibrillation (AFIB)
bull Smartphone-based machine learning validates quality of trace
assesses possibility of AFIB classifies normal abnormal sinus rhythm
bull EKGs can be sent to a board-certified cardiologist for interpretation
for $19
Sources alivecorcom Medtroniccom Health Care IT Advisor research and
analysis
AliveC
orK
ard
iaM
obile
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
30
How Machines Learn
Similar Methods Differing Goals
Build the Model
Source Advisory Board research and analysis
Input Data
bull The more the better
bull Labeled with key indicators
Adjust the Model
bull Automated trial and error
bull Cross-breeding successful
candidates
Repeat
Supervised Learning
Evaluate models against the ldquocorrectrdquo answers
provided with the data The learning process
focuses on minimizing errors
Unsupervised Learning
Evaluate models against an abstract algorithmic
goal Example Divide this population of diabetics
into a handful of subpopulations that have similar
future risk profiles
Reinforcement Learning
Evaluate models by giving a reward for
desirable outcomes and a penalty for negative
outcomes Models are trained to maximize their
total rewards over timeReady for Review
Evaluate Performance
bull Against what goal
bull Forces explicit
tradeoffs (eg cost
vs quality)
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
32
Black Boxes Shades of Grey
Differing Views on the Importance of Explainability
Incorporate into Workflow
Mike Wall PharmD MBA
Chief Analytics Officer
The University of Chicago Medicine
The opaqueness of black-box models is
really scary When we canrsquot tell a
clinician why a patient wonrsquot do well on a
certain medication that can be a tough
thing to blindly trust from a clinical
perspective We want to reach a place
where we augment our practice
through AI rather than dictatingrdquo
Nigam Shah MBBS PhD
Associate Professor of Medicine
(Biomedical Informatics)
Stanford University
I am pushing to evaluate these models
via a similar set of mechanisms as RCT1
or AB testing We should evaluate
against the current status-quo and do a
beforeafter studyhellipWe have to convince
ourselves of the utility of these models
We should focus on utility before
explainabilityrdquo
Surgery is the
best option ()
Data
Black Box Model
1) RCT = Randomized controlled trial
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
33
Workflow Is Key to Adoption
Incorporate into Workflow
The Right Information
To the Right Person
In the Right Form
By the Right Channel
At the Right Time
Case in Brief Mayo Breast Oncology
Clinic Trials Recruitment
bull Up to 100 concurrent drug therapy clinical trials are
available to Mayo Clinic breast cancer patients making
it challenging to consistently identify candidates
bull New clinical trial matching technology developed in
collaboration with IBM Watson Health evaluates charts
to identify eligible patients
bull Launched in January 2016 as a point-of-care solution
for oncology providers initial adoption was low due to
time pressures and fell to zero by February
bull Rework by health systems engineers resulted in a July
2016 relaunch screening moved to occur in
advance of patient visits so clinicians know of trial
matches before opening a patientrsquos chart
bull New process enrolls significantly more patients in
clinical trials than the prior manual process success led
to additional coordinator funding
The Five Rights of
Clinical Decision Support
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
34
Health Carersquos Field Agents
Medical Devices Become Intelligent Partners
Incorporate into Workflow
1) EKG = Electrocardiogram
ME
DT
RO
NIC
MIN
IME
D 6
70G
Case in Brief Artificial Pancreas and Smart Infusion Pumpsmdash
Medtronic MiniMed Connect
bull SMARTGUARD mimics some functions of a healthy pancreas
measures and predicts glucose level drift and adjusts dosing
bull Insulin pump and continuous glucose monitoring can send data directly
to smartphone
bull Trials showed a 44 reduction in hypoglycemia and reduction in
average A1C values from 74 to 69
Case in Brief AliveCor KardiaMobile
bull Portable consumer-grade single-lead EKG1 retails for $100
bull Intended for self-capture of intermittent cardiac events such as atrial
fibrillation (AFIB)
bull Smartphone-based machine learning validates quality of trace
assesses possibility of AFIB classifies normal abnormal sinus rhythm
bull EKGs can be sent to a board-certified cardiologist for interpretation
for $19
Sources alivecorcom Medtroniccom Health Care IT Advisor research and
analysis
AliveC
orK
ard
iaM
obile
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
32
Black Boxes Shades of Grey
Differing Views on the Importance of Explainability
Incorporate into Workflow
Mike Wall PharmD MBA
Chief Analytics Officer
The University of Chicago Medicine
The opaqueness of black-box models is
really scary When we canrsquot tell a
clinician why a patient wonrsquot do well on a
certain medication that can be a tough
thing to blindly trust from a clinical
perspective We want to reach a place
where we augment our practice
through AI rather than dictatingrdquo
Nigam Shah MBBS PhD
Associate Professor of Medicine
(Biomedical Informatics)
Stanford University
I am pushing to evaluate these models
via a similar set of mechanisms as RCT1
or AB testing We should evaluate
against the current status-quo and do a
beforeafter studyhellipWe have to convince
ourselves of the utility of these models
We should focus on utility before
explainabilityrdquo
Surgery is the
best option ()
Data
Black Box Model
1) RCT = Randomized controlled trial
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
33
Workflow Is Key to Adoption
Incorporate into Workflow
The Right Information
To the Right Person
In the Right Form
By the Right Channel
At the Right Time
Case in Brief Mayo Breast Oncology
Clinic Trials Recruitment
bull Up to 100 concurrent drug therapy clinical trials are
available to Mayo Clinic breast cancer patients making
it challenging to consistently identify candidates
bull New clinical trial matching technology developed in
collaboration with IBM Watson Health evaluates charts
to identify eligible patients
bull Launched in January 2016 as a point-of-care solution
for oncology providers initial adoption was low due to
time pressures and fell to zero by February
bull Rework by health systems engineers resulted in a July
2016 relaunch screening moved to occur in
advance of patient visits so clinicians know of trial
matches before opening a patientrsquos chart
bull New process enrolls significantly more patients in
clinical trials than the prior manual process success led
to additional coordinator funding
The Five Rights of
Clinical Decision Support
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
34
Health Carersquos Field Agents
Medical Devices Become Intelligent Partners
Incorporate into Workflow
1) EKG = Electrocardiogram
ME
DT
RO
NIC
MIN
IME
D 6
70G
Case in Brief Artificial Pancreas and Smart Infusion Pumpsmdash
Medtronic MiniMed Connect
bull SMARTGUARD mimics some functions of a healthy pancreas
measures and predicts glucose level drift and adjusts dosing
bull Insulin pump and continuous glucose monitoring can send data directly
to smartphone
bull Trials showed a 44 reduction in hypoglycemia and reduction in
average A1C values from 74 to 69
Case in Brief AliveCor KardiaMobile
bull Portable consumer-grade single-lead EKG1 retails for $100
bull Intended for self-capture of intermittent cardiac events such as atrial
fibrillation (AFIB)
bull Smartphone-based machine learning validates quality of trace
assesses possibility of AFIB classifies normal abnormal sinus rhythm
bull EKGs can be sent to a board-certified cardiologist for interpretation
for $19
Sources alivecorcom Medtroniccom Health Care IT Advisor research and
analysis
AliveC
orK
ard
iaM
obile
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
33
Workflow Is Key to Adoption
Incorporate into Workflow
The Right Information
To the Right Person
In the Right Form
By the Right Channel
At the Right Time
Case in Brief Mayo Breast Oncology
Clinic Trials Recruitment
bull Up to 100 concurrent drug therapy clinical trials are
available to Mayo Clinic breast cancer patients making
it challenging to consistently identify candidates
bull New clinical trial matching technology developed in
collaboration with IBM Watson Health evaluates charts
to identify eligible patients
bull Launched in January 2016 as a point-of-care solution
for oncology providers initial adoption was low due to
time pressures and fell to zero by February
bull Rework by health systems engineers resulted in a July
2016 relaunch screening moved to occur in
advance of patient visits so clinicians know of trial
matches before opening a patientrsquos chart
bull New process enrolls significantly more patients in
clinical trials than the prior manual process success led
to additional coordinator funding
The Five Rights of
Clinical Decision Support
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
34
Health Carersquos Field Agents
Medical Devices Become Intelligent Partners
Incorporate into Workflow
1) EKG = Electrocardiogram
ME
DT
RO
NIC
MIN
IME
D 6
70G
Case in Brief Artificial Pancreas and Smart Infusion Pumpsmdash
Medtronic MiniMed Connect
bull SMARTGUARD mimics some functions of a healthy pancreas
measures and predicts glucose level drift and adjusts dosing
bull Insulin pump and continuous glucose monitoring can send data directly
to smartphone
bull Trials showed a 44 reduction in hypoglycemia and reduction in
average A1C values from 74 to 69
Case in Brief AliveCor KardiaMobile
bull Portable consumer-grade single-lead EKG1 retails for $100
bull Intended for self-capture of intermittent cardiac events such as atrial
fibrillation (AFIB)
bull Smartphone-based machine learning validates quality of trace
assesses possibility of AFIB classifies normal abnormal sinus rhythm
bull EKGs can be sent to a board-certified cardiologist for interpretation
for $19
Sources alivecorcom Medtroniccom Health Care IT Advisor research and
analysis
AliveC
orK
ard
iaM
obile
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
34
Health Carersquos Field Agents
Medical Devices Become Intelligent Partners
Incorporate into Workflow
1) EKG = Electrocardiogram
ME
DT
RO
NIC
MIN
IME
D 6
70G
Case in Brief Artificial Pancreas and Smart Infusion Pumpsmdash
Medtronic MiniMed Connect
bull SMARTGUARD mimics some functions of a healthy pancreas
measures and predicts glucose level drift and adjusts dosing
bull Insulin pump and continuous glucose monitoring can send data directly
to smartphone
bull Trials showed a 44 reduction in hypoglycemia and reduction in
average A1C values from 74 to 69
Case in Brief AliveCor KardiaMobile
bull Portable consumer-grade single-lead EKG1 retails for $100
bull Intended for self-capture of intermittent cardiac events such as atrial
fibrillation (AFIB)
bull Smartphone-based machine learning validates quality of trace
assesses possibility of AFIB classifies normal abnormal sinus rhythm
bull EKGs can be sent to a board-certified cardiologist for interpretation
for $19
Sources alivecorcom Medtroniccom Health Care IT Advisor research and
analysis
AliveC
orK
ard
iaM
obile
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
35
Who Watches the Machines
Treat Artificial Intelligence Models Like Junior Staff
Monitor Performance and Revisit
Governance is focused on
quantifiable target
outcomes
Automated
processes need
human oversight
ldquoCircuit breakerrdquo
rules can limit
potential harm
Periodically
evaluate processes
against goals
ldquoTo err is human but to really foul things up you need a computerrdquo
William E Vaughan Columnist
Kansas City Star
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
ROAD MAP36
The Data Dilemma1
2 AI and the New Machine Age
3 How to Get There
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
41
Major Challenges to AI in Health Care
Business Challenges Legal and Ethical Challenges
Complexity Medical issues do not
appear in isolation and coordination
of care is difficult
Threat to Human Jobs Strong fear
associated with technology displacing
human workers
Workflow How do AI solutions fit
into existing workflows How much
effort is required to use it Does it
interfere or annoy unnecessarily
Competing Priorities We are still in
the midst of installing basic and
foundational systems (eg EMRs)
while addressing regulatory and
other pressing PHM initiatives
Regulation Health IT regulations
are hotly debated at the national
level Finding the right balance of
public health protection and fostering
innovation are key
Legal Juries still award large sums
when health care is not applied
properly or expected outcomes are
not achieved
Liability How do we deal with
computer failings Even if AI
approaches are statistically better
there may be liability when it fails
Human Touch How will we interact
with AI How strongly will we require
the human touch and human
compassion in health care
Cost The high costs for
developing testing certifying and
implementing can be a barrier
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
43
Cut Through the AI Clutter
Is Your Vendor Ready for AI Are You
1) Accuracy is the portion of cases a model correctly predicts
2) Sensitivity also known as recall is the probability a model can detect the condition it was designed to find
3) Specificity also known as precision is the probability a case is truly positive when the model says it is
Ask YourselfhellipAsk Your Vendorhellip
If you arenrsquot listening to what your data tells you about the present
yoursquore not ready to use it to predict the future
Health System ReadinessVendor Readiness
bull What outcomes are we trying to
impact How are they measured
today
bull Do we have access to the data and
talent we need to execute
bull How will AI interact with human
decision makers What workflows
will change
bull Who will oversee the system over
time Are there circuit breakers
bull Has their approach been deployed
with similar data and populations
bull Does using AI improve upon
conventional approaches (eg
statistical models visualization)
bull Who will manage the training and
ongoing validation of models
bull Go beyond accuracy1 get details
on sensitivity2 and specificity3
Source Health Care IT Advisor research and analysis
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
45
Analytics is an essential tool for improvement of health carersquos
economics quality of clinical care and operational efficiency
New data sources present opportunities for better decision making
but people are already overwhelmed Analytics distills raw data
into actionable insights and machine learning provides deeper
analysis and faster model development
Narrow task-focused administrative and operational
decisions are an immediate opportunity for AIML applications
Task-focused clinical applications are inevitable
Build a strong foundation in governance data-driven decision
culture and skills before you pursue advanced analytical
techniques including artificial intelligence
Analytics only delivers value when coupled with broader workflow
and process changes
Questions amp Key TakeawaysThe Decision Machine
Use our AI Cheat SheetTo explain key concepts and takeaways to non-IT leaders
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
46
Develop
Market-Leading Strategy
Enhance
Team Effectiveness
Accelerate
Performance Improvement
Letrsquos Keep the Conversation Going
bull Staff training and development
bull News digests and analysis
including the IT Forefront blog
bull Health care cheat sheets
bull Ready-made presentations
bull Presentations on critical IT-
related topics to get IT and
non-IT leaders on the same
page
bull On-call expert consultations
bull Strategic planning tools
bull Executive briefings reports
and presentations
bull Decision guides
bull Tools
bull Templates
ITSuiteEventsadvisorycom or leave a
comment in the survey
To access these resources contact us at
Next Steps Available to Advisory Board Members
To see the latest in your HCITA
membership visit advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
47
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
ArenaIOBaltimore Maryland
Michael Finn
Michael Rosenbaum
AthenahealthWatertown Massachusetts
Girish Venkatachaliah
AyasdiPalo Alto California
Jonathan Symonds
Childrenrsquos Hospital of
Eastern OntarioOttawa Ontario Canada
Christina Honeywell
CollibraNew York New York
Chris Cooper
Dan Scholler
Dignity Health San Francisco California
Dr Gurmeet Sran
Duke University
Health SystemDurham North Carolina
Dr Mark Sendak
Forecast Health Durham North Carolina
Michael Cousins
Geisinger Health SystemDanville Pennsylvania
Kirk Hanson
Health Catalyst Salt Lake City Utah
Dale Sanders
Levi Thatcher
HealthDataVizWaltham Massachusetts
Sandy Lawson
Kathy Rowell
Intermountain Healthcare Salt Lake City Utah
Dr Nathan Dean
Jeffrey Ferraro
Lonny Northrup
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
48
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
Jewish General Hospital Montreacuteal Queacutebec Canada
Dr Lawrence Rosenberg
Phil Troy
Mayo ClinicRochester Minnesota
Dr Tufia Haddad
Memorial Sloan Kettering
Cancer CenterNew York New York
Ari Caroline
Isaac Wagner
Microsoft Seattle Washington
John Doyle
Tom Lawry
Dennis Schmuland
New York PresbyterianNew York New York
Dr Peter Fleischut
Dr David Tsay
PaxataRedwood City California
Nenshad Bardoliwalla
Jayanta Bhowmik
Pieces Technologies Dallas Texas
Dr Ruben Amarasingham
Shannon Kmak
QventusLos Altos California
Mudit Garg
Venkat Mocherla
Royal Free London NHS
Foundation TrustSurrey London UK
Glenn Winteringham
SalesforceSan Francisco California
Jayesh Govindarajan
Stanford University
Medical CenterStanford California
Dr Matthew Lungren
Dr Nigam Shah
Taunton and Somerset NHS
Foundation TrustSomerset England UK
Andrew Forrest
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
49
Advisors to Our Work
With Sincere Appreciation
The Health Care IT Advisor program would like to express deep gratitude to
the individuals and organizations that shared their insights analysis and time
with us The research team would especially like to recognize the following
contributors for being particularly generous with their time and expertise
The MetroHealth SystemCleveland Ohio
Dr Robert Fergueson
Dr David Kaelber
The University of
Chicago MedicineChicago Illinois
Mike Wall
University Hospital
BirminghamBirmingham England UK
Steven Chilton
Paul Jennings
Barnaby Waters
University Hospitals of
LeicesterLeicester England UK
Andrew Carruthers
University of CambridgeCambridge England UK
Pietro Lio
University of LeedsLeeds England UK
Owen Johnson
University of North
Carolina Healthcare Chapel Hill North Carolina
Jason Burke
University of Pittsburg
Medical Center Pittsburgh Pennsylvania
Terri Mikol
Don Riefner
Dr Rasu Shreshta
Washington University
Medical CenterSt Louis Missouri
Dr Brad Racette
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
copy2018 Advisory Board bull All Rights Reserved bull advisorycom bull 36516D
LEGAL CAVEAT
Advisory Board is a division of The Advisory Board Company Advisory Board has
made efforts to verify the accuracy of the information it provides to members This
report relies on data obtained from many sources however and Advisory Board
cannot guarantee the accuracy of the information provided or any analysis based
thereon In addition Advisory Board is not in the business of giving legal medical
accounting or other professional advice and its reports should not be construed
as professional advice In particular members should not rely on any legal
commentary in this report as a basis for action or assume that any tactics
described herein would be permitted by applicable law or appropriate for a given
memberrsquos situation Members are advised to consult with appropriate professionals
concerning legal medical tax or accounting issues before implementing any of
these tactics Neither Advisory Board nor its officers directors trustees
employees and agents shall be liable for any claims liabilities or expenses
relating to (a) any errors or omissions in this report whether caused by Advisory
Board or any of its employees or agents or sources or other third parties (b) any
recommendation or graded ranking by Advisory Board or (c) failure of member
and its employees and agents to abide by the terms set forth herein
The Advisory Board Company and the ldquoArdquo logo are registered trademarks of The
Advisory Board Company in the United States and other countries Members are
not permitted to use these trademarks or any other trademark product name
service name trade name and logo of Advisory Board without prior written consent
of Advisory Board All other trademarks product names service names trade
names and logos used within these pages are the property of their respective
holders Use of other company trademarks product names service names trade
names and logos or images of the same does not necessarily constitute (a) an
endorsement by such company of Advisory Board and its products and services or
(b) an endorsement of the company or its products or services by Advisory Board
Advisory Board is not affiliated with any such company
IMPORTANT Please read the following
Advisory Board has prepared this report for the exclusive use of its members
Each member acknowledges and agrees that this report and the information
contained herein (collectively the ldquoReportrdquo) are confidential and proprietary to
Advisory Board By accepting delivery of this Report each member agrees to
abide by the terms as stated herein including the following
1 Advisory Board owns all right title and interest in and to this Report Except
as stated herein no right license permission or interest of any kind in this
Report is intended to be given transferred to or acquired by a member
Each member is authorized to use this Report only to the extent expressly
authorized herein
2 Each member shall not sell license republish or post online or otherwise this
Report in part or in whole Each member shall not disseminate or permit the
use of and shall take reasonable precautions to prevent such dissemination or
use of this Report by (a) any of its employees and agents (except as stated
below) or (b) any third party
3 Each member may make this Report available solely to those of its employees
and agents who (a) are registered for the workshop or membership program of
which this Report is a part (b) require access to this Report in order to learn
from the information described herein and (c) agree not to disclose this Report
to other employees or agents or any third party Each member shall use and
shall ensure that its employees and agents use this Report for its internal use
only Each member may make a limited number of copies solely as adequate
for use by its employees and agents in accordance with the terms herein
4 Each member shall not remove from this Report any confidential markings
copyright notices andor other similar indicia herein
5 Each member is responsible for any breach of its obligations as stated herein
by any of its employees or agents
6 If a member is unwilling to abide by any of the foregoing obligations then
such member shall promptly return this Report and all copies thereof to
Advisory Board
Health Care IT Advisor
Research DirectorGreg Kuhnen
kuhnengadvisorycom
Research TeamJacqueline Beltejar
Bethany Jones
Nouran Ragaban
Sophie Ranen
Andrew Rebhan
Program LeadershipJim Adams
Design ConsultantKevin Matovich
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
2445 M Street NW Washington DC 20037
P 2022665600 F 2022665700 advisorycom
top related