supervised machine learning & the automated-ml platform
TRANSCRIPT
Hooman H. Rashidi, MD, MS
Professor & Vice Chair of GME
Vice Chair of Informatics &
Computational Pathology
University of California, Davis School of Medicine
Supervised Machine Learning & the Automated-ML platform MILO (Machine Intelligence Learning Optimizer)
My background
• Practicing Physician • Clinical Hematopathologist
• My graduate studies (UCSD) was in Bioinformatics (Masters Degree)
• Extensive background in Bioinformatics
• Hence the strong interest in Artificial Intelligence (AI) and Machine Learning (ML)
• PI of multiple ML research studies• 30+ members within the research
group
Disclosures
• Systems And Methods For Machine Learning-based Identification Of Acute Kidney Injury In Trauma Surgery And Burned Patients (Ea r l y P r e d i c t i o n O f A k i W i t h M a c h i n e L e a r n i n g ) : C o - i nv e n t o rs ( H o o m a n H . R a s h i d i M D M S & N a m Tra n P h D )
• U n i ve rs i t y O f C a l i fo r n i a I n t e l l e c t ua l P ro p e r t y
• Systems And Methods For Machine Learning-based Identification Of Sepsis (Ea r l y P r e d i c t i o n O f S e p s i s W i t h M a c h i n e L e a r n i n g ) : C o - i nv e n t o rs ( H o o m a n H . R a s h i d i M D M S & N a m Tra n P h D )
• U n i ve rs i t y O f C a l i fo r n i a I n t e l l e c t ua l P ro p e r t y
• Systems And Methods For Automated Machine Learning (M I LO: M a c h i n e I n t e l l i g e n c e L e a r n i n g O p t i m i ze r )
• U n i ve rs i t y O f C a l i fo r n i a I n t e l l e c t ua l P ro p e r t y ( Pa te nt Pe n d i ng )
• C o - i nve nto rs O f M I LO
• H o o m a n H . R a s h i d i M D M S
• S a m e r A l b a h ra M D
• N a m Tra n P h D
Talk Outline
• Machine Learning (ML) overview• ML classification
• Supervised Machine Learning (ML)
• Non-Image ML models: • Our AKI ML studies
• Automated Machine Learning• MILO: Machine Intelligence Learning
Optimizer
What is Artificial Intelligence / Machine Learning?
Artificial Intelligence
What is Artificial Intelligence / Machine Learning?
Artificial Intelligence
Machine Learning
“AI is the capability for machines to imitate intelligent human behavior, while ML is an application of AI that allows computer systems to automatically learn from experience without explicit programming. Paraphrasing Arthur Samuel and others, ML models are constructed by a set of data points and trained through mathematical and statistical approaches that ultimately enable prediction of new previously unseen data without being explicitly programmed to do so”
Rashidi HH, et al. Academic Pathology, 2019
Human Learning versus Machine Learning
Human learns through “experiences” and forms neuronal connections to help recall
Machine learns by experiences AKA “DATA” to build neuronal connections to be able to recall
Examples of AI and ML in our daily life
• Siri and Alexa• spam filtering• photo organizers• Facebook or LinkedIn people connectors• Amazon recommendations, etc.
Is there a difference in machine learning for medical
applications?
YES, Absolutely !
What is so different about our field (medicine) ?
• Practice of medicine is still a balance between art and science • most fields are experience driven
• Since the gold standard for how we practice is based on expertise that is experience driven, the data collected will have more variations
• Ultimately increases the chance of interobserver variability
• Hence the data sets used in our fields are not as easily reproducible as in other fields that employ machine learning
What ML approaches are used the most in medicine / pathology?
• Supervised Machine Learning
Rashidi, HH et al. Academic Pathology, 2019
The “real world” can be a tough place!
The “real world” can be a tough place!
• MD Anderson partners with IBMWatson to use “Oncology ExpertAdvisor” for targeting cancer therapy.
The “real world” can be a tough place!
• MD Anderson partners with IBMWatson to use “Oncology ExpertAdvisor” for targeting cancer therapy.
• “A new era of computing has emerged,in which cognitive systems“understand” the context within users’questions, uncover answers from BigData, and improve in performance bycontinuously learning fromexperiences”
The “real world” can be a tough place!
The “real world” can be a tough place!
$62 million wasted without achieving goals“Treating cancer is more complex than winning a trivia game, and the “vast universe of medical knowledge” may not be as significant as purveyors of artificial intelligence make it out to be…”
https://www.healthnewsreview.org/2017/02/md-anderson-cancer-centers-ibm-watson-project-fails-journalism-related/
WHERE DO WE GO NOW?What are some potential ”safe” applications of AI/ML in health care?
Realistic Opportunities for Healthcare AI/ML?
AI/ML Enhanced Detection of Burn Related AKI: A Proof of ConceptTran NK, Sen S, Palmieri TL, Lima K, Falwell S, Wajda J, Rashidi H. Burns 2019
• Goal: To build Models that predict Acute kidney Injury (AKI)
Current Standard for AKI diagnosis
• Kidney Disease and Improving Global Outcome (KDIGO)• Based on Serial Creatinine (Cr) and Urine Outputs (UOP)• Takes days (since it’s on serial Cr and UOP measurements)• Sensitivity in 50s
Here comes NGAL to the rescue
• NGAL (Neutrophil Gelatinase Associated Lipocalin)• Used in Europe• Reportedly in the process of being FDA cleared in US
AI/ML Enhanced Detection of Burn Related AKI: A Proof of ConceptTran NK, Sen S, Palmieri TL, Lima K, Falwell S, Wajda J, Rashidi H. Burns 2019
The ”Burns” population proof of concept
• Showed that ML (specifically a K-NN model) can enhance NGAL’s performance for
• By combining it with other markers • BNP• Cr• UOP
• Sensitivity and accuracy in low 90s
Our Follow up AKI ML study
• Can the Burns-derived AKI ML model predict AKI in Non-Burn Trauma patient population
Early Recognition of Burn- and trauma-Related Acute Kidney injury: A pilot comparison of Machine Learning techniques. Hooman H. Rashidi*, Soman Sen, Tina L. Palmieri, Thomas Blackmon, Jeffery Wajda & Nam K. Tran*. Nature Scientific Reports Jan 2020
Rashidi H. et. al. Nature’s Scientific Reports, 2020
Rashidi H. et. al. Nature’s Scientific Reports, 2020
Creatinine + UOP vs Creatinine Only vs UOP Only
What Happens when you Introduce NGAL
Rashidi H. et. al. Nature’s Scientific Reports, 2020
Rashidi H. et. al. Nature’s Scientific Reports, 2020
In summary
• The AKI ML models trained on the Burn Population were able to predict AKI in Non-Burn trauma and Burn patient populations
• ML enhances the predictive capability of NGAL and NGAL combined with other markers (esp. Cr and BNP)
• Most importantly:• The AI/ML algorithm helped predict AKI 61.8 (32.5) hours faster than the
KDIGO criteria for burn and non-burned trauma patients
Rashidi H. et. al. Nature’s Scientific Reports, 2020
AI/ML Real World Application for Burn AKI?Tran N et. al. Burns 2019Rashidi H. et. al. Nature’s Scientific Reports, 2020
AI/ML Real World Application for Burn AKI?
~24 hours earlier!
Tran N et. al. Burns 2019Rashidi H. et. al. Nature’s Scientific Reports, 2020
Can this Machine Learning process that was used in this study be automated?
•So that ALL investigators can have EASY access to these Machine Learning methods
•How to automate this process so that• No ML expertise is required• No programming or Software Engineering
background is needed
M I L O
Some Basic Facts:
• Artificial intelligence / Machine learning (ML) is a very powerful tool
• ML is starting to get incorporated into various aspects of health care and health science disciplines
Rashidi, HH et al. Academic Pathology, 2019
Current Challenges with ML:
• Can be intimidating • requires a team with
ML, statistics and programming expertise
• Can be very time consuming and not easily accessible
• Final ML model can be very challenging to deploy
• requires software engineering expertise to make an App or Website
Imagine a world where Artificial Intelligence (AI) / Machine Learning (ML) studies are as EASY as using a website on your laptop or even your smart phone
NO ML or Statistics expertise needed
NO Software engineering expertise needed
NO Programming required
It’s just plain easy to do !!!
OUR SOLUTION: MILO
MILO : Machine Intelligence Learning Optimizer
Your Fully Automated Machine Learning (Auto-ML) Solution
MILO makes AI/ML accessible for ALL
No Coding, No Programming No Machine Learning expertise required
All the heavy lifting is done by MILO!
MILO’s key highlights?
Expedites the machine learning process
Drastically reduces the time required to complete an ML project• Much faster than the
current traditional ML approach
Improves the performance outcome of the machine learning models
Builds a much larger number of ML models compared to
the current ML approachFinds the best ML model for a given study compared to the
current ML approach
A very simple User Interface (UI)
MILO’s Web-App makes the AI/ML study super easy to
operateAll the heavy lifting is done
through MILO’s UI
MILO Expedites the Machine Learning Process
• Traditional ML study approach• Acute Kidney Injury (AKI)
project required ~400 man hours (4 months) to complete
• Sepsis project required ~300 man hours (3 months) to complete
• MILO’s Auto-ML Approach• Acute Kidney Injury (AKI)
project through MILO was completed in <20 hours
• Sepsis project through MILO was completed in <18 hours
Tran et. al. Burns 2019.
Rashidi et. al. Nature’s Scientific Reports 2020.
MILO: A Validated Platform
Validated on 10 different IRB approved studies• Acute kidney injury (AKI) predictor (study 1): • AKI predictor (study 2): Burns Surgery• Sepsis predictor: Burns• Delayed Graft Function predictor: Kidney transplant • Cath Result - PET predictor: Cardiology• TB (11 Ag): Global Health Infectious disease• TB(31 Ag): Global Health Infectious disease• Massive Transfusion Protocol (MTP): Trauma• Step 1 Test Early intervention Predictor: Med school• Step 1 Test Late intervention Predictor: Med School
AI/ML Enhanced Detection of Burn Related AKI: A Proof of ConceptTran et. al. Burns 2019
Early Recognition Of Burn- And Trauma-related Acute Kidney Injury: A Pilot Comparison Of Machine Learning TechniquesRashidi et. al. Nature’s Scientific Reports 2020
How does MILO work?
• MILO makes NO Assumptions• About your dataset
• MILO enables each Unique Dataset to findits most suitable ML Algorithm/Model
• rather than having a data scientist and their team choosing an ML algorithm that they feel is most suitable for finding the best model for the dataset
• Follows Industry’s CRISP-DM Approach
MILO: Machine Intelligence Learning Optimizer
Curr
ent M
L Ap
proa
ch
49,940 models~400 hours (~4 months)
MORE MODELS – LESS TIME – MORE OPPORTUNITIES
ROC-AUC 94
Curr
ent M
L Ap
proa
ch
? ?Are there any other potential models?
MORE MODELS – LESS TIME – MORE OPPORTUNITIES
49,940 models~400 hours (~4 months)
ROC-AUC 94
Curr
ent M
L Ap
proa
ch
284,670 models
24 hours
MIL
O
? ?Are there any other potential models?
MORE MODELS – LESS TIME – MORE OPPORTUNITIES
49,940 models~400 hours (~4 months)
ROC-AUC 94
Curr
ent M
L Ap
proa
chM
ILO
? ?Are there any other potential models?
MORE MODELS – LESS TIME – MORE OPPORTUNITIES
MILO found six additional better models not found by the Current Traditional ML approach
49,940 models~400 hours (~4 months)
284,670 models
<24 hours
ROC-AUC 94
ROC-AUC 97 ROC-AUC 98 ROC-AUC 98
ROC-AUC 98 ROC-AUC 98 ROC-AUC 94
How does MILO achieve this?
Builds 300,000+ ML models
Optimized to find the best hyperparameters and feature sets with our proprietary embedded approach & FOLLOWS ML STUDY BEST PRACTICES (SIMILAR TO THE FOLLOWING RECENT MANUSCRIPTS)
Rashidi et. al. Nature’s Scientific Reports Jan 2020
Rashidi, HH et al. Academic Pathology, 2019
QUICK DEMO OF MILO
BUILD A MODEL TO PREDICT SEPSISWITH MILO
MILO: Machine Intelligence Learning Optimizer
Import in the Balanced Training Dataset (Dataset 1)
MILO: Machine Intelligence Learning Optimizer
Import Generalization Dataset (Dataset 2): Representing the true prevalence
MILO IN ACTIONQuick Demo
INSERT MILO MP4 VIDEO 1
MILO is also very transparent
INSERT MILO MP4 VIDEO 2
In Summary: MILO• It’s super easy to use
• No Machine Learning (ML) expertise needed
• No Programming or software engineering expertise needed• MILO is your personal ML & Software engineering Expert Team
• MILO• Follows ML Study Best practices • Finds your Best ML Model and does it Much Faster • Most importantly it’s a validated platform
Our Studies & Collaborators
Our AKI & Sepsis ML Team• Nam Tran• Tina Palmieri• Soman Sen• Samer Albahra• Jeff Wajda• Hooman Rashidi
Collaborators for our other validation studies• Joe Galante & Shawn Tejiram: MTP• Imran Khan : TB• Kuang yu Jen: DGF (transplant)• Thomas Smith & William Wung
(Cath results: Cardiology)• Erin Griffin, Sharad Jain & Kristin
Olson (USMLE step 1 studies)
MILO’s Core Team
Samer Albahra MD Hooman H. Rashidi MD MS Nam Tran, PhD
Thank you for your attention