![Page 1: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease](https://reader030.vdocuments.us/reader030/viewer/2022040610/5ed14ab0a225a048a515d98c/html5/thumbnails/1.jpg)
Convolutional Neural Networks For Modeling Temporal Biomarkers And
Disease Predictions
Narges Razavian New York University Langone Medical Center
GTC 2017
In collaboration with: David SontagPhD, Saul BleckerMD, Ann-Marie SchmidtMD, Enrico BertiniPhD, Rahul Krishnan, YD Choi, Josua Krause, Somesh Nigam, Aaron Smith-McLallen, Ravi Chawla
![Page 2: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease](https://reader030.vdocuments.us/reader030/viewer/2022040610/5ed14ab0a225a048a515d98c/html5/thumbnails/2.jpg)
Deep learning progress Healthcare world getting digital
Parallel Developments
EHR adoption by healthcare centers in the US
Error rate on Image-Net object recognition challenge
![Page 3: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease](https://reader030.vdocuments.us/reader030/viewer/2022040610/5ed14ab0a225a048a515d98c/html5/thumbnails/3.jpg)
What is captured in the EHR?
Source: healthcare.gov
![Page 4: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease](https://reader030.vdocuments.us/reader030/viewer/2022040610/5ed14ab0a225a048a515d98c/html5/thumbnails/4.jpg)
Healthcare has joined the data-rich world
![Page 5: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease](https://reader030.vdocuments.us/reader030/viewer/2022040610/5ed14ab0a225a048a515d98c/html5/thumbnails/5.jpg)
Moving from Treatment to Prevention
Challenges: Each Individual has a different ‘healthy’ baseline.
- Temporal Patterns/Trends are predictive Each biomarker varies at a different speed in our bodies Measurements are sparse, asynchronous and correlated Many correlated outcomes are observed per patient
- Can we leverage this correlation?
![Page 6: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease](https://reader030.vdocuments.us/reader030/viewer/2022040610/5ed14ab0a225a048a515d98c/html5/thumbnails/6.jpg)
Biomarkers and Outcomes
Biomarkers measurements
over time
![Page 7: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease](https://reader030.vdocuments.us/reader030/viewer/2022040610/5ed14ab0a225a048a515d98c/html5/thumbnails/7.jpg)
Biomarkers and Outcomes
Biomarkers measurements
over time
Phenotype (diseases) over time
![Page 8: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease](https://reader030.vdocuments.us/reader030/viewer/2022040610/5ed14ab0a225a048a515d98c/html5/thumbnails/8.jpg)
Biomarkers and Outcomes
Biomarkers measurements
over time
Phenotype (diseases) over time
![Page 9: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease](https://reader030.vdocuments.us/reader030/viewer/2022040610/5ed14ab0a225a048a515d98c/html5/thumbnails/9.jpg)
Biomarkers and Outcomes
Biomarkers measurements
over time
Phenotype (diseases) over time
![Page 10: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease](https://reader030.vdocuments.us/reader030/viewer/2022040610/5ed14ab0a225a048a515d98c/html5/thumbnails/10.jpg)
Step 1 Learn each biomarker from other biomarkers time-series
![Page 11: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease](https://reader030.vdocuments.us/reader030/viewer/2022040610/5ed14ab0a225a048a515d98c/html5/thumbnails/11.jpg)
Kernel Regression
Observations
X
(Mea
sure
men
t Ti
me-
Ser
ies)
Time Not Observed Want to estimate
![Page 12: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease](https://reader030.vdocuments.us/reader030/viewer/2022040610/5ed14ab0a225a048a515d98c/html5/thumbnails/12.jpg)
Kernel Regression
Observations
X
(Mea
sure
men
t Ti
me-
Ser
ies)
Time Not Observed Want to estimate
E[X(v)]= xP(x |∫ t = v,Xtrain )dx
E[X | t = v,Xtrain ]= x∫ P(x, t = v | Xtrain)P(t = v | Xtrain)
dx
E[X | t = v,Xtrain ]= x∫K(x − xi,v− ti )
xi ,ti
∑
K(v− ti )ti
∑dx
![Page 13: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease](https://reader030.vdocuments.us/reader030/viewer/2022040610/5ed14ab0a225a048a515d98c/html5/thumbnails/13.jpg)
Kernel Regression
Observations
X
(Mea
sure
men
t Ti
me-
Ser
ies)
Time Not Observed Want to estimate
E[X(v)]= xP(x |∫ t = v,Xtrain )dx
E[X | t = v,Xtrain ]= x∫ P(x, t = v | Xtrain)P(t = v | Xtrain)
dx
E[X | t = v,Xtrain ]= x∫K(x − xi,v− ti )
xi ,ti
∑
K(v− ti )ti
∑dx
E[X | t = v,Xtrain ]=(K ⊗ Xtrain )(v)
(K ⊗ I(Xtrain :Observed))(v)
![Page 14: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease](https://reader030.vdocuments.us/reader030/viewer/2022040610/5ed14ab0a225a048a515d98c/html5/thumbnails/14.jpg)
Use convolution framework to LEARN those kernels
![Page 15: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease](https://reader030.vdocuments.us/reader030/viewer/2022040610/5ed14ab0a225a048a515d98c/html5/thumbnails/15.jpg)
We can learn the kernel (No need for parametric forms and cross validations) Easily extendible to multivariate!
Unsupervised: All needed is (asynchronous) sequence of observations. Fast to train. Fast to apply.
Benefits
![Page 16: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease](https://reader030.vdocuments.us/reader030/viewer/2022040610/5ed14ab0a225a048a515d98c/html5/thumbnails/16.jpg)
Data:30KIndividualsfromtheoriginaltrainingset.Datasetsplitequallybetweentrain,testandvalidateset.Loss:MSE.Trainandevaluateonlyon(lab,person)withmorethan1observaGon.
Mul$variateKernelslearnedforeachinputdimension(total18)
![Page 17: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease](https://reader030.vdocuments.us/reader030/viewer/2022040610/5ed14ab0a225a048a515d98c/html5/thumbnails/17.jpg)
More details in our ICLR paper
Narges Razavian, David Sontag Temporal Convolutional Neural Networks for Diagnosis from Lab Tests http://arxiv.org/abs/1511.07938 Open Source code available (torch/lua implementation): https://github.com/clinicalml/deepDiagnosis
![Page 18: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease](https://reader030.vdocuments.us/reader030/viewer/2022040610/5ed14ab0a225a048a515d98c/html5/thumbnails/18.jpg)
Step 2 Predict 200+ correlated outcomes using multi-resolution convolutional neural networks and multi-task learning
![Page 19: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease](https://reader030.vdocuments.us/reader030/viewer/2022040610/5ed14ab0a225a048a515d98c/html5/thumbnails/19.jpg)
Multi-Resolution Convolution Networks The Architecture - model (1)
![Page 20: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease](https://reader030.vdocuments.us/reader030/viewer/2022040610/5ed14ab0a225a048a515d98c/html5/thumbnails/20.jpg)
Multi-Resolution Convolution Networks The Architecture - model (2)
![Page 21: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease](https://reader030.vdocuments.us/reader030/viewer/2022040610/5ed14ab0a225a048a515d98c/html5/thumbnails/21.jpg)
Prediction AUCs on the held-out test set
![Page 22: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease](https://reader030.vdocuments.us/reader030/viewer/2022040610/5ed14ab0a225a048a515d98c/html5/thumbnails/22.jpg)
More details in our JMLR paper
Narges Razavian, Jake Marcus, David Sontag, Multi-task Prediction of Disease Onsets from Longitudinal Lab Tests JMLR, 2016 http://arxiv.org/abs/1608.00647 Open Source code available (torch/lua implementation): https://github.com/clinicalml/deepDiagnosis
![Page 23: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease](https://reader030.vdocuments.us/reader030/viewer/2022040610/5ed14ab0a225a048a515d98c/html5/thumbnails/23.jpg)
Following up in clinical world • Prediction models built and deployed for
– Nurse calls and home visits for 250,000+ NYUMC patients at high risk for a number of these outcomes
– Improved documentation in EHR • Automation of mandatory visits/screening/follow-ups • Best practice alerts • Reimbursement for intense lifestyle management programs
• Extending to broader outcomes and domains
![Page 24: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease](https://reader030.vdocuments.us/reader030/viewer/2022040610/5ed14ab0a225a048a515d98c/html5/thumbnails/24.jpg)
New York University (i2b2) Database
![Page 25: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease](https://reader030.vdocuments.us/reader030/viewer/2022040610/5ed14ab0a225a048a515d98c/html5/thumbnails/25.jpg)
New York University (i2b2) Database Nuclear Medicine Procedures
Magnetic Resonance Imaging
![Page 26: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease](https://reader030.vdocuments.us/reader030/viewer/2022040610/5ed14ab0a225a048a515d98c/html5/thumbnails/26.jpg)
Conclusions • Applications of deep learning in healthcare are unlimited
• Unsupervised learning + back-propagation + deep learning can recover biomarker models from asynchronous high-dimensional time-series data
• Multi-task learning benefits prediction tasks with smaller datasets.
![Page 27: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease](https://reader030.vdocuments.us/reader030/viewer/2022040610/5ed14ab0a225a048a515d98c/html5/thumbnails/27.jpg)
Thanks!
Questions/comments: [email protected]
Open Source Package: https://github.com/clinicalml/deepDiagnosis
![Page 28: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease](https://reader030.vdocuments.us/reader030/viewer/2022040610/5ed14ab0a225a048a515d98c/html5/thumbnails/28.jpg)