july 10th, 2020 a brief tutorial on vikram voleti neural odes€¦ · vikram voleti a brief...
TRANSCRIPT
![Page 1: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/1.jpg)
A brief tutorial on
Neural ODEs
Vikram Voleti
July 10th, 2020
@ (virtual) Mila, Montreal, Canadavoletiv.github.io
PhD student - Mila, University of MontrealVisiting Researcher - University of Guelph
Prof. Christopher PalProf. Graham Taylor
![Page 2: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/2.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
1. Ordinary Differential Equations (ODEs)
○ Initial Value Problems
○ Numerical Integration methods
○ Fundamental theorem of ODEs
2. Neural ODEs
3. Later research
2
![Page 3: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/3.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Ordinary Differential Equations (ODEs)
1st order Ordinary Differential Equation:
3
![Page 4: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/4.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Ordinary Differential Equations (ODEs)
Initial value problem:
Many physical processes follow this template!
4
![Page 5: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/5.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Ordinary Differential Equations (ODEs)
Initial value problem:
Solution:
5
![Page 6: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/6.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Ordinary Differential Equations (ODEs)
Initial value problem:
Solution:
What if this cannot be
analytically integrated?
6
![Page 7: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/7.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Approximations to
i.e. Numerical Integration :
● Euler method
● Runge-Kutta methods
● ...
Ordinary Differential Equations (ODEs)
Initial value problem:
Solution:
7
![Page 8: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/8.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Ordinary Differential Equations (ODEs)
1st-order Runge-Kutta / Euler’s method:
Step size
Update using derivative
Initial value problem:
Solution:
https://guide.freecodecamp.org/mathematics/differential-equations/eulers-method/
8
![Page 9: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/9.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Ordinary Differential Equations (ODEs)
1st-order Runge-Kutta / Euler’s method:
Initial value problem:
Solution:
9
![Page 10: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/10.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Ordinary Differential Equations (ODEs)
https://lpsa.swarthmore.edu/NumInt/NumIntFirst.html
Initial value problem:
Solution:
1st-order Runge-Kutta / Euler’s method:
Step size matters!
10
![Page 11: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/11.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Ordinary Differential Equations (ODEs)
Initial value problem:
Solution:
Default ODE solver used in MATLAB:https://blogs.mathworks.com/loren/2015/09/23/ode-solver-selection-in-matlab/
4th-order Runge-Kutta method:
11
![Page 12: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/12.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Ordinary Differential Equations (ODEs)
Initial value problem:
Solution:
Any ODE solver of our choice!
Differential
Initial value
Initial time
Final time
12
![Page 13: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/13.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Ordinary Differential Equations (ODEs)
Initial value problem:
Solution:
Fundamental Theorem of ODEs
1. The solution curves for this differential equation completely fill the plane, and
2. Solution curves for different solutions do not intersect.
https://openreview.net/pdf?id=B1e9Y2NYvShttp://faculty.bard.edu/belk/math213/InitialValueProblems.pdf
13
![Page 14: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/14.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
1. Ordinary Differential Equations (ODEs)
○ Initial Value Problems
○ Numerical Integration methods
○ Fundamental theorem of ODEs
2. Neural ODEs (Chen et al., NeurIPS 2018)
○ Adjoint method
○ Applications
3. Later research
14
![Page 15: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/15.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Neural ODEs (Chen et al., NeurIPS 2018)
Initial value problem:
Solution:
is a neural network!
https://arxiv.org/pdf/1806.07366.pdf
Paradigm shift: whereas earlier f was pre-defined/hand-designed according to the domain,here we would like to estimate an f that suits our objective.
15
![Page 16: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/16.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Neural ODEs (Chen et al., 2018)
https://arxiv.org/pdf/1806.07366.pdf
Euler discretization
ODEsResidual
networks
Skip connection
Vector notation
https://arxiv.org/pdf/1512.03385.pdf
16
![Page 17: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/17.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Neural ODEs (Chen et al., 2018)
https://arxiv.org/pdf/1806.07366.pdf
ODEsResidual
networks
Update to reduce .Update to reduce .
Skip connection
Stacked ResBlocks
Euler discretization
Forward propagation:
How to compute this?
https://arxiv.org/pdf/1512.03385.pdf
16
![Page 18: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/18.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Neural ODEs (Chen et al., 2018)
https://arxiv.org/pdf/1806.07366.pdf
ODEs
High memory cost -
need to save all activations of all
iterations of ODESolve.
Can we do better?
Yes.
Update to reduce .
Euler discretization
Forward propagation:
17
Back-propagate through the ODE Solver!
![Page 19: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/19.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Neural ODEs (Chen et al., 2018)
https://arxiv.org/pdf/1806.07366.pdf
Adjoint method (Pontryagin et al., 1962)
18
Forward propagation:
? ?
![Page 20: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/20.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Neural ODEs (Chen et al., 2018)
https://arxiv.org/pdf/1806.07366.pdf Initial value is 0
Adjoint method (Pontryagin et al., 1962)
Forward propagation:
Back-propagation:
18
![Page 21: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/21.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Neural ODEs (Chen et al., 2018)
https://arxiv.org/pdf/1806.07366.pdf
Forward propagation:
Back-propagation:
Update to reduce .
19
![Page 22: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/22.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Neural ODEs (Chen et al., 2018)https://arxiv.org/pdf/1806.07366.pdf
Neural ODE
https://openreview.net/pdf?id=B1e9Y2NYvS
Neural ODEs describe a homeomorphism (flow).
● They preserve dimensionality.
● They form non-intersecting trajectories.
20
![Page 23: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/23.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Neural ODEs (Chen et al., 2018)https://arxiv.org/pdf/1806.07366.pdf
Neural ODE
Neural ODEs are reversible models!Just integrate forward/backward in time.
Neural ODE
21
![Page 24: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/24.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Supervised Learning
Neural ODEs (Chen et al., 2018)https://arxiv.org/pdf/1806.07366.pdf
Applications
Neural ODE
Linear
~ Replacement for ResNets
ODE-Net:
Generative Latent ModelsContinuous Normalizing Flows
22
Test error # Params Memory Time
1-layer MLP 1.60% 0.24 M - -
ResNet 0.41% 0.60 M
RK-Net 0.47% 0.22 M
ODE-Net 0.42% 0.22 M
Table 1: Performance on MNIST.
![Page 25: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/25.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Continuous Normalizing FlowsSupervised Learning
Neural ODEs (Chen et al., 2018)https://arxiv.org/pdf/1806.07366.pdf
Applications
Noise distributionTarget distribution
(such asreal image manifold)
Generative Latent Models
Sample fromnoise distribution(such as Gaussian)
Sample from target distribution(such as an image)
23
Neural ODE
![Page 26: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/26.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Continuous Normalizing FlowsSupervised Learning
Neural ODEs (Chen et al., 2018)https://arxiv.org/pdf/1806.07366.pdf
Applications
Neural ODE
Likelihood estimationusing Change of Variables formula
Generative Latent Models
24
Noise distributionTarget distribution
Train f to maximize the likelihood of the samples from target distribution
![Page 27: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/27.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Continuous Normalizing FlowsSupervised Learning
Neural ODEs (Chen et al., 2018)https://arxiv.org/pdf/1806.07366.pdf
Applications
Neural ODE
Likelihood estimationusing Change of Variables formula
Generative Latent Models
25
Noise distributionTarget distribution
Generate samples
Sample from the noise distribution, transform it into a sample from the target distributionusing the trained Neural ODE.
![Page 28: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/28.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Supervised Learning
Neural ODEs (Chen et al., 2018)
Generative Latent Models
https://arxiv.org/pdf/1806.07366.pdf
Applications
Continuous Normalizing Flows
26
![Page 29: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/29.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Supervised Learning
Neural ODEs (Chen et al., 2018)
Generative Latent Models
https://arxiv.org/pdf/1806.07366.pdf
Applications
Continuous Normalizing Flows
Encode the observed data
26
![Page 30: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/30.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Supervised Learning
Neural ODEs (Chen et al., 2018)
Generative Latent Models
https://arxiv.org/pdf/1806.07366.pdf
Applications
Continuous Normalizing Flows
Encode into a latent distribution (such as Gaussian)
26
![Page 31: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/31.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Supervised Learning
Neural ODEs (Chen et al., 2018)
Generative Latent Models
https://arxiv.org/pdf/1806.07366.pdf
Applications
Continuous Normalizing Flows
Sample from the latent distribution
26
![Page 32: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/32.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Supervised Learning
Neural ODEs (Chen et al., 2018)
Generative Latent Models
https://arxiv.org/pdf/1806.07366.pdf
Applications
Continuous Normalizing Flows
Use the sample asinitial value to a Neural ODE
26
![Page 33: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/33.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Supervised Learning
Neural ODEs (Chen et al., 2018)
Generative Latent Models
https://arxiv.org/pdf/1806.07366.pdf
Applications
Continuous Normalizing Flows
Solve the Neural ODE forlatent points in later time steps
26
![Page 34: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/34.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Supervised Learning
Neural ODEs (Chen et al., 2018)
Generative Latent Models
https://arxiv.org/pdf/1806.07366.pdf
Applications
Continuous Normalizing Flows
Compute loss
Decode the latent points
26
![Page 35: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/35.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Supervised Learning
Neural ODEs (Chen et al., 2018)
Generative Latent Models
https://arxiv.org/pdf/1806.07366.pdf
Applications
Continuous Normalizing Flows
Future generation!
Solve the Neural ODE forlatent points in future time steps
Decode the latent points
26
![Page 36: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/36.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
1. Ordinary Differential Equations (ODEs)
○ Initial Value Problems
○ Numerical Integration methods
○ Fundamental theorem of ODEs
2. Neural ODEs (Chen et al., NeurIPS 2018)
○ Adjoint method
○ Applications
3. Later research
27
![Page 37: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/37.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Later research
FFJORD: Free-form Continuous Dynamics For Scalable Reversible Generative Models (Grathwohl et al., ICLR 2019)
https://arxiv.org/pdf/1810.01367.pdf
● Essentially a better Continuous Normalizing Flow.
● Makes a better estimate for the log determinant term.
● “We demonstrate our approach on high-dimensional
density estimation, image generation, and variational
inference, achieving the state-of-the-art among exact
likelihood methods with efficient sampling.”
28
![Page 38: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/38.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Later research
Augmented Neural ODEs (Dupont et al., NeurIPS 2019)
● Shows that Neural ODEs cannot model non-homeomorphisms (non-flows)
● Augments the state with additional dimensions to cover non-homeomorphisms
● Performs ablation study on toy examples and image classification
https://arxiv.org/pdf/1904.01681.pdf
29
![Page 39: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/39.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Later research
ANODEV2: A Coupled Neural ODE Evolution Framework(Zhang et al., NeurIPS 2019)
● Network weights are also a function of time
● Separate “weight network” generates the weights of the function network
at a given time
https://arxiv.org/pdf/1906.04596.pdf
30
![Page 40: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/40.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Later research
Latent ODEs for Irregularly-Sampled Time Series(Rubanova et al., NeurIPS 2019)
https://arxiv.org/pdf/1907.03907.pdf
● Improves the generative latent variable framework for irregularly-sampled time series
● Essentially uses an ODE in the encoder where samples are missing
● Shows results on toy data, MuJoCo, PhysioNet
31
![Page 41: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/41.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Later research
Simple Video Generation using Neural ODEs(David Kanaa*, Vikram Voleti*, Samira Kahou, Christopher Pal; NeurIPS 2019 Workshop)
● Video generation as a generative latent
variable model using Neural ODEs
https://sites.google.com/view/neurips2019lire/accepted-papers?authuser=0
32
![Page 42: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/42.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Later research
ODE2VAE: Deep generative second order ODEs with Bayesian neural networks (Yildiz et al., NeurIPS 2019)
● Uses 2nd-order Neural ODE
● Uses a Bayesian Neural Network
● Showed results modelling video generation
as a generative latent variable model using
(2nd-order Bayesian) Neural ODE
https://papers.nips.cc/paper/9497-ode2vae-deep-generative-second-order-odes-with-bayesian-neural-networks.pdf
33
![Page 43: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/43.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Later research
Neural Jump Stochastic Differential Equations (Jia et al., NeurIPS 2019)
● Models continuous + discrete dynamics of
a hybrid system
● Discontinuities are modelled as stochastic
events
● Show results on real-world and synthetic
point process datasets
34
https://papers.nips.cc/paper/9177-neural-jump-stochastic-differential-equations.pdf
![Page 44: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/44.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Later research
Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise (Liu et al., 2019)
● Random noise injection into Neural ODEs
● Adds a diffusion term into the Neural ODE
formulation, denoting a continuous time
stochastic process
● Makes a case for robustness
35
https://arxiv.org/pdf/1906.02355.pdf
![Page 45: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/45.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Later research
On Robustness of Neural Ordinary Differential Equations(Yan et al., ICLR 2020)
● Ablation study on adversarial attacks on ODE-Nets
● Introduces new regularization term to improve
robustness
https://arxiv.org/pdf/1910.05513.pdf, https://openreview.net/pdf?id=B1e9Y2NYvS
36
![Page 46: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/46.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Later research
Approximation Capabilities of Neural ODEs and Invertible Residual Networks (Zhang et al., ICML 2020)
● Provides guarantees on
modelling capability of
homeomorphisms v/s the
capacity of the Neural ODE
https://arxiv.org/pdf/1907.12998.pdf
37
![Page 47: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/47.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Later research
How to Train Your Neural ODE : the world of Jacobian and kinetic regularization (Finlay et al., ICML 2020)
● Makes a link between the flow in Neural
ODEs and optimal transport
● Introduces two new regularization terms to
constrain flows to straight lines
● Speeds up training of Neural ODEs
https://arxiv.org/pdf/2002.02798.pdf
38
![Page 48: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/48.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Later research
Scalable Gradients for Stochastic Differential Equations(Li et al., AISTATS 2020)
● Generalizes the adjoint method to stochastic dynamics defined by SDEs :
“stochastic adjoint sensitivity method.”
● PyTorch Implementation of Differentiable SDE Solvers:
https://github.com/google-research/torchsde
https://arxiv.org/pdf/2001.01328.pdf
39
![Page 49: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/49.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Later research
“Bullshit that I and others have said about Neural ODEs”(David Duvenaud, ML Retrospectives Workshop @ NeurIPS 2019)
● Summarizes what went right and wrong in the
publication of the Neural ODEs paper
https://www.youtube.com/watch?v=YZ-_E7A3V2w
40
![Page 50: July 10th, 2020 A brief tutorial on Vikram Voleti Neural ODEs€¦ · Vikram Voleti A brief tutorial on Neural ODEs / 40 Ordinary Differential Equations (ODEs) Initial value problem:](https://reader035.vdocuments.us/reader035/viewer/2022071001/5fbdc005ac6e7b78b249cd6b/html5/thumbnails/50.jpg)
Vikram Voleti A brief tutorial on Neural ODEs / 41
Additional References
● http://faculty.bard.edu/belk/math213/InitialValueProblems.pdf
● ODE Solvers: https://math.temple.edu/~queisser/assets/files/Talk3.pdf
● Textbook : https://users.math.msu.edu/users/gnagy/teaching/ode.pdf
● https://lpsa.swarthmore.edu/NumInt/NumIntFirst.html
● Excellent blog post on ODE solvers: https://blogs.mathworks.com/loren/2015/09/23/ode-solver-selection-in-matlab/
● Autodiff tutorial: http://www.cs.toronto.edu/~rgrosse/courses/csc421_2019/readings/L06%20Automatic%20Differentiation.pdf
● Course on Neural Networks & Deep Learning by Roger Grosse & Jimmy Ba, University of Toronto - http://www.cs.toronto.edu/~rgrosse/courses/csc421_2019/
● Official Neural ODE code torchdiffeq : https://github.com/rtqichen/torchdiffeq
● DiffEqML’s torchdyn : https://github.com/DiffEqML/torchdyn
● TorchSDE : https://github.com/google-research/torchsde
41