![Page 1: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/1.jpg)
LEARNING AND LEARNING TO SOLVE PDES Bin Dong (董彬)• Beijing International Center for Mathematical Research, Peking University
• Laboratory for Biomedical Image Analysis & Laboratory of Deep Learning Research, Beijing Institute of Big Data Research
![Page 2: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/2.jpg)
OUTLINE
Overview and Motivations
Deep Neural Networks (DNNs) and PDEs
Learning PDEs: PDE-Net for Inverse Problems
Learning to solve PDEs: A Reinforcement Learning Framework
2
Learning Learning to solve
![Page 3: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/3.jpg)
OVERVIEW
Motivations and Intuitions
![Page 4: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/4.jpg)
DEEP LEARNING REVOLUTION
4
Biomedicine
Computer Vision
![Page 5: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/5.jpg)
DEEP LEARNING FROM MATHEMATICS PERSPECTIVE
What are still challenging
Theoretical guidance
Transparency, interpretability, robustness
Idea: find links of DL with mathematics
Our perspective: control
5
![Page 6: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/6.jpg)
DEEP LEARNING FROM MATHEMATICS PERSPECTIVE
Control Perspective:
6
• E, CMS, 5(1):1–11, 2017.• Li, Chen, Tai, E, JMLR, 18(1), 2017.• Haber, Ruthotto, IP, 34(1), 2017.• Lu, Zhong, Li, Dong, ICML 2018. • Gregor, LeCun, ICML 2010. • Yang, Sun, Li, Xu, NIPS 2016. • Li, Tai, E, ICML 2017.• Liu, Theodorou, arXiv:1908.10920.
Fokker-Planck equation:
Dynamics:
Training loss:
Training loss:
• Chen, Wei, Pock, CVPR 2015.• Long, Lu, Ma, Dong, ICML 2018. • Long, Lu, Dong, JCP, 2019.• Ray, Hesthaven, JCP, 367, 2018• Han, Jentzen, E, PNAS, 115(34), 2018.• Wang, Shen, Long, Dong, arXiv: 1905.11079.• Li, Shi, arXiv:1708.05115.• Sun, Tao, Du, arXiv:1812.00174.
Credit: Pengfei Jin @ PKUSee also: Haber & Ruthotto, arXiv:1705.03341
![Page 7: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/7.jpg)
DEEP LEARNING FROM MATHEMATICS PERSPECTIVE
Control Perspective: Supervised Learning (SL)、Reinforcement Learning (RL)、Meta Learning (Meta)
7
Dynamics:
Training loss:
Typical SL-control
Typical RL-control
Typical Meta-control
Solver:
• BP/PMP
• DP
Common:Uncommon:
![Page 8: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/8.jpg)
OTHER RELATED WORKS
Architecture design Wang B, Yuan B, Shi Z, et al. EnResNet: ResNet Ensemble via the
Feynman-Kac Formalism. arXiv:1811.10745, 2018. Tao Y, Sun Q, Du Q, et al. Nonlocal Neural Networks, Nonlocal
Diffusion and Nonlocal Modeling. NeurIPS 2018. Zhu M, Chang B, Fu C. Convolutional Neural Networks combined
with Runge-Kutta Methods. arXiv:1802.08831, 2018. Zhang L, Schaeffer H. Forward Stability of ResNet and Its Variants.
arXiv:1811.09885, 2018. Sun Q, Tao Y, Du Q. Stochastic Training of Residual Networks: a
Differential Equation Viewpoint. arXiv:1812.00174, 2018. Li H, Yang Y, Chen D and Lin Z. Optimization Algorithm Inspired
Deep Neural Network Structure Design. ACML 2018. He J, Xu J. MgNet: A Unified Framework of Multigrid and
Convolutional Neural Network. arXiv:1901.10415, 2019. Theory
E W., Han J, Li Q. A mean-field optimal control formulation of deep learning. Research in the Mathematical Sciences, vol. 6, no. 10, pp. 1–41, 2019.
Thorpe M, van Gennip Y. Deep Limits of Residual Neural Networks. arXiv:1810.11741, 2018.
8
![Page 9: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/9.jpg)
OTHER RELATED WORKS
Optimization Li Q, Hao S. An optimal control approach to deep learning and applications to
discrete-weight neural networks. ICML 2018. Chen T Q, Rubanova Y, Bettencourt J, et al. Neural ordinary differential equations.
NeurIPS 2018. (Best paper) Parpas P, Muir C. Predict Globally, Correct Locally: Parallel-in-Time Optimal
Control of Neural Networks. arXiv:1902.02542. Zhang D, Zhang T, Lu Y, Zhu Z, Dong B, You Only Propagate Once: Accelerating
Adversarial Training via Maximal Principle, NeurIPS 2019.
Applications Chen Y, Yu W, and Pock T. On learning optimized reaction diffusion processes
for effective image restoration. CVPR 2015. Sun, Li, and Xu. Deep ADMM-net for compressive sensing MRI. NIPS 2016. Zhang S, Lu Y, Liu J, Dong B. Dynamically Unfolding Recurrent Restorer: A
Moving Endpoint Control Method for Image Restoration. ICLR 2019 (arXiv:1805.07709).
Zhang H, Dong B and Liu B, JSR-Net: A Deep Network for Joint Spatial-Radon Domain CT Reconstruction from incomplete data, IEEE-ICASSP, 2019
Lu Y, He D, Li Z, Sun Z, Dong B, Qin T, Wang L, Liu T-Y. Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View. arXiv: 1906.02762, 2019.
9
![Page 10: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/10.jpg)
BRIDGING DIFFERENTIAL EQUATIONS WITH DEEP NETWORKS
DNNs and numerical PDEs: Learning PDEs• Zichao Long, Yiping Lu, Xianzhong Ma and Bin Dong, PDE-Net:
Learning PDEs from Data, ICML 2018. (arXiv:1710.09668)• Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs
from Data with A Numeric-Symbolic Hybrid Deep Network, accepted by Journal of Computational Physics, 2019 (arXiv:1812.04426).
![Page 11: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/11.jpg)
PDE-NET: LEARNING PDES FROM DATA
Can we learn principles (e.g. PDEs) from data?
11Biology
Meteorology
Computer Graphics
![Page 12: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/12.jpg)
PDE-NET: LEARNING PDES FROM DATA
12
Earlier work
Other earlier attempts:• Bongard & Lipson,
PNAS, 2007• Liu, Lin, Zhang & Su.
ECCV 2010.
![Page 13: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/13.jpg)
PDE-NET: LEARNING PDES FROM DATA
• S. Brunton, J. L. Proctor and J. N. Kutz Proceedings of the National Academy of Sciences, 2016• Samuel H Rudy, Steven L Brunton, Joshua L Proctor, and J Nathan Kutz. Science Advances, 3(4), 2017.• Hayden Schaeffer. Proc. R. Soc. A, volume 473, The Royal Society, 2017.
13
![Page 14: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/14.jpg)
PDE-NET: LEARNING PDES FROM DATA
Room for improvements
Can we go beyond sparse coding framework
(linear dictionary)?
—— Bigger model class with less prior knowledge
Can we learn discrete forms of differential
operators and does it help?
——More accurate estimation of the PDE and
prediction 14
![Page 15: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/15.jpg)
PDE-NET: LEARNING PDES FROM DATA
Assuming:
15
![Page 16: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/16.jpg)
PDE-NET: LEARNING PDES FROM DATA
PDE-Net 2.0
16
Assuming:
![Page 17: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/17.jpg)
PDE-NET: LEARNING PDES FROM DATA
PDE-Net 2.0
17
identity identity
More Constraints:• Pseudo-upwind• Sparsity on moment matrices• Sparsity on the symbolic network
![Page 18: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/18.jpg)
PDE-NET: LEARNING PDES FROM DATA
1st orderlearnable
2st orderlearnable
1st orderfrozen
18
• J.F. Cai, B. Dong, S. Osher and Z. Shen, Journal of the American Mathematical Society, 2012.• B. Dong, Q. Jiang and Z. Shen, Multiscale Modeling & Simulation, 2017
![Page 19: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/19.jpg)
PDE-NET: LEARNING PDES FROM DATA
19Prediction
Model recovery
![Page 20: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/20.jpg)
PDE-NET: LEARNING PDES FROM DATA
20
Model Recovery
![Page 21: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/21.jpg)
BRIDGING DIFFERENTIAL EQUATIONS WITH DEEP NETWORKS
DNNs and numerical PDEs: Learning to solve PDEs• Yufei Wang, Ziju Shen, Zichao Long and Bin Dong, Learning to
Discretize: Solving 1D Scalar Conservation Laws via Deep Reinforcement Learning, arXiv: 1905.11079, 2019.
![Page 22: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/22.jpg)
NEURAL NETWORKS (NNS) AND NUMERICAL PDES – SOME RECENT DEVELOPMENTS
NNs as a new ansatz: M. Raissi, P. Perdikaris, G. E. Karniadakis. arXiv:1711.10561 & arXiv:1711.10566,
2017. J. Magiera, D. Ray, J. S Hesthaven, Rohde. arXiv:1904.12794, 2019. C. Michoski, M. Milosavljevic, T. Oliver, D. Hatch. arXiv:1905.04351, 2019. G.-J. Both, S. Choudhury, P. Sens, R. Kusters. arXiv:1904.09406, 2019.
NNs to approximate complex solution mappings Y. Khoo, J. Lu, L. Ying. arXiv:1707.03351, 2017. Y. Khoo, L. Ying. arXiv:1810.09675, 2018. Y. Li, J. Lu, A. Mao. arXiv:1905.02789, 2019. Y. Fan, L. Lin, L. Ying, L. Zepeda-Núnez. arXiv:1807.01883, 2018.
NNs to solve very high dimensional PDEs C. Beck, W. E, A. Jentzen. JNS, 1–57, 2017. W. E and B. Yu, CMS, 6(1), 1-12, 2018. J. Han, A. Jentzen, W. E. PNAS, 115(34):8505–8510, 2018. D. Pfau, J.S. Spencer, A.G. Matthews, W. M. Foulkes, arXiv:1909.02487. W. Cai and Z. Xu, arXiv:1910.11710
NNs integrated with classical solvers N. Discacciati, J. S Hesthaven, D. Ray. Technical report, 2019. D. Ray, J. S Hesthaven. JCP, 367:166–191, 2018.
![Page 23: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/23.jpg)
LEARNING TO DISCRETIZE
Roe speed:
23
![Page 24: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/24.jpg)
LEARNING TO DISCRETIZE
Reinforcement learning (RL)
• RL is to learn to make sequential decisions by interacting with the environments (learn from rewards).
• Can be modeled as a finite Markov Decision Process (MDP):
24
![Page 25: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/25.jpg)
LEARNING TO DISCRETIZE
CL solver revisited
25
Action:
State transition
![Page 26: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/26.jpg)
LEARNING TO DISCRETIZE
RL-WENO
26
![Page 27: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/27.jpg)
LEARNING TO DISCRETIZE
RL-WENO v.s. WENO-5: Overall accuracy:
Comparable with WENO-5; Generalize well to other mesh sizes, flux, temporal
discretization, and terminal time.
Near singularities
27
![Page 28: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/28.jpg)
LEARNING TO DISCRETIZE
28
Action
![Page 29: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/29.jpg)
LEARNING TO DISCRETIZE
![Page 30: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/30.jpg)
CONCLUSIONS
30
![Page 31: PDE...2019/10/31 · Learning PDEs from Data, ICML 2018. (arXiv:1710.09668) • Zichao Long, Yiping Lu and Bin Dong, PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid](https://reader033.vdocuments.us/reader033/viewer/2022042215/5ebb60ef6a466e14fa1fb847/html5/thumbnails/31.jpg)
THANKS FOR YOUR ATTENTION!
MY WEBPAGE:HTTP://BICMR.PKU.EDU.CN/~DONGBIN
Jointly with
Quanzheng Li@ Harvard MGH
Jiaying Liu@ PKU ICST
Xiaoshuai Zhang@ PKU ICST
Zichao Long@ PKU SMS
Xianzhong Ma@ PKU SMS
Aoxiao Zhong@ Harvard SEAS
Zhanxing Zhu@ PKU SMS
Jikai Hou (SMS)Tianyuan Zhang (ICST)Dinghuai Zhang (SMS)
@ PKU
Yufei Wang@ PKU Yuanpei College
Ziju Shen@ PKU AAIS
Liwei Wang@ PKU CS
Tie-Yan Liu@ Microsoft