Data-driven approaches to physical modelling
(in CFD)
July 2021
Richard P. Dwight
Aerodynamics Group/Wind energy Group, Aerospace, TU Delft
1
Contents
I Data-driven modelling
I Method 1: Physics-Informed Neural Networks (PINNs)
I Method 2: Sparse Identification of Nonlinear Dynamical
systems (SINDy)
I Method 3: Field-Inversion Machine-Learning (FIML) OR
Sparse-regression of residuals (SpaRTA)
I Outlook
2
(Turbulence) modelling in the age of data
[Data] has historically been used to calibrate simple engineer-
ing models... with the availability of large and diverse datasets, re-
searchers have begun to explore methods to systematically inform
turbulence models with data.
– (Duraisamy, Iaccarino, and Xiao 2019)
Open question
How can we exploit large high-resolution data-sets to improve the
physical understanding and modelling of the physics (of
turbulence)?
Amorphous data Physical understanding 3
Basic problem statement: Regression
Deterministically:
I Inputs/features x i ∈ RNx known at M
I Outputs/data y ∈ RNy
I Find q : RNx → RNy minimizing the residual sum-of-squares:
L(q) :=1
M
M∑k=1
(yi − q(xi ))2
Statistically:
I Random variables X : Ω→ RNx and Y : Ω→ RNy with joint
density P(X ,Y )
I Find q : RNx → RNy minimizing the expected prediction error:
EX ,Y (Y − q(X ))2 :=
∫(y − q(x))2 dP(X ,Y )
I With exact solution q(x) = E(Y | X = x) 4
Method 1: Physics-Informed Neural
Networks (PINNs)
Neural Networks - Definition
Neuron
σ(x) : R→ R
q := σ
(N∑i=1
wixi + b
)
Network
Single “layer” ` ∈ 1, . . . , L:x` := σ
(W ` · x`−1 + b`
)x`−1, x`,b` ∈ RN W ` ∈ RN×N
Network:
q := σ(W Lσ(W L−1σ(· · · ) + bL−1) + bL)5
Neural Networks - Fitting/training/learning
Standard linear (aka sane) regression
Function representation (P basis functions ϕp(·)):
qLIN(x) :=P∑
p=1
θpϕp(x)
Solve, given M data-pairs (xk , yk):
minθ∈RP
M∑k=1
|yk − qLIN(xk)|2
Quadratic optimization problem =⇒ ATAθ = ATy .
Neural networks, solve:
minW 0,...,W L∈RN×N ;
b0,...,bL∈RN
M∑k=1
|yk − qNN(xk)|2
6
Physics informed Neural Networks (PINNs)
Given a nonlinear PDE (e.g. Navier-Stokes) in x ∈ Ω ⊂ R3,
N (u) = (u · ∇)u +∇p − ν∇2u = 0,
with Dirichlet boundary-conditions u = u∂Ω on the boundary of Ω.
Then represent u as a NN u(x), and solve:
minW 0,...,W L∈RN×N ;
b0,...,bL∈RN
‖u − u∂Ω‖2∂Ω + ‖N (u)‖2
Ω
or approximately:
minW 0,...,W L∈RN×N ;
b0,...,bL∈RN
ND∑i=1
|u(x i )− u∂Ω,i |2 +
NC∑j=1
|N (u)j |2
M. Raissi, P. Perdikaris, and G.E. Karniadakis (2019). “Physics-informed
neural networks”. In: Journal of Computational Physics 378, pp. 686–707.
issn: 0021-9991. doi: 10.1016/j.jcp.2018.10.045 7
Action of a single neuron
8
Action of a network
N. Terleth (2019). “Artificial Neural Networks for Flow Field Inference”.
MA thesis. TU Delft http://resolver.tudelft.nl/uuid:
d69a58c4-91ea-4590-9153-c6fa35f374e5 9
Example: Lid-driven cavity
I 3-layer, 10-neural fully-connected network, tanh activation
I NC = 1000 collocation points in interior, ND = 200 boundary
points
I Error computed with respect to OpenFOAM.
10
PINNs - Prognosis
Excellent:
I Very flexible framework for eqns + data (data-assimiliation!)
I Always gives a solution!
I Fun!
Gloomy:
I Which equation do you want to solve? (Can’t do all!)
I Always gives a solution!
I Inherits all the problems of Least-squares finite-elements
I Training is significantly harder than standard losses
I Much slower than a standard solver
11
Method 2: Sparse Identification of
Nonlinear Dynamical systems
(SINDy)
Sparse regression
Standard linear regression
Model: qLIN(x) :=∑P
p=1 θpϕp(x)
Solve: minθ∈RP
∑Mk=1 |yk − qLIN(xk)|2 = minθ∈RP L(θ)
Regularization (option A = L2):
minθ∈RP
[L(θ) + λ‖θ‖2
2
], ‖θ‖2
2 :=P∑
p=1
|θp|2
Regularization (option B = L1):
minθ∈RP
[L(θ) + λ‖θ‖1] , ‖θ‖1 :=P∑
p=1
|θp|
12
Sparse regression (interpretation)
L2 regularized least-squares
(“ridge regression”)
minL(θ) s.t. ‖θ‖22 ≤ t
L1 regularized least-squares
(“lasso”)
minL(θ) s.t. ‖θ‖1 ≤ t
Hastie, Tibshirani, and Friedman (2009). The Elements of Statistical
Learning. Springer Series in Statistics13
Dictionary methods
Dictionary methods
1. Generate masses of basis functions ϕp(x), P 100 in
q(x) :=P∑
p=1
θpϕp(x)
2. Solve
minL(θ) + λ‖θ‖1
with sufficiently large λ so that most of the θp = 0.
I Dictionary generation, e.g. given features (x1, . . . , xN), and
integers (Q1, . . . ,QN) ∈ NN , choose:
ϕp(x) := xQ11 · x
Q12 · · · x
QNN
I Or also combined with fundamental functions sin, exp, etc. 14
Sparse identification of nonlinear dynamical systems (SINDy)
Rudy et al. (Apr. 2017). “Data-driven discovery of partial differential
equations”. In: Science Advances 3.4, e1602614. issn: 2375-2548. doi:
10.1126/sciadv.1602614 15
SINDy - Prognosis
Excellent:
I Equations directly from data!
I Very low cost
Gloomy:
I Need measurements of complete state
I Need ability to represent exact model in dictionary
I Sensitivity to noise
I Not really demonstrated outside of “toy” problems
I Not clear how to introduce a priori knowledge of equations
16
Method 3: Field-Inversion
Machine-Learning (FIML)
OR
Sparse regression of residuals
(SpaRTA)
Field-inversion
I Consider some PDE model e.g. conservation law
N (u;κ(u)) = 0
with an unknown constitutive relationship κ(u), solution u(ξ).
I We have M (filtered/incomplete/noisy measurements) of the
state yi ' u(ξi ) for some particular testcase.
Field-inversion
1. Approximate κ(u) ' β(ξ) for this case, ξ ∈ R3 (space)
2. Solve
βbest(ξ) := arg minβ
M∑k=1
|yi − u(ξi )|2 s.t. N (u;β(ξ)) = 0.
with β defined at (e.g.) every meshpoint.17
PDE-constrained optimization
This is a high-dimensional PDE-constrained optimization problem...
minq
M∑k=1
|yi − u(ξi )|2 s.t. N (u;β(ξ)) = 0.
... need dJ /dβ to solve efficiently.
Gradient calculation:
Get out your Lagrangian:
L(u, λ) :=M∑k=1
|yi − u(ξi )|2 + λN (u;β(ξ))
and build an equation for the adjoint variable λ
∂N∂u
T
λ +∂J∂u
= 0, =⇒ dJdβ
=∂J∂β
+ λ∂N∂β
18
Example: Turbulence closure modelling
Reynolds decomposition u = U + u′(t) with averaging 〈·〉substituted into N-S:
N (U + u′) = (U · ∇)U +∇p − ν∇2U + 〈u′iu′j〉
with 〈u′iu′j〉 is the Reynolds stress tensor.
Turbulence closure problem
Model 〈u′iu′j〉 in terms of resolved flow U.
E.g. “Boussinesq”:
〈u′iu′j〉 ' −νT (∇U +∇UT )
with some transport equation for νT
NT (U, νT ;β) = 019
Example corrective fields (RANS turbulence closures)
I Partial correction of closure-model error - β(ξ)
Zeno Belligoli, RPD, and Georg Eitelberg (Mar. 2021). “Reconstruction of
Turbulent Flows at High Reynolds Numbers Using Data Assimilation
Techniques”. In: AIAA Journal 59.3, pp. 855–867. issn: 1533-385X. doi:
10.2514/1.j059474
20
Prediction: Field-inversion Machine-learning
I So far no prediction!
Idea: Model β in terms of u
I Standard regression problem where β(ξ) is the data:
minθ
M∑k=1
|β(ξk)− q(uk ;θ)|2
I q(u;θbest) is a new consistutive model =⇒ apply anywhere!
Eric J. Parish and Karthik Duraisamy (Jan. 2016). “A paradigm for
data-driven predictive modeling using field inversion and machine learning”. In:
Journal of Computational Physics 305, pp. 758–774. issn: 0021-9991. doi:
10.1016/j.jcp.2015.11.012
21
Application: Wind-farm wake modelling
Training and test data - LES as ground-truth:
−1 0 1 2 3 4 5 6
−4
−3
−2
−1
0
1
2
Case A
Case B
Case C
I OpenFOAM-6.0 with SOWFA-6 (from NREL)
I Neutral ABL based on precursor
I Turbine, inflow properties were taken from the wind-tunnel
experiment of Chamorro and Porte-Agel (Chamorro and
Porte-Agel 2010).22
Inverse problem: Optimal correction fields β(ξ)
R
|bijΔ|
Corrections:
I R corrects turbulence kinetic energy budget
I b∆ corrects the anisotropy of the Reynolds-stress 23
Regressing corrections - β(U)
I Dictionary approach with elastic-net
I Result:
b∆ij (q) = [1.62 · 10−1 · q1/2
TI · q1/2F
+4.84 · 10−3 · q1/2TI · I 1/2
1
+2.51 · 10−2 · q1/2F
+2.00 · 10−3 · I 1/21 ] · T (1)
ij
R(q) = 8.06 · 10−5 · I 1/21 · q3
ν · k · T (1)ij ∂jui+
[−2.91 · 101 · q1/2TI · qF · I 1/2
1
+4.28 · 10−1 · q2⊥ · qF · I 1/2
1
−1.22 · qF · qγ+2.30 · q2
F · I2] · ε
Schmelzer, RPD, and Cinnella (2019). “Discovery of Algebraic
Reynolds-Stress Models Using Sparse Symbolic Regression”. In: Flow,
Turbulence and Combustion 104.2-3, pp. 579–603. issn: 1573-1987. doi:
10.1007/s10494-019-00089-x
24
Predictions of correction R (Train A,B; Test C)R
refe
rence
R m
odelle
d
Steiner, RPD, and Vire (2020a). “Data-driven RANS closures for wind
turbine wakes under neutral conditions”. In: arXiv 2009.1081625
Predictions (Train A,B; Test C) - Mean U
-0.5
0.0
0.5
(z zhub)/DCase C: T1 -1D Case C: T1 1D Case C: T1 2D
0.0 U/Uhub 1.0
-0.5
0.0
0.5
(z zhub)/DCase C: T2 2D
0.0 U/Uhub 1.0
Case C: T2 4D
0.0 U/Uhub 1.0
Case C: T2 6D
LES
26
Predictions (Train A,B; Test C) - Mean U
-0.5
0.0
0.5
(z zhub)/DCase C: T1 -1D Case C: T1 1D Case C: T1 2D
0.0 U/Uhub 1.0
-0.5
0.0
0.5
(z zhub)/DCase C: T2 2D
0.0 U/Uhub 1.0
Case C: T2 4D
0.0 U/Uhub 1.0
Case C: T2 6D
LESRANS baseline
26
Predictions (Train A,B; Test C) - Mean U
-0.5
0.0
0.5
(z zhub)/DCase C: T1 -1D Case C: T1 1D Case C: T1 2D
0.0 U/Uhub 1.0
-0.5
0.0
0.5
(z zhub)/DCase C: T2 2D
0.0 U/Uhub 1.0
Case C: T2 4D
0.0 U/Uhub 1.0
Case C: T2 6D
LESRANS baseline
RANS corrected
26
Predictions (Train A,B; Test C) - T.K.E. k
-0.5
0.0
0.5
(z zhub)/DCase C: T1 -1D Case C: T1 1D Case C: T1 2D
0.0 k/khub 5.0
-0.5
0.0
0.5
(z zhub)/DCase C: T2 2D
0.0 k/khub 5.0
Case C: T2 4D
0.0 k/khub 5.0
Case C: T2 6D
LES
27
Predictions (Train A,B; Test C) - T.K.E. k
-0.5
0.0
0.5
(z zhub)/DCase C: T1 -1D Case C: T1 1D Case C: T1 2D
0.0 k/khub 5.0
-0.5
0.0
0.5
(z zhub)/DCase C: T2 2D
0.0 k/khub 5.0
Case C: T2 4D
0.0 k/khub 5.0
Case C: T2 6D
LESRANS baseline
27
Predictions (Train A,B; Test C) - T.K.E. k
-0.5
0.0
0.5
(z zhub)/DCase C: T1 -1D Case C: T1 1D Case C: T1 2D
0.0 k/khub 5.0
-0.5
0.0
0.5
(z zhub)/DCase C: T2 2D
0.0 k/khub 5.0
Case C: T2 4D
0.0 k/khub 5.0
Case C: T2 6D
LESRANS baseline
RANS corrected
27
Predictions (Train C; Test A) - T.K.E. k
Correction model
28
FIML - Prognosis
Excellent:
I Formally complete/consistent
I Use any data!
Gloomy:
I Data might be not informative
I Inverse problem is consistently a nightmare to solve
Steiner, RPD, and Vire (2021). “Classifying regions of high model error
within a data-driven RANS closure: Application to wind turbine wakes”. In:
arXiv 2106.15593
29
Conclusions
30
References i
References
Belligoli, Zeno, RPD, and Georg Eitelberg (Mar. 2021). “Reconstruction of
Turbulent Flows at High Reynolds Numbers Using Data Assimilation
Techniques”. In: AIAA Journal 59.3, pp. 855–867. issn: 1533-385X. doi:
10.2514/1.j059474.
Brunton, Proctor, and Kutz (2016). “Discovering governing equations from
data by sparse identification of nonlinear dynamical systems”. In:
Proceedings of the National Academy of Sciences 113.15, pp. 3932–3937.
doi: 10.1073/pnas.1517384113.
31
References ii
Chamorro and Porte-Agel (2010). “Effects of Thermal Stability and Incoming
Boundary-Layer Flow Characteristics on Wind-Turbine Wakes: A
Wind-Tunnel Study”. In: Boundary-Layer Meteorology 136.3, pp. 515–533.
doi: 10.1007/s10546-010-9512-1.
Duraisamy, Karthik, Gianluca Iaccarino, and Heng Xiao (2019). “Turbulence
Modeling in the Age of Data”. In: Annual Review of Fluid Mechanics 51.1,
pp. 357–377. doi: 10.1146/annurev-fluid-010518-040547.
Edeling, Cinnella, and RPD (2014). “Predictive RANS simulations via Bayesian
Model-Scenario Averaging”. In: Journal of Computational Physics 275,
pp. 65–91.
Edeling, RPD, and Cinnella (2014). “Bayesian estimates of the parameter
variability in the k − ε turbulence model”. In: Journal of Computational
Physics 258, pp. 73–94.
32
References iii
Edeling, Schmelzer, et al. (2018). “Bayesian Predictions of Reynolds-Averaged
Navier-Stokes Uncertainties Using Maximum a Posteriori Estimates”. In:
AIAA Journal 56.5, pp. 2018–2029. doi: 10.2514/1.J056287.
Hastie, Tibshirani, and Friedman (2009). The Elements of Statistical Learning.
Springer Series in Statistics.
Huijing, Jasper P., RPD, and Martin Schmelzer (July 2021). “Data-driven
RANS closures for three-dimensional flows around bluff bodies”. In:
Computers and Fluids 225, p. 104997. issn: 0045-7930. doi:
10.1016/j.compfluid.2021.104997.
Kumar, Schmelzer, and RPD (2018). “Stochastic turbulence modeling in
RANS simulations via Multilevel Monte Carlo”. In: arXiv 1811.00872.
Parish, Eric J. and Karthik Duraisamy (Jan. 2016). “A paradigm for
data-driven predictive modeling using field inversion and machine learning”.
In: Journal of Computational Physics 305, pp. 758–774. issn: 0021-9991.
doi: 10.1016/j.jcp.2015.11.012.
33
References iv
Porte-Agel, Fernando, Majid Bastankhah, and Sina Shamsoddin (2019).
“Wind-Turbine and Wind-Farm Flows: A Review”. In: Boundary-Layer
Meteorology 174.1, pp. 1–59. issn: 1573-1472. doi:
10.1007/s10546-019-00473-0.
Raissi, M., P. Perdikaris, and G.E. Karniadakis (2019). “Physics-informed
neural networks”. In: Journal of Computational Physics 378, pp. 686–707.
issn: 0021-9991. doi: 10.1016/j.jcp.2018.10.045.
Rudy et al. (Apr. 2017). “Data-driven discovery of partial differential
equations”. In: Science Advances 3.4, e1602614. issn: 2375-2548. doi:
10.1126/sciadv.1602614.
Schmelzer, RPD, and Cinnella (2019). “Discovery of Algebraic Reynolds-Stress
Models Using Sparse Symbolic Regression”. In: Flow, Turbulence and
Combustion 104.2-3, pp. 579–603. issn: 1573-1987. doi:
10.1007/s10494-019-00089-x.
34
References v
Steiner, RPD, and Vire (2020a). “Data-driven RANS closures for wind turbine
wakes under neutral conditions”. In: arXiv 2009.10816.
– (2020b). “Data-driven turbulence modeling for wind turbine wakes under
neutral conditions”. In: Journal of Physics: Conference Series 1618,
p. 062051. issn: 1742-6596. doi: 10.1088/1742-6596/1618/6/062051.
– (2021). “Classifying regions of high model error within a data-driven RANS
closure: Application to wind turbine wakes”. In: arXiv 2106.15593.
Terleth, N. (2019). “Artificial Neural Networks for Flow Field Inference”.
MA thesis. TU Delft.
35