Data-driven approaches to physical modelling

(in CFD)

July 2021

Richard P. Dwight

Aerodynamics Group/Wind energy Group, Aerospace, TU Delft

1

Contents

I Data-driven modelling

I Method 1: Physics-Informed Neural Networks (PINNs)

I Method 2: Sparse Identification of Nonlinear Dynamical

systems (SINDy)

I Method 3: Field-Inversion Machine-Learning (FIML) OR

Sparse-regression of residuals (SpaRTA)

I Outlook

2

(Turbulence) modelling in the age of data

[Data] has historically been used to calibrate simple engineer-

ing models... with the availability of large and diverse datasets, re-

searchers have begun to explore methods to systematically inform

turbulence models with data.

– (Duraisamy, Iaccarino, and Xiao 2019)

Open question

How can we exploit large high-resolution data-sets to improve the

physical understanding and modelling of the physics (of

turbulence)?

Amorphous data Physical understanding 3

Basic problem statement: Regression

Deterministically:

I Inputs/features x i ∈ RNx known at M

I Outputs/data y ∈ RNy

I Find q : RNx → RNy minimizing the residual sum-of-squares:

L(q) :=1

M

M∑k=1

(yi − q(xi ))2

Statistically:

I Random variables X : Ω→ RNx and Y : Ω→ RNy with joint

density P(X ,Y )

I Find q : RNx → RNy minimizing the expected prediction error:

EX ,Y (Y − q(X ))2 :=

∫(y − q(x))2 dP(X ,Y )

I With exact solution q(x) = E(Y | X = x) 4

Method 1: Physics-Informed Neural

Networks (PINNs)

Neural Networks - Definition

Neuron

σ(x) : R→ R

q := σ

(N∑i=1

wixi + b

)

Network

Single “layer” ` ∈ 1, . . . , L:x` := σ

(W ` · x`−1 + b`

)x`−1, x`,b` ∈ RN W ` ∈ RN×N

Network:

q := σ(W Lσ(W L−1σ(· · · ) + bL−1) + bL)5

Neural Networks - Fitting/training/learning

Standard linear (aka sane) regression

Function representation (P basis functions ϕp(·)):

qLIN(x) :=P∑

p=1

θpϕp(x)

Solve, given M data-pairs (xk , yk):

minθ∈RP

M∑k=1

|yk − qLIN(xk)|2

Quadratic optimization problem =⇒ ATAθ = ATy .

Neural networks, solve:

minW 0,...,W L∈RN×N ;

b0,...,bL∈RN

M∑k=1

|yk − qNN(xk)|2

6

Physics informed Neural Networks (PINNs)

Given a nonlinear PDE (e.g. Navier-Stokes) in x ∈ Ω ⊂ R3,

N (u) = (u · ∇)u +∇p − ν∇2u = 0,

with Dirichlet boundary-conditions u = u∂Ω on the boundary of Ω.

Then represent u as a NN u(x), and solve:

minW 0,...,W L∈RN×N ;

b0,...,bL∈RN

‖u − u∂Ω‖2∂Ω + ‖N (u)‖2

Ω

or approximately:

minW 0,...,W L∈RN×N ;

b0,...,bL∈RN

ND∑i=1

|u(x i )− u∂Ω,i |2 +

NC∑j=1

|N (u)j |2

M. Raissi, P. Perdikaris, and G.E. Karniadakis (2019). “Physics-informed

neural networks”. In: Journal of Computational Physics 378, pp. 686–707.

issn: 0021-9991. doi: 10.1016/j.jcp.2018.10.045 7

Action of a single neuron

8

Action of a network

N. Terleth (2019). “Artificial Neural Networks for Flow Field Inference”.

MA thesis. TU Delft http://resolver.tudelft.nl/uuid:

d69a58c4-91ea-4590-9153-c6fa35f374e5 9

Example: Lid-driven cavity

I 3-layer, 10-neural fully-connected network, tanh activation

I NC = 1000 collocation points in interior, ND = 200 boundary

points

I Error computed with respect to OpenFOAM.

10

PINNs - Prognosis

Excellent:

I Very flexible framework for eqns + data (data-assimiliation!)

I Always gives a solution!

I Fun!

Gloomy:

I Which equation do you want to solve? (Can’t do all!)

I Always gives a solution!

I Inherits all the problems of Least-squares finite-elements

I Training is significantly harder than standard losses

I Much slower than a standard solver

11

Method 2: Sparse Identification of

Nonlinear Dynamical systems

(SINDy)

Sparse regression

Standard linear regression

Model: qLIN(x) :=∑P

p=1 θpϕp(x)

Solve: minθ∈RP

∑Mk=1 |yk − qLIN(xk)|2 = minθ∈RP L(θ)

Regularization (option A = L2):

minθ∈RP

[L(θ) + λ‖θ‖2

2

], ‖θ‖2

2 :=P∑

p=1

|θp|2

Regularization (option B = L1):

minθ∈RP

[L(θ) + λ‖θ‖1] , ‖θ‖1 :=P∑

p=1

|θp|

12

Sparse regression (interpretation)

L2 regularized least-squares

(“ridge regression”)

minL(θ) s.t. ‖θ‖22 ≤ t

L1 regularized least-squares

(“lasso”)

minL(θ) s.t. ‖θ‖1 ≤ t

Hastie, Tibshirani, and Friedman (2009). The Elements of Statistical

Learning. Springer Series in Statistics13

Dictionary methods

Dictionary methods

1. Generate masses of basis functions ϕp(x), P 100 in

q(x) :=P∑

p=1

θpϕp(x)

2. Solve

minL(θ) + λ‖θ‖1

with sufficiently large λ so that most of the θp = 0.

I Dictionary generation, e.g. given features (x1, . . . , xN), and

integers (Q1, . . . ,QN) ∈ NN , choose:

ϕp(x) := xQ11 · x

Q12 · · · x

QNN

I Or also combined with fundamental functions sin, exp, etc. 14

Sparse identification of nonlinear dynamical systems (SINDy)

Rudy et al. (Apr. 2017). “Data-driven discovery of partial differential

equations”. In: Science Advances 3.4, e1602614. issn: 2375-2548. doi:

10.1126/sciadv.1602614 15

SINDy - Prognosis

Excellent:

I Equations directly from data!

I Very low cost

Gloomy:

I Need measurements of complete state

I Need ability to represent exact model in dictionary

I Sensitivity to noise

I Not really demonstrated outside of “toy” problems

I Not clear how to introduce a priori knowledge of equations

16

Method 3: Field-Inversion

Machine-Learning (FIML)

OR

Sparse regression of residuals

(SpaRTA)

Field-inversion

I Consider some PDE model e.g. conservation law

N (u;κ(u)) = 0

with an unknown constitutive relationship κ(u), solution u(ξ).

I We have M (filtered/incomplete/noisy measurements) of the

state yi ' u(ξi ) for some particular testcase.

Field-inversion

1. Approximate κ(u) ' β(ξ) for this case, ξ ∈ R3 (space)

2. Solve

βbest(ξ) := arg minβ

M∑k=1

|yi − u(ξi )|2 s.t. N (u;β(ξ)) = 0.

with β defined at (e.g.) every meshpoint.17

PDE-constrained optimization

This is a high-dimensional PDE-constrained optimization problem...

minq

M∑k=1

|yi − u(ξi )|2 s.t. N (u;β(ξ)) = 0.

... need dJ /dβ to solve efficiently.

Gradient calculation:

Get out your Lagrangian:

L(u, λ) :=M∑k=1

|yi − u(ξi )|2 + λN (u;β(ξ))

and build an equation for the adjoint variable λ

∂N∂u

T

λ +∂J∂u

= 0, =⇒ dJdβ

=∂J∂β

+ λ∂N∂β

18

Example: Turbulence closure modelling

Reynolds decomposition u = U + u′(t) with averaging 〈·〉substituted into N-S:

N (U + u′) = (U · ∇)U +∇p − ν∇2U + 〈u′iu′j〉

with 〈u′iu′j〉 is the Reynolds stress tensor.

Turbulence closure problem

Model 〈u′iu′j〉 in terms of resolved flow U.

E.g. “Boussinesq”:

〈u′iu′j〉 ' −νT (∇U +∇UT )

with some transport equation for νT

NT (U, νT ;β) = 019

Example corrective fields (RANS turbulence closures)

I Partial correction of closure-model error - β(ξ)

Zeno Belligoli, RPD, and Georg Eitelberg (Mar. 2021). “Reconstruction of

Turbulent Flows at High Reynolds Numbers Using Data Assimilation

Techniques”. In: AIAA Journal 59.3, pp. 855–867. issn: 1533-385X. doi:

10.2514/1.j059474

20

Prediction: Field-inversion Machine-learning

I So far no prediction!

Idea: Model β in terms of u

I Standard regression problem where β(ξ) is the data:

minθ

M∑k=1

|β(ξk)− q(uk ;θ)|2

I q(u;θbest) is a new consistutive model =⇒ apply anywhere!

Eric J. Parish and Karthik Duraisamy (Jan. 2016). “A paradigm for

data-driven predictive modeling using field inversion and machine learning”. In:

Journal of Computational Physics 305, pp. 758–774. issn: 0021-9991. doi:

10.1016/j.jcp.2015.11.012

21

Application: Wind-farm wake modelling

Training and test data - LES as ground-truth:

−1 0 1 2 3 4 5 6

−4

−3

−2

−1

0

1

2

Case A

Case B

Case C

I OpenFOAM-6.0 with SOWFA-6 (from NREL)

I Neutral ABL based on precursor

I Turbine, inflow properties were taken from the wind-tunnel

experiment of Chamorro and Porte-Agel (Chamorro and

Porte-Agel 2010).22

Inverse problem: Optimal correction fields β(ξ)

R

|bijΔ|

Corrections:

I R corrects turbulence kinetic energy budget

I b∆ corrects the anisotropy of the Reynolds-stress 23

Regressing corrections - β(U)

I Dictionary approach with elastic-net

I Result:

b∆ij (q) = [1.62 · 10−1 · q1/2

TI · q1/2F

+4.84 · 10−3 · q1/2TI · I 1/2

1

+2.51 · 10−2 · q1/2F

+2.00 · 10−3 · I 1/21 ] · T (1)

ij

R(q) = 8.06 · 10−5 · I 1/21 · q3

ν · k · T (1)ij ∂jui+

[−2.91 · 101 · q1/2TI · qF · I 1/2

1

+4.28 · 10−1 · q2⊥ · qF · I 1/2

1

−1.22 · qF · qγ+2.30 · q2

F · I2] · ε

Schmelzer, RPD, and Cinnella (2019). “Discovery of Algebraic

Reynolds-Stress Models Using Sparse Symbolic Regression”. In: Flow,

Turbulence and Combustion 104.2-3, pp. 579–603. issn: 1573-1987. doi:

10.1007/s10494-019-00089-x

24

Predictions of correction R (Train A,B; Test C)R

refe

rence

R m

odelle

d

Steiner, RPD, and Vire (2020a). “Data-driven RANS closures for wind

turbine wakes under neutral conditions”. In: arXiv 2009.1081625

Predictions (Train A,B; Test C) - Mean U

-0.5

0.0

0.5

(z zhub)/DCase C: T1 -1D Case C: T1 1D Case C: T1 2D

0.0 U/Uhub 1.0

-0.5

0.0

0.5

(z zhub)/DCase C: T2 2D

0.0 U/Uhub 1.0

Case C: T2 4D

0.0 U/Uhub 1.0

Case C: T2 6D

LES

26

Predictions (Train A,B; Test C) - Mean U

-0.5

0.0

0.5

(z zhub)/DCase C: T1 -1D Case C: T1 1D Case C: T1 2D

0.0 U/Uhub 1.0

-0.5

0.0

0.5

(z zhub)/DCase C: T2 2D

0.0 U/Uhub 1.0

Case C: T2 4D

0.0 U/Uhub 1.0

Case C: T2 6D

LESRANS baseline

26

Predictions (Train A,B; Test C) - Mean U

-0.5

0.0

0.5

(z zhub)/DCase C: T1 -1D Case C: T1 1D Case C: T1 2D

0.0 U/Uhub 1.0

-0.5

0.0

0.5

(z zhub)/DCase C: T2 2D

0.0 U/Uhub 1.0

Case C: T2 4D

0.0 U/Uhub 1.0

Case C: T2 6D

LESRANS baseline

RANS corrected

26

Predictions (Train A,B; Test C) - T.K.E. k

-0.5

0.0

0.5

(z zhub)/DCase C: T1 -1D Case C: T1 1D Case C: T1 2D

0.0 k/khub 5.0

-0.5

0.0

0.5

(z zhub)/DCase C: T2 2D

0.0 k/khub 5.0

Case C: T2 4D

0.0 k/khub 5.0

Case C: T2 6D

LES

27

Predictions (Train A,B; Test C) - T.K.E. k

-0.5

0.0

0.5

(z zhub)/DCase C: T1 -1D Case C: T1 1D Case C: T1 2D

0.0 k/khub 5.0

-0.5

0.0

0.5

(z zhub)/DCase C: T2 2D

0.0 k/khub 5.0

Case C: T2 4D

0.0 k/khub 5.0

Case C: T2 6D

LESRANS baseline

27

Predictions (Train A,B; Test C) - T.K.E. k

-0.5

0.0

0.5

(z zhub)/DCase C: T1 -1D Case C: T1 1D Case C: T1 2D

0.0 k/khub 5.0

-0.5

0.0

0.5

(z zhub)/DCase C: T2 2D

0.0 k/khub 5.0

Case C: T2 4D

0.0 k/khub 5.0

Case C: T2 6D

LESRANS baseline

RANS corrected

27

Predictions (Train C; Test A) - T.K.E. k

Correction model

28

FIML - Prognosis

Excellent:

I Formally complete/consistent

I Use any data!

Gloomy:

I Data might be not informative

I Inverse problem is consistently a nightmare to solve

Steiner, RPD, and Vire (2021). “Classifying regions of high model error

within a data-driven RANS closure: Application to wind turbine wakes”. In:

arXiv 2106.15593

29

Conclusions

30

References i

References

Belligoli, Zeno, RPD, and Georg Eitelberg (Mar. 2021). “Reconstruction of

Turbulent Flows at High Reynolds Numbers Using Data Assimilation

Techniques”. In: AIAA Journal 59.3, pp. 855–867. issn: 1533-385X. doi:

10.2514/1.j059474.

Brunton, Proctor, and Kutz (2016). “Discovering governing equations from

data by sparse identification of nonlinear dynamical systems”. In:

Proceedings of the National Academy of Sciences 113.15, pp. 3932–3937.

doi: 10.1073/pnas.1517384113.

31

References ii

Chamorro and Porte-Agel (2010). “Effects of Thermal Stability and Incoming

Boundary-Layer Flow Characteristics on Wind-Turbine Wakes: A

Wind-Tunnel Study”. In: Boundary-Layer Meteorology 136.3, pp. 515–533.

doi: 10.1007/s10546-010-9512-1.

Duraisamy, Karthik, Gianluca Iaccarino, and Heng Xiao (2019). “Turbulence

Modeling in the Age of Data”. In: Annual Review of Fluid Mechanics 51.1,

pp. 357–377. doi: 10.1146/annurev-fluid-010518-040547.

Edeling, Cinnella, and RPD (2014). “Predictive RANS simulations via Bayesian

Model-Scenario Averaging”. In: Journal of Computational Physics 275,

pp. 65–91.

Edeling, RPD, and Cinnella (2014). “Bayesian estimates of the parameter

variability in the k − ε turbulence model”. In: Journal of Computational

Physics 258, pp. 73–94.

32

References iii

Edeling, Schmelzer, et al. (2018). “Bayesian Predictions of Reynolds-Averaged

Navier-Stokes Uncertainties Using Maximum a Posteriori Estimates”. In:

AIAA Journal 56.5, pp. 2018–2029. doi: 10.2514/1.J056287.

Hastie, Tibshirani, and Friedman (2009). The Elements of Statistical Learning.

Springer Series in Statistics.

Huijing, Jasper P., RPD, and Martin Schmelzer (July 2021). “Data-driven

RANS closures for three-dimensional flows around bluff bodies”. In:

Computers and Fluids 225, p. 104997. issn: 0045-7930. doi:

10.1016/j.compfluid.2021.104997.

Kumar, Schmelzer, and RPD (2018). “Stochastic turbulence modeling in

RANS simulations via Multilevel Monte Carlo”. In: arXiv 1811.00872.

Parish, Eric J. and Karthik Duraisamy (Jan. 2016). “A paradigm for

data-driven predictive modeling using field inversion and machine learning”.

In: Journal of Computational Physics 305, pp. 758–774. issn: 0021-9991.

doi: 10.1016/j.jcp.2015.11.012.

33

References iv

Porte-Agel, Fernando, Majid Bastankhah, and Sina Shamsoddin (2019).

“Wind-Turbine and Wind-Farm Flows: A Review”. In: Boundary-Layer

Meteorology 174.1, pp. 1–59. issn: 1573-1472. doi:

10.1007/s10546-019-00473-0.

Raissi, M., P. Perdikaris, and G.E. Karniadakis (2019). “Physics-informed

neural networks”. In: Journal of Computational Physics 378, pp. 686–707.

issn: 0021-9991. doi: 10.1016/j.jcp.2018.10.045.

Rudy et al. (Apr. 2017). “Data-driven discovery of partial differential

equations”. In: Science Advances 3.4, e1602614. issn: 2375-2548. doi:

10.1126/sciadv.1602614.

Schmelzer, RPD, and Cinnella (2019). “Discovery of Algebraic Reynolds-Stress

Models Using Sparse Symbolic Regression”. In: Flow, Turbulence and

Combustion 104.2-3, pp. 579–603. issn: 1573-1987. doi:

10.1007/s10494-019-00089-x.

34

References v

Steiner, RPD, and Vire (2020a). “Data-driven RANS closures for wind turbine

wakes under neutral conditions”. In: arXiv 2009.10816.

– (2020b). “Data-driven turbulence modeling for wind turbine wakes under

neutral conditions”. In: Journal of Physics: Conference Series 1618,

p. 062051. issn: 1742-6596. doi: 10.1088/1742-6596/1618/6/062051.

– (2021). “Classifying regions of high model error within a data-driven RANS

closure: Application to wind turbine wakes”. In: arXiv 2106.15593.

Terleth, N. (2019). “Artificial Neural Networks for Flow Field Inference”.

MA thesis. TU Delft.

35