1
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Construction de réseaux de neurones dynamiqueset application à la modélisation de phénomènes de
thermique transitoires
Jonathan Guerra1 3, Patricia Klotz1,Béatrice Laurent2 and Fabrice Gamboa2
13 Novembre 2014
1ONERA2Institut de Mathématiques de Toulouse3Epsilon Ingénierie
JSO ONERA Jonathan Guerra
1
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Table of Contents
1 Context
2 Construction of a spatio-temporal surrogate modelMultilevel optimizationRobust model selectionSensitivity Analysis to reduce input dimension
3 Application to transient thermal engineering
JSO ONERA Jonathan Guerra
1
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Table of Contents
1 Context
2 Construction of a spatio-temporal surrogate model
3 Application to transient thermal engineering
JSO ONERA Jonathan Guerra
2
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Physical problem: Electronic equipment in the avionic bay
• Physical modeling of an avionic bay isn’t easy: numerousinteractions between the equipment, fluid dynamics, radiation...
• The equations to solve are Navier-Stokes ones coupled with heatequation.
→ Thermo-fluidic modeling to perform. Several tools exist:
� Commercial softwares (FLOTHERM, Fluent)� ONERA softwares (CEDRE coupling CHARME and ACACIA)� Physical reduced models such as nodal models
JSO ONERA Jonathan Guerra
2
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Physical problem: Electronic equipment in the avionic bay
• Physical modeling of an avionic bay isn’t easy: numerousinteractions between the equipment, fluid dynamics, radiation...
=⇒ Need of a surrogate model in thermal engineering:
1 Optimization (of the lifetime of the equipment for instance)2 Multilevel simulation of the avionic bay
JSO ONERA Jonathan Guerra
2
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Physical problem: Electronic equipment in the avionic bay
• Physical modeling of an avionic bay isn’t easy: numerousinteractions between the equipment, fluid dynamics, radiation...
=⇒ This complex modeling implies for the surrogate:
� Few learning trajectories� The input/output dimension can be important� Construction time must be reasonable� Long-term in time prediction
JSO ONERA Jonathan Guerra
2
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Table of Contents
1 Context
2 Construction of a spatio-temporal surrogate modelMultilevel optimizationRobust model selectionSensitivity Analysis to reduce input dimension
3 Application to transient thermal engineering
JSO ONERA Jonathan Guerra
3
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Surrogate model for transient thermal engineering
The phenomenon follows heat equation so :
∂T∂t
(x , t) = D∆T (x , t) + C
By discretizing temporally this equation tk = k∆t with k ∈ [0,Nt ]:
T (x , tk) = F[T (x , tk−1)
]An interesting mathematical formulation for the surrogate F is:(
T k1 , · · · ,T k
Np
)= F
(T k−11 , · · · ,T k−1
Np, exogenous variables at tk
)(1)
with Np the number of points of interest
JSO ONERA Jonathan Guerra
4
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Dynamical model for spatio-temporal prediction
• Principle: a recursive formulation is used
• It means that the outputs at time tk−1 become the inputs of the samemodel at time tk
• It can be written{yk = F
(yk−1, uk ,∆tk ; w
)y0 = y0
• The parameters w are theone minimizing:
Elearning =
Ny∑j=1
l∑sample
Nt∑k=1
∥∥∥yk,lj − yk,l
j
∥∥∥2
2
JSO ONERA Jonathan Guerra
5
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Definition of an artificial neural networkMathematically, an artificial neural network can be written:
{s}j∈J1,NsK = {F (e;w)}j∈J1,NsK =
{Nn∑n=1
w0,n,jφ
(Ne∑i=1
wi,n,0ei + w0,n,0
)+ w0,0,j
}j∈J1,NsK
Derivative of the output sj relatively to its weights:
∂sj∂w
=∂Fj (e ; w)
∂w
JSO ONERA Jonathan Guerra
φ is the logistic function
6
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Neural network for spatio-temporal predictionFor spatio-temporal prediction:{
yk = F(yk−1, uk ,∆tk ; w
)y0 = y0
Derivative of the output of the network relatively to its weights:
∂ykj
∂w=
∂Fj (uk , x,∆tk ; w)
∂w
∣∣∣∣∣x=yk−1
+
Ny∑m=1
∂Fj(uk , yk−1,∆tk ; w
)∂yk−1
m
∂yk−1m
∂w
JSO ONERA Jonathan Guerra
6
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Multilevel optimization
Table of Contents
1 Context
2 Construction of a spatio-temporal surrogate modelMultilevel optimizationRobust model selectionSensitivity Analysis to reduce input dimension
3 Application to transient thermal engineering
JSO ONERA Jonathan Guerra
7
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Multilevel optimization
Multilevel optimization - Justification with 2 outputs and 4 neurons
Theoretically, the optimization of the weights should be performedon the following complete network:
yk1 = F1
(yk−11 , yk−1
2 ; w)
yk2 = F2
(yk−12 , yk−1
1 ; w)
y0j = y0j
minw
2∑
j=1
l∑sample
k∑time
∥∥∥yk,lj − yk,l
j
∥∥∥2
2
JSO ONERA Jonathan Guerra
7
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Multilevel optimization
Multilevel optimization - Justification with 2 outputs and 4 neurons
Theoretically, the optimization of the weights should be performedon the following complete network:
yk1 = F1
(yk−11 , yk−1
2 ; w)
yk2 = F2
(yk−12 , yk−1
1 ; w)
y0j = y0j
minw
2∑
j=1
l∑sample
k∑time
∥∥∥yk,lj − yk,l
j
∥∥∥2
2
But it requires to solve a high dimensional optimization problem:for instance with 6 outputs, 4 exogenous variables and 10 neurons,there are 176 weights to fit.
JSO ONERA Jonathan Guerra
7
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Multilevel optimization
Multilevel optimization - Justification with 2 outputs and 4 neurons
Theoretically, the optimization of the weights should be performedon the following complete network:
yk1 = F1
(yk−11 , yk−1
2 ; w)
yk2 = F2
(yk−12 , yk−1
1 ; w)
y0j = y0j
minw
2∑
j=1
l∑sample
k∑time
∥∥∥yk,lj − yk,l
j
∥∥∥2
2
⇒ This great dimension implies a degraded solving of the weights.Solution: A multilevel framework has to be introduced to overcomethis problem : the optimization is decomposed output by output
JSO ONERA Jonathan Guerra
8
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Multilevel optimization
Initialization : Optimization with measured inputs
∀j ∈ J1,Ny K, the weights w0j are optimized for each output yj . The
number of neurons is chosen at this step.Example with two outputs and two neurons per network: Back
JSO ONERA Jonathan Guerra
9
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Multilevel optimization
Iteration it > 0 : Optimization with predicted inputs
• Only weights wj that are used to compute the output yj are beingoptimized at iteration it while the other outputs are computedthanks to the weights optimized at the last step it − 1. Back
JSO ONERA Jonathan Guerra
10
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Multilevel optimization
Multilevel optimization - Recap of the process
1 Initialization of the process:Complexity selection andoptimization of the weights onone-output networks withmeasured inputs (except for theone corresponding to the outputcomputed by the network)
Detail
2 For it > 0: The network isinitialized with w it−1 optimizedat it − 1, and the weightsinvolved in the computation ofeach yj are separately optimized
Detail
JSO ONERA Jonathan Guerra
10
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Robust model selection
Table of Contents
1 Context
2 Construction of a spatio-temporal surrogate modelMultilevel optimizationRobust model selectionSensitivity Analysis to reduce input dimension
3 Application to transient thermal engineering
JSO ONERA Jonathan Guerra
11
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Robust model selection
Model Selection by Cross Validation• Model selection gives an estimation of the generalization errorwhich is used to� Choose the complexity (number of neurons)� Choose the best model for each complexity
• Why Cross Validation?� It uses all the samples at disposal by resampling
• Principle :� Samples at disposal are splitted in 5 groups (also called folds)� One of the folds is only used to test (test samples) the network
built with the 4 other ones (learning samples)=⇒ 5 models are constructed
• The final model is obtained by averaging the outputs of those5 models :
Fj =15
5∑fold=1
F foldj
JSO ONERA Jonathan Guerra
12
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Robust model selection
Cross Validation- How to split spatio-temporal examples?• Whole trajectories: problem of robustness (because some foldcan poorly fill the entire input space)
• Subtrajectories: problem of proximity between test folds andlearning folds
JSO ONERA Jonathan Guerra
12
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Robust model selection
Cross Validation- How to split spatio-temporal examples?• Whole trajectories: problem of robustness (because some foldcan poorly fill the entire input space)→ Solution: stop considering entire trajectories, and split the
time steps into the folds (Subtrajectories)
• Subtrajectories: problem of proximity between test folds andlearning folds
JSO ONERA Jonathan Guerra
12
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Robust model selection
Cross Validation- How to split spatio-temporal examples?
• Whole trajectories: problem of robustness (because some foldcan poorly fill the entire input space)→ Solution: stop considering entire trajectories, and split the
time steps into the folds (Subtrajectories)
• Subtrajectories: problem of proximity between test folds andlearning folds
JSO ONERA Jonathan Guerra
12
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Robust model selection
Cross Validation- How to split spatio-temporal examples?
• Whole trajectories: problem of robustness (because some foldcan poorly fill the entire input space)→ Solution: stop considering entire trajectories, and split the
time steps into the folds (Subtrajectories)
• Subtrajectories: problem of proximity between test folds andlearning folds→ Solution: get rid of the redundant points of the learning
examples before dividing them into the folds Criterions
JSO ONERA Jonathan Guerra
12
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Sensitivity Analysis to reduce input dimension
Table of Contents
1 Context
2 Construction of a spatio-temporal surrogate modelMultilevel optimizationRobust model selectionSensitivity Analysis to reduce input dimension
3 Application to transient thermal engineering
JSO ONERA Jonathan Guerra
13
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Sensitivity Analysis to reduce input dimension
Sensitivity Analysis - Need
• The number of input dimensions can be quite important. Thisimplies:⇒ An important number of weights to optimize⇒ A degraded solution when they are optimized
• Moreover, the aim is to build a parcimonious model (a point ofinterest is not equally influenced by the other ones!)
• Solution: sensitivity analysis is used on the surrogate model toquantify the impact of each input. Once the non-influentinputs are detected, a new network is built without thoseinputs.
JSO ONERA Jonathan Guerra
14
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Sensitivity Analysis to reduce input dimension
Process
Steps
1 Once a first network is built, non-influent inputs aredetermined (thanks to sensitivity analysis)
2 A new network is built without the non-influent inputs of step1 (so with less dimensions)
3 The result after and before Sensitivity Analysis can becompared
JSO ONERA Jonathan Guerra
15
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Sensitivity Analysis to reduce input dimension
Sensitivity Analysis - Choice of the method
• Tool: DGSM (Derivative-based Global Sensitivity Measures)
• Idea: Because it does not depend on time, the sensitivityanalysis will be applied directly on the artificial neural network
• Limit: strong correlations between the inputs of each models(the signification of those coefficient is not obvious in thiscase)
JSO ONERA Jonathan Guerra
16
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Sensitivity Analysis to reduce input dimension
Sensitivity Analysis - Definition of DGSM
• Explanation: it is based on the fact that if the derivative of themodel output relatively to one of its inputs is important, itmeans that this input has a great influence on this output (atleast locally)
• It is defined as follows:To measure the influence of an input xj ∈ H on the outputy = F(x ; w), one has to compute:
νj = E
[(∂F(X ; w)
∂xj
)2]
=
∫HNx
(∂F(x ; w)
∂xj
)2
dµ(x)
• With the hypothesis that ∂F(X ;w)∂xj
is square-integrable, thosecoefficients exist
JSO ONERA Jonathan Guerra
17
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Sensitivity Analysis to reduce input dimension
Sensitivity Analysis - Application to the network
The derivative of a neural network relatively to one of itsinputs has an analytical formulation:
∂Fk (x ; w)
∂xj=
Nn∑n=1
w0,n,kwj ,n,0φ′
(Ne∑i=1
wi ,n,0xi + w0,n,0
)
� k being the index of the output� xj being an exogenous variable or an output at the previous
time step
⇒ The mathematical definition of those coefficient gives a meaningto them
JSO ONERA Jonathan Guerra
17
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Table of Contents
1 Context
2 Construction of a spatio-temporal surrogate model
3 Application to transient thermal engineering
JSO ONERA Jonathan Guerra
18
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Presentation of the test case• To evaluate the presented methodology, a mathematical modelof an equipment is used (Thermal nodal network model):
• The neural network built has4 exogenous inputs and 6outputs which represent:
1 T upper wall2 T left wall3 T radiator4 T air5 T component6 T wall next to the
componentJSO ONERA Jonathan Guerra
18
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Presentation of the test case• To evaluate the presented methodology, a mathematical modelof an equipment is used (Thermal nodal network model):
• The neural network built has4 exogenous inputs and 6outputs which represent:
1 T upper wall2 T left wall3 T radiator4 T air5 T component6 T wall next to the
componentJSO ONERA Jonathan Guerra
19
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Design of Experiment
• The learning samples is composed of 37 trajectories of 300seach.
• They are generated from the following set of inputs in thethermal network model: (T (t = 0),Tamb, I ,P,Flow rate).
JSO ONERA Jonathan Guerra
19
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Design of Experiment• The learning samples is composed of 37 trajectories of 300seach.
JSO ONERA Jonathan Guerra
19
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Design of Experiment• The learning samples is composed of 37 trajectories of 300seach.
JSO ONERA Jonathan Guerra
20
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Test sample - Result of Multilevel optimization• The test sample is an in-flight profile which has the followingexogenous inputs:
• After optimization, it gives this result on the six outputs:
JSO ONERA Jonathan Guerra
20
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Test sample - Result of Multilevel optimization• The test sample is an in-flight profile which has the followingexogenous inputs:
• After optimization, it gives this result on the six outputs:
JSO ONERA Jonathan Guerra
21
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Multilevel Optimization result
• To proove the benefits of the methodology, multilevel splittingand construction without it are compared on this test case.Here are the results :
0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Relative error
P[e
rro
r]<
Re
lative_e
rro
r
Without methodology
Initialization
2nd Iteration
3rd Iteration
• Without methodology, the result is obtained with 30 neurons.With the methodology, it is able to go up to 47
JSO ONERA Jonathan Guerra
22
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Sensitivity Analysis: Computation of the coefficients• The DGSM coefficients are computed on the outputs of thepreviously built network
This gives the “matrix of influences”:
Tamb →P →Q →I →
T1 →T2 →T3 →T4 →T5 →T6 →
1 1 1 1 1 11 1 1 1 1 11 1 1 1 1 11 1 1 1 1 11 1 0 1 1 10 1 0 1 0 01 0 1 1 1 10 0 0 1 0 01 0 1 1 1 10 1 1 1 1 1↓ ↓ ↓ ↓ ↓ ↓T1 T2 T3 T4 T5 T6
• Remark: The air temperature needs all the other outputs to be computed,
but the other outputs don’t need it in input (the flow rate and ambienttemperature suffice in this case)
JSO ONERA Jonathan Guerra
22
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Sensitivity Analysis: Computation of the coefficients• The DGSM coefficients are computed on the outputs of thepreviously built network
only the inputs j with νj > 0.005 are kept:
This gives the “matrix of influences”:
Tamb →P →Q →I →
T1 →T2 →T3 →T4 →T5 →T6 →
1 1 1 1 1 11 1 1 1 1 11 1 1 1 1 11 1 1 1 1 11 1 0 1 1 10 1 0 1 0 01 0 1 1 1 10 0 0 1 0 01 0 1 1 1 10 1 1 1 1 1↓ ↓ ↓ ↓ ↓ ↓T1 T2 T3 T4 T5 T6
• Remark: The air temperature needs all the other outputs to be computed,
but the other outputs don’t need it in input (the flow rate and ambienttemperature suffice in this case)
JSO ONERA Jonathan Guerra
22
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Sensitivity Analysis: Computation of the coefficients• The DGSM coefficients are computed on the outputs of thepreviously built network
only the inputs j with νj > 0.005 are kept:This gives the “matrix of influences”:
Tamb →P →Q →I →
T1 →T2 →T3 →T4 →T5 →T6 →
1 1 1 1 1 11 1 1 1 1 11 1 1 1 1 11 1 1 1 1 11 1 0 1 1 10 1 0 1 0 01 0 1 1 1 10 0 0 1 0 01 0 1 1 1 10 1 1 1 1 1↓ ↓ ↓ ↓ ↓ ↓T1 T2 T3 T4 T5 T6
• Remark: The air temperature needs all the other outputs to be computed,but the other outputs don’t need it in input (the flow rate and ambienttemperature suffice in this case)
JSO ONERA Jonathan Guerra
22
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Sensitivity Analysis: Computation of the coefficients• The DGSM coefficients are computed on the outputs of thepreviously built network
only the inputs j with νj > 0.005 are kept:This gives the “matrix of influences”:
Tamb →P →Q →I →
T1 →T2 →T3 →T4 →T5 →T6 →
1 1 1 1 1 11 1 1 1 1 11 1 1 1 1 11 1 1 1 1 11 1 0 1 1 10 1 0 1 0 01 0 1 1 1 10 0 0 1 0 01 0 1 1 1 10 1 1 1 1 1↓ ↓ ↓ ↓ ↓ ↓T1 T2 T3 T4 T5 T6
• Remark: The air temperature needs all the other outputs to be computed,
but the other outputs don’t need it in input (the flow rate and ambienttemperature suffice in this case)
JSO ONERA Jonathan Guerra
23
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Sensitivity Analysis: Result
• A new network is built with less inputs for each output.
• The result before and after sensitivity analysis can now becompared:Error in ◦C T1 T2 T3 T4 T5 T6
RMSE before S-A 0.45 0.35 0.47 0.07 0.46 0.49RMSE after S-A 0.38 0.53 0.33 0.04 0.33 0.37
• By decreasing the size of the optimization problem (378weights instead of 695 or 245 if we get rid of T2 and T4), thesolution obtained is better
JSO ONERA Jonathan Guerra
23
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Sensitivity Analysis: Result• A new network is built with less inputs for each output.
• By decreasing the size of the optimization problem, thesolution obtained is better
JSO ONERA Jonathan Guerra
24
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Conclusions and perspectives
• This presentation has introduced an innovative construction ofspatio-temporal surrogate model based on :� A multilevel framework to optimize the weights� Cross validation for the model selection� Sensitivity analysis to reduce the input dimension
• The main perspective is the construction of a spatio-temporaldesign of experiment (in progress):� Mathematical form of the exogenous inputs (sinusoidal, crenel
functions, both?)� Choice of the best trajectories to compose it
JSO ONERA Jonathan Guerra
25
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
THANK YOU
Questions?
JSO ONERA Jonathan Guerra
26
Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering
Criterions to exclude some points1 First one: The points tk are kept if:
∣∣∣∣∂yj∂t (tk)
∣∣∣∣+
∣∣∣∣∂yj∂t (tk+1)
∣∣∣∣ > k1 if k 6= Nt
∣∣∣∣∂yj∂t (tNt )
∣∣∣∣ > k2 elsewhere
2 Second one: time between two time steps kept cannotexceeds
threshold = 5 ·∆t = 5 · tNt − t0Nt
3 Third one: if ti is time step kept and if tj is as j > i + 1 anddoes not verify the two first conditions, tj is kept if :∣∣∣∣∣
j∑k=i+1
∂yj
∂t(tk)
∣∣∣∣∣ > k3
JSO ONERA Jonathan Guerra
Back