hpc enabling of openfoam for cfd applications
TRANSCRIPT
HPC enabling of OpenFOAM R© forCFD applications
HPC simulation of volcanic ash plumes and application ofOpenFOAM to CFD volcanological problems
06-08 April 2016, Casalecchio di Reno, BOLOGNA.
Matteo Cerminara – [email protected] Esposti Ongaro – [email protected]
Mattia de’ Michieli Vitturi – [email protected]
Istituto Nazionale di Geofisica e VulcanologiaIstituto Nazionale di Oceanografia e Geofisica Sperimentale
Effusive and explosive eruptions:
The example of Etna volcano
effusive (lava flows)
(2006)
(lava fountains)
(2014)
explosive (ash plumes)
(2015)
Explosivity is mostly controlled by multiphase processes in the magma(melt+gas+crystals):
• Gas phase transitions (gas exsolution, bubble nucleation andexpansion).
• Non-linear magma rheology (brittle transition = fragmentation).
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 2 / 44
Challenges in
magma ascent modeling with OpenFOAM
Effusive regime
• Introduce multicomponent physics and phase transitions
• Bubble nucleation
• Manage phase separation and degassing
Explosive regime
• Wave propagation
• Fragmentation conditions
• Manage different domains (below/above fragmentation)
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 3 / 44
TwoPhaseChangeEulerFoam
A new multiphase multicomponent model with phase change has beendeveloped. Each phase is the mixture of several components andexsolution/evaporation laws can be defined for each component.
Test 1 (decompression experiment):
• validation through comparisonswith decompression experimentsperformed at LMU, Munich.
• silicon oil chosen as analogue formagmatic melt, and saturatedwith Argon at 10MPA
• slow decompression toatmospheric conditions
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 4 / 44
TwoPhaseChangeEulerFoam
A new multiphase multicomponent model with phase change has beendeveloped. Each phase is the mixture of several components andexsolution/evaporation laws can be defined for each component.
Test 1 (decompression experiment):
• validation through comparisonswith decompression experimentsperformed at LMU, Munich.
• silicon oil chosen as analogue formagmatic melt, and saturatedwith Argon at 10MPA
• slow decompression toatmospheric conditions
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 5 / 44
TwoPhaseChangeEulerFoam
A new multiphase multicomponent model with phase change has beendeveloped. Each phase is the mixture of several components andexsolution/evaporation laws can be defined for each component.
Test 2 (gas-driven effusive eruption):
• rise of hot and high-viscositymagmatic mixture:
• liquid phase: melt, dissolvedH2O, dissolved CO2;
• gas phase: exsolved H2O,exsolved CO2, air;
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 6 / 44
Explosive eruptions:
Volcanic plumes
Weak Plume
(Etna, 2011)
Strong Plume
(Chaiten, 2008)
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 7 / 44
Multiscale dynamics
Spatial and temporal scales
• Mass flow rate: Q ∼ UD2
• Reynolds number: Re = UDµ ∼ 109
• Typical Large-Eddy Scale: L ∼ (Str)D
• Kolmogorov’ length: η ∼ L−3/4 ∼ L× 10−7
• Integral length scale (plume height): H ∼ F 1/4S−3/4
(F = g ′UD2 = g ′Q; S = −g/ρ dρdz )
• Integral time scale: τ ∼√
Tv
(1+n)Γg ∼ 1− 5× 102 s
Number of degrees of freedom
• Minimum grid size: ∆x ∼ L
• Minimum number of cells: N ∼ (H/L)3 ∼ Q−1/4
• Minimum time-step ∆t ∼ ∆x/U
• Ntot(Weak) ∼ 1014; Ntot(Strong) ∼ 1011
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 8 / 44
Challenges in
volcanic plume modeling with OpenFOAM
Multiphase flow
• Describe non-equilibrium gas–particle dynamics
• Manage compressibility, turbulence, heat exchange
• Model subgrid turbulence
Numerical solution• Time-step constrained by vent conditions
• Compressible turbulence needs appropriate discretization schemes
• Number of cells is necessarily large! Effective parallelism.
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 9 / 44
Model-related uncertainty
Weak Plume Strong Plume
Does uncertainty blur accuracy?
• A ”blind” intercomparison test (4 models)
• Model-related uncertainty is still much larger than• Error associated with grid size• Error associated with SGS model
• Measurement uncertainty is within the model error
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 10 / 44
Table of contents
1 The ASHEE model
2 Verification and validation study
3 Volcanologic application
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 11 / 44
Eulerian fields
A new model ASHEE (Ash Equilibrium Eulerian) has been developedtaking advantage of the OpenFOAM infrastructure
• it models a mixture of I gas species and J solid particle species, eachwith diameter dj
• the ith gas species is characterized by the following fields in eachpoint (x, t)
• bulk density ρi• velocity ug
• temperature Tg
• the jth solid species is characterized by the following fields in eachpoint (x, t)
• bulk density ρj• velocity uj
• temperature Tj
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 12 / 44
Mass averaged Eulerian fields
The conservation equations for mass, momentum and energy are writtenfocusing on the mass averaged mixture properties:
Mass averaged field ψm
Firstly we define the mixture density as ρm =∑I
i=1 ρi +∑J
j=1 ρj , so thatthe mass fractions of each phase are defined:
• yi = ρi/ρm
• yj = ρj/ρm.
Thus, given a generic field ψ(x, t) for the ith gas species (ψi ) or for thejth solid species (ψj), we define
ψm =I∑
i=1
yiψi +J∑
j=1
yjψj
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 13 / 44
Physical configuration
The Eulerian model in “mixture” formulation
∂tρm +∇ · (ρmum) =∑j∈J
Sj ;
∂t(ρmyi ) +∇ · (ρmugyi ) = 0, i ∈ I ;
∂t(ρmyj) +∇ · (ρmujyj) = Sj , j ∈ J ;
∂t(ρmum) +∇ · (ρmum ⊗ um + ρmTr) =
= −∇p +∇ · T + ρmg +∑j∈J
Sjuj ;
∂t(ρmhm) +∇ · [ρmhm(um + vh)] +
+ ∂t(ρmKm) +−∇ · [ρmKm(um + vK )] =
= ∂tp +∇ · (T · ug − q) + ρm(g · um) +∑j∈J
Sj(hj + Kj) .
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 14 / 44
Physical configuration
• The “mixture” formulation is useful in two-way coupled multiphasesystems, where the mass of the dispersed phase has a non-negligibleeffect on the dynamics.
• The problem is formulated in the dispersed regime εs . 10−3,collisions between particles are disregarded
• It focuses the mathematical problem on the mass averaged fields:ρm , um , hm (improving stability).
• All the effects due to the kinematic decoupling are confined in theterms Tr(yj , vj) , vh(yj , vj , hj , hm) , vK (yj , vj ,Kj ,Km) , keeping intoaccount the effect of the relative velocity vj = uj − ug
• No explicit dependence on the drag functional expression isnecessary at this level
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 15 / 44
Equilibrium Eulerian model
• if particles are perfectly coupled both thermally (Tj = Tg = T ) andkinematically (uj = ug = u) we recover the dusty gas model, whereall the decoupling terms are zero
• in ASHEE we adopt the equilibrium-Eulerian model, where thesystem is thermally perfectly coupled while the kinematic decouplingis approximated via an asymptotic expansion relative to the Stokestime τj :
uj = ug + wj − τj(ag + wj · ∇ug) + O(τj)
where wj = τjg is the settling velocity and ag is the gas phaseacceleration
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 16 / 44
Equilibrium Eulerian model
• The contribution of the particle inertia can be accurately taken intoaccount by using standard Navier-Stokes numerical algorithm,without needing to implicitly solve the drag term
• Particle decoupling and preferential concentration well modeled upto St . 0.2, keeping the advantages of the dusty gas model
• Total number of equation highly reduced for a polydispersed mixture(4J PDEs less):
• Eulerian: I + 3 + 5J• Equilibrium-Eulerian: I + 3 + J.
• Allows to solve efficiently the multiphase dynamics at geophysicalscale for particles of size up to ' 1 mm.
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 17 / 44
Paper on the ASHEE model
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 18 / 44
Numerical configuration
We implemented the following subgrid-scale LES models for the subgridterms of the compressible equilibrium-Eulerian model:
• Compressible Smagorinsky (static and dynamic)
• Turbulent Kinetic Energy model (static and dynamic)
• WALE model (static and dynamic)
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 19 / 44
Numerical configuration
We modified the compressible monophase PISO-PIMPLE algorithm:
• predictor for the mixture density ρm• PIMPLE loop
• solve for the mass fractions yi,j• predictor for the mixture velocity um
• solve for the enthalpy hm, fixing the compressibility• PISO loop
• solve the decoupling• solve for pressure p• correct velocity and fluxes
• solve LES models
• correct mixture density
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 20 / 44
Table of contents
1 The ASHEE model
2 Verification and validation study
3 Volcanologic application
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 21 / 44
DHITDecaying Homogeneous and Isotropic Turbulence (2563 cells)The solver is able to simulate accurately the turbulence.
k
E(k)
100 101 10210-10
10-9
10-8
10-7
10-6
10-5
10-4
10-3
10-2
DNS Bernardini & Pirozzoli
OpenFoam
Figure: Test case validation: comparison with an eight order DNS after one large-eddy turnovertime (10000 time steps).
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 22 / 44
Scalability
64 128 256 512 10240,000,100,200,300,400,500,600,700,800,901,001,10
FermiPLX
# cores
Efficiency
64 128 256 512 10240
200
400
600
800
1000
1200
Fermi
Ideal
PLX
# coresSpeedup
Figure: Scalability test on PLX and FERMI environments (with little outputwork). Collaboration between us and Paride Dagna.
In order to fix ideas, the solver reach a velocity in the range 1÷10 Mcells/s on 1024 cores
(multiphase÷monophase) .
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 23 / 44
SGS LES models
1e-06
1e-05
0.0001
0.001
1 10
ener
gy s
pect
rum
E(k
)
wavenumber k
[noM] 2563
[noM] 323
[sma] 323
[oneEqEddy] 323
[wale] 323
Figure: DHIT using static SGS LES models in a box with 323 cells
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 24 / 44
SGS LES models
1e-06
1e-05
0.0001
0.001
1 10
ener
gy s
pect
rum
E(k
)
wavenumber k
[noM] 2563
[dynSma] 643
[moin] 323
[dynSma] 323
[dynOneEqEddy] 323
[dynWale] 323
[cubic] [dynOneEqEddy] 323
Figure: DHIT using dynamic SGS LES models in a box with 323 cells
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 25 / 44
Kinematic decoupling
Slice of the turbulent box at t/τe ' 2.2. The two panels representrespectively a logarithmic color map of yj=2 (Stmax = 0.5) and of |ag|
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 26 / 44
Kinematic decoupling
1e-05
0.0001
0.001
0.01
0.1
1
0.01 0.1 1
τ j τξ ⟨P⟩ j
(LE
S);
τ
j τη ⟨
P⟩ j
(D
NS
)
Stξ (LES); Stη (DNS)
[noM], 2563
[dynWale], 323
1.52*St2
Rani 2003
fit of Rani 2003
Figure: Evolution of the degree of preferential concentration with Stξ (LES) orStη (DNS). We obtain a good agreement between equilibriumEulerianLES/DNS and Lagrangian DNS simulations.
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 27 / 44
Advection schemes
Figure: Advection test case from Holzmann-cfd solved with ASHEE
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 28 / 44
Advection schemes
-0.2
0
0.2
0.4
0.6
0.8
1
1.2
-0.02 -0.015 -0.01 -0.005 0 0.005 0.01 0.015 0.02
trac
er
time
upwindlinearUpwind
linearlimitedLinear
limitedLimitedLinearvanLeer
limitedVanLeercubic
limitedCubicUMISTSFCD
QUICKfilteredLinear
Figure: Tracer mass fraction along the cavity section.
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 29 / 44
Advection schemes
10-6
10-5
10-4
10-3
1 10
E(k
)
k
[noM], 2563
[dynWale], 323
[dynWale], [MUSCL], 323
[dynWale], [UMIST], 323
[dynWale], [vanLeer], 323
[dynWale], [rk4blended], 323
[dynWale], [filtlin0200], 323
Figure: Performances of OpenFOAM schemes in DHIT LES
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 30 / 44
Other benchmarks
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
ρ
x/ct
analytic solutionrhoCentralFoam
ASHEE
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
ρ
x/ct
Sod’s shock tube
1
10
10 100 1000
Nu
t [s]
Ra = 1e6, Nu = 8.800
Ra = 1e5, Nu = 4.519
Ra = 1e4, Nu = 2.243
Ra = 1e3, Nu = 1.118
Natural convection
Experimental plumeMixing
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 31 / 44
Table of contents
1 The ASHEE model
2 Verification and validation study
3 Volcanologic application
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 32 / 44
Plinian eruption
We have used ASHEE to simulate volcanic eruptions (scale ≈ 100 km)from the vent (mass eruption rate up to Mton/s) to the atmosphere
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 33 / 44
Volcanic plumes
We are participating to an international benchmark initiative involvingtwo numerical simulations:
weakPlume• duration: 0.2 hours
• Mass flow rate: 1.5 ∗ 106 kg/s
• Exit velocity: 135 m/s
• Exit temperature: 1273 K
• Exit gas fraction: 3 wt%
• Grain size distribution:• coarse: 1 mm• fine: 62.5 µm
strongPlume
• duration: 2.5 hours
• Mass flow rate: 1.5 ∗ 109 kg/s
• Exit velocity: 275 m/s
• Exit temperature: 1053 K
• Exit gas fraction: 5 wt%
• Grain size distribution:• coarse: 0.5 mm• fine: 15.6 µm
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 34 / 44
Effects of resolution and SGS modelTime and horizontal average of the velocity field, with 8, 16 and 32 cellsin a vent diameter; with and without decoupling model (weakPlume)
z [k
m]
U [m/s]
[dynWale], high res., [eqEu][dynWale], mid. res., [eqEu][dynWale], low res., [eqEu][dynWale], low res., [dusty]
0
2
4
6
8
10
12
14
0 20 40 60 80 100 120 140
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 35 / 44
Effects of resolutionTime and horizontal average of the velocity field, with 8, 16 and 32 cellsin a vent diameter (strongPlume)
z [k
m]
U [m/s]
[dynWale], high res., [eqEu][dynWale], mid. res., [eqEu][dynWale], low res., [eqEu]
0
5
10
15
20
25
30
35
40
45
0 50 100 150 200 250 300
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 36 / 44
Effects of SGS modelTime and horizontal average of the velocity field, with 16 and 32 cells ina vent diameter; with different SGS models (strongPlume)
z [k
m]
U [m/s]
[dynWale], high res., [eqEu][noM], high res., [eqEu]
[dynWale], mid. res., [eqEu][moin], mid. res., [eqEu]
[oneEqEddy], mid. res., [eqEu]
0
5
10
15
20
25
30
35
40
45
0 50 100 150 200 250 300
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 37 / 44
Volcanic infrasound
Compressibility and turbulence generate infrasound
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 38 / 44
Observations & simulations
Observation of 8 March 2005 MountSt. Helens eruption (Matoza et al.2009) Simulation with ASHEE of the
strongPlume eruption
0.001
0.01
0.1
1
10
100
0.01 0.1 1 10
20
30
40
50
60
70 0.01 0.1 1
Ep(
Str)
[Pa
]
Ep(
Str)
[dB
re
20 µ
Pa]
Str
f [Hz]
-11/4
Ep
-0.6-0.4-0.2
0 0.2 0.4 0.6 0.8
1 1.2
0 100 200 300 400 500 600 700 800
p [k
Pa]
t [s]
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 39 / 44
Summary of the second part
• We have developed the compressible version of theequilibrium-Eulerian model (ASHEE model)
• We have implemented it into the OpenFOAM infrastructure
• the solver has been tested up to 1024 cores. It shows a reasonableefficiency (> 60%) on the Cineca Fermi infrastructure
• The numerical scheme has been tested and chosen to maximizeaccuracy and stability of dynamic LES simulations
• A number of different benchmarks have been performed satisfactorily
• The new solver is able to accurately and efficiently captureclustering, preferential concentration and settling up to 1 mmparticles
• Applications to volcanological scale have been performed, comparingresults with other models and with observations
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 40 / 44
Volcanic hazard assessment:
an HPC/HTC challenge
Complex physics, multiscale dynamics
• Non-newtonian (non-linear) rheology
• Multiphase flows
• Broad range of spatial/temporal scales
Uncertainty on initial/boundary conditions
• Partial knowledge of the initial state
• Catastrophic dynamics
• Difficulty of measurements
• Repeatability issues
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 41 / 44
Future perspectives
Magma ascent modeling
• Implementation of fragmentation conditions
• More complex geometries
• Fluid-structure interaction
Volcanic plume modeling
• Add an arbitrary wind field
• Add volcano topography
• Add micro-physics, as water condensation, ash particles shape andaggregation
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 42 / 44
Future perspective
Numerical issues• Non-reflecting boundary conditions
• Improve the advection scheme
• Implement a shock capturing correction
• Optimize parallel linear algebra in OpenFOAM
Volcanic hazard assessment• Sensitivity and uncertainty analysis (coupled with Dakota).
• Coupling with meteorological solvers.
• ”Nowcasting” of volcanic plume scenarios.
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 43 / 44
Thank You!
Contacts• research web-page:https://sites.google.com/site/matteocerminara
• CFD movies:https://www.youtube.com/user/MatteoCerminara
Acknowledgments: we acknowledge CINECA for the availability ofhigh-performance computing resources and technical support on portingOpenFOAM on HPC architectures in the framework of ISCRA projects:IsB06 VolcFOAM, IsC26 VolcAshP and IsC07 GEOFOAM.
Matteo Cerminara et al. / HPC enabling of OpenFOAM for CFD applications / Modeling 44 / 44