numerical simulations in fluid dynamics using gpu a practical
TRANSCRIPT
Numerical Simulations in Fluid Dynamics
using GPU: a Practical Introduction
Dr Tomasz P Bednarz, Con Caris, Dr John A Taylor
22 September 2010
Content (behind the scene)
β’ Introduction
β’ What is Computational Fluid Dynamics (CFD) and where is it used?
β’ Governing equations
β’ Navier-Stokes equations for conservation of mass, momentum &
energy equation for thermal fluid flow
β’ Discretisation
β’ Rectangular and Boundary Fitted Coordinates
β’ Algorithms
β’ HSMAC methodology, UPWIND and UTOPIA schemes.
β’ Verification
β’ Driven Cavity and Natural Convection case.
β’ Migration to GPU
β’ Migration of the existing code to OpenCL.
β’ Applications
β’ Scaling Analysis, Magneto-thermal convection, exchange flows.
β’ Closing Remarks
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
CSIRO overview
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
6500+ staff over 55 locations
CSIRO today: a snapshot
160+ active licences of CSIRO innovation
20+ spin-off companies in six years
Ranked in top 1% in 14 research fields
One of the largest & most diverse in the world
Australiaβs national science agency
Building national prosperity and wellbeing
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
Fluid Dynamics
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
Introduction
β’ Experiments and CFD?
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
Courtesy of High Field Magnet Laboratory, NL
Air
Air
N2
gas
Wakayama Jet
Exchange Flows in Reservoirs β Cooling Case
β’ Water circulation in reservoirs is driven by thermal gradients changing
during day and night cycles.
0
1
2
3
4
5
0 10 20 30 40 50
t [s]
dT
[m
m]
dT ~ (k t )0.5 at Gr = 7.82x103 and Pr = 6.91
dT ~ (k t )0.5 at Gr = 2.04x104 and Pr = 8.05
CSIRO. Insert presentation title, do not remove CSIRO from start of footer
dT ~ (k Β· t)1/2 Scaling relation:
Exchange Flows in Reservoirs β Cooling Case
t = 100[s]
t = 200[s]
t = 500[s]
t = 700[s]
t = 1000[s]
t = 4000[s]
17
18
19
20
21
22
23
24
0 7 14 21 28 35 42 49 56 63 70
Time [min]
Tem
pera
ture
[oC
] a
b
c
d
e
f
g
h
i
j
k
Exchange Flows in Reservoirs β Diurnal Case
Pr = 6.82,
Gr = 3.52Γ104
9
Dt
PIV result
unsharp mask
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
DEM and SPH examples
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
Courtesy of Paul Cleary Computational Modelling
Group
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
Equations Discretisation
Implementation Verification
Results Transfer CPU code to
GPU code Verification Results!
Process
Governing Equations,
Discretisation Procedure,
Algorithms.
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
Governing equations (dimensional)
β’ Continuity equation
ππ’
ππ₯+ππ£
ππ¦= 0
ππ’
ππ‘+ π’
ππ’
ππ₯+ π£
ππ’
ππ¦= β
1
π
ππ
ππ₯+ Ξ½
π2π’
ππ₯2 +π2π’
ππ¦2 + ππππ¦_πππππ_π₯
β’ Momentum equations
ππ£
ππ‘+ π’
ππ£
ππ₯+ π£
ππ£
ππ¦= β
1
π
ππ
ππ¦+ Ξ½
π2π£
ππ₯2 +π2π£
ππ¦2 + ππππ¦_πππππ_π¦
β’ Energy equation
ππ
ππ‘+ π’
ππ
ππ₯+ π£
ππ
ππ¦= πΌ
π2π
ππ₯2 +π2π
ππ¦2
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
unsteady
acceleration
convective
acceleration pressure
gradient
viscous
term body
force
β’ Continuity equation
ππ
ππ+ππ
ππ= 0
Governing equations (non-dimensional)
β’ Momentum equations
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
ππ
ππ+ π
ππ
ππ+ π
ππ
ππ= βπ΄
ππ
ππ+ π΅
π2π
ππ2 +π2π
ππ2 + πΆπΉπ
ππ
ππ+ π
ππ
ππ+ π
ππ
ππ= βπ΄
ππ
ππ+ π΅
π2π
ππ2 +π2π
ππ2 + πΆπΉπ
ππ
ππ+ π
ππ
ππ+ π
ππ
ππ= π·
π2π
ππ2 +π2π
ππ2
β’ Energy equation
π =π’
π’β
β’ Where:
π =π£
π£β π =
π₯
π₯β π =π¦
π¦β π =
π
πβ π =
π‘
π‘β π =
π β πβ
πβ
Staggered grid
ππ,π ππ+1,π
ππ,π+1 ππ+1,π+1
π£π,π π£π+1,π
π£π+1,π+1 π£π,π+1
π’π,π
π’π,π+1
π’π+1,π+1
π’π+1,π
π π + π
π
π + π
ππππ(π, π)
β’ Different variables are located at
different locations:
β’ scalar variables (pressure,
temperature, concentration) are
stored at the centre of the cell,
β’ the horizontal velocity component is
sampled at the centre of the vertical
cell face,
β’ the vertical velocity component is
sampled at the centre of the
horizontal cell face.
β’ The discrete values of u, v and p can be thought as located on three
separate grids, each shifted by half grid spacing to the bottom, to the
left, and to the lower left respectively.
β’ The staggered arrangement of the unknowns prevents possible
pressure oscillations.
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
Staggered grid
β’ In 3D, the grid is set up exactly the same way.
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
Finite Difference (FD) method
β’ The continuous problem domain is replaced by a finite-difference
mesh or grid.
ππ,π ππ+1,π ππβ1,π
ππ,π+1
ππ,πβ1
βπ
βπ
β’ If we think about Ui,j as U(X0,Y0) then:
ππ+1,π = π(π0 + βπ, π0)
ππ,π+1 = π(π0, π0 + βπ)
ππβ1,π = π(π0 β βπ, π0)
ππ,πβ1 = π(π0, π0 β βπ)
β’ The idea of FD representation for derivative can
be introduced by recalling the definition of the
derivative for the function U(X,Y) at X0 and Y0:
ππ
ππ= lim
βπβ0
π π0 + βπ, π0 β π(π0, π0)
βπ
β’ The difference approximation can be put on a formal basis through
the use of a Taylor-series expansion for U(X0+βX,Y0) about (X0,Y0):
π π0 + βπ, π0 = π π0, π0 +ππ
ππ 0
βπ +π2π
ππ2 0
βπ 2
2!+π3π
ππ3 0
βπ 3
π!+ β―
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
Computational domain
π = π π = π π = π π = π π = π π = π π = π π = π
π = π
π = π
π = π
π = π
π = π
π΅πΏ = π; π΅π = π
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
boundary strip
boundary of computational domain
computational domain
πΏ
π
Computational domain
π = π π = π π = π π = π π = π π = π π = π π = π
π = π
π = π
π = π
π = π
π = π
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
π΅πΏ = π; π΅π = π
πΏ
π
Discretisation of the continuity equation
β’ Continuity equation:
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
ππ
ππ+ππ
ππ= 0
ππ
ππ=
ππ β ππ€
βπ
ππ
ππ=
ππ β ππ βπ
ππ
ππ+ππ
ππ=
ππ,π β ππβ1,π
βπ+ππ,π β ππ,πβ1
βπ= 0
β’ All derivatives are calculated at the centre of the cell.
βπ
ππ
ππ
βπ
ππ ππ€ ππ = ππ,π
ππ€ = ππβ1,π
ππ = ππ,π ππ = ππ,πβ1
Discretisation of the x-momentum equation
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
β’ All derivatives are calculated at the right vertical face of the cell.
ππ
ππ+ π
ππ
ππ+ π
ππ
ππ= βπ΄
ππ
ππ+ π΅
π2π
ππ2 +π2π
ππ2 + πΆπΉπ
π πΈ π
π
π
πΈπ
πΈπ
ππ
ππ
π π€ ππ
ππ
ππ
π€π
π€π
π
π
ππ
π€π
π€π
ππ
ππ
π€ππ πππ
πππ
πππ
ππ
ππ=
πππ+1 β ππ
π
βπ
βπ
βπ
β’ Time advancement.
Discretisation of the x-momentum equation
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
β’ All derivatives are calculated at the right vertical face of the cell.
ππ
ππ+ π
ππ
ππ+ π
ππ
ππ= βπ΄
ππ
ππ+ π΅
π2π
ππ2 +π2π
ππ2 + πΆπΉπ
π πΈ π
π
π
πΈπ
πΈπ
ππ
ππ
π π€ ππ
ππ
ππ
π€π
π€π
π
π
ππ
π€π
π€π
ππ
ππ
π€ππ πππ
πππ
πππ
ππ
ππ=
πππ+1 β ππ
π
βπ
πΉππ = πππ
ππ= ππ
πππ β ππ€
2βπ
βπ
βπ
β’ This term can be improved using
upwind scheme !!!
Discretisation of the x-momentum equation
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
β’ All derivatives are calculated at the right vertical face of the cell.
ππ
ππ+ π
ππ
ππ+ π
ππ
ππ= βπ΄
ππ
ππ+ π΅
π2π
ππ2 +π2π
ππ2 + πΆπΉπ
π πΈ π
π
π
πΈπ
πΈπ
ππ
ππ
π π€ ππ
ππ
ππ
π€π
π€π
π
π
ππ
π€π
π€π
ππ
ππ
π€ππ πππ
πππ
πππ
ππ
ππ=
πππ+1 β ππ
π
βπ
πΉππ = πππ
ππ= ππ
πππ β ππ€
2βπ
πΉππ = πππ
ππ=
ππ + πππ + πππ + ππ 4
πππ β πππ
2βπ
βπ
βπ
β’ This term can be improved using
upwind scheme !!!
Discretisation of the x-momentum equation
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
β’ All derivatives are calculated at the right vertical face of the cell.
ππ
ππ+ π
ππ
ππ+ π
ππ
ππ= βπ΄
ππ
ππ+ π΅
π2π
ππ2 +π2π
ππ2 + πΆπΉπ
π πΈ π
π
π
πΈπ
πΈπ
ππ
ππ
π π€ ππ
ππ
ππ
π€π
π€π
π
π
ππ
π€π
π€π
ππ
ππ
π€ππ πππ
πππ
πππ
ππ
ππ=
πππ+1 β ππ
π
βπ
πΉππ = πππ
ππ= ππ
πππ β ππ€
2βπ
πΉππ = πππ
ππ=
ππ + πππ + πππ + ππ 4
πππ β πππ
2βπ
ππ πΈπππ = π΄ππ
ππ= π΄
ππΈπ+1 β ππ
π+1
βπ
βπ
βπ
Discretisation of the x-momentum equation
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
β’ All derivatives are calculated at the right vertical face of the cell.
ππ
ππ+ π
ππ
ππ+ π
ππ
ππ= βπ΄
ππ
ππ+ π΅
π2π
ππ2 +π2π
ππ2 + πΆπΉπ
π πΈ π
π
π
πΈπ
πΈπ
ππ
ππ
π π€ ππ
ππ
ππ
π€π
π€π
π
π
ππ
π€π
π€π
ππ
ππ
π€ππ πππ
πππ
πππ
ππ
ππ=
πππ+1 β ππ
π
βπ
πΉππ = πππ
ππ= ππ
πππ β ππ€
2βπ
πΉππ = πππ
ππ=
ππ + πππ + πππ + ππ 4
πππ β πππ
2βπ
ππ πΈπππ = π΄ππ
ππ= π΄
ππΈπ+1 β ππ
π+1
βπ
π·πΌπΉπΉπ = π΅πππ β ππ
βπβππ β ππ€
βπ
1
βπ+
πππ β ππ
βπβππ β πππ
βπ
1
βπ
βπ
βπ
ππ
ππ+ π
ππ
ππ+ π
ππ
ππ= βπ΄
ππ
ππ+ π΅
π2π
ππ2 +π2π
ππ2 + πΆπΉπ
Discretisation of the y-momentum equation
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
π πΈ π
π
π
πΈπ
πΈπ
ππ
ππ
π π€ ππ
ππ
ππ
π€π
π€π
π
π
ππ
π€π
π€π
ππ
ππ
π€ππ πππ
πππ
πππ
π·πΌπΉπΉπ = π΅πππ β ππ
βπβππ β ππ€π
βπ
1
βπ+
πππ β ππβπ
βππ β ππ βπ
1
βπ
βπ
βπ ππ πΈπππ = π΄ππ
ππ= π΄
πππ+1 β ππ
π+1
βπ
πΉππ = πππ
ππ= ππ
πππ β ππ 2βπ
πΉππ = πππ
ππ=
ππ€π + πππ + ππ + ππ€
4
πππ β ππ€π
2βπ
ππ
ππ=
πππ+1 β ππ
π
βπ
β’ All derivatives are calculated at the upper horizontal face of the cell.
ππ
ππ+ π
ππ
ππ+ π
ππ
ππ= π·
π2π
ππ2 +π2π
ππ2
Discretisation of the energy equation
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
π πΈ π
π
π
πΈπ
πΈπ
ππ
ππ
π π€ ππ
ππ
ππ
π€π
π€π
π
π
ππ
π€π
π€π
ππ
ππ
π€ππ πππ
πππ
πππ
π·πΌπΉπΉπ = π·ππΈ β ππ
βπβππ β ππ
βπ
1
βπ+
ππ β ππβπ
βππ β ππβπ
1
βπ
βπ
βπ
ππ
ππ=
πππ+1 β ππ
π
βπ
β’ All derivatives are calculated in the centre of the cell.
πΉππ = πππ
ππ=
ππ + ππ€
2
ππΈ β ππ2βπ
πΉππ = πππ
ππ=
ππ + ππ 2
ππ β ππ2βπ
Velocity iteration
ππ
ππ+ π
ππ
ππ+ π
ππ
ππ= βπ΄
ππ
ππ+ π΅
π2π
ππ2 +π2π
ππ2 + πΆπΉπ
ππ
ππ+ π
ππ
ππ+ π
ππ
ππ= βπ΄
ππ
ππ+ π΅
π2π
ππ2 +π2π
ππ2 + πΆπΉπ
ππΌπΏ ππΌπ π·πΉπ¬πΊπΊπΌ π«π°πππΌ ππΏ
ππ½πΏ ππ½π π·πΉπ¬πΊπΊπ½ π«π°πππ½ ππ
β’ Momentum equations are rewritten to get velocities:
ππ,ππ+1 = ππ,π
π + βπ βπΉππ β πΉππ β ππ πΈπππ + π·πΌπΉπΉπ + πΉπ
ππ,ππ+1 = ππ,π
π + βπ βπΉππ β πΉππ β ππ πΈπππ + π·πΌπΉπΉπ + πΉπ
πΌπ,ππ+π
π½π,ππ+π
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
ππ
ππ+ π
ππ
ππ+ π
ππ
ππ= π·
π2π
ππ2 +π2π
ππ2
Temperature iteration
ππ»πΏ ππ»π π«π°πππ»
β’ Energy equation is rewritten to get temperature field:
ππ,ππ+1 = ππ,π
π + βπ βπΉππ β πΉππ + π·πΌπΉπΉπ
π½π,ππ+π
β’ Energy equation:
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
OUTER LOOP
Flow Chart of the CPU Fluid Solver
START
Read Configuration
Allocate Memory for Flow
Field Tables
Apply Boundary
Conditions (BC)
Calculate Tentative
Velocities (U, V)
Calculate Pressure
Correction
Correct Pressure and
Velocity Fields, Apply BC
Converged? NO
Calculate Energy
Equation (if required)
YES
Final time
achieved?
Advance All Flow Tables
and Increase Time
NO
Save Results: Flow
Fields, etc...
FINISH
YES
INNER LOOP
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
HSMAC
β’ HSMAC (Highly Simplified Marker and Cell): a method to solve
for pressure and velocity components.
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
ππ
ππ+ π
ππ
ππ+ π
ππ
ππ= βπ΄
ππ
ππ+ π΅
π2π
ππ2 +π2π
ππ2 + πΆπΉπ
ππ
ππ+ π
ππ
ππ+ π
ππ
ππ= βπ΄
ππ
ππ+ π΅
π2π
ππ2 +π2π
ππ2 + πΆπΉπ
β’ Momentum equations:
β’ Can be rewritten as follows:
ππ,ππ+1 = ππ,π
π + βπ πΉπ,ππ β π΄
ππ+1,ππ+1 β ππ,π
π+1
βπ
ππ,ππ+1 = ππ,π
π + βπ πΊπ,ππ β π΄
ππ,π+1π+1 β ππ,π
π+1
βπ
β’ Once new pressure are given, we can get new velocity components
from these equations.
Harlow and Welch, Physics of Fluids 8, 2182-2189, 1965. Amsden and Welch, JPC 10, 324-325, 1970.
HSMAC
β’ These velocity components are required to satisfy the equation of
continuity:
ππ
ππ+ππ
ππ= 0
β’ The solution is sought with the Newton-Raphson method in
stepwise way:
β’ At first, with the tentative value for pressure, the tentative velocity
components are computed:
πππ,ππ+1 = ππ,π
π + βπ πΉπ,ππ β π΄
πππ+1,ππ+1 β πππ,π
π+1
βπ
πππ,ππ+1 = ππ,π
π + βπ πΊπ,ππ β π΄
πππ,π+1π+1 β πππ,π
π+1
βπ
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
HSMAC
β’ The tentative velocity and the pressure fields need to be corrected to
give better approximation at step m+1:
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
ππ+1 π+1 = ππ π+1 + πβ²
ππ+1 π+1 = ππ π+1 + πβ²
ππ+1 π+1 = ππ π+1 + πβ²
β’ Then, the LHS of continuity equation is expressed as:
π·π+1π,ππ+1 =
ππ+1π,ππ+1 β ππ+1
πβ1,ππ+1
βπ+
ππ+1π,ππ+1 β ππ+1
π,πβ1π+1
βπ
π·ππ,ππ+1 =
πππ,ππ+1 β ππ
πβ1,ππ+1
βπ+
πππ,ππ+1 β ππ
π,πβ1π+1
βπ
β’ The equation of continuity is to be satisfied in the new time step (π + 1)βπ
so let us assume π·π,ππ+1 = 0 though π·π,π
π β 0 with the non-convergent
incorrect velocity components for π and π.
β’ Newton-Raphson method provides the stepwise approach for the
solution to satisfy π·π,ππ+1 = 0.
HSMAC
β’ These are inserted in continuity equation:
πππ,ππ+1 = ππ,π
π + βπ πΉπ,ππ β
πππ+1,ππ+1 β πππ,π
π+1
βπ
πππ,ππ+1 = ππ,π
π + βπ πΊπ,ππ β
πππ,π+1π+1 β πππ,π
π+1
βπ
ππ,ππ+1 β ππβ1,π
π+1
βπ+ππ,ππ+1 β ππ,πβ1
π+1
βπ=
ππ,ππ β ππβ1,π
π
βπ+ππ,ππ β ππ,πβ1
π
βπ
+ βπ πΉπ,ππ β πΉπβ1,π
π
βπ+πΊπ,ππ β πΊπ,πβ1
π
βπ
+ βππ+1,ππ+1 β ππ,π
π+1
βπ+ππ,ππ+1 β ππβ1,π
π+1
βπ
1
βπ
+ βππ,π+1π+1 β ππ,π
π+1
βπ+ππ,ππ+1 β ππ,πβ1
π+1
βπ
1
βπ
π«π,ππ+π
π«π,ππ = π
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
HSMAC
β’ Calculate ππ so π·ππ+1 becomes zero using Newton-Raphson method:
π+1ππ,ππ+1 = π ππ,π
π+1 β ππ·π,π
π+1
πππ·π,π
πππ,π
π+1 = π ππ,ππ+1 + ππ,π
β²
π·ππ,ππ+1 =
πππ,ππ+1 β ππ
πβ1,ππ+1
βπ+
πππ,ππ+1 β ππ
π,πβ1π+1
βπ= π·π·
β’ where:
πππ·π,π
πππ,π
π+1
= βπ1
βπ
1
βπ+
1
βπ+
1
βπ
1
βπ+
1
βπ= π·πΈπΏ
β’ so:
ππ,πβ² = β
π·π·
π·πΈπΏ
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
HSMAC
β’ Since the pressure is corrected, the velocity components should be
corrected to reflect this correction in pressure.
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
β’ Taylor expansion of velocity components with pressure in terms of
small pressure increment ππβ² :
π+1ππ,ππ+1 = πππ,π
π+1 + ππππ,π
πππ,π
π+1
ππ,πβ² +β― = πππ,π
π+1 + ππ,πβ² +β―
π+1ππ,ππ+1 = π ππ,π
π+1 + ππππ,π
πππ,π
π+1
ππ,πβ² +β― = π ππ,π
π+1 + ππ,πβ² +β―
ππβ² =
βπ
βπππ,πβ² ππ
β² =βπ
βπππ,πβ²
ππ€β² = β
βπ
βπππ,πβ² ππ
β² = ββπ
βπππ,πβ²
Non-slip boundary conditions
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
fluid
boundary
wall
β’ To satisfy the non-slip condition, the continuous velocities should
vanish at the boundary:
U V
ππππ’πππππ¦ = 0; ππππ’πππππ¦ = 0
β’ Therefore:
π = π π = π
π0,π = 0; π0,π = βπ1,π
π π0,π
π0,π π1,π
Free-slip boundary conditions
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
fluid
boundary
wall
β’ The velocity component normal to boundary and normal derivative of
velocity component tangent to boundary should vanish:
U V
ππππ’πππππ¦ = 0; ππ ππ πππ’πππππ¦ = 0
β’ Therefore:
π = π π = π
π0,π = 0; π0,π = π1,π
π π0,π
π0,π π1,π
Case 1: Lid Driven Cavity
β’ Used as a validation case for new codes (same here?).
β’ Fluid is contained in a square cavity with three rigid walls (bottom, left and
right) and one moving wall (top).
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
ππ
ππ+ π
ππ
ππ+ π
ππ
ππ= β
ππ
ππ+
1
π π
π2π
ππ2+π2π
ππ2
ππ
ππ+ π
ππ
ππ+ π
ππ
ππ= β
ππ
ππ+
1
π π
π2π
ππ2+π2π
ππ2
ππ
ππ+ππ
ππ= 0
β’ Reynolds number is a non dimensional parameter that
measure of the ratio of inertial forces to viscous forces:
π π =
πππππ‘πππ πππππ
π£ππ πππ’π πππππ=
ππΏ
π£
Ve
locity
field
Case 1: Lid Driven Cavity
Re = 100 Re = 400 Re = 1000
Driven Cavity Test at Re = 1000
-0.7
-0.5
-0.3
-0.1
0.1
0.3
0.5
0.7
0.9
1.1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
x and y location (respectively)
v a
nd
u v
elo
cit
y c
om
po
ne
nts
(re
sp
ec
tiv
ely
)
u- along y=0.5 (Ghia)
u- along y=0.5 (my code)
v- along x=0.5 (my code)
v- along x=0.5 (Ghia)
v- velocity component along y=0.5
u- velocity component along x=0.5
Re = 1000
str
eam
fu
nctio
n
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
Ve
locity
field
Accuracy improvements
ππ
ππ+ π
ππ
ππ+ π
ππ
ππ= β
ππ
ππ+
1
π π
π2π
ππ2+π2π
ππ2
β’ Upwind (central difference replaced by one sided-differences),
β’ Upwind + Central Difference,
β’ QUICK,
Driven Cavity Test at Re = 1000
-0.55
-0.45
-0.35
-0.25
-0.15
-0.05
0.05
0.15
0.25
0.35
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
x location
v v
elo
cit
y c
om
po
ne
nt
Ghia
my old FV code (SIMPLE + QUICK)
my code: Central
my code: Upwind
Fluent: Upwind
Fluent: 2nd upwind
my code: Up + Ce
my code: UTOPIA
Driven Cavity at Re = 1000
β’ UTOPIA:
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
Case 2: Natural Convection
β’ Governing equations:
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
Prandtl number describes the relative strength of the
diffusion of momentum to that of heat.
Rayleigh number expresses the ratio of the buoyancy
forces to viscous forces.
ππ
ππ+ π
ππ
ππ+ π
ππ
ππ= β
ππ
ππ+ ππ
π2π
ππ2+π2π
ππ2
ππ
ππ+ π
ππ
ππ+ π
ππ
ππ= β
ππ
ππ+ ππ
π2π
ππ2+π2π
ππ2+ π ππππ
ππ
ππ+ππ
ππ= 0
ππ
ππ+ π
ππ
ππ+ π
ππ
ππ=
π2π
ππ2+π2π
ππ2
β’ Characteristic non dimensional variables:
Conv. if Dr 0
π π =ππ½(πβ β ππ)π
3
πΌπ
ππ =π
πΌ
πβ ππΆ
ππ
ππ= 0
ππ
ππ= 0
π = π = 0 on all walls
Temperature (TLC)
Case 2: Natural Convection
πΉπ = πππ, π·π = π. ππ
πΉπ = πππ, π·π = π. ππ
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
128x128
Max of V- at Y = 0.5:
HSMAC: 19.56 at x = 0.1114 err 0.2906%
Benchmark: 19.617 at x = 0.119
Max of U- at X = 0.5:
HSMAC: 16.108 at y = 0.8324 err 0.4327%
Benchmark: 16.178 at y = 0.823
Max of V- at Y = 0.5:
HSMAC: 219.5 at x = 0.03622 err 0.0638%
Benchmark: 219.36 at x = 0.03790
Max of U- at X = 0.5:
HSMAC: 64.492 at y = 0.85 err 0.2135%
Benchmark: 64.630 at y = 0.850
π π = 104
π π = 106
β’ Variables to be compared: temperature and velocity profiles, stream function, Nusselt number
Te
mp
era
ture
T
em
pe
ratu
re
Velo
city
Stre
am
functio
n
Case 3: Magneto-thermal convection
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
Magnetic body force acting on electrically non-conducting materials:
Magnetic suceptibility of paramagnetic substances () is inversly
proportional to its absolute temperature q - Curie law:
Using Curieβs law we can obtain the magnetic force (similar to Bousinessq approximation):
q < q0: attractive force
q > q0: repulsive force
β’ Suppression of the convection:
(a) 0 [T], Dq = 3.44 [K]
(b) 10 [T], Dq = 5.27 [K]
Te
mp
era
ture
Boundary Fitted Coordinates (BFC)
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
β’ If we want to solve more complex geometries?
β’ We need to generate a smooth orthogonal grid system :
β’ Specify boundaries in the physical plane,
β’ Solve:
β’ Or easier:
Boundary Fitted Coordinates (BFC)
β’ Sample grids
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
Boundary Fitted Coordinates (BFC)
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
π π = 104; ππ = 0.71
β’ Now, if we have our continuity equation in physical plane:
ππ
ππ+ππ
ππ= 0
XXX
YYY
2221
1211
aa
aa
Y
X
with transformation between physical and computational plane
β’ Using chain rules of differential calculus, we have:
YXYX
YY
XXYXJ -
,
,
fa
fa
X
f1211
fa
fa
Y
f2221
the Jacobian of the transformation represents the ratio of the volume
of a grid call in physical space to one in computational space.
fa
faa
fa
faa
XXX
f1211121211112
2
attic space simulation
Boundary Fitted Coordinates (BFC)
β’ Two more sample simulations:
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
Applications: scaling analysis
β’ Used to predict behaviour of the fluid flow for different cases.
β’ Approach: by comparing terms of governing equations.
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
β’ Transient flow:
HSMAC on GPU
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
OpenCLβ’
β’ OpenCL (Open Compute Language): an open, royalty-free standard
for parallel programming of heterogeneous systems that include
multi-core processors (CPUs), graphics processing units (GPUs),
and other accelerators such as Cell and digital signal processors
(DSPs).
β’ OpenCL has complex platform and device management model that
reflects its support for multi-platform and multi-vendor portability.
β’ Uniform programming environment to write efficient, portable code
for HPC servers, desktop computer systems and handheld devices.
β’ The standard includes an API for coordinating execution between
devices and a cross-platform parallel programming language.
β’ Initiated by Apple (now specification editor) and developed by the
Khronos Group.
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
OpenCL Platform Model
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
COMPUTE UNIT
COMPUTE UNIT
COMPUTE UNIT
COMPUTE UNIT
COMPUTE DEVICE
PROCESSING ELEMENT
COMPUTE UNIT
COMPUTE UNIT
COMPUTE UNIT
COMPUTE DEVICE
.....
COMPUTE UNIT
COMPUTE UNIT
COMPUTE UNIT
COMPUTE DEVICE
.....
.....
.....
Tested GPUs
β’ GPU Featuresβ¦
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
GT 330M QFX4600 QFX4800 460GTX S2050
Streaming Multiprocessors (SM) 6 12 24 7 14
Streaming Processors (SP) / Cores 48 96 192 224 448
Registers per SM 16K 8K 16K 32KB 32KB
Shared Memory per SM 16KB 16KB 16KB 48KB 48KB
Processor Clock [MHz] 990 1200 1204 810 1147
Work item size 512/512/64 512/512/64 512/512/64 1024/1024/64 1024/1024/64
Work group size 512 512 512 1024 1024
CUDA Compute Capability 1.2 1.0 1.3 2.x 2.x
OpenCL execution model 1-D
β’ Number of work items = 4096:
NDRANGE
DEVICE get_group_id(0) = 2
0 1 2 15
get_global_size(0) = 4096
get_local_size(0) = 256
3 β¦ 4
get_global_id(0) = 1792
WORK GROUP WORK ITEM get_local_id(0) = 255
0 1 β¦ 255
β’ Number of work items = 4100:
0 1 2 15
get_local_size(0) = 256
3 β¦ 4
0 1 255
16
2 3 β¦ 4
PADDING WASTE
get_global_size(0) = 4352
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
(A kernel is executed in each point of a problem domain)
OpenCL execution model 2-D
β’ Number of work items to execute 128 x 128 = 16384:
NDRANGE
DEVICE
0,0 1,0 2,0
0,1
0,2
1,1
β¦
β¦
7,0
7,7 0,7
3,4
0,0 1,0 2,0
0,1
0,2
1,1
β¦
β¦
15,0
0,15
c
WORK GROUP WORK ITEMS
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
2,2
4,1
get_global_size(0)
ge
t_glo
bal_
siz
e(1
) ge
t_lo
ca
l_siz
e(1
)
get_local_size(0)
get_group_id(0),get_group_id(1)
get_local_id(0),get_local_id(1) get_global_id(0),get_global_id(1)
.
(A kernel is executed in each point of a problem domain)
HOST OUTER LOOP
Flow Chart of the OpenCL Fluid Solver
START
Read Configuration
Allocate Memory for Flow
Field Tables
Apply Boundary
Conditions (BC)
Calculate Tentative
Velocities (U, V)
Calculate Pressure
Correction
Correct Pressure and
Velocity Fields, Apply BC
Converged? NO
Calculate Energy
Equation (if required)
YES Final time
achieved?
Advance All Flow Tables
and Increase Time
NO
Save Results: Flow
Fields, etc...
FINISH
YES
HOST INNER LOOP
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
Get OpenCL Platform,
Initialize Device, Create
Context, Create
Command Queue
Create OCL Memory
Buffers
Create OCL Program,
Create Kernels and Set
Kernel Arguments
Setup Boundary Flags
Copy Host Memory
Buffers to GPU Buffers
krnl_bc_u krnl_bc_v krnl_bc_p
krnl_momentum_u krnl_momentum_v
krnl_pressure_velocity_correction
krnl_bc_u krnl_bc_v krnl_bc_p
krnl_energy
krnl_advance_u krnl_advance_v
Copy GPU Buffers to
Host Memory Buffers
Release Command
Queue, Context, Kernels,
Programs, Memory
Visualisation
Before we startβ¦
β’ Get OpenCL platform and initialise computing devices:
β’ Create context and create command queue.
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
Before we startβ¦
β’ Create OpenCL program, kernel and set kernel arguments:
β’ Create OpenCL memory buffers:
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
OpenCL Class Diagram
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
β’ UML class diagram of OpenCL: relationships between classes,
β’ Solid diamonds (aggregations), inheritance (open arrowhead),
cardinality: * many, 1 one and only one, 0..1 optionally one, 1..* one
or more.
Reconstructed from Figure 2.1 of the Khronos OpenCL Specification.
ComputeEngine Class
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
β’ Example:
β’ Kernels call:
Boundary Conditions on GPU
β’ Setup boundary conditions
β’ Create βghostβ cells for velocity, pressure, temperature and
concentration fields.
β’ Below: ghost cells to assist scalar computations.
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
π = π π = π π = π π = π π = π π = π π = π
π = π
π = π
π = π
π = π
π = π
π = π
πππππ = π
πππππ = π
πππππ = π
πππππ = π
π πππππ
π΅πΏ = π; π΅π = π
π΅πΏπ = π; π΅ππ = π
Boundary conditions on GPU
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
π π π π π π π π
π
π
π
π
π
π π π π π π π π
π
π
π
π
βπ
β’ Velocity boundary conditions
β’ For simplicity, the memory buffer for vectors is of the same size as
for scalar variables.
β’ For horizontal velocity component there is one column of waste.
β’ For vertical velocity components there is one row of waste.
π΅πΏ = π; π΅π = π
π΅πΏπ = π; π΅ππ = π
Memory Buffers Structure
π π π π π π π π π π π = π π = π π = π π = π π = π
π = π
π = π
π = π
π = π
π = π
π΅πΏ = π; π΅π = π; π΅πΏπ = π; π΅ππ = π
P
PG
UN, UO
UG
VN, VO
VG
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
Apply boundary conditions
π π π π π π π π π π π = π π = π π = π π = π π = π
π = π
π = π
π = π
π = π
π = π
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
Calculate tentative velocity component U
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
ππ
ππ+ π
ππ
ππ+ π
ππ
ππ= βπ΄
ππ
ππ+ π΅
π2π
ππ2 +π2π
ππ2
ππ,ππ+1=
ππ,ππ + βπ βπΉππ β πΉππ β ππ πΈπππ + π·πΌπΉπΉπ
β’ From X- momentum equation
π πΈ π
π
π
πΈπ
πΈπ
ππ
ππ
π π€ ππ
ππ
ππ
π€π
π€π
π
π
ππ
π€π
π€π
ππ
ππ
π€ππ πππ
πππ
πππ
βπ
βπ
π πΈ π
π
π
πΈπ
πΈπ
ππ
ππ
π π€ ππ
ππ
ππ
π€π
π€π
π
π
ππ
π€π
π€π
ππ
ππ
π€ππ πππ
πππ
πππ
βπ
βπ
ππ,ππ+1 =
ππ,ππ + βπ βπΉππ β πΉππ β ππ πΈπππ + π·πΌπΉπΉπ
ππ
ππ+ π
ππ
ππ+ π
ππ
ππ= βπ΄
ππ
ππ+ π΅
π2π
ππ2 +π2π
ππ2
Calculate tentative velocity component V
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
β’ From X- momentum equation
Pressure and velocity corrections
π·ππ,ππ+1 =
πππ,ππ+1 β ππ
πβ1,ππ+1
βπ+
πππ,ππ+1 β ππ
π,πβ1π+1
βπ= π·
πππ·π,π
πππ,π
π+1
=
βπ1
βπ
1
βπ+
1
βπ+
1
βπ
1
βπ+
1
βπ= π·πΈπΏ
ππβ² =
βπ
βπππ,πβ²
ππβ² =
βπ
βπππ,πβ²
ππ€β² = β
βπ
βπππ,πβ²
ππ β² = β
βπ
βπππ,πβ²
ππβ² = β
π·
π·πΈπΏ
π πΈ π
π
π
πΈπ
πΈπ
ππ
ππ
π π€ ππ
ππ
ππ
π€π
π€π
π
π
ππ
π€π
π€π
ππ
ππ
π€ππ πππ
πππ
πππ
βπ
βπ
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
Advance flow tables
β’ Advance velocity:
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
β’ And for instance, this can be optimised explicitly:
0
500
1000
1500
2000
2500
0 64 128 192 256 320
Tim
e [
s]
Number of cells (K)
S2050
GT 330M
Measures for Execution Speedup
β’ Driven Cavity
5
10
15
20
25
30
35
32 64 96 128
Sp
eed
up
X (
GP
U/C
PU
)
Grid Size
GPU/CPU
GeForce GT 330M
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
β’ Natural Convection
CPU: Intel Core i7 2.66Ghz
~7-9 x
Grid size GT 130M S2050
32 x 32 40.64 s 8.17 s
64 x 64 83.54 s 11.37 s
128 x 128 218.27 s 24.14 s
256 x 256 678.58 s 93.61 s
512 x 512 2477.87 s 353.72 s
Execution times:
CSIRO GPU Cluster
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
CSIRO GPU Cluster Launch (25 November 2009)
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
CSIRO GPU Cluster Configuration
GPU Cluster Configuration
The new CSIRO GPU cluster will deliver up
to 264 plus Teraflops of single precision
computing performance and will consist of:
β’128 Dual Xeon E5462 Compute Nodes
(i.e. a total of 1024 2.8GHz compute cores)
with 16 or 32GB of RAM, 500GB SATA
storage and DDR InfiniBand interconnect
β’64 Tesla S250βs (256 GPUs with a total of
114,688 CUDA processor cores)
β’144 port DDR InfiniBand Switch
β’80 Terabyte Hitachi NAS file system.
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
CSIRO GPU Cluster
Linux and Windows
Production
Environment
Linux and Windows
Test & Training
Environment
Linux and Windows
Management
Nodes
Currently
1024 CPU cores
64 Tesla s1070βs
(256 GPUs with
61,440 CUDA cores)
CSIRO GPU Cluster
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
NVIDIA CUDA Research Center
β’ CSIRO, in recognition of its leadership in massively parallel computing, is an inaugural member of the NVIDIA CUDA Research Center program
β’ Institutions identified as CUDA Research Centers are doing world-changing research by leveraging CUDA and NVIDIA GPUs. CUDA Research Centers and PIs:
β’ CSIRO (PI: John Taylor)
β’ ICHEC (PI: Gilles Civario)
β’ Johns Hopkins University (PI: Alex Szalay)
β’ Mass General-Harvard Medical School (PI: Homer Pien)
β’ Nanyang University (PI: Bertil Schmidt)
β’ SINTEF (PI: Trond Runar Hagan)
β’ Technical University of Ostrava (PI: Eduard Sojka)
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
CSIRO Computational and Simulation Sciences
Science: Emerging Science Challenges
Square Kilometre Array β Potential to
process 10 million Gb of data per hour
(source: Au SKA website)
Visualization capabilities β To
illuminate high-performance
computing data from large-scale climate
simulations. (source: US Department of
Energy)
Computational Fluid Dynamics β CMIS Fast Flow computer environment
(source: CMIS website)
Predictive Mineral
Discovery β
(source:CSIRO E&M)
WRON β Using
advanced sensor
networks to
monitor scarce
water resources
via the internet (source: WRON
website)
Summary
β’ CFD does not need to be scary !!!
β’ We used the following approach:
β’ Equations Discretisation Implementation Verification
Results Transfer CPU code to GPU code Verification
Results.
β’ Methods presented:
β’ Finite Difference Method (for discretisation),
β’ HSMAC Method (for mutual iteration of pressure/velocity fields),
β’ BFC (for solving Navier-Stokes equations on complex geometry),
β’ Upwind and UTOPIA (for getting more stable and accurate results),
β’ OpenCL (for speeding-up computing process).
β’ The methodology presented, can be extended into 3D and
applied to compute different cases in real-time, e.g., smoke,
free surface flows, ventilation, turbulent flows, etc.
β’ Source-code to be released online.
CSIRO. Numerical Simulations in Fluid Dynamics using GPU: a Practical Introduction.
Contact Us
Phone: 1300 363 400 or +61 3 9545 2176
Email: [email protected] Web: www.csiro.au
Thank youβ¦ β¦
Earth Science and Resource Engineering
Dr Tomasz P Bednarz
3D Visualisation Engineer
Mining Automation
Phone: +61 429 153 274
Email: [email protected]
Web: www.csiro.au/org/CESRE.html
www.tomaszbednarz.com
Acknowledgments: Dr J.Taylor, C.Caris, Prof. H.Ozoe, Prof. H.Hirano, Prof. T.Tagawa, Prof. J.C.Patterson,
Prof. Chengwang Lei, Prof. J.S.Szmyd, Dr P.Cleary, Dr M.Rudman, Dr J.Ralston,
Dr L.Domanski, Dr J.Malos, J.Craig, Dr S.Zawlodzka-Bednarz