CSC – Finnish research, education, culture and public administration ICT knowledge center

Elmer/Ice advanced workshop 2017

Grenoble, France

Including pre- and postprocessing in parallel

Thomas Zwinger

Lecturers (in alphabetical order):Jaakko LeinonenAtte SillanpääThomas Zwinger

Parallel strategies in Elmer/Ice

SOLUTION

VISUALIZATION

ASSEMBLY

PARTITIONING

MESHING GMSH

Parallel concept of Elmer

MPI

Parallel concept of Elmer

Trilinos

HUTIter

MATC

BLAS LAPack UMFPack

cPardiso

Hypre

SuperLU

CholMOD

MUMPS

ScaLAPack

BLACS

Parallel Concept of Elmer

• Domain decomposition

• Additional pre-processing step

(splitting)

• Every domain is running its ”own”

ElmerSolver

• Parallel process communication:

Message Passing Interface (MPI)

Parallel Concept of Elmer

• Ice sheet = flat geometry

• Meshing tools (e.g., Gmsh) have

difficulties with aspect ratios

deviating from unity

• Creating footprint mesh in 2D

• Extruding it to 3D

Pre-processing in Elmer/Ice (focus parallel runs)

Internal Extrusion

• Implemented as an internal strategy in Elmer (2013)

oJuha, Peter & Rupert

• First partition a 2D mesh, then extrude into 3D

• Implemented also for partitioned meshes

oExtruded lines belong to the same partition by construction!

• Deterministic, i.e. element and node numbering determined

by the 2D mesh

oComplexity: O(N)

• Neecessary, if using parallel runs with

StructuredMeshMapper (see later)

Internal extrusion

Extruded Mesh Levels = 11

By default z in [0,1]

Else give:

Extruded Max Coordinate = Real 10000.0

Extruded Min Coordinate = Real -100.0

Internal extrusion

Extruded Mesh Levels = 11

Extruded Mesh Density = Variable Coordinate 1

Real MATC "0.2+sin(pi*tx)"

Any functional dependenceis ok as long as it is positive!

The optimal division is founditeratively using Gauss-Seidel type of iteration and large variations make the iterations converge slowly.

Internal Extrusion

• StructuredMesh Mapper

to impose geometry to a

topological prism (3D)

• Define functions at

bottom:

Bottom Surface = Variable

"Coordinate 1“

Real MATC "0.1*cos(5*tx)"

• And surface:

Bottom Surface = Variable

"Coordinate 1“

Solver 2

Equation = "MapCoordinate"

Procedure = "StructuredMeshMapper"

"StructuredMeshMapper"

Active Coordinate = Integer 2

End

ElmerSolver parallel

• Same executable: ElmerSolver

• Depending on platform/MPI: mpirun –np N

$ mpirun –np 6 ElmerSolver

• Needs information for different processes, which SIF to load:

ELMERSOLVER_STARTINFO

• User defined functions/routines usually do not need special

rewriting for MPI

Improvements on block preconditioners

Massive parallel Stokes problem

• Solving Stokes equations:

• No connection between pressure and density – saddle point

problem:

Massive parallel Stokes problem

• Needs stabilization to avoid null-space:

• Yet, this matrix usually has a bad condition number:

oDirect parallel solver (expensive): MUMPS, cPardiso (N2 –algorithm)

oGood pre-conditioner + iterative solvers (N log(N) –algorithm)

Block pre-conditioner

• Using following form:

oM is a by viscosity weighted unit-matrix

• Yet, we need the inverse of P, which requires exact solutions

of linear systems of the blocks A and M

• Those systems are

oway smaller sub-problems

obetter conditioned

osolvable using effective linear Algebra (=Krylov subspace methods) to solve them

ParStokes

• The previously described method is implemented in the

ParStokes-solver (Mika Malinen, CSC)

• ParStokes is nowadays part of the Elmer-distribution

oNo extra compilation is needed

o Interface in SIF has been tremendously simplified

oStill needs extra dummy-routines for providing the solution space for the preconditioning blocks (pressure,velocity); but their syntax is simplified. Basically only procedure call, if happy with default settings

oFull documentation under ElmerModelsManual (chapter 24)

oEplained on elmerice-Wiki

oTest-case: elmerfem/fem/tests/PasStokes_ISMIP_HOM_A010

(New) direct parallel solver: cPardiso

• Solver very similar to MUMPS

• Included in MKL (comes with Intel compiler suite) – not open

source (to my knowledge)

• cPardiso can run mixed MPI/OpenMP codes, which Elmer is

oAlso OK to just use MPI side, like we can test in a session on Friday

• Simply include these lines into Solver section:

Linear System Solver = Direct

Linear System Direct Method = "cPardiso"

Future developments

• Modern CPU’s have 12+ cores

• Modern CPU’s have inproved vector

units

• Multi-threading (OpenMP) might

replace MPI for workstations

• Optimizing for that, using SIMD-

instructions within the code

Multi-threaded and SIMD version of advection/diffusion solver

Optimization of Elmer(/Ice) using OpenMP Threading and SIMD

• Article in the proceedings which appear in the Lecture Notes

in Computer Science (vol 10468), Springer

• The following slides are an excerpt of a presentation shown at

this year’s 13th International Workshop on OpenMP (IWOMP)

in Stony Brook University, NY, USA

• Presentation permission granted by Intel Corp.

Byckling, M., J. Kataja, M. Klemm and T. Zwinger, 2017. OpenMP SIMD Vectorization and Threading of the Elmer Finite Element Software, Proceedings 13th International Workshop on OpenMP, Springer Lecture Notes in Computer Sicnece,123-137, doi:10.1007/978-3-319-65578-9_9

Suggested exercise

Bueler pofile parallel runs

• Based on the analytic equilibrium profile (SIA) from Bueler

• Training accounts + remote desktop on taito.csc.fi

1. Creating the footprint mesh

o Partitioning of mesh

2. Setting up diagnostic parallel run using

o Internal extrusion

o Free surface imposed by external routine buelerprofile.f90

3. Running the case

o With classic Navier-Stokes solver (MUMPS + cPardiso)

o With ParStokes Solver

Bueler pofile parallel runs

• Two different kind of footprint meshes

1. footprint_bueler.geo for small runs on laptop (ParStokesmight not scale)

2. footprint_bueler_f.geo for node-size runs on supercomputer (< 24 partitions)

3. footprint_bueler_ff.geo for (massive) parallel runs

• Create mesh from gmsh:

$ gmsh -2 footprint_bueler.geo

• Convert mesh into Elmer mesh

$ ElmerGrid 14 2 footprint_bueler.msh –autoclean

Add –metis 24 4 at the end to create a 24 partition parallel mesh

• Using GetEmergenceVelocity

after the Stokes solution to

display the corresponding

equilibrium accumulation-

ablation function of the

diagnostic solution

• Use footprint_f.geo

Bueler pofile parallel runs

MUMPS cPardiso ParStokes

> 30 mins > 30 mins 132.66 s

Bueler pofile parallel runs

End of session