parallelization of an extended finite element two-phase flow solver · 2011. 7. 5. · chair for...

CHAIR FOR

COMPUTATIONAL

ANALYSIS OF

TECHNICAL

SYSTEMS

Parallelization of an Extended Finite Element

Two-Phase Flow Solver

H. Sauerland and T.-P. Fries

July 20, 20109. WCCM

Sydney, Australia

History of our XFEM flow solver

Navier-Stokes solver for quadrilateral/hexahedral meshes withsemi-implicit time integration.

←−

increasingNo.

DOF

serial direct solverC, UMFPACK1

⇓

parallel direct solverMPI, MUMPS2

⇓

parallel iterative solverMPI, PETSc3

1http://www.cise.ufl.edu/research/sparse/umfpack/

2http://graal.ens-lyon.fr/MUMPS/

3http://www.mcs.anl.gov/petsc/

H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 2

Outline

Challenges in two-phase flows

Introduction to the XFEM

Iterative solvers with the XFEMBlocking degrees of freedomPreconditioning: Diagonal scalingConvergence study: Blocking DOFs, diagonal scalingStudy on iterative solvers and preconditioners

Parallelization in the XFEM contextDynamic load balancing

Scale test

Conclusion






Scale test

Conclusion



Moving interface:

Topological changes of the phases can occur→ Level-set method.

Discontinuous field variables:

Density differences

→ kink in the velocity/pressure field,Viscosity differencesSurface tension → jump in the pressure field→ XFEM.

up

ρ2, µ2

ρ1, µ1

u

p

ρ2, µ2

ρ1, µ1

γ






Scale test

Conclusion



Interface is implicitly described using the level-set method.

Discontinuities inside elements accounted for with the XFEM.

Here: Pressure approximation space is locally enriched:

ph(x) =∑

i∈I

Ni (x)pi

︸︷︷︸

strd. FE approx.

+∑

i∈I⋆

Ni (x) · [ψ(x, t)− ψ(xi , t)] ai

︸︷︷︸

enrichment

.

Sign-enrichment:

ψ(x, t) = sign (φ(x, t)) =

−1 : φ < 0,

0 : φ = 0,

1 : φ > 0.

I ⋆

φ(x, t) = 0






Scale test

Conclusion


Iterative solvers with the XFEM

Occurrences of enrichment basis functions with very smallsupport seriously increase the condition number of the systemmatrix → slow/no convergence with iterative solvers.

Spectral condition number of the system matrix K:

κ(K) =

∣∣∣∣

λmax(K)

λmin(K)

∣∣∣∣, λ: Eigenvalues

Case κ(K)

Standard FEM ∼ 15XFEM (p-enriched) ∼ 97444

Possibilities: Blocking degrees of freedom,’special’ (additional) preconditioning.






Scale test

Conclusion


Blocking degrees of freedom

Reusken (CVS, 2008) introduced a modified XFEM space,excluding the basis functions with very small support.

Here: Identify basis functions with very small support simplyusing the area ratio of the two phases occupying the element:

Aphase 1

Aphase 2< Cblock ≪ 1, where Aphase 1 < Aphase 2.

Spectral condition number of the system matrix K:

Case κ(K)

Standard FEM ∼ 15XFEM (p-enriched) ∼ 97444Cblock = 0.01 ∼ 22849 Cblock = 0.01






Scale test

Conclusion


Preconditioning: Diagonal scaling

Bechet et al. (IJNME, 2005) developed a preconditioner forthe XFEM based on a local Cholesky decomposition.

Here: Simple diagonal scaling of the linear system:

D−1/2KD−1/2(D1/2x

)= D−1/2b, with D = diag(K)

Applied in addition to the general purpose preconditioners.

Spectral condition number of the system matrix:

Case κ(K)

Standard FEM ∼ 15XFEM (p-enriched) ∼ 97444Cblock = 0.01 ∼ 22849Diagonal scaling ∼ 27






Scale test

Conclusion


Convergence study: Blocking DOFs, diagonal scaling

Steady state test case with surface tension:

ρ1/ρ2 = 1.0,

µ1/µ2 = 1.0,

γ = 1 kg/s2.

Exact solution given by:

u(x) = 0, p2 − p1 = γ ·1

r.

Ω1

Ω2

2r4r

4r

Solver: GMRES, LU preconditioner.


Convergence study: Blocking DOFs, diagonal scaling

No convergence to a steady state without blocking andwithout diagonal scaling.


10−2

10−1

10−4

10−3

10−2

10−1

100

Element size h

Pre

ssur

e L2

−no

rm

dia. scal.c=0.01c=0.1c=0.5c=1

1/15 1/20 1/40 1/80 1/1600

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Element size h

Rat

io o

f num

ber

of b

lock

ed a

nd e

nric

hed

node

s

c=0.01c=0.1c=0.5c=1

Conclusion: Diagonal scaling is very powerful in this case.






Scale test

Conclusion


Study on iterative solvers and preconditioners

PETSc allows to change solver/preconditioner on execution.

Considered Krylov-Subspace methods:

GMRES(m), l-GMRES, f-GMRES,BiCGstab, i-BiCGstab, BiCGstab-l.

GMRES: satisfies an optimality condition; Comp. costs risewith the iteration count.

BiCGstab: Comp. costs independent of the iteration count;does not satisfy an optimality condition.

Considered preconditioners:

ASM, KSP, LU, ICC, ILU, SOR, SSOR.




ρ1/ρ2 = 1.0,

µ1/µ2 = 1.0,

γ = 1 kg/s2,

80× 80 elements.

Exact solution given by:

u(x) = 0, p2 − p1 = γ ·1

r.

Ω1

Ω2

2r4r

4r

Convergence criteria:Residual of the solution vector: ε ≤ 10−10.



GMRES

PC ε× 10−10 # t [s]lu 0.00060 528 7.92

f-GMRES

PC ε× 10−10 # t [s]lu 0.00057 522 7.84

l-GMRES

PC ε× 10−10 # t [s]asm 0.00190 1473 7.13ilu 0.00058 1524 6.93lu 0.00058 453 7.78

i-BiCGstab

PC ε× 10−10 # t [s]asm 3083. 3410 22.03ilu 8455. 3410 19.72

BiCGstab-l

PC ε× 10−10 # t [s]asm 3992. 2936 20.32lu 12. 1000 24.29ssor 7604. 4894 19.61

BiCGstab

PC ε× 10−10 # t [s]asm 1918. 3059 21.25ilu 4542. 3085 19.15ssor 875. 7678 27.29

GMRES methods converge very fast.

BiCGstab methods do not reach the residual bound.



Tank sloshing test case:

ρ1/ρ2 = 1000.0,

µ1/µ2 = 0.01,

∆t = 0.05 s,

80× 120 elements,400 time-steps.

Ω2

Ω1

1.0

1.5

g

Convergence criteria: ε ≤ 10−4.



Tank sloshing test case:

Solver PC t [min]

GMRES lu 36f-GMRES lu 34

l-GMRESasm 37ilu 33lu 36

i-BiCGstab asm 25BiCGstab-l lu 19BiCGstab ssor 30

0 2 4 6 8 10 12 14 16 18 200.9

0.95

1

1.05

1.1

1.15

Time

Hei

ght

BiCGstab methods perform better here, as the convergencecriteria is not that strong as in the steady case.



Rising bubble test case:

ρ1/ρ2 = 1000.0,

µ1/µ2 = 0.01,

γ = 0.001 kg/s2 ,

r = 0.1m,

∆t = 0.025 s,

80× 160 Elements,1000 time-steps.

Ω1

Ω2

4r

8r

g

2r

Convergence criteria: ε ≤ 10−4.



Rising bubble test case:

Solver PC t [min] Cblock

l-GMRES asm 159 0.0BiCGstab-l lu - 0.0BiCGstab-l lu 118 0.001

GMRESBiCGs

Diagonal scaling and blocking with small Cblock makes sensehere.






Scale test

Conclusion


Parallelization in the XFEM context

Basic approach is the same as for standard FEM:

Partitioning(ParMETiS)

Assembly onpartition-level

Assembly in parallelPETSc matrix

Parallel solver



XFEM: Enriched elements are computationally more expensive(more integration points, more shape function evaluations andthe additional DOFs)

Tank sloshing for 2 PEs

PE0: 75 Elements (12 enriched)PE1: 75 Elements (0 enriched)→ Imbalance



Solution: Assign weights to the elements before the partitionphase (larger ones for enriched).

ParMETiS uses the weights as an additional constraint.

Tank sloshing for 2 PEs (enriched elements weighted with 10):

PE0: 48 Elements (9 enriched)PE1: 102 Elements (3 enriched)→ Balanced?

Reasonable values for the weights have to be determined.

E.g. for the rising bubble test case a static load balancing willnot be sufficient (also cf. adaptive meshes).


Dynamic load balancing

Dynamic load balancing:

Adapt the partitioning duringexecution to maintain a good loadbalance.

Specify criteria when to perform a load balancing step (ideallytaking into account migration/rebuilding costs).






Scale test

Conclusion


Scale test

Olmsted dam on the Ohio river (Illinois, USA):

http://www.lrl.usace.army.mil/olmsted_specproj/

Computational domain:

195m

1.16m3/s →


http://www.lrl.usace.army.mil/olmsted_specproj/

Scale test

140000 DOFs(performed on an Intel Xeon cluster with 1840 cores in total).

Interface motion and pressure:


Scale test

Parallel performance:

Speedup:

S(n) =T (1)

T (n).

Efficiency:

E (n) =T (1)

n · T (n),

n: Number of processors,

n ts/h S(n) E(n)

1 1.5 - -2 2.8 1.87 0.9354 5.1 3.38 0.8458 8.6 5.69 0.71116 15.3 10.10 0.63132 16.6 10.90 0.340

T (i): Execution time with i processors.






Scale test

Conclusion


Conclusion

Usage of standard iterative solvers and preconditioners withthe XFEM is difficult.

Diagonal scaling of the system matrix significantly improvesthe condition number.

Additional blocking of DOFs makes sense in extreme cases.

Dynamic load balancing is required for the XFEM in order toachieve optimal parallel performance.

Current implementation already scales quite good.

Outlook: Complete dynamic load balancing.

Parallelize our adaptive mesh refinement.


Thank you for your attention.

www.xfem.rwth-aachen.de


parallelization of an extended finite element two-phase flow solver · 2011. 7. 5. · chair for...

Documents