overview of trilinos and pt-scotch - uni-graz.at · average edge expansion/dilation: exp def= p f c...

Overview of Trilinos and PT-Scotch

Markus Hopfer, Karl-Franzens University, Graz

29.03.2012

PT-ScotchThe Trilinos Library

Outline

1 PT-ScotchMapping and Graph Partitioning

The Dual Recursive Bipartitioning AlgorithmParallel Graph Bipartitioning Methods

Parallel Sparse Matrix Ordering

2 The Trilinos LibraryOverview of the Trilinos PackagesExamples on using Trilinos

Markus Hopfer, Karl-Franzens University, Graz Overview of Trilinos and PT-Scotch


Mapping and Graph PartitioningParallel Sparse Matrix Ordering

PT-Scotch




The Scotch Project

The Scotch Project

Laboratoire Bordelais de Recherche en Informatique (LaBRI), UniversiteBordeaux I / ScALApplix project of INRIA Bordeaux Sud-Ouest

applications of graph theory to scientific computingdevelopement of the Dual Recursive Bipartitioning Mapping Algorithmgraph bipartitioning heuristicscomputation of high-quality vertex separators for the ordering of sparsematriceshypergraph partitioning algorithms

PT-Scotch 5.1 ⇔ parallel graph ordering/partitioning features

parallel static mapping will be available in the next release




Parallel Static Mapping and Sparse Matrix Ordering

Sparse Matrix Ordering

scientific/engineering problems ⇔ sparse linear systems direct oriterative methods to solve the systems

direct solversminimize the fill-in by re-ordering of the linear systemparallel graph ordering tools ⇔ PT-Scotch

Static Mapping

determine best start time for processes on host machine(s) ⇔ scheduling

determine best distribution of processes on host machine(s) ⇔ mappingbalance computational weight of processes between processorsreducing the cost of communicationkeep intensively intercommunicating processes on nearby processors

computational structure of parallel program ⇔ graphvertices ⇔ processes that handle distributed dataedges ⇔ data dependencies/communication

static mapping ⇔ computed prior to execution of the program




Parallel Static Mapping by Dual Recursive Bipartitioning

Static Mapping

we want to map the parallel program on a target architecture

⇒ parallel program is modelled by a graph S (source graph)vertices vS ⇔ processes vertex number ωS (vS ) ∈ Zedges eS ⇔ communication between processes edge number ωS (eS ) ∈ ZωS (vS ) . . . estimation of computation weight for processωS (eS ) . . . estimate amount of communication between processes

⇒ target architecture is modelled by a graph T (target graph)vertices υT ⇔ processors vertex number ωT (υT ) ∈ Zedges eT ⇔ communication abilities edge number ωT (eT ) ∈ ZωT (υT ) . . . estimation of computational power of corresponding processorωT (eT ) . . . estimate costs for communication between processors

if target is homogeneous ⇒ all ωT (υT ) are equal




Mapping of the Source Graph S to the Target Graph T

vertex mapping: τS,T : V (S)→ V (T )

edge mapping: ρS,T : E(S)→ P(E(T ))P(E(T )) . . . set of all simple loopless paths which can be build from E(T )

examples:τS,T (υS ) = υT . . . if process υS of S is mapped onto processor υT of T

ρS,T (eS ) = {e1T , e

2T , . . . , e

nT } . . . if communication channel eS of S is routed

through communication links e1T , e

2T , . . . , e

nT of T

|ρT ,S(eS)| . . . dilation of edge eS , i.e. the number of necessary edges toroute eS on graph T




Cost Function and Performance Criteria

communication cost function:

fC (τS,T , ρS,T )def=

∑eS∈E(S)

ωS(eS)|ρS,T (eS)|

goal: minimize cost function, while keeping good load balance

average load per computational power unit:

µmapdev=

∑υS∈V (S)

ωS(υS)∑υT∈V (T )

ωT (υT )

load imbalance ratio:

δmapdev=

∑υT∈V (T )

∣∣∣∣∣∣∣ 1ωT (υT )

∑υS∈V (S)

τS,T (υS )=υT

ωS(υS)

− µmap

∣∣∣∣∣∣∣∑υS∈V (S)

ωS(υS)

maximum load imbalance ratio δmaxmap provided by user as input




Cost Function and Performance Criteria

average edge expansion/dilation:

µexpdef=

fC∑eS∈E(S)

ωS(eS)µdil

dev=

∑eS∈E(S)

|ρS,T (eS)|

|E(S)|

δexpdev=

µexp

µdil< 1 if mapper succeeds in putting heavily

intercommunicating processes closer together




The Dual Recursive Bipartitioning Algorithm

recursively allocate subsets of processes to subsets of processors1 take all processors of target (domain) & take all processes to map2 bipartition a yet unprocessed domain into two disjoint subdomains3 call a graph bipartitioning algorithm to split the subset of processes

associated with the domain across the two subdomains

sketch of the algorithm:mapping (D, P);Set_of_Processors D;Set_of_Processes P;{

Set_of_Processors D0, D1;Set_of_Processes P0, P1;if (P == 0) return; /* If nothing to do. */if (D == 1) { /* If one processor in D */

result (D, P); /* P is mapped onto it. */return;

}

(D0, D1) = processor_bipartition (D);(P0, P1) = process_bipartition (P, D0, D1);mapping (D0, P0); /* Perform recursion. */mapping (D1, P1);

}




Partial Cost Function

cost function for bipartitioning of a subgraph S ′ of S

f ′C (τS,T , ρS,T )def=

∑υ∈V (S′)

{υ,υ′}∈E(S)

ωS({υ, υ′})|ρS,T ({υ, υ′})|

{υ, υ′} . . . cocycles, i.e. connections between S and S ′

taking into account results of previous bipartitionings to avoid localchoices that might be globally disadvantageous




Parallel Graph Bipartitioning Methods

Dual Parallel Recursive Mapping Algorithm uses graph bipartitioningmethods as black boxes

allows to run any type of graph bipartitioning method which isconformal with cost function

Kernighan-Lin-type algorithmssimulated annealingquadratic assignmentgenetic algorithmsetc.

define mapping strategiesdifferent methods for different domainsapply different methods in sequence

currently implemented graph bipartitioning methodsBand method: meta-algorithm which supports other partitioning algorithmsto perform betterDiffusion: global optimization methodMulti-Level: in combination with banded diffusion method to refine theprojected partitions a every level




Multi-level Framework for Graph Partitioning

Figure: Multi-level framework for computing a bipartition of a graph.




Figure: Partition of graph bump into 8 parts:(upper) Scotch 4.0 with un-banded Fiduccia-Mattheyses refinement(lower) Scotch 5.0 with multi-level banded diffusion scheme




Multi-Level Banded Diffusion Scheme

The Jug(s) of the Danaides

nodes ⇔ leaking barrels of ∞capacity

graph edges ⇔ pipes ofsection equal to their weight

two sorts of liquids (annihilateeach other)

w h i l e ( number o f p a s s e s to do ) {r e s e t c o n t e n t s o f new a r r a y to 0 ;o l d [ s0 ] ← o l d [ s0 ]−|V | / 2 ; /∗ R e f i l l s o u r c e b a r r e l s ∗/o l d [ s1 ] ← o l d [ s1 ]+|V | / 2 ;f o r ( a l l v e r t i c e s v i n graph ) {

c ← o l d [ v ] ; /∗ Get c o n t e n t s o f b a r r e l ∗/i f ( | c|>w e i g h t [ v ] ) { /∗ I f not a l l c o n t e n t s have l e a k e d ∗/

c ← c−w e i g h t [ v ]∗ s i g n ( c ) ; /∗ Compute what w i l l remain ∗/σ ←

∑e=(v,v′) w e i g h t [ e ] ; /∗ Sum w e i g h t s o f a l l a d j a c e n t edges ∗/

f o r ( a l l edges e=(v , v ’ ) ) { /∗ For a l l edges a d j a c e n t to v ∗/f ← c∗w e i g h t [ e ] /σ ; /∗ F r a c t i o n to be s p r e a d to v ’ ∗/new [ v ’ ]←new [ v ’ ]+ f ; /∗ Accumulate s p r e a d e d c o n t r i b u t i o n s ∗/

} } }swap o l d and new a r r a y s ;

}




Parallel Sparse Matrix Ordering Capabilities of PT-Scotch

Deferred to Second Talk



Overview of the Trilinos PackagesExamples on using Trilinos

The Trilinos Software Package




Trilinos Packages

Epetrabasic classes for distributed matrices and vectors, linear operators and linearproblems, time/FLOP measuring, etc.each Trilinos package accepts Epetra objects

Triutilscollection of utilities for software developementcommand line parser, matrix generator

AztecOOlinear solver package based on preconditioned Krylov methodsobject-oriented version of Aztec

Belosprovides next-generation iterative linear solvers and a powerful linear solverframework

IFPACKperforms various incomplete factorizationsused with AztecOO

Isorropia and Zoltanpartitioning and load balancing

Teuchosclasses for advanced code developement




Trilinos Packages

MLalgebraic multilevel and domain decomposition preconditioner packagescalable preconditioning capabilities for a variety of problems

Amesosprovides common interface to certain sparse linear solvers (generallyavailable outside the Trilinos framework)

Anazasiprovides a common interface to parallel eigenvalue and eigenvector solversfor both symmetric and non-symmetric linear problems

NOXcollection of non-linear solversdesigned to be easily integrated into an application or combined with linearsolvers

Tpetranext-generation, templated version of Petrataking advantage of the newer advanced features of C++

Didaskocontains all the examples from the tutorial




Example: Epetra Objects

#i n c l u d e ” E p e t r a C o n f i g D e f s . h”#i f d e f HAVE MPI#i n c l u d e ”mpi . h”#i n c l u d e ”Epetra MpiComm . h”#e l s e#i n c l u d e ” Epet ra Ser ia lComm . h”#e n d i f

// . . o t h e r i n c l u d e f i l e and o t h e r s . . .

i n t main ( i n t argv , c h a r ∗a r g v [ ] ) {

// . . some d e c l a r a t i o n s h e r e . . .

#i f d e f HAVE MPIM P I I n i t (& argc , &a r g v ) ;Epetra MpiComm Comm(MPI COMM WORLD ) ;

#e l s eEpet ra Ser ia lComm Comm;

#e n d i f

// . . o t h e r code f o l l o w s . . .

#i f d e f HAVE MPIM P I F i n a l i z e ( ) ;

#e n d i f

r e t u r n 0 ;}




Example: Solving a Linear System Ax = b with AztecOO

create an Epetra object for the linear system

E p e t r a L i n e a r P r o b l e m Problem(&A, &x , &b ) ;

create an AztecOO object

AztecOO S o l v e r ( Problem ) ;

define option & parameter vector to override default by desired values

i n t o p t i o n s [ AZ OPTIONS SIZE ] ;d o u b l e params [ AZ PARAMS SIZE ] ;A Z d e f a u l t s ( o p t i o n s , params ) ;

S o l v e r . S e t A l l A z t e c O p t i o n s ( o p t i o n s ) ;S o l v e r . S e t A l l A z t e c P a r a m s ( params ) ;

alternatively: set options explicitly → Jacobi preconditioner

S o l v e r . S e t A z t e c O p t i o n ( AZ precond , AZ Jacob i ) ;

solve the linear system with a maximum of 1550 iterations and a residualerror norm threshold of 10−9

S o l v e r . I t e r a t e (1550 , 1E−9);




Thank you for your attention !


overview of trilinos and pt-scotch - uni-graz.at · average edge expansion/dilation: exp def= p f c...

Documents