1. implementation: target architectures · implementation:... risc technology ... • different...

Implementation: . . .

RISC Technology

Pipelining

Superscalar Processors

Cache Memory

Memory Hierarchy

Parallel Computers – . . .

Flynn’s Classification . . .

Memory Access . . .

Parallelization

The Programming . . .

MPI Messages

Programming with MPI

Load Distribution

Designing Load . . .

Classification of . . .

Examples of LD- . . .

Performance Evaluation

Introduction to Scientific Computing

9. ImplementationMiriam Mehl

1. Implementation: Target Architectures

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• different target architectures for numerical simulations:

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– monoprocessors

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– monoprocessors

– supercomputers

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– monoprocessors

– supercomputers

• modern microprocessors:

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– monoprocessors

– supercomputers

– obvious trends:

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– monoprocessors

– supercomputers

– obvious trends:

* increasing clock rates (> 2GHz almost standard)

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– monoprocessors

– supercomputers

– obvious trends:

* more MIPS, more FLOPS

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– monoprocessors

– supercomputers

– obvious trends:

* very-, ultra-, and ???-large scale integration; hence, moretransistors and more functionality on the chip

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– monoprocessors

– supercomputers

– obvious trends:

* longer words: 64 Bit architectures are standard (work-stations) or coming (PCs)

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– monoprocessors

– supercomputers

– obvious trends:

– important features:

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– monoprocessors

– supercomputers

– obvious trends:

* RISC (Reduced Instruction Set Computer) technology

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– monoprocessors

– supercomputers

– obvious trends:

* well-developed pipelining

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– monoprocessors

– supercomputers

– obvious trends:

* superscalarprocessor organization

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– monoprocessors

– supercomputers

– obvious trends:

* cachingand multi-level memory hierarchy

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– monoprocessors

– supercomputers

– obvious trends:

* cachingand multi-level memory hierarchy

* VLIW, Multi Thread Architecture, On-chip multiproces-sors, ...

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

2. RISC Technology

• counter-trend to CISC: more and more complex instructions en-tailing microprogramming

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

2. RISC Technology

• now instead:

– relatively small number of instructions (tens)

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

2. RISC Technology

• now instead:

– simple machine instructions, fixed format, few address modes

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

2. RISC Technology

• now instead:

– load-and-storeprinciple: only explicit LOAD/WRITE instruc-tions have memory access

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

2. RISC Technology

• now instead:

– load-and-storeprinciple: only explicit LOAD/WRITE instruc-tions have memory access

– no more need for microprogramming

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

3. Pipelining

• decompose instructions into simple steps involving different partsof the CPU:

– load,

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

3. Pipelining

– load,

– decode,

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

3. Pipelining

– load,

– decode,

– reserve registers,

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

3. Pipelining

– load,

– decode,

– execute,

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

3. Pipelining

– load,

– decode,

– execute,

– write results

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

3. Pipelining

– load,

– decode,

– execute,

– write results

• further improvement: reorder steps of an instruction (LOAD asearly as possible, WRITE as late as possible: avoids risk of idlewaiting time)

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

3. Pipelining

– load,

– decode,

– execute,

– write results

• best case: identical instructions to be pipelined/overlapped, asin vector processors

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

3. Pipelining

– load,

– decode,

– execute,

– write results

• best case: identical instructions to be pipelined/overlapped, asin vector processors

• pipelining needs different functional units in the CPU that candeal with the different steps in parallel; therefore:

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

4. Superscalar Processors

• several parts of the CPU are available in more than 1 copy

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• example: MIPS R10000 has 5 execution pipelines

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– one for FP-multiplication, one for FP-addition

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– two integer ALU (arithmetic-logical units)

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– two integer ALU (arithmetic-logical units)

– one address pipeline

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

5. Cache Memory

• CPU performance increased faster than memory access speed

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

5. Cache Memory

• thus: reduce memory access time / latency

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

5. Cache Memory

• cache memory: small and fast on-chip memory, keeps part ofthe main memory

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

5. Cache Memory

• optimum: needed data is always available in cache memory

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

5. Cache Memory

• look for strategies to ensure hit-probability p close to 1:

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

5. Cache Memory

– choice of section: what to be kept in cache?

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

5. Cache Memory

– ensure locality of data (instructions in cache need data incache)

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

5. Cache Memory

– strategies for fetching, replacement, and updating

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

5. Cache Memory

– association: how to check whether data are available incache?

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

5. Cache Memory

– association: how to check whether data are available incache?

– consistency: no different versions in cache and main mem-ory

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

6. Memory Hierarchy

• today: several cache levels → memory hierarchy:

– register,

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

6. Memory Hierarchy

– register,

– (level-1/2/3) cache,

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

6. Memory Hierarchy

– register,

– main memory,

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

6. Memory Hierarchy

– register,

– main memory,

– hard disk,

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

6. Memory Hierarchy

– register,

– main memory,

– hard disk,

– remote memory

the faster, the smaller

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

6. Memory Hierarchy

– register,

– main memory,

– hard disk,

– remote memory

• notion of the target computer’s memory hierarchy is importantfor numerical algorithms’ efficiency:

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

6. Memory Hierarchy

– register,

– main memory,

– hard disk,

– remote memory

– example: matrix-vector product Ax with A too large for cache

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

6. Memory Hierarchy

– register,

– main memory,

– hard disk,

– remote memory

– standard algorithm:

* outer loop over rows of A,

* inner loop for scalar product of one row of A with x

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

6. Memory Hierarchy

– register,

– main memory,

– hard disk,

– remote memory

– if current contents of cache are some rows of A, it’s OK

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

6. Memory Hierarchy

– register,

– main memory,

– hard disk,

– remote memory

– if current contents of cache are some rows of A, it’s OK

– if current contents of cache are some columns of A: slow!

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

6. Memory Hierarchy

– register,– (level-1/2/3) cache,– main memory,– hard disk,– remote memory

– example: matrix-vector product Ax with A too large for cache– standard algorithm:

– if current contents of cache are some rows of A, it’s OK– if current contents of cache are some columns of A: slow!– tuning crucial: peak performance up to 4 orders of magni-

tude higher than performance observed in practice (withouttuning)

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

7. Parallel Computers – Topologies

• parallel computers – distributed systems: frontier?

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• different possibilities of arrangement:

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– static network topologies:

* bus, ring, grid, or torus

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

* binary tree or fat tree

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

* hypercube

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

* hypercube

– dynamic network topologies:

* crossbar switch

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

* hypercube

* crossbar switch

* shuffle exchange network

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

* hypercube

* crossbar switch

• crucial quantities:

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

* hypercube

* crossbar switch

– diameter (longest path between two processors)

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

* hypercube

* crossbar switch

– number of network connections (ports) per processor

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

* hypercube

* crossbar switch

– parallel communications possible?

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

* hypercube

* crossbar switch

– parallel communications possible?

– existence of bottlenecks?

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

8. Flynn’s Classification (1972)

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• SISD: Single Instruction Single Data

– classical von-Neumann monoprocessor

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• SIMD : Single Instruction Multiple Data

– vector computers: extreme pipeling, one instruction appliedto a sequence (vector) of data (CRAY 1,2,X,Y,J/C/T90,. . . )

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– array computers: array of processors, concurrency (Think-ing Machines CM-2, MasPar MP-1, MP-2)

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• MIMD : Multiple Instruction Multiple Data

– multiprocessors:

* distributed memory(loose coupling, explicit communica-tion; Intel Paragon, IBM SP-2) or

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

* shared memory(tight coupling, global address space, im-plicit communication; most workstation servers) or

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

* nets/clusters

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

* nets/clusters

• MISD : Multiple Instruction Single Data: rare

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

9. Memory Access Classification

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• other criteria for classification:

scalability (S), programming model (PM), portability (P), and loaddistribution (L)

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• UMA : Uniform Memory Access

– shared memory systems: SMP (symmetric multiprocessors,parallel vector processors); PC- and WS-servers, CRAYYMP

– advantage: P, PM, L; drawback: S

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• NORMA : No Remote Memory Access

– distributed memory systems; clusters, IBM SP-2, iPSC/860

– advantage: S; drawback: P, PM, L

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• NORMA : No Remote Memory Access

– distributed memory systems; clusters, IBM SP-2, iPSC/860

– advantage: S; drawback: P, PM, L

• NUMA : Non-Uniform Memory Access

– systems with virtually shared memory; KSR-1, CRAY T3D/T3E,CONVEX SPP

– Advantage: PM, S, P; drawback: cache-coherence, com-mun.

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

10. Parallelization

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

10. Parallelization

• classical programming paradigms are, in principle, all well-suitedfor explicit or implicit parallelization:

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

10. Parallelization

– imperative: FORTRAN, C (dominant male, recently withsome OO-touch like in C++)

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

10. Parallelization

– logical/relational: PROLOG

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

10. Parallelization

– object-oriented: SMALLTALK

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

10. Parallelization

– functional/applicative: LISP

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

10. Parallelization

• implicit parallelization typically via special compilers

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

10. Parallelization

• explicit parallelization typically via linked communication libraries

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

10. Parallelization

• traditional way in Scientific Computing: FORTRAN code,vectorizing compiler, CRAY, wait for results

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

10. Parallelization

• traditional way in Scientific Computing: FORTRAN code,vectorizing compiler, CRAY, wait for results

• explicit parallelization often difficult (cf. Gauß-Seidel), this makesnon-conventional approaches attractive

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

11. The Programming Model MPI

• How to write parallel programs?

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– UMA systems: simple answer – just as sequential ones

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– distributed memory systems: MPI model or standard

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

* Message Passing Interface

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

* originally for clusters, today used even on massivelyparallel computers, too

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

* MPI-1 developed 1992-1994

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

* explicit exchange of messages: higher amount of pro-gramming work, but increasing possibilities of tuning andoptimizing

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• MPI Features:

– parallel program: n processes, separate address spaces,no remote access

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• MPI Features:

– message exchange via system calls sendand receive

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• MPI Features:

– message exchange via system calls sendand receive

– MPI-kernel: library of communication routines, allowing tointegrate MPI commands into standard languages

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

12. MPI Messages

• messages consist of a

– header (recipient, buffer, type, context of communication)and of their

– body(contents)

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

12. MPI Messages

– body(contents)

• messages are buffered (send buffer, receive buffer)

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

12. MPI Messages

– body(contents)

• sending a message can be

– blocking(finished only after message has left node) or

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

12. MPI Messages

– body(contents)

– non-blocking(finished immediately, message may be sentlater)

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

12. MPI Messages

– body(contents)

• the same holds for receiving a message:

– blocking: waiting;

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

12. MPI Messages

– body(contents)

– non-blocking: looking for it from time to time

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

12. MPI Messages

– body(contents)

– non-blocking: looking for it from time to time

cost of passing a message (length N, buffer cap. K):

t(N) = α · NK

+ β ·Ninitializing cost/time α, transportation cost β

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

13. Programming with MPI

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• a simple example:

P1: compute something P2: compute somethingstore result in SBUF store result in SBUFSendBlocking(P2,SBUF) SendBlocking(P1,SBUF)RecBlocking(P2,RBUF) RecBlocking(P1,RBUF)read data in RBUF read data in RBUFcompute again compute again

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• without buffering: deadlocks possible

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– nothing specified: buffering possible, but not imperative

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– never: no buffering (efficient, but risky)

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– always: secure, but sometimes costly

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– always: secure, but sometimes costly

• collective communication features available:

– broadcast, gather, gather-to-all, scatter, all-to-all,. . .

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

14. Load Distribution

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• load: amount of work on processors

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– optimum: minimize idle times; needs estimates and moni-toring

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– strategy: load balancingor load distribution or scheduling

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– important: avoid overhead

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• one distinguishes

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– scheduling:

* global: where do which processes run?

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– scheduling:

* local: when does which processor which process

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– scheduling:

– load balancing:

* static: a priori

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– scheduling:

– load balancing:

* static: a priori

* dynamic: during runtime

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– scheduling:

– load balancing:

* static: a priori

• in Scientific Computing applications load is often not predictable:

– adaptive refinement of a finite element mesh,

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– scheduling:

– load balancing:

* static: a priori

– convergence behaviour of iterations may differ

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– scheduling:

– load balancing:

* static: a priori

– convergence behaviour of iterations may differ

– thus: static load balancing not sufficient

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

15. Designing Load Distribution

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• Which are the primary objectives?

– optimization of system loador application runtime?

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– placementof new processes or migration of running pro-cesses?

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• Which is the level of integration?

– Who initiates actions (measure load, chose strategy)?

* application program

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

* runtime system

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

* runtime system

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

* runtime system

• Any special features of the application to be considered?

– restrictions in allocation process-to-processor frequent inS.C.

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

* runtime system

• Which units shall be distributed or displaced?

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

* runtime system

– whole processes (coarse grain)

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

* runtime system

– threads (fine grain)

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

* runtime system

– threads (fine grain)

– objects or data (typical for simulation applications)

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

16. Classification of Strategies

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• origin of the idea:

from physics (diffusion model), from combinatorics (graph the-ory), economics (bidding, brokerage)

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• for networks, for bus topologies

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• data represented as grids, trees, sets, or . . .

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• distribution mechanisms:

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– load handed over to neighbouring nodes only?

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

– just distribution of new units or migration of running ones(how?)?

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• flow of information:

to whom is load communicated, from where comes information?

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• coordination:

who makes decisions? autonomous/cooperative/competitive?

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• coordination:

who makes decisions? autonomous/cooperative/competitive?

• algorithms:

who initiates measures? adaptivity? costs relevant? evalua-tion?

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

17. Examples of LD-Strategies

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• diffusion model:

permanent balancing process between neighbours

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• bidding model:

supply and demand, establishment of some market

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• bidding model:

• broker model:

– esp. for heterogeneous hierarchical topologies, scalable

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• bidding model:

• broker model:

– broker with partial knowledge, budget-based decision whetherlocal processing or looking for better offers

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• bidding model:

• broker model:

– prices for use of resources and brokerage

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• bidding model:

• broker model:

• matching model:

construct matching in topology graph, balance along edges

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

• bidding model:

• broker model:

• matching model:

construct matching in topology graph, balance along edges

• balanced allocation, space-filling curves, . . .

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

18. Performance Evaluation

• performance evaluation of algortihms and computers

• average parallelism(for p processors):

A(p) =sum of processor runtimes

parallel runtime

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

parallel runtime

• speedup S: S =sequential runtime

parallel runtime

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

parallel runtime

• efficiency E: E = Sp

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

parallel runtime

• Amdahl’s Law :

assumption: each program has some part 0 < seq < 1 that canonly be treated in a sequential way

S ≤ 1

seq+ 1−seqp

≤ 1seq

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

parallel runtime

S ≤ 1

seq+ 1−seqp

≤ 1seq

• another important quantity: CCR (Communication-to-ComputationRatio)

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

parallel runtime

S ≤ 1

seq+ 1−seqp

≤ 1seq

– CCR often increases with increasing p and constant prob-lem size (example: iterative methods for Ax = b)

RISC Technology

Pipelining

Cache Memory

Memory Hierarchy

Memory Access . . .

Parallelization

MPI Messages

Load Distribution

parallel runtime

• Amdahl’s Law :assumption: each program has some part 0 < seq < 1 that canonly be treated in a sequential way

S ≤ 1

seq+ 1−seqp

≤ 1seq

– CCR often increases with increasing p and constant prob-lem size (example: iterative methods for Ax = b)

– therefore: do not compare speedups for different p, butsame problem size

1. implementation: target architectures · implementation:... risc technology ... • different...

Documents

implementation of particle filter-based target...

design and implementation of efficient multiplier...

module 3 - iptv architectures and implementation

landmine detection architectures and their implementation on

fpga based implementation of fft processor using different...

the implementation of sdgs goal 15 target 2 in …

design and implementation a different architectures of mix...

an efficient hardware implementation of target recognition

efficient implementation of reductions on gpu architectures

system and network architectures and...

lunar architectures - usra · lunar architectures paul d....

performance portability across diverse computer...

understanding the implementation of system architectures...

advoss software architectures. target markets wimax/4g...

heterogeneous architectures for implementation of · pdf...

tqm - national productivity centre · 2019-07-19 ·...

dual-mode fpga implementation of target and anomaly...

an efficient fpga implementation of multifunction …...

design and implementation of parallel memory...

architectures for efficient implementation of particle...