![Page 1: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/1.jpg)
1Copyright © 2010, Elsevier Inc. All rights Reserved
Chapter 3
Distributed Memory Programming with
MPI
An Introduction to Parallel ProgrammingPeter Pacheco
![Page 2: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/2.jpg)
2Copyright © 2010, Elsevier Inc. All rights Reserved
Roadmap
Writing your first MPI program. Using the common MPI functions. The Trapezoidal Rule in MPI. Collective communication. MPI derived datatypes. Performance evaluation of MPI programs. Parallel sorting. Safety in MPI programs.
# Chapter S
ubtitle
![Page 3: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/3.jpg)
3
A distributed memory system
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 4: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/4.jpg)
4
A shared memory system
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 5: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/5.jpg)
5
Hello World!
Copyright © 2010, Elsevier Inc. All rights Reserved
(a classic)
![Page 6: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/6.jpg)
6
Identifying MPI processes
Common practice to identify processes by nonnegative integer ranks.
p processes are numbered 0, 1, 2, .. p-1
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 7: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/7.jpg)
7
Our first MPI program
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 8: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/8.jpg)
8
Compilation
Copyright © 2010, Elsevier Inc. All rights Reserved
mpicc -g -Wall -o mpi_hello mpi_hello.c
wrapper script to compile
turns on all warnings
source file
create this executable file name
(as opposed to default a.out)
producedebugging
information
![Page 9: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/9.jpg)
9
Execution
Copyright © 2010, Elsevier Inc. All rights Reserved
mpiexec -n <number of processes> <executable>
mpiexec -n 1 ./mpi_hello
mpiexec -n 4 ./mpi_hello
run with 1 process
run with 4 processes
![Page 10: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/10.jpg)
10
Execution
Copyright © 2010, Elsevier Inc. All rights Reserved
mpiexec -n 1 ./mpi_hello
mpiexec -n 4 ./mpi_hello
Greetings from process 0 of 1 !
Greetings from process 0 of 4 !
Greetings from process 1 of 4 !
Greetings from process 2 of 4 !
Greetings from process 3 of 4 !
![Page 11: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/11.jpg)
11
MPI Programs Written in C.
Has main. Uses stdio.h, string.h, etc.
Need to add mpi.h header file. Identifiers defined by MPI start with “MPI_”. First letter following underscore is
uppercase. For function names and MPI-defined types. Helps to avoid confusion.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 12: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/12.jpg)
12
MPI Components
MPI_Init Tells MPI to do all the necessary setup.
MPI_Finalize Tells MPI we’re done, so clean up anything
allocated for this program.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 13: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/13.jpg)
13
Basic Outline
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 14: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/14.jpg)
14
Communicators
A collection of processes that can send messages to each other.
MPI_Init defines a communicator that consists of all the processes created when the program is started.
Called MPI_COMM_WORLD.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 15: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/15.jpg)
15
Communicators
Copyright © 2010, Elsevier Inc. All rights Reserved
number of processes in the communicator
my rank (the process making this call)
![Page 16: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/16.jpg)
16
SPMD
Single-Program Multiple-Data We compile one program. Process 0 does something different.
Receives messages and prints them while the other processes do the work.
The if-else construct makes our program SPMD.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 17: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/17.jpg)
17
Communication
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 18: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/18.jpg)
18
Data types
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 19: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/19.jpg)
19
Communication
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 20: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/20.jpg)
20
Message matching
Copyright © 2010, Elsevier Inc. All rights Reserved
MPI_Send
src = q
MPI_Recv
dest = r
r
q
![Page 21: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/21.jpg)
21
Receiving messages
A receiver can get a message without knowing: the amount of data in the message, the sender of the message, or the tag of the message.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 22: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/22.jpg)
22
status_p argument
Copyright © 2010, Elsevier Inc. All rights Reserved
MPI_SOURCE
MPI_TAG
MPI_ERROR
MPI_Status*
MPI_Status* status;
status.MPI_SOURCE
status.MPI_TAG
![Page 23: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/23.jpg)
23
How much data am I receiving?
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 24: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/24.jpg)
24
Issues with send and receive
Exact behavior is determined by the MPI implementation.
MPI_Send may behave differently with regard to buffer size, cutoffs and blocking.
MPI_Recv always blocks until a matching message is received.
Know your implementation;don’t make assumptions!
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 25: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/25.jpg)
25
TRAPEZOIDAL RULE IN MPI
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 26: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/26.jpg)
26
The Trapezoidal Rule
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 27: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/27.jpg)
27
The Trapezoidal Rule
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 28: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/28.jpg)
28
One trapezoid
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 29: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/29.jpg)
29
Pseudo-code for a serial program
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 30: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/30.jpg)
30
Parallelizing the Trapezoidal Rule
1. Partition problem solution into tasks.
2. Identify communication channels between tasks.
3. Aggregate tasks into composite tasks.
4. Map composite tasks to cores.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 31: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/31.jpg)
31
Parallel pseudo-code
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 32: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/32.jpg)
32
Tasks and communications for Trapezoidal Rule
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 33: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/33.jpg)
33
First version (1)
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 34: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/34.jpg)
34
First version (2)
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 35: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/35.jpg)
35
First version (3)
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 36: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/36.jpg)
36
Dealing with I/O
Copyright © 2010, Elsevier Inc. All rights Reserved
Each process justprints a message.
![Page 37: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/37.jpg)
37
Running with 6 processes
Copyright © 2010, Elsevier Inc. All rights Reserved
unpredictable output
![Page 38: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/38.jpg)
38
Input
Most MPI implementations only allow process 0 in MPI_COMM_WORLD access to stdin.
Process 0 must read the data (scanf) and send to the other processes.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 39: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/39.jpg)
39
Function for reading user input
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 40: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/40.jpg)
40
COLLECTIVE COMMUNICATION
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 41: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/41.jpg)
41
Tree-structured communication
1. In the first phase: (a) Process 1 sends to 0, 3 sends to 2, 5 sends to 4, and 7 sends to 6. (b) Processes 0, 2, 4, and 6 add in the received values. (c) Processes 2 and 6 send their new values to processes 0 and 4, respectively.(d) Processes 0 and 4 add the received values into their new values.
2. (a) Process 4 sends its newest value to process 0.(b) Process 0 adds the received value to its newest value.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 42: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/42.jpg)
42
A tree-structured global sum
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 43: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/43.jpg)
43
An alternative tree-structured global sum
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 44: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/44.jpg)
44
MPI_Reduce
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 45: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/45.jpg)
45
Predefined reduction operators in MPI
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 46: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/46.jpg)
46
Collective vs. Point-to-Point Communications
All the processes in the communicator must call the same collective function.
For example, a program that attempts to match a call to MPI_Reduce on one process with a call to MPI_Recv on another process is erroneous, and, in all likelihood, the program will hang or crash.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 47: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/47.jpg)
47
Collective vs. Point-to-Point Communications
The arguments passed by each process to an MPI collective communication must be “compatible.”
For example, if one process passes in 0 as the dest_process and another passes in 1, then the outcome of a call to MPI_Reduce is erroneous, and, once again, the program is likely to hang or crash.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 48: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/48.jpg)
48
Collective vs. Point-to-Point Communications
The output_data_p argument is only used on dest_process.
However, all of the processes still need to pass in an actual argument corresponding to output_data_p, even if it’s just NULL.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 49: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/49.jpg)
49
Collective vs. Point-to-Point Communications
Point-to-point communications are matched on the basis of tags and communicators.
Collective communications don’t use tags. They’re matched solely on the basis of the
communicator and the order in which they’re called.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 50: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/50.jpg)
50
Example (1)
Copyright © 2010, Elsevier Inc. All rights Reserved
Multiple calls to MPI_Reduce
![Page 51: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/51.jpg)
51
Example (2)
Suppose that each process calls MPI_Reduce with operator MPI_SUM, and destination process 0.
At first glance, it might seem that after the two calls to MPI_Reduce, the value of b will be 3, and the value of d will be 6.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 52: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/52.jpg)
52
Example (3)
However, the names of the memory locations are irrelevant to the matching of the calls to MPI_Reduce.
The order of the calls will determine the matching so the value stored in b will be 1+2+1 = 4, and the value stored in d will be 2+1+2 = 5.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 53: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/53.jpg)
53
MPI_Allreduce
Useful in a situation in which all of the processes need the result of a global sum in order to complete some larger computation.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 54: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/54.jpg)
54Copyright © 2010, Elsevier Inc. All rights Reserved
A global sum followedby distribution of theresult.
![Page 55: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/55.jpg)
55Copyright © 2010, Elsevier Inc. All rights Reserved
A butterfly-structured global sum.
![Page 56: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/56.jpg)
56
Broadcast
Data belonging to a single process is sent to all of the processes in the communicator.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 57: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/57.jpg)
57Copyright © 2010, Elsevier Inc. All rights Reserved
A tree-structured broadcast.
![Page 58: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/58.jpg)
58
A version of Get_input that uses MPI_Bcast
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 59: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/59.jpg)
59
Data distributions
Copyright © 2010, Elsevier Inc. All rights Reserved
Compute a vector sum.
![Page 60: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/60.jpg)
60
Serial implementation of vector addition
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 61: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/61.jpg)
61
Different partitions of a 12-component vector among 3 processes
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 62: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/62.jpg)
62
Partitioning options
Block partitioning Assign blocks of consecutive components to
each process. Cyclic partitioning
Assign components in a round robin fashion. Block-cyclic partitioning
Use a cyclic distribution of blocks of components.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 63: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/63.jpg)
63
Parallel implementation of vector addition
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 64: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/64.jpg)
64
Scatter
MPI_Scatter can be used in a function that reads in an entire vector on process 0 but only sends the needed components to each of the other processes.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 65: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/65.jpg)
65
Reading and distributing a vector
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 66: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/66.jpg)
66
Gather
Collect all of the components of the vector onto process 0, and then process 0 can process all of the components.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 67: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/67.jpg)
67
Print a distributed vector (1)
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 68: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/68.jpg)
68
Print a distributed vector (2)
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 69: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/69.jpg)
69
Allgather
Concatenates the contents of each process’ send_buf_p and stores this in each process’ recv_buf_p.
As usual, recv_count is the amount of data being received from each process.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 70: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/70.jpg)
70
Matrix-vector multiplication
Copyright © 2010, Elsevier Inc. All rights Reserved
i-th component of yDot product of the ithrow of A with x.
![Page 71: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/71.jpg)
71
Matrix-vector multiplication
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 72: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/72.jpg)
72
Multiply a matrix by a vector
Copyright © 2010, Elsevier Inc. All rights Reserved
Serial pseudo-code
![Page 73: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/73.jpg)
73
C style arrays
Copyright © 2010, Elsevier Inc. All rights Reserved
stored as
![Page 74: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/74.jpg)
74
Serial matrix-vector multiplication
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 75: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/75.jpg)
75
An MPI matrix-vector multiplication function (1)
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 76: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/76.jpg)
76
An MPI matrix-vector multiplication function (2)
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 77: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/77.jpg)
77
MPI DERIVED DATATYPES
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 78: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/78.jpg)
78
Derived datatypes
Used to represent any collection of data items in memory by storing both the types of the items and their relative locations in memory.
The idea is that if a function that sends data knows this information about a collection of data items, it can collect the items from memory before they are sent.
Similarly, a function that receives data can distribute the items into their correct destinations in memory when they’re received.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 79: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/79.jpg)
79
Derived datatypes
Formally, consists of a sequence of basic MPI data types together with a displacement for each of the data types.
Trapezoidal Rule example:
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 80: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/80.jpg)
80
MPI_Type create_struct
Builds a derived datatype that consists of individual elements that have different basic types.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 81: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/81.jpg)
81
MPI_Get_address
Returns the address of the memory location referenced by location_p.
The special type MPI_Aint is an integer type that is big enough to store an address on the system.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 82: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/82.jpg)
82
MPI_Type_commit
Allows the MPI implementation to optimize its internal representation of the datatype for use in communication functions.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 83: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/83.jpg)
83
MPI_Type_free
When we’re finished with our new type, this frees any additional storage used.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 84: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/84.jpg)
84
Get input function with a derived datatype (1)
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 85: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/85.jpg)
85
Get input function with a derived datatype (2)
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 86: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/86.jpg)
86
Get input function with a derived datatype (3)
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 87: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/87.jpg)
87
PERFORMANCE EVALUATION
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 88: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/88.jpg)
88
Elapsed parallel time
Returns the number of seconds that have elapsed since some time in the past.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 89: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/89.jpg)
89
Elapsed serial time
In this case, you don’t need to link in the MPI libraries.
Returns time in microseconds elapsed from some point in the past.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 90: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/90.jpg)
90
Elapsed serial time
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 91: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/91.jpg)
91
MPI_Barrier
Ensures that no process will return from calling it until every process in the communicator has started calling it.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 92: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/92.jpg)
92
MPI_Barrier
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 93: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/93.jpg)
93
Run-times of serial and parallel matrix-vector multiplication
Copyright © 2010, Elsevier Inc. All rights Reserved
(Seconds)
![Page 94: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/94.jpg)
94
Speedup
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 95: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/95.jpg)
95
Efficiency
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 96: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/96.jpg)
96
Speedups of Parallel Matrix-Vector Multiplication
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 97: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/97.jpg)
97
Efficiencies of Parallel Matrix-Vector Multiplication
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 98: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/98.jpg)
98
Scalability
A program is scalable if the problem size can be increased at a rate so that the efficiency doesn’t decrease as the number of processes increase.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 99: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/99.jpg)
99
Scalability
Programs that can maintain a constant efficiency without increasing the problem size are sometimes said to be strongly scalable.
Programs that can maintain a constant efficiency if the problem size increases at the same rate as the number of processes are sometimes said to be weakly scalable.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 100: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/100.jpg)
100
A PARALLEL SORTING ALGORITHM
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 101: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/101.jpg)
101
Sorting
n keys and p = comm sz processes. n/p keys assigned to each process. No restrictions on which keys are assigned
to which processes. When the algorithm terminates:
The keys assigned to each process should be sorted in (say) increasing order.
If 0 ≤ q < r < p, then each key assigned to process q should be less than or equal to every key assigned to process r.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 102: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/102.jpg)
102
Serial bubble sort
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 103: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/103.jpg)
103
Odd-even transposition sort
A sequence of phases. Even phases, compare swaps:
Odd phases, compare swaps:
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 104: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/104.jpg)
104
Example
Start: 5, 9, 4, 3
Even phase: compare-swap (5,9) and (4,3) getting the list 5, 9, 3, 4
Odd phase: compare-swap (9,3)getting the list 5, 3, 9, 4
Even phase: compare-swap (5,3) and (9,4) getting the list 3, 5, 4, 9
Odd phase: compare-swap (5,4)getting the list 3, 4, 5, 9
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 105: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/105.jpg)
105
Serial odd-even transposition sort
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 106: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/106.jpg)
106
Communications among tasks in odd-even sort
Copyright © 2010, Elsevier Inc. All rights Reserved
Tasks determining a[i] are labeled with a[i].
![Page 107: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/107.jpg)
107
Parallel odd-even transposition sort
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 108: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/108.jpg)
108
Pseudo-code
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 109: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/109.jpg)
109
Compute_partner
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 110: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/110.jpg)
110
Safety in MPI programs
The MPI standard allows MPI_Send to behave in two different ways: it can simply copy the message into an MPI
managed buffer and return, or it can block until the matching call to
MPI_Recv starts.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 111: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/111.jpg)
111
Safety in MPI programs
Many implementations of MPI set a threshold at which the system switches from buffering to blocking.
Relatively small messages will be buffered by MPI_Send.
Larger messages, will cause it to block.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 112: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/112.jpg)
112
Safety in MPI programs
If the MPI_Send executed by each process blocks, no process will be able to start executing a call to MPI_Recv, and the program will hang or deadlock.
Each process is blocked waiting for an event that will never happen.
Copyright © 2010, Elsevier Inc. All rights Reserved
(see pseudo-code)
![Page 113: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/113.jpg)
113
Safety in MPI programs
A program that relies on MPI provided buffering is said to be unsafe.
Such a program may run without problems for various sets of input, but it may hang or crash with other sets.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 114: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/114.jpg)
114
MPI_Ssend
An alternative to MPI_Send defined by the MPI standard.
The extra “s” stands for synchronous and MPI_Ssend is guaranteed to block until the matching receive starts.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 115: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/115.jpg)
115
Restructuring communication
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 116: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/116.jpg)
116
MPI_Sendrecv
An alternative to scheduling the communications ourselves.
Carries out a blocking send and a receive in a single call.
The dest and the source can be the same or different.
Especially useful because MPI schedules the communications so that the program won’t hang or crash.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 117: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/117.jpg)
117
MPI_Sendrecv
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 118: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/118.jpg)
118
Safe communication with five processes
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 119: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/119.jpg)
119
Parallel odd-even transposition sort
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 120: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/120.jpg)
120
Run-times of parallel odd-even sort
Copyright © 2010, Elsevier Inc. All rights Reserved
(times are in milliseconds)
![Page 121: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/121.jpg)
121
Concluding Remarks (1)
MPI or the Message-Passing Interface is a library of functions that can be called from C, C++, or Fortran programs.
A communicator is a collection of processes that can send messages to each other.
Many parallel programs use the single-program multiple data or SPMD approach.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 122: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/122.jpg)
122
Concluding Remarks (2)
Most serial programs are deterministic: if we run the same program with the same input we’ll get the same output.
Parallel programs often don’t possess this property.
Collective communications involve all the processes in a communicator.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 123: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/123.jpg)
123
Concluding Remarks (3)
When we time parallel programs, we’re usually interested in elapsed time or “wall clock time”.
Speedup is the ratio of the serial run-time to the parallel run-time.
Efficiency is the speedup divided by the number of parallel processes.
Copyright © 2010, Elsevier Inc. All rights Reserved
![Page 124: 1 Copyright © 2010, Elsevier Inc. All rights Reserved Chapter 3 Distributed Memory Programming with MPI An Introduction to Parallel Programming Peter Pacheco](https://reader033.vdocuments.us/reader033/viewer/2022061306/55147983550346ea6e8b45ea/html5/thumbnails/124.jpg)
124
Concluding Remarks (4)
If it’s possible to increase the problem size (n) so that the efficiency doesn’t decrease as p is increased, a parallel program is said to be scalable.
An MPI program is unsafe if its correct behavior depends on the fact that MPI_Send is buffering its input.
Copyright © 2010, Elsevier Inc. All rights Reserved