Parallel Computing

• A task is broken down into tasks, performed by separate workers or processes

• Processes interact by exchanging information

• What do we basically need?– The ability to start the tasks– A way for them to communicate

Communication

• Cooperative– All parties agree to transfer data– Message passing is cooperative

• Data must be explicitly sent and received• Any change in the receiver’s memory is made with the

receiver’s participation

• One-sided– One worker performs the transfer of data– Data can be accessed without waiting for another

process

What is MPI?

• A message passing library specification– Message-passing model– Not a compiler specification (i.e. not a

language)– Not a specific product

• Designed for parallel computers, clusters, and heterogeneous networks

The MPI Process

• Development began in early 1992• Open process/Broad participation

– IBM,Intel, TMC, Meiko, Cray, Convex, Ncube– PVM, p4, Express, Linda, …– Laboratories, Universities, Government

• Final version of draft in May 1994• Public and vendor implementations are now

widely available

Point to Point Communication

• A message is sent from a sender to a receiver

• There are several variations on how the sending of a message can interact with the program

Synchronous

• A synchronous communication does not complete until the message has been received– A FAX or registered

mail

Asynchronous

• An asynchronous communication completes as soon as the message is on the way.– A post card or email

Blocking and Non-blocking

• Blocking operations only return when the operation has been completed– Normal FAX machines

• Non-blocking operations return right away and allow the program to do other work– Receiving a FAX

Collective Communications

• Point-to-point communications involve pairs of processes.

• Many message passing systems provide operations which allow larger numbers of processes to participate

Types of Collective Transfers

• Barrier– Synchronizes processors– No data is exchanged but the barrier blocks until all

processes have called the barrier routine

• Broadcast (sometimes multicast)– A broadcast is a one-to-many communication– One processor sends one message to several

destinations

• Reduction– Often useful in a many-to-one communication

What’s in a Message?

• An MPI message is an array of elements of a particular MPI datatype

• All MPI messages are typed– The type of the contents must be specified in

both the send and the receive

Basic C Datatypes in MPI

MPI Datatype C datatype

MPI_CHAR signed char

MPI_SHORT signed short int

MPI_INT signed int

MPI_LONG signed long int

MPI_UNSIGNED_CHAR unsigned char

MPI_UNSIGNED_SHORT unsigned short int

MPI_UNSIGNED_INT unsigned int

MPI_UNSIGNED_LONG unsigned long int

MPI_FLOAT float

MPI_DOUBLE double

MPI_LONG_DOUBLE long double

MPI_BYTE

MPI_PACKED

MPI Handles

• MPI maintains internal data-structures which are referenced by the user through handles

• Handles can be returned by and passed to MPI procedures

• Handles can be copied by the usual assignment operation

MPI Errors

• MPI routines return an int that can contain an error code

• The default action on the detection of an error is to cause the parallel operation to abort– The default can be changed to return an error

code

Initializing MPI

• The first MPI routine called in any MPI program must be the initialization routine MPI_INIT

• MPI_INIT is called once by every process, before any other MPI routines

int mpi_Init( int *argc, char **argv );

Skeleton MPI Program

#include <mpi.h>

main( int argc, char** argv ) { MPI_Init( &argc, &argv );

/* main part of the program */

MPI_Finalize();}

Communicators

• A communicator handle defines which processes a particular command will apply to

• All MPI communication calls take a communicator handle as a parameter, which is effectively the context in which the communication will take place

• MPI_INIT defines a communicator called MPI_COMM_WORLD for each process that calls it

Communicators

• Every communicator contains a group which is a list of processes

• The processes are ordered and numbered consecutively from 0.

• The number of each process is known as its rank– The rank identifies each process within the

communicator

• The group of MPI_COMM_WORLD is the set of all MPI processes

Point-to-point Communication

• Always involves exactly two processes• The destination is identified by its rank within the

communicator• There are four communication modes provided by

MPI (these modes refer to sending not receiving)– Buffered– Synchronous– Standard– Ready

Standard Send

• When using standard-mode send– It is up to MPI to decide whether outgoing messages will

be buffered.– Completes once the message has been sent, which may or

may not imply that the massage has arrived at its destination

– Can be started whether or not a matching receive has been posted. It may complete before a matching receive is posted.

– Has non-local completion semantics, since successful completion of the send operation may depend on the occurrence of a matching receive.

Standard Send

MPI_Send( buf, count, datatype, dest, tag, comm )

Where– buf is the address of the data to be sent– count is the number of elements of the MPI datatype which buf

contains– datatype is the MPI datatype– dest is the destination process for the message. This is specified by the

rank of the destination within the group associated with the communicator comm

– tag is a marker used by the sender to distinguish between different types of messages

– comm is the communicator shared by the sender and the receiver

Synchronous Send

• MPI_Ssend( buf, count, datatype, dest, tag, comm )

– can be started whether or not a matching receive was posted

– will complete successfully only if a matching receive is posted, and the receive operation has started to receive the message sent by the synchronous send.

– provides synchronous communication semantics: a communication does not complete at either end before both processes rendezvous at the communication.

– has non-local completion semantics.

Buffered Send

• A buffered-mode send– Can be started whether or not a matching receive has

been posted. It may complete before a matching receive is posted.

– Has local completion semantics: its completion does not depend on the occurrence of a matching receive.

– In order to complete the operation, it may be necessary to buffer the outgoing message locally. For that purpose, buffer space is provided by the application.

Ready Mode Send

• A ready-mode send– completes immediately– may be started only if the matching receive has

already been posted.– has the same semantics as a standard-mode

send.– saves on overhead by avoiding handshaking

and buffering