introduction to mpi nischint rajmohan [email protected] 5 november 2007
TRANSCRIPT
What can you expect ? Overview of MPI Basic MPI commands How to parallelize and execute a program using MPI
& MPICH2
What is outside the scope ? Technical details of MPI MPI implementations other than MPICH Hardware specific optimization techniques
Overview of MPI
• MPI stands for Message Passing Interface• What is Message Passing Interface?
It is not a programming language or compiler specification It is not a specific implementation or product MPI is a specification for the developers and users of message
passing libraries. By itself, it is NOT a library - but rather the specification of what such a library should be.
The specifications lets you create libraries that allow you to do problems in parallel using message passing to communicate between processes
It provides binding for widely used programming languages like Fortran, C/C++
Background on MPI• Early vendor systems (Intel’s NX, IBM’s EUI,
TMC’s CMMD) were not portable (or very capable)
• Early portable systems (PVM, p4, TCGMSG, Chameleon) were mainly research efforts
– Did not address the full spectrum of issues
– Lacked vendor support– Were not implemented at the most
efficient level• The MPI Forum organized in 1992 with broad
participation by:– vendors: IBM, Intel, TMC, SGI, Convex,
Meiko– portability library writers: PVM, p4– users: application scientists and library
writers– finished in 18 months– Library standard defined by a
committee of vendors, implementers, and parallel programmers
Reasons for using a MPI standard
• Standardization - MPI is the only message passing library which can be considered a standard. It is supported on virtually all HPC platforms. Practically, it has replaced all previous message passing libraries.
• Portability - There is no need to modify your source code when you port your application to a different platform that supports (and is compliant with) the MPI standard.
• Performance Opportunities - Vendor implementations should be able to exploit native hardware features to optimize performance.
• Functionality - Over 115 routines are defined in MPI-1 alone.
• Availability - A variety of implementations are available, both vendor and public domain.
MPI Operation
0
1
2
3
0
1
2
3
B1B2
B4B3
A A1A2
A4A3
B
Send
Receive
Processing
Communicator
MPI Programming Model
MPI Library
MPI Library
Environment
Management
Routines
Point to Point
Communication
Routines
Collective
Communication
Routines
Ex: MPI_INIT,
MPI_Comm_size,
MPI_Comm_rank,
MPI_Finalize
Non-Blocking
Routines
Blocking
Routines
Ex: MPI_Barrier,
MPI_Bcast
Ex: MPI_Send,
MPI_Recv
Ex:MPI_Isend,
MPI_Irecv
Environment Management Routines
• MPI_Init– Initializes the MPI execution environment. This
function must be called in every MPI program, must be called before any other MPI functions and must be called only once in an MPI program. For C programs, MPI_Init may be used to pass the command line arguments to all processes, although this is not required by the standard and is implementation dependent.
C: MPI_Init (&argc,&argv)
Fortran: MPI_INIT (ierr)
Environment Management Routines contd.
• MPI_Comm_Rank– Determines the rank of the calling process within the
communicator. Initially, each process will be assigned a unique integer rank between 0 and number of processors - 1 within the communicator MPI_COMM_WORLD. This rank is often referred to as a task ID. If a process becomes associated with other communicators, it will have a unique rank within each of these as well.
C: MPI_Comm_rank (comm,&rank) FORTRAN: MPI_COMM_RANK (comm,rank,ierr)
Environment Management Routines contd.
• MPI_Comm_size– Determines the number of processes in the group associated
with a communicator. Generally used within the communicator MPI_COMM_WORLD to determine the number of processes being used by your application.
C: MPI_Comm_size (comm,&size) Fortran: MPI_COMM_SIZE (comm,size,ierr)
• MPI_Finalize– Terminates the MPI execution environment. This function should
be the last MPI routine called in every MPI program - no other MPI routines may be called after it.
C: MPI_Finalize ()Fortran: MPI_FINALIZE (ierr)
MPI Sample Program: Environment Management Routines
! In C! /* the mpi include file */ #include "mpi.h"#include <stdio.h>int main( argc, argv )int argc;char *argv[];{ int rank, size;!/* Initialize MPI */ MPI_Init( &argc, &argv );
!/* How many processors are there?*/ MPI_Comm_size( MPI_COMM_WORLD,
&size );
!/* What processor am I (what is my rank)? */
MPI_Comm_rank( MPI_COMM_WORLD, &rank );
printf( "I am %d of %d\n", rank, size );
MPI_Finalize(); return 0;}
! In Fortran
program main
! /* the mpi include file */
include 'mpif.h'
integer ierr, rank, size
!/* Initialize MPI */
call MPI_INIT( ierr )
!/* How many processors are there?*/
call MPI_COMM_SIZE( MPI_COMM_WORLD, size, ierr )
!/* What processor am I (what is my rank)? */
call MPI_COMM_RANK( MPI_COMM_WORLD, rank, ierr )
print *, 'I am ', rank, ' of ', size
call MPI_FINALIZE( ierr )
end
Point to Point Communication Routines
MPI_Send • Basic blocking send operation. Routine returns only after the
application buffer in the sending task is free for reuse. Note that this routine may be implemented differently on different systems. The MPI standard permits the use of a system buffer but does not require it.
C:
MPI_Send (&buf,count,datatype,dest,tag,comm)
Fortran:
MPI_SEND(buf,count,datatype,dest,tag,comm,ierr)
Point to Point Communication Routines contd.
MPI_Recv • Receive a message and block until the requested data is
available in the application buffer in the receiving task.
C:MPI_Recv(&buf,count,datatype,source,tag,comm)
Fortran:MPI_RECV(buf,count,datatype,source,tag,comm,ierr)
MPI Sample Program: Send and Receive
! In C #include <stdio.h> #include "mpi.h" main(int argc, char** argv){
int my_PE_num, numbertoreceive, numbertosend=42; MPI_Status status;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &my_PE_num);
if (my_PE_num==0){ MPI_Recv( &numbertoreceive,
1, MPI_INT, MPI_ANY_SOURCE, MPI_ANY_TAG,
MPI_COMM_WORLD, &status); printf("Number received is:
%d\n", numbertoreceive);} else MPI_Send( &numbertosend, 1, MPI_INT, 0, 10, MPI_COMM_WORLD);
MPI_Finalize(); }
! In Fortran program shifter
implicit none
include 'mpif.h'
integer my_pe_num, errcode, numbertoreceive, numbertosend
integer status(MPI_STATUS_SIZE)
call MPI_INIT(errcode)
call MPI_COMM_RANK(MPI_COMM_WORLD, my_pe_num, errcode)
numbertosend = 42
if (my_PE_num.EQ.0) then
call MPI_Recv( numbertoreceive, 1, MPI_INTEGER,MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, status, errcode)
print *, 'Number received is:‘,numbertoreceive
Endif
if (my_PE_num.EQ.1) then
call MPI_Send( numbertosend, 1,MPI_INTEGER, 0, 10, MPI_COMM_WORLD,errcode)
endif
call MPI_FINALIZE(errcode)
end
Collective Communication Routines
• MPI_Barrier– Creates a barrier synchronization in a group. Each task, when
reaching the MPI_Barrier call, blocks until all tasks in the group reach the same MPI_Barrier call.
C: MPI_Barrier (comm) Fortran: MPI_BARRIER (comm,ierr)
• MPI_Bcast – Broadcasts (sends) a message from the process with rank "root"
to all other processes in the group. C: MPI_Bcast (&buffer,count,datatype,root,comm)
Fortran: MPI_BCAST (buffer,count,datatype,root,comm,ierr)
• Send a large message from process 0 to process 1– If there is insufficient storage at the destination, the send must wait for the user to provide
the memory space (through a receive)
• What happens with this code?
Sources of Deadlocks
Process 0
Send(1)Recv(1)
Process 1
Send(0)Recv(0)
• This is called “unsafe” because it depends on the availability of system buffers
MPICH – MPI Implementation
• MPICH is a freely available, portable implementation of MPI
• MPICH acts as the middleware between the MPI parallel library API and the hardware environment
• MPICH build is available for Unix based systems and also as an installer for Windows.
• MPICH2 is latest version of the implementation
• http://www-unix.mcs.anl.gov/mpi/mpich/
JobScheduler
ProcessManager
Message Progression
Communication Component
MPI Middleware
Parallel Application
Parallel Application
Resource Resource Resource Resource Resource
Parallel Application
Parallel Application
MPI Parallel Library API
MPI Program Compilation (Unix)
Fortran
mpif90 –c hello_world.f mpif90 –o hello_world hello_world.o
C mpicc -c hello_world.cc
mpicc –o hello_world hello_world.o
MPI Program Execution
Fortran/C
mpiexec –n 4 ./hello_world
mpiexec is the command for execution in parallel environment
used for specifying number of processors
mpiexec -help
This command should give all the options available to run mpi programs
If you don’t have mpiexec installed on your system, use mpirun and use –np instead of -n
MPI Program Execution contd.
mpiexec –machinefile hosts –n 7 ./hello_world
This flag allows you to specify a file containing the host name of the processors you want to use
master
master
node2
node3
node3
node5
node6
Sample hosts file :
MPICH on Windows
• Installing MPICH2
1. Download the Win32-IA32 version of MPICH2 from:
http://www-unix.mcs.anl.gov/mpi/mpich2/
2. Run the executable, mpich2-1.0.3-1-win32-ia32.msi (or a more recent version). Most likely it will result in the following error:
3. To download version1.1 use this link:
http://www.microsoft.com/downloads/details.aspx?FamilyId=262D25E3-F589-4842-8157-034D1E7CF3A3&displaylang=en
MPICH on Windows contd.
4. Install the .NET Framework program 5. Install the MPICH2 executable. Write down the passphrase for future reference.
The passphrase must be consistent across a network.6. Add the MPICH2 path to Windows:
1. Right click “My Computer” and pick properties2. Select the Advanced Tab3. Select the Environment Variables button4. Highlight the path variable under System Variables and click edit. Add
“C:\MPICH2\bin” to the end of the list, make sure to separate this from the prior path with a semicolon.
7. Run the example executable to insure correct installation. 8. mpiexec –n 2 cpi.exe9. If installed on a dual processor machine, verify that both processors are being
utilized by examining “CPU Usage History” in the Windows Task Manager.10. The first time each session mpiexec is run it will ask for username and password.
To prevent being asked for this in the future, this information can be encrypted into the Windows registry by running:
11. mpiexec –register12. The username and password are your Windows XP logon information.
MPICH on Windows contd.
• Compilation (Fortran)
ifort /fpp /include:”C:MPICH2/INCLUDE” /names/uppercase /iface:cref /libs:static /threads /c hello_world.f– The above command will compile the parallel program and create a .obj
file.
ifort –o hello_world.exe hello_world.obj C:/MPICH2/LIB/cxx.lib C:/MPICH2/LIB/mpi.lib C:/MPICH2/LIB/fmpich2.lib C:/MPICH2/LIB/fmpich2s.lib C:/MPICH2/LIB/fmpich2g.lib– The above command will link the object file and create the executable.
The executable is run in the same way as specified before using mpiexec command
THE END
• Useful Sources:
http://www.llnl.gov/computing/tutorials/mpi/#LLNL
http://www-unix.mcs.anl.gov/mpi/
• CS 6290 - High Performance Computing & Architecture
• For more assistance, you can contact
Nischint Rajmohan
MK402
404-894-6301