how to make mpi awesome: a proposal for mpi sessions

HowtomakeMPIAwesome:MPISessions

WesleyBland,IntelRyanGrant,SandiaNa=onalLaboratory

DanHolmes,EdinburghParallelCompu=ngCenterKathrynMohror,LawrenceLivermoreLaboratoryMar=nSchulz,LawrenceLivermoreLaboratory

AnthonySkjellum,AuburnUniversityJeffSquyres,CiscoSystems,Inc.

^

more

Follow-ontoJeff’scrazythoughtsdiscussedatEuroMPI2015inBordeaux,France

Whatwewant

•  FixMPI-3.1limita=ons:– Cannotini=alizeMPIfromdifferenten==eswithinaprocesswithoutaprioriknowledge/coordina=on

– Cannotini=alizeMPImorethanonce– Cannotre-ini=alizeMPIa\erithasbeenfinalized– CannotseterrorbehaviorofMPIini=aliza=on

Whatwewant•  Anythread(e.g.,library)canuseMPIany=meitwants•  Buts=llbeabletototallycleanupMPIif/whendesired

MPIProcess

// Library 3MPI_Init(…);

// Library 4MPI_Init(…);

// Library 5MPI_Init(…);

// Library 6MPI_Init(…);// Library 7

MPI_Init(…);

// Library 8MPI_Init(…);

// Library 9MPI_Init(…);

// Library 10MPI_Init(…);

// Library 11MPI_Init(…);

// Library 12MPI_Init(…);// Library 1

MPI_Init(…);// Library 2MPI_Init(…);

Howdowegetthosethings?

Newconcept:“session”

•  AlocalhandletotheMPIlibrary–  Implementa=onintent:lightweight/usesveryfewresources

– Canalsocachesomelocalstate

•  Canhavemul=plesessionsinanMPIprocess– MPI_Session_init(…,&session);– MPI_Session_finalize(…,&session);

MPISession

MPIProcess

oceanlibrary

MPI_SESSION_INIT(…)

atmospherelibrary

MPI_SESSION_INIT(…)

MPIlibrary

MPISession

MPIProcess

oceanlibrary

atmospherelibrary

MPIlibrary

oceansession

atmos-pheresession

UniquehandlestotheunderlyingMPIlibrary

MPISession

MPIProcess

oceanlibrary

atmospherelibrary

MPIlibrary

oceanErrorsreturn

atmos-phereErrorsabort

Uniqueerrorhandlers,info,localstate,etc.

Great.Ihaveasession.Nowwhat?

Fairwarning

•  TheMPIrun=mehaslong-sincebeenabastardstepchild–  Barelyacknowledgedinthestandard

– Mainlyintheformofnon-norma=vesugges=ons

•  It’s&metochangethat

Workwiththerun=me

•  Generalscheme:–  Querytheunderlyingrun-=mesystem•  Geta“set”ofprocesses

–  Determinetheprocessesyouwant•  CreateanMPI_Group

–  Createacommunicatorwithjustthoseprocesses•  CreateanMPI_Comm

Queryrun=meforsetofprocesses

MPI_Group

MPI_Comm

MPI_Session

Run=meexposessetsofprocesses

•  Setsareiden=fiedbystringname•  Twosetsaremandated– “mpi://WORLD”– “mpi://SELF”

•  Otherimplementa=on-definedsetscanbereported,too

Examplesofsets

MPIprocess0 MPIprocess1 MPIprocess2 MPIprocess3

mpi://WORLD

Examplesofsets

MPIprocess0 MPIprocess1 MPIprocess2 MPIprocess3

mpi://WORLD

arch://x86_64

Examplesofsets

MPIprocess0 MPIprocess1 MPIprocess2 MPIprocess3

mpi://WORLD

job://12942

arch://x86_64

Examplesofsets

MPIprocess0 MPIprocess1 MPIprocess2 MPIprocess3

loca=on://rack/17 loca=on://rack/23

Well,thatallsoundsgreat.

…butwhocallsMPI_INIT?

AndwhatsessiondoesMPI_COMM_WORLD/

MPI_COMM_SELFbelongto?

Newconcept:nolongerrequireMPI_INIT/MPI_FINALIZE

Theoveralltheme

•  JustuseMPIfunc=onswheneveryouwant–  Ini=aliza=onessen=allybecomesanimplementa=ondetail

•  Finaliza=onwilloccurwheneveralluser-definedhandlesaredestroyed

Exampleint main() { // Create a datatype – initializes MPI MPI_Type_contiguous(2, MPI_INT, &mytype);

Thecrea=onofthefirstuser-definedMPIobjectini=alizesMPI

Ini=aliza=oncanbealocalac=on!

Exampleint main() { // Create a datatype – initializes MPI MPI_Type_contiguous(2, MPI_INT, &mytype); // Free the datatype – finalizes MPI MPI_Type_free(&mytype);}

Thedestruc=onofthelastuser-definedMPIobjectfinalizesMPI.

Exampleint main() { // Create a datatype – initializes MPI MPI_Type_contiguous(2, MPI_INT, &mytype); // Free the datatype – finalizes MPI MPI_Type_free(&mytype);

// Re-initialize MPI! MPI_Type_dup(MPI_INT, &mytype);

Wecanalsore-ini=alizeMPI!(it’stransparenttotheuser–sowhynot?)

Keyinsight:SplitMPIAPIsintotwosets

Performancedoesn’tma/er(asmuch)

•  Func=onsthatcreate/query/destroy:–  MPI_Comm–  MPI_File–  MPI_Win–  MPI_Info–  MPI_Op–  MPI_Errhandler–  MPI_Datatype–  MPI_Group–  MPI_Session–  Arributes–  Processes

•  MPI_T

Performanceabsolutelyma/ers

•  Pointtopoint•  Collec=ves•  I/O•  RMA•  Test/Wait•  Handlelanguagexfer

Keyinsight:SplitMPIAPIsintotwosets

Performancedoesn’tma/er(asmuch)

•  Func=onsthatcreate/query/destroy:–  MPI_Comm–  MPI_File–  MPI_Win–  MPI_Info–  MPI_Op–  MPI_Errhandler–  MPI_Datatype–  MPI_Group–  MPI_Session–  Arributes–  Processes

•  MPI_T

Performanceabsolutelyma/ers

•  Pointtopoint•  Collec=ves•  I/O•  RMA•  Test/Wait•  Handlelanguagexfer

EnsurethatMPIisini=alized(and/orfinalized)bythese

func=ons

Thesefunc=onss=llcan’tbeusedunlessMPIis

ini=alized

Keyinsight:SplitMPIAPIsintotwosets

Performancedoesn’tma/er(asmuch)

•  Func=onsthatcreate/query/destroy:–  MPI_Comm–  MPI_File–  MPI_Win–  MPI_Info–  MPI_Op–  MPI_Errhandler–  MPI_Datatype–  MPI_Group–  MPI_Session–  Arributes–  Processes

•  MPI_T

Performanceabsolutelyma/ers

•  Pointtopoint•  Collec=ves•  I/O•  RMA•  Test/Wait•  Handlelanguagexfer

Thesefunc=onsinit/finalizeMPItransparently

Thesefunc=onscan’tbecalledwithoutahandlecreatedfrom

thele\-handcolumn

Waitaminute–WhataboutMPI_COMM_WORLD?

int main() { // Can’t I do this? MPI_Send(…, MPI_COMM_WORLD);

Thiswouldbecallinga“performancemarers”

func=onbeforea“performancedoesn’tmarer”

func=on

I.e.,MPIhasnotini=alizedyet

Waitaminute–WhataboutMPI_COMM_WORLD?

int main() { // This is valid MPI_Init(NULL, NULL); MPI_Send(…, MPI_COMM_WORLD);

Re-defineMPI_INITandMPI_FINALIZE:constructoranddestructorfor

MPI_COMM_WORLDandMPI_COMM_SELF

INITandFINALIZE

•  INIT/FINALIZEcreateanimplicitsession– YoucannotextractanMPI_SessionhandlefortheimplicitsessioncreatedbyMPI_INIT[_THREAD]

•  Yes,youcanuseINIT/FINALIZEinthesameMPIprocessasothersessions

(Abbreviated)Summary

•  MPIsession:aunitofisola=on– Allow“ocean”and“atmosphere”scenarios

•  Definesomeinterac=onswiththerun=me– Queryprocesssets,makegroups,makecommunicators

•  NolongerrequireMPI_INIT/FINALIZE– Keyinsight:cannotinvoke“performancemarers”func=onswithoutahandle

Moreideasanddetailsinthefullslidedeck

hrp://blogs.cisco.com/performance/“MPISessions:

AproposaltotheMPIForum”BlogentryfromMarch2,2016