a spmd model for ocr (with collectives) sanjay chatterjee 2/9/2015 intel confidential1

Intel Confidential 1

A SPMD Model for OCR(with collectives)

Sanjay Chatterjee2/9/2015


OCR SPMD model• A SPMD context in OCR is a collection of individual logical execution units called ranks• A rank has a unique id within a SPMD context and can be viewed as a sequential chain of SPMD-EDTs• SPMD EDTs have special semantics and can exist only within a SPMD context• A SPMD context includes two kinds of SPMD EDT templates: compute and sync• SPMD ranks collectively start computation by individually calling COMPUTE• SPMD ranks collectively synchronize by individually calling SYNC• A SPMD EDT restarts itself by calling NEXT

SPMD CONTEXT

RANK 1I1

C10C11C12

S10

S11

C13

RANK 0I0

C00

S00S01S02S03

C01

RANK 2I2

C20C21C22

S20S21

S22

C23

RANK 3I3

C30C31C32

S30

S31

C33

COMPUTE PHASE

COLLECTIVE SYNCPHASE

COMPUTE PHASE

RANKMESSAGE

NEXTNEXT

SYNC

COMPUTE

COMPUTE

NEXT


Creating and launching a SPMD Context• u8 ocrSpmdLaunch(u64 numRanks, ocrInitFunc_t funcPtr, ocrGuid_t initDb,

ocrGuid_t computeTemplate, ocrGuid_t syncTemplate, ocrGuid_t outputEvent);• [in] numRanks : Number of ranks in the SPMD context• [in] funcPtr : Rank initialization function

• typedef void (*ocrInitFunc_t)(void *initDbPtr)• [in] initDb : DB that is passed to the initialization function on every rank.

• Every rank gets a private copy of this DB which is destroyed after the init function• [in] computeTemplate : Compute SPMD EDT template• [in] syncTemplate : Collective synchronization SPMD EDT template• [in] outputEvent : SPMD output event


SPMD EDTs vs Regular EDTs

• SPMD EDTs are similar to regular EDTs with some differences• SPMD EDTs are anonymized i.e they do not have a guid• A SPMD EDT only lives within a SPMD context and is associated with a rank• Returning from a SPMD EDT will exit the rank from the SPMD context• A SPMD EDT can restart itself by calling NEXT• A SPMD EDT is created as either a compute or synchronization EDT

• A compute EDT can call SYNC to exit itself and start a new sync EDT on the same rank• A sync EDT can call COMPUTE to exit itself and start a new compute EDT on the same rank• A compute EDT calling COMPUTE or a sync EDT calling SYNC is an error

• A SPMD EDT in one rank can communicate with another rank using rank messages• A SPMD EDT can add a self dependence


Creating SPMD EDTs in a rank

• Similar to regular EDT create, except no output guid or input template parameter.• u8 ocrComputeSpmdEdtCreate(u32 paramc, u64* paramv, u32 depc,

ocrGuid_t *depv, u16 properties, ocrGuid_t affinity, ocrGuid_t *outputEvent);• u8 ocrSyncSpmdEdtCreate(u32 paramc, u64* paramv, u32 depc, ocrGuid_t

*depv, u16 properties, ocrGuid_t affinity, ocrGuid_t *outputEvent);

• Has to be created inside the initialization function before making the first call to COMPUTE or SYNC


SPMD Rank Messages• SPMD rank messages support point-to-point communication between ranks• Messages can be communicated only between the same kind of SPMD EDT templates

• Compute SPMD EDTs on one rank can only send/receive messages to/from compute SPMD EDTs on other ranks• Sync SPMD EDTs on one rank can only send/receive messages to/from sync SPMD EDTs on other ranks

• Message ordering at source rank is guaranteed to be maintained at destination rank depv slot

• u8 ocrSend(u64 dstRank, u64 dstSlot, ocrGuid_t db);• [in] dstRank: rank id of message destination rank• [in] dstSlot: slot id at destination rank• [in] db: Guid of the datablock communicated• Called by message source• Message send is guaranteed to be complete after NEXT is called• Another send to the same location and slot is permitted only after calling NEXT

• u8 ocrRecv(u64 srcRank, u64 dstSlot);• [in] srcRank: rank id of the message source rank• [in] dstSlot: slot id in current rank where message will be received• Called by message destination• DB at destination can be accessed in slot after calling NEXT


SPMD EDT Self Dependence

• ocrAddSelfDependence(ocrGuid_t source, u32 slot, ocrDbAccessMode_t mode);• [in] source: Source of the dependence edge. Maybe event or DB.• [in] slot: Slot in the current SPMD EDT that will be satisfied by the dependence• [in] mode: The access mode on the DB attached to the slot

• Adds a dependence to an event or DB source• Allows SPMD EDT to wait for an event

• NEXT has to be called for completion of the wait on the satisfaction of the dependence

• The data from the source is visible only after calling NEXT


API for NEXT

• void ocrNext();• exits and restarts current SPMD EDT• All sends and receives called before ocrNext are guaranteed to be complete

before the EDT restarts• After restart, the depv slots that receive messages are updated with new DB.

• All other depv slots and params maintain their state from previous ocrNext


API for COMPUTE

• void ocrCompute();• Creates and launches a new SPMD EDT in the current rank from the compute

template of the SPMD context• Can be called from either the initialization function or a sync SPMD EDT• All compute EDTs in a rank share the same paramv and depv state

• setup during the initialization function• can be updated during the lifetime of the rank


API for SYNC

• void ocrSync(ocrCollective_t colType, ocrGuid_t db, bool reqResult);• [in] colType: type of collective synchronization to be performed.

• E.g: sum-reduction, barrier, etc• [in] db: DB that the current rank gives to the collective. Placed into depv[0] of sync EDT• [in] reqResult: Boolean to indicate if current rank needs the result of the collective• Creates and launches a new SPMD EDT in the current rank from the sync template of the

SPMD context• Can be called from initialization function or compute SPMD EDT

• u8 ocrSyncResult(ocrGuid_t *db);• [out] db: DB of the result from the collective• Can be called only from a compute SPMD EDT• Call will result in error if the previous ocrSync was called with reqResult as FALSE


Other API supported inside a SPMD context• u64 ocrGetRank() – returns the current rank• u64 ocrNumRanks() – returns total number of ranks in the SPMD

context


Example: Sum-Reduction…ocrEdtTemplateCreate(& syncRedTempl, syncRed, 2, 2);ocrEventCreate(& outputRed, OCR_EVENT_STICKY_T, TRUE);ocrSpmdLaunch(NUM_RANKS, initRed, NULL_GUID, NULL_GUID, syncRedTempl, outputRed);…}

void initRed(void *dbPtr) {ocrDbCreate(&elementDb, …);u64 paramv[2]; paramv[0] = ocrGetRank();paramv[1] = 1;ocrSyncSpmdEdtCreate(2, &paramv, 2, NULL, 0, NULL_GUID, NULL);ocrSync(SUM_REDUCTION_BINARY, elementDb, FALSE)}

ocrGuid_t syncRed ( u32 paramc, u64* paramv, u32 depc, ocrEdtDep_t depv[]) { u64 myRank = ocrGetRank(); u64 numRanks = ocrNumRanks(); u64 tree_counter = paramv[0]; if (tree_counter % 2) == 0) { //reduce: depv[0] = depv[0] + depv[1];

u64 srcRank = myRank + paramv[1]; if (srcRank >= numRanks) break; ocrRecv(srcRank, i);

paramv[0] = tree_counter / 2; paramv[1] *= 2; ocrNext(); } else { u64 dstRank = myRank - paramv[1]; //ASSERT(dstRank >= 0 && dstRank < numRanks); ocrSend(dstRank, 1, depv[0].guid); }return depv[0].guid;}

a spmd model for ocr (with collectives) sanjay chatterjee 2/9/2015 intel confidential1

Documents