Hardware Acceleration of Barrier Communicationfor Large Scale Parallel Computer

Pang Zhengbin, Wang Shaogang, Wu Dan, Lu PingjingSchool of Computer, National University of Defense Technology

Changsha, Hunan, China, 410073zbpang@163.com, {wshaogang,daisydanwu,lupingjing}@nudt.edu.cn

Abstract—MPI collective communication overhead dominatesthe communication cost for large scale parallel computers,scalability and operation latency for collective communicationis critical for next generation computers. This paper proposes afast and scalable barrier communication offload approach whichsupports millions of compute cores. Following our approach, thebarrier operation sequence is packed by host MPI driver intothe barrier "descriptor", which is pushed to the NIC (Network-Interfaces). The NIC can complete the barrier automaticallyfollowing its algorithm descriptor. Our approach acceleratesboth intra-node and inter-node barrier communication. Weshow that our approach achieves both barrier performanceand scalability, especially for large scale computer system. Thispaper also proposes an extendable and easy-to-implement NICarchitecture supporting barrier offload communication and alsoother communication pattern.

I. INTRODUCTION

Collective communication (barrier, broadcast, reduce, all-to-all) is very important for scientific applications running onparallel computers, it has been shown that the collective com-munication overhead could take over 80% communication costfor large scale super computers[1]. The barrier operation is themost common used collective communication, its performanceis critical for most MPI parallel applications. In this paper, wefocus on the implementation of fast barrier for large scaleparallel systems.

For next generation exascale computers, the system couldhave over 1 million cores, a good barrier implementationshould achieve both low latency and scalability[2]. In order toachieve the overlapping of communication and computation,offload the collective communication to hardware could haveobvious benefits for these systems. Present barrier offload tech-nique, like Core-Direct [3], TIANHE-1A[4], uses a triggeredpoint-to-point communication approach, Core-Direct softwareinitiates multiple point-to-point communication requests tothe hardware and sets the request to be triggered by othermessages, in this way, the whole collective communication canbe handled by hardware without further software intervention.

We observe that present barrier offload method may sufferfrom long delay and poor scalability. The Core-Direct mustpush many working-queue-element for a single barrier oper-ation in each node, e.g., for barrier group with 4096 nodes,each node needs to push 12 work requests to the hardware[5],we observe that this incurs long host-NIC (Network-Interface-Card) communication.

For next-generation computer networks, its point-to-pointcommunication delay is usually high as its topology usuallyuses the torus mode. But each chip’s network bandwidth ishigh due to the technology advances in serdes. In this paper,we propose a new barrier communication offload approach,which fits well for the next generation system networking.

For next generation super computer, the processor usuallyincorporates many cores, so each NIC must support moreMPI threads. We see that the NIC is difficult to supportthe communication requirement for too many threads. In thispaper, we leverage a hierarchical approach which the inter-processor threads perform the barrier through dedicate NIChardware, and the node leader thread communication withother nodes through inter-node barrier algorithm which isexecuted fully by hardware.

Compared with other approach, our approach only requiresthe host to push 1 communication descriptor to the NIC in eachnode. The barrier descriptor covers both intra-node and inter-node barrier algorithm. The hardware can follow the descriptorto automatically finish the full barrier algorithm. We also givethe NIC hardware architecture that smoothly supports the newbarrier offload approach, due to the simple barrier enginearchitecture, the NIC can dedicate more hardware resources tocollective communication. From simulation results, we showthat our approach performs better than present barrier offloadtechnique.

II. BARRIER OFFLOAD ALGORITHM

Our approach offloads the MPI barrier operation throughthe following steps:

step1: each barrier node’s MPI driver calculates the barriercommunication sequence,i.e, the communication pattern per-formed by the barrier algorithm, and packs it into the barrierdescriptor. All descriptor is packed following the enhanceddissemination algorithm, while the host does not perform anyreal communication during this step.

step2: the descriptor is sent to the NIC, our approachenables the NIC to complete the real barrier communicationautomatically without any further host intervention, the barriercommunication is performed solely by NIC hardware.

step3: when the NIC completes the whole barrier commu-nication, it informs the host through the host-NIC communi-cation.

2013 8th International Conference on Communications and Networking in China (CHINACOM)

978-1-4799-1406-7 © 2013 IEEE610

N ←number of barrier nodesrank ← my local rankround← −1repeat

round← round+ 1sendpeer1← rank + 3round mod psendpeer2← rank − (3round + p) mod precvpeer1← rank + 3round mod psendpeer1← rank − (3round + p) mod psend barrier msg to sendpeer1 with round idsend barrier msg to sendpeer2 with round idreceiv barrier msg from recvpeer1 with round idreceiv barrier msg from recvpeer2 with round id

until round ≥ log(3, N)− 1

Fig. 1. 2-way dissemination algorithm

A. Inter-Node Barrier Algorithm

The dissemination algorithm is a common used barriermethod[6], [7]. It supports the barrier group with arbitrarynumber of nodes. The basic dissemination barrier algorithmrequires multiple rounds, each round sends and receives onebarrier message from other node. The following round com-munication can be initiated only after the previous round hasbeen finished. We observe that in large scale systems, thenetwork system usually uses the torus topology, the networkpoint-to-point communication delay is high as it may requiremany hops to reach the dest node. For these systems, the basicdissemination algorithm is not efficient. In most cases, everynode takes long time waiting for the source barrier messagein each round.

To efficiently hide the barrier message delay, our NIChardware uses an enhanced K-way dissemination algorithmto offload the barrier communication. The modified algorithmis able to sends and receives K messages parallel in eachround. Our approach defines a new message type which is usedfor solely barrier communication. The new barrier messageis very small, so even the NIC does not support multi-portsparallel message processing, the barrier messages can be sentand received very fast. The example 2-way disseminationalgorithm is shown in 1.

We can prove that the 2-way dissemination algorithm re-quires total log(3, N) rounds to complete the barrier for Nnodes. The obvious benefit of the new algorithm is that it cangreatly reduce the number of communication rounds, e.g., forthe 2-way dissemination algorithm, the algorithm rounds canbe reduced from log(2, N) to log(3, N), we observe that thewhole barrier delay can benefit from less algorithm rounds.

B. Inter-Node Barrier Algorithm for Fat Tree

For super computer which uses the fat tree interconnection,the topology is fast on the broadcast operation. For example,the TIANHE super computer, each board is equipped with 8NICs, and all the on-board NICs are connected to one NR,

NIC NIC NIC NIC

NR

Fig. 2. the NIC architecture supporting collective communication

Counter 0

Inc by 1

Thread 0

Counter 1

Inc by 1

Thread 1

Counter 2

Inc by 1

Thread 2

Counter 3

Inc by 1

Thread 3

Counter n

Inc by 1

Thread n

Barrier Group

Counter

Group Config

Register

Barrier Group 0

Fig. 3. the NIC architecture supporting collective communication

which is a fat tree router chip. We see that, for the fat treetopology, the basic scatter-broadcast barrier algorithm is fasterthan pair-wise algorithm.

We leverages the scatter-broadcast approach to perform thebarrier communication within the board. Each board selects aleader NIC, which all on-board NICs send barrier notificationto it. The leader NIC is then communication with other leaderNIC to finish the whole system barrier operation. On receivingall the notification messages, the leader NIC broadcasts thebarrier reach notification to all other NICs.

C. Intra-Node Barrier Algorithm

For the MPI threads that are resident on the same processor,We use a fence counter to accelerate the barrier operation.After intra-node MPI threads has finished the communication,our approach select a leader thread, through which it commu-nicates with other MPI leader thread in other nodes.

To accelerate intra-node barrier, the NIC hardware incorpo-rates several groups of fence counter, each group is used tosupport one MPI communicator. Within the group, there areseveral counters, each counter is bonded with one MPI thread,when the thread reaches the barrier, it host driver increasesits own fence counter, and the group counter which is thesmallest value of thread counter, when the group counter isincreased by 1, the MPI threads that resides in its host processall reach the barrier point. The group config register is used todesignate which thread is participated in the barrier operation.The barrier group is assigned to a MPI communicator whenit is created by MPI driver, and the group counter is reset toinitial value when the group is re-assigned.

III. BARRIER ALGORITHM DESCRIPTOR

When offloading the collective communication to the NIChardware, there is one approach that offloads the full collec-tive algorithm to the hardware. For example, the collectiveoptimization over Infiniband[8] uses an embedded processorto execute the algorithm. We observe that this approach will

611

greatly complicate the NIC design. The embedded processoris usually limited by its performance, it is far slow comparedwith NIC’s bandwidth and the host processor’s performance.

We propose a new approach that does not require thehardware to execute the full barrier algorithm, instead, thebarrier’s communication sequence is calculated by host’s MPIdriver, and the hardware simply follows the operation sequenceto handle the real communication. We see that this will leadto simple hardware design, through which more hardware canbe dedicated to the real collective communication.

For any node in the barrier group, we can see from thedissemination algorithm that even before real communication,each round’s source and dest nodes can be statically deter-mined. Our approach leverages each node’s MPI driver tocalculate the barrier sequences and pack it to the algorithmdescriptor. After the descriptor is generated, it is pushed tothe NIC through its host interface. The NIC hardware canfollow the descriptor to automatically communicate with othernodes, after the sequence is completed, the whole barrier iscompleted. The host interface may be varied for differentsystems, for example, the command queue residents in thehost memory, or the descriptor is directly written to NIC on-chip RAM through PCIE write command.

An example structure of barrier descriptor supporting 2-way dissemination algorithm is shown in figure 4. The DTypefield indicates descriptor type, along with the barrier de-scriptor, the system may support other collective or point-to-point communication type, the following descriptor fieldis interpreted according to its descriptor type. the BID isa system wide barrier ID, and it is predefined when thecommunication group is created, each node’s barrier descriptorfor the same barrier group uses the same barrier ID. Thebarrier message uses the BID to match the barrier descriptorfor the target node. The SendVec field is a bit vector, its widthequals the maximum barrier algorithm round, and each bitindicates whether the corresponding round should send outbarrier messages to the target node. The RC1, RC2,.. RC16shows the number of barrier message in each algorithm roundit should received. only after receiving all the source barriermessages and sending out all the barrier messages, the NICbarrier engine proceeds to the next algorithm round. SendPeerindicates the target node ID for each communication round.

The S flag is used to synchronize multiple barrier descriptor,when the S flag is set, it waits for previous descriptors to becompleted before issuing to the barrier processing hardware.The M flag is used to mark current NIC to be the barrierleader, this flag is used when the underlying topology is fattree. The V flag is used to indicate whether to perform the fattree topology optimization. The BVEC flag is used to indicatewhich thread is participating the barrier communication usingthe fence counter approach. When BVEC is set to all zero, thefence counter optimization is disabled.

We see that this should be easy to support the next-generation systems. The host-NIC communication cost is lowas it only requires each node to push 1 descriptor to the node.The barrier descriptor is small compared with most standard

DType BarrierID SendVec RESV S V M BVEC

8920203

63~0

RC19

2

RC20

2

RC18

2

RC17

2

RC15

2

RC16

2

RC14

2

RC13

2

RC12

2

RC11

2

RC9

2

RC10

2

RC8

2

RC7

2

RC5

2

RC6

2

RC4

2

RC3

2

RC2

2

RC1

2

RESV

24

DST

18

RID

5

DST

18

RID

5

RESV

18

127~64

191~128

DST

18

RID

5

DST

18

RID

5

RESV

18

255~192

DST

18

RID

5

DST

18

RID

5

RESV

18

319~256

DST

18

RID

5

DST

18

RID

5

RESV

18

383~320

DST

18

RID

5

DST

18

RID

5

RESV

18

511~384

Fig. 4. example descriptor structure for 2-way dissemination algorithm

point-to-point message descriptor, for example, the TIANHE-1A computer’s MP (Message Passing) descriptor has 1024bits[4].

Each node has its own barrier descriptor, and it shouldbe generated completely through node’s local information.The target and source rank id for each barrier round can beeasy generated if local process rank and the barrier groupsize is known. But for the NIC hardware to perform thereal communication, the NIC should know the target andsource node id, our approach leverages the MPI driver totranslate the process rank id to the physical node id. Thetranslation process should be handled by local node withoutany communication, so our approach requires that the rank tonode mapping information is saved in each node’s memorywhen the MPI communicator group is created.

The barrier message takes the information on BID, RID,DestID to the dest node. Through these information, the destnode can easily determine which point it has reached for thealgorithm. If the dest node has not reached the barrier, thebarrier messages are saved in temporal buffer and wait for thedest node’s own descriptor to be pushed to the NIC. When theNIC has completed the sequence defined in the descriptor, thegroup barrier communication is finished. It then informs thehost that all other nodes have reached the barrier.

IV. THE HARDWARE IMPLEMENTATION

A. Barrier Engine Architecture

For the barrier communication, a complicated case is todeal with different processing arriving patterns. The timingdifference between collective communication group nodes canhave a significant impact on the performance of the operation,it requires the hardware to be carefully designed to avoidperformance degression.

To handle this problem, BE leverages the DAMQ(Dynamically-Allocated Multi Queue)[10] to hold incomingbarrier messages. The packets stored in this queue can beprocessed out of order. If the barrier reaches the target nodewhose has not reached the barrier, BE saves the packets in theDAMQ buffer.

We use a simple barrier message handshake protocol tobarrier between two nodes. In the barrier descriptor, if therecvpeer is valid, this indicates that local node should wait

612

the source node to reach barrier; if the sendpeer is valid,this indicates that local node should tell the dest node thatit reaches the barrier. If the barrier messages from sendpeerreaches the target node, but the target node has not reached thebarrier, the barrier messages are saved in a DAMQ buffer, thenthe target barrier engine sends back the BarrierRsp message.On receiving the BarrierRsp message, the source node knowsthat the target node is sure to get the message. When DAMQbuffer is full, the target node nacks the source node withBarrierNack message, on receiving this message, the sourcenode will resend the barrier message after a predefined delay.

When the barrier descriptor reaches the NIC, the barrierengine will first check its local DAMQ buffer to see if thereare any previously reached barrier messages. If there are any,BE processes these messages immediately.

Note that each barrier message takes the information on thebarrioud id BID, through which it is matched with dest node’sdescriptor. If the message’s BID equals the descriptor BID, thesource node and target node are from the same barrier group.The BID could be derived from the MPI communicator groupid, it is required that all the nodes on the same barrier groupagree on the BID. This BE engine supports multi barrier runin parallel, with each barrier uses different ID.

The structure of the barrier engine is shown in figure5. The logic is separated by the barrier message sending(TE) and receiving module (SE). The SDQ and HDQ aredescriptor queue. SDQ is resident in host memory, and HDQis resident in on-chip RAM. The HDQ is mainly used for fastcommunication and fully controlled by host MPI driver. TheOF is the fetching module which reads from descriptor queueand dispatches the descriptor to the barrier engine.

The SE module is responsible for receiving network barriermessage that comes from sendpeer. The barrier engine savesthe messages in the DAMQ buffer; To reduce hardware re-quirement, all barrier group messages are saved in one buffer,and the DAMQ buffer can be handled out of order. Whenthe receiving DAMQ receives one message, it directly sendsback the rsp reply to the sender. For local node, it does notneed to known the barrier message is from which node, so thebarrier descriptor only holds the number of messages it shouldreceived. The SE module uses a booking table to holds thenumber of source barrier messages for each round. Becauseeach node may reach the barrier in the arbitrary sequence,the receiving messages may reaches local node out of order,so current receiving round id is RID, where from round 1 toRID-1, all barrier messages have received.

The TE module is responsible for sending barrier messagesto target nodes following the sequence defined in the descrip-tor. For algorithm round i, barrier messages to node sendpeerare sent if: the SE module has received all barrier messagesbefore round i, and the sendpeer is valid for round i. Thebarrier messages for current round are sent in pipeline beforetheir responsive messages are received. If target node replieswith nack message, the barrier message is resent after a pre-configured delay.

TE and SE are running in parallel and independently. Be-

DAMQTE

(Target Engine)

SE

(Source Engine)

DAMQ

BarrierRsp

Barrier

BarrierRsp

SDQ

HDQ

Ne

two

rk In

terfa

ce

OFSource Round

Barrier

Fig. 5. barrier engine architecture

0

1

2

3

4

5

6

7

8

4 10 20 40 80 100 128

speedup over software

Fig. 6. NIC barrier offload speedup over software only approach

cause the TE module needs to know current receiving round,SE module directly gives this information through moduleports.

V. EXPERIMENTS

We implemented the barrier engine using the SystemVer-ilog language, and integrated the barrier engine module intoTIANHE-1 NIC’s RTL model. TIANHE-1’s point-to-pointcommunication engine uses the descriptor for MP (Message-Passing) and RDMA (Remote-Direct-Memory-Access)[4], andwe add the new barrier descriptor type. From our experiencesthat the barrier engine is easy to design, we model the barrierengine with less than 6000 SystemVerilog code lines.

The new NIC model is simulated by synopsys VCS simu-lator. We test the barrier latency for different sized barrier. Tosimulate large scale barrier groups, we designed a simplifiedNIC model using SystemVerilog language, the simplifiedmodel requires less simulation resources and runs more fast,yet its processing delay is similar with the real RTL model.

To simulate the network, we use a general model whichroute point-to-point message to the target node, the point-to-point delay is calculated based on the number of hops for the2D torus network.

A. Barrier Delay Compared With Software Only Approach

In this section, we test the average barrier speedup over thesoftware only approach. The software approach is simulatedby modifying the timing parameters collected from the realhardware. The software only barrier delay is compared withour approach, shown in figure 6.

613

0.5

0.6

0.7

0.8

0.9

1

1.1

1 4 10 20 40 80 100 128

one node late

two node late

Fig. 7. barrier communication delay

Our barrier offload approach gets obvious delay reductioncompared with the software only approach, and shows muchbetter performance scalability. For the barrier group with128 nodes, our NIC offload approach is 7.6x faster than thesoftware only approach. The performance benefits come fromthe following reasons:

1) The software approach uses standard MP message forbarrier communication, we see that the MP packet istoo large for barrier communication, it incurs long NICprocessing delay.

2) The host-NIC communication cost is high, it is evenworse than the Core-Direct approach which uses thetriggered point-to-point operations. For each point-to-point message, the host needs to push the MP descriptorto the NIC, and waits for the NIC’s completion events.

3) The offload approach permits more communication andcomputation overlapping, the performance benefits maydepends on the applications.

B. Process Arriving Patterns

Barrier performance is greatly impacted by the processarriving patterns. We give some sample processing arrivingpatterns and shows its impact on total barrier delay. The perfor-mance results is shown in figure 7. The simulation is conductedon the barrier group with 1-node and 2-node arriving late,while other nodes arrive the barrier at the same time. Theperformance is compared with the baseline simulation whenall the barrier nodes reach the barrier at the same time.

We obtain from the test result that the barrier delay is greatlyimpacted by the process arriving pattern, when the barriergroup size grows, its impact is more obvious. This behavior isbecause the barrier algorithm is executed following the roundsequence. if one node does not reach the barrier, it will notsend out the barrier messages to its targets, so all the followingbarrier communication must wait for this node to reach thebarrier.

We compare the results with test results from [12], it hasbeen shown that our approach is less affected by the processarriving pattern. This is because the hardware can resume thebarrier algorithm more quickly and automatically without anyhost intervention, but for the software approach, it must takelong time on host-NIC communication when the late node

reaches the barrier.

VI. CONCLUSION

We propose a new barrier offload approach, with the newhardware-software interfaces, the barrier engine is . The NIChardware follows the descriptor to executes the complex K-way dissemination algorithm. Simulation results show that ourapproach reduces barrier delay efficiently and achieves goodcomputation and communication overlap. From our experi-ences, the barrier engine is easy to implement and requiresless chip resources, so the NIC can dedicate more logicfor real communication, this is important for next-generationsuper computer, where each NIC must support more processorthreads.

VII. ACKNOWLEDGEMENT

This research is sponsored by Natural ScienceFoundation of China(61202124), Chinese 863 project(2013AA014301), Hunan Provincial Natural ScienceFoundation of China(13JJ4007).

REFERENCES

[1] R. Thakur, R. Rabenseifner, and W. Gropp, “Optimization of CollectiveCommunication Operations in MPICH,” International Journal of HighPerformance Computing Applications, vol. 19, no. 1, pp. 49–66, Feb.2005.

[2] H. Miyazaki, Y. Kusano, N. Shinjou, F. Shoji, M. Yokokawa, andT. Watanabe, “Overview of the K computer System,” FUJITSU Sci.Tech. J, vol. 48, no. 3, pp. 255–265, 2012.

[3] M. G. Venkata, R. L. Graham, J. Ladd, and P. Shamis, “Exploring theAll-to-All Collective Optimization Space with ConnectX CORE-Direct,”in 2012 41st International Conference on Parallel Processing (ICPP).IEEE, pp. 289–298.

[4] M. Xie, Y. Lu, L. Liu, H. Cao, and X. Yang, “Implementation andEvaluation of Network Interface and Message Passing Services forTianHe-1A Supercomputer,” in 2011 IEEE 19th Annual Symposium onHigh-Performance Interconnects (HOTI). IEEE, pp. 78–86.

[5] K. S. Hemmert, B. Barrett, and K. D. Underwood, “Using triggered oper-ations to offload collective communication operations,” in EuroMPI’10:Proceedings of the 17th European MPI users’ group meeting conferenceon Recent advances in the message passing interface. Springer-Verlag,Sep. 2010.

[6] T. Hoefler, T. Mehlan, F. Mietke, and W. Rehm, “Fast barrier syn-chronization for InfiniBandTM,” in IPDPS’06: Proceedings of the 20thinternational conference on Parallel and distributed processing. IEEEComputer Society, Apr. 2006.

[7] D. Hensgen, R. Finkel, and U. Manber, “Two algorithms for barrier syn-chronization,” International Journal of Parallel Programming, vol. 17,no. 1, Feb. 1988.

[8] A. R. Mamidala, “Scalable and High Performance Collective Communi-cation for Next Generation Multicore Infiniband Clusters,” Phd Thesis,2008.

[9] F. Sonja, “Hardware Support for Efficient Packet Processing,” PhdThesis, pp. 1–207, Mar. 2012.

[10] Y. Tamir and G. L. Frazier, “Dynamically-allocated multi-queue buffersfor VLSI communication switches,” Computers, IEEE Transactions on,vol. 41, no. 6, pp. 725–737, 1992.

[11] V. Tipparaju, W. Gropp, H. Ritzdorf, R. Thakur, and J. L. Traff, “Inves-tigating High Performance RMA Interfaces for the MPI-3 Standard,” in2009 International Conference on Parallel Processing (ICPP). IEEE,pp. 293–300.

[12] “Efficient Barrier and Allreduce on InfiniBand Clusters using HardwareMulticast and Adaptive Algorithms*,” pp. 1–10, May 2005.

614