lhcb daq protocol independent performance evaluator: daqpipe
DESCRIPTION
LHCb DAQ Protocol Independent Performance Evaluator: DAQPIPE. Daniel Hugo Cámpora Pérez. LHCb DAQ workshop , 27-05-2013. What is it, anyway. DAQPIPE is an evaluation tool designed to measure the performance of a generic DAQ system. EM. Daniel Hugo Cámpora Pérez. - PowerPoint PPT PresentationTRANSCRIPT
LHCb DAQ Protocol Independent Performance
Evaluator: DAQPIPE
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
1
What is it, anyway
● DAQPIPE is an evaluation tool designed to measure the performance of a generic DAQ system.
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
2
EM
What we want to measure
● Network transfer protocols● Network topologies● Event-Building protocols
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
3
Network transfer protocols
● Preliminary tests with TCP/UDP over 40G.● Preparing IB FDR tests.● Testing
– 40G Ethernet UDP– iWarp– ROCE– IB OpenMPI (full hardware support)– IB RDMA
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
4
InfiniBand? Ethernet? Upcoming ones?
Network topologies (recap)
Unidirectional flow
● Simple
● Current solution, scale up
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
5
Network topologies
Bidirectional uniform flow
● Full-Duplex on core – mixed-servers.
● Implementation details: RUs sit on servers.
● Fraction of computing done in receiving nodes, offloading done to FUs.
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
6
Event-Building protocols
● PUSH● PUSH with Barrel Shifting● PULL
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
7
Push
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
8
Barrel shifting
● Equal-length receiving queues, asymmetric sending queues.
● Potential dramatic decrease of congestion in DAQ network.
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
9
Pull
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
10
Let's talk software
● Test suite should be– Portable– Flexible– C-like entry point (interfaceable)– Scalable– Efficient!
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
11
Output
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
● Logs & rates– Effective throughput– Maximum trigger rate– Error rate– Incomplete event rate
12
Preliminary tests
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
● PTP allows for microsecond-level synchronization.
● RDMA over IB CPU consumption is negligible.
13
Preliminary tests
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
● 40GbE
14
Tx Rate CPU Consumption
40G (TCP), eight streams
36 Gb/s 5 – 8%
4 x 10G (TCP)
36 Gb/s
40G (UDP), four streams
32 Gb/s 4%
HP 59002x Mellanox Technologies MT27500 Family ConnectX-3
8x Chelsio Communications Inc T420-CR
Thanks!
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
15
Backup
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
16
Network topologies
Bidirectional mixed flow
● Full-Duplex on core - TOR connections.
● Port-density requirements for core switches significantly reduced.
● Server-wise identical to unidirectional solution.
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
17
Network topologies
Bidirectional uniform flow
● Relatively high load per mixed server (RU/BU/FU) – 4 x rate
– Input – RU
– Input / Output – BU
– Output - FU
Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013
18