simulations and prototyping of the lhcb l1 and hlt triggers
DESCRIPTION
Simulations and Prototyping of the LHCb L1 and HLT triggers. Tara Shears For A. Barczyk, J.P. Dufey, B. Jost, T. Kechadi, R.McNulty, N. Neufeld, T. Shears On behalf of the LHCb Collaboration. Overview. Introduction to LHCb trigger and data transfer requirements Network simulation studies - PowerPoint PPT PresentationTRANSCRIPT
Simulations and Prototyping of the LHCb
L1 and HLT triggers
Tara Shears
For A. Barczyk, J.P. Dufey, B. Jost, T. Kechadi, R.McNulty, N. Neufeld, T. Shears
On behalf of the LHCb Collaboration
Overview
• Introduction to LHCb trigger and data transfer requirements
• Network simulation studies• Evaluating performance in software• Evaluating performance in hardware• Conclusions
LHCb Trigger System
Must reduce 10MHz visible interaction rate 200 Hz output rate
3 level trigger using hardware + software:L01 MHz (~4 s)L1 40 kHz (~50 ms)HLT 200 Hz
Average event size - L1 / HLT
5 kB / 40 kB
Events per MEP - L1 / HLT
25 / 10
Total bandwidth of readout network
7.1 GB/s
Data sources – L1 / HLT
126 / 323
Data destinations (SFC)
94
At L1, 125kB must be received each 25 s by one of the destinations
Data FlowLevel 0 trigger
1MHz
L1 / HLT Farm
MultiplexingLayer
FE FE FE FE FE FE FE FE FE FE FE FE
Switch Switch
Level-1Traffic
HLTTraffic
126Links
44 kHz5.5 GB/s
323Links4 kHz
1.6 GB/s29 Switches
32 Links
94 SFCs
Front-end Electronics
Gb EthernetLevel-1 Traffic
Mixed TrafficHLT Traffic
94 Links7.1 GB/s
TRM
Sorter
TFCSystem
L1-Decision
StorageSystem
Readout Network
Switch Switch Switch
SFC
Switch
CPUCPU
CPU
SFC
Switch
CPUCPUCPU
SFC
Switch
CPUCPU
CPU
SFC
Switch
CPUCPU
CPU
SFC
Switch
CPUCPU
CPUCPUFarm
62 Switches
64 Links88 kHz
~1800 CPUs
Big disk
Methods of evaluating trigger networks
Software simulation:– Parametrise dataflow, switch response
predicted system behaviour – Two approaches; Custom (models specific
network topology), Ptolemy
Hardware simulation:– Evaluate network / switching response in situ– Can examine microscopic or macroscopic
properties of system
Software Simulation Studies
Objectives:
Simulate performance of switch + cpu network arrangement
Predict data loss, latency, network robustness
Provide facility for testing alternative network arrangements
Approach:
Parametrise data packing / transfer with specified network to study response in custom simulation
Develop alternative, flexible simulation to allow other networks to be studied (Ptolemy)
Simulation; Custom code
Existing custom code:Implemented in C LHCb TDR network
configuration modelled and response studied (LHCb_2003_079)
Network simulated by parametrising data packing / queuing
sources destinationsmultiplexors
Crossbar switches
Custom simulation results
L1 latency for events in subfarm:
• TDR network configuration modelled
• 25 events in a MEP
• Simulated processing times cut off at 50 ms
If processing time cut at 30 ms < 0.5% of events lost.
multiplexors
Simulation; Ptolemy
Ptolemy:Freeware Java based
packageUses ‘actors’, self
contained units that execute simple tasks when called upon
Program then depends upon the way in which these actors are linked together in the graphical editor
Crossbar switches
Ptolemy simulation results
Investigate buffer occupancies:– Change inter-trigger
delays – Change data routing
source – destination– Essentially L1 data flow.
• Results of custom code/Ptolemy identical so far Facility available to test
any network configuration
Custom
Ptolemy
(Example buffer occupancy)
Hardware simulation studies
Switch fabric and logic – hard to simulate
Another approach is to characterise performance in hardware- Can evaluate each network component
separately- Or evaluate performance of given network
configuration (cpu + ethernet link + switch)- Results:
- parameters for input to software simulation- study errors / data loss for LHCb traffic patterns- performance characterisation in own right
Testing switch behaviour
Data source: – PCI card with NP4GS3
network processor, 3 GbE ports per card.
– Programmable data patterns– Up to 24 sources available
Synchronisation:– frame rate: up to ~75kHz– Synchronisation: O(100ns)
Software control:– Use Python interface– RH9 + 2.6.0 kernel
Connect ≥24 port switch to data sources to test switch characteristics
DST
SRC SRC
SW
125MB/s
125MB/s125MB/s
+ …. ( 24)
Results
Check buffer occupancies, switch latencies, packet loss and error rates.
Eg. Buffer occupancy
4MB (as expected) for this switch
Eg. Switch latency
linear, switching time negligible wrt.L1 latency for this switch
Studies ongoing
Large-scale network behaviour
Hardware:
100 3GHz cpu (source or dest.)
500 Mb/s cpu ethernet links
168 port switch (1 Gb/s links)
Software:
Synchronise data sends / receives with MPI
RH9 2.4.24 kernel
Tests:
Measure data transfer rate
Investigate different traffic patterns
500 Mb/s
Results
~LHCb L1 rate
(1 kB sent per source takes 1.25 ms for 50 destinations to receive)
Median receive time for 50 destinations [s]
Source data size [B]
50 sources send synchronously to each destination in turn– Find median time for data to
be received at all 50 destinations
– Each experiment repeated 1000 times
– Different data sizes testedResults:
– Measurements limited by system bandwidth
– Data transfer rates ~2.8 GB/s achieved at LHCb L1 timings (1KB/25s per source)
• Studies ongoing
Conclusions
• Software simulation:– Simulations of LHCb trigger network developed to
evaluate response / performance– LHCb TDR network parametrised and studied in custom
simulation– Alternative Ptolemy-based simulation developed
• Hardware simulation:– Allow simulation of network response where details of
fabric unknown– Testbed developed to analyse switch characteristics – Large scale testbed devised to study general network
response
Backup
Example: Building a switch
• Implementation– Routing– Output queuing– Timing– Buffer occupancy– Scaleable
Banyan Network Topology
Multiplexors Crossbars DestinationsSources