data acquisition backbone core j. adamczewski-musch, n. kurz, s. linev gsi, experiment electronics,...
TRANSCRIPT
Data Acquisition Backbone Core
J. Adamczewski-Musch, N. Kurz, S. Linev
GSI, Experiment Electronics, Data processing group
9.02.2011
DABC
S. Linev http://dabc.gsi.de 2
DAQ architectures
Detector
FEE buffer
Readoutbuffer
Switch
Processorfarm
Storage
Self-triggered Front-endall hits shipped to DAQ.Data push architecture
High-throughputEvent building
First event selectiondone in processor farm.
Readout buffer outside radiation area. Many Gbyte
storage easily possible. Allows L1 decision times of
10-100 ms
Fast links
Conventional DAQ
L1trigger
L1trigger
L1trigger
L1trigger
HLTHLTHLTHLTHLTHLTHLTL1
CBM DAQ
9.02.2011
DABC
S. Linev http://dabc.gsi.de 3
InfiniBand performance studies
0 1 2 3 4
Nominal QDR
Unidir. QDR
Bidir. QDR
Nominal DDR
Unidir. DDR
Bidir. DDR
Nominal SDR
Unidir. SDR
Bidir. SDR
GByte/s
2006*
2007**
2011***
Measurements: * GSI, 4-nodes SDR IB cluster
** Uni Mainz, 120-nodes DDR IB cluster
*** LOEWE-CSC, Höchst, 836-nodes QDR IB cluster (preliminary)
9.02.2011
DABC
S. Linev http://dabc.gsi.de 4
DAQ requirements
Flexible – adopt different kinds of soft- and hardware
Compact – use only necessary code
Scalable – from small detector tests to many-nodes clusters
Performing – low framework overhead
Multiprocessing & multithreading
9.02.2011
DABC
S. Linev http://dabc.gsi.de 5
Data Acquisition Backbone Core
Controls
core-testnet-testbnet-testmbsbnet-mbs
Slim(batch)
DIM
Java GUI
ApplicationsPlugins
socketIB verbsbnetmbsbnet-mbs
Core
Plugins: Implementation of applications (programmers)
Applications: Mainly setup or testing programs (users)
DABC
Download via http://dabc.gsi.de
9.02.2011
DABC
S. Linev http://dabc.gsi.de 6
DABC dataflow concept
UserModule
“Pool”
“Input”
“Output0”
“Output1”
File
UserModule
“Pool”
“Input”
“Output0”
“Output1”
UserModule
“Pool”
“Input”
“Output0”
“Output1”
File
DataServer
NetworkDevice
NetworkDevice
PCIeDevice
node1
node2
Transport
Transport
Transport
Transport
Transport
Transport
Transport
Transport:• manages buffers queue• runs in own thread• decouples user code from actual transport functionality
Device:• represents hardware items• manages several transports
9.02.2011
DABC
S. Linev http://dabc.gsi.de 7
User code - modules
dabc::Module
dabc::PoolHandle
dabc::Timer
dabc::Port
dabc::Port
dabc::Port
dabc::Parameter dabc::Module provides:• I/O ports for communications
• Pools handles to request memory
• Timers for timeouts processing
• Configuration & monitoring parameters
9.02.2011
DABC
S. Linev http://dabc.gsi.de 8
Many other features
Configuration with XML files
Controlling interface and GUI
Factories (plugins) architecture
Application and state machine
Event building network (BNET)
Runs on 32/64 bit Linux
9.02.2011
DABC
S. Linev http://dabc.gsi.de 9
CBM hardware prototypes
nXYTER, 128-channel self-triggered
readout chip with few ns time
resolution, DEFNI collaboration
CBM Readout controller (ROC),
Kirchhoff Institut für Physik, Heidelberg
9.02.2011
DABC
S. Linev http://dabc.gsi.de 10
DABC as access layer to ROC
file I/O online monitor
ROC-udpplugin
ROC-opticplugin
Use
r acc
ess
la
yer
DABCUsers
scripts, GUIs
optic
PCIe
Ethernet
9.02.2011
DABC
S. Linev http://dabc.gsi.de
ROC software components
DABC-based components roc-udp plugin roc-optic plugin combiner and sorter modules readout application
rocGui and rocutil for configuration
go4-based online analysis
https://subversion.gsi.de/cbm/ROC
9.02.2011
DABC
S. Linev http://dabc.gsi.de
CBM beam test in CERN, Nov.10
ROCEth
SYNC
A
B ROCEth
SYNC
A
B
ROCEth
SYNC
A
B
STS
stati
on
RIC
H
MBSEth
SYNCsender
SusiboUSB
SYNCSPA
DIC
SusiboUSB
SYNCSPA
DIC
SusiboUSB
SYNCSPA
DIC
SusiboUSB
SYNCSPA
DIC
SusiboUSB
SYNCSPA
DIC
SusiboUSB
SYNCSPA
DIC
TR
D1
TR
D2
TR
D3
TR
D4
Beam
monit
or
PC
DABC
Eth
USB
Control room
storagego4 online monitor
• Three different readouts:• ROC UDP-based readout• SPADIC/Susibo USB readout• MBS readout
• Combination of free-running and triggered electronics
Cave
9.02.2011
DABC
S. Linev http://dabc.gsi.de
node3
node2
node1
DABC application
SPADICcombiner
Input1
OutputInput0
Input2
Input3
Input4
Input5
Supercombiner
MBS
Output0ROC
SPADIC
Output1
ROCcombiner
Input1
OutputInput0
Input2go4 online
monitor
lmd filestorage
MBS
RO
Cs
SPA
DIC
s/Susi
bo
9.02.2011
DABC
S. Linev http://dabc.gsi.de
CBM beam test in COSY, Dec.10
ROCSFP
SYNC
A
BData
combinerboard
SFP
SFP
SFP
SFP
SFP
SFP
ROCSFP
SYNC
A
B
ROCSFP
SYNC
A
B ROCSFP
SYNC
A
B
ROCSFP
SYNC
A
B ROCSFP
SYNC
A
B
ROCSFP
SYNC
A
B ROCSFP
SYNC
A
B
Datacombiner
boardSFP
SFP
SFP
SFP
SFP
SFP
STS
stati
on 1
STS
stati
on 2
GEM
stati
on 1
GEM
stati
on 2
MBSEth
SYNCsender
Beam
monit
or
PC
DABC
AVNet
PCIeSFP
SFP
Eth
PC
DABC/ EPICS
AVNet
PCIeSFP
SFP
Eth
• data readout and control via optic• clock synchronization via optic• optional MBS readout• optional EPICS readout
Cave Control room20 m
9.02.2011
DABC
S. Linev http://dabc.gsi.de
DABC application
Supercombiner
MBS
Output0ROC
EPICS
Output1
go4 onlinemonitor
Lmd filestorage
MBS
RO
Cs
ROCcombiner
Input1
OutputInput0
Input2
Input3
Input4
Input5
Input6
Input7
DABC/EPICS
ROC datasplitter
Output1
Input Output0
Output2
Output3
Output4
Output5
Output6
Output7
opticreadout
9.02.2011
DABC
S. Linev http://dabc.gsi.de
Plans for version 2
More scalability and flexibility
Failure detection and recovery
Redesign of control interface
9.02.2011
DABC
S. Linev http://dabc.gsi.de 17
Conclusion
DABC is general-purpose DAQ framework
Can be used for different purposes – from small detectors tests to multi-nodes application
Because of its plugin architecture can be easy extended to specific needs
Open for improvements and new ideas