na49-future meeting, january 26, 20071ervin dénes, kfki - rmki date the daq s/w for alice...
Post on 14-Jan-2016
216 Views
Preview:
TRANSCRIPT
NA49-future Meeting, January 26, 2007 1 Ervin Dénes, KFKI - RMKI
DATE
the DAQ s/w for ALICE (Birmingham, Budapest, CERN, Istanbul, Mexico, Split, Zagreb collaboration)
and its possible application at NA49-future
NA49-future Meeting, January 26, 2007 2 Ervin Dénes, KFKI - RMKI
ALICE Experiment
Counting
rooms
Cavern
Computing
Center
DDLs
10Gb Ethernet
Machine: LHC @ CERN, Point 2ALICE detectors: 17Online Systems: ECS, DAQ, TRG, HLT, DCS Readout: ~400 DDLs @ 2125 MbpsStorage rate: max. 1.25 GB/sEvents: 2.5 MB (p-p), 87 MB (Pb-Pb)DAQ Software: DATE V5
DAQ
NA49-future Meeting, January 26, 2007 3 Ervin Dénes, KFKI - RMKI
GDC GDCGDCGDC
ALICE DAQ architecture
CTP
LTU
TTC
FERO FERO
LTU
TTC
FERO FERO
LDCLDC
BUSY BUSY
Rare/All
Event Fragment
Sub-event
Event
File
Storage Network
TDS
PDS
L0, L1a, L2
L0, L1a, L2
262 DDLs
EDM
LDCLoad Bal. LDC LDC
HLT Farm
FEPFEP
DDL
H-RORC
10 DDLs
10 D-RORC
10 HLT LDC
123 DDLs
TDS
DS DS
Event Building Network
329 D-RORC
175 Detector LDC
50 GDC25 TDS
5 DSS
4NA49-future Meeting, January 26, 2007 Ervin Dénes, KFKI - RMKI
TriggerLevel 0,1
TriggerLevel 2
High-LevelTrigger
Decision
Decision
Decision
Data
ALICE online logical modelDetector
Digitizers
Front-end Pipeline/Buffer
Decision
Readout Buffer
Decision
Subevent Buffer
Event Buffer
Storage
Local Data Concentrator
(LDC)
Global DataCollector (GDC)
Data transfer
Detector DataLink (DDL)
DAQ Read-Out Receiver Card
(D-RORC)
ALICETerminology
Transient Data
Storage (TDS)
Permanent Data
Storage (PDS)
Decision
Event Building
NA49-future Meeting, January 26, 2007 5 Ervin Dénes, KFKI - RMKI
ALICE Detector Data Link
NA49-future Meeting, January 26, 2007 6 Ervin Dénes, KFKI - RMKI
LDC
DDL architecture
PCI Bus
Front-End Read-Out
DAQ Read-out Receiver
Card (D-RORC)
SourceInterface
Unit
ForwardChannel
(Raw data)
BackwardChannel
(Pedestals, control)
DestinationInterface
Unit
Detector Data Link (DDL) :- Source Interface Unit- Transmission media- Destination Interface Unit
Standarddetector/DAQ interface
100 MHz64-bit
PCI card
NA49-future Meeting, January 26, 2007 7 Ervin Dénes, KFKI - RMKI
0,00
50,00
100,00
150,00
200,00
250,00
300,00
10 100 1000 10000 100000 1000000
Block size (32-bit words)
Ba
nd
wid
th (
MB
/s)
DDLD-RORC
DAQ Read-Out Receiver Cards (D-RORC)
• PCI-X adapter for 2 DDL
• PCI master:autonomous DMA
NA49-future Meeting, January 26, 2007 8 Ervin Dénes, KFKI - RMKI
Readout System Performance
• Motherboard with dual Xeon CPUs @ 2.4 GHz
• Six PCI-X slots, 4 bus segments (3+1+1+1), 2 controllers
• Linux OS
• ALICE Data-Acquisition software (DATE)
NA49-future Meeting, January 26, 2007 9 Ervin Dénes, KFKI - RMKI
Performance: 6 D-RORCs
• Testing the fully populated PC using data source internal to PCI interface
• Interoperability test
• Measure the maximal input bandwidth
PCI #6
PCI #5
PCI #4
PCI #3
PCI #2
PCI #1Seg
men
t #1
#2#3
#4
Con
trol
ler
#1#2
1 Ch 1 Ch
1 Ch 1 Ch 1 Ch
1 Ch 1 Ch 1 Ch 1 Ch
1 Ch 1 Ch
1 Ch 1 Ch 1 Ch
1 Ch 1 Ch 1 Ch 1 Ch 1Ch 1 Ch 1 Ch
264 464 424 528 792 1045 840
264 232 141.3 264 264 261.3 140
Aggregate Bandwidth [MB/s]Aggregate Bandwidth [MB/s]
Normalized Bandwidth Normalized Bandwidth [MB/s/Ch][MB/s/Ch]
NA49-future Meeting, January 26, 2007 10 Ervin Dénes, KFKI - RMKI
DAQ software
• DAQ software• DDL software we can freely use
• DAQ framework (DATE) does we need it?
• Performance Monitoring (AFFAIR)
• Data quality monitoring (MOOD)
• Information dissemination• Documentation: 4 DAQ sw packages + ECS: fully documented. User’s guide
released and printed
• Regular DAQ trainings (70 people in last 3 years)
• Linux• Linux SLC4 now.
• Process defined to produce, distribute, install new versions• Code management system: CVS. Release packaging and distribution: RPM (~20
MB)
• Automatic installation of Linux and DATE on DAQ nodes
NA49-future Meeting, January 26, 2007 11 Ervin Dénes, KFKI - RMKI
RORCRORC
Memory Management’s Principle
FirmwareFirmware
datalength
transferstatus
Ready FIFOReady FIFO
Transfer status possible values:ffffffffffffffff unloaded (set by sw)0000000000000000 loaded, no DTSTW (set by fw)elseelse loaded, DTSTW (set by fw)
Free FIFOFree FIFO
blocksize
Index ofReady FIFO
startaddress
Page aligned continuous user memoryoutside Linux’s memory space,reachable for DMA and user as well.(physmem module necessary)
Free blocks for events or event fragments
Free FIFO / Ready FIFO depth: Free FIFO / Ready FIFO depth: 128128
Maximum block size: Maximum block size: 2Mbyte2Mbyte
NA49-future Meeting, January 26, 2007 12 Ervin Dénes, KFKI - RMKI
The Free FIFO
D-RORC PC memory bankFirmware
readout
page address
page address
page address
Free FIFO
PC CPU
Allocation of free pages
NA49-future Meeting, January 26, 2007 13 Ervin Dénes, KFKI - RMKI
Direct Memory Access
D-RORCFirmware
PC memory bank
DDL
No involvement
PC CPU
NA49-future Meeting, January 26, 2007 14 Ervin Dénes, KFKI - RMKI
The Ready FIFO
D-RORC PC memory bank
readout
DDL
Ready FIFOFirmware
addresspage status
addresspage status
addresspage status
Delivery of filled pages
PC CPU
NA49-future Meeting, January 26, 2007 15 Ervin Dénes, KFKI - RMKI
DDL Software• All functions accessible as
interactive commands or API• Script-based interpreter for
sequence of operations:• Sending command to the FEE• Reading FEE status
• printing the status• comparing the status• polling the status
• Downloading data into the FEE from a file• Reading data from the FEE
• writing data into a file• comparing data with data in a file
• TPC configuration: < 0.3 s• DDL performance• Parallelism
FEROFERO
DDL
define pedestal_addr 0x1FFF
define enable_pedestal 0x2C
reset SIU
write_command enable_pedestal
write_block pedestal_addr pedestal.hex %x
read_and_check_block pedestal_addr pedestal.hex %x
define pedestal_addr 0x1FFF
define enable_pedestal 0x2C
reset SIU
write_command enable_pedestal
write_block pedestal_addr pedestal.hex %x
read_and_check_block pedestal_addr pedestal.hex %x
D-RORCD-RORC
LDCLDC
NA49-future Meeting, January 26, 2007 16 Ervin Dénes, KFKI - RMKI
DATE Run Control
•Control• Configuration
• Main parameters
•Display• SMI State of all nodes
• Main counters of all nodes
•Compatible with ALICE ECS
NA49-future Meeting, January 26, 2007 17 Ervin Dénes, KFKI - RMKI
DATE Configuration Database
•Database content• DATE Roles
Actors of DATE system:LDCs, GDCs…
• TriggerTrigger masks
• DetectorsFront-end equipment of LDCs
• Event building controlEvent building rules
• BanksMemory banks to operate DATE
NA49-future Meeting, January 26, 2007 18 Ervin Dénes, KFKI - RMKI
DATE Infologger (1/2)
•Online view• Online selection of visible fields
• Online selection of message displayed
NA49-future Meeting, January 26, 2007 19 Ervin Dénes, KFKI - RMKI
DATE Infologger (2/2)
•Offline view• Queries in the message archives
• Selection of visible fields and of message displayed
NA49-future Meeting, January 26, 2007 20 Ervin Dénes, KFKI - RMKI
Data quality monitoring: MOOD
• MOOD: Monitoring Of Online Data
• DATE + ROOT environments
• MOOD framework• Interfaces to detector code
• Applications:• Raw data integrity
• Detector performance
NA49-future Meeting, January 26, 2007 21 Ervin Dénes, KFKI - RMKI
Performance monitoring: AFFAIR
• System view / Aggregated performances• Event building bandwidth
• Event numbers
• Individual nodes view• CPU usage
• Input/Ouput
• Node status
NA49-future Meeting, January 26, 2007 22 Ervin Dénes, KFKI - RMKI
Event Building Switch
• ALICE baseline:• TCP/IP over switched Ethernet
• Computing Data Challenges
• Event-Building Switch• CERN frame contracts
• Needs of IT and experiments
• Good prices
• On site maintenance (company+IT)
• Force 10 Model 1200
NA49-future Meeting, January 26, 2007 23 Ervin Dénes, KFKI - RMKI
Qualification of event-building switch
NA49-future Meeting, January 26, 2007 24 Ervin Dénes, KFKI - RMKI
Storage Arrays Performance
• Local disk buffer at Pt2
• Storage network:
• Fibre Channel currently FC 2G or 4G(Brocade,QLogic)
• 16 ports switches with 4 uplinks for interconnect
NA49-future Meeting, January 26, 2007 25 Ervin Dénes, KFKI - RMKI
Storage Arrays Performance• Transient Data Storage
• Storage arrays(e.g. Infortrend A16F 2 FC 2G ports, 16 SATA II HD)
• Performance for a RAID 5 set of 5 disks
NA49-future Meeting, January 26, 2007 26 Ervin Dénes, KFKI - RMKI
DDL and DATE for NA49-future? (1/2)
•Data transfer• DDL and D-RORC produced and used for longtime
• Match speed requirements
• Few (< 5) PCs are enough for the task
• We have the knowledge how to implement to NA49-future• About 1 man-year of work
• Problems with sub-detectors using VME• In ALICE exist VME boards where DDL SIU cards are attached,
but interfaces are specific to the given ALICE sub-detector
• Or the design VME – DDL interface is necessary for TOF and CAMACs
• Interconnect with the trigger system• Trigger “box” has to be designed
NA49-future Meeting, January 26, 2007 27 Ervin Dénes, KFKI - RMKI
DDL and DATE for NA49-future? (2/2)
•DAQ software• Software (DATE V5) released and documented.
Process defined to produce, distribute, install new versions.
• Everything (ECS, monitoring, etc.) included
• Can we get it from ALICE? • They give it only with support
• They can not support us during this year(integration with 17 ALICE sub-detectors)
• Modifications necessary if TOF and CAMACs are used without the DDL
• Is it shooting at sparrows with a high caliber gun ?
•DAQ fabric hardware• Event building switch is necessary
• Simple storage network has to be added
NA49-future Meeting, January 26, 2007 28 Ervin Dénes, KFKI - RMKI
Thank you
NA49-future Meeting, January 26, 2007 29 Ervin Dénes, KFKI - RMKI
Detector Data Link (DDL)
• Detector readout: fast data transfer to PC memory• Electronics configuration: pedestals download• Interface and data-transfer detector/DAQ• Radiation tolerant card
All components are radiation tolerant including FPGA (ACTEL)
• 10 year doses in ALICE (at TPC inner radius): total ionising dose 1.6 krad neutron fluence 3.9·1011 cm-2
charged hadron fluence 8·109 cm-2 • ACTEL ProASIC adopted as baseline
NA49-future Meeting, January 26, 2007 30 Ervin Dénes, KFKI - RMKI
DATE V5 SoftwareALICE Data Acquisition and Test
Environment (DATE): software framework for the ALICE DAQ system
Latest DATE version V5: DATE V5.x kits:
RPM package (~20 MB), User’s Guide (~470 pages)
Platforms:IA32 family, SLC4 using a 2.6 kernel
Development:~20 packages, mostly in C, using CVS
Auxiliary Software:DIM, SMI, MySQL, Tcl/Tk, libshift, ROOT, VMEbus driver
Testing:reference system, on-line data challenges, test beams
DATE V5
Packages
top related