na49-future meeting, january 26, 20071ervin dénes, kfki - rmki date the daq s/w for alice...

NA49-future Meeting, January 26, 2007 1 Ervin Dénes, KFKI - RMKI

DATE

the DAQ s/w for ALICE (Birmingham, Budapest, CERN, Istanbul, Mexico, Split, Zagreb collaboration)

and its possible application at NA49-future


ALICE Experiment

Counting

rooms

Cavern

Computing

Center

DDLs

10Gb Ethernet

Machine: LHC @ CERN, Point 2ALICE detectors: 17Online Systems: ECS, DAQ, TRG, HLT, DCS Readout: ~400 DDLs @ 2125 MbpsStorage rate: max. 1.25 GB/sEvents: 2.5 MB (p-p), 87 MB (Pb-Pb)DAQ Software: DATE V5

DAQ


GDC GDCGDCGDC

ALICE DAQ architecture

CTP

LTU

TTC

FERO FERO

LTU

TTC

FERO FERO

LDCLDC

BUSY BUSY

Rare/All

Event Fragment

Sub-event

Event

File

Storage Network

TDS

PDS

L0, L1a, L2

L0, L1a, L2

262 DDLs

EDM

LDCLoad Bal. LDC LDC

HLT Farm

FEPFEP

DDL

H-RORC

10 DDLs

10 D-RORC

10 HLT LDC

123 DDLs

TDS

DS DS

Event Building Network

329 D-RORC

175 Detector LDC

50 GDC25 TDS

5 DSS

4NA49-future Meeting, January 26, 2007 Ervin Dénes, KFKI - RMKI

TriggerLevel 0,1

TriggerLevel 2

High-LevelTrigger

Decision

Decision

Decision

Data

ALICE online logical modelDetector

Digitizers

Front-end Pipeline/Buffer

Decision

Readout Buffer

Decision

Subevent Buffer

Event Buffer

Storage

Local Data Concentrator

(LDC)

Global DataCollector (GDC)

Data transfer

Detector DataLink (DDL)

DAQ Read-Out Receiver Card

(D-RORC)

ALICETerminology

Transient Data

Storage (TDS)

Permanent Data

Storage (PDS)

Decision

Event Building


ALICE Detector Data Link


LDC

DDL architecture

PCI Bus

Front-End Read-Out

DAQ Read-out Receiver

Card (D-RORC)

SourceInterface

Unit

ForwardChannel

(Raw data)

BackwardChannel

(Pedestals, control)

DestinationInterface

Unit

Detector Data Link (DDL) :- Source Interface Unit- Transmission media- Destination Interface Unit

Standarddetector/DAQ interface

100 MHz64-bit

PCI card


0,00

50,00

100,00

150,00

200,00

250,00

300,00

10 100 1000 10000 100000 1000000

Block size (32-bit words)

Ba

nd

wid

th (

MB

/s)

DDLD-RORC

DAQ Read-Out Receiver Cards (D-RORC)

• PCI-X adapter for 2 DDL

• PCI master:autonomous DMA


Readout System Performance

• Motherboard with dual Xeon CPUs @ 2.4 GHz

• Six PCI-X slots, 4 bus segments (3+1+1+1), 2 controllers

• Linux OS

• ALICE Data-Acquisition software (DATE)


Performance: 6 D-RORCs

• Testing the fully populated PC using data source internal to PCI interface

• Interoperability test

• Measure the maximal input bandwidth

PCI #6

PCI #5

PCI #4

PCI #3

PCI #2

PCI #1Seg

men

t #1

#2#3

#4

Con

trol

ler

#1#2

1 Ch 1 Ch

1 Ch 1 Ch 1 Ch

1 Ch 1 Ch 1 Ch 1 Ch

1 Ch 1 Ch

1 Ch 1 Ch 1 Ch

1 Ch 1 Ch 1 Ch 1 Ch 1Ch 1 Ch 1 Ch

264 464 424 528 792 1045 840

264 232 141.3 264 264 261.3 140

Aggregate Bandwidth [MB/s]Aggregate Bandwidth [MB/s]

Normalized Bandwidth Normalized Bandwidth [MB/s/Ch][MB/s/Ch]


DAQ software

• DAQ software• DDL software we can freely use

• DAQ framework (DATE) does we need it?

• Performance Monitoring (AFFAIR)

• Data quality monitoring (MOOD)

• Information dissemination• Documentation: 4 DAQ sw packages + ECS: fully documented. User’s guide

released and printed

• Regular DAQ trainings (70 people in last 3 years)

• Linux• Linux SLC4 now.

• Process defined to produce, distribute, install new versions• Code management system: CVS. Release packaging and distribution: RPM (~20

MB)

• Automatic installation of Linux and DATE on DAQ nodes


RORCRORC

Memory Management’s Principle

FirmwareFirmware

datalength

transferstatus

Ready FIFOReady FIFO

Transfer status possible values:ffffffffffffffff unloaded (set by sw)0000000000000000 loaded, no DTSTW (set by fw)elseelse loaded, DTSTW (set by fw)

Free FIFOFree FIFO

blocksize

Index ofReady FIFO

startaddress

Page aligned continuous user memoryoutside Linux’s memory space,reachable for DMA and user as well.(physmem module necessary)

Free blocks for events or event fragments

Free FIFO / Ready FIFO depth: Free FIFO / Ready FIFO depth: 128128

Maximum block size: Maximum block size: 2Mbyte2Mbyte


The Free FIFO

D-RORC PC memory bankFirmware

readout

page address

page address

page address

Free FIFO

PC CPU

Allocation of free pages


Direct Memory Access

D-RORCFirmware

PC memory bank

DDL

No involvement

PC CPU


The Ready FIFO

D-RORC PC memory bank

readout

DDL

Ready FIFOFirmware

addresspage status

addresspage status

addresspage status

Delivery of filled pages

PC CPU


DDL Software• All functions accessible as

interactive commands or API• Script-based interpreter for

sequence of operations:• Sending command to the FEE• Reading FEE status

• printing the status• comparing the status• polling the status

• Downloading data into the FEE from a file• Reading data from the FEE

• writing data into a file• comparing data with data in a file

• TPC configuration: < 0.3 s• DDL performance• Parallelism

FEROFERO

DDL

define pedestal_addr 0x1FFF

define enable_pedestal 0x2C

reset SIU

write_command enable_pedestal

write_block pedestal_addr pedestal.hex %x

read_and_check_block pedestal_addr pedestal.hex %x

define pedestal_addr 0x1FFF

define enable_pedestal 0x2C

reset SIU

write_command enable_pedestal

write_block pedestal_addr pedestal.hex %x

read_and_check_block pedestal_addr pedestal.hex %x

D-RORCD-RORC

LDCLDC


DATE Run Control

•Control• Configuration

• Main parameters

•Display• SMI State of all nodes

• Main counters of all nodes

•Compatible with ALICE ECS


DATE Configuration Database

•Database content• DATE Roles

Actors of DATE system:LDCs, GDCs…

• TriggerTrigger masks

• DetectorsFront-end equipment of LDCs

• Event building controlEvent building rules

• BanksMemory banks to operate DATE


DATE Infologger (1/2)

•Online view• Online selection of visible fields

• Online selection of message displayed


DATE Infologger (2/2)

•Offline view• Queries in the message archives

• Selection of visible fields and of message displayed


Data quality monitoring: MOOD

• MOOD: Monitoring Of Online Data

• DATE + ROOT environments

• MOOD framework• Interfaces to detector code

• Applications:• Raw data integrity

• Detector performance

http://ph-dep-aid.web.cern.ch/ph-dep-aid/MOOD/Gallery/ssd3.png

http://ph-dep-aid.web.cern.ch/ph-dep-aid/MOOD/Galleri/mood_1.png


Performance monitoring: AFFAIR

• System view / Aggregated performances• Event building bandwidth

• Event numbers

• Individual nodes view• CPU usage

• Input/Ouput

• Node status


Event Building Switch

• ALICE baseline:• TCP/IP over switched Ethernet

• Computing Data Challenges

• Event-Building Switch• CERN frame contracts

• Needs of IT and experiments

• Good prices

• On site maintenance (company+IT)

• Force 10 Model 1200


Qualification of event-building switch


Storage Arrays Performance

• Local disk buffer at Pt2

• Storage network:

• Fibre Channel currently FC 2G or 4G(Brocade,QLogic)

• 16 ports switches with 4 uplinks for interconnect


Storage Arrays Performance• Transient Data Storage

• Storage arrays(e.g. Infortrend A16F 2 FC 2G ports, 16 SATA II HD)

• Performance for a RAID 5 set of 5 disks


DDL and DATE for NA49-future? (1/2)

•Data transfer• DDL and D-RORC produced and used for longtime

• Match speed requirements

• Few (< 5) PCs are enough for the task

• We have the knowledge how to implement to NA49-future• About 1 man-year of work

• Problems with sub-detectors using VME• In ALICE exist VME boards where DDL SIU cards are attached,

but interfaces are specific to the given ALICE sub-detector

• Or the design VME – DDL interface is necessary for TOF and CAMACs

• Interconnect with the trigger system• Trigger “box” has to be designed


DDL and DATE for NA49-future? (2/2)

•DAQ software• Software (DATE V5) released and documented.

Process defined to produce, distribute, install new versions.

• Everything (ECS, monitoring, etc.) included

• Can we get it from ALICE? • They give it only with support

• They can not support us during this year(integration with 17 ALICE sub-detectors)

• Modifications necessary if TOF and CAMACs are used without the DDL

• Is it shooting at sparrows with a high caliber gun ?

•DAQ fabric hardware• Event building switch is necessary

• Simple storage network has to be added


Thank you


Detector Data Link (DDL)

• Detector readout: fast data transfer to PC memory• Electronics configuration: pedestals download• Interface and data-transfer detector/DAQ• Radiation tolerant card

All components are radiation tolerant including FPGA (ACTEL)

• 10 year doses in ALICE (at TPC inner radius): total ionising dose 1.6 krad neutron fluence 3.9·1011 cm-2

charged hadron fluence 8·109 cm-2 • ACTEL ProASIC adopted as baseline


DATE V5 SoftwareALICE Data Acquisition and Test

Environment (DATE): software framework for the ALICE DAQ system

Latest DATE version V5: DATE V5.x kits:

RPM package (~20 MB), User’s Guide (~470 pages)

Platforms:IA32 family, SLC4 using a 2.6 kernel

Development:~20 packages, mostly in C, using CVS

Auxiliary Software:DIM, SMI, MySQL, Tcl/Tk, libshift, ROOT, VMEbus driver

Testing:reference system, on-line data challenges, test beams

DATE V5

Packages

na49-future meeting, january 26, 20071ervin dénes, kfki - rmki date the daq s/w for alice...

Documents

pci cardna49future meeting

date v5daqna49future

alice birmingham

alice experiment

alice detectors

pci core v6

ddl pci master

pci core v5