stefano magni i.n.f.n milano italy - nss 2003 pomone, a pci based data acquisition system authors:...

Stefano Magni I.N.F.N Milano Italy - NSS 2003

Pomone, a PCI based data acquisition system

Authors:

G.Alimonti3, G. Chiodini1, B. Hall2, S. Magni3, D.Menasce3, L. Uplegger3, D. Zhang2

1I.N.F.N. Lecce, 2FNAL, 3I.N.F.N. Milano


Project goals

• Validation of the benefits of an Object Oriented design in a small scale project in the BTeV collaboration

• Development of a general purpose DAQ system based on PCI technology suited for test beams and laboratory measurements

• Test of a pixel detectors DAQ in untriggered mode and of an on-line, time-stamp based, event reconstruction algorithm


A PCI Test Adapter (PTA)

plug-in card,compliant with the PCI protocol

Pomone has been designed to meet the requirements of a general Test Stand hardware for testing detectors both in a laboratory environment and at test beam facilities for the BTeV experiment at Fermilab. Current implementation focuses on following hardware components:

A Programmable MezzanineCard (PMC)

An FPIX read-out

chip (ROC)

External datasource

subsystem, aFermilab Pixelreadout chip

(FPIX)

The PMC is intended to work in conjunction with a PTA and serves as an

hardware interface between the the PTA card and the external

detector.

PCI-compliant card featuring:•a programmable Altera FPGA controlling all functions•two RAM banks (1 Mb each)

Hardware components

DAQ computer

Esternal subsystem


PTA working mechanism ● A data driven DAQ: data are produced by an

external subsystem (PMC+ROC) at a variable rate while the host computer receives them at its own rate (depending upon the CPU clock and the processor current activity).

● The PTA FPGA has been programmed to get data from the external subsystem and to act as an intermediate rate-balancing buffer. This isaccomplished by a periodic switching of the data flow between two RAM banks (details later)

● The architecture has been designed to allow for a continuous, lossless sustained data-rate


Software design guidelines

• Components of the system should be loosely coupled: this allows for upgrades and changes of individual elements with little effort. It further insures smoother deployment due to minimal cross dependencies.

• Code must be robust, in principle able to whitstand change of operating system environment, extensive refurbishing and additions of algorithms.

• Code must be highly modular, able to accommodate different detectors, with different hardware and software specifications.

• Since the incoming data rate is asynchronous from the host computer clock, a data-rate compensation buffer must be provided to accommodate for fluctuations in the data rate.

• Functionality of the code must be guaranteed also in environments with minimal resources (eg. no X11graphics is available). The system should be able to perform even without a GUI for user interaction.


... solutions

• Communications between processes has been implemented througth a lightware message bus, based on system V message queues

• To allow for robustness and modularity the object oriented paradigm has been chosen. Particular care has been exercized in the design phase in order to efficiently achieve an optimal decoupling of components.

• Where possible, the system uses already existing (open source) packages or component libraries. Don’t reinvent the wheel !

• In order to accomodate an intermediate data-rate compensation buffer, the system is split in two main processes with a shared memory in the middle: the first process (called producer) gets data out of the PTA card memories placing them in the shared memory, while the second (called consumer) continuously browses the shared memory to fetch completed blocks for the event-builder to assemble hits into events.

Events are finally written by the consumer on an external data-storage.

• System logging is performed by means of a centralized logging utility, collecting messages from different cooperating DAQ processes


Producer

DAQ objects collaboration

PTA card

Hardware Software

Circular buffer

Event builder

Event buffer

Message Bus

Controller Logger

Writer

Sys

tem

m

onit

or

PTA card

PTA card

PTA card

PCI Bus


Read-out process

DAQ time evolution (0/15)• Basic mechanism of operation of the PCI card and the read-out:

Bank0 Bank1

FPGA

0

Interrupt handler

Reset interrupt

Shared memory

Disk stager

ConsumerProcess A

ConsumerProcess B


DAQ time evolution(1/15)• Basic mechanism of operation of the PCI card and the read-out:

Bank0

FPGA

Time0

Interrupt handler

Bank1Read-out process

Shared memory

Reset interruptDisk stager

ConsumerProcess A

ConsumerProcess B



Bank0

FPGA

Time0

Interrupt handler


Shared memory


ConsumerProcess A

ConsumerProcess B



Bank0

FPGA

Time0

Interrupt handler


Shared memory


ConsumerProcess A

ConsumerProcess B

User limit



Bank0

FPGA

Time0

Interrupt handler


Shared memory


ConsumerProcess A

ConsumerProcess B


DAQ time evolution(1315)• Basic mechanism of operation of the PCI card and the read-out:

Bank0

FPGA

Time0

Interrupt handler


Shared memory


ConsumerProcess A

ConsumerProcess B



Bank0

FPGA

Time0

Interrupt handler


Shared memory


ConsumerProcess A

ConsumerProcess B


A data driven (untriggered) DAQ• Hits from pixel detectors are continuously read in untriggered

mode.

• Single hit is tagged by PMC cards with a hardware generated time-stamp.

• An event, defined as “a collection of equal time-stamp hits from different detectors”, is built by the consumer process using a efficient sorting algorithm.

• The DAQ framework allows replacing and adding different event-building algorithms by defining an abstract interface to the appropriate implementation. This allows for different detectors,with different event-building strategies, to be read-out by the same DAQ system.

Frequenze tipiche sono …


Event builder

Event

Builder

Hit Buffer(Unordered data)

Event Buffer (Ordered data)

Time-stamps:

Event


DAQ control• The complete control of the DAQ system is performed through

User Interfaces

• Both a command line and a graphical user interface (based on the QT library, http://www.trolltech.com) have been developed


DAQ monitor• Acquired data and related DAQ information are streamed through

sockets to the network, where client code on remote machines can suitably make use of those informations for histogram and event display. The graphics and histogramming package of choice is ROOT (http://root.cern.ch).

• No burden is placed on the host CPU for monitoring

• An unlimited number of people can remotely monitor the system with high •efficiency


A successful Object Oriented approach

• During the development of the DAQ new versions of read-out chip have been available: thanks to the OO approach and the design of a virtual read-out chip class, inclusion of these new chips, along with additional detectors, became trivial.

• The first version of the system used a commercial general purpose PCI driver.

• After about 2 years of development of the DAQ system we decided to replace the commercial PCI driver to get free of licence issues; we developed a custom linux kernel device driver specifically suited for our cards.

• Thanks to the OO design the switching to the new driver required just one line of code change in the DAQ package !


A successful integration of open source components

In this project we followed the general philosopy to take advantage of all the existing open source libraries and components and to develop just the specific DAQ related parts.

The following open source packages has been used:

• Xerces: an xml parser by the Apache project (http://xml.apache.org/xerces-c). XML is used to provide contextual content to configuration files.

• Qt: a multiplatform, C++ framework with a set of powerful graphical widgets to develop sophisticated GUI applications (http://www.trolltech.com)

• Root: a data analysis package (http://root.cern.ch)

• QTRoot: a set of graphical analysis widgets to integrate QT and ROOT (http://www-linux.gsi.de/~go4/qtroot/html/qtroot.html)

• Doxygen: a Web auto-documentation system for C++, C, Java, IDL. (http://www.doxygen.org)


Conclusions

• A low cost PCI DAQ framework has been developed

• The system is highly flexible– new detectors can be added just by providing a suitable

mezzanine card – virtual functions for detector initialization e calibration are

available in the framework

• A complete test of an untriggered DAQ for pixel detectors has been carried on

• We exploited the advantages of a good object oriented design

stefano magni i.n.f.n milano italy - nss 2003 pomone, a pci based data acquisition system authors:...

Documents