the d0 daq and its operational control

CHEP 2000, February 7 - February 11, 2000, Padova (Italy), Abstract 379.

The D0 DAQ and its Operational Control

G. BriskinD. Cutts

A. KarachintsevS. Mattingly

G. WattsR. Zeller

Gennady Briskin, Brown University

2Gennady Briskin (Brown University)

The DØ Trigger System

Tape Output

Level 1

Level 2

Level 3

Trigger Information

Full Readout

C/R

Detector

DAQ

RIP

~50-70 Hz~50-70 Hz

~1 kHz~1 kHz

~10 kHz~10 kHz

~1 kHz~1 kHz

~7 MHz~7 MHz


Run II DAQ Numbers• Readout channels:

– Will be ~800,000 in Run 2 <250 kBytes>/event

• Data rates:– 60-80 readout crates– Many to one:

• All GS to one of 64 L3 farm nodes– Initial design capacity:

• ~1000 Hz• 250 MBytes/sec into the DAQ/l3-farm• Staged upgrades to more than 3 GB/sec


VBDVBD

VRCVRC

SBSB

MPMMPM

{ FCIETIDCI

ETGETG

VME Buffer/Driver

VBD Readout Concentrator

Segment BridgeFibre Channel InterfaceEvent Tag InterfaceData Cable Interface

Event Tag Generator

Multiport Memory

Digitizing crate

Level-3 filter node

NewNew

NewNew

NewNew

L3/DAQ Hardware Glossary


MPM MPM

SB

o o o

MPM MPM

SB

o o o

MPM

SB

o o o

MPM MPM

SB

o o o

MPM

VRC (1 of 8)

VB

D

VB

D

VB

D

o o oV

BD

VB

D

VB

Do o o

ETG

data collection pathdata collection path

primary FC Path

L3 data distribution to Level 3 farmLevels 1,2Levels 1,2trigger datatrigger data

event tagevent tagpathpath

(stage C)

• direct flow from VBD to MPM

• each data block flows freely and independently

• blocks for different events flow simultaneously

and asynchronously

• recirculation loops allow maximum use of data

path bandwidth

• Segment Bridges use Event Tag data to perform

complex realtime routing decisions

L3/DAQ Data Path


Segment Data

Cables

Segment Data

Cables

)

VRC1

)

Front EndCrate

Front EndCrate

Front EndCrate

Front EndCrate

Front EndCrate

Front EndCrate

Front EndCrate

Front EndCrate

VRC8

S(4 DATA CES

L3 Node(1 of 16)

L3 Node(1 of 16)

L3 Node(1 of 16)

SB1

SB4

ETG

Event Tag Loop

Primary Fiber Channel Loop #1

Primary Fiber Channel Loop #8

Front End TokenReadout Loop

Front End TokenReadout Loop

TriggerFramework )

L3 Node(1 of 16)

L3 Node(1 of 16)

L3 Node(1 of 16)

To Collector

Router

To Collector

Router

Ethe

rnet

Ethe

rnet


DAQ Architecture• Data flow is unidirectional

– No complex setup communication is required

• Each packet is sent only once– No retransmission if packet error is detected

• Routing is driven by the contents of the VBD header– No communication with front end crates

• Sender does not know or care where data is going

• Designed for expansion beyond the run 2 era


VME Buffer Driver Module

• Digitized data enters DAQ system via a VBD module– Typically 4-6 Kbytes per VBD/event

Dig

itiz

ers

Dig

itiz

ers

Dig

itiz

ers

VBD

VBD

Front End Crate

Token Control

Data Pathway48 MBytes/sec

From last VBD or VRC

To next VBDor VRC

•Components:–VME interface with list processing DMA–Dual SRAM buffers–External data cable interface with token arbitration logic.

•Performance:–VME: BLK mode DMA at 25-30 MB/sec–Data cable output: 48 MB/sec.–Token arbitration time: <10 usec.



• Bridges VBD readout loops to Primary Fibre Channel loop

• Provides readout token generation and general front end loop management: e.g. initialization, resets, and diagnostics

• Primary loop interface via full-duplex Fibre channel port– transmits data to first Segment Bridges (SB’s), receives

recirculated packets from last SB;

VRC

VBD

Data Cable

Token CLK

VBD

VBD

VBD

VBD

VBD

Token CLK

100 MB/s

FCI to 1st SB

100 MB/s

FCI from last SB

Data Cable

Loop #1

Loop #2



PCI/66Pentium

XXX128Mb

Ethernet

PCI

PCI

DPMem i960 Buffer

Mem

HD C A B L E

HD C A B L E

Term & FBTrans

64/26

64/26

VRC 1

FC P O R T

HD C A B L E

Term & FBTrans

64/26

64/26

FC out

FC in

DPMem i960 Buffer

Mem

HD C A B L E

HD C A B L E

Term & FBTrans

64/26

64/26

VRC 2

FC P O R T

HD C A B L E

Term & FBTrans

64/26

64/26

FC out

FC in


The Event Tag Generator• Receives L2 Accept Trigger Information

• Correlates Trigger Framework information with preprogrammed parameters for Level-3 node configuration, and then classifies event as:– generic: any segment, any node;– targeted: specific segment/node;– special function: diagnostic, broadcast, etc.

• Creates corresponding ‘event tag’ that instructs SB’s on how to handle data from that event.

• Places event tag in primary tag queue and transmits it when tag queue is available.

• Queue fill status is used to provide rate limiting feedback to Level 1 triggers– L1 disable lines


ETG System– Links L3/DAQ with the L1/L2 trigger systems– Uses trigger information for a given event to create an

“event tag” which enables complex data block routing in real time

•16 bit L3 Transfer Number•Trigger Bits

ET Creator(CAM Lookup)

SC SC SC SC

Return ETGMonitor

From Trigger FrameWork

ET Queue

L1 Disable

MonitorCPU

Ethernet

Event Tag Format:• Header

• Event Number• Flags (diagnostics or regular)

• Segment Controller 1 Block• Event Type• FEC Readout Bit Masks

• Segment Controller 2 Block• Event Type• FEC Readout Bit Masks

• etc.

}}

L3 Farm


Segment Bridge• SB routes data from the VRCs to a L3 Node(s)

– Bridge Primary Fibre Channel loop to Level-3 data cables

– Each Level-3 Segment has a Segment Bridge– ETG circulates event tag information to each SB.– ETG/SB is capable of complex node assignments

• Node assignment based on event type (calibration, etc…)• Shadow Nodes

– A second nodes receives the same event; allows testing of new L3 exes– Allows partitioning into several simultaneous data

taking runs

• SBs are serially linked together– One can expand the number of farm segments


Event-Node Assignment

ETGSB

L2 TriggerBits

Event Type

Trigger-bitevent type

lookup tables

Node Assignment

• Node Readies• Node-Event type

lookup table

Event Tag

• Regular• CAL_CALIB• Regular + Shadow #2• COSMIC


Segment Bridge

Router

DataCable Drivers

Fiber Out to next SC

or VRC

Fiber In from SC or VRC

•Receives Event Tag from ETG• accepts/declines the event based on characteristics of available L3 nodes


Dedicated 100 Mbits/sEthernet to

Online Collector/Router

L3 Farm Nodes

• Input rate starts at 1000 Hz• Output rate is ~50-70 Hz• Intel Based SMP System• Windows NT• As many L3 Filters as possible• Prototype hardware in hand

100-BaseT

Ethernet

L3FilterL3

FilterL3 FilterProcess

DMA capableVME-PCI Bridge

L3 Node FrameworkEach 48 MB/s

Control,Monitoringand ErrorModule

L3 Filter InterfaceModule

Node-VME I/O Module

Shared Memory Buffers

VME Crate

MPM

MPM

CollectorRouterModule


Validation Queue

Event Validation

• FECs presence validation• Checksum validation

L3 Filter Input Interface

Process Interface

Pool Queue

Collector/Router Network Interface•Determine where this event should be sent•Sent event to collector/router node

Filter Queue

Data to Online Host System

Data

Control

OutputEventsQueue

•Get a pointer to an event buffer •Configures MPMs for receiving new event•Wait till complete event arrives into MPM•Load event data into shared memory buffer•Insert event pointer into the next queue

L3 Supervisor Interface

L3 Monitor Interface

L3 Error Interface

Command/Monitor/ErrorShared Memory

Event BufferShared Memory

L3 Filter Process

L3 Filter Output Interface

Process Interface

OutputPool

Queue

MPM Reader


The L3 DAQ Control System Goals

•High speed configuration changes–Run start/stop–Physics Run/Calibration Run Modes

•Diagnose problems quickly


Environment•All nodes will use Commodity Hardware & Windows OS

•C++–MS Visual Studio 6.0

•Works well, IDE is a blessing.–Visual Source Safe for source control

•Very tightly integrated into the MS Visual Studio IDE

•Communication–ACE as the cross platform wrapper around TCP/IP stack–itc package to manage messages and event transport with D0 Online system (Fermi CD)–DCOM for internal control of DAQ components–XML message formatting for internal control/configuration of DAQ components


SC

The Level 3 DAQ/Trigger system

L3 FarmNode

L3 Supervisor

ETGVRCVRCVRCVRC

SCSCSB

OnlineSystem

CollectorRouter

L3 Monitor


DAQ Components States

Boot

IdleFinishRun

Start Run

Shut-down

PauseRun

•Every DAQ component can be in 1 of the 6 states


Auto Start SystemAuto Start Server

Web Server

Configuration Database

Client Machine

Auto StartService

Get Package List

Install Packages

Package Database

Configuration

Change Packages,Get Status,Reboot, etc.

Package

Running Packages

(COM Object)


The Level 3 DAQ/Supervisor System

•Translate D0 online system commands into L3 specific commands

– Begin Run– End Run– Run Script ele_high on L2 Trigger bit 2 on 6 nodes of type REGULAR.

•Report configuration errors (synchronous)– Must track a single online system command as it is distributed to DAQ components

–Maintain DAQ System state


L3 Supervisor Interface

COOR

COORCommandInterface

Current Configuration

DBResourceAllocator

CommandGeneratorSequencer

L3 NodeL3 NodeL3 NodeL3 NodeL3 Node

Supervisor Online System

CommandsConfiguration

Request

DesiredConfiguration

Data Base

DirectCommands

Commands


Monitor System

CollatorProcess

L3Monitor

Monitor Client

Process1

Process2

Process3

TransmitterProcess

Any L3 Computer

SharedMemory

SharedMemory

SharedMemory

Process1

Process2

Process3

Any L3 Computer

SharedMemory

SharedMemory

SharedMemory

TransmitterProcess

Monitor Client

Monitor Client


VRC Interface

FCI from last SB

VRCPrograms

ControlDisk

L3 SupervisorL3 Monitor

VRC Node

EmbeddedSystems

EmbeddedSystems

EmbeddedSystems

VBD Data Cables50 MB/s50 MB/s

100 MB/sFCI to 1st SB

100 MB/s

DCOMDCOM


ETG Interface

ETGPrograms

ControlDisk


ETG Node

EmbeddedSystems

EmbeddedSystems

EmbeddedSystems

TriggerFramework

Triggers TriggersDisable

DCOMDCOM


Segment Bridge Interface

SBPrograms

ControlDisk


SB Node

EmbeddedSystems

EmbeddedSystems

EmbeddedSystems

MPM Data Cables

100 MB/s FCI from VRC or SB

FCI to next SB

100 MB/s

DCOMDCOM


Conclusion• Natural upgrade of the Run I DAQ

– Allowed us to put together the framework for the L3 farm node with modest amounts of hardware effort

• New DAQ hardware permits > 10 increase in bandwidth

• DAQ control system is designed to be modular– makes use of industry standard software


The valiant warriors of D0 are strong thanks to a magic potion prepared by druid Getafix and this recipe is secret (all we know is that it has lobster and beer)

the d0 daq and its operational control

Documents