particle physics & astrophysics representing: mark freytag gunther haller ryan herbst michael...

Particle Physics & Astrophysics

Representing:Mark FreytagGunther HallerRyan HerbstMichael HufferChris O’GradyAmedeo PerazzoLeonid SapozhnikovEric Siskind

LCWS 2008 Data Acquisition LCWS 2008 Data Acquisition and Global Detector Networkand Global Detector Network

SiD ATCA DAQ System

Matt Weaver, [email protected] National Accelerator Laboratory

November 18, 2008

Talk based largely on Huffer, et al., “A new proposal for the construction of high speed,

massively parallel, ATCA based Data Acquisition Systems,” IEEE 2008 IEEE 2008 Nuclear Science Symposium.Nuclear Science Symposium.

Represents development work done at SLAC for the following projects:

PetaCache, LCLS, LSST


2

OutlineOutline

• Modular DAQ design to address new projects

• ATCA demonstration boards– Reconfigurable Cluster Element– Cluster Interconnect Module

• Example application to SiD


3

Abstract HEP Data Acquisition SystemAbstract HEP Data Acquisition System

– Analog shaping– Digitization– Transmission– Trigger pipeline buffering– Charge injection

Front-End Electronics

DataflowDetector

Back-End Systems

– High-Level Triggering– Event-Building– Archival– Data Quality Assurance

– Feature Extraction– Compression– Re-formatting– Event buffering– Aggregation– Reception & Transmission– Protocol conversion

emphasis is on this component


4

A A clustercluster of 12 elements of 12 elements

To back-end systems To back-end systems

ClusterInterconnect

Elements

From Front-End systems

switching fabric

switching fabric


5

Typical (5 slot) ATCA crateTypical (5 slot) ATCA crate

fans

“back-end” I/O

“front-end” I/O

“interconnect” board

Power supplies

“element” board Shelf

manager

Front

Rear


6

(Reconfigurable) Cluster Element (RCE) (Reconfigurable) Cluster Element (RCE)

• Bundled software:– bootstrap loader– Open Source kernel (RTEMS)

• POSIX compliant interfaces• standard I/P network stack

– exception handling support

MGTs

Core

DSP tilesCombinatoric logic

Resources

Processor450 MHZ PPC-405

512 MByte RLD-II

Boot Options Memory Subsystem Configuration

data

128 MByte Flash

Data Exchange Interface (DEI)

instruction

reset & bootstrap

options

• Class libraries (C++) provide:– DEI support– configuration interface

• Bundled software:– GNU cross-development

environment (C & C++)– remote (network) GDB

debugger– network console


7

ResourcesResources

• Multi-Gigabit Transceivers (MGTs)– up to 24 channels of:

• SER/DES• input/output buffering• clock recovery• 8b/10b encoder/decoder• 64b/66b encoder/decoder

– each channel can operate up to 6.5 gb/s– channels may be bound together for greater aggregate speed

• Combinatoric logic• gates• flip-flops (block RAM)• I/O pins

• DSP support– contains up 192 Multiple-Accumulate-Add (MAC) units


8

RCE board + RTM (Block diagram)RCE board + RTM (Block diagram)

MFD RCE

flashmemory

RCE

slice0

slice1

slice2

slice3

slice7

slice6

slice5

slice4

MFD

Fiber-optic transceiversZone2

Payload RTM

Zone3

Zone3

E0

E1

E0

E1

Basenetwork

Fabricnetwork


9

RCE board + RTMRCE board + RTM

Media Carrier with flash

Media Slice controller

RCE

Zone 1(power)

Zone 2

Zone 3

transceivers

RTM


10

Front-end Interface ExampleFront-end Interface Example

Transceiver

RegisterInterface

Clock/Reset generator

TransmitInterface

GT11CLK

MGT

Reference clock

PGPprotocol

coreMGT

GT11CLK

TriggerInterface

reset request

input[31:0]

address[23:0]

request

R/W

ACK

status

output[31:0]

context

opcode

trigger

enable

SOF

EOF

error

byte/word

data[15:0]

clock

reset

FEE logic Front-End Board

FPGA

POR


11

Cluster Interconnect board + RTM (Block diagram)Cluster Interconnect board + RTM (Block diagram)

MFD CI

Q2

P2

1-GE

10-GE XFP

XFP

Q0

Q1 Q3

XFP

XFP

XFP

XFP

XFP

XFP

XFP

Payload RTM

P3

P3

10-GE1-GE

10-GE XFP

10-GE

(fabric)

(base)

base

fabric

(fabric)

(base)


12

Cluster Interconnect board + RTMCluster Interconnect board + RTM

Zone 3

Zone 1

10 GE switch

XFP

1G Ethernet

MgmtProcessor

XFPRTM


13

SiD Data AcquisitionSiD Data Acquisition

• EMCal example

– ~100M channels– 4bytes/hit x 4hits/train (KPiX)– readout in 100ms (1/2 of inter-train spacing)– EMCal total readout data rate ~ 128Gbps


14

SiD DAQ continued…SiD DAQ continued…

to Cluster Element

FPGA

Protocol interface

One ATCA crate

EMCal(super?-) concentrators

FPGA

8 Gbytes/sec

500 Mbytes/sec

16 elements

8 RCE boards

2 CI boards

1.6 Gbytes/train

ClusterElement

ClusterInterconnect


15

StatusStatus

• SLAC is positioning itself for the production of a new generation of DAQ systems– strategy is based on the idea of modular building blocks

• inexpensive computational elements (the RCE)• interconnect mechanism (the CI)• industry standard packaging (ATCA)

• Architecture is now relatively mature– both demonstration boards (& their corresponding RTMs) are functional

• RTEMS ported & operating• network stack fully tested and functional

– performance and scaling have been demonstrated to meet expectations• Current focus is on applications using these concepts…

– LSST DAQ– LCLS DAQ– PetaCache project

• Application of same boards to SiD DAQ could yield a compact yet scalable system.


16

particle physics & astrophysics representing: mark freytag gunther haller ryan herbst michael...

Documents