adass 2007 september 24, 2007 data acquisition in high-energy physics johannes gutleber cern...

ADASS 2007ADASS 2007

September 24, 2007

Data Acquisition in

High-Energy PhysicsJohannes Gutleber

CERN

European Organization for Nuclear Research

1211 Geneva 23, Switzerland

ADASS 2007 Johannes Gutleber - CERN 2

OutlineOutline

• High-Energy Physics Experiments• Data Acquisition Systems• Technical Challenges• Operational Challenges• Organizational Challenges

Disclaimer:We give a gentle overview and pick out single issues to stimulate your appetite.

Disclaimer:We give a gentle overview and pick out single issues to stimulate your appetite.


The Size of ThingsThe Size of Things


CERNCERN

The Large Hadron Collider Project


Data from Particle CollisionsData from Particle Collisions

Particle

Proton - Proton 2804 bunch/beam

Protons/bunch 1011

Beam energy 7 TeV (7x1012 eV)Luminosity 1034cm-2s-1

Crossing rate 40 MHz(every 25 ns)

Collision rate ≈ 107-109

Parton(quark, gluon)

Proton

Event selection:1 in 10,000,000,000,000Event selection:1 in 10,000,000,000,000

ll

jetjet

Bunch

SUSY

Higgs

Zo

Zo

e+

e+

e-

e-

New physics rate ≈ .00001 Hz

An Event is a particle collision that is recorded. The rejection rate ofnon-interesting events is 99.9997%(from 40 MHz to 100 Hz). Higgs event probability is 1 in 1012 (3 d)


High Energy Physics ExperimentHigh Energy Physics Experiment


Trigger and Data AcquisitionTrigger and Data Acquisition

• Constraints

– For a reasonable amount of money

– With the highest possible performance

• The photographer analogy:

– Camera: the experiment

– Optics: the various subdetectors (per particle type)

– Trigger: the photographer/camera push-button combination

– DAQ: burning the film, rolling out the picture, storing film

– Quality of shot: number of pictures/second, number of pixels and of course the photographer

• All physics analysis runs off of the film taken

Task:Look at all bunch crossings, apply a filter to reduce data rate, combine all channels, provide it to event filter and off-line storage

Task:Look at all bunch crossings, apply a filter to reduce data rate, combine all channels, provide it to event filter and off-line storage

DAQ does not look into the dataApplies though physics plug-ins at level 1 and event filter

DAQ does not look into the dataApplies though physics plug-ins at level 1 and event filter


T = Accept

Reject

Developing the FilmDeveloping the Film

Combine data fragments -> event building


Challenges I - TechnicalChallenges I - Technical

• Operational requirementsOperational requirements

• ArchitecturesArchitectures


HEP DAQ SpecialitiesHEP DAQ Specialities

• Are embedded systems– DAQ systems never work alone,

they are always embedded in a broader context• Interfaces to (often custom) readout electronics• Interfaces to other processing systems

• Are mission critical systems– If the DAQ subsystem does not work,

the whole system does not work• Are not real-time systems

– May contain, however, real-time paths


DAQ ≠ DAQDAQ ≠ DAQ

• SCADA (Supervisory Control and Data Acquisition)

– Commercial systems and products

– Are systems to “control/manage” control systems

– Typically small data rates and volumes

– Industrial standards (OPC)

– Used at CERN for controlling the detector hardware control systems

• High Energy and Nuclear Physics DAQ

– High throughput computing (vs. parallel computing)

– High degree of distribution

– Similarities with upcoming commercial projects

• Video processing clusters

• Proteomics (MALDI-TOF, sequencing)

• Astronomy ???


Architecture EvolutionArchitecture Evolution

Event buildingEvent building

Detector ReadoutDetector Readout

On-line processingOn-line processing

Off-line data storeOff-line data store

Custom HW ReadoutBus-based event buildingMinicomputer basedEvent filteringOff-line tape store

Custom HW ReadoutNetwork event buildingMinicomputer basedEvent filteringOff-line tape store

Programmable ReadoutSwitched event buildingCluster based filteringOff-line data served by data grid infrastructure

SPS, ISR, 1980’ies LEP era, 1990’ies LHC, 2001 -

Growing degree of distributionGrowing system sizes

Growing degree of distributionGrowing system sizes


Event BuildingEvent Building

40 MHz Collision rate

100 kHz outputfrom trigger

500+500 ports switching fabric(full N x N connectivity!)

5-10 TeraOps processing cluster(take decision within deadline,

can‘t go back)

1 Terabit/sec50000 channels

~100 Gbytes/sec full connectivity

100 Mbytes/sec (1.5 Tbytes/day) output toPetabyte archive and output to world-wide data grid

Detectors

500 Readout UnitsBuffer incoming data and

Serve it upon request

500 Builder Units combinethe data from the RUs

DAQ ClusterDAQ Cluster


ProblemsProblems• COTS networks not adapted to high throughput computing

– Parallel computing - small messages, high rate (queuing theory)– Internet - high bandwidth, 30% network usage, low interconnectivity

• Memory management– Must deal with multiple types of memory concurrently

• Physical, virtual, on card (PCI, VME)– Overhead of memory copy and bus utilization– Fight memory fragmentation and growth

• Scalability– Adding elements must yield improved performance– Must be predictable

• Configuration– Need methodology for configuring large systems– Configurations may change quickly (runs)– XML as workhorse

• Error handling and monitoring

No commercial software or system support for these kind of systems exist to date

No commercial software or system support for these kind of systems exist to date


Challenges II - OperationalChallenges II - Operational

• Scale

• Architectural software support

• Control

• Monitoring

• Error handling


Meeting Performance RequirementsMeeting Performance Requirements

• Low level: Traffic shaping– Barrel shifter– Token credit scheme– Memory management (buffer loaning, zero copy)

• Medium level: Configuration– Striping over multiple output and input legs

(per message)– Multistage switches (CLOS)– Redundant paths

• High level: System organisation– Slices and partitions


ScalabilityScalability

Scalability:The system shall have the ability to scale, that is to operatewithin the requirements as it changes in size or volume andto take advantage of additional resource availability.

Scalability:The system shall have the ability to scale, that is to operatewithin the requirements as it changes in size or volume andto take advantage of additional resource availability.

bad

good within requirements

linear

System size

Perf

orm

ance


Scaling by Increasing the NetworkScaling by Increasing the Network

Network Network

Hit the limit soon depending on the switched network technology64x64, 256x256 single links (input/output devices)


Slicing (3D)Slicing (3D)

• Scale into the third dimension• Multiple slices can be connected through networks that are close to the

detectors– Routing can be hardwired to avoid inefficient usage

• Limits today: 64x64x8 = 512 inputs/512 outputs

• Reduces bottleneck• Enables staged deployment• Allowes for larger clusters


Trapezoidal Slicing (3D)Trapezoidal Slicing (3D)

• Asymmetric input and output link speeds

• Profit from higher CPU processing power at processing nodes

• Limits today: 64x256x8 = 512 inputs/2048 outputs


Scaling and StagingScaling and Staging

DAQ unit (1/8th full system):Lv-1 max. trigger rate 12.5 kHzRU Builder (64x64) .125 Tbit/sEvent fragment size 16 kBRU/BU systems 64Event filter power ≈ .5 TFlop

Data to surface:Average event size 1 MbyteNo. FED s-link64 ports > 512DAQ links (2.5 Gb/s) 512+512Event fragment size 2 kBFED builders (8x8) ≈ 64+64

Scaling & Staging


Operation SoftwareOperation Software

• Architectural Software Support– Low and medium level performance enablers– Executive approach from I2O, Infiniband, Component

CORBA, various RTOS elevated to application level• Software as an integral part of the architectural scaffolding

– uniform addressing– uniform communication (technology independent)– uniform configuration– uniform monitoring– uniform handling

• Software provides centrally designed efficiency enablers– Provide implemented patterns

Lesson learned:To build a clustered data acquisition system one needs more than a collection of computers and networks that are plugged together.

Lesson learned:To build a clustered data acquisition system one needs more than a collection of computers and networks that are plugged together.


Peer To PeerPeer To Peer

Network

Application componentImplements the data processing task on one local computer

Peer-transport componentImplements optimized access to a specific comm. technology

Executive

Computer

Cluster based systems with XDAQ:The executive framework on each computer hosts the various application components that together perform the cluster application. Exchangeable peer-transports provide the applications with data transmission services over several network technologies concurrently. All software components are dynamically loadable at run-time.

Cluster based systems with XDAQ:The executive framework on each computer hosts the various application components that together perform the cluster application. Exchangeable peer-transports provide the applications with data transmission services over several network technologies concurrently. All software components are dynamically loadable at run-time.


Chained Block OperationChained Block Operation

Sending chained buffers:Buffers can be chained (step 1). An application sends the first buffer reference and all chained buffers are sent by the peer transport (step 2). Upon successful sending of the last chain element, the chain is broken and the buffers are released into the originating poo (step 3). Such, zero-copy operation from input to output is achieved and arbitrary length messages can be created.

Sending chained buffers:Buffers can be chained (step 1). An application sends the first buffer reference and all chained buffers are sent by the peer transport (step 2). Upon successful sending of the last chain element, the chain is broken and the buffers are released into the originating poo (step 3). Such, zero-copy operation from input to output is achieved and arbitrary length messages can be created.


Web Peer (HyperDAQ)Web Peer (HyperDAQ)

Application componentIs controlled/monitored through Web protocols, cooperates with other applications through other PTs

Myrinet PTSOAP/HTTP PT

SOAP/HTTP PT:Each application becomes browsable through an embedded HTTP/SOAP peer transport.-Navigate directly to each application in the cluster-Applications implement application specific web pages and callbacks-Components become truly reflective (make themselves visible)-Gradual integration from manual to automatic through scripts and programmatic approaches

SOAP/HTTP PT:Each application becomes browsable through an embedded HTTP/SOAP peer transport.-Navigate directly to each application in the cluster-Applications implement application specific web pages and callbacks-Components become truly reflective (make themselves visible)-Gradual integration from manual to automatic through scripts and programmatic approaches


Direct Direct AccessAccessBrowsable ClusterBrowsable Cluster


Web reflective applications


ControlControl• Programmatic + GUI

– Through scripts and specialized languages– Encountered in commercial SCADA systems

(Supervisory Control and Data Acquisition)– Pro: Easy to extend, simple to use– Con: no real-time behaviour, risk of inconsistency– Used: for LHC detector control systems at high level

• State machines– Through libraries, hierarchically distributed– Pro: formal approach, well understood technologies– Con: Risk of inconsistencies, scalability problems due to hierarchies,

restrictive (boolean)– Used: for detector control subsystems, run control systems

• Workflow– Currently being investigated as alternative to state machines– Pro: well understood formalism (extends Harrel state charts), promises

improved integration capabilities of subsystems through Web protocols (BPEL), large potential for error handling subsystems

– Con: domain is not focus of COTS products (consequences: important footprint, distribution uncovered)

– Used: in R&D state

Evolu

tion


Callenges III - OrganizationalCallenges III - Organizational• 1250 scientists, 132 nations

• Unstructured organization

• Personnel turnover

• One of a kind projects

– Bleeding edge

• Financial constraints


Subsystem Integration (Hardware)Subsystem Integration (Hardware)

• Minimally standardized data link– Serial, simple protocol, high throughput

• Need to integrate many, diverse types of custom readout devices

to Central Data Acquisition> 1000 devices

Cu

sto

mC

OTS


Subsystem Integration (Software)Subsystem Integration (Software)

• Provide every participant with same software infrastructure– Uniform monitoring– Uniform error handling– Uniform configuration– Uniform communication

• Integrate Web peers– Decoupled operation

(HyperDAQ pages, speed dial Web map)– Integrated operation (SOAP, HTTP, workflows)

• System of systems– Encapsulated subsystems– Minimal interface and documentation needed

• Subsystems provide workflows• Integration of workflows

– Allow for operation with subset of subsystems• In/out configurations


HyperDAQHyperDAQMonitorMonitor

Data collectedfrom computers

Collected datarendered

graphically

Widget

Hint:Error handlingsame style


SummarySummary

• HEP DAQ is high throughput cluster computing– Data bound not computation bound– N x N full bandwidth connections– Systems built for scalability (staged commissioning)– Large system size

• Foster architectural software support– Software as integral part of the system architecture– Software as performance enabler– Little commercial software support available– Use open, self-contained packages (exchange!)

• Integration through– Simple and narrow hardware interfaces– Web peers (HyperDAQ and SOAP)


Information (XDAQ)Information (XDAQ)

http://xdaqwiki.cern.chhttp://www.sourceforge.net/projects/xdaq

[email protected]@cern.ch


SecuritySecurity

• XAccess Module

– Basic Web authentication

– IP based filtering

– Combination of both

• SSL (SOCKS)

– For server and client authentication

• Custom policy

– Any policy can be “plugged-in”

• Investigation

– COTS products (Oracle OIM)

– Web gateways

adass 2007 september 24, 2007 data acquisition in high-energy physics johannes gutleber cern...

Documents

filteringoffline data

data acquisitionconstraintsfor

data grid infrastructuresps

higgs event probability

bunchnew physics rate

physics plugins

nscollision rate

productsare systems