atlas hlt/daq stato e prospettive

ATLAS HLT/DAQ ATLAS HLT/DAQ Stato e prospettiveStato e prospettive

Valerio VercesiValerio Vercesi

CSN1 Settembre 2005CSN1 Settembre 2005

CSN1 Settembre 2005 V. Vercesi - INFN Pavia 2

OutlineOutline

Pre-series Status in USA15/SDX1 Commissioning and exploitation

Large Scale Test Activities, experiences Lessons learnt

Activities Monitoring, ROD Crate DAQ Algorithms development and deployment

Finance Accounting 2006 requests


S. Falciano (Roma1) Coordinatore Commissioning HLT A. Negri (Pavia) Coordinatore Event Filter Dataflow A. Nisati (Roma1) TDAQ Institute Board chair e Coordinatore

Muon Slice PESA F. Parodi (Genova) Coordinatore b-tagging PESA V. Vercesi (Pavia) Deputy HLT leader e Coordinatore PESA

(Physics and Event Selection Architecture) Attività italiane

Trigger di Livello-1 muoni barrel (Napoli, Roma1, Roma2) Trigger di Livello-2 muoni (Pisa, Roma1) Trigger di Livello-2 pixel (Genova) Event Filter Dataflow (LNF, Pavia) Selection software steering (Genova) Event Filter Muoni (Lecce, Napoli, Pavia, Roma1) DAQ (LNF, Pavia, Roma1) Monitoring (Cosenza, Napoli, Pavia, Pisa) Pre-series commissioning (LNF, Pavia, Roma1)


ATLAS TDAQ system ATLAS TDAQ system

Muon

ROD ROD ROD

LVL1

LVL2

Event builder network

Storage: ~ 300 MB/s

ROBROB ROBROB ROBROB

Calo Inner

PipelineMemories

ReadoutDrivers

ReadoutBuffers~1600

High-Level Trigger

LEVEL-1 TRIGGER• Hardware-Based• Coarse granularity from calorimeter & muon systems

LEVEL-2 TRIGGER• Regions-of-Interest “seeds”• Full granularity for all subdetector systems

• Fast Rejection “steering”

EVENT FILTER • Possibly “seeded” by Level 2 • Full event access• Algorithms inherited by offline

RoI

EF farm~1000 CPUs

1 selected event

every millionTDAQ ≅Rates

40 MHz

~75 kHz

~2 kHz

~200 Hz

~2 ms

~10 ms

~ 1 s

Latency

EF

LVL2farm

( )


TDAQ TDAQ

Trigger e Data Acquisition hanno da sempre in fase di commissioning un doppio ruolo Come “server” per il commissioning dei rivelatori Come “client” per utilizzare le informazioni realistiche

dell’esperimento per i propri studi di funzionalità e performance

La situazione si è già presentata durante il Combined Testbeam 2004

Il TDAQ di ATLAS è un progetto in piena evoluzione in cui development/commissioning/exploitation sono ancora fasi molto miscelate Presentazione di risultati e indicazione delle prospettive Maggiore enfasi alle componenti con forte partecipazione

italiana Descrizione del piano di commissioning generale


Pre-series designPre-series design“Module-0” of final system

8 racks (~10% of final dataflow)


Pre-series realityPre-series reality

LVL2 rack

EF rackSwitch rackOnline rack

6 racks SDX1

ROS rack


Commissioning and exploitationCommissioning and exploitation

Fully functional, small scale, version of the complete HLT/DAQ Equivalent to a detector’s ‘module 0’

Purpose and scope of the pre-series system Pre-commissioning phase

To validate the complete, integrated, HLT/DAQ functionality To validate the infrastructure, needed by HLT/DAQ, at point-1

Commissioning phase To validate a component (e.g. a ROS) or a deliverable (e.g. a Level-2

rack) prior to its installation and commissioning TDAQ post-commissioning development system

Validate new components (e.g. their functionality when integrated into a fully functional system)

Validate new software elements or software releases before moving them to the experiment


Pre-Series CommissioningPre-Series Commissioning


Commissioning LVL2+ROSCommissioning LVL2+ROS

First measurements with full LVL2 rack feeded by ROS data

Using separate Control and Data networks


Commissioning EFCommissioning EF


Large Scale TestsLarge Scale Tests Pre-serie work will help understanding the TDAQ system in terms

of functionality Forms the basis for future deployments/exploitations

Complexity of ATLAS TDAQ system arises also from the size of bulk components involved Topology of communications, size of LVL2/EF farms, software, …

Test scalability of HLT system using presently available large installations Understand issues like configuration, startup time, communication, control,

error reporting, … UCB/TRIUMF WestGrid Cluster (http://www.westgrid.ca)

60 racks x 14 nodes = 840 Dual-CPU nodes(3 GHz CPUs / 2-4 GB RAM)

CERN LXSHARE Cluster (http://batch.web.cern.ch/batch) Up to ~700 nodes (various flavours)

Reference page for all tests http://atlas-tdaq-large-scale-tests.web.cern.ch


State transitionsState transitions

Luke Warm Start Luke Warm Stop

USR_RUNNING_TIME(default is 30 s)

RUNNING

CONFIGURED

INITIAL

ABSENT

Configure

Boot

Unconfigure

Shutdown

Setup Close

Cold Start Cold Stop

configure: load configure

start: prepareForRun startTrigger

stop: stopTrigger stopFrontEnd stopDataCollection stopEventFilter stopRecording

unconfigure: unconfigure unload


LVL2 transition times LVL2 transition times

State transistion timing quite acceptable No significant differences between 2 and 3 tier Run Control

2 T ier Run Control

0

10

20

30

40

50

60

70

80

0 5 10 15 20 25

Configuration Id (# L2PU nodes 8 -> 256)

Tim

e (

se

cs

)

setup

boot

conf

start

wait

stop

unconf

shutdown

close


EF results @ LSTEF results @ LST

EF TrigMoore with Oracle/MySQL

0

100

200

300

400

500

600

700

800

0 100 200 300 400 500 600 700

Number of processes (1EFD+2PTs)

con

fig

. /

tota

l ti

mes

, s config Oracle

config MySQL

Total test Oracle

Total test MySQL

Timings HelloWorld / TrigMoore

0

50

100

150

200

250

0 100 200 300 400 500 600 700

Number of processes (1EFD+2PTs)

tim

e,

s

config HelloWorld

config TrigMoore MySQL

backend setup HelloWorld

backend setup TrigMoore MySQL

Effect of realistic algorithm:TrigMoore vs HelloWorld EF standalone 1EFD+2PTs / node up to 200 nodes MySQL as geometry DB used significant slow down due to

access/reading geom. DB MySQL vs Oracle DB in TrigMoore

EF standalone, 1EFD+2PTs / node Oracle DB – up to 160 nodes MySQL DB – up to 200 hosts MySQL works faster at “small

scales”, while Oracle looks better at higher scales - to be investigated more

not able to on higher than 200 nodes with any of both partitions – to be investigate further (do we need to replicated DBs ?)


Cosmics Tile setupCosmics Tile setup

MobiDAQ (Mobile DAQ):

read out of 8 drawers in

the pit with temporary

RODemu but real TDAQ

(tdaq-01-02-00), tests of

electronics, cosmic

muons runs

CSN1 Settembre 2005 V. Vercesi - INFN Pavia

GNAM Monitoring chainGNAM Monitoring chain

Framework per monitoring on-line a basso livello Core: trasporto di eventi, istogrammi e comandi Plugin dinamici: decodifica e istogrammazione Possibilita’ di correlazione fra diversi rivelatori

Comandi asincroni (reset, rebin, update)

Status Incluso in TDAQ da aprile Utilizza i servizi disponibili

per il monitoring Validato al CTB04; in uso in

alcuni siti di commissioning


GNAM al commissioningGNAM al commissioning Acquisizione dati nell’ambito del software TDAQ Commissioning di MDT usa GNAM per monitoring online ed analisi dati Sampling completo a livello di ROS [~2 KHz per noise test, ~200 Hz per pulser] Integrazione di librerie: completato per MDT, in via di sviluppo per RPC Stato del monitoring MDT:

Richiesta minimale di informazioni allo shifter (nomi camere) Output: file di istogrammi e file di testo con risultati di analisi dati per ciascuna camera e per

ciascun run In sviluppo:

ottimizzazione dell’analisi dati on-line installazione del presenter per la visualizzazione degli istogrammi on-line event display


PMP PresenterPMP Presenter

Visualizzazione asincrona on-line di istogrammi Interattivo (reset, rebin, zoom, fitting, ecc...) Operazioni grafiche su istogrammi (ROOT canvas) Grafica configurabile

Status Incluso in TDAQ da settembre Utilizza i servizi disponibili

per il monitoring Pienamente funzionale

al CTB04 Riprogettato per nuove

funzionalità e maggiore scalabilità


GNAM&PMP: sviluppi futuriGNAM&PMP: sviluppi futuri

GNAM Completare l’integrazione nel framework del TDAQ

Configurazione di Core e librerie dal database del TDAQ Messaggistica e gestione degli errori software

Supporto per la generazione di allarmi automatici Livelli di severità, routing

Verifica delle prestazioni e delle risorse necessarie CPU, Memoria, Banda

Studio della scalabilità

PMP Completare la nuova versione

Minimizzazione del traffico di rete Adattamento della IGUI alla nuova struttura

Generazione allarmi Plugin di analisi degli istogrammi


ROD Crate DAQROD Crate DAQ RCD is used as interface with the RODs

for Control Configuration Monitoring

Statistics Event sampling

Data readout (through VME) User guide for detectors developers

available Validation system in Bld. 40 DAQ Commissioning – Phase 1:

The ROD Emulator system will be used in order to validate all common RCD software and infrastructure

After adding and validating the detector sw and hw specific items, multi crate event building will be used in the absence of the full DAQ chain

VMEbus memory + CORBO

=

Memory +

Registers +

Interrupt capability

RCC

MEMORY

CORBO

Config & Control

Data readout

REB

Event Fragments

ROD Emulator

ROS


RCD exploitationRCD exploitation

Experience from Combined Testbeam extremely useful Recall almost all detector used it in the CTB Successful workshop to put forward new requirements

As a consequence, several improvements during last months Configurable interrupt handling Simplified user interface to access ordered event fragments Data driven event building for multicrate acquisition in the commissioning

phase Simplified ROD emulation Hardware trigger distribution

All sub-detector commissioning (but LAr…) sites use RCD MDT and RPC on the forefront

BB5 integration, Point 1 with MROD, Lab testing with ROD emulators


Algorithms: Muon sliceAlgorithms: Muon slice LVL1 simulation is the fundamental input for the measurement of

the full muon vertical slice performance LVL2 and EF Muon algorithms have been extensively tested on

data simulated in ATLAS Rome Physics Workshop: June 2005

LVL2: Fast Confirm the LVL1 trigger with a more precise PT estimation within a

Region of Interest (RoI) Global pattern recognition, track fit, fast PT estimate via Look Up Table

(LUT) with no use of time consuming fit methods Event Filter: TrigMoore

Based on offline reconstruction algorithm Moore Can run seeded (reconstruction starting from RoI of previous levels) Precise PT determination

General goal is now to achieve more realistic estimate of trigger selections and corresponding rates Real geometry, configuration and conditions database usage, …


LVL1 Coincidence WindowsLVL1 Coincidence Windows

Athena release 10.0.4 Low pT 6 GeV Threshold

Low-pT Inefficiency mapEfficiency curve


MuFast endcap extensionMuFast endcap extension Early stages of project Endcap differs from Barrel

M and O station are outside B field Inhomogeneous B field – bending is local

Algorithm Pattern recognition and fit in TGC → position and slope in EM Extrapolate segment into MDT EM/EO → Roads in EM/EO, find hits, fit

Next pattern recognition and fit in MDT as in mFast – not done yet Extrapolation into EI and LUT

-1.40E-05

-1.20E-05

-1.00E-05

-8.00E-06

-6.00E-06

-4.00E-06

-2.00E-06

0.00E+00

0 10 20 30 40

PT

Slo

pe

Forward Endcap


MuFast: MDT miscalibrationMuFast: MDT miscalibration

Commissioning the algorithms: realistic approach to data handling The plot shows the muFast resolution for two different scenarios:

the correct MDT r-t function is used, red points a systematic shift of + 0.2 mm is added to the radius returned by the correct

r-t function, blue points

+10% degradation @ 50 GeV, to be compared with a + 5% expected by a naive calculation


pT resolution (Moore)

The relative degradation in (pT) is +5% for muons with a 6 GeV/c transverse momentum, increases to +13% around 50 GeV/c

This MDT miscalibration leaves almost unaffected resolutions.

TrigMoore: MDT miscalibrationTrigMoore: MDT miscalibration

Single muons (with pT = 6, 10, 20, 40, 100 GeV/c, produced for the Rome Initial Layout) have been reconstructed in two different scenarios Using the correct MDT r-t relation function (red squares in pictures) Applying a systematic +0.2 mm shift on the radius obtained with the correct MDT

r-t function (blue circles in pictures)


LVL2 tracking: SiTrackLVL2 tracking: SiTrack

Preliminary results obtained on DC1 b-jet samples at initial luminosity


B-tagging @ LVL2B-tagging @ LVL2 Results obtained with the “standard” SiTrack algorithm on DC1 data b-tagging: likelihood ratio using transverse and longitudinal impact parameters Upgraded version to be tested soon: should improve both efficiency and track

parameters resolution and hence significantly improve the b-tagging performance


Algorithms steeringAlgorithms steering

Cluster60

eg25

eg15

eg60

iSe60

iSg60

iSg25

l2g60i

l2g25i

eg60Hy

eg25Hy

eg15Hy

T2Calo

iSg15

iSe25

iSe15

EMtrackSoft

l2e25i

l2e15i

gIsol60

gIsol25

eIsol15

eIsol25

g60Hy

g25Hy

g15Hy

e60Hy

e25Hy

e15Hy

EMtrackHard

TrackSoft15EM

l2e60ieIsol60

eg20

iSg20 l2g20i

eg20Hy

iSe20 l2e20i

gIsol20

eIsol20

g20Hy

e20Hy

EM60Hy

EM25Hy

EM15Hy

EM20Hy

EM25

EM15

EM60

EM20

l2e15ieIsol15

TrackSoft02EM

EMROI

Cluster15

Cluster25

Cluster20

TrackSoft25EM

TrackHardEM

T2Calo

T2Calo

T2Calo

EMtrackSoft

EMtrackSoft


ATLAS Commissioning PhasesATLAS Commissioning Phases

Commissioning means bringing ATLAS systems from “just installed” to “operational”. It is broken in 4 phases Subsystem standalone commissioning Integrate subsystems into full detector Cosmic rays, recording data, analyze/understand, distribute to

remote sites Single beam, first collisions, increasing rates, etc…

A consistent part of commissioning activities will be done during the installation itself

Phases will overlap since different systems may be at different development levels For the barrel calorimeter commissioning will start soon Tile calorimeter is already taking data


HLT CommissioningHLT Commissioning Commissioning is a set of activities which spans the time interval from the

installation of the HLT racks and nodes … A rack is the elementary unit for commissioning OS, Dataflow and Online software are installed

... to the phase when the HLT is filtering physics data and recording them HLT selection algorithms are installed and running stably The complete trigger menu (at least for early physics) is configured The trigger selection efficiencies and background rejection rates are understood and

can serve as input for physics measurements Phase-1 Commissioning definition is the most urgent

Heavily use the Pre-series to exercise the procedures for installation and commissioning

Important steps will cover the integration of detectors into full system Involve operations that have a very strong coupling with the offline commissioning

activities Development of specific algorithms looking at simple data decoding (cabling,…)

Final commissioning phases extend far beyond the data-taking startup (interface with run coordinator team)


Cosmic muons in ATLASCosmic muons in ATLAS

Rock ~ Silicon

600m x 600m x 200m deep

(2.33 g/cm3)

AirConcrete

Surface building

PX14/16 shielding

(2.5 g/cm3)

PX14

(18.0 m Inner Ø)

PX16

(12.6 m Inner Ø )

ATLAS

Geant Simulation Initial detector


Overall planOverall plan


OutlookOutlook A lot of work during this year, system entering phase of complete

deployment Purchase plan proceeding as scheduled, with some minor delays (Wo)manpower situation not always healthy More help and support welcome

Three big tasks awaiting us in the next months Commissioning the pre-series and extract a coherent and complete set of

system performance measurements Based on previous experiences and on already established partial results

On-line trigger selections evaluation (rates, efficiencies, physics coverage,…) evolving towards more realistic approach

Calibration, geometry “as installed”, mis-alignment, error handling, complete trigger menus, physics analysis based on trigger objects

Prepare for cosmic run next year “Cosmic” slices and trigger menu (Tile, LVL1, “digital” LVL2)


AccountingAccounting Contributo INFN alla Pre-serie

Read-Out System: 51 kCHF (ROS Racks) Online Computing System: 40 kCHF (Monitoring, Operations) Online Network System: 44 kCHF (Switches, FileServer)

Inviati al CERN a Dicembre 2004 VV riceve in copia tutte le fatturazioni dei singoli acquisti ed un sommario

mensile dello stato finanziario Contributo CORE 2005

Online Computing System: 45 kCHF (Monitoring, Operations) Inviati al CERN a Maggio 2005 Già acquisiti due file server

Read-Out System: 275 kCHF (ROS Racks) Questo acquisto si espleta secondo una gara e non con un semplice market

survey o price inquiry come fino ad ora avvenuto Richiesta alla Giunta l’autorizzazione per partecipare alla gara

Grazie a Speranza che si è prodigata per espletare le pratiche necessarie Il CERN preferisce gestire la gara su un periodo di due anni

Omogeneità dei componenti vs miglioramento delle prestazioni Si sommano i 275 kCHF del 2006


Cost Profile (kCHF)Cost Profile (kCHF)

2004 2005 2006 2007 2008 2009 Total

Pre-series 140 0 0 0 0 0 140

Detector R/O 0 275 275 0 0 0 550

LVL2 Proc 0 0 65 195 230 160 650

Event Builder 0 0 50 50 110 70 280

Event Filter 0 0 170 180 570 380 1300

Online 0 45 135 0 0 0 180

Infrastructure 0 0 80 80 20 20 200

INFN Total 140 320 775 505 930 630 3300

TDR Total 1048 3357 4087 4544 7522 4543 25101

INFN Percentage(%) 13.4 9.5 19.0 11.1 12.4 13.9 13.1


Missioni EstereMissioni Estere

LNF Commissioning HLT/DAQ Pre-serie e pit: 6 m.u.

Ferrer, Kordas (+ Miscetti, Giovannella)

Pavia VV coordinatore PESA e duputy HLT: 1 m.u. Negri A. responsabile Event Filter: 1 m.u. Scannicchio D. commissioning HLT: 2 m.u.

Roma1 Speranza responsabile commissioning HLT: 2 m.u. Leandro chair IB, coordinatore slice mu: 1 m.u. ROD crate DAQ e HLT/DAQ muoni : 4 m.u.

Pasqualucci, Di Mattia, …


MilestonesMilestones 30/06/2005

TDAQ - Installazione, test e uso della "Pre-serie" (~ 10% TDAQ slice)

“ragionevolmente” raggiunta: ritardi accumulati soprattutto sugli acquisti delle componenti

24/12/2005 TDAQ - Installazione e test dei ROS di Pixel, LAr, Tile, Muon

(interfacciamento al ROD Crate e integrazione nel DAQ) Parte del piano di commissioning in esecuzione: piccola dipendenza dalla data

di consegna dei ROS 30/04/2006

Completamento dei test sulla pre-serie e definizione delle funzionalità per il supporto al commissioning TDAQ

31/08/2006 Commissioning delle slice di ROS dei rivelatori utilizzando le funzionalità

della pre-serie (modulo-0 del sistema finale) 31/12/2006

Presa dati integrata dei rivelatori nel pozzo con raggi cosmici

atlas hlt/daq stato e prospettive

Documents

vercesi infn pavias

commissioning generalev

il tdaq

small scale

final system

muoni barrel napoli

software releases

roma1daq lnf