d.k. hasell future trends in nuclear physics computing ... · introduction triggered daq...
TRANSCRIPT
![Page 1: D.K. Hasell Future Trends in Nuclear Physics Computing ... · Introduction Triggered DAQ Traditional Triggered Data Acquisition Scheme Can not read out everything-event rate / size](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6186a8e57aa15a3569c1d/html5/thumbnails/1.jpg)
Streaming Readout
D.K. Hasell
Future Trends in Nuclear Physics ComputingThomas Jefferson National Accelerator Facility
March 16–18, 2016
![Page 2: D.K. Hasell Future Trends in Nuclear Physics Computing ... · Introduction Triggered DAQ Traditional Triggered Data Acquisition Scheme Can not read out everything-event rate / size](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6186a8e57aa15a3569c1d/html5/thumbnails/2.jpg)
Introduction Triggered DAQ
Traditional Triggered Data Acquisition Scheme
DetectorB
~5 µsPipeline
R/OBuffer ProcessorDigitize Readout
DetectorC
~5 µsPipeline
R/OBuffer ProcessorDigitize Readout
DetectorA
~5 µsPipeline
R/OBuffer ProcessorDigitize Readout
Level 1Trigger
Level 2Trigger
Level 3Trigger
10 MHz 100 kHz 1 kHz 100 Hz 5 Hz
100 kB
D.K. Hasell Streaming Readout March 17, 2016 2 / 25
![Page 3: D.K. Hasell Future Trends in Nuclear Physics Computing ... · Introduction Triggered DAQ Traditional Triggered Data Acquisition Scheme Can not read out everything-event rate / size](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6186a8e57aa15a3569c1d/html5/thumbnails/3.jpg)
Introduction Triggered DAQ
Traditional Triggered Data Acquisition Scheme
Can not read out everything
- event rate / size greater than what can be handled in time
- must reduce rate by discarding background and noise events
- and / or selecting “interesting” events
Identifying “interesting” events is the challenge
- timing with respect to beam crossing very useful
- indications of energy in calorimeter, potential track, muon signal, . . .
- tracks coming or not coming from interaction point, . . .
Problems with this approach
- often special detectors, hardware, and electronics to provide a trigger
- decision based on a subset of the data from detector
- based on a posteriori knowledge, looking for what you expect
D.K. Hasell Streaming Readout March 17, 2016 3 / 25
![Page 4: D.K. Hasell Future Trends in Nuclear Physics Computing ... · Introduction Triggered DAQ Traditional Triggered Data Acquisition Scheme Can not read out everything-event rate / size](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6186a8e57aa15a3569c1d/html5/thumbnails/4.jpg)
Introduction Red Light Trap
Red Light Camera Trap
D.K. Hasell Streaming Readout March 17, 2016 4 / 25
![Page 5: D.K. Hasell Future Trends in Nuclear Physics Computing ... · Introduction Triggered DAQ Traditional Triggered Data Acquisition Scheme Can not read out everything-event rate / size](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6186a8e57aa15a3569c1d/html5/thumbnails/5.jpg)
Introduction Red Light Trap
Might Miss the Unexpected
D.K. Hasell Streaming Readout March 17, 2016 5 / 25
![Page 6: D.K. Hasell Future Trends in Nuclear Physics Computing ... · Introduction Triggered DAQ Traditional Triggered Data Acquisition Scheme Can not read out everything-event rate / size](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6186a8e57aa15a3569c1d/html5/thumbnails/6.jpg)
Introduction LHC
The Energy Frontier
LHC - run 2 - now - luminosity = 1034 cm−2s−1
- ∼ 20 pile-up events per beam crossing, 40 MHz rate or 25 ns
- hardware level 1 trigger reduce rate to 100 kHz, decision in ∼ 2 µs
- 20,000 cores high level trigger to 1 kHz in ∼ 200 ms
LHC - run 4 - 2026 - luminosity = 1035 cm−2s−1
- ∼ 200 pile-up events per beam crossing, every 25 ns
- level 0 trigger to 1 MHz in ∼ 6 µs
- level 1 trigger to 400 kHz in ∼ 30 µs
- high level trigger to 10 kHz
Major changes to meet upgraded luminosity
Moral: Assume the luminosity will be upgraded
D.K. Hasell Streaming Readout March 17, 2016 6 / 25
![Page 7: D.K. Hasell Future Trends in Nuclear Physics Computing ... · Introduction Triggered DAQ Traditional Triggered Data Acquisition Scheme Can not read out everything-event rate / size](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6186a8e57aa15a3569c1d/html5/thumbnails/7.jpg)
Introduction EIC
EIC - Event Rates assuming 1033 cm−2s−1
D.K. Hasell Streaming Readout March 17, 2016 7 / 25
![Page 8: D.K. Hasell Future Trends in Nuclear Physics Computing ... · Introduction Triggered DAQ Traditional Triggered Data Acquisition Scheme Can not read out everything-event rate / size](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6186a8e57aa15a3569c1d/html5/thumbnails/8.jpg)
Introduction EIC
EIC - nominally 1033–1034 cm−2s−1 - but ?
D.K. Hasell Streaming Readout March 17, 2016 8 / 25
![Page 9: D.K. Hasell Future Trends in Nuclear Physics Computing ... · Introduction Triggered DAQ Traditional Triggered Data Acquisition Scheme Can not read out everything-event rate / size](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6186a8e57aa15a3569c1d/html5/thumbnails/9.jpg)
Introduction Intensity Frontier
Intensity Frontier - DarkLight
Will run at JLab on the LERF
- 6× 1035 cm−2s−1
- current design ∼ 80, 000 channels, may be drastically reduced
- 4 TB/s
- possible to also run at Cornell ERL at 10× L
D.K. Hasell Streaming Readout March 17, 2016 9 / 25
![Page 10: D.K. Hasell Future Trends in Nuclear Physics Computing ... · Introduction Triggered DAQ Traditional Triggered Data Acquisition Scheme Can not read out everything-event rate / size](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6186a8e57aa15a3569c1d/html5/thumbnails/10.jpg)
Introduction Computer Performance
Advances in Computer Performance
- http://www.cs.columbia.edu/ sedwards/classes/2012D.K. Hasell Streaming Readout March 17, 2016 10 / 25
![Page 11: D.K. Hasell Future Trends in Nuclear Physics Computing ... · Introduction Triggered DAQ Traditional Triggered Data Acquisition Scheme Can not read out everything-event rate / size](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6186a8e57aa15a3569c1d/html5/thumbnails/11.jpg)
Introduction Computer Performance
Advances in Computer Performance
Single core performance advances may be slowing down ?
- was driven by improved processes, architecture, organisation, . . .
- then limited by power, memory latency, I/O speed, . . .
- cache speeds didn’t keep up with CPU speeds
But computer performance will continue to improve ×1000 by 2022 ?
- improvements in production process
- multi-core processors, MIC, . . .
- on core DRAM
- solid state disks
Leverage computer performance/$ and disk space/$ for physics
D.K. Hasell Streaming Readout March 17, 2016 11 / 25
![Page 12: D.K. Hasell Future Trends in Nuclear Physics Computing ... · Introduction Triggered DAQ Traditional Triggered Data Acquisition Scheme Can not read out everything-event rate / size](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6186a8e57aa15a3569c1d/html5/thumbnails/12.jpg)
Streaming Readout Streaming Readout Concept
Streaming Readout Concept
DetectorB
DataMerge
FEE
DetectorC
DataProc.
FEE CPU
DetectorB FEE
DetectorA FEE
DetectorA FEE
DataProc.
DataProc.
DataProc.
DataProc.
CPU
CPU
CPU
CPU
DataMerge
Switch
DataDistr.
DataDistr.
DataDistr.
Clock
SC
D.K. Hasell Streaming Readout March 17, 2016 12 / 25
![Page 13: D.K. Hasell Future Trends in Nuclear Physics Computing ... · Introduction Triggered DAQ Traditional Triggered Data Acquisition Scheme Can not read out everything-event rate / size](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6186a8e57aa15a3569c1d/html5/thumbnails/13.jpg)
Streaming Readout Streaming Readout Components
Streaming Readout Components
Front-End Electronics
- specific for each detector type and requirements
- pre-amp, amplifier, shaper / filter, discriminator, . . .
- FADC - sampling rate and precision as needed
- can be somewhat removed from other components- accepts multiple channels as input- suppresses channels with no signal- streams multiplexed output to data processing unit
Connection to data processing
- ethernet, infiniband, or special protocol
- copper or optical fiber
- optical fiber might provide electrical isolation
- digital data so electronics can be remote from detector
- outside radiation environment, accessible, maintainable
D.K. Hasell Streaming Readout March 17, 2016 13 / 25
![Page 14: D.K. Hasell Future Trends in Nuclear Physics Computing ... · Introduction Triggered DAQ Traditional Triggered Data Acquisition Scheme Can not read out everything-event rate / size](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6186a8e57aa15a3569c1d/html5/thumbnails/14.jpg)
Streaming Readout Streaming Readout Components
Streaming Readout Components
Data processing unit - FPGA based
- accepts signals from several FEE
- de-multiplexes signal into individual channels- FPGA processes each channel to extract data
- leading edge or CF timing, time above threshold, . . .- amplitude, total charge, . . .
- maintains histograms of data for each channel
- adds time stamp and channel ID
- streams channels with valid data (more zero suppression)
Clock, slow control, and calibration channels
- all units synchronized with a common clock pulse for time stamp- 2-way slow control communication with master DAQ system
- sends histograms periodically- accepts threshold or pedestal settings for each channel- can apply calibration to the raw data
D.K. Hasell Streaming Readout March 17, 2016 14 / 25
![Page 15: D.K. Hasell Future Trends in Nuclear Physics Computing ... · Introduction Triggered DAQ Traditional Triggered Data Acquisition Scheme Can not read out everything-event rate / size](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6186a8e57aa15a3569c1d/html5/thumbnails/15.jpg)
Streaming Readout Streaming Readout Components
Streaming Readout Components
Data distribution module - FPGA based
- accepts signals from several data processing units
- sorts data according to time stamps into time windows
- sends data to different event reconstruction CPUs
- based on time window and beam crossing- time windows may overlap
Reconstruction CPUs
- collects all data from all detector for given time window
- analyses data from all detectors to accept or reject
- if accepted can build event and save
- otherwise move on to the next available time window
D.K. Hasell Streaming Readout March 17, 2016 15 / 25
![Page 16: D.K. Hasell Future Trends in Nuclear Physics Computing ... · Introduction Triggered DAQ Traditional Triggered Data Acquisition Scheme Can not read out everything-event rate / size](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6186a8e57aa15a3569c1d/html5/thumbnails/16.jpg)
Streaming Readout Alternative Streaming Readout Concept
Alternative Streaming Readout Concept
DetectorB FEE
DetectorC
DataProc.
FEE CPU
DetectorB FEE
DetectorA FEE
DetectorA FEE
DataProc.
DataProc.
DataProc.
DataProc.
CPU
CPU
CPU
CPU
Switch
CPU
CPU
CPU
Clock
SC
D.K. Hasell Streaming Readout March 17, 2016 16 / 25
![Page 17: D.K. Hasell Future Trends in Nuclear Physics Computing ... · Introduction Triggered DAQ Traditional Triggered Data Acquisition Scheme Can not read out everything-event rate / size](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6186a8e57aa15a3569c1d/html5/thumbnails/17.jpg)
Streaming Readout Alternative Streaming Readout Concept
Alternative Streaming Readout Concept
Replace data merge and data distribution nodes
- replace with computers or clusters with disk space
- specific for a given detector
- can begin reconstruction and sorting within time windows
- stores data locally for each time window
- provides data upon request to event building CPUs
- can share data with other detector CPUs
- discards data if “sell by date” expired
D.K. Hasell Streaming Readout March 17, 2016 17 / 25
![Page 18: D.K. Hasell Future Trends in Nuclear Physics Computing ... · Introduction Triggered DAQ Traditional Triggered Data Acquisition Scheme Can not read out everything-event rate / size](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6186a8e57aa15a3569c1d/html5/thumbnails/18.jpg)
Streaming Readout Streaming Readout Issues
Streaming Readout Issues
Readout scheme strongly coupled to detector technology
- FEE and Data Processor tailored to detector and its requirements
- FPGAs non-trivial to program
- code also specific to detector and requirements
- need a basic design that can be applied to several detectors
- change sampling rate, number of bits, etc. as needed- standardize code to obtaining timing, ADC, QDC, . . .
Detector must also be designed with streaming readout in mind
- detector can not require a trigger to start readout
- GEM, micromegas with APV, DREAM ASICs readout need a trigger
D.K. Hasell Streaming Readout March 17, 2016 18 / 25
![Page 19: D.K. Hasell Future Trends in Nuclear Physics Computing ... · Introduction Triggered DAQ Traditional Triggered Data Acquisition Scheme Can not read out everything-event rate / size](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6186a8e57aa15a3569c1d/html5/thumbnails/19.jpg)
Hardware Commercial
Hardware
Front-End Electronics
- currently in contact with an ASIC manufacturer
- 64+ channels, 40 MSPS, 14 bit ADC
- built-in gain and shaping controls
- zero suppression on chip
- high bandwidth serial transceiver lanes
Data processing cards exist commercially- CAEN interested in collaborative development
- desktop, NIM, and VME FPGA modules exist- 64 channel, 62.5 MSPS, 12 bit- software for peak sensing ADC, QDC, and TDC measurements
- CAEN user programmable FPGA logic module- up to 192 input channels- up to 130 output channels- 32 independent gate and delay generators
D.K. Hasell Streaming Readout March 17, 2016 19 / 25
![Page 20: D.K. Hasell Future Trends in Nuclear Physics Computing ... · Introduction Triggered DAQ Traditional Triggered Data Acquisition Scheme Can not read out everything-event rate / size](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6186a8e57aa15a3569c1d/html5/thumbnails/20.jpg)
Hardware National Laboratories
DAQ / Trigger Development at National Laboratories
SLAC - ATCA based DAQ and control systems
- RCE (Reconfigurable Cluster Element)
- plug-in architecture for applications
- firmware and software development kits
- based upon Xilinx Zynq platform
- full mesh 10G network
- 96 high speed back end links
- in use at DUNE, ATLAS, HPS, . . .
- modified HPS board at MIT for DarkLight evaluation
FNAL- CAPTAN+X - compact and programmable data acquisition
- Xilinx Virtex 7- 10 GB ethernet
- CAPTAN+X used in off-the-shelf DAQ system for test stations
D.K. Hasell Streaming Readout March 17, 2016 20 / 25
![Page 21: D.K. Hasell Future Trends in Nuclear Physics Computing ... · Introduction Triggered DAQ Traditional Triggered Data Acquisition Scheme Can not read out everything-event rate / size](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6186a8e57aa15a3569c1d/html5/thumbnails/21.jpg)
Hardware National Laboratories
DAQ / Trigger Development at National Laboratories
BNL
- not focusing on a single solution
- develop solutions for each experiment’s needs
- use commercial, off-the-shelf components
- energy frontier
- ATLAS - LAr calorimeter (digitizer), MNSW (FE ASIC), TDAQ
- intensity frontier
- MicroBooNE (FEE), SBND (TPC R/O), DUNE (cold ASIC)
- capability to produce ASIC as well as full boards useful
ANL ? JLab ? . . .
Would be nice if the national labs worked together and standardized
D.K. Hasell Streaming Readout March 17, 2016 21 / 25
![Page 22: D.K. Hasell Future Trends in Nuclear Physics Computing ... · Introduction Triggered DAQ Traditional Triggered Data Acquisition Scheme Can not read out everything-event rate / size](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6186a8e57aa15a3569c1d/html5/thumbnails/22.jpg)
Hardware Crates Standards
Crates Standards
NIM
CAMAC
FastBus
VME
- lots of crates still in use, large number of manufacturers- will stay as a legacy system for decades- parallel, asynchronous bus but VME320 only 320 MB/s- protocol easy to implement in FPGA- limited power and cooling- VXI, VPX similar
VXS
- compatible with VMS but has 10 GB agnostic differential linesD.K. Hasell Streaming Readout March 17, 2016 22 / 25
![Page 23: D.K. Hasell Future Trends in Nuclear Physics Computing ... · Introduction Triggered DAQ Traditional Triggered Data Acquisition Scheme Can not read out everything-event rate / size](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6186a8e57aa15a3569c1d/html5/thumbnails/23.jpg)
Hardware Crates Standards
Advanced Telecommunications Architecture - ATCA
Driven by telecom / internet industry
- Amazon, NetFlix, Google, . . .
- cell phones
Crate standard
- -48 V power, zone 1
- redundant cooling
- crate management
- 40 GB backplane, zone 2
Blade standard
- 400 W per blade
- hot swapable
- RTM, zone 3
D.K. Hasell Streaming Readout March 17, 2016 23 / 25
![Page 24: D.K. Hasell Future Trends in Nuclear Physics Computing ... · Introduction Triggered DAQ Traditional Triggered Data Acquisition Scheme Can not read out everything-event rate / size](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6186a8e57aa15a3569c1d/html5/thumbnails/24.jpg)
Summary
Summary
Triggered and streaming readout systems not so different
- triggered systems discard events early based on subset of data
- goal of streaming systems is to look at all the data
- barring some hardwired electronic trigger both rely on computations
Performance improvements make streaming readout more accessible
- faster FPGAs, multi-core computers, disk access, and networks
- motivation to design for the future technology not the past
Programming a streaming readout may be simpler than the alternative
D.K. Hasell Streaming Readout March 17, 2016 24 / 25
![Page 25: D.K. Hasell Future Trends in Nuclear Physics Computing ... · Introduction Triggered DAQ Traditional Triggered Data Acquisition Scheme Can not read out everything-event rate / size](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6186a8e57aa15a3569c1d/html5/thumbnails/25.jpg)
Summary
Thank You
DetectorB FEE
DetectorC
DataProc.
FEE CPU
DetectorB FEE
DetectorA FEE
DetectorA FEE
DataProc.
DataProc.
DataProc.
DataProc.
CPU
CPU
CPU
CPU
Switch
CPU
CPU
CPU
Clock
SC
D.K. Hasell Streaming Readout March 17, 2016 25 / 25