research seminar, fermi national accelerator laboratory november 2005 signal and image processing at...

Research Seminar, Fermi National Accelerator LaboratoryNovember 2005

Signal and Image Processing atLAPI / POLITEHNICA

1990 - 2005

Vasile Buzuloiu

LAPI stands for“LABORATORUL DE ANALIZA SI PRELUCRAREA

IMAGINILOR”(Image Processing and Analysis Laboratory)

from the FACULTY OF ELECTRONICS, TELECOMMUNICATIONS and INFORMATION TECHNOLOGY of theUNIVERSITATEA POLITEHNICA BUCURESTI (UPB)


The expertise we have at LAPI in the field of (uni-and-multidimensional) signal processing and statistical analysis as well as in software and algorithm development has allowed us a welcomed participation in TDAQ second level triggering for the future ATLAS experiment of LHC at CERN


LAPI pre-history (1975 - 1985)

• involvement in building image analysis systems from scratch (including video ADC/DAC, memories etc)• applications of our systems : remote sensing maps analysis, geological microscopic image analysis, dental X-ray image analysis, image compression methods)• an award of the Romanian Academy crowned the activity and also stopped it


LAPI history (1990 - 2005)old age (1990 - 2000)

Involvement at CERTN in R&D activity for the future detectors of LHC in the framework of the LAA project (led by Prof A. Zichichi) more specifically in the group of Dr. R.K. Bock and W Krischer. Also collaboration with Dr. A. Marchioro for the first level triggering.


LAPI history (1990-2005)old age (1990- 2000) (cont.)

Some results:• The ELOISATRON international workshop (in Erice) dedicated to "Image processing for future High Energy Physics Detectors“ (Nov 1991) (V. Buzuloiu director of the workshop and editor of the proceedings)• Fast and precise peak finder for the first level triggering (published in the proceedings of the CHEP 1992 - Annecy)• 4 participants from UPB over various periods between 1993 and 2000


LAPI history (1990-2005)new age (2000 - 2005)

The LAPI team starts cooperating with the group of Prof. Bob Dobinson - B. Martin working on:• High speed networking (10 Gb/s Ethernet) and long haul intercontinental connections• Quality degradation in networks• TDAQ for the Second Level Triggering of ATLAS


LAPI history (1990-2005)new age (2000- 2005) (cont.)

Research and teaching home and elsewhere :

• cooperation with CERN on the same subjects• research on various themes of image processing and analysis: filtering, enhancement, image data bases with content indexing, color and multi-component image processing, head tracking, eye tracking, digital photography tools, medical image analysis etc.


LAPI history (1990-2005)new age (2000- 2005)

Research and teaching home and elsewhere : (cont.)

• R&D in cooperation with foreign universities (Ireland, France, Germany, Italy). Exchange of professors and students• R&D in cooperation with foreign companies as e.g. for digital photography tools and eye tracking• A yearly international postgraduate, intensive school on multidimensional signal processing and analysis


The ATLAS TDAQ System

ATLASData Collection

L2PUL2PUL2PUL2PUSFISFI SFI

EF EF EF

ROBROBROBROB

Data Collection Network

Back End Network

SFOs

Massstorage

Detector buffers

ATLAS detector

~40 MHz

~100 kHz

~3 kHz

~200 Hz300MBps

~60 TBps

~150 GBps

~4.5 GBps

Event size ~= 1.5MBytes

REF

Level 1 Trigger


ATLAS DataCollection networkR OBIN

PC I

PC I

PC I

R OBIN

PC I

PC I

PC I

R OBIN

PC I

PC I

PC I

R OBIN

PC I

PC I

PC I

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

R OB

LVL2 PUs

LVL2 PUs

BackEndNetwork

3

3

136 to ROSes42 to L2PU farms46 to SFIs2 to Control switch

Total28 L2PU conc. switches

500 L2PUs92 SFIs

LVL2 PUs

LVL2 PUs

BackEndNetwork

3

3

1600 ROLsDetector (ROLs - ReadOutLinks)

136 ROSes

12 ROLs/ROS

Central Control50 GE ports 228 GE ports

24 GE ports3 up-links

18 L2PUs/switch

2 2

SFIspROSSVs(10)

SFIsDFM (35)


DataCollection componentsROB 1600 Read Out Buffer. Stores the event fragments validated by the Level1 Trigger.

RobIn 534 PCI card with 1 GE interface and input from 3 ROLs (Read Out Links). The ROLs feed validated event data to the ROBs.

ROS (sb) 20 Read Out System. Industrial PCs housing 10 RobIns. The RobIns are configured and monitored via PCI, while the data is read out via GE.

ROS (bb) 136 Read Out System. High end PCs housing 4 RobIns. The RobIns are configured, monitored and read out via PCI. Data is shipped further by the PC via 2 GE interfaces.

L2SV 10 Level 2 Supervisor. Supervises a subset of L2PUs, including algorithms for even load balancing between the L2PUs.

L2PU 500 Level 2 Processing Unit. Runs physics algorithms on a subset of the event data (RoI) indicated by the Level 1 trigger, and validates/invalidates events.

PROS 1 Pseudo ROS (Read Out System). Temporarily stores the result of the L2 analysis for validated events.

DFM 35 Data Flow Manager. Assigns SFIs for building events using a load balancing algorithm.

SFI 92 Sub Farm Interface. The events validated by L2 are built together inside the SFI, from all the ROSes.


No packet loss!

• The main data flow is based on a request–response mechanism.•Message loss implies time-outs and retries at the application level, having a great penalty on its performance. •TCP guarantees no message loss even if packet loss occurs in the network, but…

•TCP does not scale with large number of connections (~600).•TCP is CPU intensive.•TCP increases the packet rate in the network.

•UDP is the preferred protocol for the main dataflow. Packet loss in the network translates into message loss, therefore:

•Special care should be taken in order to minimize packet loss.•We must avoid multicast/broadcast traffic, as its rate is not guaranteed on all switches.


Understanding congestion

• Paper calculation based on queuing theory:•safe average link utilization as function of buffer size.

•Measurements on switches from several manufacturers with various traffic patterns:

•Throughput / packet loss•Latency•Multicast/broadcast handling.

•Measurements performed on a 10-20% size system.•Computer modeling of the system:

•Model validation by crosschecking with measurements•Model prediction for full size system


Contribution to the ATLAS TDAQ system• The issues related to the use of Ethernet as network technology have been analyzed and tested on several Ethernet switching devices.•Gigabit Ethernet traffic generators have been programmed to emulate the output from the level one trigger.•The large number of these devices allowed us to optimize not only the network topology, but also the traffic patterns and traffic shaping techniques inside the system.•Thus the performance of the DataFlow has been improved in several steps, and the scalability of the system has been proved up to one tenth of its final size.•The experience acquired in running the DataFlow system enabled us to derive the requirements for the networking equipment needed in the final system. •The Gigabit Ethernet traffic generators are currently used in evaluating the performance of sample devices from the major manufacturers.


The GETB platform• Atlas TDAQ DataFlow network

•Large high-speed Layer 2 network (~700 nodes) •Central switches: ~250 ports, chassis based•Concentrating switches: 24 - 48 ports, pizza box•Packet loss or excessive latency performance drop

•Specific performance requirements for DataFlow switches•Need to evaluate devices with realistic test scenarios•See the “Switch Features” document

•Testing equipment on the market is not fully adequate•Cost per channel is high•Not enough flexibility in defining traffic patterns•Layer 4 - 7 functionality – not essential for Atlas


GETB Platform – Hardware

•FPGA Firmware•90% Handel-C, 10% VHDL

•Commercial IP cores (IP = Intellectual Property)•Gigabit Ethernet MAC•PCI Controller

•Logic utilization – approx 85 - 90%•Single FPGA controls 2 Eth ports

•Multiple projects using the GETB•GETB Tester, Network Emulator, ROS Emulator•Common firmware / control infrastructure

GPS Clock

2 x Gigabit Ethernet

3.3V PCI

Configuration Flash

SDRAM 2 x 64Mb

SRAM 2 x 512Kb

Altera Stratix EP1S25 FPGA


GETB Platform – Control

•Server (runs on Linux)•Configures the cards•Monitors status / collects statistics•Handles remote client requests•Accesses cards using IO-RCC (DataFlow package)

•Client (runs on any platform)•Talks to servers using XML-RPC•Runs user-defined scripts•Displays statistics in a GUI•Manages cards from multiple servers

GETB Client(Control PC)

Device Under Test

(DUT)

GETB Servers

GPS Clock Distribution

Gigabit Ethernet Links

• Distributed System– 15 servers hosting 65 cards– Entirely based on Python

DUT Control


GETB platform - summary

• The GETB platform provides a flexible environment for the design and development of Gigabit Ethernet applications•A tester able to evaluate switches for the DataFlow network has been created

•128 ports running at line-speed•Client-server traffic emulation provides a way to test devices under realistic Atlas conditions

•Evaluating the device in a worst-case scenario•We have a comprehensive set of test procedures to check that devices meet our requirements


Quality degradation in networks

• Measure the quality degradation introduced by the building blocks of any network (i.e. switches, routers):

• emergent properties of traffic differentiation and scheduling mechanisms• service fairness for traffic flows of same class/priority

• Study the relationship between quality degradation in networks and application-level user-perceived quality

• for Internet Telephony (VoIP) applications• for bulk data transfers (FTP or TCP/IP)


Network Emulation• A high-performance network emulator is currently being built by two LAPI researchers, in collaboration with Predictable Network Solutions, Ltd. :

• capable of reproducing realistic network conditions (correlated loss and delay, natural induced delay variation)• implemented on a custom-design FPGA-based platform

• Emulate the quality degradation likely to appear in large networks• emulate the presence of background traffic as vacations of the service facility• use a “degradation algebra” to aggregate basic models (of queues and wires) into one single model


Long-distance networking• Transmission of native 10 Gigabit Ethernet over the

installed base of long-distance telecommunication networks – through the ESTA EU project– First transatlantic connection based on the 10 GE WAN PHY

technology: Oct. 2003, Geneva-Ottawa, live demo at the ITU Telecom World Conference

• Studies for the use of remote computing farms, in real time, in the ATLAS TDAQ system– Proof of concept in 2004, on a testbed located in Switzerland,

Canada, the United Kingdom, Denmark and Poland • the ATLAS online applications support a remote farm scenario • the network connections can assure the bandwidth for this particular

application


High-speed signal integrity

• Studies of signal integrity over high-speed backplanes for telecommunication equipment – the ESTA EU project– Simulating the behavior of different materials and

connectors through mathematical models– Developed the software to control and operate a

backplane tester device– Measurements on real hardware and reconciliation

with the simulation models


research seminar, fermi national accelerator laboratory november 2005 signal and image processing at...

Documents