1

A Log Service Package for the ATLAS TDAQ/DCS

Group

2006 Conference forComputing in High Energy and Nuclear Physics

13-17 Feb. 2006 Mumbai, India

Benedetto Gorini on behalf of Raul Murillo-Garcia

Conference for Computing in High Energy and Nuclear Physics13-17 February 2006, Mumbai

Raul Murillo-Garcia - CERN

2

Outline

Motivation Architectural overview Log Producers Log Manager Log Server Log Users Test Analysis Conclusions

Conference for Computing in High Energy and Nuclear Physics13-17 February 2006, Mumbai

Raul Murillo-Garcia - CERN

3

Motivation

Having a centralized mechanism whereby messages produced by TDAQ applications can be stored and later retrieved in an intuitive manner.

This is especially necessary for applications running in diskless and ‘consoless’ machines.

Conference for Computing in High Energy and Nuclear Physics13-17 February 2006, Mumbai

Raul Murillo-Garcia - CERN

4

Architectural overview

Conference for Computing in High Energy and Nuclear Physics13-17 February 2006, Mumbai

Raul Murillo-Garcia - CERN

5

Log Producers

Log Producers do the actual message logging via the ERS (Error Reporting System)

The ERS offers a unified system for handling error, warning and debug messages and provides the logging front-end for the TDAQ applications.

The Message Reporting System (MRS) provides the transport mechanism following the Producer-Server-Consumer scheme.

Conference for Computing in High Energy and Nuclear Physics13-17 February 2006, Mumbai

Raul Murillo-Garcia - CERN

6

Log Manager

The Log Manager is a PHP-based standalone tool that provides a web-based user-friendly interface to the Log Server database. • Filter – this is a mechanism whereby messages are

retrieved from the log server according to a qualifier-based criteria.

• Display – allows the user to view messages.• Local archive – permits the storage of messages to a

local file. • Delete – this option allows messages to be removed

remotely according to a filter. To prevent unwanted users from removing messages, the Log Service shall access the Access Manager system to restrict this option to authorized users only.

Conference for Computing in High Energy and Nuclear Physics13-17 February 2006, Mumbai

Raul Murillo-Garcia - CERN

7

Log Manager

Conference for Computing in High Energy and Nuclear Physics13-17 February 2006, Mumbai

Raul Murillo-Garcia - CERN

8

Log Manager

Conference for Computing in High Energy and Nuclear Physics13-17 February 2006, Mumbai

Raul Murillo-Garcia - CERN

9

Log Servers

The message archival is based on the MySQL DB.

An Apache HTTP server has been deployed to let Log Managers access the MySQL database.

Scalability has been taken into consideration to accommodate a multiple Log Server system. A logging policy based on the severity level of the message is deployed to distribute the messages.

Conference for Computing in High Energy and Nuclear Physics13-17 February 2006, Mumbai

Raul Murillo-Garcia - CERN

10

Log Servers

An application (Log Receiver) subscribes to the MRS server with a given criteria, archiving onto the database the messages received. An intermediate buffer is used to decouple the thread that receives the messages and the thread that does the actual logging. This buffer is a C++ List Container with the SQL INSERT queries as the entries.

Conference for Computing in High Energy and Nuclear Physics13-17 February 2006, Mumbai

Raul Murillo-Garcia - CERN

11

Log Users

A Log Service C++ API mirrors the functionality provided by the Log Manager. Remote access to the database is done through the MySQL library.

The API is intended for those applications in the TDAQ system that require access to the Log Service databases (Log Users). An example of such application could build histograms based on the number of errors occurred during a specific Run.

Conference for Computing in High Energy and Nuclear Physics13-17 February 2006, Mumbai

Raul Murillo-Garcia - CERN

12

Measurement taking: set up

Conference for Computing in High Energy and Nuclear Physics13-17 February 2006, Mumbai

Raul Murillo-Garcia - CERN

13

Test Results

Log Server 1: Pentium 4 – 2,8 GHz 256 MB RAM and Hard Disk PCI (~42 MB/sec)Transmission Rate: 2,000 msgs/sec.

0

500

1000

1500

2000

2500

3000

3500

0 45 91 136 182 227 273 318 364 409 454 500 545 591 636 682 728 773 819 864

Time (sec.)

Inco

min

g-I

nsert

Rate

(m

sg

s/s

ec)

0

100000

200000

300000

400000

500000

600000

700000

800000

900000

Bu

ffer

Occu

pan

cy

Incoming Rate (msgs/sec) Msg. Insert Rate (msgs/sec) Buffer Occupancy (msgs)

31

2

Behaviour of the low spec. Log Server for a transmission rate of 2,000 msgs/sec. This figure also shows the influence of the Log Manager (three red points). Insert Rate Upper Limit ~1,800 msgs/sec.

Conference for Computing in High Energy and Nuclear Physics13-17 February 2006, Mumbai

Raul Murillo-Garcia - CERN

14

Test ResultsBehaviour of the high spec. Log Server for a transmission rate of 3,000 msgs/sec. This figure also shows the influence of the Log Manager (three red points). Insert Rate Upper Limit ~3,800 msgs/sec.

Log Server 2: Dual CPU AMD 250–2,4 GHz with 4 GB RAM Hard Disk SATA (~85 MB/sec)Transmission Rate: 3,000 msgs/sec

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

0 30 60 90 121 151 181 212 242 272 303 333 363 393 424 454 484 515 545 575 606 636

Time (sec.)

Inco

min

g-I

nse

rt R

ate

(msg

s/se

c)

0

20000

40000

60000

80000

100000

120000

Bu

ffer

Occ

up

ancy

Incoming Rate (msgs/sec) Msg. Insert Rate (msgs/sec) Buffer Occupancy (msgs)

3

1 2

Conference for Computing in High Energy and Nuclear Physics13-17 February 2006, Mumbai

Raul Murillo-Garcia - CERN

15

Test results

The tests have hinted how influential the Log Server specifications are. Especially important is the amount of RAM available, as it is desirable to avoid swap ins/outs, which greatly slow the whole system.

The Log Manager has a visible impact on the Insert Rate, especially when the Transmission Rate is close to the Insert Rate upper limit.

The intermediate buffer is only intended to accommodate transitions during which the insert rate decreases, be it because the Log Manager is used or because the transmission rate is larger than normal.

Conference for Computing in High Energy and Nuclear Physics13-17 February 2006, Mumbai

Raul Murillo-Garcia - CERN

16

Conclusions

A Log Service has been developed to allow applications to log messages and users to browse/maintain them in an user-friendly manner.

Although the Log Server should be as powerful as affordable, one can also resort to use multiple less powerful Log Servers. The final implementation can only be devised once the scale of the information produced by the TDAQ system is known.

The Log Service has been released for general use in October 2005.