OMIS Approach to Grid Application Monitoring

Bartosz Baliś

Marian Bubak

Włodzimierz Funika

Roland Wismueller

X#

AGENDA

Introduction Monitoring architecture

– sensors (local monitors, application monitors)– service managers

Performance– efficient data gathering– scalability of grid-scale monitoring

Producer / consumer communication protocol Comparison to DATAGRID Experience Conclusion

X#

Introduction

Need for monitoring applications– improve performance– localize bugs

For these purposes – specialized tools needed– debuggers, performance analyzers, visualizers, etc.

Tools composed of two modules– user interface– monitoring module

X#

Introduction (cont’d)

Main issues of monitoring on Grid– scale of Grid enormous– many applications, many users, high distribution, high

heterogeneity– simply porting existing environments not sufficient!

A solution:– underlying universal monitoring system– well defined interface to tools

Experience with OMIS / OCM: PVM MPI, port of tools

– next step – move to Grid?

X#

Monitoring architecture

Compliance with GMA (Grid Monitoring Architecture)– producer / consumer model

Sensors – producers of performance data Tools – consumers of the data Direct communication between producers and

consumers Producers located via e.g. a directory service

X#

Sensors

Collect performance data from applications Two types of sensors

– local monitors (process sensors)– application monitors

X#

Sensors (cont’d)

Local monitors– one per node– collect data only from processes on this node– publish themselves in the directory service

Application monitors– embedded parts of applications– collect data on various events, e.g. function calls– may improve efficiency and portability– interact with local monitors

X#

Monitoring Architecture

X#

Service managers

Tool + local monitors – one consumer, multiple producers

Intermediate entity: service manager– handles requests coming from a tool– splits them into sub-requests for local monitors– collects replies from local monitors– assembles them into a single reply for the tool

Both producer (of data for tools) and consumer (of data from local monitors)

Offers the functionality of local monitors but on a per-application basis

X#

Application Monitors

Part of the monitoring system embedded in the application’s processes– have acces to the application address space!

Many possible usages– efficient data gathering and storing– may take over some of the local monitor’s tasks– may be used to dynamically load monitoring

extensions– even more for multithreaded applications

X#

Application Monitors – debugging example

A debugger wants to access a process’ address space

Standard system mechanisms: ptrace, /proc– /proc more powerful yet platfom-dependant– synchronous control

Via application monitors request from the debugger to access the data– portable, asynchronous– question: how to ensure that application monitors

are not corrupted by the application?

X#

Performance

Efficient data gathering– data production much more frequent than retrieval– frequency and time of access – difficult to predict

Scalability– grid-scale monitoring system– distributed vs. centralized

X#

Efficient data gathering

Local storing– performance data first stored locally, in the context

of application processes– on request, passed to local monitors– saves communication and context switches

between application and local monitor processes Efficient data structures

– performance data initially preprocessed– summarized information stored in e.g. counters and

integrators

X#

Scalability

Decentralization multiple service managers instead of one

Possible approaches– fixed number of service managers, each responsible

for part of the system– one service manager starting for every monitored

application

X#

Fixed number of SMs

X#

One SM per application

X#

Scalability (cont’d)

In the first approach– more tight cooperation between service managers

will be necessary

In the second approach– local monitors must have the ability to serve multiple

service managers– service managers locate local monitors via directory

service

X#

Communication protocol

Based on the OMIS specification OMIS = On-line Monitoring Interface

Specification– specification of a universal interface between tools

and a monitoring system– supports various types of tools– allows for easy extending

Necessary Grid-specific extensions (e.g. for authentication)

X#

Comparison to DATAGRID

Monitoring approach– DG: (semi-)on-line– CG: on-line

Architecture– DG: centralized distributed (local monitors and one

main monitor)– CG: distributed (local monitors and multiple service

managers)

X#

Comparison to DATAGRID (cont’d)

Data collection– DG: local storing with trace buffering or counters– CG: local storing with preprocessing (counters,

integrators)

Communication protocol– DG: Not specified– CG: OMIS

X#

Experience

OMIS-based monitoring system for clusters of workstations – OCM

OMIS-based tools – PATOP (performance analysis), DETOP (debugging), others...

Local storing and efficient data structures (counters and integrators) proved to be very efficient– full monitoring overhead of about 4%

Instrumentation techniques used induce zero-overhead when monitoring inactive

X#

Summary

Demand for accurate data from monitoring tools

Monitoring data handling: production / consumption

A general scheme of monitoring compliant with GMA

Need of an advanced monitoring infrastructure Concepts of OMIS will be extended to fit Grid