computer and automation research institute hungarian academy of sciences presentation and analysis...

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of Sciences

Presentation and Analysis of Grid Performance Data

Norbert Podhorszki and Peter Kacsuk

Laboratory of Parallel and Distributed SystemsMTA SZTAKI

Hungary

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of SciencesContents

● Relational Grid Information and Monitoring System(R-GMA)

● R-GMA Browser– Browsing status information

● Pulse– Analysis and presentation of performance information of grid

services and components

● GRM/PROVE– Performance monitoring of applications

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of SciencesR-GMA

● R-GMA is the Relational Monitoring and Information System within the EU-DataGrid project

● Gives the illusion of a virtual relational database of the information available about grid services, resources and jobs

● SQL is the information publishing and query language

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of Sciences

R-GMA

Sensor Code

ProducerAPI

ApplicationCode

ConsumerAPI

Producer Servlet

RegistryAPI

Registry

SchemaAPI

Schema

“Definition of tables”

Consumer Servlet

RegistryAPI

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of SciencesR-GMA

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of Sciences

R-GMA

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of SciencesPulse

● On-line analysis and visualisation tool for

● monitoring data of resources and services and applications.

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of SciencesPulse

● Data source components preprocessing components

view components their connections are defined in XML

● Separates data model from visualisation modules

● based on Model-View-Controller paradigm

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of SciencesPulse

● Data model: meta-data (name, type, unit, range) and interface to the data source.

● Controller: preprocessing, analysis components

● View: displaying components

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of SciencesExample: CPU Load

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of SciencesExample: CPU Load

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of SciencesExample: CPU Load

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of SciencesData model

● Data channel

– Name

– Data type

– Unit

– Range of valid data

LoadChannelMetaData(

name=”load1”,

type=float,

unit=””,

range=[0.0-100.0+0.01] )

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of SciencesInput components

● Input component (sensor): an interface to some kind of external data source (file, service, an executable, etc.)

● A sensor– Reads data from the external source– Translates data into a channel based representation– Publishes complex information in several channels– Provides one piece of information at a time

● Buffered sensor– Buffering capability to a sensor

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of SciencesPresentation components

● To show the results to the user

● Tabular view, Histogram plotter

● Presentation components subscribe to the sensor's notification service ->they are notified about the status changes in the sensor ->display updates are controlled by the sensors (incoming data)

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of SciencesAnalysis components

● Essentially:

– Take data from other components (sensors)

– Do something with the data

– Forward it to other components (plotter)

● E.g.

– Channel selector (for histogram plotter)

– Merge data from several sensors

– Compute statistical values on data

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of SciencesConfiguration of Pulse

● XML configuration to define

– Components

– Connections between the components

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of SciencesPulse: conclusion

● Pulse can be used to build an analysis chain to process dynamic information and present results in a graphical form

● Flexible data representation

● Already it can be used as a tool for the R-GMA grid monitoring and information system

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of SciencesApplication monitoring

● Performance monitoring (not job status!) of

● Parallel applications (MPI, PVM, P-GRADE)

● Large volume of trace information

● On-line transfer of trace to the user

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of SciencesGRM at EuroPar’2001

Main MonitorMM

Application Process

Application Process

Site 1

Local Host

Host 1 Host 2 Host 1

Local MonitorLM

Local MonitorLM

Site 2

Local MonitorLM

Site MonitorSM

Site MonitorSM

Appl. Process

Appl. Process

shm shm shm

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of SciencesTools

● GRM– Instrumentation library and trace collector

● Mercury monitor– Transfer trace data from the application processes

to GRM

● R-GMA– Information system for finding the application and

the monitor components

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of SciencesMercury monitor

in the GridLab project

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of SciencesGRM+Mercury+R-GMA

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of SciencesMercury monitor

● To deliver trace data from the application to the user efficiently.

– Uses TCP Socket communication

– Data in XDR format and could be optimised for TCP transmission

– Two sw. hops between application and GRM: local and main monitors

– One hw. hop: host of main monitor

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of SciencesR-GMA

● To find the application by the user/GRM

– Where is it running? -> machineX.siteY

– What is its global job id? -> GID

● To find the monitor to be connected

– What is the address of Mercury monitor running at siteY?

● To find the application by the monitor

– What processes (PIDs) belong to application GID?

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of SciencesGRM

● Instrumentation library

– Connect and publish trace to Mercury monitor

● GRM and PROVE

– Look for application asking R-GMA

– Connect to Mercury monitor and query for trace

– Visualise trace information with PROVE

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of SciencesCurrent status

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of SciencesFuture work

● Intrusion: replace socket communication between LM and application process for shared-memory buffering (as in original GRM)

● Scalability: data reduction at LMs and MM.

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of SciencesConclusion

● Different monitoring information can be presented with different techniques

– Web browser is used to search among available information and static (or slowly changing) behaviour of the grid

– Pulse can be used to build specific analysis chains to process dynamic information about the services and resources and to create graphical plots

– GRM is a tool for performance monitoring of parallel applications, connected to Mercury and R-GMA to work for grid.

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of Sciences

Thank you

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of SciencesR-GMA Schema Browser

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of SciencesR-GMA Schema Browser