computer and automation research institute hungarian academy of sciences presentation and analysis...
TRANSCRIPT
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of Sciences
Presentation and Analysis of Grid Performance Data
Norbert Podhorszki and Peter Kacsuk
Laboratory of Parallel and Distributed SystemsMTA SZTAKI
Hungary
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesContents
● Relational Grid Information and Monitoring System(R-GMA)
● R-GMA Browser– Browsing status information
● Pulse– Analysis and presentation of performance information of grid
services and components
● GRM/PROVE– Performance monitoring of applications
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesR-GMA
● R-GMA is the Relational Monitoring and Information System within the EU-DataGrid project
● Gives the illusion of a virtual relational database of the information available about grid services, resources and jobs
● SQL is the information publishing and query language
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of Sciences
R-GMA
Sensor Code
ProducerAPI
ApplicationCode
ConsumerAPI
Producer Servlet
RegistryAPI
Registry
SchemaAPI
Schema
“Definition of tables”
Consumer Servlet
RegistryAPI
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesR-GMA
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of Sciences
R-GMA
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesPulse
● On-line analysis and visualisation tool for
● monitoring data of resources and services and applications.
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesPulse
● Data source components preprocessing components
view components their connections are defined in XML
● Separates data model from visualisation modules
● based on Model-View-Controller paradigm
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesPulse
● Data model: meta-data (name, type, unit, range) and interface to the data source.
● Controller: preprocessing, analysis components
● View: displaying components
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesExample: CPU Load
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesExample: CPU Load
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesExample: CPU Load
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesData model
● Data channel
– Name
– Data type
– Unit
– Range of valid data
LoadChannelMetaData(
name=”load1”,
type=float,
unit=””,
range=[0.0-100.0+0.01] )
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesInput components
● Input component (sensor): an interface to some kind of external data source (file, service, an executable, etc.)
● A sensor– Reads data from the external source– Translates data into a channel based representation– Publishes complex information in several channels– Provides one piece of information at a time
● Buffered sensor– Buffering capability to a sensor
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesPresentation components
● To show the results to the user
● Tabular view, Histogram plotter
● Presentation components subscribe to the sensor's notification service ->they are notified about the status changes in the sensor ->display updates are controlled by the sensors (incoming data)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesAnalysis components
● Essentially:
– Take data from other components (sensors)
– Do something with the data
– Forward it to other components (plotter)
● E.g.
– Channel selector (for histogram plotter)
– Merge data from several sensors
– Compute statistical values on data
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesConfiguration of Pulse
● XML configuration to define
– Components
– Connections between the components
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesPulse: conclusion
● Pulse can be used to build an analysis chain to process dynamic information and present results in a graphical form
● Flexible data representation
● Already it can be used as a tool for the R-GMA grid monitoring and information system
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesApplication monitoring
● Performance monitoring (not job status!) of
● Parallel applications (MPI, PVM, P-GRADE)
● Large volume of trace information
● On-line transfer of trace to the user
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesGRM at EuroPar’2001
Main MonitorMM
Application Process
Application Process
Site 1
Local Host
Host 1 Host 2 Host 1
Local MonitorLM
Local MonitorLM
Site 2
Local MonitorLM
Site MonitorSM
Site MonitorSM
Appl. Process
Appl. Process
shm shm shm
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesTools
● GRM– Instrumentation library and trace collector
● Mercury monitor– Transfer trace data from the application processes
to GRM
● R-GMA– Information system for finding the application and
the monitor components
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesMercury monitor
in the GridLab project
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesGRM+Mercury+R-GMA
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesMercury monitor
● To deliver trace data from the application to the user efficiently.
– Uses TCP Socket communication
– Data in XDR format and could be optimised for TCP transmission
– Two sw. hops between application and GRM: local and main monitors
– One hw. hop: host of main monitor
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesR-GMA
● To find the application by the user/GRM
– Where is it running? -> machineX.siteY
– What is its global job id? -> GID
● To find the monitor to be connected
– What is the address of Mercury monitor running at siteY?
● To find the application by the monitor
– What processes (PIDs) belong to application GID?
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesGRM
● Instrumentation library
– Connect and publish trace to Mercury monitor
● GRM and PROVE
– Look for application asking R-GMA
– Connect to Mercury monitor and query for trace
– Visualise trace information with PROVE
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesCurrent status
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesFuture work
● Intrusion: replace socket communication between LM and application process for shared-memory buffering (as in original GRM)
● Scalability: data reduction at LMs and MM.
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesConclusion
● Different monitoring information can be presented with different techniques
– Web browser is used to search among available information and static (or slowly changing) behaviour of the grid
– Pulse can be used to build specific analysis chains to process dynamic information about the services and resources and to create graphical plots
– GRM is a tool for performance monitoring of parallel applications, connected to Mercury and R-GMA to work for grid.
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of Sciences
Thank you
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesR-GMA Schema Browser
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesR-GMA Schema Browser