© jboss inc. 2005 application and system monitoring using jboss application server nicholas...
TRANSCRIPT
© JBoss Inc. 2005
Application and System Monitoring Using JBoss
Application Server
Nicholas WhiteheadRobert QuinnPatrick Urban
March 1, 2005
2
Agenda
• Holistic Application and System Performance Monitoring
• Challenges In Collecting and Analyzing Performance Data
• A Proposed Architecture for a Performance Management Data Platform
• An Implementation of the Platform• Questions (and tentative answers)
3
Performance Monitoring & Analysis
4
Performance Monitoring & Analysis
• Heterogeneous Monitoring Targets Network (LAN, WAN, Routers, Switches) Hosts (CPU, Memory, Disk, Threads) Various Operating Systems Application Server (JMX, Instrumentation,
Pooling) Middleware (Message Throughput) JVM (Memory, GC) Database (Exec Plans, Memory, Concurrency,
Locks) Files (ad nauseum) Applicaton Specific Structures
5
Challenges
• Phone Call: “System is slow” Why and what do you do about it ?
• Prevention: Where are your weak spots ?• Planning: How many servers do I need to
get us through the next n months ?• What is the relationship between system
load and: Number of users Day of the week Month of the year My astrological sign ?
6
So Many Monitor Points, So Little Time
7
Techniques For Performance Monitoring
• Commercial Enterprise Frameworks Expensive Not Always Complete
• Manual Data Collection & Merging No Real Time Vizualization Inflxexible High Maintenance
• Adhoc Monitoring No Cross Referencing Multiple Monitors Load Monitored System
8
A Cascading Performance Emergency
• Users Report System is slow.
Where’s Waldo ?
9
Anatomy of a Perfomance Hit
There He Is !
10
A Consolidated Platform For Peformance Data Acquisition
11
Key Attributes
• Central, globally available clearing house for performance and system status data.
• Scale to large numbers of data sources and client dashboards.
• Supply dashboards, reports and status summaries to dashboards.
• Provide platform for post processing acquired performance and status data.
• Monitor Proxy: Multiplex multiple monitor clients to one set of predictable and controllable data acquisition streams.
• Support Correlation of Heterogeneous Metrics
12
JMX – Perfect For The Job
• Extensible and Standardized Container for In-Memory State• Robust, Flexible API• Rich and Dynamic Meta Data• Organized Domain Structure• Built In Monitors• Attribute Change and other Notifications• Wide Support for 3rd Party Tools• Rich Lifecycle Control
13
Services
• Simple State: Holds the last reading for a specific metric. e.g. The last CPU utilization reading on webserver1.
• Data History Buffer: The last n readings of a specific metric. Shows flat, upward or downward trend.
• Persistence: Store performance data for reporting, historical analysis and forecasting.
• Charting: Real time data visualization.• Alerts: Alerting services for specific conditions,
monitor triggers or failed readings.• Boundary Tracking: Track high, low and average
values for performance statistics.• Data Summarization: Aggregation of large
quantities of data into understandble summarized reportlets.
14
Simple State & State History Buffer
• JMX Attributes Contain the Latest Scanned Value of a Performance Metrics
• History Buffer Show the Last n Readings.
15
Persistence & Reporting
• Store performance data for: Reporting Forecasting Capacity Planning Trend Analysis
16
Charting
• Visualization of real time performance data.
• Side by side or large and rich displays containing many graphs for visual correlation.
17
Alerts
• Alerting by email and pager when established threshholds are crossed.
Nagios Monitoring System Service Status Screen
www.nagios.org
18
Boundary Tracking
• Track boundaries of selected metrics High Low Average
19
Summarization & Aggregation
• Large quantities of performance data can be aggregated and summarized to an understandable reportlet.
\\IISWEBSERVER01\Thread(inetinfo/0)\Thread State: 5\\IISWEBSERVER01\Thread(inetinfo/0)\Thread Wait Reason: 0\\IISWEBSERVER01\Thread(inetinfo/1)\Thread State: 5\\IISWEBSERVER01\Thread(inetinfo/1)\Thread Wait Reason:6\\IISWEBSERVER01\Thread(inetinfo/2)\Thread State: 5\\IISWEBSERVER01\Thread(inetinfo/2)\Thread Wait Reason: 0\\IISWEBSERVER01\Thread(inetinfo/3)\Thread State: 5\\IISWEBSERVER01\Thread(inetinfo/3)\Thread Wait Reason:6\\IISWEBSERVER01\Thread(inetinfo/4)\Thread State: 5\\IISWEBSERVER01\Thread(inetinfo/4)\Thread Wait Reason: 0\\IISWEBSERVER01\Thread(inetinfo/5)\Thread State: 5\\IISWEBSERVER01\Thread(inetinfo/5)\Thread Wait Reason:6\\IISWEBSERVER01\Thread(inetinfo/6)\Thread State: 5\\IISWEBSERVER01\Thread(inetinfo/6)\Thread Wait Reason: 0\\IISWEBSERVER01\Thread(inetinfo/7)\Thread State: 5\\IISWEBSERVER01\Thread(inetinfo/7)\Thread Wait Reason:6
20
Helios – An Implementation
• An Open Source Implementation• Powered by JBoss Application Server• Performance Data Service & Clearing
House• Heterogeneous Data Sources• Extensible Monitors and Post
Processing• Customizable and Easy to Configure
Dashboards
21
Evolution of Helios
Original State of Affairs• Multi-tool data collection• Manual Data Merge• No Proxy. All monitor clients put load
on servers.
22
Evolution of Helios
Simple Automation• Automated multi-tool data collection• Daily merge• Manual Reports and Charts
23
Evolution of Helios
Simple Dashboard• Numeric and character data only. • Web page console.• Limited data.
24
Evolution of Helios
Evolving Architecture• Implementation of MBean Proxies.• MBean generated graphics.• Single console still requires rebuild and redeploy to
change.
25
Helios Version 0.1
New Architecture• Dynamic Dashboards
that can be saved and shared.
• Dashboard display widgets can be resized and moved.
• Configurable, extensible and hot deplyable data source monitors.
• Extensible post-processing using plugins.
• External client API for push data.
• More…..
26
Helios Architecture
• Extensible and Robust JMX (Remote Notification Really Work)
• Hibernate Managed Persistence (& HARs !!)
• Scalable and Clusterable Services
• Multi-Protocol Remoting
• Great Platform To Work On
• Free Platform To Work On
27
Monitors - Overview
• Implemented as a JMX MBean Acts as a proxy to a performance data source Performance data is available as attributes Configured using XML in jboss-service.xml Can be configured to pass results to a
collection of plugins Broadcasts notification of new scans and
attribute changes Active Monitors scan a performance data
source (Pull) Passive Monitors receive performance data
from an External Client Agent (Push)
28
Monitors - Types• Currently Available
JMXMonitor - Monitors JMX Attributes WindowsMonitor - Monitors Windows Performance Manager Stats SQLMonitor - Monitors JDBC Data Sources YahooFinancialMonitor - Monitors Yahoo Finance Securities PassiveMonitor – Creates attributes for holding results from a remote agent JRat Java Byte Code Instrumentation – Acquires Java Method Elapsed Times
• In Progress LinuxMonitor: Monitors Linux /proc file performance data JSR77 Autoconfiguring Monitor Text Stream Reader
• File Scraper (Log Analyzer) • Telnet Data Acquisition • SSH Data Acquisition
• Planned OpenArm Helios Mediator: Application Response Measurement Data Acquisition GkRellm Data Acquisition NetPerf Data Acquisition SNMP Data Acquisition PING Monitor
29
Monitors – Sample Configuration
30
Plugins - Overview
• Configured for each Monitor Receives a copy of each scan result Extends a monitor’s functionality Configured using XML in jboss-service.xml Can create a new result and register it an
attribute in the Monitor (back register)
31
Plugins - Types• Currently Available
PersistScalar – persists scalar results PersistSimpleArray – persists simple arrays RollingHistory – Tracks ‘n’ previous versions of a scan VariaGraphPlugin – Creates line graphs (JPG, PNG or SVG) TrackAverageNumber- Computes averages BasicSQLHTMLFormatter – Formats ResultSets as HTML J2EEStatsHTMLFormatter – Formats JSR77 results as HTML TrackHighNumber – Maintains a maximum value TrackLowNumber – Maintains a minimum value TrackAverageNumber – Maintains the average value
• In Progress Graphing Library Nagios External Event Updates File Logging Event
• Planned Customizable Event Broadcaster For Integrating to 3rd Parties Customizable HTML Renderer
32
Plugins – Sample Configuration
33
Subscription Service - Overview
• Allows a client to subscribe to and receive updates for Monitor attributes Implemented as an MBean HTTP Façaded provide for web clients Clients establish a “session” Server Side Proxy manages client’s
subscriptions Subscription listen on the notification bus and
filter attributes notifications based on client’s subscription
Client can subscribe/unsubscribe to attributes Client “polls” and receives all available updates
34
Subscription Service - Components
Http
Session..
.
Clients
Subscription ServiceHTTP Facade
Subscription Service(Mbean)
Client Proxy (Mbean)
Client Proxy (Mbean)
Client Proxy (Mbean)
HTTPSessionListener manages thelifecycle of the Client Proxy.
Non Http clients must manage theclient proxy lifecycle themselvs
SubscribernSubscriberSubscriber
Client Info
Notification Bus
Monitors
JMX Notifications
Client Proxy is responsible formanaging the subscriptions and theDOM
Subscriber is responsible formanaging a single subscription. Allfiltering and/or managing of messagefrequency “differences” are handledhere.
xferDOM
Subscribe/Unsubscribes
getUpdates
Attribute Subscriptions update theDOM directly.
getUpdates marshals the DOM of allavailable updates
35
Subscription Service – Basic Use Cases
Client
Subscribe
Subscription Facade Subscription Service
Subscribe
Client Proxy
Create
MBean Server
Subscribe
Create
Subscriber
Subscribe
NotifyUpdate
Update Notify
Update Notify
Xfer DOM
Create
Subscribe
Subscribe
SubscribeCreate
Subscriber
Subscribe
NotifyUpdate
NotifyUpdate
NotifyUpdate
HttpSessionListener
getUpdates
getUpdates
read
36
Visualizing The Data
Dashboard Manager A WYSIWYG tool for managing and sharing
views of your information real-time via “Dashboards”.
37
The Goals Of Helios Dashboard Manager
• Easy Creation
• Easy Viewing
• Easy Sharing
38
Easy Creation
• All available “Monitors” displayed for selection in a tree style view called the Dictionary
39
Easy Creation Cont.
• Drag and Drop style dashboard creator• Supports resizing and overlapping
40
Easy Viewing
• View dashboards using a browser Currently supports IE 5.5+ and Firefox 1.0
• Supports full-screen viewing and slide show style for multiple dashboards.
• Live Data Updates. Supported output type include… Single Value Array Rich HTML Image .gif, .jpg, .png SVG
• View on any platform via Firefox
• No additional client installs or plugins required
• Clean presentation ready look
41
Easy Sharing
• Dashboards support custom HTML content via “Labels” that allow the creator to annotate a dashboard as necessary.
• Underlying XML descriptor allows for dashboard definitions to be imported and exported between Helios server instances.
42
Future Plans
• Library of Client-Side Rendered Graphs via SVG• More Dashboard Customization Options
Look and Feel Behavior
• Security/Dashboard Ownership Integration
43
Client API
• Remote API support for: Creating New Passive
Monitors. Creating New Monitor
Attributes Updating Monitor
Attributes
• Protocols: RMI JMS Simple XML over
HTTP
• XML over HTTP allows non-Java scripting languages such as Perl and TCL to perform remote data acquisition for Helios.
44
Thanks !
Helios: Greek God Of The Sun. He sees everything that happens and is often called
upon to shed light on events.
Helios. See All. Hear All. Understand All.
https://helios.dev.java.net/index.html