monitoring system performance and health of ibm i tim · pdf file1 monitoring system...

1

Monitoring System Performance and Health of IBM i

Tim Rowe – [email protected]

Business Architect -

Systems Management

Application Development

© 2014 International Business Machines Corporation 2

Many Thanks to …..

Dawn May – [email protected]@DawnMayiCan

IBM Systems Lab Services & Training

2

Proactive versus Reactive

� Prevent potential problems before they become real problems

– Understand system performance in real-time

• What is consuming CPU, memory, or disk?

– Automatically notify an operator when a condition is detected

– Know immediately when a message is sent

� This session will show you how to automate monitoring so you can focus on other aspects of your job.

� This presentation will help you:– Create and use system monitors and message monitors– Learn about watches– Understand Health Indicators with Navigator for i


Agenda

� Monitoring with Navigator for i

� Performance Health Indicators

� Watches

� Monitoring with Management Central


3

Monitoring Interfaces

� IBM Navigator for i

– Monitors: System and Message – new in IBM i 7.2!

– Performance Tasks - Performance Data Investigator

• Health Indicators

• Monitor

� Command interface

– Watches

� System i Navigator (aka, “iSeries Navigator”, “Operations

Navigator”)

– Management Central Monitors - Stablized

5© 2014 International Business Machines Corporation

Monitors with IBM Navigator for i

List of system monitors on the system

Select what you want to monitor

Set monitoring intervals

Set thresholds

Define actions taken when a threshold is reached

Manage event logs


4

Notes: Management Applications - where to get them?

IBM Navigator for i is the Web console for IBM i. It is included with IBM i; no installation is necessary, all you need in a Web browser.

Navigator for I provides system monitors and message monitors. It has very basic support for system groups for PTF compare and install.

System i Navigator, is included with IBM i at no additional cost. The IBM i function is integrated into the base of the operating system. The client function is shipped as part of System i Access for Windows.

Note: 7.1 was the last release of System i Navigator – you should be moving to Navigator for i. However, there are still a few functions in System i Navigator that are not yet in Navigator for i. As you will notice, 7.2 has made this difference smaller.

Management Central is a technology integrated into System i Navigator and is not directly installed. When installing System iAccess for Windows, choose 'Custom Install'. Expand the System i Navigator option tree and select the appropriate components such as Monitors, Commands.....

The general rule for Management Central functions for connectivity is that N-2 and N+2 releases are supported. However, for the best performance and most functions available it is strongly recommended that your IBM System i Navigator and your Central System be at the highest release you have available.

Your endpoint systems can then be at a mix of previous releases.

IBM Systems Director is a browser-based management tool. You need to download and install the management server (it does not run on i) as well as install the agents required on each endpoint. For monitoring on i, you need to install the 5770-UME licensed program product and 5733-SC1, Option 1 (SSH) on each IBM i partition you want to monitor.IBM Systems Director can be downloaded from http://www-03.ibm.com/systems/software/director/downloads/index.html


Navigator Monitors

• System Monitor

– Monitoring IBM i

– Custom event actions

• Message Monitor

• Single partition only

New in 7.2!


5

Monitoring with System Monitors

• System Monitors gather and present real-time performance data– Monitor the health of your system

– Identify potential performance problems before they become serious issues

• System Monitors show high-level performance information– Navigator’s System Monitors use the Performance Data Investigator to visualize and provide drill-

down capabilities

– Management Central System Monitors have basic levels of detail

• Thresholds can be defined to trigger an action when a system wide performance metric exceeds the defined “comfort level”

– For example, when CPU Utilization exceeds 80%, send a message to notify the operator

• System Monitors provide powerful capabilities to monitor what is happening on your system

– BUT finding out what caused the problem often requires other performance analysis tools


Create New System Monitor


6

Metrics to MonitorLink to configure thresholds and actions


Navigator - System Monitor Metrics

• CPU Utilization (Average)

• CPU Utilization (Interactive Jobs)

• CPU Utilization(Uncapped)

• CPU Utilization(SQL)

• Interactive Response Time (Average and Maximum)

• Transaction Rate (Interactive)

• Batch Logical Database I/O

• Disk Arm Utilization (Average and Maximum)

• Disk Arm Utilization for User/System/Independent ASP (Average and Maximum)

• Disk Storage Utilization (Average and Maximum)

• Disk Storage Utilization for User/System/Independent ASP (Average and Maximum)

• Communications Line Utilization (Average and Maximum)

• LAN Utilization (Maximum and Average)

• Machine Pool Faults

• User Pool Faults (Maximum and Average)

• Spool File Creation Rate

• Shared Processor Pool Utilization (Virtual and Physical)

• Temporary Storage Utilization

Red are new with 7.2 Navigator Monitors


7

Configure Metric


Metric

Configure Metric Examples


8

Command Prompt

Basic

Advanced


Monitor Summary


9

Monitor Actions

You can take the following actions on a monitor:

• Start

• Stop

• Display the properties and/or change the configuration

• Monitors must be stopped to make changes

• Create one like

• View the event log

• Investigate the monitor data for the desired metric


Threshold Triggered


10

Event Log


Event Properties


11

Investigate Data - Monitor

• Investigate Monitor Data starting from the monitor


Investigate Data - Monitor

• Investigate Monitor Data via the Performance Data Investigator

– This interface allows you to view monitor data without having set up a system monitor


12

Investigate Monitor Data

Monitor collections begin with “R”No automatic refresh (yet….)


Monitor Drill-down Actions

With Navigator System Monitors, you use the Performance Data Investigator to view the graphs.

Drill-down to find contributing jobs


13

Changing a Monitor Properties

You can only change the properties when a monitor is stopped


System Monitoring

Collection Services can be configured to collect system monitor data 24x7. (System policy for “real-time” data collection)

• Data to support system monitoring can be available without depending on a monitor function

– CS starts at IPL, data is available at IPL

• Similar to Management Central, a monitor can tell Collection Services what data it needs and that data is collected and stored in the *MGTCOL just like before.


14

System Monitoring

Configure Collection Services GUI for the System Monitor support:


Collections when Monitoring

• You will see three collections when you start monitoring on 7.2:

– The management collection object

– Two file-based collections

� Q* - the traditional collection services collection

� R* - the monitoring collection


15

PDI: Monitor Perspectives


Monitor Perspectives

You can:• Collect Monitor data• Investigate Monitor perspectiveswith Collection Services

However… to define thresholds and actions you must create and start a system monitor


16

Navigator

Message Monitors


Monitor Message Queues


17



18


Start the Monitor


19

Message Monitors require 5770UME

• 5770UME is shipped with the operating system

– (no additional charge)

• You must install this product to use Navigator Message Monitors

– Starts a TCP/IP Server named *CIMOMSTRTCPSVR *CIMOM

– Server jobs are named QUME* and run in QSYSWRK


Triggers and Event Log


20

Trigger


Trigger Example


21

Monitoring System Performance with Navigator for i


Navigator for iPerformance Tasks


22

Health IndicatorsManually Monitor your System Performance


Health IndicatorsCustomize Health Indicator Thresholds


23

Monitoring with Watches


Monitoring with Watches

� Watches can be use to automate the actions taken when the

following occur:

– Message

– Licensed Internal Code Log (LIC Log)

– Problem Activity Log Entry (PAL entry)

� Start Watch (STRWCH) command or API (QSCSWCH)

� End Watch (ENDWCH) command or API (QSCEWCH)

� When the condition being watched occurs, your program gets control and you can take any action you want

http://publib.boulder.ibm.com/infocenter/iseries/v7r1m0/topic/rzahb/rzahb_eventfunction.htmhttp://ibmsystemsmag.blogs.com/i_can/2010/01/i-can-automate-monitoring-with-watches.html


24

Watches

� Low Overhead

– Watches are an exit

– Almost no overhead until the watched condition occurs

– Your program gets control to determine what action to take

– For message watches

• Can watch for messages sent to any message queue, including

� QSYSOPR, History Log

• Can watch for messages sent to any job log

� Can specify generic job name

� Can specify *ALL to watch for a message to all job logs


Start Watch Command

Start Watch (STRWCH)

Type choices, press Enter.

Session ID . . . . . . . . . . . Name, *GEN

Watch program . . . . . . . . . Name

Library . . . . . . . . . . . *LIBL Name, *LIBL, *CURLIB

Call watch program . . . . . . . *WCHEVT *WCHEVT *STRWCH *ENDWCH

Watch for message:

Message to watch . . . . . . . *NONE Name, generic*, *NONE...

Comparison data . . . . . . .

Compare against . . . . . . . *MSGDTA, *FROMPGM, *TOPGM

Message type . . . . . . . . . *ALL, *COMP, *DIAG...

Relational operator . . . . . *GE, *EQ, *GT, *LT, *LE

Severity code . . . . . . . . 0-99

+ for more values

Bottom

F3=Exit F4=Prompt F5=Refresh F12=Cancel F13=How to use this display

F24=More keys


25

Start Watch Command



Watched message queue:

Message queue . . . . . . . . *SYSOPR Name, *SYSOPR, *JOBLOG...

Library . . . . . . . . . . *LIBL Name, *LIBL

+ for more values

Watched job:

Job name . . . . . . . . . . . * Name, generic*, *, *ALL

User . . . . . . . . . . . . Name, generic*, *ALL

Number . . . . . . . . . . . 000001-999999, *ALL

+ for more values

Watch for LIC log entry:

Major code . . . . . . . . . . *NONE 0000-FFFF, *ALL, *NONE

Minor code . . . . . . . . . . 0000-FFFF, *ALL

Comparison data . . . . . . .

Compare against . . . . . . . *ALL, *TDENBR, *TASKNAME...

+ for more values

More...


F24=More keys


Start Watch Command



Watch for PAL entry:

System reference code . . . . *NONE Character value, *NONE, *ALL

Comparison data . . . . . . . Character value, *NONE

Compare against . . . . . . . *RSCNAME, *RSCTYPE, *RSCMODEL

+ for more values

Run priority . . . . . . . . . . 25 1-99

Bottom


F24=More keys


26

Management Central Monitors


Management Central

� IBM System i Navigator -Management Central Monitors

• Diagnosis based on information from the Central System and the Endpoint System

� Logical flow of a diagnosis:

• Endpoint reports condition to central system

• Central system notifies administrator

• Administrator looks at the endpoint

� with the central system observing

• Administrator may need to examine details on the endpoint.

So what's this event?

Create event about Partition1

Something is wrong!


27

Notes - Terminology

IBM System i Navigator provides a graphical user interface to IBM i® . It comes in 2 complimentary options: Windows and Web.System i Navigator for Windows is in installed on the PC so the user can have a rich graphical interface to interact with their systems.System i Navigator tasks on the Web perform a subset of Navigator tasks through an Internet Web browser. These are URL-addressable links only.

Navigator for i is the browser-based console that has much of the function of System i Navigator; monitors however,are not available through this interface. The Performance tasks is only available through this Web console.

IBM® i® integrated Web application server - The i integrated Web application server (5761-SS1) integrates an OSGi-based Web-servlet container with the i operating system. (5.4 and later)

Central System helps to manager your other systems (called endpoints) and stores most management information (inventory, command, package, product, and user definitions, etc).

Endpoints are the systems which your PC does not need to be in direct contact with in order to "manage".

Source System is the system from which objects, files and information are sent using Management Central's send tasks. The Source System is the source of the objects, files and information being sent.

Model System has all and only desired fixes installed or has all system values set properly for the targets.

Target System is where objects, files and information are sent within Management Central's send tasks. The Target Systems are the destinations of the objects, files and information being sent. Target Systems (and more generally, endpoint systems) are often grouped into System Groups.


Real-time performance monitoring with

System Monitors


28

Management Central System Monitors


Select 'New Monitor...' and specify General properties

Define A Monitor


29

Notes: Select 'New Monitor...' & specify General properties

You need to name your monitor – specific names work well. You can select one or multiple metrics in a single monitor

Question: How many metrics do you put in a single monitor?

Answer: It depends

1. do you like to see all monitors on a single screen?

2. do you prefer to have more granular monitor notifications

There is no limit on the number of endpoint systems that a monitor can be started on. However you do get into usability issues when displaying the graph, too many systems on a graph and it might get to difficult to view.

General - The General page for New Monitor or Monitor Properties allows you to view and change general information about the monitor. The general information includes the name of the monitor, a brief description of the monitor.

Name - The unique name of the monitor. You can change the name, using up to 64 characters for the new name. Do not use any of the following characters: asterisk (*), backslash (\), colon (:), greater than (>), less than (<), question mark (?), quotation mark (“), slash (/), or vertical bar (|).

Description - A brief description to help you identify this monitor in a list of monitors. You can change the description, using up to 64 characters for the new description.© 2014 International Business Machines Corporation 57

Select 'Metrics to monitor' and press OK to create

Define A Monitor

What to monitor

How often

Verticalaxis

Horizontalaxis


30

Notes: Define A Monitor

The Metrics page for New Monitor or Monitor Properties allows you to select the metrics that you want to monitor. You can view and change information about the collection interval, the maximum graphing value, and the display time for each metric. You can also click Threshold 1 or Threshold 2 to specify information about the thresholds for each metric.

Metrics is the piece of information to collect. Possible values are:

CPU Utilization (Average) Communications IOP Utilization (Average)

CPU Utilization (Interactive Jobs) Communications IOP Utilization (Maximum)

CPU Utilization (Interactive Feature) Communications Line Utilization (Average)

CPU Utilization Basic (Average) Communications Line Utilization (Maximum)

CPU Utilization (Secondary Workloads) LAN Utilization (Average)

CPU Utilization (Database Capability) LAN Utilization (Maximum)

Interactive Response Time (Average) Machine Pool Faults

Interactive Response Time (Maximum) User Pool Faults (Average)

Transaction Rate (Average) User Pool Faults (Maximum)

Transaction Rate (Interactive) Disk Storage (Average)

Batch Logical Database I/O Disk Storage (Maximum)

Disk Arm Utilization (Average) Disk IOP Utilization (Average)

Disk Arm Utilization (Maximum) Disk IOP Utilization (Maximum)

Collection Interval is the time to wait in-between each collection of data.

Maximum graphing value is the highest value to be displayed on the vertical axis of the graph.

Display time is how many minutes you want displayed on the horizontal axis of the graph.© 2014 International Business Machines Corporation 59

System MonitorsSetting Thresholds

Conditionindicating

problem exists

Conditionindicating

problem resolved

Automation

What


31

Notes: Setting Thresholds

Threshold - A threshold is a setting for a metric that is being collected by a monitor. This setting allows you to specify actions to be taken when a specified value (called the trigger value) is reached. You can also specify actions to be taken when a second value (called the reset value) is reached. For example, you can specify a CL command that stops any new jobs from starting when CPU utilization reaches 90% and another command that allows new jobs to start when CPU utilization falls to less than 70%. You can also choose to add an event to the Event Log whenever the trigger value or the reset value is reached. You can set up to two thresholds for each metric that the monitor is collecting. Trigger - considered bad (usually high but can be low), reset - consider good (opposite of trigger)

The two Thresholds tabs on the metrics page provide a place for you to specify whether or not you want to monitor this metric for a particular threshold. You must check the Enable threshold box before you can specify the conditions to trigger and to reset this threshold. You can also specify the action to be taken when the threshold is triggered and when it is reset. The action that you specify must be a CL command. When you click OK, this metric will be actively monitored for this threshold if the monitor is currently running. If the monitor is not currently running, this metric will be monitored for this threshold the next time the monitor is started

You can specify the following conditions and commands for Threshold trigger and for Threshold reset:

Value - Specifies the condition that must be met to trigger or to reset this threshold.

Duration - Specifies the number of consecutive collection intervals that the value must meet the criterion to cause a threshold trigger or reset event. Specifying a higher number of collection intervals for Duration helps to avoid unnecessary threshold activity due to frequent spiking of values.

i command - Specifies the command to be run on the i endpoint system when the threshold is triggered or reset on that endpoint. This command can be as simple as sending a message, or as complex as submitting or calling a program.


Notes: Threshold variables

System Monitor Replacement Variables:

Parameter Passed Data&DATE The Date the monitor triggered or reset&INTVL Collection interval: How often the monitor collected data (in seconds)&MON The Monitor name&RDUR Reset duration: How many intervals does the reset value have to be met before the monitor resets.&RVAL Reset value: The value that the metric was monitoring for when the monitor reset&SEQ Sequence number: A unique, incrementing number assigned to each collection interval. Can be used in

a program to compare when triggers happened and in what sequence.&TDUR Trigger duration: How many intervals does the trigger value have to be met before the monitor triggers&TIME The time the monitor triggered or reset&TVAL Trigger value: The value that the metric was monitoring for when the monitor triggered&VAL Current value: The actual value of the metric when the monitor triggered (2)

Note: A couple of things to note about system monitor replacement parameters:- The dollar sign ($) that was available in previous releases is still supported, for example, $TIME.- The wording is a bit different on some metrics and values:- Batch I/O is shown as I/O operations rather than transactions per second.- Transaction rates are shown as transactions rather than transactions per second.- Interactive response times (both average and maximum) are shown in milliseconds rather than

seconds.


32

Select the monitor, then the start button to select systems/groups

Start A Monitor


Notes: Start A Monitor

The Start Monitor dialog allows you to select the endpoint systems and system groups on which you want to start the monitor (if no endpoint systems or system groups have been previously selected for this monitor).

To add a system or group to the Selected systems and groups list, select it in the Available systems and groups list, and then click Add. If a monitor is started and then a system is added, the monitor will be started on that endpoint system automatically.

To remove a system or group from the Selected systems and groups list, select it in the list, and then click Remove. If a monitor is started and then a system is removed, the monitor will be stopped on that endpoint system automatically.

Available systems and groups - A list of endpoint systems and system groups from which you can select a system or group. Click the plus sign (+) next to any group to see the systems that are included in the group.

Monitor data is collected and stored on the endpoint system. A minimum amount of data is actually sent back to the client when viewing the graph, The more specific, detailed data is only sent to the client when the graphs are open

PC is not required to be connected once monitor is started. The graph window can also be minimized and the monitor will still be active.

The data shown in the graph is obtained from Collection Services. Collection Services houses the data in management collection objects. This data is used by system monitors, job monitors and other performance tools.


33

System Monitors

Overall status

View the status


Notes: View Status for a MonitorThe Status dialog allows you to see the current status of each endpoint system and system group associated

with a monitor. The status of each system and group is updated automatically as changes occur. You can expand any group in the System or Group list to see the status of individual systems in the group. By clicking the Restart button, you can restart the monitor on any systems on which it has failed.

Overall status - The current status of the monitor. Possible values are:

x thresholds triggered - The number of thresholds that are currently active for the monitor (that is, x represents the number of thresholds that have been triggered but have not been reset).

Started on x of y systems - The monitor is collecting data on x of y endpoint systems, where x represents the number of systems where the monitor is running and y represents the number of systems where you requested to start the monitor. The monitor is in the process of starting on the remaining systems.

Started - The monitor is collecting data on all endpoint systems where you requested to start the monitor.

Starting - The monitor is in the process of starting.

Stopping - The monitor is in the process of stopping.

Stopped - The monitor is no longer collecting data.

Failed - An attempt was made to start the monitor on the specified systems or groups, but the monitor was not started on any systems. The failure may have occurred because the systems were not running when you tried to start the monitor, or it may be because a connection was lost or a server was not started. Click Restart to try starting this monitor again.

Failed on x of y systems - The monitor has failed to start or unexpectedly stopped working on x of y systems (where x is the number of systems on which work has stopped and y is the total number of systems on which the monitor is to be run). The monitor is starting or started on the remaining systems. The failure may have occurred because the systems were not running when you tried to start the monitor, or it may be because a connection was lost or a server was not started. Click Restart to try starting this monitor again. See the System or Group Status for a list of the endpoint systems and system groups associated with the monitor and the current status of each system and group.


34

Metric data

View A Monitor


View A MonitorClick on graph point to see upper right pane

Click on object graph in upper right pane to see details in lower

right


System Monitors provide multiple levels of performance informationLevel 1 – System wide performance metrics such as CPU Utilization, Disk Utilization, etc.Level 2 – A list of items that are contributing most to the Level 1 metric

For CPU Utilization, it’s a list of jobs that are consuming the most CPUFor Disk Utilization, it’s a list of disk arms that are the busiest

Level 3 – A list of performance metrics and properties for the Level 2 items

35

System Monitors

Threshold Indicators

Viewing the thresholds


Changing Thresholds

Properties

Active Control

Menu


36

Notes: Changing ThresholdsYou can change the thresholds several ways.

Properties

Active graphical control

Menu items

You can change thresholds while a monitor is started, e.g., you do not need to stop the monitor to change the thresholds. The general properties of the monitor can be accessed view the toolbar or menu items for making any changes or additions to the thresholds and values.

To change the trigger value or the reset value for a threshold using the active graphical control, place the mouse pointer on the threshold indicator. When the ToolTip indicates Trigger, hold the mouse button down and move up or down to change the trigger value. The changing values are shown in the ToolTip. When the ToolTip indicates Reset, hold the mouse button down and move up or down to change the reset value . Click any collection point on a Monitor graph line to see Details of the data associated with the collection point.

By accessing the menu items, you will taken directly to the thresholds page in properties to make any changes.

There are several visual indicators when a threshold occurs:

Status in the toolbar area.

Upper Left corner icon will change.

Line in the graph will change to red.

Metric graph title will change to red with icon indicator


Threshold Actions

IBM i

PC Client


37

Notes: Threshold Actions

The Actions page for Monitor Properties allows you to specify the actions to occur when a threshold is triggered and when a threshold is reset which apply to all metrics.

Log event - Adds an entry to the Event Log on the central system indicating that the threshold was triggered. The entry also includes the date and time the event occurred, the endpoint system being monitored, the metric being collected, and the monitor that logged the event.

Open Event Log - Displays the Event Log, which is a list of threshold trigger and reset events that have occurred.

Open monitor - Displays a graphical view of the metrics as they are being collected.

Sound alarm - Sounds an alarm on the PC.

Threshold commands will be run under the monitor's owner's user profile.

When a threshold gets triggered/reset, your PC client does not need to be up and running to run the Operating System command. However, if it is not up the corresponding PC action will not happen.


Viewing Events From Thresholds


38

Notes: Event Logs

The Event Log window displays a list of threshold trigger and reset events for all of your monitors. You can specify on the Properties page for each monitor whether or not you want events added to the Event Log. To see the Properties page for any monitor, select the monitor in the Monitors window and then select Properties from the File menu. The list of events is arranged in order by date and time by default, but you can change the order by clicking on any column heading. For example, to sort the list by the endpoint system where the event occurred, click on System.

An icon to the left of each event indicates the type of event::

A red circle with white x - indicates that this event is a trigger event for which you did not specify a host command to be run when the threshold was triggered.

A yellow circle with red x - Indicates that this event is a trigger event for which you specified a host command to be run when the threshold was triggered.

A white check with a check mark -indicates that this event is a threshold reset event.

You can customize the list of events to include only those that meet specific criteria by selecting Options from the menu bar and then selecting Include. You can have more than one Event Log window open at the same time, and you can work with other windows while the Event Log windows are open. Event Log windows are updated continuously as events occur.


Customize Event Log Information


39

Notes: Options – Include

Options menu choices

Click Options on the menu bar to display the actions you can perform to change what information is displayed. The possible choices are:

Include...

Displays the Include dialog, which allows you to specify which events you want to display in the list.

Columns...

Displays the Columns dialog, which allows you to specify which columns of information you want to display in the list. You can also specify the order in which you want the columns to be displayed.


Event Properties - Trigger (reset similar)


40

Notes: Event Properties

The Trigger/Reset page for Event Properties allows you to view additional information about the event. This information includes the value, the duration, the Operating System command and the sequence number of the event.

Trigger/Reset value - The value specified in the monitor properties.

Actual value - The actual value that exceeded the trigger value and caused the trigger event.

Duration - The number of collection intervals specified for the duration in the monitor properties.

Operating System command - The command that was run on the endpoint system when the event occurred.

The General page for Event Properties allows you to view general information about the event. The general information includes the type of event (trigger or reset), the date and time the event occurred, the endpoint system that the event occurred on, the metric that was being collected, and the name of the monitor that logged the event.

For more information, select the following:

Event type System

Date Time

Monitor Metric


Management Central Job Monitors

� Monitor the system by selecting ...

– Criteria to subset data• Job criteria - subsystem, job name, job type or user

• Server name - web server, ftp server...

– Predefined metrics• Job Count, Thread count, CPU Utilization, …

� Enabled with...

– Event logging– Trigger/Reset notification

– Command Automation– System Actions

• Job actions such hold, release...


41

Job MonitorsSelect 'New Monitor...' and specify General properties


Job Monitors

Problem Condition

Automation

Select 'Metrics to monitor' and thresholds then press OK to create


Different Metrics Levels

42

Notes: Define A Monitor

The Metrics page for New Monitor or Monitor Properties allows you to selectthe metrics that you want to monitor. You can view and change information foreach metric. You can also click Threshold 1 or Threshold 2 to specifyinformation about the thresholds for each metric.

Metrics are the pieces of information to collect. Possible values are:Job Count, Job Log Message and Job StatusJob Numeric Values:CPU Percent UtilizationLogical I/O RateDisk I/O RateCommunications I/O RateTransaction RateTransaction TimeThread CountPage Fault RateSummary Numeric Values (same a job level)



Job Monitor Replacement Variables:

Parameter Passed Data &DATE The Date the monitor triggered or reset&INTVL Collection interval: How often the monitor collected data (in seconds)&MON The monitor name&TIME The time the monitor triggered or reset&ENDPOINT The endpoint system name&EVENTTYPE Event type: The type of trigger or reset that is happening, defined as

follows:Triggered Event = 1Auto Reset Event = 2Manual Reset Event = 3

&JOBNAME The job name of the job causing the trigger/reset&JOBNUMBER The job number of the job causing the trigger/reset&JOBSTATUS The job status causing a trigger/reset

&JOBTYPE The job type of the job causing the trigger/reset&JOBUSER The job user of the job causing the trigger/reset


43


Job Monitor Replacement Variables (continued):

Parameter Passed Data &METRICTYPE The category of the metric. For a Job monitor, the categories are as follows:

Status Metric = 10010Message Metric = 10020Numeric Metric = 10030

&METRIC Metric that has triggered/reset, defined as follows:Job CPU Utilization = 1010 Summary Comm I/O = 2040Job Logical I/O = 1020 Summary Trans. Rate = 2050Job Disk I/ = 1030 Summary Trans. Time = 2060Job Comm I/O = 1040 Summary Thread Cnt = 2070Job Transaction Rate = 1050 Summary Page Faults = 2080Job Transaction Time = 1060 Job Status = 3010Job Thread Count = 1070 Job Log Messages = 3020Job Page Faults = 1080 Summary Job Count = 4010Summary CPU Utilization = 2010Summary Logical I/O = 2020Summary Disk I/O = 2030



Job Monitor Replacement Variables (continued):

Parameter Passed Data&NUMCURRENT Current numeric value &NUMRESET Threshold value to cause auto-reset of numeric

metric&NUMTRIGGER Threshold value to cause trigger of a numeric

metric&OWNER Monitor owner&RDUR Reset duration, in intervals, as set in the threshold&RESETTYPE Reset type and defined as follows:

Manual reset = 1Automatic reset = 2

&SBS Subsystem of the job causing the trigger/reset&SERVER Server type of the job causing the trigger/reset.

Note: Not supported for summary metrics.&TDUR Trigger duration, in intervals, as set in the threshold &THRESHOLD Threshold number causing the trigger&MSGID Message ID causing the trigger/reset &MSGSEV Message severity causing the trigger/reset

&MSGTYPE Message type causing the trigger/reset


44


Invalid Job Monitor Replacement Variable Combinations:

Job Monitor substitution parameter notes:•If a monitor is triggered and the user performs a manual reset ("Reset with Commands" or "Reset Only") , there is no substitution value for the Parm &NUMRESET, &RDUR. It will only have a value if the reset is automated.•&MSGID, &MSGSEV, or &MSGTYPE you need to be monitoring the 'Job Log Message' metric - otherwise there is no substitution value for these. Additionally, these are only valid in the trigger and reset commands of Job Log Messages thresholds.•&RESETTYPE only has a valid substitution value on a reset command. Constant values are used to determine whether the reset type is manual or automated.•&EVENTTYPE is valid for all substitution and has constant values that are used to determine the type of monitor event that occurred (automated trigger, automated reset, or manual reset). In a trigger command, the value is always the trigger constant; in a reset command, it can either be the automated reset or manual reset constant.•&TDUR, &NUMTRIGGER, and &NUMCURRENT only have valid substitution when a trigger occurs, in the trigger command.•&NUMTRIGGER, &NUMCURRENT, and &NUMRESET only have valid substitution when a "numeric" metric is being monitored, in the trigger and reset commands of numeric metric thresholds.•&JOBSTATUS only has valid substitution when the Job Status metric is monitored, in the trigger and reset commands of Job Status thresholds.•Job Count metric not valid with: &JOBNAME, &JOBUSER, &JOBNUMBER, &JOBTYPE, &SBS, &SERVER, &MSGID, &MSGSEV, &MSGTYPE, AND &JOBSTATUS•Job Log Message metric not valid with: &RDUR, &NUMRESET, &TDUR, &NUMTRIGGER, &NUMCURRENT, and &JOBSTATUS•Job Status metric not valid with: &NUMRESET, &NUMTRIGGER, &NUMCURRENT, &MSGID, &MSGSEV, AND &MSGTYPE•The 'Job Numeric Values' metrics of CPU Percent Utilization, Logical I/) Rate, Disk I/O Rate, Communications I/) Rate, Transaction Rate, Transaction Time, Thread Count, and Page Fault Rate are not valid with: &MSGID, &MSGSEV, &MSGTYPE AND &JOBSTATUS•The 'Summary Numeric Values' metrics of CPU Percent Utilization, Logical I/) Rate, Disk I/O Rate, Communications I/) Rate, Transaction Rate, Transaction Time, Thread Count, and Page Fault Rate are not valid with: &JOBNAME, &JOBUSER, &JOBNUMBER, &JOBTYPE, &SBS, &SERVER &MSGID, &MSGSEV, &MSGTYPE AND &JOBSTATUS


Job Monitors

Caution!

Uses system resources!

Setting collection interval


How often?

5,15, 30 minutes or 1 hour

Tuning options

45

Notes: Job Monitors and System Resources

Job monitors connect to a QZRCSRVS job for each job that is being monitored for the Job Log Messages

and the Job Status metrics.

QZRCSRVS jobs are not Management Central jobs. They are IBM i TCP Remote Command Server jobs

that the Management Central Java server uses for calling commands and APIs. In order to process the

API calls for the Job Log Messages and Job Status metrics in a timely fashion within the job monitor’s

interval length, the APIs are called for each job concurrently at interval time.

When both metrics are specified on the same monitor, two QZRCSRVS jobs are started for each job. For

example, if 5 jobs are monitored for Job Log Messages, 5 QZRCSRVS jobs are started to support the

monitor. If 5 jobs are monitored for Job Log Messages and Job Status, then 10 QZRCSRVS jobs are started.

Thus, it is recommended that for standard systems, when you are using the Job Log Message and Job

Status metrics, you limit the number of jobs monitored on a small system to 40 jobs or less.


Job Monitors

PC Client

Server

Data will collect without

thresholds and actions

Job, Message and File Monitors

Actions


46

Job MonitorsSelect the monitor, then the start button


Job Monitors

Restart on failed systems

Overall status

View the status


47

Job Monitors

Status

Actions

DetailedInformation

Viewing the system and job information


Message Monitors

� Monitor the system by selecting ...

– A single message queue

– Message criteria• ID, severity or type

– Predefined metrics• Count...

� Enabled with...

– Event logging

– Trigger/Reset notification

– Command Automation

– System Actions• Message reply, delete...

Message monitors can be used to view messages across systems that match monitor criteria. You can work with the messages listed in the monitor (display details, reply, and delete).


48

Message MonitorsSelect 'New Monitor...' and specify General properties


Message MonitorsSelect 'Metrics to monitor' and thresholds then press OK to create

Messages


49


Message Monitor Replacement Variables:

Parameter Passed Data&DATE Date&MON Monitor name&INTVL Collection interval length in seconds&TIME Time&ENDPOINT Endpoint system name&EVENTTYPE Event type and defined as follows:

Triggered Event = 1Manual Reset Event = 3

&FRMJOBNUMBER Job number for the job causing the triggering message&FRMJOBNAME Job name for the job causing the triggering message&FRMUSER User owning the job causing the triggering message&FRMPROGRAM Name of the program causing the triggering message&MSGKEY 4-byte message key for the message causing the trigger (as a hex string)&MSGID Message ID causing the trigger&MSGSEV Message severity causing the trigger&MSGTYPE Message type causing the trigger&MSGCOUNT Current message count (that caused the trigger)&OWNER Monitor owner&THRESHOLD Threshold number causing the trigger&TOLIB Message queue's library to which this message was sent (the library of

the queue being monitored)&TOMSGQ Message queue name to which this message was sent (the queue being

monitored)


Message Monitors

Actions

Viewing the system and message information

Status


50

File Monitors

� Monitor the system by

selecting ...

– History Log (QHST) or specific files

– File criteria

• File location

– Predefined metrics

• Status, size and text

� Enabled with...

– Event logging

– Trigger/Reset notification

– CL Command Automation

You can use a file monitor to notify you whenever a selected file has changed, reached a specified size, or for specified text strings.


File MonitorsYou can select to monitor all system log files or selected files.


51

File Monitors

� Metrics

– Text

– File Status

– Size


Automatic Reset for Message and File Monitor Triggers

� Some metrics can be reset automatically after a trigger command runs

– Only available for Message and File monitors


52

Monitors

� Remaining properties

– Collection Interval

– Actions

– Systems and Groups

– Sharing


Sharing of Monitors

� Monitors can be shared

� Owner

– User that created the monitor

� None

– No one else can see it

� Read-only

– Others can see it

• i.e. view properties & copy it

� Controlled

– Other can perform actions

• i.e. start and stop


53

Notes: Sharing

The owner has specified one of the following levels of sharing:

NoneOther users cannot view this item.

Read-Only

Other users can view this item and use it (but can not start or stop it). Other users can create a new item based on this one and make changes to the new one as needed. However, other users cannot delete or change this item in any way. If you are the owner of a monitor and have specified actions (such as opening the event log window or sounding an alarm on the PC), these actions occur for all users of the monitor whenever a threshold is triggered or reset. The other users cannot change these actions.

Controlled

Other users can start and stop this item. Only the owner can change the level of sharing. Other users can also view this item and use it to create a new item based on this one. If you are the owner of a monitor and have specified actions (such as opening the event log window or sounding an alarm on the PC), these actions occur for all users of the monitor whenever a threshold is triggered or reset. The other users cannot change these actions. Actions are run under the profile of owner!© 2014 International Business Machines Corporation 105

Monitor Restart Options


54

Notes: Changing the Monitor Restart Options

Restarting Monitors

Monitor Restart was added to provide a way to automatically restart monitors when the Management Central servers have been interrupted. These interruptions could be as simple as the MC Central Server or MC Endpoint Server being restarted, or something more dramatic such as the temporary loss of communications between the Central Server and an Endpoint Server or a system being IPLed.

If you select to have the system automatically attempt to restart your monitors, you may also specify how long you want the central system to keep trying to restart the monitors and how often you want the system to try during that time period.

For example, if you want the system to try to restart monitors every five minutes for a period of 3 hours, you select 'Automatically restart monitors on failed systems' and then specify 180 minutes for 'How long to attempt restart' and 5 minutes for 'How often to attempt restart'.A change to this setting takes effect the next time the Management Central servers are restarted.All Monitors support the restart option.

Default behavior is OFF.

Documentation on how to automatically restart Management Central Monitors:https://www-304.ibm.com/support/entdocview.wss?uid=nas16e5b0871315547a68625729e004737ce


Monitoring with IBM Systems Director

Tips for Managing IBM i with IBM Systems DirectorWednesday at 8:00


55

Monitor Your i


IBM i Monitors


56

IBM i Monitors

� 30+ Metrics for common monitoring scenarios



57

Create Your Own Monitor


Director Agent Monitors (requires CAS agent)


58



59

Monitor Your IOA Cache Batteries

http://ibmsystemsmag.blogs.com/i_can/2012/04/monitoring-cache-battery-status-with-ibm-systems-director-63.html


Monitoring IBM i Processes

� Create a process monitor from the Manage Processes task

� Once a process monitor is createdadditional metrics can be monitored for that processusing the Create monitortask

� Requires CAS agent


60

Monitor Thresholds

� The process of setting threshold and enabling automation is the same regardless of the metric type

� For any metric that you want to monitor, you can set thresholds

� Once thresholds are set, you then can create an event filter for the event that occurs when the threshold setting is hit

� Automation plans complete the set up to enable automatic notification that the threshold was hit


Monitor Thresholds


61

Monitor Thresholds


Event Automation Plans


62

Event Actions


Monitoring Messages with IBM Systems Director Events


63

Event Filters for QSYSOPR messages


Monitoring QSYSOPR .... Event Automation Plan

� After you establish what you want to monitor by creating the

event filter for IBM i messages ...

� You must also create an event automation plan for the action

Director is to take when those events occur

Even though these events are generated by their respective operating systems (or an optional layer that is installed on the operating system), IBM Systems Director does not process these events unless you create an event automation plan to do so.


64

www.ibm.com/power/i


References• IBM i Information Center

http://publib.boulder.ibm.com/iseries/

• IBM Knowledge Center– http://www-01.ibm.com/support/knowledgecenter/ssw_ibm_i_71/welcome.html

– http://www-01.ibm.com/support/knowledgecenter/ssw_ibm_i_72/rzahg/ic-homepage.htm

• developerWorks - http://www.ibm.com/developerworks/ibmi/

• Navigator for i on developerWorkshttps://www.ibm.com/developerworks/mydeveloperworks/wikis/home?lang=en#/wiki/IBM%20i%20Technology%20Updates/page/IBM%20Navigator%20for%20i

• IBM Systems Directorhttp://www-03.ibm.com/systems/software/director/http://pic.dhe.ibm.com/infocenter/director/pubs/index.jsp


65

Special notices

This document was developed for IBM offerings in the United States as of the date of publication. IBM may not make these offerings available in other countries, and the information is subject to change without notice. Consult your local IBM business contact for information on the IBM offerings available in your area.

Information in this document concerning non-IBM products was obtained from the suppliers of these products or other public sources. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.

IBM may have patents or pending patent applications covering subject matter in this document. The furnishing of this document does not give you any license to these patents. Send license inquires, in writing, to IBM Director of Licensing, IBM Corporation, New Castle Drive, Armonk, NY 10504-1785 USA.

All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.

The information contained in this document has not been submitted to any formal IBM test and is provided "AS IS" with no warranties or guarantees either expressed or implied.

All examples cited or described in this document are presented as illustrations of the manner in which some IBM products can be used and the results that may be achieved. Actual environmental costs and performance characteristics will vary depending on individual client configurations and conditions.

IBM Global Financing offerings are provided through IBM Credit Corporation in the United States and other IBM subsidiaries and divisions worldwide to qualified commercial and government clients. Rates are based on a client's credit rating, financing terms, offering type, equipment type and options, and may vary by country. Other restrictions may apply. Rates and offerings are subject to change, extension or withdrawal without notice.

IBM is not responsible for printing errors in this document that result in pricing or information inaccuracies.

All prices shown are IBM's United States suggested list prices and are subject to change without notice; reseller prices may vary.

IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.

Any performance data contained in this document was determined in a controlled environment. Actual results may vary significantly and are dependent on many factors including system hardware configuration and software design and configuration. Some measurements quoted in this document may have been made on development-level systems. There is no guarantee these measurements will be the same on generally-available systems. Some measurements quoted in this document may have been estimated through extrapolation. Users of this document should verify the applicable data for their specific environment.

Revised September 26, 2006


Special notices (cont.)IBM, the IBM logo, ibm.com AIX, AIX (logo), AIX 5L, AIX 6 (logo), AS/400, BladeCenter, Blue Gene, ClusterProven, DB2, ESCON, i5/OS, i5/OS (logo), IBM Business Partner (logo), IntelliStation, LoadLeveler, Lotus, Lotus Notes, Notes, Operating System/400, OS/400, PartnerLink, PartnerWorld, PowerPC, pSeries, Rational, RISC System/6000, RS/6000, THINK, Tivoli, Tivoli (logo), Tivoli Management Environment, WebSphere, xSeries, z/OS, zSeries, Active Memory, Balanced Warehouse, CacheFlow, Cool Blue, IBM Systems Director VMControl, pureScale, TurboCore, Chiphopper, Cloudscape, DB2 Universal Database, DS4000, DS6000, DS8000, EnergyScale, Enterprise Workload Manager, General Parallel File System, , GPFS, HACMP, HACMP/6000, HASM, IBM Systems Director Active Energy Manager, iSeries, Micro-Partitioning, POWER, PowerExecutive, PowerVM, PowerVM (logo), PowerHA, Power Architecture, Power Everywhere, Power Family, POWER Hypervisor, Power Systems, Power Systems (logo), Power Systems Software, Power Systems Software (logo), POWER2, POWER3, POWER4, POWER4+, POWER5, POWER5+, POWER6, POWER6+, POWER7, System i, System p, System p5, System Storage, System z, TME 10, Workload Partitions Manager and X-Architecture are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries.

A full list of U.S. trademarks owned by IBM may be found at: http://www.ibm.com/legal/copytrade.shtml.

Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.

AltiVec is a trademark of Freescale Semiconductor, Inc.

AMD Opteron is a trademark of Advanced Micro Devices, Inc.

InfiniBand, InfiniBand Trade Association and the InfiniBand design marks are trademarks and/or service marks of the InfiniBand Trade Association.

Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.

IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency which is now part of the Office of Government Commerce.

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates.

Linear Tape-Open, LTO, the LTO Logo, Ultrium, and the Ultrium logo are trademarks of HP, IBM Corp. and Quantum in the U.S. and other countries.

Linux is a registered trademark of Linus Torvalds in the United States, other countries or both.

Microsoft, Windows and the Windows logo are registered trademarks of Microsoft Corporation in the United States, other countries or both.

NetBench is a registered trademark of Ziff Davis Media in the United States, other countries or both.

SPECint, SPECfp, SPECjbb, SPECweb, SPECjAppServer, SPEC OMP, SPECviewperf, SPECapc, SPEChpc, SPECjvm, SPECmail, SPECimap and SPECsfs are trademarks of the Standard Performance Evaluation Corp (SPEC).

The Power Architecture and Power.org wordmarks and the Power and Power.org logos and related marks are trademarks and service marks licensed by Power.org.

TPC-C and TPC-H are trademarks of the Transaction Performance Processing Council (TPPC).

UNIX is a registered trademark of The Open Group in the United States, other countries or both.

Other company, product and service names may be trademarks or service marks of others.

Revised December 2, 2010


monitoring system performance and health of ibm i tim · pdf file1 monitoring system...

Documents