document version [0.5] - ibm · pdf filethe agent monitors oracle alert log by reading alert...

Best Practice of Oracle Monitoring by ITCAM Extended Agent Jan 2015

Best Practices of Oracle Database Monitoring by ITCAM Extended Agent

Document version [0.5]

Tivoli China Development Lab

© Copyright International Business Machines Corporation 2010, 2015.US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

of 52

CONTENTS

Contents.............................................................................................................................iii

List of Figures....................................................................................................................vi

List of Tables.....................................................................................................................vii

Revision History...............................................................................................................viii

1Overview..........................................................................................................................1

1.1Planning your enterprise monitoring infrastructure............................................1

1.2Reference sample Oracle database environment.............................................1

2Strategy of Monitoring.......................................................................................................3

2.1Local Monitoring or Remote Monitoring.............................................................3

2.2Oracle Instant Client or Oracle Home................................................................3

2.3Database Connection Number per Agent Instance...........................................4

3Agent Installation and Configuration.................................................................................5

3.1Remote installation............................................................................................5

3.2Apply the latest Fix Pack and Interim Fix..........................................................5

3.3Oracle Instant Client..........................................................................................5

3.4Mount Oracle alert log directory from remote DB server...................................6

3.5Configure LISTENER.ORA and TNSNAMES.ORA files in agent side...............6

iii

3.6Monitor IPC/TCPS listener................................................................................8

3.7Agent configuration from command line............................................................8

3.8ASM database monitoring.................................................................................9

3.9Dataguard database monitoring......................................................................10

3.10[Optional] Use non-root user to startup agent to monitor Oracle database....11

3.11Check the status for connections..................................................................12

4Customization.................................................................................................................14

4.1Extend agent timeout......................................................................................14

4.2Listener Monitoring..........................................................................................14

4.2.1Disable the Listener/Net Service Monitoring..................................................14

4.2.2Selected Listener/Net Services to be monitored............................................14

4.3Alert Log Monitoring........................................................................................15

4.4Customized SQL.............................................................................................16

4.5Product Predefined SQL Customization..........................................................16

4.6Cluster Support...............................................................................................16

4.7Usage of KRZ_SQL_PREFILTER=TRUE/FALSE...........................................16

5FAQ................................................................................................................................18

5.1The SQL query the agent uses.......................................................................18

5.2How the agent checks net service status........................................................18

iv

5.3About RDB_Instance_Info/ASM_Instance attribute groups.............................19

6Oracle Monitoring Concept and Situations......................................................................20

6.1Monitor Oracle Availability...............................................................................20

6.1.1Monitor Oracle Instance Status......................................................................20

6.1.2Monitor Oracle Tablespace............................................................................22

6.1.3Monitor Oracle Automatic Storage Management (ASM)................................23

6.1.4Monitor Oracle Real Application Cluster (RAC).............................................24

6.1.5Monitor Oracle Database...............................................................................25

6.1.6Monitor Oracle Alert Log................................................................................26

6.2Monitor Oracle Performance...........................................................................27

6.2.1Buffer Cache Hit Ratio...................................................................................27

6.2.2System Global Area (SGA)............................................................................29

6.2.3Top SQL........................................................................................................31

6.2.4Lock Contention and Dead Locks..................................................................32

7Problem Determination...................................................................................................35

Appendix............................................................................................................................vi

8Appendix A. How to use Oracle Instant Client for Tivoli Extended Oracle Agent..............vi

References.........................................................................................................................8

v

LIST OF FIGURES

vi

LIST OF TABLES

Oracle Node, Instance Name, Server Name and the Net Service name in the Listener...................1

Compare of Local monitoring and Remote monitoring.....................................................................2

Environment variable for agent timeout.............................................................................12

Environment variables for alert log monitoring..................................................................13Agent Affinities...................................................................................................................14

vii

REVISION HISTORY

Date Version Revised By Comments

2011/4/27 0.1 [email protected] initials

2011/10/17 0.2 [email protected] Add more best practices items

2012-4-30 0.3 [email protected] Add ASM/Dataguard

10/15/12 0.4 [email protected] Add sample monitoring situations

1/26/15 0.5 [email protected] Revise and add new items

viii

1 OverviewThe IBM Tivoli Composite Application Manager Extended Agent for Oracle Database provides you with the capability to monitor Oracle Database. You can also use the agent to take basic actions with the Oracle Database.

IBM Tivoli Monitoring is the base software for the Oracle Database Extended agent. The Oracle Database Extended agent monitors general Oracle database performance, Oracle RAC performance, Oracle ASM performance, and Data Guard performance.

You can get detail about agent’s features from web link below:

http://www-01.ibm.com/support/knowledgecenter/SS3JRN_7.2.1/com.ibm.itcama.doc_7.2.1/oracleextended/fac_landing_install.html?cp=SS3JRN_7.2.1.1%2F10-0&lang=en

Oracle Real Application Cluster Database (RAC) support users to access the same data from multiple instances on different servers. In RAC environment, the instance level information includes multiple instances information; the database level information has only one copy.

For example, you can get following information for each instance: Instance Name, SGA, Session, and Process. The database level information includes Tablespace, Table, Index and Datafile, such kind of database level information is consistent and shared by all instances in a RAC environment.

This document will introduce the best practices in using the Oracle Extended Agent to monitor general Oracle database, RAC database, ASM database or Dataguard database.

1.1 Planning your enterprise monitoring infrastructureThe Oracle Extended Agent v6.3.1 requires IBM Tivoli Monitoring (ITM) Infrastructure version 622FP2 or higher.

1.2 Reference sample Oracle database environmentThe following Oracle Database environment will be used as sample environment in this document.

The RAC database is installed into two Solaris Sparc servers as two nodes (RAC1 and RAC2). The listener is running on port 1521 in both two nodes.

The Oracle cluster ware (including ASM) is installed at /opt/oracle/grid, the Oracle database software is installed at /opt/oracle/db. The Oracle database server’s alert log location is /opt/oracle/base/diag/rdbms/rac1/trace/alert_rac1.log. The Oracle software version is 11.2.0.1.

The Oracle RAC database myrac is created with two RAC instances, one instance name is rac1 in the RAC1 node, and the other instance name is rac2 in the RAC2 node.

1




The Oracle ASM database +ASM is created and there are also two instance, one instance is +asm1 in the RAC1 node, the other is +asm2 in the RAC2 node.

The Oracle Dataguard database is installed into two Solaris Sparc Sun OS 5.10 servers as Primary Node in the machine sunpri and standby node in the machine sunstd.

Table 1: Oracle Node, Instance Name, Server Name and the Net Service name in the Listener

Oracle Node Type Oracle Instance Name Server Name Net Service Name in Listener

RAC1 rac1 sol_rac1 myrac

RAC2 rac2 sol_rac2 myrac

ASM1 +ASM1 sol_rac1 +ASM

ASM2 +ASM2 sol_rac2 +ASM

Dataguard Primary oradb sol_dg1 dbpri

Dataguard Standby oradb sol_dg2 dbstd

2

2 Strategy of MonitoringThis chapter introduces the monitoring strategy with the Oracle Database Extended Agent about the agent capacity, limitation and scalability.

2.1 Local Monitoring or Remote MonitoringOracle extended agent support local monitoring and remote monitoring. Following table compares these two different monitoring ways.

Table 2: Compare of local monitoring and remote monitoring

Category Local Monitoring Remote MonitoringInstallation in Oracle Server machine

Yes No

Oracle physical disk usage monitoring

Yes Not support

Oracle alert log monitoring

Yes Yes, need to mount the remote machine’s log directory to local, and need to specify mount path at agent configuration

IPC listener monitoring Yes Not supportOther monitoring features

Yes Yes

Maintenance effort Big, need to apply agent patch in each Oracle server machines.

Little, only need to apply patch at agent installation machine once.

Note: For a RAC environment, we recommend installing the agent into one node of the RAC environment, and then the agent performs the remote monitoring. The disk usage for local node can still be monitored; the alert log for remote nodes can be monitored after mount the alert log directory at the remote node.

2.2 Oracle Instant Client or Oracle Home

The Oracle Database Extended agent is built with Oracle Call Interface (OCI) technology, the agent can use the dynamic library provided as Oracle Instant Client or Oracle server installation (as Oracle home).

We recommend using the Oracle instant client, because the user can always use the latest version of Oracle Instant Client package which contains the latest Oracle client dynamic library code fixes. By using the latest Oracle Instant Client, the user does not need to upgrade the Oracle server in case there are some known bugs in Oracle Client dynamic library.

Oracle Instant Client can be downloaded from the Oracle web site: http://www.oracle.com/technetwork/database/features/instant-client/index-097480.html

3

2.3 Database Connection Number per Agent InstanceWe suggest each agent instance monitor no more than 10 database connections in case there are no more than 5 situations or historical collections in one agent.

This will ensure the total request number in an agent instance is less than 50(10 connection * 5 situations/historical collections). If there are more situations or historical collections, then we suggest reducing the db connection number for each agent instance and adding more agent instances.

The number of agent instances is affected by ITM limitation. Including other agents, a maximum of 15 agent instances can be started on a single system by default. This is a common ITM limitation, but the limitation can be extended by changing agent instance’s configuration, pls refer to “Resolving the problem” in the below ITM technote:

No more that 15 IP.PIPE processes can be active on a single system image.http://www-01.ibm.com/support/docview.wss?uid=swg21263550

4

3 Agent Installation and Configuration

3.1 Remote installationThe Oracle Database Extended agent supports remote installation and you can install the agent into a separated remote machine with Oracle servers.

Example The agent installation directory is /opt/IBM/Tivoli.

3.2 Apply the latest Fix Pack and Interim FixBy Jan.2015, the latest version of the agent is 06.31.02.02.

We strongly recommend you apply the latest interim fix 6.3.1.2-TIV-ITM_KRZ-IF0002 if the agent version is in 06.31.02.00 in your environment, since 6.3.1.2-TIV-ITM_KRZ-IF0002 includes important fixes.

6.3.1-TIV-ITM_EXT-FP0002:http://www-01.ibm.com/support/docview.wss?uid=swg24035571

6.3.1.2-TIV-ITM_KRZ-IF0002:http://www-01.ibm.com/support/docview.wss?uid=swg24038442

For the most current information about the latest fix pack or interim fix, see the agent’s wiki page:https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Tivoli%20Composite%20Application%20Manager/page/Oracle_Extended_Agent

3.3 Oracle Instant ClientIf there is no Oracle Home or Oracle Client installed in the agent machine by the Oracle OUI installer, then you need to download the Oracle Instant Client package and extract it to one readable directory in the agent machine, for example, /opt/IBM/Tivoli/instantclient_11_2.

Note: The LITE version of Oracle Instant Client is NOT supported.

You need to create a soft link from libclntsh.so.11.1 to file libclntsh.so in the Oracle Instant Client directory.

Example #ln -s /opt/IBM/Tivoli/instantclient_11_2/libclntsh.so.11.1 /opt/IBM/Tivoli/instantclient_11_2/libclntsh.so

Note: There is a full step-by-step guide for the Oracle Instant Client downloading in the Appendix A. How to use Oracle Instant Client for Oracle Database Extended Agent.

5

3.4 Mount Oracle alert log directory from remote DB serverThe agent monitors Oracle alert log by reading alert log file configured in agent instance configuration item - Oracle Alert Log File Paths (KRZ_LOG_PATHS).

If the agent is installed in the remote server, Oracle alert log directory (for example /opt/oracle/base/diag) in the Oracle server need to be mounted to the agent machine.

Following are sample steps:

Step 1) Check if the NFS server is started in the Oracle server, or run the following command as user ‘root’.

# /etc/init.d/nfs.server start

Step 2) Check alert log directory permission in the Oracle server and add the “rx” permission for directory recursively to ensure the mount client can access the mounted alert log files, run the following command as user ‘root’.

# chmod -R +rx /opt/oracle/base/diag

Step 3) Share the alert log directory with following command in the Oracle server rac1.

# share -F nfs -o ro=rac1 /opt/oracle/base/diag

Step 4) Create a local directory and mount the remote alert log directory into local directory.

# mkdir /mnt/rac1# mount rac1:/opt/oracle/base/diag /mnt/rac1

3.5 Configure LISTENER.ORA and TNSNAMES.ORA files in agent side

The agent gets listeners’ and Net Service names’ information from listener definition file – listener.ora and Net Service name definition file – tnsnames.ora, and can monitor the status of listeners and Net Service names.

There are 3 places where the agent tries to read listener.ora and tnsnames.ora:

If you configure the agent instance with Oracle Home (KRZ_ORACLE_HOME), the agent tries to read listener.ora and tnsnames.ora from KRZ_ORACLE_HOME/network/admin on Unix or KRZ_ORACLE_HOME\network\admin on Windows.

If the environment variable TNS_ADMIN is defined in agent running environment or agent configuration files, the agent also tries to read listener.ora and tnsnames.ora from TNS_ADMIN/network/admin on Unix or TNS_ADMIN\network\admin on Windows.

You can also copy listener.ora and tnsnames.ora files from remote DB server systems or create self-defined listener.ora and tnsnames.ora files, put them into a directory on the system where the agent is installed, and configure their directory path into agent instance configuration item - Net Configuration Files Directories (KRZ_TNS_PATHS). Multi-directories are allowed to be configured. The agent tries to read listener.ora and tnsnames.ora from here as well.

6

Following is sample step to create self-defined tnsnames.ora:

Use editor like Vi to create a new file tnsnames.ora file in the server tivsun21 at /opt/IBM/Tivoli/config directory with following sample lines. The file permission should be set as rx-rx-rx.

Note: Connection name of TIVRAC must start from the far left of the line; other lines should NOT start from left of the line.

Example # cat /opt/IBM/Tivoli/config/tnsnames.ora

#RAC database connection stringTIVRAC =

(DESCRIPTION= (LOAD_BALANCE=ON) (FAILOVER=on)

(ADDRESS = (PROTOCOL=tcp)(HOST=sol_rac1)(PORT=1521))

(ADDRESS = (PROTOCOL=tcp)(HOST=sol_rac2)(PORT=1521))

(CONNECT_DATA=(SERVICE_NAME=myrac) )

)#ASM database connection stringTIVASM =

(DESCRIPTION= (ADDRESS = (PROTOCOL=tcp)(HOST=sol_rac1)(PORT=1521

)) (ADDRESS = (PROTOCOL=tcp)(HOST=sol_rac2)(PORT=1521

)) (CONNECT_DATA=(SERVICE_NAME=+asm)

) )#Dataguard database primary node connection stringTIVDGPRI =

(DESCRIPTION= (ADDRESS = (PROTOCOL=tcp)(HOST=sol_dg1)(PORT=1521)

) (CONNECT_DATA=(SERVICE_NAME=dgpri)

) )#Dataguard database standby node connection stringTIVDGSTD =

(DESCRIPTION=

7

(ADDRESS = (PROTOCOL=tcp)(HOST=sol_dg2)(PORT=1521))

(CONNECT_DATA=(SERVICE_NAME=dgstd) )

)

3.6 Monitor IPC/TCPS listener

For Inter-process communication (IPC) protocol listener monitoring. Only local agent (local agent means the agent is installed on the same system where the monitoring Oracle server resides) is able to monitor IPC listener; remote agent is unable to monitor IPC protocol listener, because the IPC listener only works on the local Oracle server.

For TCPS listener monitoring. You need to set TNS_ADMIN environment variable into agent running environment or agent configuration files. The TNS_ADMIN path need to include net service configuration file sqlnet.ora, and the sqlnet.ora file need to include necessary information for TCPS listener monitoring, for example WALLET_LOCATION. If sqlnet.ora does not include necessary information, the TCPS listener status is shown as Inactive, and you will see the following error message in agent log file:

"Oracle error at ServerAttach during Net Service checking" error code is:28759

3.7 Agent configuration from command line

Launch agent configuration from command line and input the specific value as following.

• Launch configuration tool from command line.

Example #/opt/IBM/Tivoli/bin/itmcmd config -A rz

• When ask for the input filed of “Oracle Home Directory”, input a blank space to make sure there is no valid value for “Oracle Home Directory”

Example Oracle home directory: (default is: ): [Input space and press Enter]

• When ask for the input field of “Oracle Instant Client Installation Directory”, input the path where the Oracle Instant Client is extracted.

Example Oracle instant client installation directory: (default is: ): /opt/IBM/Tivoli/instantclient_11_2

• When ask for selection to edit the database connection settings, input 1

Example Edit 'Database connection' settings, [1=Add, 2=Edit, 3=Del, 4=Next, 5=Exit] (default is: 5): 1

• When ask for the input filed of “Database Connection Name”, input the database name “rac”.

Example Database connection name: (default is: ): rac

8

• When ask for the input field of “Oracle Connection String”, input the tnsnames.ora connection identifier which is defined in the file /opt/IBM/Tivoli/config/tnsnames.ora, current value is “TIVRAC”. Or you can input the full connection string directly.

Example Oracle connection string: (default is: ): TIVRAC

Example Oracle connection string: (default is: ): (DESCRIPTION=(LOAD_BALANCE=ON)(FAILOVER=ON)(ADDRESS=(PROTOCOL=tcp)(HOST=sol_rac1)(PORT=1521))(ADDRESS=(PROTOCOL=tcp)(HOST=sol_rac2)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=myrac)))

• When ask for the input filed of Oracle alert log file paths, input the local alert log file full paths which is mounted from remote Oracle servers.

Example The absolute file path of mapped alert log files of remote database instances in this database connection. Multiple files are separated by ";" on Windows systems, or ":" on UNIX systems. Each file is matched to a database instance, by the file name pattern alert_<instance>.log, or ignored if unmatched. Local database instance alert log files can be discovered automatically. If this item was not configured, the alter logs of remote database instances would be unable to be collected. Oracle alert log file paths: (default is: ): /mnt/sol_rac1/rdbms/rac/rac1/trace/alert_rac1.log: /mnt/sol_rac2/rdbms/rac/rac2/trace/alert_rac2.log

3.8 ASM database monitoringYou can use existing agent instance to add one new connection to monitor one ASM database or create a new agent instance with a new connection.

• When ask for selection to edit the database connection settings, input 1


• When ask for the input field of “Database Connection Name”, input the connection name like “asm”.

Example Database connection name: (default is: ): asm

Note: Only letters, Arabic number, underline and minus characters can be used in the connection name. Other connection name like “+asm” is invalid.

• When ask for the input field of “Oracle Connection String”, input the tnsnames.ora connection identifier which is defined in the file /opt/IBM/Tivoli/config/tnsnames.ora, current value is “TIVASM”. Or you can input the full connection string directly.

9

Example Oracle connection string: (default is: ): TIVASM

Example Oracle connection string: (default is: ): (DESCRIPTION=(LOAD_BALANCE=ON)(FAILOVER=ON)(ADDRESS=(PROTOCOL=tcp)(HOST=sol_rac1)(PORT=1521))(ADDRESS=(PROTOCOL=tcp)(HOST=sol_rac2)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=+ASM)))

• When ask for the input field of “Username”, “Password” and “Role”, input the user information with SYSDBA role.

Example The set of privileges to be associated with the connection. For a user that was granted the SYSDBA system privilege, you can specify a connection that includes the privilege. If this item was not defined, 'DEFAULT' would be used as the role of the user. Role: [ 1=SYSDBA, 2=SYSOPER, 3=SYSASM, 4=DEFAULT ] (default is: 4): 1

Note: In Oracle 10g ASM database, a new Oracle user cannot be created in the sqlplus directly, you need to create the new Oracle user and grant as SYSDBA role in the RDBMS sqlplus and copy the RDB password file to the ASM password file, reference the below technote:

Create a database user for ASM with Oracle 10ghttp://www-01.ibm.com/support/docview.wss?uid=swg21589173

3.9 Dataguard database monitoring

You can use existing agent instance to add two new connections for Dataguard Primary database and Standby database separately, or you can create a new agent instance with two new connections.

• When ask for selection to edit the database connection settings, input 1 for Primary Node connection.


• When ask for the input filed of “Database Connection Name”, input the connection name “dgpri” for the primary node connection.

Example Database connection name: (default is: ): dgpri

• When ask for the input field of “Oracle Connection String”, input the tnsnames.ora connection identifier which is defined in the file /opt/IBM/Tivoli/config/tnsnames.ora, current value is “TIVDGPRI”. Or you can input the full connection string directly.

Example Oracle connection string: (default is: ): TIVDGPRI

Example Oracle connection string: (default is: ): (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=sol_dg1)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=dgpri)))

10

• When ask for the input field of “Username”, “Password” and “Role”, input the user with SYSDBA role. The user with SYSDBA role is not mandatory for Dataguard Primary Node,

• When ask for the selection to edit the database connection settings, input 1 for the Standby Node connection.


• When ask for the input filed of “Database Connection Name”, input the connection name “dgpri” for the primary node connection.

Example Database connection name: (default is: ): dgstd

• When ask for the input field of “Oracle Connection String”, input the tnsnames.ora connection identifier which is defined in the file /opt/IBM/Tivoli/config/tnsnames.ora, current value is “TIVDGPRI”.

Example Oracle connection string: (default is: ): TIVDGSTD

Example Oracle connection string: (default is: ): (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=sol_dg1)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=dgstd)))

• When ask for the input field of “Username”, “Password” and “Role”, input the user with SYSDBA role.

Example The set of privileges to be associated with the connection. For a user that was granted the SYSDBA system privilege, you can specify a connection that includes the privilege. If this item was not defined, 'DEFAULT' would be used as the role of the user. Role: [ 1=SYSDBA, 2=SYSOPER, 3=SYSASM, 4=DEFAULT ] (default is: 4): 1

3.10 [Optional] Use non-root user to startup agent to monitor Oracle databaseIf you want to use ITM utility $ITMHOME/bin/itmcmd to startup agent appending with agent instance name in the -o option by non-root user, eg. tivmon, you need to grant the r+w permission of files “installdir/config/.Config/RunInfo*” and “installdir/logs/*rz*” to the non-root user before starting the agent.

Example #su - tivmon -c “/opt/IBM/Tivoli/bin/itmcmd agent -o inst1 start rz”

3.11 Check the status for connections

Go to the Oracle Database Extended agent root node and check the “Database Status” column in the view of “Database Connection”.

11

If the “Database Status” is not “Active” or there is not data in the view of “Database Connection”, then check the “Error Message” column and “Suggestion” column in the Agent Event workspace.

Figure 1: Blank Database Connection and Agent Event

Figure 2: Dataguard Database Connections

12

Figure 3: ASM Database Connection

13

4 Customization

4.1 Extend agent timeoutYou can set the environment variable below to extend the agent itmeout from default 60 seconds to more.

Table 3: Environment variable for agent timeout

Variable Description

CDP_COLLECTION_TIMEOUT The maximum time of data collection. If the execution time exceeded, the agent stops the execution, and reports timeout. The default value is 60 (in seconds). The minimum value is 5 (in seconds).

4.2 Listener Monitoring

4.2.1 Disable the Listener/Net Service Monitoring

The Oracle Database Extended agent version 06.31.00.01 and above versions provide configuration parameters to disable Listener/Net Service monitoring:

Enhancement: INTERNAL

Abstract: Providing the option to control whether monitoring the listener and Oracle net-service.

Additional Information: The listener monitoring and net-service monitoring are started by default. We provide the option that if you do not want this monitoring; you can turn off the related monitoring functions.

The environment variables KRZ_LISTENER_PING_INTERVAL and KRZ_TNS_PING_INTERVAL control the interval. If the value is less than 1, then it exits the thread and stops checking.

Example KRZ_LISTENER_PING_INTERVAL=0

Example KRZ_TNS_PING_INTERVAL=0

4.2.2 Selected Listener/Net Services to be monitored

If the Oracle home directory is selected during the agent configuration, then agent will load Listener definition from file $ORACLE_HOME/network/admin/listener.ora and load Net Service definition from file $ORACLE_HOME/network/admin/tnsnames.ora automatically.

14

If user do not want to monitor all Listener or Net Service from ORA files in the Oracle home directory and do not want to revise those files too, customer can specify the TNS_PATH during the agent configuration and create new listener.ora file and tnsnames.ora files in the new path and only the targeted Listener or Net Service is specified in the new files.

Example This is the directory that contains Oracle database net configuration file. This directory is defined by the TNS_ADMIN environment variable for each Oracle database instance. The default directory is $ORACLE_HOME/network/admin on UNIX or Linux systems, and %ORACLE_HOME%\NETWORK\ADMIN on Windows systems. If there are multiple net configuration file directories, use ";" on Windows systems, or ":" on UNIX systems, to separate the directories. If this item was not configured, the default directory would be used. Net configuration files directories: (default is: ): /opt/IBM/Tivoli/config/myrac

4.3 Alert Log MonitoringTo customize the severity of Oracle alert log messages, reference Info Center:

http://www-01.ibm.com/support/knowledgecenter/SS3JRN_7.2.1/com.ibm.itcama.doc_7.2.1/oracleextended/fac_config_agentspecific_customizingoraclealertlog.html?cp=SS3JRN_7.2.1.1%2F10-0-2-2-6&lang=en

You can set environment variable in the table below for the alert log monitoring.

Table 4: Environment variables for alert log monitoring

Variable Description

KRZ_LOG_MAXREAD Only the latest part of alert logs is scanned by the Oracle Database Extended agent while the agent starts. The size of the scanned part is specified by the KRZ_LOG_MAXREAD attribute. The default value is 5M. The value must be positive. The valid data suffixes are M/m presenting megabyte, and K/k presenting kilobyte.

KRZ_LOG_INTERVAL This attribute defines the search interval (in seconds) for alert log files. The default value is 300 (seconds). Setting the value to 0 disables search. The valid range is from 60 to 3600.

KRZ_LOG_NOREPEAT If the value is true, the Oracle Database Extended agent does not re-send alert log message entries that were found in the last running of the Oracle Database Extended agent. The default value is FALSE. The valid values are TRUE and FALSE.

15

4.4 Customized SQLTo define a customized SQL statement to monitor in the Customized SQL workspace, reference Info Center:

http://www-01.ibm.com/support/knowledgecenter/SS3JRN_7.2.1/com.ibm.itcama.doc_7.2.1/oracleextended/fac_config_agentspecific_definingsql.html?cp=SS3JRN_7.2.1.1%2F10-0-2-2-8&lang=en

4.5 Product Predefined SQL CustomizationYou can use the environment variable KRZ_CUSTOM_SQLXML to specify a customized krzsql_custom.xml with the revised SQL query for the existing agent attribute groups like RDB_Session_Detail, reference below technote:

http://www.ibm.com/support/docview.wss?uid=swg21588392

4.6 Cluster SupportReference to below technote:

Agent clustering and positioning in the TEP Navigatorhttp://www-01.ibm.com/support/docview.wss?uid=swg21326517

Table 5: Agent Affinities

Affnities Node in the TEP

%IBM.OracleAgents Oracle Database Extended

%IBM.OracleAgentRDB Oracle RDBMS

%IBM.OracleAgentASM Oracle ASM

%IBM.OracleAgentDG Oracle Dataguard

4.7 Usage of KRZ_SQL_PREFILTER=TRUE/FALSEIt is default behavior that the Oracle Database Extended agent adds prefilter into SQL query that is sent to the Oracle database for data processing, and then returns data to the Tivoli Enterprise Monitoring Server with records matched with the filter condition. The purpose of this behavior is to improve the agent performance by reducing the data to be queried and sent.

The prefilter information is usually gotten from situation's formula. When the agent receives a request from the situation, this condition will be combined into the SQL query.

For example (Example 1):

A situation with formula tablespace_used_percentage >= 80%

The formula '>=80%' will be used in the where clause as below:

16

SELECT tablespace_name, tablespace_used_percentage, ... FROM gv$tablespace_table WHERE tablespace_used_percentage >= 80;

But the agent has problem to handle the situation which has override formula/includes the original situation's formula condition info. The root cause is that the override formula info cannot be passed to agent in current ITM agent factory framework.

For example (Example 2):

A situation with formula tablespace_used_percentage >= 80% and the override formula is tablespace_name = 'TEST' and tablespace_used_percentage >= 50.

Only the formula '>=80%' will be used in the where clause as below, because override formula info cannot be passed to agent in current ITM agent factory framework:

SELECT tablespace_name, tablespace_used_percentage, ... FROM gv$tablespace_table WHERE tablespace_used_percentage >= 80;

In above example, agent will only query data from Oracle which matches with original formula of >=80%, so the other tablespace which matches with override formula will not be queried and it will not be fired in the TEMS anyway. This is a potential drawback, as it can cause alerts to be missed.

To resolve this issue, This parameter KRZ_SQL_PREFILTER has been introduced by Oracle Extended Agent 6.3.1 Fix Pack 2. The valid values are TRUE and FALSE. To be consistent with previous releases, the default value is TRUE, and the agent has same behavior as previous releases. If you have override situation defined which is similar with example 2, you need to set KRZ_SQL_PREFILTER to FALSE in agent instance configuration file, and restart the agent instance. When KRZ_SQL_PREFILTER is set to FALSE, the agent will not use any prefilter, all data for the attribute group will be queried from database and be parsed to TEMS.

17

5 FAQ

5.1 The SQL query the agent usesYou can get query the agent uses for most attribute groups from the below file on the system where the agent is installed.UNIX:<CANDLEHOME>/<Arch>/rz/bin/krzsql.xmlWINDOWS:<CANDLEHOME>\TMAITM6(_x64)\krzsql.xml

Take "ASM Disk Group Capacity" workspace view as example, the corresponding attribute group is ASM_DiskGroup. Search ID="ASM_DiskGroup" from krzsql.xml, you can get the query for it:

For Oracle 10.1 and above version:<SQL DBVersion="101">...



5.2 How the agent checks net service statusFor agent v6.31.00.02 and above. If you configure the agent instance with Oracle Home (KRZ_ORACLE_HOME) and does not set KRZ_USE_ORACLE_TNSPING=FALSE, the agent will try to use KRZ_ORACLE_HOME/bin/tnsping on Unix or KRZ_ORACLE_HOME\bin\tnsping.exe on Windows to test the status of listeners by analyzing the output of tnsping utility. If you configure the agent instance with Oracle Instant Client, the agent will use Oracle OCI library API to get the status of listeners.

The agent always use Oracle OCI library API to get the status of net service names.

If the environment variable TNS_ADMIN is NOT set or there is not sqlnet.ora file in TNS_ADMIN, the agent generates net service configuration file <Installation_Dir>/logs/<Host_Name>_rz_<Agent_Instance_Name>/sqlnet.ora, for example,

-bash-3.2# cat sqlnet.ora# SQLNET.ORA Logging and Tracing parameters, generated by Oracle Extended AgentDIAG_ADR_ENABLED=offLOG_DIRECTORY_CLIENT=<Installation_Dir>/logs/<Host_Name>_rz_<Agent_Instance_Name>LOG_FILE_CLIENT=krzsqlnet.log

18

The agent auto-generated sqlnet.ora only contains basic configuration information of net service.

If the environment variable TNS_ADMIN is set, the agent copies listener.ora/tnsnames.ora/sqlnet.ora to <Installation_Dir>/logs/<Host_Name>_rz_<Agent_Instance_Name>. By default, LOG_DIRECTORY_CLIENT and LOG_FILE_CLIENT in sqlnet.ora will be modified to <Installation_Dir>/logs/<Host_Name>_rz_<Agent_Instance_Name>/krzsqlnet.log in order to ensure net service log information is generated in a file which the agent is able to control when the agent checks net service status. If you do not want the agent do this change, change KRZ_REDIRECT_TNS configuration item to FALSE.

Example of net service log information in net service log file:

Fatal NI connect error 12514, connecting to:

(DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=orcl)(CID=(PROGRAM=krzstart)(HOST=tivagent)(USER=root)))(ADDRESS=(PROTOCOL=TCP)(HOST=rac1.cn.ibm.com)(PORT=1521)))

5.3 About RDB_Instance_Info/ASM_Instance attribute groupsThe RDB_Instance_Info and ASM_Instance attribute groups were originally designed to only show information of active instances.

For agent version 6.31.00.01 and above versions, The RDB_Instance_Info and ASM_Instance attribute groups were enhanced to show both active and inactive instances. For inactive instance, the status is shown as ‘OFFLINE’. But there is a prerequisite to show inactive instance, the instance need to be active when the agent instance is started, otherwise, the agent is unable to save its information into cache and cannot show its information.

Enhancement: INTERNAL 145418Abstract: support instance OFFLINE status for RDB_Instance_Info and ASM_Instance_InfoAdditional Information: With this enhancement, RDB_Instance_Info and ASM_Instance_Info attribute groups can show instance information with 'OFFLINE' Status.

19

6 Oracle Monitoring Concept and Situations

6.1 Monitor Oracle Availability

6.1.1 Monitor Oracle Instance Status

The Oracle database is made of a set of operating system files containing data entered by users or applications and structural information about the database itself called metadata. Information is stored persistently in these files.

To enable users and applications to view or update data in the database, Oracle must start a set of processes, called background processes, and must allocate some memory to be used during database operation. The background processes and memory allocated by Oracle together make up an instance. An instance must be started to read and write information to the database. However, having a database is not necessary to run an instance.

Example Instance Overview workspace about Oracle Instance Status

Figure 4: Navigate to Instance Overview workspace

20

Figure 5: Instance Status table in the Instance Overview workspace:

Figure 6: Navigate to product predefined situations from Instance Node

21

Figure 7: Product predefined situations for Oracle Inactive Instance

6.1.2 Monitor Oracle Tablespace

Tablespace is the logical storage unit that groups related logical structure together.

Example Create a sample tablespace in the sqlplus:SQL> CREATE TABLESPACE tbs_3 DATAFILE 'E:\tbs_3.dbf' size 10M AUTOEXTEND ON MAXSIZE 80M;

Tablespace created.

You can navigate to the Tablespace Usage workspace from the Tablespace node.

22

Figure 8: Navigate to Tablespace Usage workspace

Then you can get the usage percentage for all tablespaces like TBS_3 which is created in the example above.

Figure 9: Non-Temporary Tablespace Usage table in the Tablespace Usage workspace

If the option of AUTOEXTEND for the Tablespace is OFF, Maximum File Size is equal to the Allocation Size, then %Free_To_Maximum is equal to the %Free_To_Allocated also.

% Free To Maximum = (Maximum file size-Used size)/Maximum file size*100%

% Free To Allocated = (Allocation size-Used size)/Allocation size*100%

In the sample Tablespace TBS_3, the Maximum file size is 80M, the Allocation size is10M and the Used size is 1M. So its %Free_To_Maximum is (80-1)/80*100%=98.75% which is greater than the %Free_To_Allocated of (10-1)/10*100%=90%.

6.1.3 Monitor Oracle Automatic Storage Management (ASM)

23

ASM is a vertical integration of both the file system and the volume manager built specifically for Oracle database files. It extends the concept of stripe and mirrors everything to optimize performance, while removing the need for manual I/O tuning.

Oracle storage is added and removed from ASM disk groups in units of ASM disks.

ASM disks can be monitored in the ASM Disk Capacity workspace linked from the Automatic Storage Management node.

Figure 10: navigate to the ASM Disk Capacity workspace

Figure 11: ASM Disk Capacity table in the ASM Disk Capacity workspace

6.1.4 Monitor Oracle Real Application Cluster (RAC)

RAC is an option that allows multiple concurrent instances to share a single physical database.

There are several workspaces for the RAC monitoring in the Node of Real Application Cluster.

24

Figure 12: Navigate to RAC related workspaces from Real Application Cluster node

Figure 13: GCS CR Latency Report for RAC monitoring

6.1.5 Monitor Oracle Database

Oracle is a relational database. In a relational database, all data is stored in two-dimensional tables that are composed of rows and columns. The Oracle Database enables you to store data, update it, and efficiently retrieve it.

Oracle provides software to create and manage the Oracle database. The database consists of physical and logical structures in which system, user, and control information is stored. The software that manages the database is called the Oracle database server. Collectively, the software that runs Oracle and the physical database are called the Oracle database system.

25

Figure 14: Navigate from Database node

Figure 15: Global Resources Consumption table in the Resource Limitation workspace

6.1.6 Monitor Oracle Alert Log

26

Each Oracle instance you run produces an alert log, which is a sequential log of text messages pertaining to overall database operations including Oracle startup and shutdown, major database events such as log archiving and tablespace definition, and certain categories of errors. Submitting a copy of your alert log data is required in many problem determination situations. Refer to the Oracle Database Administrator's Guide for general information on the alert log.

Figure 16: Alert Log workspace and its tables

6.2 Monitor Oracle Performance

6.2.1 Buffer Cache Hit Ratio

The buffer cache hit ratio calculates how often a requested block has been found in the buffer cache without requiring disk access. This ratio is computed using data selected from the dynamic performance view V$SYSSTAT. The buffer cache hit ratio can be used to verify the physical I/O as predicted by V$DB_CACHE_ADVICE.

Example below has been simplified by using values selected directly from the V$SYSSTAT table, rather than over an interval. It is best to calculate the delta of these statistics over an interval while your application is running, then use them to determine the hit ratio.

27

SQL> SELECT NAME,VALUE FROM V$SYSSTAT

2 WHERE NAME IN ('db block gets from cache', 'consistent gets from cache', 'physical reads cache');

NAME VALUE

--------------------------------------------- ----------

db block gets from cache 81182

consistent gets from cache 419831

physical reads cache 13286

Using the values in the output of the query, calculate the hit ratio for the buffer cache with the following formula:

1 - (('physical reads cache') / ('consistent gets from cache' + 'db block gets from cache')

28

Figure 17: Buffer Cache performance table from Cache node

6.2.2 System Global Area (SGA)

SGA is a group of shared memory structures that contain data and control information for one Oracle database instance. If multiple users are concurrently connected to the same instance, then the data in the instance's SGA is shared among the users. Consequently, the SGA is sometimes referred to as the shared global area.

29

Figure 18: Navigate to SGA workspace from Memory node

Figure 19: SGA Usage bar chart in the SGA Detail workspace

There are two different concepts about SGA free memory with the Oracle Database Extended agent. Here is a clarification of the two attributes, this is normal behavior.

In Figure 20, the "Free Total SGA (MB)" attribute in the "SGA Allocation" attribute group uses the v$sgastat Oracle performance view. It means the free memory of allocated memory in the buffer cache, shared pool, etc.

30

Figure 20: Free Total SGA (MB) in the SGA Allocation workspace

In Figure 21, the “Free SGA Memory Available (MB)" attribute in the "SGA Usage" attribute group uses the v$sgainfo Oracle performance view. It means the free SGA memory which is still not allocated to buffer cache, shared pool, etc.

31

Figure 21: Free SGA Memory Available (MB) in the SGA Usage workspace

6.2.3 Top SQL

The Top SQL section describes the top SQL statements of the sampled session activity. Use this information to identify high-load SQL statements that may be the cause of the transient performance problem.

32

Figure 22: Top SQL workspace

Figure 23: Top SQL By Elapsed Time workspace

6.2.4 Lock Contention and Dead Locks

When two Oracle users are attempting to change the same row, there is a lock contention with the lock type of TX. A deadlock can occur when two or more Oracle users are waiting for data locked by each other. Deadlocks prevent some transactions from continuing to work. Figure below is a hypothetical illustration of two transactions in a deadlock.

33

Figure 24: Two Transaction in a deadlock

In Figure 24, no problem exists at time point A, as each transaction has a row lock on the row it attempts to update. Each transaction proceeds without being terminated. However, each tries next to update the row currently held by the other transaction. Therefore, a deadlock results at time point B, because neither transaction can obtain the resource it needs to proceed or terminate. It is a deadlock because no matter how long each transaction waits, the conflicting locks are held.

Figure 25: Navigate to Lock Contention from Contention node

34

Figure 26: Lock Conflict table with the deadlock record in Lock Contention workspace

35

7 Problem Determination

Troubleshooting: Slow Database Monitoring

http://www-01.ibm.com/support/docview.wss?uid=swg21508178

Question:

How do you extended the timeout period of the oracle agent?

Cause:

The oracle database is slow on the system and the agent often times out during monitoring.

Answer:

Update the CDP_COLLECTION_TIMEOUT environment variable. The default is 60 seconds and the agent needs more time to verify database connection.

Troubleshooting: TOP SQL historical collection

http://www-01.ibm.com/support/docview.wss?uid=swg21511760

Question:

How to configure the TOP SQL attribute group historical collection for the RZ agent

Answer:

To configure the historical collection for the TOP SQL attribute group, the user needs to click on the third tab ("Filters") and fill the requested filters: "begin hour" ,"end hour", "order by" and "row order". This setting is required to get the historical collection enabled for this attribute group.

Historical collection settings:

36

Figure 27: History Collection Configuration for TOPSQL

Troubleshooting: DB connection fails during Oracle Extended Agent configuration

Problem (Abstract):

Attempts to configure result in error "ORA-01722: invalid number" despite IF0001 is already applied for the RZ agent v6.3.1.

Symptom:

Error message "ORA-01722: invalid number"

Resolving the problem:

If the IF0001 has been applied, you can safely ignore the error message for test connection, as follows.

1. Set "NLS_NUMERIC_CHARACTERS" in <AGENT_INSTALL_HOME>/config/rz.ini

NLS_NUMERIC_CHARACTERS=.

Reconfigure agent and ignore the above error message.

Try to start this agent, if agent is not started, try next step (2).

2. Set "NLS_NUMERIC_CHARACTERS" in <AGENT_INSTALL_HOME>/config/rz.ini

NLS_NUMERIC_CHARACTERS=.,

Reconfigure agent and ignore the above error message.

37

Try to start this agent, if agent is not started, please collect the debugging data for the IBM support investigation as explained at next step (3).

3. Set "KBB_RAS1" in <AGENT_INSTALL_HOME>/config/rz.ini

KBB_RAS1=ERROR (UNIT:krz all)

Start agent and send to IBM Support the output of "digup -a" command.

Also, please use "sqlplus" to connect to database directly: issue following SQL and send the output to the IBM Support

select * from nls_database_parameters;

select * from nls_instance_parameters;

select * from nls_session_parameters;

select userenv('language') from dual;

select * from v$nls_valid_values;

Then quit "sqlplus" and setup the environment as follows:

NLS_LANG=AMERICAN_AMERICA.AL32UTF8 && export NLS_LANG

NLS_NUMERIC_CHARACTERS=. && export NLS_NUMERIC_CHARACTERS

Use again "sqlplus" to connect to oracle db and issue:

select userenv('language') from dual

select * from nls_session_parameters

Then quit "sqlplus" and setup system environment again as follows:

NLS_NUMERIC_CHARACTERS='.,' && export NLS_NUMERIC_CHARACTERS

Finally, issue the last SQL command:

Select * from nls_session_parameters

Please send the outputs from all the above SQLs statements to the IBM Support for more investigation.

Troubleshooting: Monitoring Oracle alert logs with remote RZ agent

Question:

How to remotely monitor the Oracle alert log on AIX with an RZ agent installed on Windows?

38

Answer:

For monitoring remote alert logs on AIX, UNIX or Linux systems, the file system containing the logs needs to be locally mounted on the RZ agent machine.

To mount AIX file systems in Windows, you need to install Samba on AIX.

Troubleshooting: Oracle Extended Agent (rz) fails loading Oracle OCI library

Problem (Abstract):

The Oracle Extended Agent is not running properly and produces the error: Loading Oracle OCI library failed!

Symptom:

The Oracle Extended Agent starts and connects to TEMS but the Agent does not display Oracle data.

Cause:

This error occurs because the Oracle Extended Agent(rz) can not use the Oracle Client Interface(OCI) to connect to the Oracle instance.

Diagnosing the problem:

The Oracle Extended Agent on unix/linux starts the krzagent and then launches the krzclient process. The krzclient process fails when it tries to connect to the Oracle database.

Example instance_rz_server_krzclient_timestamp.log reflects the following errors:

Fail to dynamically load library libclntsh.so.10.1, return code 8

Try to load another possible Oracle OCI library

Fail to dynamically load library libclntsh.so.11.1, return code 2

Loading Oracle OCI library failed!

Resolving the problem:

Reconfigure the agent and correct Oracle home or Oracle Instant Client Library. Reconfigure the agent and restart.

39

APPENDIX

8 Appendix A. How to use Oracle Instant Client for Tivoli Extended Oracle AgentStep 1) open Oracle Instant Client download page from Oracle Site below:

http://www.oracle.com/technetwork/database/features/instant-client/index-097480.html

Step 2) click one suitable platform from list, especially for the 32-bit and 64-bit platforms.

Figure 28: Selection of Oracle Instant Client Downloads

Step 3) select ‘Accept License Agreement’. For example:

vi

“Instant Client Downloads for Solaris Operating System(SPARC 64-bit)

Step 4) download Instant Client Package - Basic: All files requires to run OCI, OCCI, and JDBC-OCI applications.

Example basic-11.2.0.2.0-solaris-sparc64.zip (64,102,971 bytes) (cksum - 4103480643)

vii

REFERENCES

[Reference List] (Use bibliography format. Style automatically creates a hanging indent when text runs over to the next line.)

Best Practices for Oracle Monitoring with Oracle Extended Agent.错误！文档中没有指定样式的文字。

®

© Copyright IBM Corporation 2009IBM United States of AmericaProduced in the United States of America US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not grant you any license to these patents. You can send license inquiries, in writing, to:

IBM Director of LicensingIBM CorporationNorth Castle DriveArmonk, NY 10504-1785 U.S.A.

The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law:INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PAPER “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes may be made periodically to the information herein; these changes may be incorporated in subsequent versions of the paper. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this paper at any time without notice.

Any references in this document to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.

IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to:

IBM Director of LicensingIBM Corporation4205 South Miami BoulevardResearch Triangle Park, NC 27709 U.S.A.

All statements regarding IBM's future direction or intent are subject to change or withdrawal without notice, and represent goals and objectives only.

This information is for planning purposes only. The information herein is subject to change before the products described become available.

If you are viewing this information softcopy, the photographs and color illustrations may not appear.

IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml.

==> The listed trademarks of the following companies require marking and attribution only if you mention any of them in your paper:

9

http://www.ibm.com/legal/copytrade.shtml

Best Practices for Oracle Monitoring with Oracle Extended Agent.错误！文档中没有指定样式的文字。

Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.

IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency which is now part of the Office of Government Commerce

Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.

Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office

UNIX is a registered trademark of The Open Group in the United States and other countries.

Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom.

Java and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

Other company, product, or service names may be trademarks or service marks of others.

10

document version [0.5] - ibm · pdf filethe agent monitors oracle alert log by reading alert...

Documents