IBM Tivoli Technical User’s Conference

© IBM Corporation 2006

Chicago, IL

C09

Brent Kellis

IBM Tivoli Monitoring 6.1 Intro to Deployment and Troubleshooting

April 24-27, 2006

®

®

IBM Software Group

© IBM Corporation

Session C09

ITM v6.1: Intro to Deployment and Troubleshooting

Brent Kellis bkellis@us.ibm.com

IBM Software Group

3

Topics

ITM v6.1 Deployment Overview

ITM v6.1 Intro to Troubleshooting & Serviceability

®

IBM Software Group

© IBM Corporation

Section 1

ITM v6.1 Deployment Introduction

IBM Software Group

5

Deployment / Installation Methods

ITM 6.1 installation consists of the several supported methods of installing the solution. Local Installation

Performed at the console or server (remote access or on location)

Installs components on one host at a time.

Agent Deployment – Remote Installation (new in ITM 6.1) Uses packages or Agent Bundles that can be rolled out to the

enterprise from a central location.

Silent Installation Inputs installation information once through a response file instead of

repeatedly through a GUI or CLI.

This presentation is not intended to be a comprehensive installation and configuration guide.

IBM Software Group

6

Planning Considerations

Approach the deployment from a business objective perspective. What conditions impact the business that ITM can address? How can we architect the solution to mesh into our existing business

processes? How people use the solution (help desk, command center,

administrators, management, etc.) Logical mapping and navigation to match business applications and

objectives.

Understand the technical environment in which the implementation will take place. Physical topology. Applications being monitored, complexity, and required automation. Number and types of TEP users. Estimated event volume. Integration with other tools and/or consoles.

IBM Software Group

7

Planning Considerations

There is a fair amount planning that should be done well in advance to ever spinning a CD-ROM.

Once the planning is complete and has been validated, the installation process should be fairly straight forward.

Start small with a few key objectives/milestones to work out any problems and gain confidence from your internal customers.

You can spend your time planning on the front-end, or troubleshooting on the back-end. It’s your choice!

IBM Software Group

8

Typical ITM 6.1 Architecture

Much of this has been already been reviewed in TTEC calls, Tivoli Advisor, etc.

`

TEMA

TEP Desktop

TEP Browser

TEPS

TEPS DBTEMS DB

TEMS

WarehouseProxy Agent

RTEMS

`

TEMA

`

Summarization& Pruninng

Agent

TDW 2.1 DB

ITM 6.1 Architecture with Tivoli Data Warehouse Components

`

UA TEMA*- SNMP- HTTP

* The Universal Agent has numerous data providers, many of which can provide remote or “agentless” monitoring functionality.

IBM Software Group

9

Installation Prerequisites

Ensure you understand and meet the pre-requisites Hardware

OS Versions, PTF’s, Service Packs, etc.

Required database and appropriate version.

Communication (network) protocols, ports, firewalls.

Ensure you have the correct binaries & distribution media for the applications you plan to monitor.

There are special provisions made for upgrading from DM, ITM5, and OMEGAMON XE/DE. Make sure you have planned accordingly.

IBM Software Group

10

Local Windows Installation

The installation is composed of an InstallShield based installer and several programs used for configuring and managing ITM services. Platform and Agent installs are all based on the same code

Installs GSKit 7.0.3.18 and IBM JRE 1.4.2 as prerequisites. For a few TEMA’s, the JRE is not a prerequisite and not installed.

IBM Tivoli Monitoring provides security through the use of Global Security Toolkit (GSKit) for SSL communications.

For purposes of this session, we will assume the appropriate database has been previously installed, configured, created and running.

IBM Software Group

11

Local Windows Installation

The installation “Destination Folder” does not support spaces!

You will need to provide an encryption key that will be used for all SSL connections.

During a modification of an existing install, if a user deselects anything, that item will be removed.

IBM Java 1.4.2 is automatically installed if any component being installed requires it. All servers require it as well as some agents.

IBM Software Group

12

Local Windows Installation – TEPS Data Source

This datasource (TEPS2 for DB2 and TEPS for MSSQL) is used to store the TEP Server workspaces.

This dialog is being controlled by DB2DataSource.exe (for DB2) or CNPSDataSource.exe (for MSSQL).

During an upgrade, the existing datasource will be used.

If using MS SQL Server, make sure the JDBC driver has been downloaded, installed.

IBM Software Group

13

Local Service Configuration and Management

Primarily done through Manage Tivoli Enterprise Monitoring Services (kinconfg.exe) – also referred to as MTEMS.

These programs are run either silently or graphically during in the install when the user is asked for configuration information

After installation, agents and servers can be managed and reconfigured if needed through the MTEMS interface.

IBM Software Group

14

Local UNIX Installation

The Install is composed of a command line interface (CLI). Although the GUI interface is still present via Candle utilities, it is no

longer recommended or supported.

Use the same Installation Prerequisites and Planning Considerations reviewed earlier in the presentation.

The overall flow of the installation performed on UNIX is similar to an MS Windows installation. Some configuration tasks are initiated in a different order.

Install one product / component at a time.

IBM Software Group

15

Local UNIX Installation

User will need to provide: Encryption key.

UNIX platform type (usually pre-selected by installation script).

Products or components to be installed.

Tivoli Enterprise Management Server (TEMS) name.

Some ITM 6.1 infrastructure components are currently not supported on a UNIX platform.

IBM Software Group

16

Local UNIX Installation – Screen Captures

Installation interface started by executing install.sh

IBM Software Group

17

Local UNIX Installation & Management

Install install.sh – Install products

uninstall.sh – Uninstall products

Config itmcmd config – Configure agents (including TEP/TEPS) or servers

itmcmd support – Add product support for TEMS

Manage itmcmd agent – Start/stop an agent (including TEP/TEPS)

itmcmd server – Start/stop a TEMS

itmcmd has many other variations for the UNIX platforms.

IBM Software Group

18

Agent Deployment – Introduction

Agent Deployment provides the ability to deploy and/or update agents from a central location.

There are two types of agent deployments OS agent deployment using the CreateNode command.

Non-OS agent (such as the DB2 agent) deployment using the Tivoli Enterprise Portal GUI (for other non-OS agents) or the addSystem command.

Non-OS agent deployment requires an existing OS Agent to be installed and running on the target host.

IBM Software Group

19

Agent Deployment – Component Overview

ITM 6.1 OS TEMA’s Have been enhanced to handle agent deployment activities at the agent end.

Launches local installation and configuration commands

Sends status information to the deployment controller

Agent Depot A repository of agent bundles that a Management Server can access during

agent deployment. (tacmd viewdepot or tacmd listbundles)

Deployment Controller A service on the Management Server whose role is to act as the driver for

the deployment.

The Deployment Controller queries the contents of the agent depot and transfers agent bundles using Remote Procedure Calls (RPC)

IBM Software Group

20

Agent Deployment - Process

Agent deployment consists of carrying out a list of generic activities that need to take place in order to set up a functioning monitoring agent on a machine.

These activities consist of: determining which agents are already deployed

transferring necessary agent bundles

installing agent bundles

configuring agent instances

starting monitoring agent instances.

IBM Software Group

21

Application Agent Deployment – TACMD CLI

Using the tacmd command, users can Perform agent deployment

Perform administrative tasks on agents an

Maintain agent depot

Perform situation maintenance and basic operations

tacmd uses the Simple Object Access Protocol (SOAP) service on TEMS.

IBM Software Group

22

OS Agent Deployment – TACMD CLI

ITM 6.1 OS agents are required to accommodate the deployment process. The OS agent installs the required infrastructure for remote deployment and

maintenance.

Installing the initial OS agent can be done remotely, but follows a specific process using the CreateNode command after doing a tacmd login.

IBM Software Group

23

Agent Deployment – Add via TEP

Agent deployment options are displayed for systems that have deployment-enabled OS agents.

IBM Software Group

24

Agent Deployment – Manage via TEP

Agent administration options can be executed on agents that are “deployment-enabled”

IBM Software Group

25

Agent Deployment & Management – Which method to use? Initial Deploy

Non-OS agent – TEP or tacmd addSystem OS agent – tacmd createnode command

Remove/Uninstall - TEP

Upgrade – TEP or tacmd updateAgent

Start/Stop/Restart – TEP or tacmd startAgent, stopAgent, restartAgent

Add/Remove Instance TEP or tacmd addSystem TEP for Remove

Reconfigure – TEP or tacmd configureSystem

IBM Software Group

26

Silent Installation

This method of installation is useful for advanced users who prefer to input installation information once through a response file instead of repeatedly through a GUI or CLI.

Can be used for Windows and UNIX installations

A sample silent installation response file (silent.txt)is provided on the product CD.

Complete instructions are documented in the manual as well as in silent.txt.

Running the silent installation

Setup /z"/sfC:\temp\SILENT.TXT" /s /f2"C:\temp\silent_setup.log"

(see doc for parm definition)

IBM Software Group

27

Silent Installation - silent.txt

®

IBM Software Group

© IBM Corporation

Section 2

ITM 6.1 Intro to Troubleshooting & Serviceability

IBM Software Group

29

Introduction

Serviceability One of the goals of Serviceability is to improve customer

satisfaction by making problem resolution more efficient and accurate

Provide tools to troubleshoot and get the product deployed

General rule: 90 percent of occurring problems are related in misunderstanding or in the environment of the installation. Only 10 percent are actual defects… (remember the planning slide!)

IBM Software Group

30

Considerations

General troubleshooting skills are essential

Rule out general issues Network issues

IP/Name resolution

Protocols in use

Firewalls/port restrictions

Database issues Database running

Database access

Invalid Queries

User issues User ID/password

Tracing and Logs TEMS, TEPS, Agents

IBM Software Group

31

Network Issues

Name resolution Ping the server/agent using name and IP If configured with fully qualified hostname ensure it can be resolved

There is a 32 character Managed System name limitation

Protocols in use IP.PIPE, IP.SPIPE, IP.UDP & SNA Ensure that the components use a protocol that is configured.

For example: Agent configured to use IP.SPIPE make sure that the TEMS is configured with at least one protocol to be IP.SPIPE.

Firewalls TEPS access configured using KFW_INTERFACES, KFW_INTERFACE_name_PORT See ITM 6.1 Installation Guide section on firewall scenarios for TEPS

Ports Ensure the ports used by the product are not blocked 1918 default for TEMS 1920 default URL port to connect to TEPS 15001 default TEPS port for IOR 3601 default Warehouse proxy port (Historical collection) 1919 default UA port for primary UA

Multiple NIC’s and Virtual Machines

IBM Software Group

32

Database Issues

Ensure that the database server is running:TEPS (example DB2)

db2 connect to teps user db2admin using <passwd> db2 select * from teps.kfwdbverRLSESTAMP MIGRATIONDATE FROMRLS

v610_kcj5293b 2005-11-29-13.52.41.571001 v610_kcj5293b

Has the password been reset on the TEPS user?+438BC885.0000 SQLSTATE: 08001, ERR: -30082, MSG: [IBM][CLI Driver] SQL30082N Attempt to establish connection failed with security reason "24" ("USERNAME AND/OR PASSWORD INVALID"). SQLSTATE=08001

IBM Software Group

33

Database Issues – SQL Errors

Some views within a workspace are either unpopulated or non-existent. Usually implies application support has not been added for the products your

are attempting to monitor. Logs will indicate bad queries being made or SQL errors.

Make sure you have added application support to the TEMS/TEPS for the TEMA’s you have installed.

Similar problems can introduced in upgraded environments (OMXE ITM 6.1) and is frequently caused by mis-matched versions of application support. Ensure application support is applied uniformly to the environment after

upgrading components. Also referred to as data definition files or seed data in the documentation.

IBM Software Group

34

User Issues – TEPS login Does the User ID exist as a user in the TEPS?

Check the existence of the user in the TEPS

Can also check in the TEPS database: select id, name from teps.kfwuser

Is security enabled on the TEMS? If yes, then make sure that the user also exists on the TEMS

Currently the TEMS uses native operating system for authentication Also uses Windows Domain Accounts

User ID is limited to 10 characters without spaces

IBM Software Group

35

Trace Facilities and Logs

Trace settings Default is normally ERROR

Trace levels are relatively easy to change and can be set dynamically in many cases.

ini & config files

GUI interface for some components & TEMA’s

ITM Service Console

Improperly set or left unchecked, some trace levels can have a severe impact on host performance. Thus, IBM currently recommends setting trace levels only under the direction of Tivoli Support.

IBM Software Group

36

!438C503E.0000!========================> IBM Tivoli RAS1 Service Log <========================

+438C503E.0000 System Name: REDBEARD Process ID: 752

+438C503E.0000 Program Name: kfwservices User Name: SYSTEM

+438C503E.0000 Task Name: kfwservices System Type: WinXP;5.1-SP1

+438C503E.0000 MAC1_ENV Macro: 0xC112 Start Date: 2005/11/29

+438C503E.0000 Start Time: 07:57:34 CPU Count: 1

+438C503E.0000 Page Size: 4K Phys Memory: 2047M

+438C503E.0000 Virt Memory: 2048M Page Space: 3944M

+438C503E.0000 Service Point: system.redbeard_cq UTC Start Time: 438c503e

+438C503E.0000 ITM Home: C:\IBM\ITM ITM Process: redbeard_cq

+438C503E.0000 Executable Name: C:\IBM\ITM\CNPS\kfwservices.exe

+438C503E.0000 KBB_RAS1: ERROR (UNIT:ctdatabus INPUT,ERROR) (UNIT:ctsql INPUT,ERROR)

+438C503E.0000 KBB_RAS1_LOG: C:\IBM\ITM\logs\REDBEARD_cq_438c503e-.log INVENTORY=C:\IBM\ITM\logs\REDBEARD_cq.inv MAXFILES=32 LIMIT=5 COUNT=5 PRESERVE=1

+438C503E.0000 KBB_ENVPATH: C:\IBM\ITM\CNPS\KFWENV

+438C503E.0000 ================================================================================

(438C503E.0000-2F4:RAS1,400,"CTBLD")

+438C503E.0000 Component: kbb

+438C503E.0000 Driver: d5298a/3581245.1

+438C503E.0000 Timestamp: Oct 25 2005 21:05:42

+438C503E.0000 Target: wv7i386

(438C503E.0001-2F4:RAS1,400,"CTBLD")

+438C503E.0001 Component: kcj

+438C503E.0001 Driver: kcj/CTTools/ctbld00.ctb, CTTools, TOP_dev_stage2, v610_kcj5293b/3581245.23

Trace Log – Sample Header TEPS Log

Build level

Trace setting

Component (TEPS)

Installation directory

Env file

IBM Software Group

37

Select Log File Locations

Component

Component code(three letter/two

letter) Location

Tivoli Enterprise Portal Server (TEPS) - Windows

KFW/FW (logs use the UNIX code of CQ) install_dir\logs\hostname_cq_timestamp-##.log

Tivoli Enterprise Portal Server (TEPS) - Linux KCQ/CQ install_dir/logs/hostname_cq_timestamp-##.log

Tivoli Enterprise Monitoring Server (TEMS) - Windows and UNIX/Linux KMS/MS install_dir/logs/hostname_ms_timestamp-##.log

Tivoli Enterprise Monitoring Agents (TEMA) - Windows Depends

install_dir\tmaitm6\logs (varies by agent) - RAS1 - hostname_xx_component_timestamp-##.log - Operations - *.LG0 and *.LG1

Tivoli Enterprise Monitoring Agents (TEMA) - UNIX/Linux Depends

install_dir/logs (varies by agent) - RAS1 - hostname_xx_component_timestamp-##.log - Operations - *.LG0 and *.LG1

Warehouse Proxy - Windows only KHD/HD install_dir/logs/hostname_hd_timestamp-##.log

Command line - tacmd - Windows KUI/UI install_dir\bin\kuiras1.log

Command line - tacmd - UNIX/Linux KUI/UI install_dir/logs/kuiras1.log

IBM Software Group

38

Select Log File Locations - continued

ITM 5.x Monitoring Agent KTM/TMEndpoint logs: $LCF_DATDIR/LCFNEW/KTM/logs/epname_tm_timestamp-xx.log

    TMR logs: $DBDIR/KTM/logs

    Windows - use %DBDIR% or %LCF_DATDIR%

Seeding process - Windows KCI/CI install_dir\CNPS\logs\seedPPC.log - PPC is the component code

    install_dir InstallITM\logs\CMSSeed.log for the TEMS

Seeding process - UNIX/Linux KCI/CI install_dir/logs/hostname_ci_<tems pid>.log

Summarization and Pruning Agent KSY/SY install_dir/logs/hostname_sy_timesta mp-xx.log

Installation logs - Windows   install_dir\InstallITM

    IBM Tivoli Monitoring date PID.log - Main installation log.

    TEPS_ODBC.log - Configuration of the TEPS ODBC connection.

    Warehouse_Configuration.log - Configuration of the Warehouse proxy.

Installation logs - UNIX/Linux   install_dir/logs

    candle_installation.log - Main installation log.

    InstallPresentation.log - TEPS seeding log.

    db2prep.log - TEPS configuration log - assuming DB2 is the database.

IBM Software Group

39

Key ITM 6.1 Processes

Windows TEP browser client iexplore.exe TEP desktop clientjava.exe TEP Server KfwServices.exe TEMS kdsmain.exe and cms.exe Windows Agent kntcma.exe Universal Agent kuma610.exe Other Agents <pc>*.exe where <pc>=product code MTEMS kinconfg.exe

UNIX TEMS kdsmain and cms UNIX OS Agent kuxagent UNIX log Agent kulagent

IBM Software Group

40

Additional Sources for Information

Tivoli Advisor, Issue 6, September 2005

ITM 6 Installation and Setup Guide, GC32-9407-00

IBM Redbook – Getting Started with IBM Tivoli Monitoring v6.1 on Distributed Environments, SG24-7143-00.

IBM Redbook - Deployment Guide Series: IBM Tivoli Monitoring 6.1, SG24-7188-00.

IBM Redbook - Certification Guide Series: IBM Tivoli Monitoring 6.1, SG24-7187-00.

IBM Tivoli Monitoring 6.1 On-Line Documentation:

http://publib.boulder.ibm.com/infocenter/tivihelp/v3r1/index.jsp?toc=/com.ibm.itm.doc/toc.xml

IBM Tivoli Monitoring Public Website:

http://www-306.ibm.com/software/tivoli/products/monitor/

IBM Software Group

41

Questions……