advancedzenosstraining

Copyright © 2009 Zenoss, Inc. All rights reserved.This work is licensed under a Creative Commons Attribution Share Alike 3.0 License. To view a copy of this license, visit

http://creativecommons.org/licenses/by-sa/3.0/; or send a letter to Creative Commons, 171 2nd Street, Suite 300, San Francisco, California, 94105, USA.

//

1

Zenoss Advanced Administration Training



//

2

Section 1: Preparing the Environment

What this section will cover:– Connectivity Preparations

– Managed Device Preparation

– Preparation Testing



//

3

Connectivity Preparation

From Zenoss Server to internet– Outbound http/s (80/443) to update zenoss packages– Outbound ssh (22) if zenoss interactive support is desired

Connectivity and Topology– Inbound from infrastructure to zenoss server

• Syslog (UDP 514), SNMP Traps (UDP 162)

– Outbound from zenoss server to infrastructure• ICMP echo request inbound echo reply• TCP ports for all monitored IPServices (smtp,http,etc)• DCOM & RPC = Dynamic Port allocation• SNMP (UDP 161) and SSH (TCP 22)



//

4

Managed Device Preparation

Unix Servers– SNMP agent with community and ACL Net-SNMP recommended– Syslog.conf that points back to Zenoss server (*.info @zenoss)– Optional public key configured with Zenoss server– Optional install of zenplugins for “external” monitoring

Network Devices (and other SNMP devices)– SNMP agent with community string and ACL– Optional SNMP trap that points back to Zenoss server

• For instance APC and other network devices

– Optional syslog that points back to Zenoss server• Recommended for Cisco devices



//

5

Managed Device Preparation

Windows Servers– Windows SNMP Agent installed with community and ACL– Login account for WMI access (Domain or Local Administrator)– Optional SNMP-Informant DLL (for CPU, Memory, Disk I/O)

• http://www.wtcs.org/informant/download.htm

Dell Servers– Dell Open Manage installed

HP/Compaq Servers– Insight Manager installed



//

6

Preparation Testing

Testing SNMP connections from Zenoss server– snmpwalk -v1 -c COMMUNITY HOST ip– Should return listing of IP addresses

Testing WMI from a windows System– Run wbemtest– Click “Connect…”– In Namespace field enter \\HOST\root\cimv2– Enter login information in “User” and “Password” fields– Click “Query…” button– Enter “select * from win32_service”– You should get a dialog with list of services on box



//

7

Section 2: Administration

What this section will cover:– Detailed Architecture (Review)

– Controlling Zenoss Daemons

– Minimal Zenoss for Debugging

– Backup and Restore

– Troubleshooting Daemons

– Escalating Issues and Contacting Support



//

8

Detailed Architecture



//

9

Controlling Zenoss Daemons

status - check the run state of a daemon

start - start daemon in background

stop - stop a running daemon

restart - reload a daemon’s configuration

run - run the daemon in the foreground for debugging

help - list all daemon options– Option list valid for configuration file



//

10

Minimal Zenoss for Debugging

ZEO

Zope

MySQL



//

11

Backup and Restore

ZEO

ZenRRD

MySQL

zenbackup

backup_19680101.tgz



//

12

Escalating Issues and Contacting Support

When escalating an issue please provide:– Steps to reproduce the problem

– Look for stack traces

– Gather version information (Zenoss, OS, etc)

– Escalate the Problem

• Email: [email protected]

• AIM: zenosssupport

• Phone: 410-990-0274



//

13

Administration Activities



//

14

Section 3: Inventory and Configuration

What this section will cover:– Modeling Daemons (Review)

– Modeling Process

– Controlling Modeling

– Plugin Selection

– Controlling SNMP Connections

– Inventory and Configuration Activities



//

15

Modeling Daemons (Review)

Unified Configuration

Model

ZenModel

ZEO

ZenModel

ZenDiscAuto-Discovery

ZenModelerDetailed Modeling

ZenWinModelerWindows Modeling

ICMP

SNMP/SSH/Telnet

WMI



//

16

Modeling Process

Database server

eth0

/dev/sda

mysql 3601

/dev/sdb

Web server

eth0

/

apache80

zope

switch1 2 3 4 5 6 7 8

ZenModel

ZenModeler

Collector Plug-ins

Transports

diskinterface service

SNMP SSH Telnet

8080

WMI Collect Data

Normalize DataTrack Changes



//

17

Controlling Modeling

Plugin Names– zenoss.snmp.InterfaceMap, zenoss.cmd.ifconfig_an– Stored in $ZENHOME/Products/DataCollector/plugins

Plugin Selection– zCollectorCollectPlugins - Regular expression that explicitly selects

plugins to be run. – zCollectorIgnorePlugins - Regular expression that excludes a

plugin– zTransportPreference - if two plugins perform the same function

which one should be used



//

18

Controlling SNMP Connections

zSnmpCommunities– List of SNMP Communities to try when first discovering a device

zSnmpCommunity– Community that will be used to connect to a device

zSnmpPort– Port to use when connecting to a device

zSnmpTimeout– Timeout in seconds before retransmit of an SNMP command. Should never

be over 15 secondszSnmpTries

– Number of times to retry a command before marking it as failed. On a poor quality link this value should be increased. High end is 4 - 5

zSnmpVer– SNMP version v1 or v2c. V2c must be used if 64-bit counters are being

collected. Also v2c allows for bulk walks which improves modeling performance significantly



//

19

Inventory and Configuration Activities



//

20

Section 4: TALES Expressions

What this section will cover:– What is a TALES expression

– Where are they Used

– Using zLinks

– Custom User Commands

– TALES Activities



//

21

What is a TALES Expression

A string substitution mechanismEvaluated in the context of an objectCommon variables

– here - object on which the expression is evaluated– evt - object pointing to the current event– dev - object pointing to the current device

Syntax for substitution– ${here/id} when evaluated on a device will be the device name

– ${here/getHWProductName} can call attribute or method

– ${here/cMyCustomField} or a custom defined field



//

22

Where are they Used

User Interface Commands– Evaluated in device context e.g. here == device

zLinks– Any HTML markup evaluated in device context

Performance Templates– Command field evaluated in device context

Event Commands– Evaluated in an event context with device– evt == the event – dev == the device



//

23

Using zLinks

zProperty like any other zProperty in /Devices

Uses TALES expressions

Any HTML text can be added

Used for many things– Create link to external management system– Insert graph on to front page of device



//

24

Custom User Commands

Commands that are run by user through web UI

Defined on any level of DMD hierarchy– Highest level in /Settings/Manage

Uses TALES expressions



//

25

TALES Activities



//

26

Section 5: Event Management

What this section will cover:– Event Architecture

– Begin / End Correlation (Review)

– Classification Process

– Event Class Mapping

– Event Commands

– SNMP Traps and Transformations

– Event Activities



//

27

Event Architecture

Events DBMySQLZenActions

Ajax Web ConsoleEvent List

SNMP Traps

•Email and Pager Alerts•Command Execution•Event Aging

ZenSyslog

ZenEventlog

Syslog

WMI

Process Events

ZenTrap



//

28

Begin/End Correlation

Ping DownStatus

History

Severity = Critical Class = /Status/PingDevice = build.zenoss.loc

Ping UpSeverity = ClearClass = /Status/PingDevice = build.zenoss.loc

Severity = Critical Class = /Status/PingDevice = build.zenoss.loc

Clear



//

29

Classification Process

New Event

Status

Parse Event

Insert Event

Find Class

Apply Context

Key + Regex or Expression

Severity = Critical

Tokenize Syslog



//

30

Event Class Mapping

Parsed Event eventClassKey

Class List

Transform

Class ListClass Maps

Regex or Test

Apply Context



//

31

Event Commands

Ping Down

Ping Up

echo “device 1.2.3.4 is down”

echo “device 1.2.3.4 is up”

Event Command

Where:class = /Status/Ping

Command:echo ${evt/summary}

Clear Command:echo ${evt/summary}



//

32

SNMP Traps and Transformations

PowerNet-MIB DEFINITIONS ::= BEGIN

apc OBJECT IDENTIFIER ::= { enterprises 318 }products OBJECT IDENTIFIER ::= { apc 1 }apcmgmt OBJECT IDENTIFIER ::= { apc 2 }

upsAdvBatteryCapacity OBJECT-TYPESYNTAX GaugeACCESS read-onlySTATUS mandatoryDESCRIPTION

"The remaining battery capacity expressed in percent of full capacity."

::= { upsAdvBattery 1 }

zenmibupsAdvBatteryCapacity -> .1.3.6.1.4.1.318.1.3.1.0upsAdvBatteryTemperature -> .1.3.6.1.4.1.318.1.6.1.0

.

.

.

DMD

OID Index

ZenTrapSNMP Trap

Res

olve

Classify & Transform Event Database

load



//

33

Event Monitoring Activities



//

34

Section 6: ZenCommands

What this section will cover:– ZenCommand Templates

– Running a ZenCommand

– ZenCommand API

– Using ZenPlugins



//

35

ZenCommand Templates

Data Source – Type = COMMAND– Component = unique name on device– Event Class = /Cmd/Fail– SSH if desired

Data Point– One per monitored output value– Make sure that data type is correct



//

36

Running ZenCommand

ZenCommand

SSH

/bin/dfRemote Box

/bin/df

exec

exec

avail:34554

avail:35654



//

37

ZenPlugins

ZenCommand Plugins for Remote Monitoring

Must be Installed on Remote Machine

Executed over SSH

Used for Devices – Outside the Firewall– Without proper SNMP Agent



//

38

ZenCommand Activities



//

39

Section 7: SNMP Performance Monitoring

What this section will cover:– What is SNMP

– Performance Daemons

– Template Binding (Review)

– Templates and Inheritance

– SNMP Templates

– Two SNMP Performance Collectors

– SNMP Performance Monitoring Activities



//

40

What is SNMP?

Simple Network Management Protocol– As in light weight not easy to understand :)

First Standard Management Protocol

Widely implemented on network devices

Servers implemented primarily at OS layer– Windows - Pretty OS support. Missing CPU, Memory, Disk I/O

– Net-SNMP - Good OS support. Varies on platform.

– Native OS agents vary widely.



//

41

SNMP Tree

A tree structure like DNS

Described by ‘.’ delimited number sequence

Each number in the OID has a corresponding name

Schema defined in Management Information Base (MIB)

Naming Conventions– RAW OID - .1.3.6.1.2.2.2.1.2– Full Name - iso.org.dod.internet.mib-2.interfaces.ifTable.ifEntry.ifDescr– Short hand - SNMPv2-MIB::ifDescr

Worthless numbers -> iso(1) org(3) dod(6) internet(1)



//

42

SNMP Single Values and Tables��

Single Values – OID points directly to a value– Usually, but not always, ends with .0– Zero must be added SNMPv2-MIB::sysDescr.0

Tables– OID points to a table or table column– OID ends with the row index– Row indexes are often single numbers– Row indexes are sometimes a series of numbers

• An IP Address - RFC1213-MIB::ipAdEntAddr.172.16.56.1• An ASCII String - SNMPv2-SMI::enterprises.9600.1.1.5.1.2.6.95.84.111.116.97.108



//

43

SNMP Tree

mib-2(1)

interfaces(2)system(1)

ifTable(2)sysDescr.0 = linux

ifDescr.1 = eth0

ifDescr.2 = eth1

sysUpTime.0 = 3 days

ifEntry(1)



//

44

Important Parts of the Tree

Standard MIB-2 – Things like IP Addresses, Interfaces, Routes, and Services

Host Resources– Things like Hardware, File Systems, Processes, and Software

Enterprises – Values defined by companies or other non-standard MIBs– Company numbers assigned by IANA

• http://www.iana.org/assignments/enterprise-numbers

– Common Numbers• IBM = 2, Cisco = 9, HP = 11, Dell = 674, Net-SNMP = 2021• SNMP-Informant = 9600, Zenoss = 14296



//

45

SNMP Commands

SNMP Get– Pull a single value from an agent– Ends with index which often 0 for single item or row for table

SNMP Walk– Steps through the values published by an agent– Used to investigate or trouble shoot agent

SNMP Trap– Message sent to management station– Has unique OID based identifier– Has a series of key value pares



//

46

SNMP Command Examples

Walk basic system MIB– snmpwalk -v1 -cpublic build.zenoss.loc system

Walk interface descriptions– snmpwalk -v1 -cpublic build.zenoss.loc ifDescr

Get a single value– snmpget -v1 -cpublic build.zenoss.loc ifDescr.2

Detailed description of OID value– snmptranslate -Td RFC1213-MIB::ifDescr

Convert name to raw OID– snmptranslate -On RFC1213-MIB::ifDescr

Convert raw OID to short name– snmptranslate -OS .1.3.6.1.2.1.2.2.1.2



//

47

Performance Daemons

Unified Configuration

Model

ZenModel

RRDTool

ZenRRD

ZenPerfSNMPSNMP Collection

ZenCommandNagios/Cacti

ZenPerfXMLRPCXML-RPC Collection

SNMP

Exec / SSH

XML-RPC



//

48

Template Binding (Review)

Device: build

FileSystem: /

ethernetCsmacd: eth0

Device

FileSystem

Templates

ethernetCsmacd



//

49

Templates and Inheritance

/Server/Linux

build.zenoss.loc

/Server/Linux/Remote

dev.zenoss.org

/DevicesFileSystem

•SNMP usedBlocks

Device•SNMP sysUpTime

•SNMP sysUpTime•SNMP ssCpuRawSystem•SNMP memAvailReal

Device

•SSH uptime_sysUpTime•SSH cpu_ssCpuRawSystem•SSH mem_memAvailReal

DeviceFileSystem

•SSH disk_usedBlocks

tilde.zenoss.loc

www.zenoss.com



//

50

Performance Templates

Data Source– How to collect information: SNMP, COMMAND, XMLRPC

Data Point– Parse Data Source output and assign type information– Custom rrd creation command allowed

Threshold– Min / Max applied to Data Points– Threshold Event Information

Graph– Combine Data Points into a graph– Many graph attributes can be selected– If default format doesn’t work custom RRD command can be used



//

51

Two Performance Monitors

gate.zenoss.loc

win2003.zenoss.loc

tilde.zenoss.loc

localhostZenPerfSNMP

localhost

build.zenoss.loc

esx.zenoss.loc

dev.zenoss.loc

remote1ZenPerfSNMP

localhost2

zenperfsnmp.conf

zenperfsnmp2.conf

bin/zenperfsnmp2

bin/zenperfsnmp

DMD



//

52

SNMP Monitoring Activities



//

53

Section 8: Programming Zenoss

What this section will cover:– Customizing the User Interface

– External APIs

– Device API

– Using ZenDMD

– Creating ZenCommands

– Custom Reports



//

54

Customizing the User Interface

Zenoss uses Zope Skins

Zenoss pages are build using ZPT

Skins are defined in file system– Can be viewed in /zport/portal_skins/manage

Any Zenoss ZPT file can be overridden– Find file in portal_skins– Click Custom button– Navigate to custom folder and make modifications

Images can be overridden in same way



//

55

External Zenoss API

Zope publishes objects and their methods

Objects are navigated to by paths like a file system– /zport/dmd/Devices/build.zenoss.loc

Methods called on objects– /zport/dmd/Devices/build.zenoss.loc/setLocation

Parameters are URL arguments– …/build.zenoss.loc/setLocation?newlocation=/Maryland

Methods can also be called using XML-RPC



//

56

Device API

Device Add– /zport/dmd/DeviceLoader?deviceName=‘build.zenoss.loc’&devicePath=/Server

Device Edit– Need to have the path to the device– /zport/dmd/Devices/Server/devices/build.zenoss.loc?setLocation=/Maryland

Device Delete– /zport/dmd/Devices/Server/devices/build.zenoss.loc/deleteDevice



//

57

Using ZenDMD

DMD$ zendmd

Welcome to zenoss dmd command shell!use zhelp() to list commands>>> d = find(‘build.zenoss.loc”)>>> d.os.interfaces.objectIds()['eth0', 'eth1', 'lo', 'sit0', 'vmnet1', 'vmnet8']>>> for d in dmd.Devices.getSubDevices():>>> print d.id, d.getManageIp()



//

58

Report Plugins

Build any report possible with information in DMD

Consists of two parts– Python code to collect information– ZPT to display information

Debug collection code using zendmd



//

59

Creating ZenCommands

ZenCommands implement Nagios and Cacti APIs

Input are command line arguments

Output is exit code and one line of standard out

Standard output line– Nagios = message|key=value[UOM];[warn];[crit];[min];[max]– Cacti = key:value key2:value

Full API defined herehttp://nagiosplug.sourceforge.net/developer-guidelines.html



//

60

Programming Activities

advancedzenosstraining

Documents

zenoss daemons minimal

zenoss daemonsstatus

zenoss server syslog

zenoss interactive support

zenoss server snmpwalk

nd street

san francisco

zenoss optional public